Skip to contentSkip to conversation

btrfs check (not lowmem) and OOM-like hangs (4.17.6)

DISCUSSION Marc MERLIN 11 participants 455 messages Jul 17, 2018 – Jun 16, 2022 6 messages not in the archive Btrfs file system Page 2 of 3

lore
Share & tools
lore
#60 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 38 lines · Marc MERLIN
Quoted 7 lines · Zygo Blaxell

Based on the history, I'd expect the filesystem is missing some number of tree nodes, from a few dozen to thousands, depending on how many writes were dropped after the 2nd drive failure before it was detected. Since the array was also degraded at that time, with 4 drives in raid5, there's 3 data drives, and if one of them was offline then we'd have a 2/3 success rate reading metadata blocks and 1/3 garbage. That's definitely in the "we need to write new software to recover from this" territory.

I think your conclusion is correct. I'm very dismayed that the filesystem didn't go read only right away.
Hell, the mdadm block device should have done read only as soon as it lost more than one drive.
Why were any writes allowed once more than one drive was missing?

Let's look at this for a second:

Mar 28 02:28:11 gargamel kernel: [1512988.446844] sd 6:1:8:0: Device offlined - not ready after error recovery
Mar 28 02:28:11 gargamel kernel: [1512988.475270] sd 6:1:8:0: rejecting I/O to offline device
Mar 28 02:28:11 gargamel kernel: [1512988.491531] blk_update_request: I/O error, dev sdi, sector 261928312 op 0x0:(READ) flags 0x84700
 phys_seg 42 prio class 0
Mar 28 02:28:11 gargamel kernel: [1512988.525073] blk_update_request: I/O error, dev sdi, sector 261928824 op 0x0:(READ) flags 0x80700
 phys_seg 5 prio class 0
Mar 28 02:28:12 gargamel kernel: [1512988.579667] blk_update_request: I/O error, dev sdi, sector 261927936 op 0x0:(READ) flags 0x80700
 phys_seg 47 prio class 0
(..)
Mar 28 02:28:12 gargamel kernel: [1512988.615910] md: super_written gets error=10
Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Disk failure on sdi1, disabling device.
Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Operation continuing on 4 devices.
Mar 28 02:28:21 gargamel kernel: [1512998.170192] usb 2-1.6-port1: disabled by hub (EMI?), re-enabling...
Mar 28 02:28:21 gargamel kernel: [1512998.240404] print_req_error: 134 callbacks suppressed
Mar 28 02:28:21 gargamel kernel: [1512998.240406] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x807
00 phys_seg 1 prio class 0
Mar 28 02:28:21 gargamel kernel: [1512998.243415] ftdi_sio 2-1.6.1:1.0: device disconnected
Mar 28 02:28:21 gargamel kernel: [1512998.341221] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x0 p
(...)
Mar 28 02:28:22 gargamel kernel: [1512998.716351] blk_update_request: I/O error, dev sdi, sector 2058 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar 28 02:28:22 gargamel kernel: [1512998.716362] blk_update_request: I/O error, dev sdi, sector 2059 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Quoted 8 lines

Ok, one drive died, but raid5 continues degraded. md7 : active raid5 sdi1[5](F) sdo1[7] sdg1[6] sdj1[3] sdh1[1]

      23441561600 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
      bitmap: 0/44 pages [0KB], 65536KB chunk

not sure what these were:

Mar 29 00:00:08 gargamel kernel: [1590505.415665] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore
Mar 29 00:00:09 gargamel kernel: [1590505.866094] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore

Readahead can fail for a number of reasons that aren't real problems, e.g. if it's necessary to swap or evict pages to do the readahead, the kernel doesn't bother, and fails the readahead operation instead. Another common case is lower layers of block IO stack, where a readahead request can't be satisfied because of some geometry constraint specific to one device in a multi-device LV. e.g. a drive is no longer present so it can't receive block IO requests.

Normally readahead errors are ignored. If there's a real problem, the application will issue a normal read later on and the error will be reported then. Or the readahead might have been entirely speculative, but the app never tries to read the data at all.

Quoted 73 lines

9H later a 2nd drive dies just when I'm replacing the failed one:

Mar 29 09:30:12 gargamel kernel: [1624709.301830] sd 6:1:5:0: [sdh] tag#523 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s
Mar 29 09:30:12 gargamel kernel: [1624709.331812] sd 6:1:5:0: [sdh] tag#523 CDB: Read(16) 88 00 00 00 00 00 00 26 3f d0 00 00 00 18 00 00
Mar 29 09:30:12 gargamel kernel: [1624709.359459] blk_update_request: I/O error, dev sdh, sector 2506704 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
Mar 29 09:30:12 gargamel kernel: [1624709.359465] md/raid:md7: read error not correctable (sector 2504656 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.359471] md/raid:md7: read error not correctable (sector 2504664 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.359472] md/raid:md7: read error not correctable (sector 2504672 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.359486] md/raid:md7: read error not correctable (sector 2504656 on sdh1).
Mar 29 09:30:12 gargamel kernel: [1624709.455886] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.681637] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.695785] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.717624] md/raid:md7: read error not correctable (sector 2504664 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.739746] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.757206] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.770348] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.790298] md/raid:md7: read error not correctable (sector 2504672 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.812546] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.825856] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.839006] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.866656] md/raid:md7: read error not correctable (sector 24815552 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.898796] md/raid:md7: read error not correctable (sector 24815552 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624709.921135] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.934315] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.947500] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624709.988589] md/raid:md7: read error not correctable (sector 1763985936 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624710.036204] md/raid:md7: read error not correctable (sector 1763985936 on sdh1).
Mar 29 09:30:13 gargamel kernel: [1624710.059121] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624710.088858] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624710.102026] md: super_written gets error=10
Mar 29 09:30:13 gargamel kernel: [1624710.158830] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.096055] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:37 gargamel kernel: [1625094.122910] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
Mar 29 09:36:37 gargamel kernel: [1625094.153249] md/raid:md7: read error not correctable (sector 6562801616 on sdh1).
Mar 29 09:36:37 gargamel kernel: [1625094.176011] md/raid:md7: read error not correctable (sector 6562801624 on sdh1).
Mar 29 09:36:37 gargamel kernel: [1625094.223351] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.250628] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.263726] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.276989] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.290121] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.303267] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.325084] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.342083] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.355206] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.368394] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.383304] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.396423] md: super_written gets error=10
Mar 29 09:36:37 gargamel kernel: [1625094.409498] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:37 gargamel kernel: [1625094.436355] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Mar 29 09:36:37 gargamel kernel: [1625094.466729] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:37 gargamel kernel: [1625094.493600] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
Mar 29 09:36:37 gargamel kernel: [1625094.523998] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:36:38 gargamel kernel: [1625094.550938] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 4, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.066422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.093309] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 5, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.124768] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.151651] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 6, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.182803] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.209677] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 7, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.239972] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.266862] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 8, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.297234] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.324094] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 9, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.354422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.381286] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 10, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.411926] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.438770] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 11, flush 0, corrupt 0, gen 0
Mar 29 09:37:34 gargamel kernel: [1625151.469361] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:34 gargamel kernel: [1625151.496269] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 12, flush 0, corrupt 0, gen 0
Mar 29 09:37:35 gargamel kernel: [1625151.527455] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:35 gargamel kernel: [1625151.554360] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 13, flush 0, corrupt 0, gen 0
Mar 29 09:37:35 gargamel kernel: [1625151.584963] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable
Mar 29 09:37:35 gargamel kernel: [1625151.611842] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 14, flush 0, corrupt 0, gen 0
Quoted 5 lines

24 seconds are lost before btrfs notices anything, and then it seems to continue without going read only for another full minute before things stopped.

Why would the raid5 not go read only immediately after 2 drives are lost?

Once a degraded array has an IO failure, mdadm assumes you're in a data-scraping recovery mode and simply passes errors through. bcache does something similar when the backing store fails.

btrfs is the agent that has to stop attempting writes, and make sure any transaction in progress doesn't get committed. ext4 has similar responsibility and implements its own force-read-only feature.

I note that btrfs is reporting only read errors here.

There's a possibility that the drive dropped the writes during the bus reset at the start of the second drive failure. If write caching was enabled in the drive, and the drive has a firmware bug that drops the write cache contents (or it's just failing hardware, i.e. the CPU running the drive firmware is getting killed by electrical problems on the controller board, causing both the bus drop and the loss of write cache contents), then writes in the drive's cache could be lost _after_ mdadm, bcache and btrfs had been told by the drive that they were completed.

If the failing drive also reorders cached writes across flush commands, then we go directly to parent transid verify failed in btrfs. Parent transid verification is designed to detect this exact firmware failure mode, and it usually works as intended. It's a pretty direct and reliable signal that write ordering is broken in a lower level of the storage stack, and must be fixed or disabled before trying again.

Even if a single drive doesn't reorder writes, multiple drives in raid effectively reorder writes between drives as each drive has its own distinct write queue. A dropped write that would be harmless in a single-device filesystem could be harmful in a multi-device array as the non-failing drives will have writes that occurred after the lost writes on the failing drive. Normally mdadm enforces flush ordering across all component devices so that this isn't a problem, but if you have optimistic firmware and a drive failure after the flush command returns success, the inter-drive ordering enforcement fails and the result is the same as if an individual drive had a write/flush reordering bug.

Firmware bugs are fairly rare across drive models, but failing drive models are vastly overrepresented in the field. I've tested hundreds of firmware revisions and only about 4% fail with bad write reordering. The few firmwares that do fail are found in popular low-end consumer drives and cheap enterprise drives (they're the same drive underneath, just a different label and price), so there are a lot of bad drives in the field. Once we identify a bad firmware revision, we avoid buying any more of it, disable write caching in the firmware on the drives we already have, and try to find a workload for the drive that is less sensitive to failure (or performance--fsync latency goes up when the drive can't use its cache for writes).

Some years ago we did a fleetwide audit of bad firmware drives and found about a third of our drives were bad. We disabled write cache on all affected drives and prioritized replacement of the affected models. This stopped the worst btrfs problems we had been having up to that point. It also stopped a lot of strange one-off application data corruptions we had been chasing for years before on ext4/mdadm arrays and in postgresql databases. We'd assumed those were some kind of application bug, but we never found an app bug, and the problem disappeared once we turned off the write cache.

Quoted 13 lines
Quoted 4 lines

If we don't have any intact subvol trees (or the subvol trees we really want aren't intact), then we can't recover this way. Instead we'd have to scrape the disk looking for metadata leaf nodes, and try to reinsert those into a new tree structure. The trick here is that we'll have the

Quoted 8 lines

duplicated and inconsitent nodes and we won't have some nodes at all, and we'll have to make sense of those (or pass it to the existing btrfs check and hope it can cope with them). I'm guessing that a simplified version of this is what Josef is building at this point, or will be building soon if we aren't extremely lucky and find an intact subvol tree. After building an intact subvol tree (even with a few garbage items in it as long as check can handle them) we can go back to the --init-extent-tree step and rebuild the rest of the filesystem.

Quoted 4 lines

I see. In that case, I'm still happy to help, to help improve the tools, but if I'm looking at some amount of non trivial loss/corruption, at soe point I'll go back to backups, since they'll be more intact than this now damaged filesystem.

Yeah, experimental recovery code is fun, but robust backups and a working disaster recovery plan is usually better. Even if the filesystem is up and running again, I'd want to compare all the files against backups because I'd trust nothing on a filesystem after fsck touched it.

On the other hand, if you get lucky and the filesystem isn't too badly damaged, then comparing the data with backups will be more convenient than starting over with a new filesystem.

Quoted 6 lines

Thanks for the detailed answers.
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#426 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?

Quoted 7 lines
Quoted 2 lines

Here is regular check:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a

Woo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,

took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?

Should I run

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
now?

output:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outm
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55cf0ce435f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55cf0ce435f0
fixed slot 0
fixed slot 2
fixed slot 5
fixed slot 7
fixed slot 12
fixed slot 27
fixed slot 28
fixed slot 32
fixed slot 34
fixed slot 35
fixed slot 36
fixed slot 45
fixed slot 48
fixed slot 49
fixed slot 50
fixed slot 52
fixed slot 53
fixed slot 54
fixed slot 55
fixed slot 56
fixed slot 60
fixed slot 61
fixed slot 63
fixed slot 76
fixed slot 77
fixed slot 85
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 98
fixed slot 99
fixed slot 100
fixed slot 103
fixed slot 104
fixed slot 106
fixed slot 107
fixed slot 108
fixed slot 109
fixed slot 110
fixed slot 111
fixed slot 112
fixed slot 114
fixed slot 120
fixed slot 135
fixed slot 140
fixed slot 141
fixed slot 142
fixed slot 146
fixed slot 155
fixed slot 156
fixed slot 158
fixed slot 159
fixed slot 160
fixed slot 161
fixed slot 165
fixed slot 166
fixed slot 171
fixed slot 173
fixed slot 174
fixed slot 180
fixed slot 181
fixed slot 182
fixed slot 183
fixed slot 184
fixed slot 185
fixed slot 186
fixed slot 212
fixed slot 213
fixed slot 221
fixed slot 222
fixed slot 224
fixed slot 226
fixed slot 227
fixed slot 228
fixed slot 229
fixed slot 231
fixed slot 232
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 244
fixed slot 245
fixed slot 248
fixed slot 252
fixed slot 259
fixed slot 260
Couldn't find a replacement block for slot 261
fixed slot 263
fixed slot 264
fixed slot 265
fixed slot 267
fixed slot 269
fixed slot 271
fixed slot 273
fixed slot 274
fixed slot 276
fixed slot 277
fixed slot 278
fixed slot 279
fixed slot 282
fixed slot 283
fixed slot 285
fixed slot 286
fixed slot 287
fixed slot 288
fixed slot 289
fixed slot 291
fixed slot 293
fixed slot 294
fixed slot 296
fixed slot 297
fixed slot 298
fixed slot 300
fixed slot 301
fixed slot 302
fixed slot 303
fixed slot 304
fixed slot 305
fixed slot 306
fixed slot 307
fixed slot 308
fixed slot 309
fixed slot 311
fixed slot 312
fixed slot 313
fixed slot 314
fixed slot 315
fixed slot 316
fixed slot 317
fixed slot 318
fixed slot 319
fixed slot 320
fixed slot 321
gargamel:/var/local/src/btrfs-progs-josefbacik#
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#427 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 20 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 6 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks,

Done, so now I can run
btrfs check --repair /dev/mapper/dshelf1a ?

Quoted 7 lines
Quoted 2 lines

Here is regular check:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a

Woo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks,

Quoted 6 lines

took a few hours to run, but it's done now. Looks like it fixed things a 2nd time?

Should I run

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
now?

Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,

Josef

#61 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Zygo Blaxell
Quoted 1 line · Zygo Blaxell

Readahead can fail for a number of reasons that aren't real problems,

thanks.

Quoted 3 lines

Once a degraded array has an IO failure, mdadm assumes you're in a data-scraping recovery mode and simply passes errors through. bcache does something similar when the backing store fails.

shouldn't it go read only also?
I haven't found a setting to tell it to do that if it's not the default.

Quoted 3 lines

btrfs is the agent that has to stop attempting writes, and make sure any transaction in progress doesn't get committed. ext4 has similar responsibility and implements its own force-read-only feature.

Agreed, but I like the defense in multiple layers approach mdadm knows that any data going to be written is going to be incomplete due to the 2nd missing drive, and there are few to no scenarios where continuing to write is a good thing.

Quoted 1 line

I note that btrfs is reporting only read errors here.

Yeah, me too. That seems very bad and likely the cause or the corruption I got.
A double raid failure (I've had them before) should be almost a non event. The first write after that should fail and that should be the end of it.
This underlying problem makes me very uncomfortable.
As much as I'm super thankful for Josef's efforts, and ultimately it will make btrfs recovery better, which is a win for all, my situation shouldn't have happened from what I can tell.

Quoted 8 lines

There's a possibility that the drive dropped the writes during the bus reset at the start of the second drive failure. If write caching was enabled in the drive, and the drive has a firmware bug that drops the write cache contents (or it's just failing hardware, i.e. the CPU running the drive firmware is getting killed by electrical problems on the controller board, causing both the bus drop and the loss of write cache contents), then writes in the drive's cache could be lost _after_ mdadm, bcache and btrfs had been told by the drive that they were completed.

That's true, but I've seen btfrs remount read only before, and it didn't there. Shouldn't hard IO errors immediately cause btrfs to go read only?

Write caching is on all those drives though:

gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh
/dev/sdh:
 multcount     = 16 (on)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 729601/255/63, sectors = 11721045168, start = 0
 write-caching =  1 (on)

I haven't heard that these drives have broken caching, but maybe they do?

Device Model:     ST6000VN0041-2EL11C
Serial Number:    ZA18TVFZ
LU WWN Device Id: 5 000c50 0a4d9b49c
Firmware Version: SC61
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Quoted 6 lines

If the failing drive also reorders cached writes across flush commands, then we go directly to parent transid verify failed in btrfs. Parent transid verification is designed to detect this exact firmware failure mode, and it usually works as intended. It's a pretty direct and reliable signal that write ordering is broken in a lower level of the storage stack, and must be fixed or disabled before trying again.

Agreed, I do want to fix the underlying problem here. What I can do is

1) disable write caching on the drives

2) disable bcache by removing the caching device

3) change the cache mode

gargamel:/sys/block/md7/bcache# cat cache_mode 
[writethrough] writeback writearound none
writethrough should be safe, but I could use writearound instead

4) if I end up wiping my device, I can just remove the bcache layer altogether.

Quoted 3 lines

Even if a single drive doesn't reorder writes, multiple drives in raid effectively reorder writes between drives as each drive has its own distinct write queue. A dropped write that would be harmless in a

very good point.

Quoted 7 lines

single-device filesystem could be harmful in a multi-device array as the non-failing drives will have writes that occurred after the lost writes on the failing drive. Normally mdadm enforces flush ordering across all component devices so that this isn't a problem, but if you have optimistic firmware and a drive failure after the flush command returns success, the inter-drive ordering enforcement fails and the result is the same as if an individual drive had a write/flush reordering bug.

That's all fair, but it feels like FS kept writing way longer than it was supposed to and that is what worries me the most.

Quoted 2 lines

Some years ago we did a fleetwide audit of bad firmware drives and found about a third of our drives were bad. We disabled write cache on

Looks like I just want to disable write caching then. Correctness beats speed for sure.

Quoted 4 lines

Yeah, experimental recovery code is fun, but robust backups and a working disaster recovery plan is usually better. Even if the filesystem is up and running again, I'd want to compare all the files against backups because I'd trust nothing on a filesystem after fsck touched it.

That is totally my plan. Given the output that I'm seeing, I'll definitely do at least a diff of all files between the backup and the array being recovered. I might even do an rsync, forcing md5 checksums, but if I do that, it will take days and longer than restoring the backup.

Quoted 3 lines

On the other hand, if you get lucky and the filesystem isn't too badly damaged, then comparing the data with backups will be more convenient than starting over with a new filesystem.

Agreed.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#428 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 3 lines · Josef Bacik

Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,

Sounds good. Is mode=lowmem a thing of the past by the way? (I remember that the old regular repair would eat up my 32GB of RAM and take the machine down)

The current one failed quickly though

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55e2e75defc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x55e2e75defc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#62 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 11 lines · Marc MERLIN
Quoted 1 line · Zygo Blaxell

Readahead can fail for a number of reasons that aren't real problems,

thanks.

Quoted 3 lines

Once a degraded array has an IO failure, mdadm assumes you're in a data-scraping recovery mode and simply passes errors through. bcache does something similar when the backing store fails.

shouldn't it go read only also?
I haven't found a setting to tell it to do that if it's not the default.

bcache in writethrough mode could leave cached blocks dirty as long as the SSD completes the write. It should be reporting the write errors back to upper layers.

Whether it actually does...I don't know, I haven't run that kind of stress test on bcache. I usually pair bcache SSD 1:1 with HDD, so if either the SSD or HDD fails, I scrub or replace them both as a single logical btrfs device, which makes this kind of question mostly irrelevant. Unfortunately this approach doesn't work with raid5--the striping cuts IO requests into tiny fragments, so the cache will try to cache everything.

Quoted 8 lines
Quoted 3 lines

btrfs is the agent that has to stop attempting writes, and make sure any transaction in progress doesn't get committed. ext4 has similar responsibility and implements its own force-read-only feature.

Agreed, but I like the defense in multiple layers approach mdadm knows that any data going to be written is going to be incomplete due to the 2nd missing drive, and there are few to no scenarios where continuing to write is a good thing.

True, there's no way the parity would make any sense with two drives in a stripe missing during an update. So maybe there weren't any writes going through mdadm at that point, and everything failed at the drive-offline event. Or writes completed on the working drives before the failing drive's failure was detected, messing up the data there. A typical drive bus timeout is a long time, and a lot of raid stripes could be modified on other drives while that happens.

Quoted 11 lines
Quoted 8 lines

There's a possibility that the drive dropped the writes during the bus reset at the start of the second drive failure. If write caching was enabled in the drive, and the drive has a firmware bug that drops the write cache contents (or it's just failing hardware, i.e. the CPU running the drive firmware is getting killed by electrical problems on the controller board, causing both the bus drop and the loss of write cache contents), then writes in the drive's cache could be lost _after_ mdadm, bcache and btrfs had been told by the drive that they were completed.

That's true, but I've seen btfrs remount read only before, and it didn't there. Shouldn't hard IO errors immediately cause btrfs to go read only?

No, only hard write IO errors on all metadata mirror drives, and cases where btrfs needs to CoW a page or read free space tree, and can't find an intact mirror. Anything less is correctable (write failure on some mirrors) or can be retried (any read failure) if the raid profile has redundancy.

Quoted 9 lines

Write caching is on all those drives though:

gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh
/dev/sdh:
 multcount     = 16 (on)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 729601/255/63, sectors = 11721045168, start = 0
 write-caching =  1 (on)
Quoted 6 lines

I haven't heard that these drives have broken caching, but maybe they do?

Device Model:     ST6000VN0041-2EL11C
Serial Number:    ZA18TVFZ
LU WWN Device Id: 5 000c50 0a4d9b49c
Firmware Version: SC61
User Capacity:    6,001,175,126,016 bytes [6.00 TB]

I've heard of problems with SC60, but IIRC SC61 was supposed to be the fix for them. I had some SC60 drives for a while without encountering issues, but I didn't have any bus timeouts with them. When they failed, they had a burst of 10k+ UNC sectors and then stopped spinning up, with no bus timeouts.

Quoted 35 lines
Quoted 6 lines

If the failing drive also reorders cached writes across flush commands, then we go directly to parent transid verify failed in btrfs. Parent transid verification is designed to detect this exact firmware failure mode, and it usually works as intended. It's a pretty direct and reliable signal that write ordering is broken in a lower level of the storage stack, and must be fixed or disabled before trying again.

Agreed, I do want to fix the underlying problem here. What I can do is

1) disable write caching on the drives

2) disable bcache by removing the caching device

3) change the cache mode

gargamel:/sys/block/md7/bcache# cat cache_mode 
[writethrough] writeback writearound none
writethrough should be safe, but I could use writearound instead

4) if I end up wiping my device, I can just remove the bcache layer altogether.

Quoted 3 lines

Even if a single drive doesn't reorder writes, multiple drives in raid effectively reorder writes between drives as each drive has its own distinct write queue. A dropped write that would be harmless in a

very good point.

Quoted 7 lines

single-device filesystem could be harmful in a multi-device array as the non-failing drives will have writes that occurred after the lost writes on the failing drive. Normally mdadm enforces flush ordering across all component devices so that this isn't a problem, but if you have optimistic firmware and a drive failure after the flush command returns success, the inter-drive ordering enforcement fails and the result is the same as if an individual drive had a write/flush reordering bug.

Quoted 2 lines

That's all fair, but it feels like FS kept writing way longer than it was supposed to and that is what worries me the most.

The log excerpt I saw didn't show any write errors from btrfs, only from mdadm. That suggests to me that the failure happened earlier when it was already too late to respond. Also we can't rule out that bcache is doing something stupid.

Quoted 29 lines
Quoted 2 lines

Some years ago we did a fleetwide audit of bad firmware drives and found about a third of our drives were bad. We disabled write cache on

Looks like I just want to disable write caching then. Correctness beats speed for sure.

Quoted 4 lines

Yeah, experimental recovery code is fun, but robust backups and a working disaster recovery plan is usually better. Even if the filesystem is up and running again, I'd want to compare all the files against backups because I'd trust nothing on a filesystem after fsck touched it.

That is totally my plan. Given the output that I'm seeing, I'll definitely do at least a diff of all files between the backup and the array being recovered. I might even do an rsync, forcing md5 checksums, but if I do that, it will take days and longer than restoring the backup.

Quoted 3 lines

On the other hand, if you get lucky and the filesystem isn't too badly damaged, then comparing the data with backups will be more convenient than starting over with a new filesystem.

Agreed.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#429 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 44 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks,

Sounds good. Is mode=lowmem a thing of the past by the way? (I remember that the old regular repair would eat up my 32GB of RAM and take the machine down)

The current one failed quickly though

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55e2e75defc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x55e2e75defc0

Quoted 9 lines

Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8
ERROR: failed to repair root items: Input/output error

Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,

Josef

#430 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Josef Bacik

Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,

Thanks for sticking with me all this time.
So, it was pretty quick before it failed:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x556ddb2fffc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x556ddb2fffc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#431 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik

Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks,

Quoted 3 lines

Thanks for sticking with me all this time.
So, it was pretty quick before it failed:

Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,

Josef

#64 lore raw

Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption

In reply to Marc MERLIN

This is an interesting discussion, so let's make a new thread out of it. TL;DR: I think btrfs may have failed to go read only earlier, causing more damage than needed to be, or some block layers just held enough data in flight that partial data got written, causing more damage than expected.
Figuring out the underlying problem would be good to avoid this again.

Quoted 1 line · Zygo Blaxell

Readahead can fail for a number of reasons that aren't real problems,

thanks.

Quoted 3 lines

Once a degraded array has an IO failure, mdadm assumes you're in a data-scraping recovery mode and simply passes errors through. bcache does something similar when the backing store fails.

shouldn't it go read only also?
I haven't found a setting to tell it to do that if it's not the default.

Quoted 3 lines

btrfs is the agent that has to stop attempting writes, and make sure any transaction in progress doesn't get committed. ext4 has similar responsibility and implements its own force-read-only feature.

Agreed, but I like the defense in multiple layers approach mdadm knows that any data going to be written is going to be incomplete due to the 2nd missing drive, and there are few to no scenarios where continuing to write is a good thing.

Quoted 1 line

I note that btrfs is reporting only read errors here.

Yeah, me too. That seems very bad and likely the cause or the corruption I got.
A double raid failure (I've had them before) should be almost a non event. The first write after that should fail and that should be the end of it.
This underlying problem makes me very uncomfortable.
As much as I'm super thankful for Josef's efforts, and ultimately it will make btrfs recovery better, which is a win for all, my situation shouldn't have happened from what I can tell.

Quoted 8 lines

There's a possibility that the drive dropped the writes during the bus reset at the start of the second drive failure. If write caching was enabled in the drive, and the drive has a firmware bug that drops the write cache contents (or it's just failing hardware, i.e. the CPU running the drive firmware is getting killed by electrical problems on the controller board, causing both the bus drop and the loss of write cache contents), then writes in the drive's cache could be lost _after_ mdadm, bcache and btrfs had been told by the drive that they were completed.

That's true, but I've seen btfrs remount read only before, and it didn't there. Shouldn't hard IO errors immediately cause btrfs to go read only?

Write caching is on all those drives though:

gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh
/dev/sdh:
 multcount     = 16 (on)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 729601/255/63, sectors = 11721045168, start = 0
 write-caching =  1 (on)

I haven't heard that these drives have broken caching, but maybe they do?

Device Model:     ST6000VN0041-2EL11C
Serial Number:    ZA18TVFZ
LU WWN Device Id: 5 000c50 0a4d9b49c
Firmware Version: SC61
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Quoted 6 lines

If the failing drive also reorders cached writes across flush commands, then we go directly to parent transid verify failed in btrfs. Parent transid verification is designed to detect this exact firmware failure mode, and it usually works as intended. It's a pretty direct and reliable signal that write ordering is broken in a lower level of the storage stack, and must be fixed or disabled before trying again.

Agreed, I do want to fix the underlying problem here. What I can do is

1) disable write caching on the drives

2) disable bcache by removing the caching device

3) change the cache mode

gargamel:/sys/block/md7/bcache# cat cache_mode 
[writethrough] writeback writearound none
writethrough should be safe, but I could use writearound instead

4) if I end up wiping my device, I can just remove the bcache layer altogether.

Quoted 3 lines

Even if a single drive doesn't reorder writes, multiple drives in raid effectively reorder writes between drives as each drive has its own distinct write queue. A dropped write that would be harmless in a

very good point.

Quoted 7 lines

single-device filesystem could be harmful in a multi-device array as the non-failing drives will have writes that occurred after the lost writes on the failing drive. Normally mdadm enforces flush ordering across all component devices so that this isn't a problem, but if you have optimistic firmware and a drive failure after the flush command returns success, the inter-drive ordering enforcement fails and the result is the same as if an individual drive had a write/flush reordering bug.

That's all fair, but it feels like FS kept writing way longer than it was supposed to and that is what worries me the most.

Quoted 2 lines

Some years ago we did a fleetwide audit of bad firmware drives and found about a third of our drives were bad. We disabled write cache on

Looks like I just want to disable write caching then. Correctness beats speed for sure.

Quoted 4 lines

Yeah, experimental recovery code is fun, but robust backups and a working disaster recovery plan is usually better. Even if the filesystem is up and running again, I'd want to compare all the files against backups because I'd trust nothing on a filesystem after fsck touched it.

That is totally my plan. Given the output that I'm seeing, I'll definitely do at least a diff of all files between the backup and the array being recovered. I might even do an rsync, forcing md5 checksums, but if I do that, it will take days and longer than restoring the backup.

Quoted 3 lines

On the other hand, if you get lucky and the filesystem isn't too badly damaged, then comparing the data with backups will be more convenient than starting over with a new filesystem.

Agreed.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#63 lore raw

Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption

In reply to Zygo Blaxell

(Zygo, answer for you below)

Quoted 6 lines · Josef Bacik

We can't do anything about the disks lying to us. If a disk has a wonky FUA/FLUSH implementation then we're just sort of screwed. Unfortunately because our metadata moves around a lot we're waaaaay more susceptible to this failure case than ext4 or xfs, their metadata is relatively static they can put humpty dumpty back together again relatively simply.

Yeah, that's clearly been my experience. The average end user (which kind of includes me), totally understands that last data written, includig potentially directories, will get damaged, potentially beyond repair, but losing the entire filesystem when 99.99%+ never got touched by the last operations, is unexpected and undesired.
That said, yes, I understand that btrfs does things that are both cool and complex, and they are less tolerant of corruption.

Quoted 2 lines

2. Put a lot more effort into disaster recovery. What I've written for you is an idea I've had in my head for a while. Some of this

I truly appreciate the time you've been spending with me on this. In the end, I do have a backup, but I could not have one, or it could fail when I'm reading back from it, so being able to recover the filesystem to some workable state, is definitely a big plus. Honestly I don't even really care if I lose 0.1%, 1%, or 5% of the data (even if probably only <0.001% of the data got touched in the minute+ that btrfs was still apparently trying to use the filesystem after the double failure), but a recovery that takes at most hours (for 24TB) and not days, or never (never is a multiple day restore over the network for me), will definitely be a great improvement.

Quoted 6 lines

3. Test these btrfs+dmcrypt+mdraid setups. Every time I notice one of these catastrophic failures it generally involves btrfs+<something else>. This is likely just because it's a timing thing, you put more layers you get a wider window in per-io races, you're more likely to be sad in the event of a failure. However it would be good to make sure these layers are doing the correct thing themselves.

Totally agreed, I'm actually the worst case scenario, I have

disk hw write cache + swraid5 + bcache + dmcrypt + btrfs

that's clearly a lot of layers. This is now my 4 or 5th pretty catastrophic btrfs failure over 2 arrays with those amount of layers over the last 8 years, but the last one was 2-3 years ago, so there is that :)
I think it's pretty clear that having all those layers is significantly increasing my risk factor.

Quoted 3 lines

We need to be better about this scenario, both in making sure we don't have bugs that contribute to the problem, but also that we have the tools necessary to recover when things go wrong. Thanks,

agreed to both, thanks again for your time on this

Quoted 7 lines · Zygo Blaxell
Quoted 2 lines

shouldn't it go read only also?
I haven't found a setting to tell it to do that if it's not the default.

bcache in writethrough mode could leave cached blocks dirty as long as the SSD completes the write. It should be reporting the write errors back to upper layers.
Whether it actually does...I don't know, I haven't run that kind of

agreed.

Quoted 8 lines
Quoted 2 lines

That's true, but I've seen btfrs remount read only before, and it didn't there. Shouldn't hard IO errors immediately cause btrfs to go read only?

No, only hard write IO errors on all metadata mirror drives, and cases where btrfs needs to CoW a page or read free space tree, and can't find an intact mirror. Anything less is correctable (write failure on some mirrors) or can be retried (any read failure) if the raid profile has redundancy.

Fair enough. And as you said, my errors were probably already staged in multiple layers and flushing before I got the first error reported back, so even by then it mau have been too late to stop it.

Quoted 8 lines
Quoted 6 lines

I haven't heard that these drives have broken caching, but maybe they do?

Device Model:     ST6000VN0041-2EL11C
Serial Number:    ZA18TVFZ
LU WWN Device Id: 5 000c50 0a4d9b49c
Firmware Version: SC61
User Capacity:    6,001,175,126,016 bytes [6.00 TB]

I've heard of problems with SC60, but IIRC SC61 was supposed to be the

Well, there you go, just found out that I have a mix of SC60 and SC61. Ok, I'll just turn off write caching on them all.

Quoted 4 lines

The log excerpt I saw didn't show any write errors from btrfs, only from mdadm. That suggests to me that the failure happened earlier when it was already too late to respond. Also we can't rule out that bcache is doing something stupid.

Yeah, at this point I think the logs agree with that statement. Also, I wasn't actually using that array, stuff doesn't really get written to it unless I manually start copies.

Thanks for looking this over and your opinion/sanity check.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#432 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 2 lines · Josef Bacik

Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,

Not sure it helped.

gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs
-rwxr-xr-x 1 root staff 6217688 Apr  6 13:38 btrfs*
gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit d51753abbc0ce345b9e163fe0fdbff086c9c609d
Author: Josef Bacik <[email protected]>
Date:   Wed Apr 6 16:37:17 2022 -0400

i need the owner

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a 
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55b85082dfc0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55b85082dfc0
Checking filesystem on /dev/mapper/dshelf1a
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#65 lore raw

Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption

In reply to Marc MERLIN
Quoted 8 lines · Marc MERLIN

This is an interesting discussion, so let's make a new thread out of it. TL;DR: I think btrfs may have failed to go read only earlier, causing more damage than needed to be, or some block layers just held enough data in flight that partial data got written, causing more damage than expected.
Figuring out the underlying problem would be good to avoid this again.

There's a lot here, and there's a lot of good information, but in general btrfs should never fail like this. We have a lot of power fail testing and error handling testing, but this is very limited. I found a bug last year where we would write an updated super block that pointed to garbage if we happened to be fsync'ing at the same time we were committing the transaction and got an error during the transaction commit phase.

We can't do anything about the disks lying to us. If a disk has a wonky FUA/FLUSH implementation then we're just sort of screwed. Unfortunately because our metadata moves around a lot we're waaaaay more susceptible to this failure case than ext4 or xfs, their metadata is relatively static they can put humpty dumpty back together again relatively simply.

Btrfs needs to

1. Go whole hog on error injection testing. I only barely scratched the surface with my bpf error injection stuff. This is on our roadmap and I plan on devoting developer time to this, but clearly that doesn't help you right now.
2. Put a lot more effort into disaster recovery. What I've written for you is an idea I've had in my head for a while. Some of this failures aren't catastrophic, we can generally pretty easily put back together a file system that resembles something sane by simply stitching together blocks that we find that are close enough to what we wanted. Unfortunately this gets back-burnered because in reality this doesn't happen that often.
3. Test these btrfs+dmcrypt+mdraid setups. Every time I notice one of these catastrophic failures it generally involves btrfs+<something else>. This is likely just because it's a timing thing, you put more layers you get a wider window in per-io races, you're more likely to be sad in the event of a failure. However it would be good to make sure these layers are doing the correct thing themselves.

We need to be better about this scenario, both in making sure we don't have bugs that contribute to the problem, but also that we have the tools necessary to recover when things go wrong. Thanks,

Josef

#433 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks,

Not sure it helped.

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Josef

#434 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Josef Bacik

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Different output going back and forth, expected?

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55ba30af45f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55ba30af45f0
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116

(still running)

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#435 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Different output going back and forth, expected?

Yup it's iterating over all the blocks, so it's going to find different slots in different blocks that are fucked.

You are having a lot of ones we can't find good matches for, depending on what those look like I may need to adjust the code to simply delete slots we don't find a good match for, which will be fine since we're going to clear this tree anyway. But let it finish and we'll see how the repair goes and I can do that if we need to. Thanks,

Josef

#436 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Josef Bacik
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik

Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks,

Different output going back and forth, expected?

Quoted 9 lines

Yup it's iterating over all the blocks, so it's going to find different slots in different blocks that are fucked.

You are having a lot of ones we can't find good matches for, depending on what those look like I may need to adjust the code to simply delete slots we don't find a good match for, which will be fine since we're going to clear this tree anyway. But let it finish and we'll see how the repair goes and I can do that if we need to. Thanks,

It took a lot longer, until it mostly hosed my machine by taking all the memory (32GB)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x55ba30af45f0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x55ba30af45f0
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slo^[[2~Timeout, server gargamel not responding.
[729314.382500] systemd-udevd[25274]: dm-13: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729314.490868] systemd-udevd[25258]: dm-12: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729351.745188] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 is taking a long time
[729352.745164] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 is taking a long time
[729352.774312] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 is taking a long time
[729354.745172] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 is taking a long time
[729354.745174] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] is taking longer than 59s to complete
[729354.745175] systemd-udevd[25275]: memory229: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26401] is taking longer than 59s to complete
[729354.859212] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 is taking a long time
[729354.889966] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 is taking a long time
[729354.920686] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 is taking a long time
[729354.951365] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 is taking a long time
[729357.745162] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 is taking a long time
[729357.774644] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 is taking a long time
[729357.803818] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 is taking a long time
[729471.745176] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 killed
[729472.745177] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 killed
[729472.769590] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 killed
[729474.745171] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] timed out after 2min 59s, killing
[729474.745175] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 killed
[729474.812102] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 killed
[729474.838576] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 killed
[729474.864496] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 killed
[729474.890429] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 killed
[729474.916324] systemd-udevd[2257]: Worker [25275] terminated by signal 9 (KILL)
[729477.442170] systemd-udevd[25244]: sdd4: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729477.465202] systemd-udevd[25248]: sdd1: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729477.475194] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 killed
[729477.531960] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 killed
[729477.556548] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 killed
[729494.433414] systemd-udevd[25271]: sdd2: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729512.577884] systemd-udevd[25265]: sdd3: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729572.745171] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 is taking a long time
[729692.745172] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 killed
[729828.852640] systemd-udevd[25271]: LNXSYSTM:00: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.880567] systemd-udevd[25294]: cstate_core: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.882126] systemd-udevd[25265]: breakpoint: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.885879] systemd-udevd[25244]: cpu: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.885886] systemd-udevd[25339]: kprobe: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.886061] systemd-udevd[25252]: parport0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.886527] systemd-udevd[25258]: cstate_pkg: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.888578] systemd-udevd[25255]: 0000:00:00.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.888587] systemd-udevd[25274]: i915: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729828.905903] systemd-udevd[25264]: msr: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[729829.377862] elogind-daemon[9054]: Watching system buttons on /dev/input/event0 (Power Button)
[729829.469868] elogind-daemon[9054]: Watching system buttons on /dev/input/event1 (Power Button)
[729829.833865] elogind-daemon[9054]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard)
[729830.359362] BTRFS info (device dm-12): devid 1 device path /dev/mapper/vgds2-1Appliances changed to /dev/dm-12 scanned by systemd-udevd (25274)
[729889.745171] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 is taking a long time
[730009.745168] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 killed
[730009.770590] systemd-udevd[2257]: Worker [25264] terminated by signal 9 (KILL)
[730009.793062] systemd-udevd[2257]: 4-1.6.3: Worker [25264] failed
[730009.813461] systemd-udevd[29139]: 4-1.6.3:1.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
[732952.473804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 35s!
[732952.498625] Showing busy workqueues and worker pools:
[732952.514833] workqueue events: flags=0x0
[732952.527430]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732952.527442]     pending: cache_reap
[732952.527447]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[732952.527449]     pending: drm_fb_helper_damage_work [drm_kms_helper]
[732952.527472] workqueue writeback: flags=0x4a
[732952.615583]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[732952.615587]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[732952.615608] workqueue btrfs-delalloc: flags=0xe
[732952.667841]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732952.667843]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[732952.667859] workqueue btrfs-worker: flags=0xe
[732952.723431]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732952.723433]     in-flight: 25134:btrfs_work_helper
[732952.723436] workqueue btrfs-endio-write: flags=0xe
[732952.774041]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732952.774043]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[732952.774047] workqueue btrfs-delalloc: flags=0xe
[732952.830056]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732952.830058]     in-flight: 25030:btrfs_work_helper
[732952.830070] workqueue cifsiod: flags=0xc
[732952.878011]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732952.878013]     in-flight: 9189:cifs_echo_request [cifs]
[732952.878056] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=36s workers=2 manager: 17834
[732952.878059] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 30021
[732983.193804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 66s!
[732983.218550] Showing busy workqueues and worker pools:
[732983.234716] workqueue events: flags=0x0
[732983.247202]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732983.247205]     pending: cache_reap
[732983.247209]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[732983.247211]     pending: drm_fb_helper_damage_work [drm_kms_helper]
[732983.247233] workqueue writeback: flags=0x4a
[732983.334945]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[732983.334948]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[732983.334971] workqueue btrfs-delalloc: flags=0xe
[732983.386983]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732983.386985]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[732983.387000] workqueue btrfs-worker: flags=0xe
[732983.442412]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732983.442415]     in-flight: 25134:btrfs_work_helper
[732983.442417] workqueue btrfs-endio-write: flags=0xe
[732983.492919]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[732983.492921]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[732983.492925] workqueue btrfs-delalloc: flags=0xe
[732983.548832]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[732983.548843]     in-flight: 25030:btrfs_work_helper
[732983.548854] workqueue cifsiod: flags=0xc
[732983.596634]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2
[732983.596637]     in-flight: 9189:cifs_echo_request [cifs]
[732983.596680] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=66s workers=2 manager: 17834
[732983.596683] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903
[733013.909809] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 97s!
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#437 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Damn, it's hosed so badly that even sysrq-o isn't working, never seen that before.

[733866.717217] sysrq: Terminate All Tasks
[733866.740424] elogind-daemon[9054]: Received signal 15 [TERM]
[733868.540292] sysrq: Emergency Remount R/O
[733870.249149] sysrq: Emergency Sync
[733904.793816] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 988s!
[733904.818717] Showing busy workqueues and worker pools:
[733904.834771] workqueue events: flags=0x0
[733904.847171]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6
[733904.847174]     pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work
[733904.847205] workqueue writeback: flags=0x4a
[733904.917846]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[733904.917849]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[733904.917868] workqueue btrfs-delalloc: flags=0xe
[733904.969635]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733904.969637]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[733904.969652] workqueue btrfs-worker: flags=0xe
[733905.024811]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733905.024813]     in-flight: 25134:btrfs_work_helper
[733905.024817] workqueue btrfs-endio-write: flags=0xe
[733905.075033]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733905.075035]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[733905.075039] workqueue btrfs-delalloc: flags=0xe
[733905.130695]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733905.130697]     in-flight: 25030:btrfs_work_helper
[733905.130709] workqueue cifsiod: flags=0xc
[733905.178289]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4
[733905.178291]     in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs]
[733905.178363] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=988s workers=2 manager: 17834
[733905.178366] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903
[733909.916792] sysrq: Power Off
[733966.261816] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x1 nice=0 stuck for 55s!
[733966.286477] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 1049s!
[733966.311622] Showing busy workqueues and worker pools:
[733966.327664] workqueue events: flags=0x0
[733966.340063]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6
[733966.340067]     pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work
[733966.340093]   pwq 0: cpus=0 node=0 flags=0x1 nice=0 active=3/256 refcnt=4
[733966.340095]     in-flight: 17295:do_poweroff
[733966.340098]     pending: vmstat_shepherd, cache_reap
[733966.340106] workqueue writeback: flags=0x4a
[733966.461521]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4
[733966.461524]     in-flight: 8793:wb_workfn, 8613:wb_workfn
[733966.461543] workqueue btrfs-delalloc: flags=0xe
[733966.513325]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733966.513328]     in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper
[733966.513344] workqueue btrfs-worker: flags=0xe
[733966.568487]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733966.568489]     in-flight: 25134:btrfs_work_helper
[733966.568492] workqueue btrfs-endio-write: flags=0xe
[733966.618683]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4
[733966.618685]     in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper
[733966.618698] workqueue btrfs-delalloc: flags=0xe
[733966.674361]   pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3
[733966.674364]     in-flight: 25030:btrfs_work_helper
[733966.674376] workqueue cifsiod: flags=0xc
[733966.721929]   pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4
[733966.721932]     in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs]
[733966.722002] pool 0: cpus=0 node=0 flags=0x1 nice=0 hung=55s workers=2 manager: 9899
[733966.722006] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=1049s workers=2 manager: 17834
[733966.722008] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#448 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#438 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN

Hi!

Marc MERLIN - 07.04.22, 06:40:06 CEST:

Quoted 2 lines

Damn, it's hosed so badly that even sysrq-o isn't working, never seen that before.

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Best,

Signature · 1 line
Martin
#66 lore raw

Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption

In reply to Josef Bacik
Quoted 9 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN

This is an interesting discussion, so let's make a new thread out of it. TL;DR: I think btrfs may have failed to go read only earlier, causing more damage than needed to be, or some block layers just held enough data in flight that partial data got written, causing more damage than expected.
Figuring out the underlying problem would be good to avoid this again.

Quoted 6 lines

We can't do anything about the disks lying to us. If a disk has a wonky FUA/FLUSH implementation then we're just sort of screwed. Unfortunately because our metadata moves around a lot we're waaaaay more susceptible to this failure case than ext4 or xfs, their metadata is relatively static they can put humpty dumpty back together again relatively simply.

Disks are pretty good at FUA/FLUSH bugs these days, though you can still buy new old drives that have them.

SSDs have a whole lot of bugs, especially if they get old. have a few bits fail, and try to recover themselves, but blow up the whole filesystem by losing a few pages. These ones get through firmware qualification testing because they only misbehave when they're a year or two old.

Quoted 13 lines

Btrfs needs to

1. Go whole hog on error injection testing. I only barely scratched the surface with my bpf error injection stuff. This is on our roadmap and I plan on devoting developer time to this, but clearly that doesn't help you right now.
2. Put a lot more effort into disaster recovery. What I've written for you is an idea I've had in my head for a while. Some of this failures aren't catastrophic, we can generally pretty easily put back together a file system that resembles something sane by simply stitching together blocks that we find that are close enough to what we wanted. Unfortunately this gets back-burnered because in reality this doesn't happen that often.

This does seem to be an important missing piece at the moment. In practice, when we see "parent transid verify failed," we go directly to mkfs + restore backups, as none of the existing tools will touch it. They all want intact interior nodes in some tree to work, and that's the one thing that FUA/FLUSH bugs destroy.

It doesn't solve the underlying problem--the drive will still trash the filesystem every other month until its write ordering gets fixed or worked around--but at least it lets users fix the filesystem in place without mkfs + restoring a full set of backups.

Quoted 6 lines

3. Test these btrfs+dmcrypt+mdraid setups. Every time I notice one of these catastrophic failures it generally involves btrfs+<something else>. This is likely just because it's a timing thing, you put more layers you get a wider window in per-io races, you're more likely to be sad in the event of a failure. However it would be good to make sure these layers are doing the correct thing themselves.

bcache and dmcache layers in series with the underlying storage multiply the failure rates (including software regressions) of the individual components, and mdadm blends firmware bugs from all the drives together. Statistically we're always going to see more problems if there are more moving parts in the system, especially if they aggregate risks and break fault isolation.

FWIW I'm not seeing a difference in failure rate between "btrfs + mdadm" and "btrfs alone", but I'm intentionally avoiding the many mdadm configurations that introduce new failure modes by design (e.g. raid5 without ppl or journal, raid1 without component device integrity, or single/raid1 SSD cache in writeback mode running over multiple drives). There isn't a fix for those except to not use them. In most cases there's a better way to do the same thing, though there are some gaps (e.g. there's no working solution for writeback SSD caching on raid5).

Thought experiment: if you have 2 drives from vendor A and 2 drives from vendor B, and you want to build two filesystems that replicate to each other, where do you put the drives if you know one vendor has a firmware bug but you don't know which one? With mdadm, you build filesystems with AA and BB drives, because that way a firmware bug corrupts one filesystems and leaves the other intact. With btrfs, you build two filesystems with AB and AB drives, because that way a firmware bug gets autocorrected by btrfs and both filesystems remain intact. If you put btrfs on mdadm on AB drives, you combine the risk from both vendors, lose both filesystems, and skew the failure statistics.

Quoted 5 lines

We need to be better about this scenario, both in making sure we don't have bugs that contribute to the problem, but also that we have the tools necessary to recover when things go wrong. Thanks,

Josef

#439 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Martin Steigerwald
Quoted 5 lines · Martin Steigerwald

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.

Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.

Overnight, it ran stable at 8GB:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root

But the bad news is that it didn't complete, and it's not looking like it's converging.

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1 2>&1 |tee /tmp/outo
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
parent transid verify failed on 22216704 wanted 1600938 found 1602177
FS_INFO IS 0x56055fb76600
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297
FS_INFO AFTER IS 0x56055fb76600
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 430
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 41
fixed slot 63
fixed slot 33
fixed slot 36
Couldn't find a replacement block for slot 47
fixed slot 52
fixed slot 58
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 64
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 223
fixed slot 76
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 92
fixed slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 245
fixed slot 153
Couldn't find a replacement block for slot 115
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
fixed slot 125
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 185
fixed slot 200
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 204
Couldn't find a replacement block for slot 103
fixed slot 249
Couldn't find a replacement block for slot 250
fixed slot 284
fixed slot 305
fixed slot 128
fixed slot 144
Couldn't find a replacement block for slot 342
fixed slot 164
fixed slot 204
fixed slot 58
fixed slot 132
Couldn't find a replacement block for slot 161
fixed slot 177
fixed slot 209
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 96
fixed slot 5
fixed slot 114
Couldn't find a replacement block for slot 90
fixed slot 237
fixed slot 19
fixed slot 7
Couldn't find a replacement block for slot 27
fixed slot 48
fixed slot 53
Couldn't find a replacement block for slot 45
Couldn't find a replacement block for slot 225
fixed slot 50
Couldn't find a replacement block for slot 52
fixed slot 57
fixed slot 208
fixed slot 210
fixed slot 78
fixed slot 230
fixed slot 61
Couldn't find a replacement block for slot 81
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 174
fixed slot 96
fixed slot 63
fixed slot 113
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 53
fixed slot 78
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 174
fixed slot 182
Couldn't find a replacement block for slot 183
fixed slot 184
Couldn't find a replacement block for slot 185
fixed slot 199
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 213
fixed slot 215
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 124
fixed slot 125
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 139
fixed slot 140
Couldn't find a replacement block for slot 148
Couldn't find a replacement block for slot 150
Couldn't find a replacement block for slot 156
Couldn't find a replacement block for slot 166
fixed slot 179
fixed slot 186
fixed slot 111
fixed slot 0
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 44
fixed slot 49
Couldn't find a replacement block for slot 67
fixed slot 68
fixed slot 69
fixed slot 70
fixed slot 76
fixed slot 80
fixed slot 83
Couldn't find a replacement block for slot 86
fixed slot 113
fixed slot 140
fixed slot 143
fixed slot 144
fixed slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
Couldn't find a replacement block for slot 196
fixed slot 197
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 199
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 239
fixed slot 245
fixed slot 247
fixed slot 248
fixed slot 260
fixed slot 20
fixed slot 28
fixed slot 44
fixed slot 45
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 69
fixed slot 82
Couldn't find a replacement block for slot 86
fixed slot 90
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 97
fixed slot 110
fixed slot 116
fixed slot 121
fixed slot 138
fixed slot 150
fixed slot 158
fixed slot 160
fixed slot 162
Couldn't find a replacement block for slot 168
Couldn't find a replacement block for slot 169
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 173
fixed slot 175
fixed slot 179
fixed slot 181
Couldn't find a replacement block for slot 182
fixed slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
fixed slot 193
fixed slot 205
fixed slot 215
Couldn't find a replacement block for slot 224
fixed slot 233
fixed slot 237
fixed slot 243
fixed slot 256
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 15
fixed slot 21
fixed slot 26
Couldn't find a replacement block for slot 32
fixed slot 50
fixed slot 55
Couldn't find a replacement block for slot 57
fixed slot 112
fixed slot 113
fixed slot 114
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 127
fixed slot 131
Couldn't find a replacement block for slot 140
fixed slot 146
Couldn't find a replacement block for slot 153
fixed slot 154
Couldn't find a replacement block for slot 159
fixed slot 163
fixed slot 164
fixed slot 177
fixed slot 202
fixed slot 214
Couldn't find a replacement block for slot 220
fixed slot 230
Couldn't find a replacement block for slot 246
fixed slot 251
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 261
Couldn't find a replacement block for slot 262
Couldn't find a replacement block for slot 263
fixed slot 264
fixed slot 266
fixed slot 114
fixed slot 5
fixed slot 24
fixed slot 72
fixed slot 85
fixed slot 93
fixed slot 98
Couldn't find a replacement block for slot 111
fixed slot 128
fixed slot 130
Couldn't find a replacement block for slot 131
fixed slot 133
fixed slot 145
fixed slot 146
Couldn't find a replacement block for slot 75
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 77
Couldn't find a replacement block for slot 83
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 37
Couldn't find a replacement block for slot 38
fixed slot 111
fixed slot 118
fixed slot 119
fixed slot 206
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 376
fixed slot 377
Couldn't find a replacement block for slot 60
fixed slot 122
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 180
fixed slot 126
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 132
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 141
fixed slot 129
fixed slot 134
Couldn't find a replacement block for slot 100
fixed slot 101
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 367
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 244
Couldn't find a replacement block for slot 308
Couldn't find a replacement block for slot 309
Couldn't find a replacement block for slot 310
Couldn't find a replacement block for slot 311
Couldn't find a replacement block for slot 312
Couldn't find a replacement block for slot 326
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 49
fixed slot 13
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 145
Couldn't find a replacement block for slot 76
fixed slot 143
Couldn't find a replacement block for slot 28
fixed slot 159
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 126
fixed slot 148
Couldn't find a replacement block for slot 150
fixed slot 155
fixed slot 161
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 199
fixed slot 161
Couldn't find a replacement block for slot 108
Couldn't find a replacement block for slot 110
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 263
fixed slot 169
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 51
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 138
fixed slot 157
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 207
fixed slot 210
fixed slot 212
fixed slot 213
fixed slot 216
fixed slot 219
Couldn't find a replacement block for slot 235
fixed slot 242
fixed slot 243
fixed slot 244
Couldn't find a replacement block for slot 245
Couldn't find a replacement block for slot 269
fixed slot 170
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 279
Couldn't find a replacement block for slot 280
Couldn't find a replacement block for slot 281
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 9
fixed slot 37
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 114
fixed slot 115
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 118
fixed slot 127
Couldn't find a replacement block for slot 135
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 219
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 235
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 264
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 267
fixed slot 16
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 41
fixed slot 44
Couldn't find a replacement block for slot 52
fixed slot 55
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 97
Couldn't find a replacement block for slot 98
fixed slot 100
Couldn't find a replacement block for slot 118
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 125
fixed slot 150
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 208
fixed slot 211
fixed slot 213
Couldn't find a replacement block for slot 218
fixed slot 221
Couldn't find a replacement block for slot 231
fixed slot 248
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 256
fixed slot 173
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 5
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
fixed slot 39
Couldn't find a replacement block for slot 41
fixed slot 55
Couldn't find a replacement block for slot 78
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 99
fixed slot 102
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 2(... still running)
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#440 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 30 lines · Marc MERLIN
Quoted 5 lines · Martin Steigerwald

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.

Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.

Quoted 7 lines

Overnight, it ran stable at 8GB:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root

But the bad news is that it didn't complete, and it's not looking like it's converging.

Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while. Thanks,

Josef

#441 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Josef Bacik - 07.04.22, 19:07:14 CEST:

Quoted 5 lines

Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while.

You guys are having fun with that, it seems.

Signature · 1 line
Martin
#442 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 40 lines · Josef Bacik
Quoted 30 lines · Marc MERLIN
Quoted 5 lines · Martin Steigerwald

I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list.

Good luck with restoring your filesystem or at least recovering all data from it!

Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case.

Quoted 7 lines · Marc MERLIN
Quoted 1 line · Marc MERLIN
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo

there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight.

Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome).
btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault.

Quoted 7 lines

Overnight, it ran stable at 8GB:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
31873 root      20   0 8024m 7.8g    0 R  99.9 25.2 667:57.24 btrfs-find-root

But the bad news is that it didn't complete, and it's not looking like it's converging.

Quoted 7 lines

Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while. Thanks,

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

Josef

#443 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

it's stlil churning along and looping. I have no idea if it's converging, or not.

parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56055fb76600
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x56055fb76600
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 62
fixed slot 88
fixed slot 235
fixed slot 237
Couldn't find a replacement block for slot 238
Couldn't find a replacement block for slot 243
fixed slot 33
fixed slot 35
fixed slot 32
fixed slot 133
fixed slot 136
fixed slot 207
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 484
Couldn't find a replacement block for slot 76
fixed slot 229
fixed slot 172
fixed slot 174
fixed slot 36
Couldn't find a replacement block for slot 20
fixed slot 24
Couldn't find a replacement block for slot 28
fixed slot 115
fixed slot 148
fixed slot 9
Couldn't find a replacement block for slot 318
fixed slot 245
fixed slot 1
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 134
fixed slot 159
fixed slot 170
fixed slot 204
fixed slot 149
fixed slot 185
fixed slot 257
fixed slot 46
fixed slot 60
fixed slot 91
fixed slot 92
fixed slot 95
fixed slot 247
fixed slot 105
Couldn't find a replacement block for slot 109
fixed slot 124
fixed slot 191
fixed slot 244
fixed slot 9
fixed slot 57
Couldn't find a replacement block for slot 166
fixed slot 163
fixed slot 78
fixed slot 179
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 218
fixed slot 26
fixed slot 75
Couldn't find a replacement block for slot 68
fixed slot 56
fixed slot 60
Couldn't find a replacement block for slot 231
fixed slot 141
fixed slot 151
fixed slot 156
fixed slot 159
fixed slot 163
fixed slot 171
fixed slot 200
fixed slot 5
Couldn't find a replacement block for slot 157
fixed slot 224
fixed slot 44
fixed slot 217
fixed slot 122
fixed slot 59
Couldn't find a replacement block for slot 120
fixed slot 181
fixed slot 187
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 220
fixed slot 60
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 192
fixed slot 9
fixed slot 7
Couldn't find a replacement block for slot 226
fixed slot 24
fixed slot 49
fixed slot 22
Couldn't find a replacement block for slot 186
fixed slot 149
fixed slot 150
Couldn't find a replacement block for slot 42
fixed slot 218
fixed slot 223
fixed slot 228
fixed slot 230
fixed slot 233
fixed slot 234
fixed slot 240
fixed slot 155
fixed slot 24
fixed slot 30
fixed slot 376
fixed slot 94
fixed slot 104
fixed slot 197
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 191
fixed slot 139
fixed slot 128
Couldn't find a replacement block for slot 143
fixed slot 16
fixed slot 17
fixed slot 114
fixed slot 28
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 83
fixed slot 86
fixed slot 92
fixed slot 149
fixed slot 117
Couldn't find a replacement block for slot 31
fixed slot 188
Couldn't find a replacement block for slot 102
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 79
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 21
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 276
Couldn't find a replacement block for slot 282
Couldn't find a replacement block for slot 283
Couldn't find a replacement block for slot 284
Couldn't find a replacement block for slot 285
Couldn't find a replacement block for slot 286
Couldn't find a replacement block for slot 287
Couldn't find a replacement block for slot 289
Couldn't find a replacement block for slot 290
Couldn't find a replacement block for slot 291
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 231
fixed slot 208
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 208
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 130
fixed slot 138
fixed slot 33
Couldn't find a replacement block for slot 179
fixed slot 154
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 129
fixed slot 111
Couldn't find a replacement block for slot 103
Couldn't find a replacement block for slot 192
fixed slot 12
Couldn't find a replacement block for slot 205
fixed slot 24
fixed slot 28
fixed slot 31
fixed slot 38
fixed slot 47
fixed slot 62
fixed slot 75
fixed slot 86
fixed slot 89
fixed slot 93
fixed slot 108
Couldn't find a replacement block for slot 115
fixed slot 146
fixed slot 218
fixed slot 288
fixed slot 33
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 176
fixed slot 240
Couldn't find a replacement block for slot 241
fixed slot 34
fixed slot 35
Couldn't find a replacement block for slot 71
Couldn't find a replacement block for slot 140
fixed slot 213
Couldn't find a replacement block for slot 155
fixed slot 210
fixed slot 111
fixed slot 147
fixed slot 150
Couldn't find a replacement block for slot 234
Couldn't find a replacement block for slot 236
fixed slot 240
fixed slot 250
Couldn't find a replacement block for slot 259
fixed slot 269
fixed slot 283
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 48
fixed slot 135
fixed slot 192
fixed slot 206
fixed slot 43
fixed slot 111
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 45
fixed slot 47
fixed slot 212
fixed slot 49
fixed slot 190
fixed slot 4
fixed slot 40
fixed slot 110
fixed slot 227
fixed slot 290
fixed slot 296
fixed slot 297
fixed slot 302
fixed slot 349
fixed slot 357
Couldn't find a replacement block for slot 53
fixed slot 304
Couldn't find a replacement block for slot 328
fixed slot 376
fixed slot 54
fixed slot 125
fixed slot 195
Couldn't find a replacement block for slot 89
fixed slot 177
fixed slot 197
Couldn't find a replacement block for slot 219
fixed slot 233
fixed slot 36
fixed slot 52
fixed slot 61
fixed slot 71
fixed slot 113
fixed slot 119
fixed slot 138
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 200
fixed slot 215
Couldn't find a replacement block for slot 173
fixed slot 315
fixed slot 320
fixed slot 326
fixed slot 328
fixed slot 335
fixed slot 65
fixed slot 32
Couldn't find a replacement block for slot 33
Couldn't find a replacement block for slot 34
Couldn't find a replacement block for slot 37
fixed slot 40
Couldn't find a replacement block for slot 179
fixed slot 75
fixed slot 198
fixed slot 79
fixed slot 16
fixed slot 89
fixed slot 111
fixed slot 152
fixed slot 219
fixed slot 220
fixed slot 43
fixed slot 70
fixed slot 85
fixed slot 102
fixed slot 130
fixed slot 145
fixed slot 146
fixed slot 161
fixed slot 188
fixed slot 202
Couldn't find a replacement block for slot 9
fixed slot 62
fixed slot 122
fixed slot 128
fixed slot 137
fixed slot 142
fixed slot 216
Couldn't find a replacement block for slot 26
fixed slot 143
fixed slot 152
fixed slot 174
fixed slot 185
fixed slot 76
fixed slot 87
fixed slot 26
fixed slot 28
fixed slot 31
fixed slot 33
fixed slot 35
fixed slot 43
fixed slot 46
fixed slot 47
fixed slot 56
fixed slot 62
fixed slot 69
fixed slot 73
fixed slot 76
fixed slot 78
fixed slot 81
fixed slot 83
fixed slot 90
fixed slot 92
fixed slot 93
fixed slot 94
fixed slot 98
fixed slot 110
fixed slot 113
fixed slot 114
fixed slot 115
fixed slot 117
fixed slot 118
fixed slot 119
fixed slot 120
fixed slot 121
fixed slot 122
fixed slot 125
fixed slot 142
fixed slot 146
fixed slot 147
fixed slot 148
fixed slot 179
fixed slot 182
fixed slot 185
fixed slot 101
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 89
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 430
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 41
fixed slot 63
fixed slot 33
fixed slot 36
Couldn't find a replacement block for slot 47
fixed slot 52
fixed slot 58
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 64
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 223
fixed slot 76
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 92
fixed slot 228
Couldn't find a replacement block for slot 109
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 245
fixed slot 153
Couldn't find a replacement block for slot 115
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
fixed slot 125
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 185
fixed slot 200
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 204
Couldn't find a replacement block for slot 103
fixed slot 249
Couldn't find a replacement block for slot 250
fixed slot 284
fixed slot 305
fixed slot 128
fixed slot 144
Couldn't find a replacement block for slot 342
fixed slot 164
fixed slot 204
fixed slot 58
fixed slot 132
Couldn't find a replacement block for slot 161
fixed slot 177
fixed slot 209
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 96
fixed slot 5
fixed slot 114
Couldn't find a replacement block for slot 90
fixed slot 237
fixed slot 19
fixed slot 7
Couldn't find a replacement block for slot 27
fixed slot 48
fixed slot 53
Couldn't find a replacement block for slot 45
Couldn't find a replacement block for slot 225
fixed slot 50
Couldn't find a replacement block for slot 52
fixed slot 57
fixed slot 208
fixed slot 210
fixed slot 78
fixed slot 230
fixed slot 61
Couldn't find a replacement block for slot 81
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 174
fixed slot 96
fixed slot 63
fixed slot 113
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 53
fixed slot 78
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 174
fixed slot 182
Couldn't find a replacement block for slot 183
fixed slot 184
Couldn't find a replacement block for slot 185
fixed slot 199
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 213
fixed slot 215
Couldn't find a replacement block for slot 106
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 124
fixed slot 125
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 139
fixed slot 140
Couldn't find a replacement block for slot 148
Couldn't find a replacement block for slot 150
Couldn't find a replacement block for slot 156
Couldn't find a replacement block for slot 166
fixed slot 179
fixed slot 186
fixed slot 111
fixed slot 0
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 44
fixed slot 49
Couldn't find a replacement block for slot 67
fixed slot 68
fixed slot 69
fixed slot 70
fixed slot 76
fixed slot 80
fixed slot 83
Couldn't find a replacement block for slot 86
fixed slot 113
fixed slot 140
fixed slot 143
fixed slot 144
fixed slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
Couldn't find a replacement block for slot 196
fixed slot 197
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 199
fixed slot 202
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 239
fixed slot 245
fixed slot 247
fixed slot 248
fixed slot 260
fixed slot 20
fixed slot 28
fixed slot 44
fixed slot 45
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 69
fixed slot 82
Couldn't find a replacement block for slot 86
fixed slot 90
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 97
fixed slot 110
fixed slot 116
fixed slot 121
fixed slot 138
fixed slot 150
fixed slot 158
fixed slot 160
fixed slot 162
Couldn't find a replacement block for slot 168
Couldn't find a replacement block for slot 169
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 173
fixed slot 175
fixed slot 179
fixed slot 181
Couldn't find a replacement block for slot 182
fixed slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
fixed slot 193
fixed slot 205
fixed slot 215
Couldn't find a replacement block for slot 224
fixed slot 233
fixed slot 237
fixed slot 243
fixed slot 256
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 15
fixed slot 21
fixed slot 26
Couldn't find a replacement block for slot 32
fixed slot 50
fixed slot 55
Couldn't find a replacement block for slot 57
fixed slot 112
fixed slot 113
fixed slot 114
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 127
fixed slot 131
Couldn't find a replacement block for slot 140
fixed slot 146
Couldn't find a replacement block for slot 153
fixed slot 154
Couldn't find a replacement block for slot 159
fixed slot 163
fixed slot 164
fixed slot 177
fixed slot 202
fixed slot 214
Couldn't find a replacement block for slot 220
fixed slot 230
Couldn't find a replacement block for slot 246
fixed slot 251
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 261
Couldn't find a replacement block for slot 262
Couldn't find a replacement block for slot 263
fixed slot 264
fixed slot 266
fixed slot 114
fixed slot 5
fixed slot 24
fixed slot 72
fixed slot 85
fixed slot 93
fixed slot 98
Couldn't find a replacement block for slot 111
fixed slot 128
fixed slot 130
Couldn't find a replacement block for slot 131
fixed slot 133
fixed slot 145
fixed slot 146
Couldn't find a replacement block for slot 75
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 77
Couldn't find a replacement block for slot 83
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 37
Couldn't find a replacement block for slot 38
fixed slot 111
fixed slot 118
fixed slot 119
fixed slot 206
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 376
fixed slot 377
Couldn't find a replacement block for slot 60
fixed slot 122
Couldn't find a replacement block for slot 92
fixed slot 96
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 171
Couldn't find a replacement block for slot 172
Couldn't find a replacement block for slot 178
Couldn't find a replacement block for slot 180
fixed slot 126
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 120
Couldn't find a replacement block for slot 121
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 132
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 141
fixed slot 129
fixed slot 134
Couldn't find a replacement block for slot 100
fixed slot 101
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 367
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 244
Couldn't find a replacement block for slot 308
Couldn't find a replacement block for slot 309
Couldn't find a replacement block for slot 310
Couldn't find a replacement block for slot 311
Couldn't find a replacement block for slot 312
Couldn't find a replacement block for slot 326
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 49
fixed slot 13
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 145
Couldn't find a replacement block for slot 76
fixed slot 143
Couldn't find a replacement block for slot 28
fixed slot 159
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 126
fixed slot 148
Couldn't find a replacement block for slot 150
fixed slot 155
fixed slot 161
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 214
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 243
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 199
fixed slot 161
Couldn't find a replacement block for slot 108
Couldn't find a replacement block for slot 110
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 263
fixed slot 169
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 51
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 124
Couldn't find a replacement block for slot 138
fixed slot 157
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 170
Couldn't find a replacement block for slot 207
fixed slot 210
fixed slot 212
fixed slot 213
fixed slot 216
fixed slot 219
Couldn't find a replacement block for slot 235
fixed slot 242
fixed slot 243
fixed slot 244
Couldn't find a replacement block for slot 245
Couldn't find a replacement block for slot 269
fixed slot 170
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 279
Couldn't find a replacement block for slot 280
Couldn't find a replacement block for slot 281
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 9
fixed slot 37
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 74
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 113
Couldn't find a replacement block for slot 114
fixed slot 115
Couldn't find a replacement block for slot 116
Couldn't find a replacement block for slot 117
Couldn't find a replacement block for slot 118
fixed slot 127
Couldn't find a replacement block for slot 135
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 167
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 219
Couldn't find a replacement block for slot 221
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 235
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 264
Couldn't find a replacement block for slot 265
Couldn't find a replacement block for slot 267
fixed slot 16
Couldn't find a replacement block for slot 23
Couldn't find a replacement block for slot 26
Couldn't find a replacement block for slot 41
fixed slot 44
Couldn't find a replacement block for slot 52
fixed slot 55
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 97
Couldn't find a replacement block for slot 98
fixed slot 100
Couldn't find a replacement block for slot 118
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 125
fixed slot 150
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 198
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 208
fixed slot 211
fixed slot 213
Couldn't find a replacement block for slot 218
fixed slot 221
Couldn't find a replacement block for slot 231
fixed slot 248
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 256
fixed slot 173
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 5
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
fixed slot 39
Couldn't find a replacement block for slot 41
fixed slot 55
Couldn't find a replacement block for slot 78
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 98
Couldn't find a replacement block for slot 99
fixed slot 102
Couldn't find a replacement block for slot 114
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 128
Couldn't find a replacement block for slot 138
Couldn't find a replacement block for slot 139
Couldn't find a replacement block for slot 140
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 217
Couldn't find a replacement block for slot 222
Couldn't find a replacement block for slot 223
Couldn't find a replacement block for slot 224
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 226
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 253
Couldn't find a replacement block for slot 254
fixed slot 256
Couldn't find a replacement block for slot 11
fixed slot 28
Couldn't find a replacement block for slot 33
fixed slot 39
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 72
fixed slot 119
Couldn't find a replacement block for slot 146
fixed slot 156
Couldn't find a replacement block for slot 179
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 197
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 250
Couldn't find a replacement block for slot 252
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 18
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 53
Couldn't find a replacement block for slot 56
Couldn't find a replacement block for slot 57
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 86
Couldn't find a replacement block for slot 87
fixed slot 99
Couldn't find a replacement block for slot 100
Couldn't find a replacement block for slot 101
fixed slot 108
fixed slot 141
fixed slot 155
fixed slot 178
Couldn't find a replacement block for slot 181
Couldn't find a replacement block for slot 197
fixed slot 177
fixed slot 193
Couldn't find a replacement block for slot 194
Couldn't find a replacement block for slot 195
fixed slot 226
Couldn't find a replacement block for slot 273
fixed slot 302
Couldn't find a replacement block for slot 69
Couldn't find a replacement block for slot 70
Couldn't find a replacement block for slot 143
Couldn't find a replacement block for slot 144
fixed slot 204
Couldn't find a replacement block for slot 129
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 214
fixed slot 248
fixed slot 220
fixed slot 204
Couldn't find a replacement block for slot 205
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 210
Couldn't find a replacement block for slot 211
Couldn't find a replacement block for slot 215
Couldn't find a replacement block for slot 220
fixed slot 234
Couldn't find a replacement block for slot 235
fixed slot 236
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 267
fixed slot 276
fixed slot 288
Couldn't find a replacement block for slot 317
fixed slot 330
Couldn't find a replacement block for slot 337
Couldn't find a replacement block for slot 345
Couldn't find a replacement block for slot 346
fixed slot 361
Couldn't find a replacement block for slot 365
Couldn't find a replacement block for slot 366
Couldn't find a replacement block for slot 369
Couldn't find a replacement block for slot 370
Couldn't find a replacement block for slot 372
Couldn't find a replacement block for slot 373
fixed slot 379
Couldn't find a replacement block for slot 380
Couldn't find a replacement block for slot 381
Couldn't find a replacement block for slot 382
Couldn't find a replacement block for slot 387
Couldn't find a replacement block for slot 388
Couldn't find a replacement block for slot 389
Couldn't find a replacement block for slot 390
Couldn't find a replacement block for slot 394
fixed slot 395
Couldn't find a replacement block for slot 396
fixed slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 410
Couldn't find a replacement block for slot 411
Couldn't find a replacement block for slot 412
Couldn't find a replacement block for slot 425
Couldn't find a replacement block for slot 426
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 428
fixed slot 221
fixed slot 0
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 20
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 49
Couldn't find a replacement block for slot 50
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 67
fixed slot 83
Couldn't find a replacement block for slot 84
Couldn't find a replacement block for slot 119
Couldn't find a replacement block for slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 126
fixed slot 206
Couldn't find a replacement block for slot 207
Couldn't find a replacement block for slot 220
fixed slot 224
fixed slot 143
fixed slot 158
Couldn't find a replacement block for slot 159
Couldn't find a replacement block for slot 160
Couldn't find a replacement block for slot 199
Couldn't find a replacement block for slot 204
Couldn't find a replacement block for slot 227
Couldn't find a replacement block for slot 228
fixed slot 229
fixed slot 225
fixed slot 95
fixed slot 104
Couldn't find a replacement block for slot 105
Couldn't find a replacement block for slot 125
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 141
Couldn't find a replacement block for slot 163
Couldn't find a replacement block for slot 165
fixed slot 171
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 184
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 203
fixed slot 205
Couldn't find a replacement block for slot 213
Couldn't find a replacement block for slot 215
fixed slot 221
Couldn't find a replacement block for slot 225
fixed slot 226
fixed slot 231
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 242
fixed slot 243
fixed slot 254
Couldn't find a replacement block for slot 255
Couldn't find a replacement block for slot 257
Couldn't find a replacement block for slot 258
Couldn't find a replacement block for slot 229
fixed slot 233
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 153
Couldn't find a replacement block for slot 154
fixed slot 155
Couldn't find a replacement block for slot 182
Couldn't find a replacement block for slot 183
Couldn't find a replacement block for slot 187
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 190
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 192
Couldn't find a replacement block for slot 193
Couldn't find a replacement block for slot 201
Couldn't find a replacement block for slot 228
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 240
Couldn't find a replacement block for slot 242
Couldn't find a replacement block for slot 1
Couldn't find a replacement block for slot 2
Couldn't find a replacement block for slot 3
Couldn't find a replacement block for slot 4
Couldn't find a replacement block for slot 12
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 19
Couldn't find a replacement block for slot 22
fixed slot 45
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
Couldn't find a replacement block for slot 54
Couldn't find a replacement block for slot 59
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 76
Couldn't find a replacement block for slot 78
fixed slot 81
Couldn't find a replacement block for slot 85
Couldn't find a replacement block for slot 87
fixed slot 90
Couldn't find a replacement block for slot 91
Couldn't find a replacement block for slot 92
fixed slot 94
Couldn't find a replacement block for slot 104
fixed slot 170
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 208
fixed slot 210
fixed slot 218
Couldn't find a replacement block for slot 256
Couldn't find a replacement block for slot 271
fixed slot 235
fixed slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 24
Couldn't find a replacement block for slot 25
Couldn't find a replacement block for slot 233
fixed slot 245
Couldn't find a replacement block for slot 246
Couldn't find a replacement block for slot 247
Couldn't find a replacement block for slot 248
Couldn't find a replacement block for slot 249
Couldn't find a replacement block for slot 285
fixed slot 286
Couldn't find a replacement block for slot 287
fixed slot 49
Couldn't find a replacement block for slot 53
fixed slot 63
Couldn't find a replacement block for slot 67
Couldn't find a replacement block for slot 68
Couldn't find a replacement block for slot 70
fixed slot 78
Couldn't find a replacement block for slot 101
Couldn't find a replacement block for slot 102
fixed slot 105
Couldn't find a replacement block for slot 114
fixed slot 122
Couldn't find a replacement block for slot 123
Couldn't find a replacement block for slot 131
Couldn't find a replacement block for slot 133
Couldn't find a replacement block for slot 142
Couldn't find a replacement block for slot 144
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 155
fixed slot 179
fixed slot 196
fixed slot 199
Couldn't find a replacement block for slot 225
Couldn't find a replacement block for slot 259
Couldn't find a replacement block for slot 260
Couldn't find a replacement block for slot 261
fixed slot 29
Couldn't find a replacement block for slot 136
Couldn't find a replacement block for slot 206
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 429
fixed slot 289
Couldn't find a replacement block for slot 195
fixed slot 172
Couldn't find a replacement block for slot 173
Couldn't find a replacement block for slot 295
fixed slot 4
fixed slot 7
Couldn't find a replacement block for slot 11
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
fixed slot 16
Couldn't find a replacement block for slot 35
Couldn't find a replacement block for slot 36
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 40
Couldn't find a replacement block for slot 41
Couldn't find a replacement block for slot 46
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 48
fixed slot 49
Couldn't find a replacement block for slot 154
Couldn't find a replacement block for slot 253
fixed slot 7
Couldn't find a replacement block for slot 189
Couldn't find a replacement block for slot 191
Couldn't find a replacement block for slot 62
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 168
fixed slot 172
Couldn't find a replacement block for slot 180
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 186
Couldn't find a replacement block for slot 203
Couldn't find a replacement block for slot 214
fixed slot 267
fixed slot 268
Couldn't find a replacement block for slot 269
Couldn't find a replacement block for slot 270
Couldn't find a replacement block for slot 280
fixed slot 281
Couldn't find a replacement block for slot 282
fixed slot 13
Couldn't find a replacement block for slot 30
Couldn't find a replacement block for slot 31
Couldn't find a replacement block for slot 69
fixed slot 70
Couldn't find a replacement block for slot 90
Couldn't find a replacement block for slot 111
Couldn't find a replacement block for slot 130
Couldn't find a replacement block for slot 146
Couldn't find a replacement block for slot 147
Couldn't find a replacement block for slot 156
fixed slot 173
Couldn't find a replacement block for slot 174
Couldn't find a replacement block for slot 175
Couldn't find a replacement block for slot 185
Couldn't find a replacement block for slot 188
Couldn't find a replacement block for slot 209
Couldn't find a replacement block for slot 220
Couldn't find a replacement block for slot 221
fixed slot 229
Couldn't find a replacement block for slot 230
Couldn't find a replacement block for slot 231
Couldn't find a replacement block for slot 233
Couldn't find a replacement block for slot 237
Couldn't find a replacement block for slot 239
Couldn't find a replacement block for slot 241
Couldn't find a replacement block for slot 244
fixed slot 3
fixed slot 4
Couldn't find a replacement block for slot 5
Couldn't find a replacement block for slot 7
Couldn't find a replacement block for slot 8
Couldn't find a replacement block for slot 9
Couldn't find a replacement block for slot 13
Couldn't find a replacement block for slot 14
Couldn't find a replacement block for slot 15
Couldn't find a replacement block for slot 16
Couldn't find a replacement block for slot 17
Couldn't find a replacement block for slot 18
fixed slot 25
Couldn't find a replacement block for slot 28
Couldn't find a replacement block for slot 40
fixed slot 41
Couldn't find a replacement block for slot 42
Couldn't find a replacement block for slot 43
Couldn't find a replacement block for slot 44
Couldn't find a replacement block for slot 47
Couldn't find a replacement block for slot 55
fixed slot 60
Couldn't find a replacement block for slot 61
Couldn't find a replacement block for slot 63
Couldn't find a replacement block for slot 66
Couldn't find a replacement block for slot 73
Couldn't find a replacement block for slot 74
fixed slot 78
Couldn't find a replacement block for slot 80
Couldn't find a replacement block for slot 82
Couldn't find a replacement block for slot 92
Couldn't find a replacement block for slot 93
Couldn't find a replacement block for slot 99
fixed slot 103
Couldn't find a replacement block for slot 104
Couldn't find a replacement block for slot 106
fixed slot 109
Couldn't find a replacement block for slot 110
fixed slot 111
fixed slot 120
Couldn't find a replacement block for slot 121
fixed slot 125
Couldn't find a replacement block for slot 126
Couldn't find a replacement block for slot 127
Couldn't find a replacement block for slot 128
fixed slot 301
Couldn't find a replacement block for slot 19
fixed slot 22
fixed slot 28
Couldn't find a replacement block for slot 29
Couldn't find a replacement block for slot 30
fixed slot 31
Couldn't find a replacement block for slot 38
Couldn't find a replacement block for slot 39
fixed slot 48
Couldn't find a replacement block for slot 51
fixed slot 53
Couldn't find a replacement block for slot 55
Couldn't find a replacement block for slot 56
fixed slot 62
fixed slot 77
Couldn't find a replacement block for slot 78
Couldn't find a replacement bl (waiting for more output)

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#444 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Josef Bacik
Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)

Couldn't find a replacement block for slot 361
Couldn't find a replacement block for slot 363
fixed slot 372
Couldn't find a replacement block for slot 374
Couldn't find a replacement block for slot 375
Couldn't find a replacement block for slot 403
Couldn't find a replacement block for slot 404
Couldn't find a replacement block for slot 405
Couldn't find a replacement block for slot 406
Couldn't find a replacement block for slot 407
Couldn't find a replacement block for slot 408
Couldn't find a replacement block for slot 409
Couldn't find a replacement block for slot 413
Couldn't find a replacement block for slot 414
Couldn't find a replacement block for slot 415
Couldn't find a replacement block for slot 416
Couldn't find a replacement block for slot 417
Couldn't find a replacement block for slot 418
Couldn't find a replacement block for slot 419
Couldn't find a replacement block for slot 420
Couldn't find a replacement block for slot 421
Couldn't find a replacement block for slot 422
Couldn't find a replacement block for slot 423
Couldn't find a replacement block for slot 424
Couldn't find a replacement block for slot 427
Couldn't find a replacement block for slot 437
Couldn't find a replacement block for slot 438
Couldn't find a replacement block for slot 443
Couldn't find a replacement block for slot 444
Couldn't find a replacement block for slot 445
fixed slot 446
fixed slot 447
gargamel:/var/local/src/btrfs-progs-josefbacik#

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1  
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x5641ceb95fd0
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8
parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x5641ceb95fd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
Creating a new extent tree
checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0 checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718 checksum verify failed on 15645365747712 wanted 0x3bdf1829 found 0xdbfcbd25 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 bad tree block 15645365747712, bad level, 240 > 8
Error reading tree block owner 7
error pinning down used bytes
ERROR: commit_root already set when starting transaction
extent buffer leak: start 13577814573056 len 16384 gargamel:/var/local/src/btrfs-progs-josefbacik#

gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro /dev/mapper/dshelf1 /mnt/mnt
mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error.
gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail
[106691.309851] BTRFS info (device dm-1): disk space caching is enabled
[106691.363587] BTRFS info (device dm-1): has skinny extents
[106691.394712] verify_parent_transid: 4 callbacks suppressed
[106691.394716] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[106691.443339] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177
[106691.474616] BTRFS error (device dm-1): failed to read chunk tree: -5
[106691.501647] BTRFS error (device dm-1): open_ctree failed
[106691.519613] systemd-udevd[31783]: btrfs-19: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1.
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#445 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)

Yup we're getting there, I'm about 90% done with the new stuff, and that'll delete block pointers that we can't find good matches for.

Also it'll cache the results of the disk wide search, so it'll have to do it once per root instead of for every bad block we find, which will drastically cut down on the runtime. I'll let you know when it's ready, figure another hour or so. Thanks,

Josef

#446 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks,

It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :)

Quoted 8 lines

Yup we're getting there, I'm about 90% done with the new stuff, and that'll delete block pointers that we can't find good matches for.

Also it'll cache the results of the disk wide search, so it'll have to do it once per root instead of for every bad block we find, which will drastically cut down on the runtime. I'll let you know when it's ready, figure another hour or so. Thanks,

Course the last 10% takes the longest, but I corrupted a local file system and ran it to shake out all the stupid bugs. Go ahead and pull and run

./btrfs rescue tree-recover /dev/whatever

and then *hopefully* you can just run btrfs check --repair, but it may fail out with a "btrfs unable to find ref byte", which is what I was seeing locally. I'm fixing that but it's tricky and may be a while. If you hit that then go ahead and use --init-extent-tree --init-csum-tree and let it ride. Thanks,

Josef

#447 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik

Course the last 10% takes the longest, but I corrupted a local file system and ran it to shake out all the stupid bugs. Go ahead and pull and run

./btrfs rescue tree-recover /dev/whatever

and then *hopefully* you can just run btrfs check --repair, but it may fail out with a "btrfs unable to find ref byte", which is what I was seeing locally. I'm fixing that but it's tricky and may be a while. If you hit that then go ahead and use --init-extent-tree --init-csum-tree and let it ride. Thanks,

Sorry for the delay, too many things this weekend and I didn't do Email.

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5
commit 580e1b4db52a0ec98661bec7310450d84ea1f441
Author: Josef Bacik <[email protected]>
Date:   Fri Apr 8 16:07:48 2022 -0400

handle fstree's differently

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
Ignoring transid failure
FS_INFO IS 0x558174d9cbc0
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Couldn't find the last root for 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x558174d9cbc0
deleting slot 94 in block 21069824
FS_INFO IS 0x558174e1f430
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Ignoring transid failure
Couldn't find the last root for 8
ERROR: failed to read block groups: Input/output error
FS_INFO AFTER IS 0x558174e1f430
Found completely clean tree for 3 in backup root, replacing
ERROR: Still have update after a repair loop, bailing
Tree recover failed

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#68 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 120 lines · Marc MERLIN

Back on list, Josef made a lot of changes to btrfs-progs for me (thanks Josef)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55c6d219bbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55c6d219bbc0
Checking root 2
Checking root 4
Checking root 5
Checking root 7
Checking root 9
Checking root 11221
Checking root 11222
Checking root 11223

After that ./btrfs check --init-extent-tree /dev/mapper/dshelf1 has been running more for than 48H now

Starting repair.
Opening filesystem to check...
checksum verify failed on 15645878108160 found 00000027 wanted 0000001B
checksum verify failed on 11821979287552 found 000000A9 wanted FFFFFFFB
checksum verify failed on 11822142046208 found 000000AC wanted FFFFFFC4
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Failed to find [13576823668736, 168, 16384]
btrfs unable to find ref byte nr 13577801809920 parent 0 root 1 owner 0 offset 0 path->slots[0]: 134 path->nodes[0]:
leaf 49709056 items 169 free space 7975 generation 1602090 owner EXTENT_TREE
leaf 49709056 flags 0x0() backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
item 0 key (13400696422400 BLOCK_GROUP_ITEM 1073741824) itemoff 16259 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 1 key (13401770164224 BLOCK_GROUP_ITEM 1073741824) itemoff 16235 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
(...)
Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
(...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
ref mismatch on [12582912 4096] extent item 0, found 1
data backref 12582912 root 11223 owner 260 offset 131072 num_refs 0 not found in extent tree incorrect local backref count on 12582912 root 11223 owner 260 offset 131072 found 1 wanted 0 back 0x5604b40697a0 backpointer mismatch on [12582912 4096]
adding new data backref on 12582912 root 11223 owner 260 offset 131072 found 1
Repaired extent references for 12582912
ref mismatch on [12587008 4096] extent item 0, found 1
data backref 12587008 root 11223 owner 261 offset 20480 num_refs 0 not found in extent tree incorrect local backref count on 12587008 root 11223 owner 261 offset 20480 found 1 wanted 0 back 0x5604b4069a00 backpointer mismatch on [12587008 4096]
adding new data backref on 12587008 root 11223 owner 261 offset 20480 found 1
Repaired extent references for 12587008
ref mismatch on [12591104 4096] extent item 0, found 1
(...)
incorrect local backref count on 20963328 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6d80 backpointer mismatch on [20963328 4096]
adding new data backref on 20963328 parent 33718272 owner 0 offset 0 found 1
Repaired extent references for 20963328
ref mismatch on [20967424 4096] extent item 0, found 1
data backref 20967424 parent 33718272 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 20967424 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6fe0 backpointer mismatch on [20967424 4096]
adding new data backref on 20967424 parent 33718272 owner 0 offset 0 found 1
Repaired extent references for 20967424
ref mismatch on [20971520 16384] extent item 0, found 1
tree backref 20971520 root 3 not found in extent tree
backpointer mismatch on [20971520 16384]
adding new tree backref on start 20971520 len 16384 parent 0 root 3
Repaired extent references for 20971520
ref mismatch on [20987904 16384] extent item 0, found 1
tree backref 20987904 root 3 not found in extent tree
backpointer mismatch on [20987904 16384]
adding new tree backref on start 20987904 len 16384 parent 0 root 3
Repaired extent references for 20987904
ref mismatch on [21004288 16384] extent item 0, found 1
tree backref 21004288 root 3 not found in extent tree
(..)
adding new tree backref on start 44531712 len 16384 parent 50872320 root 50872320
Repaired extent references for 44531712
ref mismatch on [44548096 16384] extent item 0, found 1
tree backref 44548096 parent 47398912 not found in extent tree
backpointer mismatch on [44548096 16384]
adding new tree backref on start 44548096 len 16384 parent 47398912 root 47398912
Repaired extent references for 44548096
ref mismatch on [44564480 16384] extent item 0, found 1
tree backref 44564480 root 7 not found in extent tree
backpointer mismatch on [44564480 16384]
adding new tree backref on start 44564480 len 16384 parent 0 root 7
Repaired extent references for 44564480
ref mismatch on [44580864 16384] extent item 0, found 1
tree backref 44580864 root 7 not found in extent tree
backpointer mismatch on [44580864 16384]
adding new tree backref on start 44580864 len 16384 parent 0 root 7
Repaired extent references for 44580864
ref mismatch on [44597248 16384] extent item 0, found 1
tree backref 44597248 root 7 not found in extent tree
backpointer mismatch on [44597248 16384]
adding new tree backref on start 44597248 len 16384 parent 0 root 7
Repaired extent references for 44597248
ref mismatch on [44613632 16384] extent item 0, found 1
tree backref 44613632 parent 47398912 not found in extent tree
backpointer mismatch on [44613632 16384]
adding new tree backref on start 44613632 len 16384 parent 47398912 root 47398912
Repaired extent references for 44613632
ref mismatch on [44630016 16384] extent item 0, found 1
tree backref 44630016 parent 47398912 not found in extent tree
backpointer mismatch on [44630016 16384]
adding new tree backref on start 44630016 len 16384 parent 47398912 root 47398912
(...)

Quoted 9 lines
gargamel:/var/local/src# grep -c 'adding new tree backref on start' checkrepair1
9026

It's been running for close to 3 days, and I'm a bit confused that it's repairing so many things when it wast just a minute's worth of potential corruption.

Do I keep running for many more days to see where this goes, or at this point 73,860 lines of output is not a bad sign and I should look at restoring from backup?

We're doing the --init-extent-tree thing, which deletes the whole extent tree and rebuilds it. This isn't the fast path, 3 days is super shitty tho. I don't want to stop it and try and make it faster because once it's done you should just be golden. It looks scary, but that's because it just clears the extent tree and then lets the normal repair thing do its thing, so it yells loudly about all the messed up references and then fixes it.

Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks,

Josef

#69 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Josef Bacik

We're doing the --init-extent-tree thing, which deletes the whole extent tree and rebuilds it. This isn't the fast path, 3 days is super shitty tho. I don't want to stop it and try and make it faster because once it's done you should just be golden. It looks scary, but that's because it just clears the extent tree and then lets the normal repair thing do its thing, so it yells loudly about all the messed up references and then fixes it.

Got it.

Quoted 4 lines

Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks,

Makes sense. I don't need faster, and it may not be able to go faster anyway, it's a lot of data. Just wanted to make sure the output and relative slow results were expected.
And I'll go with slower but likely to work vs faster but very experimental :)

That said, as Ted T'so said once, "anything with a progress bar is faster".
So if it's possible in the future to add some completion percent, even if approximate, that will help when it can take multiple days (or week+) to run.

Happy to wait in the meantime, thanks.
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#67 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN

Back on list, Josef made a lot of changes to btrfs-progs for me (thanks Josef)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55c6d219bbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55c6d219bbc0
Checking root 2
Checking root 4
Checking root 5
Checking root 7
Checking root 9
Checking root 11221
Checking root 11222
Checking root 11223

After that ./btrfs check --init-extent-tree /dev/mapper/dshelf1 has been running more for than 48H now

Starting repair.
Opening filesystem to check...
checksum verify failed on 15645878108160 found 00000027 wanted 0000001B
checksum verify failed on 11821979287552 found 000000A9 wanted FFFFFFFB
checksum verify failed on 11822142046208 found 000000AC wanted FFFFFFC4
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Failed to find [13576823668736, 168, 16384]
btrfs unable to find ref byte nr 13577801809920 parent 0 root 1 owner 0 offset 0 path->slots[0]: 134 path->nodes[0]:
leaf 49709056 items 169 free space 7975 generation 1602090 owner EXTENT_TREE
leaf 49709056 flags 0x0() backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a
item 0 key (13400696422400 BLOCK_GROUP_ITEM 1073741824) itemoff 16259 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 1 key (13401770164224 BLOCK_GROUP_ITEM 1073741824) itemoff 16235 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
(...)
Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent
Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent
Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk.
Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk.
Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk.
Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk.
Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk.
(...)
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
ref mismatch on [12582912 4096] extent item 0, found 1
data backref 12582912 root 11223 owner 260 offset 131072 num_refs 0 not found in extent tree incorrect local backref count on 12582912 root 11223 owner 260 offset 131072 found 1 wanted 0 back 0x5604b40697a0 backpointer mismatch on [12582912 4096]
adding new data backref on 12582912 root 11223 owner 260 offset 131072 found 1
Repaired extent references for 12582912
ref mismatch on [12587008 4096] extent item 0, found 1
data backref 12587008 root 11223 owner 261 offset 20480 num_refs 0 not found in extent tree incorrect local backref count on 12587008 root 11223 owner 261 offset 20480 found 1 wanted 0 back 0x5604b4069a00 backpointer mismatch on [12587008 4096]
adding new data backref on 12587008 root 11223 owner 261 offset 20480 found 1
Repaired extent references for 12587008
ref mismatch on [12591104 4096] extent item 0, found 1
(...)
incorrect local backref count on 20963328 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6d80 backpointer mismatch on [20963328 4096]
adding new data backref on 20963328 parent 33718272 owner 0 offset 0 found 1
Repaired extent references for 20963328
ref mismatch on [20967424 4096] extent item 0, found 1
data backref 20967424 parent 33718272 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 20967424 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6fe0 backpointer mismatch on [20967424 4096]
adding new data backref on 20967424 parent 33718272 owner 0 offset 0 found 1
Repaired extent references for 20967424
ref mismatch on [20971520 16384] extent item 0, found 1
tree backref 20971520 root 3 not found in extent tree
backpointer mismatch on [20971520 16384]
adding new tree backref on start 20971520 len 16384 parent 0 root 3
Repaired extent references for 20971520
ref mismatch on [20987904 16384] extent item 0, found 1
tree backref 20987904 root 3 not found in extent tree
backpointer mismatch on [20987904 16384]
adding new tree backref on start 20987904 len 16384 parent 0 root 3
Repaired extent references for 20987904
ref mismatch on [21004288 16384] extent item 0, found 1
tree backref 21004288 root 3 not found in extent tree
(..)
adding new tree backref on start 44531712 len 16384 parent 50872320 root 50872320
Repaired extent references for 44531712
ref mismatch on [44548096 16384] extent item 0, found 1
tree backref 44548096 parent 47398912 not found in extent tree
backpointer mismatch on [44548096 16384]
adding new tree backref on start 44548096 len 16384 parent 47398912 root 47398912
Repaired extent references for 44548096
ref mismatch on [44564480 16384] extent item 0, found 1
tree backref 44564480 root 7 not found in extent tree
backpointer mismatch on [44564480 16384]
adding new tree backref on start 44564480 len 16384 parent 0 root 7
Repaired extent references for 44564480
ref mismatch on [44580864 16384] extent item 0, found 1
tree backref 44580864 root 7 not found in extent tree
backpointer mismatch on [44580864 16384]
adding new tree backref on start 44580864 len 16384 parent 0 root 7
Repaired extent references for 44580864
ref mismatch on [44597248 16384] extent item 0, found 1
tree backref 44597248 root 7 not found in extent tree
backpointer mismatch on [44597248 16384]
adding new tree backref on start 44597248 len 16384 parent 0 root 7
Repaired extent references for 44597248
ref mismatch on [44613632 16384] extent item 0, found 1
tree backref 44613632 parent 47398912 not found in extent tree
backpointer mismatch on [44613632 16384]
adding new tree backref on start 44613632 len 16384 parent 47398912 root 47398912
Repaired extent references for 44613632
ref mismatch on [44630016 16384] extent item 0, found 1
tree backref 44630016 parent 47398912 not found in extent tree
backpointer mismatch on [44630016 16384]
adding new tree backref on start 44630016 len 16384 parent 47398912 root 47398912
(...)

gargamel:/var/local/src# grep -c 'adding new tree backref on start' checkrepair1 
9026

It's been running for close to 3 days, and I'm a bit confused that it's repairing so many things when it wast just a minute's worth of potential corruption.

Do I keep running for many more days to see where this goes, or at this point 73,860 lines of output is not a bad sign and I should look at restoring from backup?

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#70 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to Marc MERLIN
Quoted 8 lines · Marc MERLIN
Quoted 4 lines

Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks,

Makes sense. I don't need faster, and it may not be able to go faster anyway, it's a lot of data. Just wanted to make sure the output and relative slow results were expected.

Looking at the output, is there any way I can figure out if it's at 5% or 80% completion?

tree backref 238026752 parent 236814336 not found in extent tree backpointer mismatch on [238026752 16384]
adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 Repaired extent references for 238026752
ref mismatch on [238043136 16384] extent item 0, found 1
tree backref 238043136 parent 236814336 not found in extent tree backpointer mismatch on [238043136 16384]
adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 Repaired extent references for 238043136
ref mismatch on [238059520 16384] extent item 0, found 1
tree backref 238059520 parent 236814336 not found in extent tree backpointer mismatch on [238059520 16384]
adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#71 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 14 lines · Marc MERLIN
Quoted 8 lines · Marc MERLIN
Quoted 4 lines

Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks,

Makes sense. I don't need faster, and it may not be able to go faster anyway, it's a lot of data. Just wanted to make sure the output and relative slow results were expected.

Looking at the output, is there any way I can figure out if it's at 5% or 80% completion?

Quoted 14 lines

tree backref 238026752 parent 236814336 not found in extent tree backpointer mismatch on [238026752 16384]
adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 Repaired extent references for 238026752
ref mismatch on [238043136 16384] extent item 0, found 1
tree backref 238043136 parent 236814336 not found in extent tree backpointer mismatch on [238043136 16384]
adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 Repaired extent references for 238043136
ref mismatch on [238059520 16384] extent item 0, found 1
tree backref 238059520 parent 236814336 not found in extent tree backpointer mismatch on [238059520 16384]
adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336

Hmm I don't know, that's byte 227, but you're in fs trees now, so hopefully soon? I've got some free time, let me rewrite this to be less stupid and see if I can get it done before your thing finishes, and I'll add some sort of progress indicator. Thanks,

Josef

#72 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 31 lines · Josef Bacik
Quoted 14 lines · Marc MERLIN
Quoted 8 lines · Marc MERLIN
Quoted 4 lines

Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks,

Makes sense. I don't need faster, and it may not be able to go faster anyway, it's a lot of data. Just wanted to make sure the output and relative slow results were expected.

Looking at the output, is there any way I can figure out if it's at 5% or 80% completion?

Quoted 14 lines

tree backref 238026752 parent 236814336 not found in extent tree backpointer mismatch on [238026752 16384]
adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 Repaired extent references for 238026752
ref mismatch on [238043136 16384] extent item 0, found 1
tree backref 238043136 parent 236814336 not found in extent tree backpointer mismatch on [238043136 16384]
adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 Repaired extent references for 238043136
ref mismatch on [238059520 16384] extent item 0, found 1
tree backref 238059520 parent 236814336 not found in extent tree backpointer mismatch on [238059520 16384]
adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336

Quoted 5 lines

Hmm I don't know, that's byte 227, but you're in fs trees now, so hopefully soon? I've got some free time, let me rewrite this to be less stupid and see if I can get it done before your thing finishes, and I'll add some sort of progress indicator. Thanks,

Alright you can kill the command, pull my tree, then run

btrfs rescue init-extent-tree <blah>

This do the straight re-init of the extent tree without all the looping and extra stuff that fsck does, so hopefully will be faster? Also it keeps track of how many bytes we've processed, as well as how many bytes we may have to process, so you'll get a better feel for the progress. I have a scratch fs here that has like 30gib of metadata with 1tib of data and it took around 30 seconds to run. It has no snapshots, I ran it on a fsstress generated fs with like 5k snapshots and it took about a minute, but with significantly less metadata/data.

Once you run this you'll still have to run --repair, but it shouldn't have to touch the extent tree. If you see it messing with the extent tree stop it and let me know what it's complaining about and I'll fix it up. I'm pretty sure my thing is going to work, but your fs is more complex than I can generate on the fly.

Also, the indicator may show "processed" as less than "possible" bytes when it moves onto the next root, this is normal for snapshots, if we've already processed a subtree we won't walk down it, so it's fine if you see that. Thanks,

Josef

#73 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 38 lines · Josef Bacik
Quoted 31 lines · Josef Bacik
Quoted 14 lines · Marc MERLIN
Quoted 8 lines · Marc MERLIN
Quoted 4 lines

Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks,

Makes sense. I don't need faster, and it may not be able to go faster anyway, it's a lot of data. Just wanted to make sure the output and relative slow results were expected.

Looking at the output, is there any way I can figure out if it's at 5% or 80% completion?

Quoted 14 lines

tree backref 238026752 parent 236814336 not found in extent tree backpointer mismatch on [238026752 16384]
adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 Repaired extent references for 238026752
ref mismatch on [238043136 16384] extent item 0, found 1
tree backref 238043136 parent 236814336 not found in extent tree backpointer mismatch on [238043136 16384]
adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 Repaired extent references for 238043136
ref mismatch on [238059520 16384] extent item 0, found 1
tree backref 238059520 parent 236814336 not found in extent tree backpointer mismatch on [238059520 16384]
adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336

Quoted 5 lines

Hmm I don't know, that's byte 227, but you're in fs trees now, so hopefully soon? I've got some free time, let me rewrite this to be less stupid and see if I can get it done before your thing finishes, and I'll add some sort of progress indicator. Thanks,

Quoted 3 lines

Alright you can kill the command, pull my tree, then run

btrfs rescue init-extent-tree <blah>

Thanks. Tried the new version

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55bf50e98bc0
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55bf50e98bc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found
0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found
0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
parent transid verify failed on 58736640 wanted 1619007 found 1619009
processed 16384 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
processed 32768 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58753024 wanted 1619007 found 1619009
parent transid verify failed on 58753024 wanted 1619007 found 1619009
processed 49152 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
corrupt leaf: root=2 block=58736640 physical=1140867072 slot=107, bad
key order, prev (169143697408 192 1073741824) current (12582912 192
8388608)
kernel-shared/extent-tree.c:1287: btrfs_inc_extent_ref: BUG_ON `err`
triggered, value -1
./btrfs(+0x29683)[0x55bf4fea1683]
./btrfs(btrfs_inc_extent_ref+0x522)[0x55bf4fea3061]
./btrfs(+0x8ba18)[0x55bf4ff03a18]
./btrfs(+0x8bb27)[0x55bf4ff03b27]
./btrfs(+0x8bd8f)[0x55bf4ff03d8f]
./btrfs(btrfs_init_extent_tree+0x226)[0x55bf4ff04092]
./btrfs(+0x83931)[0x55bf4fefb931]
./btrfs(handle_command_group+0x49)[0x55bf4fe9017b]
./btrfs(main+0x94)[0x55bf4fe90275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7faebe73a7fd]
./btrfs(_start+0x2a)[0x55bf4fe8fe1a]
Aborted

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#74 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 94 lines · Marc MERLIN
Quoted 38 lines · Josef Bacik
Quoted 31 lines · Josef Bacik
Quoted 14 lines · Marc MERLIN
Quoted 8 lines · Marc MERLIN
Quoted 4 lines

Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks,

Makes sense. I don't need faster, and it may not be able to go faster anyway, it's a lot of data. Just wanted to make sure the output and relative slow results were expected.

Looking at the output, is there any way I can figure out if it's at 5% or 80% completion?

Quoted 14 lines

tree backref 238026752 parent 236814336 not found in extent tree backpointer mismatch on [238026752 16384]
adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 Repaired extent references for 238026752
ref mismatch on [238043136 16384] extent item 0, found 1
tree backref 238043136 parent 236814336 not found in extent tree backpointer mismatch on [238043136 16384]
adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 Repaired extent references for 238043136
ref mismatch on [238059520 16384] extent item 0, found 1
tree backref 238059520 parent 236814336 not found in extent tree backpointer mismatch on [238059520 16384]
adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336

Quoted 5 lines

Hmm I don't know, that's byte 227, but you're in fs trees now, so hopefully soon? I've got some free time, let me rewrite this to be less stupid and see if I can get it done before your thing finishes, and I'll add some sort of progress indicator. Thanks,

Quoted 3 lines

Alright you can kill the command, pull my tree, then run

btrfs rescue init-extent-tree <blah>

Thanks. Tried the new version

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x55bf50e98bc0
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55bf50e98bc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found
0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found
0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
parent transid verify failed on 58736640 wanted 1619007 found 1619009
processed 16384 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
processed 32768 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58753024 wanted 1619007 found 1619009
parent transid verify failed on 58753024 wanted 1619007 found 1619009
processed 49152 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
parent transid verify failed on 58736640 wanted 1619007 found 1619009
corrupt leaf: root=2 block=58736640 physical=1140867072 slot=107, bad
key order, prev (169143697408 192 1073741824) current (12582912 192
8388608)
kernel-shared/extent-tree.c:1287: btrfs_inc_extent_ref: BUG_ON `err`
triggered, value -1
./btrfs(+0x29683)[0x55bf4fea1683]
./btrfs(btrfs_inc_extent_ref+0x522)[0x55bf4fea3061]
./btrfs(+0x8ba18)[0x55bf4ff03a18]
./btrfs(+0x8bb27)[0x55bf4ff03b27]
Quoted 8 lines
./btrfs(+0x8bd8f)[0x55bf4ff03d8f]
./btrfs(btrfs_init_extent_tree+0x226)[0x55bf4ff04092]
./btrfs(+0x83931)[0x55bf4fefb931]
./btrfs(handle_command_group+0x49)[0x55bf4fe9017b]
./btrfs(main+0x94)[0x55bf4fe90275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7faebe73a7fd]
./btrfs(_start+0x2a)[0x55bf4fe8fe1a]
Aborted

Argh you still have bogus stuff for some reason, maybe it's because of cancelling the rescue. Re-run btrfs rescue tree-recover, and do it a couple of times to make sure it's stopped complaining. If it does complain after the first run please let me know, it really should fix everything the first go around. If it's not then I need to try and reproduce that locally and figure out wtf it's doing. Once tree-recover runs cleanly you can run btrfs rescue init-extent-tree, then hopefully you'll be good to go. Thanks,

Josef

#75 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik

Argh you still have bogus stuff for some reason, maybe it's because of cancelling the rescue. Re-run btrfs rescue tree-recover, and do it a couple of times to make sure it's stopped complaining. If it does complain after the first run please let me know, it really should fix everything the first go around. If it's not then I need to try and reproduce that locally and figure out wtf it's doing. Once tree-recover runs cleanly you can run btrfs rescue init-extent-tree, then hopefully you'll be good to go. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x555a360efbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555a360efbc0
Checking root 2
Checking root 4
Checking root 5
Checking root 7
Checking root 9
Checking root 11221
Checking root 11222
Checking root 11223
Checking root 11224
Checking root 159785
Checking root 159787
Checking root 160494
Checking root 160496
Checking root 161197
Checking root 161199
Checking root 162628
Checking root 162632
Checking root 162645
Checking root 163298
Checking root 163302
Checking root 163303
Checking root 163316
Checking root 163318
Checking root 163916
Checking root 163920
Checking root 163921
Checking root 164620
Checking root 164624
Checking root 164633
Checking root 165098
Checking root 165100
Checking root 165198
Checking root 165200
Checking root 165294
Checking root 165298
Checking root 165299
Checking root 18446744073709551607
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x56080e544bc0
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56080e544bc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed
(...)
checksum verify failed on 13577810395136 wanted 0x4171ba0a found 0xda4963ab
checksum verify failed on 46576713728 wanted 0xbb1ee48e found 0x8809f6a6
checksum verify failed on 13577126330368 wanted 0xf7264639 found 0x8287ec94
checksum verify failed on 15645432397824 wanted 0xc706f8b6 found 0xf0425483
checksum verify failed on 15645954080768 wanted 0x3ff10ddf found 0x77ab2f14
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645262348288 wanted 0x433325c8 found 0x81ea056f
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 15646053974016 wanted 0x98bd62bc found 0xdaa7dd44
checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95
checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c
checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2
checksum verify failed on 12511656394752 wanted 0x607d7b9e found 0xec111bf5
checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]
Segmentation fault

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559add0 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559add0 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48

#1 0x000055555559afd4 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640)
at common/extent-cache.c:179
#2 0x0000555555584b67 in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584d9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x000055555558577d in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bbaf in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2222
#6  0x000055555557ed74 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434045, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b82ed0, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434045, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b82ed0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261
#9 0x00005555555dfa18 in process_eb (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcb20,

    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499
#10 0x00005555555dfb27 in process_eb (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, eb=0x55555566b790, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574

#11 0x00005555555dfd8f in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645 #12 0x00005555555e0092 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:779
#13 0x00005555555d7931 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#76 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 48 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik

Argh you still have bogus stuff for some reason, maybe it's because of cancelling the rescue. Re-run btrfs rescue tree-recover, and do it a couple of times to make sure it's stopped complaining. If it does complain after the first run please let me know, it really should fix everything the first go around. If it's not then I need to try and reproduce that locally and figure out wtf it's doing. Once tree-recover runs cleanly you can run btrfs rescue init-extent-tree, then hopefully you'll be good to go. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x555a360efbc0
Couldn't find the last root for 8
FS_INFO AFTER IS 0x555a360efbc0
Checking root 2
Checking root 4
Checking root 5
Checking root 7
Checking root 9
Checking root 11221
Checking root 11222
Checking root 11223
Checking root 11224
Checking root 159785
Checking root 159787
Checking root 160494
Checking root 160496
Checking root 161197
Checking root 161199
Checking root 162628
Checking root 162632
Checking root 162645
Checking root 163298
Checking root 163302
Checking root 163303
Checking root 163316
Checking root 163318
Checking root 163916
Checking root 163920
Checking root 163921
Checking root 164620
Checking root 164624
Checking root 164633
Checking root 165098
Checking root 165100
Checking root 165198
Checking root 165200
Quoted 8 lines
Checking root 165294
Checking root 165298
Checking root 165299
Checking root 18446744073709551607
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x56080e544bc0
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4

Ok this is sort of maddening, IDK how we'll trip over this broken block here, but not with tree-recover. Can you repull, build, and then run init-extent-tree. As soon as you see the JOSEF message you can kill it and send me the output, I need to figure out wtf is going on here. Thanks,

Josef

#77 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Josef Bacik

Ok this is sort of maddening, IDK how we'll trip over this broken block here, but not with tree-recover. Can you repull, build, and then run init-extent-tree. As soon as you see the JOSEF message you can kill it and send me the output, I need to figure out wtf is going on here. Thanks,

Mmmh, not sure I got that far.

gargamel:/var/local/src/btrfs-progs-josefbacik# grep -r JOSEF *
Binary file btrfs matches
Binary file btrfsck matches
Binary file btrfs-convert matches
Binary file btrfs-corrupt-block matches
Binary file btrfs-find-root matches
Binary file btrfs-image matches
Binary file btrfs-map-logical matches
Binary file btrfs-select-super matches
Binary file btrfstune matches
kernel-shared/ctree.c:          printf("JOSEF: read node slot on root %llu\n", root->root_key.objectid);
Binary file kernel-shared/ctree.o matches
kernel-shared/disk-io.c:                printf("JOSEF: root %llu\n", root->root_key.objectid);
Binary file kernel-shared/disk-io.o matches
Binary file mkfs.btrfs matches

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x89f5cb13 found 0x04196d22
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
(...)
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48

#1 0x000055555559b05c in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640)
at common/extent-cache.c:179
#2 0x0000555555584bea in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e1d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585800 in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bbf3 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224
#6  0x000055555557edf7 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b83090, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b83090, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261
#9 0x00005555555dfaad in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcfa0,

    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499
#10 0x00005555555dfbbc in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=0x55555566b790, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574

#11 0x00005555555dfe24 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645 #12 0x00005555555e0127 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:779
#13 0x00005555555d79c6 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#78 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 27 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik

Ok this is sort of maddening, IDK how we'll trip over this broken block here, but not with tree-recover. Can you repull, build, and then run init-extent-tree. As soon as you see the JOSEF message you can kill it and send me the output, I need to figure out wtf is going on here. Thanks,

Mmmh, not sure I got that far.

gargamel:/var/local/src/btrfs-progs-josefbacik# grep -r JOSEF *
Binary file btrfs matches
Binary file btrfsck matches
Binary file btrfs-convert matches
Binary file btrfs-corrupt-block matches
Binary file btrfs-find-root matches
Binary file btrfs-image matches
Binary file btrfs-map-logical matches
Binary file btrfs-select-super matches
Binary file btrfstune matches
kernel-shared/ctree.c:          printf("JOSEF: read node slot on root %llu\n", root->root_key.objectid);
Binary file kernel-shared/ctree.o matches
kernel-shared/disk-io.c:                printf("JOSEF: root %llu\n", root->root_key.objectid);
Binary file kernel-shared/disk-io.o matches
Binary file mkfs.btrfs matches
Quoted 6 lines

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9

Huh ok, it's the UUID tree, weird. I pushed, can you re-run tree-recover, you can stop it after it does root 9, I just want to see what bytenr it thinks the root node is at. Thanks,

Josef

#79 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Josef Bacik
Quoted 6 lines

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9

Huh ok, it's the UUID tree, weird. I pushed, can you re-run tree-recover, you can stop it after it does root 9, I just want to see what bytenr it thinks the root node is at. Thanks,

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed
checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c
checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0
checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718
checksum verify failed on 11970896232448 wanted 0x8b800530 found 0x3790ebce
checksum verify failed on 8733701898240 wanted 0x80eed7c5 found 0xd6af2972
(..)
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Clearing the extent root and re-init'ing the block groups
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48

#1 0x000055555559b05c in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640)
at common/extent-cache.c:179
#2 0x0000555555584bea in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e1d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585800 in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bbf3 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224
#6  0x000055555557edf7 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b928c0, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b928c0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261
#9 0x00005555555dfb31 in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcbe0,

    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499
#10 0x00005555555dfc40 in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=0x55555566b790, 
    current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574

#11 0x00005555555dfea8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645 #12 0x00005555555e01ab in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:779
#13 0x00005555555d79c6 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#80 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 6 lines

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9

Quoted 6 lines
Quoted 3 lines

Huh ok, it's the UUID tree, weird. I pushed, can you re-run tree-recover, you can stop it after it does root 9, I just want to see what bytenr it thinks the root node is at. Thanks,

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1

Sorry, I need tree-recover, not init-extent-tree. Thanks,

Josef

#81 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Josef Bacik
Quoted 9 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 6 lines

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Huh ok, it's the UUID tree, weird. I pushed, can you re-run tree-recover, you can stop it after it does root 9, I just want to see what bytenr it thinks the root node is at. Thanks,

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1

Sorry, I need tree-recover, not init-extent-tree. Thanks,

gdb) run rescue tree-recover /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Checking root 2 bytenr 67387392
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 13577819963392
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576862547968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781491265536
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781492101120
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781492756480
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781495197696
Checking root 163303 bytenr 15645365993472
Checking root 163316 bytenr 6781496393728
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11971021275136
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970667446272
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823685120 Tree recovery finished, you can run check now
[Inferior 1 (process 6147) exited normally]
(gdb)

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#82 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 16 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 6 lines

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9

Huh ok, it's the UUID tree, weird. I pushed, can you re-run tree-recover, you can stop it after it does root 9, I just want to see what bytenr it thinks the root node is at. Thanks,

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1

Sorry, I need tree-recover, not init-extent-tree. Thanks,

gdb) run rescue tree-recover /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1

Wtf, it's reading the right bytenr, but it's not failing here but failing when we do the init-extent-tree. I've pushed something again to force reads, maybe that's the problem, can you run tree-recover and then init-extent-tree again? Thanks,

Josef

#87 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

In reply to a message not in the archive
Quoted 17 lines · Josef Bacik
Quoted 16 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 10 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Wtf, it's reading the right bytenr, but it's not failing here but failing when we do the init-extent-tree. I've pushed something again to force reads, maybe that's the problem, can you run tree-recover and then init-extent-tree again? Thanks,

off-list:
Seeing this, that kind of worried me so I stopped it. Is it ok to continue?

Sorry, my printf was wrong, pushed. Thanks,

Quoted 8 lines
Quoted 3 lines

Better?
Do I worry about those new "had to be read from a different mirror" ?

Ooooh ok, the other mirror is bad, but it finds the right thing. Ok that makes sense. I pushed another patch just to make sure. Once this is all done you'll need to run a scrub to fix the mirrors but this is fine. Thanks,

No more crash, but:
ERROR: Error adding block group
ERROR: commit_root already set when starting transaction

(gdb) run rescue tree-recover /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
block 15645878108160 had to be read from a different mirror, ret 0
Couldn't find the last root for 8
block 58720256 had to be read from a different mirror, ret 0
block 58720256 had to be read from a different mirror, ret 0
FS_INFO AFTER IS 0x55555564cbc0
block 13577220816896 had to be read from a different mirror, ret 0
block 13577870934016 had to be read from a different mirror, ret 0
(...)
block 13577821011968 had to be read from a different mirror, ret 0
block 15645781196800 had to be read from a different mirror, ret 0
Checking root 2 bytenr 67387392
block 58720256 had to be read from a different mirror, ret 0
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
block 11970891186176 had to be read from a different mirror, ret 0
block 12511729680384 had to be read from a different mirror, ret 0
(...)
block 13577726541824 had to be read from a different mirror, ret 0
block 13577776611328 had to be read from a different mirror, ret 0
block 364637798400 had to be read from a different mirror, ret 0
block 364637962240 had to be read from a different mirror, ret 0
Checking root 7 bytenr 13577819963392
block 210337792 had to be read from a different mirror, ret 0
block 595607552 had to be read from a different mirror, ret 0
block 949698560 had to be read from a different mirror, ret 0
(...)
Checking root 165299 bytenr 13577191505920
block 364679708672 had to be read from a different mirror, ret 0
block 364713902080 had to be read from a different mirror, ret 0
(...)
block 13577821011968 had to be read from a different mirror, ret 0
block 15645781196800 had to be read from a different mirror, ret 0
Checking root 18446744073709551607 bytenr 13576823685120
Tree recovery finished, you can run check now
[Inferior 1 (process 5587) exited normally]
(gdb) quit

Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
block 15645878108160 had to be read from a different mirror, ret 0
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
(...)
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
block 58720256 had to be read from a different mirror, ret 0
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d
block 11651824091136 had to be read from a different mirror, ret 0
checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188
block 606126080 had to be read from a different mirror, ret 0
checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858
block 15645807640576 had to be read from a different mirror, ret 0
checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82
block 364863324160 had to be read from a different mirror, ret 0
(...)
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
block 15645419667456 had to be read from a different mirror, ret 0
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
block 13577821011968 had to be read from a different mirror, ret 0
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
block 15645781196800 had to be read from a different mirror, ret 0
Clearing the extent root and re-init'ing the block groups
parent transid verify failed on 29442048 wanted 1619017 found 1619015
parent transid verify failed on 29442048 wanted 1619017 found 1619015
parent transid verify failed on 29442048 wanted 1619017 found 1619015
parent transid verify failed on 29556736 wanted 1619017 found 1619013
parent transid verify failed on 29556736 wanted 1619017 found 1619013
parent transid verify failed on 29556736 wanted 1619017 found 1619013
parent transid verify failed on 29589504 wanted 1619017 found 1619013
parent transid verify failed on 29589504 wanted 1619017 found 1619013
parent transid verify failed on 29589504 wanted 1619017 found 1619013
ERROR: Error adding block group
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
extent buffer leak: start 29589504 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 34504704 len 16384
WARNING: dirty eb leak (aborted trans): start 34504704 len 16384
Init extent tree failed
[Inferior 1 (process 11217) exited with code 0357]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#88 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 23 lines · Marc MERLIN
Quoted 21 lines · Josef Bacik
Quoted 19 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Wtf, it's reading the right bytenr, but it's not failing here but failing when we do the init-extent-tree. I've pushed something again to force reads, maybe that's the problem, can you run tree-recover and then init-extent-tree again? Thanks,

Quoted 5 lines
Quoted 3 lines

off-list:
Seeing this, that kind of worried me so I stopped it. Is it ok to continue?

Sorry, my printf was wrong, pushed. Thanks,

Better?
Do I worry about those new "had to be read from a different mirror" ?

Quoted 9 lines
Quoted 4 lines

Ooooh ok, the other mirror is bad, but it finds the right thing. Ok that makes sense. I pushed another patch just to make sure. Once this is all done you'll need to run a scrub to fix the mirrors but this is fine. Thanks,

No more crash, but:
ERROR: Error adding block group
ERROR: commit_root already set when starting transaction

I feel like this thing is purposefully changing itself between each run so I can't get a grasp on wtf is going on. I pushed some stuff, lets see how that goes. Thanks,

Josef

#89 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

I feel like this thing is purposefully changing itself between each run so I can't get a grasp on wtf is going on. I pushed some stuff, lets see how that goes. Thanks,

After all the tests we did, is it possible that some damaged the FS further?

Either way:
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 block 15645878108160 had to be read from a different mirror, ret 0 Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 (...)
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks block 11651824091136 had to be read from a different mirror, ret 0 block 606126080 had to be read from a different mirror, ret 0 block 15645807640576 had to be read from a different mirror, ret 0 block 364863324160 had to be read from a different mirror, ret 0 block 364970688512 had to be read from a different mirror, ret 0 block 15645178052608 had to be read from a different mirror, ret 0 (...)
inserting block group 352753549312
inserting block group 353827291136
Ignoring transid failure
Ignoring transid failure
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920 extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384 extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384 extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384 Init extent tree failed
[Inferior 1 (process 1264) exited with code 0357]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#90 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

I feel like this thing is purposefully changing itself between each run so I can't get a grasp on wtf is going on. I pushed some stuff, lets see how that goes. Thanks,

After all the tests we did, is it possible that some damaged the FS further?

That's the thing, we're literally deleting the entire tree and starting over, it should do the same thing every time. I pushed another fix, I think I've been messing up the buffers and that's why we're getting random values. Lets try this again,

Josef

#91 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

I feel like this thing is purposefully changing itself between each run so I can't get a grasp on wtf is going on. I pushed some stuff, lets see how that goes. Thanks,

Quoted 8 lines
Quoted 3 lines

After all the tests we did, is it possible that some damaged the FS further?

That's the thing, we're literally deleting the entire tree and starting over, it should do the same thing every time. I pushed another fix, I think I've been messing up the buffers and that's why we're getting random values. Lets try this again,

Gotcha.

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
(...)
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
inserting block group 12582912
inserting block group 20971520
inserting block group 29360128
inserting block group 1103101952
(...)
inserting block group 345237356544
inserting block group 346311098368
inserting block group 347384840192
inserting block group 348458582016
inserting block group 349532323840
inserting block group 350606065664
inserting block group 351679807488
inserting block group 352753549312
inserting block group 353827291136
Ignoring transid failure
Ignoring transid failure
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
Init extent tree failed
[Inferior 1 (process 6259) exited with code 0357]
(gdb)

I ran it a second time and got the same output

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#92 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 70 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

I feel like this thing is purposefully changing itself between each run so I can't get a grasp on wtf is going on. I pushed some stuff, lets see how that goes. Thanks,

Quoted 8 lines
Quoted 3 lines

After all the tests we did, is it possible that some damaged the FS further?

That's the thing, we're literally deleting the entire tree and starting over, it should do the same thing every time. I pushed another fix, I think I've been messing up the buffers and that's why we're getting random values. Lets try this again,

Gotcha.

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
(...)
checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups
inserting block group 12582912
inserting block group 20971520
inserting block group 29360128
inserting block group 1103101952
(...)
inserting block group 345237356544
inserting block group 346311098368
inserting block group 347384840192
inserting block group 348458582016
inserting block group 349532323840
inserting block group 350606065664
inserting block group 351679807488
inserting block group 352753549312
inserting block group 353827291136
Ignoring transid failure
Ignoring transid failure
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
Init extent tree failed
[Inferior 1 (process 6259) exited with code 0357]
(gdb)

Quoted 1 line

I ran it a second time and got the same output

Well thats something at least, I've rigged it up to dump the leaf so I can see wtf is going on here. Thanks,

Josef

#93 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik
Quoted 1 line

I ran it a second time and got the same output

Well thats something at least, I've rigged it up to dump the leaf so I can see wtf is going on here. Thanks,

Sorry for the delay, took a while to run:
(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
inserting block group 253969301504
item 33 key (328057487360 BLOCK_GROUP_ITEM 1073741824) itemoff 15467 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 173 key (477844471808 BLOCK_GROUP_ITEM 1073741824) itemoff 12107 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 174 key (478918213632 BLOCK_GROUP_ITEM 1073741824) itemoff 12083 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 175 key (479991955456 BLOCK_GROUP_ITEM 1073741824) itemoff 12059 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 176 key (481065697280 BLOCK_GROUP_ITEM 1073741824) itemoff 12035 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 177 key (482139439104 BLOCK_GROUP_ITEM 1073741824) itemoff 12011 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 178 key (483213180928 BLOCK_GROUP_ITEM 1073741824) itemoff 11987 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 179 key (484286922752 BLOCK_GROUP_ITEM 1073741824) itemoff 11963 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 180 key (485360664576 BLOCK_GROUP_ITEM 1073741824) itemoff 11939 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 181 key (486434406400 BLOCK_GROUP_ITEM 1073741824) itemoff 11915 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 182 key (487508148224 BLOCK_GROUP_ITEM 1073741824) itemoff 11891 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 183 key (488581890048 BLOCK_GROUP_ITEM 1073741824) itemoff 11867 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 184 key (489655631872 BLOCK_GROUP_ITEM 1073741824) itemoff 11843 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 185 key (490729373696 BLOCK_GROUP_ITEM 1073741824) itemoff 11819 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 186 key (491803115520 BLOCK_GROUP_ITEM 1073741824) itemoff 11795 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 187 key (492876857344 BLOCK_GROUP_ITEM 1073741824) itemoff 11771 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 188 key (493950599168 BLOCK_GROUP_ITEM 1073741824) itemoff 11747 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 189 key (495024340992 BLOCK_GROUP_ITEM 1073741824) itemoff 11723 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 190 key (496098082816 BLOCK_GROUP_ITEM 1073741824) itemoff 11699 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 191 key (497171824640 BLOCK_GROUP_ITEM 1073741824) itemoff 11675 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 192 key (498245566464 BLOCK_GROUP_ITEM 1073741824) itemoff 11651 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 193 key (499319308288 BLOCK_GROUP_ITEM 1073741824) itemoff 11627 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 194 key (500393050112 BLOCK_GROUP_ITEM 1073741824) itemoff 11603 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 195 key (501466791936 BLOCK_GROUP_ITEM 1073741824) itemoff 11579 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 196 key (502540533760 BLOCK_GROUP_ITEM 1073741824) itemoff 11555 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 197 key (503614275584 BLOCK_GROUP_ITEM 1073741824) itemoff 11531 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 198 key (504688017408 BLOCK_GROUP_ITEM 1073741824) itemoff 11507 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 199 key (505761759232 BLOCK_GROUP_ITEM 1073741824) itemoff 11483 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 200 key (506835501056 BLOCK_GROUP_ITEM 1073741824) itemoff 11459 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 201 key (507909242880 BLOCK_GROUP_ITEM 1073741824) itemoff 11435 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 202 key (508982984704 BLOCK_GROUP_ITEM 1073741824) itemoff 11411 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 203 key (510056726528 BLOCK_GROUP_ITEM 1073741824) itemoff 11387 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 204 key (511130468352 BLOCK_GROUP_ITEM 1073741824) itemoff 11363 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 205 key (512204210176 BLOCK_GROUP_ITEM 1073741824) itemoff 11339 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 206 key (513277952000 BLOCK_GROUP_ITEM 1073741824) itemoff 11315 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 207 key (514351693824 BLOCK_GROUP_ITEM 1073741824) itemoff 11291 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 208 key (515425435648 BLOCK_GROUP_ITEM 1073741824) itemoff 11267 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 209 key (516499177472 BLOCK_GROUP_ITEM 1073741824) itemoff 11243 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 213 key (520794144768 BLOCK_GROUP_ITEM 1073741824) itemoff 11147 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 216 key (524015370240 BLOCK_GROUP_ITEM 1073741824) itemoff 11075 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 217 key (525089112064 BLOCK_GROUP_ITEM 1073741824) itemoff 11051 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
Init extent tree failed
[Inferior 1 (process 24213) exited with code 0357]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#94 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups?

Quoted 126 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Quoted 1 line

I ran it a second time and got the same output

Well thats something at least, I've rigged it up to dump the leaf so I can see wtf is going on here. Thanks,

Sorry for the delay, took a while to run:
(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
inserting block group 253969301504
item 33 key (328057487360 BLOCK_GROUP_ITEM 1073741824) itemoff 15467 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 173 key (477844471808 BLOCK_GROUP_ITEM 1073741824) itemoff 12107 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 174 key (478918213632 BLOCK_GROUP_ITEM 1073741824) itemoff 12083 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 175 key (479991955456 BLOCK_GROUP_ITEM 1073741824) itemoff 12059 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 176 key (481065697280 BLOCK_GROUP_ITEM 1073741824) itemoff 12035 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 177 key (482139439104 BLOCK_GROUP_ITEM 1073741824) itemoff 12011 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 178 key (483213180928 BLOCK_GROUP_ITEM 1073741824) itemoff 11987 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 179 key (484286922752 BLOCK_GROUP_ITEM 1073741824) itemoff 11963 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 180 key (485360664576 BLOCK_GROUP_ITEM 1073741824) itemoff 11939 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 181 key (486434406400 BLOCK_GROUP_ITEM 1073741824) itemoff 11915 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 182 key (487508148224 BLOCK_GROUP_ITEM 1073741824) itemoff 11891 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 183 key (488581890048 BLOCK_GROUP_ITEM 1073741824) itemoff 11867 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 184 key (489655631872 BLOCK_GROUP_ITEM 1073741824) itemoff 11843 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 185 key (490729373696 BLOCK_GROUP_ITEM 1073741824) itemoff 11819 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 186 key (491803115520 BLOCK_GROUP_ITEM 1073741824) itemoff 11795 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 187 key (492876857344 BLOCK_GROUP_ITEM 1073741824) itemoff 11771 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 188 key (493950599168 BLOCK_GROUP_ITEM 1073741824) itemoff 11747 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 189 key (495024340992 BLOCK_GROUP_ITEM 1073741824) itemoff 11723 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 190 key (496098082816 BLOCK_GROUP_ITEM 1073741824) itemoff 11699 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 191 key (497171824640 BLOCK_GROUP_ITEM 1073741824) itemoff 11675 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 192 key (498245566464 BLOCK_GROUP_ITEM 1073741824) itemoff 11651 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 193 key (499319308288 BLOCK_GROUP_ITEM 1073741824) itemoff 11627 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 194 key (500393050112 BLOCK_GROUP_ITEM 1073741824) itemoff 11603 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 195 key (501466791936 BLOCK_GROUP_ITEM 1073741824) itemoff 11579 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 196 key (502540533760 BLOCK_GROUP_ITEM 1073741824) itemoff 11555 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 197 key (503614275584 BLOCK_GROUP_ITEM 1073741824) itemoff 11531 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 198 key (504688017408 BLOCK_GROUP_ITEM 1073741824) itemoff 11507 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 199 key (505761759232 BLOCK_GROUP_ITEM 1073741824) itemoff 11483 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 200 key (506835501056 BLOCK_GROUP_ITEM 1073741824) itemoff 11459 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 201 key (507909242880 BLOCK_GROUP_ITEM 1073741824) itemoff 11435 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 202 key (508982984704 BLOCK_GROUP_ITEM 1073741824) itemoff 11411 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 203 key (510056726528 BLOCK_GROUP_ITEM 1073741824) itemoff 11387 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 204 key (511130468352 BLOCK_GROUP_ITEM 1073741824) itemoff 11363 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 205 key (512204210176 BLOCK_GROUP_ITEM 1073741824) itemoff 11339 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 206 key (513277952000 BLOCK_GROUP_ITEM 1073741824) itemoff 11315 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 207 key (514351693824 BLOCK_GROUP_ITEM 1073741824) itemoff 11291 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 208 key (515425435648 BLOCK_GROUP_ITEM 1073741824) itemoff 11267 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 209 key (516499177472 BLOCK_GROUP_ITEM 1073741824) itemoff 11243 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 213 key (520794144768 BLOCK_GROUP_ITEM 1073741824) itemoff 11147 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 216 key (524015370240 BLOCK_GROUP_ITEM 1073741824) itemoff 11075 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
item 217 key (525089112064 BLOCK_GROUP_ITEM 1073741824) itemoff 11051 itemsize 24
block group used 0 chunk_objectid 256 flags DATA|single
ERROR: Error adding block group -17
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920
extent buffer leak: start 67469312 len 16384
extent buffer leak: start 29540352 len 16384
WARNING: dirty eb leak (aborted trans): start 29540352 len 16384
extent buffer leak: start 29589504 len 16384
WARNING: dirty eb leak (aborted trans): start 29589504 len 16384
extent buffer leak: start 29655040 len 16384
WARNING: dirty eb leak (aborted trans): start 29655040 len 16384
Init extent tree failed
[Inferior 1 (process 24213) exited with code 0357]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#95 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Marc MERLIN

Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups?

Yup sorry for the radio silence, loads of meetings today, but good news is I've reproduced your problem locally, so I'm trying to hammer it out. I hope to have something useful for you today. Thanks,

Josef

#96 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 12 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN

Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups?

Yup sorry for the radio silence, loads of meetings today, but good news is I've reproduced your problem locally, so I'm trying to hammer it out. I hope to have something useful for you today. Thanks,

Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to make sure any stupidity I've caused is undone, and then run rescue init-extent-tree and then we can go from there? Thanks,

Josef

#97 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Josef Bacik
Quoted 8 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN

Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups?

Quoted 7 lines
Quoted 3 lines

Yup sorry for the radio silence, loads of meetings today, but good news is I've reproduced your problem locally, so I'm trying to hammer it out. I hope to have something useful for you today. Thanks,

Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to make sure any stupidity I've caused is undone, and then run rescue init-extent-tree and then we can go from there? Thanks,

FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Checking root 2 bytenr 67387392
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 13577819963392
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576862547968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781491265536 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781492101120 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781492756480 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781495197696 Checking root 163303 bytenr 15645365993472 Checking root 163316 bytenr 6781496393728 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11971021275136 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970667446272 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823685120 Tree recovery finished, you can run check now [Inferior 1 (process 28238) exited normally]

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Checking root 2 bytenr 67387392
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 13577819963392
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576862547968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781491265536 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781492101120 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781492756480 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781495197696 Checking root 163303 bytenr 15645365993472 Checking root 163316 bytenr 6781496393728 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11971021275136 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970667446272 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823685120 Tree recovery finished, you can run check now [Inferior 1 (process 28238) exited normally] (gdb)

        item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA|single
        item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24

inserting block group 15731991248896
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559ae9e <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559ae9e <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48

#1 0x000055555559b0a2 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99713024)
at common/extent-cache.c:179
#2 0x0000555555584c30 in set_extent_bits (tree=0x10000000060, start=99713024, end=99713023, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e63 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585846 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee3d in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434246, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b92bd0, root=<optimized out>) at kernel-shared/extent-tree.c:1084
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434246, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b92bd0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1196

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261
#9 0x00005555555dfb81 in process_eb (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfce80,

    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:502
#10 0x00005555555dfc90 in process_eb (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:577

#11 0x00005555555dfef8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:648 #12 0x00005555555e0202 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:784
#13 0x00005555555d7a0c in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#98 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 8 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN

Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups?

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Yup sorry for the radio silence, loads of meetings today, but good news is I've reproduced your problem locally, so I'm trying to hammer it out. I hope to have something useful for you today. Thanks,

Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to make sure any stupidity I've caused is undone, and then run rescue init-extent-tree and then we can go from there? Thanks,

Ok back to this problem again, I've added some debugging. Unfortunately the real bytenr is getting lost, so the new debugging will print the actual bytenr we're trying to add, and then I can add more targeted debugging to figure out whats going on. Thanks,

Josef

#99 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 16 lines · Josef Bacik
Quoted 15 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 11 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN

Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups?

Yup sorry for the radio silence, loads of meetings today, but good news is I've reproduced your problem locally, so I'm trying to hammer it out. I hope to have something useful for you today. Thanks,

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to make sure any stupidity I've caused is undone, and then run rescue init-extent-tree and then we can go from there? Thanks,

Ok back to this problem again, I've added some debugging. Unfortunately the real bytenr is getting lost, so the new debugging will print the actual bytenr we're trying to add, and then I can add more targeted debugging to figure out whats going on. Thanks,

Small typo:

  409 |  prinf("doing insert of %llu\n", key->objectid);
        |  ^~~~~
	|  printf

Result:
doing insert of 12233379401728
Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48

#1 0x000055555559b0ac in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=100564992)
at common/extent-cache.c:179
#2 0x0000555555584c3a in set_extent_bits (tree=0x10000000060, start=100564992, end=100564991, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e6d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585850 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee47 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434256, parent=0, 
    iref=0xffffffffffffffe4, path=0x555555b8cda0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434256, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cda0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262
#9 0x00005555555dfba4 in process_eb (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00cc0,

    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:503
#10 0x00005555555dfcb3 in process_eb (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:578

#11 0x00005555555dff1b in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:649 #12 0x00005555555e0225 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:785
#13 0x00005555555d7a16 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#100 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 27 lines · Marc MERLIN
Quoted 16 lines · Josef Bacik
Quoted 15 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 11 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN

Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups?

Yup sorry for the radio silence, loads of meetings today, but good news is I've reproduced your problem locally, so I'm trying to hammer it out. I hope to have something useful for you today. Thanks,

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to make sure any stupidity I've caused is undone, and then run rescue init-extent-tree and then we can go from there? Thanks,

Ok back to this problem again, I've added some debugging. Unfortunately the real bytenr is getting lost, so the new debugging will print the actual bytenr we're trying to add, and then I can add more targeted debugging to figure out whats going on. Thanks,

Quoted 8 lines

Small typo:

  409 |  prinf("doing insert of %llu\n", key->objectid);
        |  ^~~~~
        |  printf

Result:
doing insert of 12233379401728
Failed to find [7750833868800, 168, 262144]

Oh it's data, interesting. Pushed some more debugging. Thanks,

Josef

#101 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Josef Bacik
Quoted 3 lines

Result:
doing insert of 12233379401728
Failed to find [7750833868800, 168, 262144]

Oh it's data, interesting. Pushed some more debugging. Thanks,

inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? Failed to find [7750833868800, 168, 262144]
processed 1245184 of 0 possible bytesIgnoring transid failure kernel-shared/ctree.c:1940: leaf_space_used: Warning: assertion data_len < 0 failed, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1d13f)[0x55555557113f] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_leaf_free_space+0xd3)[0x555555573b36] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1fbf3)[0x555555573bf3] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2083b)[0x55555557483b] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_search_slot+0x11cd)[0x5555555767c9] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_insert_empty_items+0x76)[0x555555576e6b] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8b97d)[0x5555555df97d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bc04)[0x5555555dfc04] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bcfb)[0x5555555dfcfb] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bf59)[0x5555555dff59] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0x22d)[0x5555555e0263] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a16)[0x5555555d7a16] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a] kernel-shared/ctree.c:1940: leaf_space_used: Warning: assertion data_len < 0 failed, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1d13f)[0x55555557113f] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_leaf_free_space+0xd3)[0x555555573b36] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1fc38)[0x555555573c38] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2083b)[0x55555557483b] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_search_slot+0x11cd)[0x5555555767c9] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_insert_empty_items+0x76)[0x555555576e6b] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8b97d)[0x5555555df97d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bc04)[0x5555555dfc04] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bcfb)[0x5555555dfcfb] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bf59)[0x5555555dff59] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0x22d)[0x5555555e0263] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a16)[0x5555555d7a16] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x1000000000060, key=key@entry=0x7fffffffd8f0, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd908) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x1000000000060, key=key@entry=0x7fffffffd8f0, 
    comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd908) at common/rbtree-utils.c:48
#1  0x000055555559b0ac in search_cache_extent (tree=tree@entry=0x1000000000060, start=start@entry=1) at common/extent-cache.c:179
#2  0x0000555555584c3a in set_extent_bits (tree=0x1000000000060, start=1, end=0, bits=bits@entry=1) at kernel-shared/extent_io.c:380
#3  0x0000555555584e6d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
    at kernel-shared/extent_io.c:486
#4  0x0000555555585850 in set_extent_buffer_dirty (eb=0x555555708580) at kernel-shared/extent_io.c:976
#5  0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=<optimized out>) at kernel-shared/disk-io.c:2224
#6  0x00005555555740c3 in push_leaf_right (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, 
    path=path@entry=0x7fffffffde40, data_size=data_size@entry=49, empty=empty@entry=0) at kernel-shared/ctree.c:2106
#7  0x000055555557483b in split_leaf (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, 
    ins_key=ins_key@entry=0x7fffffffdf60, path=path@entry=0x7fffffffde40, data_size=data_size@entry=49, extend=extend@entry=0)
    at kernel-shared/ctree.c:2385
#8  0x00005555555767c9 in btrfs_search_slot (trans=0x555558a8d700, root=root@entry=0x5555559547d0, key=key@entry=0x7fffffffdf60, 
    p=p@entry=0x7fffffffde40, ins_len=ins_len@entry=49, cow=cow@entry=1) at kernel-shared/ctree.c:1432
#9  0x0000555555576e6b in btrfs_insert_empty_items (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, 
    path=path@entry=0x7fffffffde40, cpu_key=cpu_key@entry=0x7fffffffdf60, data_size=data_size@entry=0x7fffffffde3c, nr=nr@entry=1)
    at kernel-shared/ctree.c:2820
#10 0x00005555555df97d in btrfs_insert_empty_item (data_size=<optimized out>, key=0x7fffffffdf60, path=0x7fffffffde40, 
    root=0x5555559547d0, trans=0x555558a8d700) at ./kernel-shared/ctree.h:2780
#11 insert_empty_extent (trans=trans@entry=0x555558a8d700, key=key@entry=0x7fffffffdf60, generation=generation@entry=1590260, 
    flags=flags@entry=1) at cmds/rescue-init-extent-tree.c:435
#12 0x00005555555dfc04 in process_eb (trans=trans@entry=0x555558a8d700, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d04fe0, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:508
#13 0x00005555555dfcfb in process_eb (trans=trans@entry=0x555558a8d700, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:594

#14 0x00005555555dff59 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:665 #15 0x00005555555e0263 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:801
#16 0x00005555555d7a16 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#17 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#18 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#19 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #20 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#102 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Quoted 3 lines

Result:
doing insert of 12233379401728
Failed to find [7750833868800, 168, 262144]

Quoted 7 lines
Quoted 1 line

Oh it's data, interesting. Pushed some more debugging. Thanks,

inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr??

Well crap, you have bytenrs that overlap. I've adjusted init-extent-tree to dump the paths of the files that have overlapping extents, as well as the keys. Run it again so we know what files are fucked. From there you need to tell me which one you don't care about and are willing to delete, and then I'll give you the command you need to remove that bad extent and then we can go again. Thanks,

Josef

#103 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 15 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Quoted 3 lines

Result:
doing insert of 12233379401728
Failed to find [7750833868800, 168, 262144]

Quoted 7 lines
Quoted 1 line

Oh it's data, interesting. Pushed some more debugging. Thanks,

inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr??

Quoted 6 lines

Well crap, you have bytenrs that overlap. I've adjusted init-extent-tree to dump the paths of the files that have overlapping extents, as well as the keys. Run it again so we know what files are fucked. From there you need to tell me which one you don't care about and are willing to delete, and then I'll give you the command you need to remove that bad extent and then we can go again. Thanks,

I see.

Not sure if that helped. Should I have run it differently?

Author: Josef Bacik <[email protected]>
Date:   Wed Apr 27 16:19:46 2022 -0400

fix some things so dump path works

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
(...)
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48

#1 0x000055555559b09e in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=100564992) at common/extent-cache.c:179 #2 0x0000555555584c2c in set_extent_bits (tree=0x100000060, start=100564992, end=100564991, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e5f in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585842 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc10 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee39 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998756304, parent=0, 
    iref=0xffffffffffffffe3, path=0x555555b8cfd0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998756304, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cfd0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262
#9 0x00005555555dfcf4 in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00d20,

    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:555
#10 0x00005555555dfe4d in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:630

#11 0x00005555555e00b8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:701 #12 0x00005555555e03c2 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:837
#13 0x00005555555d7a08 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#104 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 33 lines · Marc MERLIN
Quoted 15 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik
Quoted 3 lines

Result:
doing insert of 12233379401728
Failed to find [7750833868800, 168, 262144]

Quoted 7 lines
Quoted 1 line

Oh it's data, interesting. Pushed some more debugging. Thanks,

inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr??

Quoted 6 lines

Well crap, you have bytenrs that overlap. I've adjusted init-extent-tree to dump the paths of the files that have overlapping extents, as well as the keys. Run it again so we know what files are fucked. From there you need to tell me which one you don't care about and are willing to delete, and then I'll give you the command you need to remove that bad extent and then we can go again. Thanks,

I see.

Not sure if that helped. Should I have run it differently?

Author: Josef Bacik <[email protected]>
Date:   Wed Apr 27 16:19:46 2022 -0400

fix some things so dump path works

Quoted 12 lines

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
(...)
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths Failed to find [7750833868800, 168, 262144]

Of course it doesn't work for you, I pushed some debug stuff. Thanks,

Josef

#105 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik
Quoted 10 lines

inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths Failed to find [7750833868800, 168, 262144]

Quoted 1 line

Of course it doesn't work for you, I pushed some debug stuff. Thanks,

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x0d38525a found 0xb3a707fa
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
(...)
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] elem_cnt 0 elem_missed 0 ret -2
doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths elem_cnt 0 elem_missed 0 ret -2
Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48

#1 0x000055555559b09e in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=100564992) at common/extent-cache.c:179 #2 0x0000555555584c2c in set_extent_bits (tree=0x100000060, start=100564992, end=100564991, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e5f in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585842 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976
#5  0x000055555557bc10 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee39 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, 
    iref=0xffffffffffffffe3, path=0x555555b8cda0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cda0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262
#9 0x00005555555dfd17 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00f00,

    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:557
#10 0x00005555555dfe70 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:632

#11 0x00005555555e00db in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:703 #12 0x00005555555e03e5 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:839
#13 0x00005555555d7a08 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#106 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 26 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines

inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths Failed to find [7750833868800, 168, 262144]

Quoted 1 line

Of course it doesn't work for you, I pushed some debug stuff. Thanks,

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
checksum verify failed on 58720256 wanted 0x0d38525a found 0xb3a707fa
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
(...)

Quoted 8 lines

inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] elem_cnt 0 elem_missed 0 ret -2
doing an insert that overlaps our bytenr 7750833627136 262144
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths elem_cnt 0 elem_missed 0 ret -2
Failed to find [7750833868800, 168, 262144]

Sigh, added another print_leaf. Thanks,

Josef

#107 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 1 line · Josef Bacik

Sigh, added another print_leaf. Thanks,

doing an insert that overlaps our bytenr 7750833627136 262144
processed 1146880 of 0 possible bytes
processed 1163264 of 0 possible bytes
processed 1179648 of 0 possible bytes
processed 1196032 of 0 possible bytes
processed 1212416 of 0 possible bytes
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths inode ref info failed???
leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE
leaf 15645023322112 flags 0x1(WRITTEN) backref revision 1
fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e
chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a

	item 0 key (5457 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1835008000
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7000 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.690807153 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 1 key (5457 EXTENT_DATA 0) itemoff 16070 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207705702400 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 2 key (5458 INODE_ITEM 0) itemoff 15910 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 46137344
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.690807153 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 3 key (5458 EXTENT_DATA 0) itemoff 15857 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207791230976 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 4 key (5459 INODE_ITEM 0) itemoff 15697 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 74448896
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 284 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.790245505 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 5 key (5459 EXTENT_DATA 0) itemoff 15644 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7314895085568 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 6 key (5460 INODE_ITEM 0) itemoff 15484 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 398721024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1521 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.596346316 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 7 key (5460 EXTENT_DATA 0) itemoff 15431 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12470912008192 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 8 key (5461 INODE_ITEM 0) itemoff 15271 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 25165824
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 96 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.790245505 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 9 key (5461 EXTENT_DATA 0) itemoff 15218 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7482517606400 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 10 key (5462 INODE_ITEM 0) itemoff 15058 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1549795328
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5912 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.694807100 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 11 key (5462 EXTENT_DATA 0) itemoff 15005 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207794606080 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 12 key (5463 INODE_ITEM 0) itemoff 14845 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 746848256
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2849 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.694807100 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 13 key (5463 EXTENT_DATA 0) itemoff 14792 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207801716736 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 14 key (5464 INODE_ITEM 0) itemoff 14632 itemsize 160
		generation 1482589 transid 1482589 size 262144 nbytes 686555136
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2619 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603031991.590888741 (2020-10-18 07:39:51)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 15 key (5464 EXTENT_DATA 0) itemoff 14579 itemsize 53
		generation 1482589 type 1 (regular)
		extent data disk byte 952344715264 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 16 key (5465 INODE_ITEM 0) itemoff 14419 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1620574208
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6182 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.698807046 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 17 key (5465 EXTENT_DATA 0) itemoff 14366 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207808765952 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 18 key (5466 INODE_ITEM 0) itemoff 14206 itemsize 160
		generation 1511028 transid 1511028 size 262144 nbytes 777256960
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2965 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1610956842.625245815 (2021-01-18 00:00:42)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 19 key (5466 EXTENT_DATA 0) itemoff 14153 itemsize 53
		generation 1511028 type 1 (regular)
		extent data disk byte 12245962477568 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 20 key (5467 INODE_ITEM 0) itemoff 13993 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1741684736
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6644 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.702806994 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 21 key (5467 EXTENT_DATA 0) itemoff 13940 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207860899840 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 22 key (5468 INODE_ITEM 0) itemoff 13780 itemsize 160
		generation 1525970 transid 1525970 size 262144 nbytes 95944704
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 366 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222802.713761123 (2021-03-08 09:00:02)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 23 key (5468 EXTENT_DATA 0) itemoff 13727 itemsize 53
		generation 1525970 type 1 (regular)
		extent data disk byte 7437273346048 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 24 key (5469 INODE_ITEM 0) itemoff 13567 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3301703680
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12595 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.702806994 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 25 key (5469 EXTENT_DATA 0) itemoff 13514 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12207862214656 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 26 key (5470 INODE_ITEM 0) itemoff 13354 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.706806940 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 27 key (5470 EXTENT_DATA 0) itemoff 13301 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12208843333632 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 28 key (5471 INODE_ITEM 0) itemoff 13141 itemsize 160
		generation 1482591 transid 1482591 size 262144 nbytes 992477184
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3786 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603032110.61310221 (2020-10-18 07:41:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 29 key (5471 EXTENT_DATA 0) itemoff 13088 itemsize 53
		generation 1482591 type 1 (regular)
		extent data disk byte 3184741777408 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 30 key (5472 INODE_ITEM 0) itemoff 12928 itemsize 160
		generation 1525968 transid 1525968 size 262144 nbytes 35389440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 135 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222730.746702497 (2021-03-08 08:58:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 31 key (5472 EXTENT_DATA 0) itemoff 12875 itemsize 53
		generation 1525968 type 1 (regular)
		extent data disk byte 8818662686720 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 32 key (5473 INODE_ITEM 0) itemoff 12715 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.706806940 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 33 key (5473 EXTENT_DATA 0) itemoff 12662 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12209607069696 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 34 key (5474 INODE_ITEM 0) itemoff 12502 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 521666560
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1990 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.710806887 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 35 key (5474 EXTENT_DATA 0) itemoff 12449 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12209981120512 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 36 key (5475 INODE_ITEM 0) itemoff 12289 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1441792000
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5500 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.710806887 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 37 key (5475 EXTENT_DATA 0) itemoff 12236 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210260660224 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 38 key (5476 INODE_ITEM 0) itemoff 12076 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1366556672
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5213 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.710806887 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 39 key (5476 EXTENT_DATA 0) itemoff 12023 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210686017536 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 40 key (5477 INODE_ITEM 0) itemoff 11863 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 7077888
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 27 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.714806834 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 41 key (5477 EXTENT_DATA 0) itemoff 11810 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210843799552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 42 key (5478 INODE_ITEM 0) itemoff 11650 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.794245452 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 43 key (5478 EXTENT_DATA 0) itemoff 11597 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7513535868928 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 44 key (5479 INODE_ITEM 0) itemoff 11437 itemsize 160
		generation 1509233 transid 1509233 size 262144 nbytes 201850880
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 770 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1610527033.221370143 (2021-01-13 00:37:13)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 45 key (5479 EXTENT_DATA 0) itemoff 11384 itemsize 53
		generation 1509233 type 1 (regular)
		extent data disk byte 11682327523328 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 46 key (5480 INODE_ITEM 0) itemoff 11224 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 968884224
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3696 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.714806834 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 47 key (5480 EXTENT_DATA 0) itemoff 11171 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12210946199552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 48 key (5481 INODE_ITEM 0) itemoff 11011 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1839464448
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 7017 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.718806781 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 49 key (5481 EXTENT_DATA 0) itemoff 10958 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12211111063552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 50 key (5482 INODE_ITEM 0) itemoff 10798 itemsize 160
		generation 1525968 transid 1525968 size 262144 nbytes 1832124416
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6989 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222730.746702497 (2021-03-08 08:58:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 51 key (5482 EXTENT_DATA 0) itemoff 10745 itemsize 53
		generation 1525968 type 1 (regular)
		extent data disk byte 8818664521728 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 52 key (5483 INODE_ITEM 0) itemoff 10585 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 6291456
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 24 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.718806781 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 53 key (5483 EXTENT_DATA 0) itemoff 10532 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12211529863168 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 54 key (5484 INODE_ITEM 0) itemoff 10372 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.722806727 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 55 key (5484 EXTENT_DATA 0) itemoff 10319 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12228941504512 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 56 key (5485 INODE_ITEM 0) itemoff 10159 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 370933760
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1415 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.722806727 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 57 key (5485 EXTENT_DATA 0) itemoff 10106 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12231089319936 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 58 key (5486 INODE_ITEM 0) itemoff 9946 itemsize 160
		generation 1525967 transid 1525967 size 262144 nbytes 1185415168
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4522 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222694.723170138 (2021-03-08 08:58:14)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 59 key (5486 EXTENT_DATA 0) itemoff 9893 itemsize 53
		generation 1525967 type 1 (regular)
		extent data disk byte 9425865584640 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 60 key (5487 INODE_ITEM 0) itemoff 9733 itemsize 160
		generation 1501750 transid 1501750 size 262144 nbytes 1346895872
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5138 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1609833640.497822971 (2021-01-05 00:00:40)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 61 key (5487 EXTENT_DATA 0) itemoff 9680 itemsize 53
		generation 1501750 type 1 (regular)
		extent data disk byte 7873777274880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 62 key (5488 INODE_ITEM 0) itemoff 9520 itemsize 160
		generation 1530421 transid 1530421 size 262144 nbytes 1375207424
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5246 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1617139918.873813209 (2021-03-30 14:31:58)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 63 key (5488 EXTENT_DATA 0) itemoff 9467 itemsize 53
		generation 1530421 type 1 (regular)
		extent data disk byte 7441055866880 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 64 key (5489 INODE_ITEM 0) itemoff 9307 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 2621440
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.726806674 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 65 key (5489 EXTENT_DATA 0) itemoff 9254 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12231929507840 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 66 key (5490 INODE_ITEM 0) itemoff 9094 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3932160
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 15 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.730806621 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 67 key (5490 EXTENT_DATA 0) itemoff 9041 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12232045236224 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 68 key (5491 INODE_ITEM 0) itemoff 8881 itemsize 160
		generation 1482911 transid 1482911 size 262144 nbytes 17301504
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 66 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169591.357269745 (2020-10-19 21:53:11)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 69 key (5491 EXTENT_DATA 0) itemoff 8828 itemsize 53
		generation 1482911 type 1 (regular)
		extent data disk byte 10889918492672 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 70 key (5492 INODE_ITEM 0) itemoff 8668 itemsize 160
		generation 1482901 transid 1482901 size 262144 nbytes 26214400
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 100 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169214.794245452 (2020-10-19 21:46:54)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 71 key (5492 EXTENT_DATA 0) itemoff 8615 itemsize 53
		generation 1482901 type 1 (regular)
		extent data disk byte 7515684741120 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 72 key (5493 INODE_ITEM 0) itemoff 8455 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 18087936
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 69 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.730806621 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 73 key (5493 EXTENT_DATA 0) itemoff 8402 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233142820864 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 74 key (5494 INODE_ITEM 0) itemoff 8242 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 54263808
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 207 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.720343072 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 75 key (5494 EXTENT_DATA 0) itemoff 8189 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12473955663872 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 76 key (5495 INODE_ITEM 0) itemoff 8029 itemsize 160
		generation 1482908 transid 1482908 size 262144 nbytes 2883584
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603169482.790704298 (2020-10-19 21:51:22)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 77 key (5495 EXTENT_DATA 0) itemoff 7976 itemsize 53
		generation 1482908 type 1 (regular)
		extent data disk byte 9382917087232 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 78 key (5496 INODE_ITEM 0) itemoff 7816 itemsize 160
		generation 1482593 transid 1482593 size 262144 nbytes 873725952
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3333 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603032208.635996917 (2020-10-18 07:43:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 79 key (5496 EXTENT_DATA 0) itemoff 7763 itemsize 53
		generation 1482593 type 1 (regular)
		extent data disk byte 4583295479808 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 80 key (5497 INODE_ITEM 0) itemoff 7603 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1531183104
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 5841 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.734806568 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 81 key (5497 EXTENT_DATA 0) itemoff 7550 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233145966592 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 82 key (5498 INODE_ITEM 0) itemoff 7390 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 18612224
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 71 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.724342967 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 83 key (5498 EXTENT_DATA 0) itemoff 7337 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12473955926016 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 84 key (5499 INODE_ITEM 0) itemoff 7177 itemsize 160
		generation 1525968 transid 1525968 size 262144 nbytes 432013312
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1648 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222730.750702446 (2021-03-08 08:58:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 85 key (5499 EXTENT_DATA 0) itemoff 7124 itemsize 53
		generation 1525968 type 1 (regular)
		extent data disk byte 8818666881024 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 86 key (5500 INODE_ITEM 0) itemoff 6964 itemsize 160
		generation 1482593 transid 1482593 size 262144 nbytes 266600448
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1017 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1603032208.639996865 (2020-10-18 07:43:28)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 87 key (5500 EXTENT_DATA 0) itemoff 6911 itemsize 53
		generation 1482593 type 1 (regular)
		extent data disk byte 4654161346560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 88 key (5501 INODE_ITEM 0) itemoff 6751 itemsize 160
		generation 1525961 transid 1525961 size 262144 nbytes 1067450368
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4072 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222478.881970135 (2021-03-08 08:54:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 89 key (5501 EXTENT_DATA 0) itemoff 6698 itemsize 53
		generation 1525961 type 1 (regular)
		extent data disk byte 7982222049280 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 90 key (5502 INODE_ITEM 0) itemoff 6538 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 562561024
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2146 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.734806568 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 91 key (5502 EXTENT_DATA 0) itemoff 6485 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233186123776 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 92 key (5503 INODE_ITEM 0) itemoff 6325 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 3145728
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.734806568 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 93 key (5503 EXTENT_DATA 0) itemoff 6272 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233187991552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 94 key (5504 INODE_ITEM 0) itemoff 6112 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1200095232
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4578 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.738806514 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 95 key (5504 EXTENT_DATA 0) itemoff 6059 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233257922560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 96 key (5505 INODE_ITEM 0) itemoff 5899 itemsize 160
		generation 1525961 transid 1525961 size 262144 nbytes 71827456
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 274 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222478.885970083 (2021-03-08 08:54:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 97 key (5505 EXTENT_DATA 0) itemoff 5846 itemsize 53
		generation 1525961 type 1 (regular)
		extent data disk byte 8013364232192 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 98 key (5506 INODE_ITEM 0) itemoff 5686 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 4718592
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.738806514 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 99 key (5506 EXTENT_DATA 0) itemoff 5633 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233379401728 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 100 key (5507 INODE_ITEM 0) itemoff 5473 itemsize 160
		generation 1525959 transid 1525959 size 262144 nbytes 294649856
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1124 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222403.234950654 (2021-03-08 08:53:23)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 101 key (5507 EXTENT_DATA 0) itemoff 5420 itemsize 53
		generation 1525959 type 1 (regular)
		extent data disk byte 7750833868800 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 102 key (5508 INODE_ITEM 0) itemoff 5260 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 362807296
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1384 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.742806462 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 103 key (5508 EXTENT_DATA 0) itemoff 5207 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233610690560 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 104 key (5509 INODE_ITEM 0) itemoff 5047 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 766246912
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 2923 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.742806462 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 105 key (5509 EXTENT_DATA 0) itemoff 4994 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233544069120 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 106 key (5510 INODE_ITEM 0) itemoff 4834 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1768685568
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 6747 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.746806408 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 107 key (5510 EXTENT_DATA 0) itemoff 4781 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233581916160 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 108 key (5511 INODE_ITEM 0) itemoff 4621 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 958398464
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 3656 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.746806408 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 109 key (5511 EXTENT_DATA 0) itemoff 4568 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233725329408 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 110 key (5512 INODE_ITEM 0) itemoff 4408 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 408420352
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 1558 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.750806355 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 111 key (5512 EXTENT_DATA 0) itemoff 4355 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233793339392 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 112 key (5513 INODE_ITEM 0) itemoff 4195 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 1253834752
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 4783 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.750806355 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 113 key (5513 EXTENT_DATA 0) itemoff 4142 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233785352192 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 114 key (5514 INODE_ITEM 0) itemoff 3982 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 8126464
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 31 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.754806301 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 115 key (5514 EXTENT_DATA 0) itemoff 3929 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233813143552 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 116 key (5515 INODE_ITEM 0) itemoff 3769 itemsize 160
		generation 1439787 transid 1439787 size 262144 nbytes 44826624
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 171 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1591038874.816340561 (2020-06-01 12:14:34)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 117 key (5515 EXTENT_DATA 0) itemoff 3716 itemsize 53
		generation 1439787 type 1 (regular)
		extent data disk byte 12474011467776 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 118 key (5516 INODE_ITEM 0) itemoff 3556 itemsize 160
		generation 1546750 transid 1546750 size 262144 nbytes 2405433344
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 9176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1621321238.520473625 (2021-05-18 00:00:38)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 119 key (5516 EXTENT_DATA 0) itemoff 3503 itemsize 53
		generation 1546750 type 1 (regular)
		extent data disk byte 12251618267136 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 120 key (5517 INODE_ITEM 0) itemoff 3343 itemsize 160
		generation 1526158 transid 1526158 size 262144 nbytes 40370176
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 154 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615276843.758806249 (2021-03-09 00:00:43)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)
	item 121 key (5517 EXTENT_DATA 0) itemoff 3290 itemsize 53
		generation 1526158 type 1 (regular)
		extent data disk byte 12233872674816 nr 262144
		extent data offset 0 nr 262144 ram 262144
		extent compression 0 (none)
	item 122 key (5518 INODE_ITEM 0) itemoff 3130 itemsize 160
		generation 1525963 transid 1525963 size 262144 nbytes 57933824
		block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
		sequence 221 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
		atime 0.0 (1969-12-31 16:00:00)
		ctime 1615222550.521041172 (2021-03-08 08:55:50)
		mtime 0.0 (1969-12-31 16:00:00)
		otime 0.0 (1969-12-31 16:00:00)

elem_cnt 0 elem_missed 0 ret -2
Failed to find [7750833868800, 168, 262144]
Segmentation fault

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#108 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Sigh, added another print_leaf. Thanks,

Quoted 9 lines

doing an insert that overlaps our bytenr 7750833627136 262144
processed 1146880 of 0 possible bytes
processed 1163264 of 0 possible bytes
processed 1179648 of 0 possible bytes
processed 1196032 of 0 possible bytes
processed 1212416 of 0 possible bytes
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths inode ref info failed???
leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE

Ooooh that explains it, it's the free space cache, that's perfect! I'll get something wired up and let you know when it's ready. Thanks,

Josef

#109 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 16 lines · Josef Bacik
Quoted 4 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Sigh, added another print_leaf. Thanks,

Quoted 9 lines

doing an insert that overlaps our bytenr 7750833627136 262144
processed 1146880 of 0 possible bytes
processed 1163264 of 0 possible bytes
processed 1179648 of 0 possible bytes
processed 1196032 of 0 possible bytes
processed 1212416 of 0 possible bytes
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths inode ref info failed???
leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE

Quoted 3 lines

Ooooh that explains it, it's the free space cache, that's perfect! I'll get something wired up and let you know when it's ready. Thanks,

Ok, lets hope for better results this time. Thanks,

Josef

#110 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 17 lines · Josef Bacik
Quoted 16 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Sigh, added another print_leaf. Thanks,

doing an insert that overlaps our bytenr 7750833627136 262144
processed 1146880 of 0 possible bytes
processed 1163264 of 0 possible bytes
processed 1179648 of 0 possible bytes
processed 1196032 of 0 possible bytes
processed 1212416 of 0 possible bytes
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths inode ref info failed???
leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE

Quoted 5 lines
Quoted 3 lines

Ooooh that explains it, it's the free space cache, that's perfect! I'll get something wired up and let you know when it's ready. Thanks,

Ok, lets hope for better results this time. Thanks,

        item 115 key (5514 EXTENT_DATA 0) itemoff 3929 itemsize 53
                generation 1526158 type 1 (regular)
                extent data disk byte 12233813143552 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 116 key (5515 INODE_ITEM 0) itemoff 3769 itemsize 160
                generation 1439787 transid 1439787 size 262144 nbytes 44826624
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 171 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1591038874.816340561 (2020-06-01 12:14:34)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)
        item 117 key (5515 EXTENT_DATA 0) itemoff 3716 itemsize 53
                generation 1439787 type 1 (regular)
                extent data disk byte 12474011467776 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 118 key (5516 INODE_ITEM 0) itemoff 3556 itemsize 160
                generation 1546750 transid 1546750 size 262144 nbytes 2405433344
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 9176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1621321238.520473625 (2021-05-18 00:00:38)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)
        item 119 key (5516 EXTENT_DATA 0) itemoff 3503 itemsize 53
                generation 1546750 type 1 (regular)
                extent data disk byte 12251618267136 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 120 key (5517 INODE_ITEM 0) itemoff 3343 itemsize 160
                generation 1526158 transid 1526158 size 262144 nbytes 40370176
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 154 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1615276843.758806249 (2021-03-09 00:00:43)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)
        item 121 key (5517 EXTENT_DATA 0) itemoff 3290 itemsize 53
                generation 1526158 type 1 (regular)
                extent data disk byte 12233872674816 nr 262144
                extent data offset 0 nr 262144 ram 262144
                extent compression 0 (none)
        item 122 key (5518 INODE_ITEM 0) itemoff 3130 itemsize 160
                generation 1525963 transid 1525963 size 262144 nbytes 57933824
                block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0
                sequence 221 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
                atime 0.0 (1969-12-31 16:00:00)
                ctime 1615222550.521041172 (2021-03-08 08:55:50)
                mtime 0.0 (1969-12-31 16:00:00)
                otime 0.0 (1969-12-31 16:00:00)

elem_cnt 0 elem_missed 0 ret -2
Failed to find [7750833868800, 168, 262144]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd740, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd758) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd740, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd758) at common/rbtree-utils.c:48

#1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=13576826748928)
at common/extent-cache.c:179
#2 0x0000555555584c67 in set_extent_bits (tree=0x100000060, start=13576826748928, end=13576826748927, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976
#5  0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998761968, parent=0, 
    iref=0xffffffffffffffe3, path=0x555555b8e5f0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998761968, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8e5f0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197

#8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, bytenr=<optimized out>,
num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262
#9 0x00005555555dfd52 in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00d20,

    current=current@entry=0x7fffffffdaf8) at cmds/rescue-init-extent-tree.c:615
#10 0x00005555555dfeab in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=0x555555944100, 
    current=current@entry=0x7fffffffdaf8) at cmds/rescue-init-extent-tree.c:690

#11 0x00005555555e0116 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:761 #12 0x00005555555e0436 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1")
at cmds/rescue-init-extent-tree.c:897
#13 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#111 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 19 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 16 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Sigh, added another print_leaf. Thanks,

doing an insert that overlaps our bytenr 7750833627136 262144
processed 1146880 of 0 possible bytes
processed 1163264 of 0 possible bytes
processed 1179648 of 0 possible bytes
processed 1196032 of 0 possible bytes
processed 1212416 of 0 possible bytes
processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths inode ref info failed???
leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Ooooh that explains it, it's the free space cache, that's perfect! I'll get something wired up and let you know when it's ready. Thanks,

Ok, lets hope for better results this time. Thanks,

Ok it should work now. Thanks,

Josef

#112 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 1 line · Josef Bacik

Ok it should work now. Thanks,

Mmmh, it got worse, unfortunately

(gdb) run rescue init-extent-tree /dev/mapper/dshelf1
Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
FS_INFO IS 0x55555564cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55555564cbc0
Walking all our trees and pinning down the currently accessible blocks
Clearing the extent root and re-init'ing the block groups

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff78e1f46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff78e1f46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x0000555555585950 in memmove (__len=<optimized out>, __src=<optimized out>, __dest=<optimized out>)

    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:36
#2  memmove_extent_buffer (dst=dst@entry=0x555555b57f60, dst_offset=<optimized out>, src_offset=<optimized out>, len=<optimized out>)
    at kernel-shared/extent_io.c:1021
#3  0x000055555557753b in btrfs_del_items (trans=trans@entry=0x555558a8d460, root=root@entry=0x55555564cde0, 
    path=path@entry=0x7fffffffdc50, slot=slot@entry=36, nr=nr@entry=1) at ./kernel-shared/ctree.h:2387
#4  0x00005555555e0627 in clear_space_cache (block_group=<optimized out>, trans=0x555558a8d460) at cmds/rescue-init-extent-tree.c:116
#5  reset_block_groups (trans=0x555558a8d460) at cmds/rescue-init-extent-tree.c:307
#6  reinit_extent_tree (fs_info=0x7fffffffdc50) at cmds/rescue-init-extent-tree.c:451
#7  btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:922
#8  0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#9  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#10 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#11 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #12 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#113 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Ok it should work now. Thanks,

Mmmh, it got worse, unfortunately

Well not worse, just failed earlier because that's where the new code is. I think I just made a simple mistake, but I added some print'fs just in case. Thanks,

Josef

#114 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Ok it should work now. Thanks,

Mmmh, it got worse, unfortunately

Well not worse, just failed earlier because that's where the new code is. I think I just made a simple mistake, but I added some print'fs just in case. Thanks,

That helped:
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 18446744073706422272 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 1707278336 of 16545742848 possible byteskernel-shared/extent-tree.c:1193: insert_inline_extent_backref: BUG_ON owner < BTRFS_FIRST_FREE_OBJECTID triggered, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2975e)[0x55555557d75e] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_inc_extent_ref+0x138)[0x55555557ed77] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8be01)[0x5555555dfe01]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c116)[0x5555555e0116] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xe96)[0x5555555e1089] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a43)[0x5555555d7a43] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd]
/var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Program received signal SIGABRT, Aborted. 0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#115 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 30 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Ok it should work now. Thanks,

Mmmh, it got worse, unfortunately

Well not worse, just failed earlier because that's where the new code is. I think I just made a simple mistake, but I added some print'fs just in case. Thanks,

That helped:
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 18446744073706422272 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 1707278336 of 16545742848 possible byteskernel-shared/extent-tree.c:1193: insert_inline_extent_backref: BUG_ON owner < BTRFS_FIRST_FREE_OBJECTID triggered, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2975e)[0x55555557d75e] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_inc_extent_ref+0x138)[0x55555557ed77] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8be01)[0x5555555dfe01]
/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab]

Quoted 9 lines

/var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c116)[0x5555555e0116] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xe96)[0x5555555e1089] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a43)[0x5555555d7a43] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a]

Cool, we're pointing at the same block in different places in the same tree. I should make tree-recover catch that and fix it, but since you're going to re-generate your csum tree anyway I've adjusted init-extent-tree to just clear the csum tree too, lets see how far we get with this. Thanks,

Josef

#116 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Josef Bacik

Cool, we're pointing at the same block in different places in the same tree. I should make tree-recover catch that and fix it, but since you're going to re-generate your csum tree anyway I've adjusted init-extent-tree to just clear the csum tree too, lets see how far we get with this. Thanks,

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48

#1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=15645286318080)
at common/extent-cache.c:179
#2 0x0000555555584c67 in set_extent_bits (tree=0x10000000060, start=15645286318080, end=15645286318079, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744, 
    parent=0, iref=0xffffffffffffffe4, path=0x555560dfe010, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555560dfe010, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfea9 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x555560e0ddf0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x55555a1be9a0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=0x555559ce9020, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734

#12 0x00005555555e026d in record_root (root=root@entry=0x555555af1ed0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#117 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 23 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik

Cool, we're pointing at the same block in different places in the same tree. I should make tree-recover catch that and fix it, but since you're going to re-generate your csum tree anyway I've adjusted init-extent-tree to just clear the csum tree too, lets see how far we get with this. Thanks,

inserting block group 15835070464000 inserting block group 15836144205824 inserting block group 15837217947648 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes

Quoted 9 lines

Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248]

Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,

Josef

#118 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,

Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?

        item 186 key (1834097 EXTENT_DATA 56098816) itemoff 5452 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451680858112 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 187 key (1834097 EXTENT_DATA 56623104) itemoff 5399 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451683733504 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684257792 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684782080 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451685568512 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457553899520 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457554423808 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457561710592 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457563021312 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457564856320 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)

elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48

#1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157752832)
at common/extent-cache.c:179
#2 0x0000555555584c67 in set_extent_bits (tree=0x100000060, start=15645157752832, end=15645157752831, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/extent_io.c:976
#5  0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528, 
    parent=0, iref=0xffffffffffffffe3, path=0x55555d2822b0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d2822b0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfea9 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x55555d291e50, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x5555564687c0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=0x555555f13e20, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734

#12 0x00005555555e026d in record_root (root=root@entry=0x555557a97ca0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#119 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,

Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?

Cool, do

./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>

Then you should be able to run the init-extent-tree and get past that part. Thanks,

Josef

#120 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,

Quoted 8 lines
Quoted 1 line

Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?

Cool, do

./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>

Then you should be able to run the init-extent-tree and get past that part. Thanks,

Sorry, I must be triggering every single unhandled path in the code :)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55fd6e82d600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fd6e82d600
kernel-shared/ctree.c:365: update_ref_for_cow: BUG_ON `ret` triggered, value -5
./btrfs-corrupt-block(+0x1137d)[0x55fd6d53237d]
./btrfs-corrupt-block(__btrfs_cow_block+0x3a9)[0x55fd6d533a90]
./btrfs-corrupt-block(btrfs_cow_block+0x73)[0x55fd6d53414f]
./btrfs-corrupt-block(btrfs_search_slot+0x11a)[0x55fd6d536b51]
./btrfs-corrupt-block(main+0x141d)[0x55fd6d56f4a9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f526bee17fd]
./btrfs-corrupt-block(_start+0x2a)[0x55fd6d52d8ba]
Aborted
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#121 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 12 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 8 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks,

Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ?

Quoted 9 lines
Quoted 6 lines

Cool, do

./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device>

Then you should be able to run the init-extent-tree and get past that part. Thanks,

Sorry, I must be triggering every single unhandled path in the code :)

"Make recovery tools more resilient" is definitely muuuuuch higher on the team priority list that's for fucking sure. Try again please, hopefully it works this time. Thanks,

Josef

#122 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

"Make recovery tools more resilient" is definitely muuuuuch higher on the team priority list that's for fucking sure. Try again please, hopefully it works this time. Thanks,

Great to hear, and obviously not just hearing, thanks for your persistence in this mega thread.

Here's the new one:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d
"3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55f248b64600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55f248b64600
Error searching to node -2

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#123 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

"Make recovery tools more resilient" is definitely muuuuuch higher on the team priority list that's for fucking sure. Try again please, hopefully it works this time. Thanks,

Great to hear, and obviously not just hearing, thanks for your persistence in this mega thread.

Quoted 9 lines

Here's the new one:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d
"3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55f248b64600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55f248b64600
Error searching to node -2

-2 is enoent, it must have committed with the deleted block, which is sort of scary but at this point I'll take it. Go ahead and do the init-extent-tree. Thanks,

Josef

#124 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

-2 is enoent, it must have committed with the deleted block, which is sort of scary but at this point I'll take it. Go ahead and do the init-extent-tree. Thanks,

Same output :( Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]

it's the one we just tried to clear, and got enoent on. Unless -r 11223 is not the correct value? (not sure where it came from)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55d8bac70600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55d8bac70600
Error searching to node -2

New run still returns the same thing:

        item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684257792 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451684782080 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7451685568512 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457553899520 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457554423808 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457561710592 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457563021312 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)
        item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53
                generation 1590259 type 1 (regular)
                extent data disk byte 7457564856320 nr 524288
                extent data offset 0 nr 524288 ram 524288
                extent compression 0 (none)

elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48

#1 0x000055555559b0e4 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157801984)
at common/extent-cache.c:179
#2 0x0000555555584c72 in set_extent_bits (tree=0x100000060, start=15645157801984, end=15645157801983, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584ea5 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585888 in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/extent_io.c:976
#5  0x000055555557bc56 in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/disk-io.c:2224
#6  0x000055555557ee7f in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232, 
    parent=0, iref=0xffffffffffffffe3, path=0x55555d273340, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d273340, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555dfeb4 in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x55555d283370, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659
#10 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x555556461260, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#11 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=0x555555f0c680, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734

#12 0x00005555555e0278 in record_root (root=root@entry=0x555556faefe0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e1104 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a4e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#125 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

-2 is enoent, it must have committed with the deleted block, which is sort of scary but at this point I'll take it. Go ahead and do the init-extent-tree. Thanks,

Same output :( Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]

Quoted 9 lines

it's the one we just tried to clear, and got enoent on. Unless -r 11223 is not the correct value? (not sure where it came from)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55d8bac70600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55d8bac70600
Error searching to node -2

Oooh I'm stupid, I thought the key it was printing out was the extent we needed to delete, but it's the extent in the extent tree. You cut off the part I need, but that's because I'm printing the leaf when I don't need to.

I've fixed the output so it should print out something like

[number, 108, number] dumping paths

that's what you want to feed into btrfs-corrupt-block, that should delete the problematic item and then we can continue. Thanks,

Josef

#126 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik

Oooh I'm stupid, I thought the key it was printing out was the extent we needed to delete, but it's the extent in the extent tree. You cut off the part I need, but that's because I'm printing the leaf when I don't need to.

I've fixed the output so it should print out something like

[number, 108, number] dumping paths

that's what you want to feed into btrfs-corrupt-block, that should delete the problematic item and then we can continue. Thanks,

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]
inode ref info failed???
elem_cnt 0 elem_missed 0 ret -2
doing an insert of the bytenr
doing an insert that overlaps our bytenr 3700677820416 262144 processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1634992128 of 1635549184 possible bytesWTF???? we think we already inserted this bytenr?? [1834097, 108, 1835008] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248]
Segmentation fault

Isn't that root 11223 and

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1

why does it say root 11223 does not exists?
FS_INFO IS 0x558818fe3600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x558818fe3600
parent transid verify failed on 13576823668736 wanted 1619060 found 1619070
parent transid verify failed on 13576823668736 wanted 1619060 found 1619070
couldn't find root 11223

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#127 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 14 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik

Oooh I'm stupid, I thought the key it was printing out was the extent we needed to delete, but it's the extent in the extent tree. You cut off the part I need, but that's because I'm printing the leaf when I don't need to.

I've fixed the output so it should print out something like

[number, 108, number] dumping paths

that's what you want to feed into btrfs-corrupt-block, that should delete the problematic item and then we can continue. Thanks,

Quoted 10 lines

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]

Oh huh, we must not have a free space object for this, in that case lets do

./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever

and then do the init. Thanks,

Josef

#128 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik
Quoted 10 lines

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]

Quoted 5 lines

Oh huh, we must not have a free space object for this, in that case lets do

./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever

and then do the init. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
FS_INFO IS 0x558c0e536600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x558c0e536600
Error searching to node -2

not good news I assume?

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#129 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 19 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]

Quoted 5 lines

Oh huh, we must not have a free space object for this, in that case lets do

./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever

and then do the init. Thanks,

Quoted 9 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
FS_INFO IS 0x558c0e536600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x558c0e536600
Error searching to node -2

not good news I assume?

Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,

Josef

#130 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 27 lines · Josef Bacik
Quoted 26 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]

Quoted 5 lines

Oh huh, we must not have a free space object for this, in that case lets do

./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever

and then do the init. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
FS_INFO IS 0x558c0e536600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x558c0e536600
Error searching to node -2
Quoted 4 lines
Quoted 2 lines

not good news I assume?

Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1
FS_INFO IS 0x557f68dd6600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x557f68dd6600

then init-extent-tree:
inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Ignoring transid failure
ERROR: root [11223 4061] level 0 does not match 2

ERROR: Error loading root
it failed?? -5
Init extent tree failed
[Inferior 1 (process 31788) exited with code 0373]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#131 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 53 lines · Marc MERLIN
Quoted 27 lines · Josef Bacik
Quoted 26 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 10 lines

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 49152 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [4088, 108, 0]

Quoted 5 lines

Oh huh, we must not have a free space object for this, in that case lets do

./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/whatever

and then do the init. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 2 /dev/mapper/dshelf1
FS_INFO IS 0x558c0e536600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x558c0e536600
Error searching to node -2
Quoted 4 lines
Quoted 2 lines

not good news I assume?

Just that I´m dumb and making silly mistakes, its -r 1, sorry about that,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1
FS_INFO IS 0x557f68dd6600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x557f68dd6600

then init-extent-tree:
inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes

Quoted 9 lines

Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Ignoring transid failure
ERROR: root [11223 4061] level 0 does not match 2

I'm going to scream. Somehow the root pointer for 11223 got messed up in all of this, do rescue tree-recover again so it can unfuck 11223, and then init-extent-tree. Thanks,

Josef

#132 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

I'm going to scream. Somehow the root pointer for 11223 got messed up in all of this, do rescue tree-recover again so it can unfuck 11223, and then init-extent-tree. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5588db830bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5588db830bc0
Checking root 2 bytenr 15645018570752
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018554368
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823668736
Repairing root 11223 bad_blocks 9 update 1
deleting slot 54 in block 11652138631168
deleting slot 54 in block 11652138631168
deleting slot 64 in block 11652138631168
deleting slot 86 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781491265536
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781492101120
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781492756480
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781495197696
Checking root 163303 bytenr 15645365993472
Checking root 163316 bytenr 6781496393728
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11971021275136
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970667446272
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823799808
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634844672 of 1635549184 possible bytesadding a bytenr that overlaps our thing, dumping paths for [1834097, 108, 1835008]
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
doing an insert of the bytenr
doing an insert that overlaps our bytenr 3700677820416 53248
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesFailed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555564fce1f0, parent_transid=parent_transid@entry=1619126)

    at kernel-shared/disk-io.c:2235
2235            ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,
(gdb) bt
#0  0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555564fce1f0, parent_transid=parent_transid@entry=1619126)
    at kernel-shared/disk-io.c:2235
#1  0x000055555557bebd in read_tree_block (fs_info=fs_info@entry=0x55555564cbc0, bytenr=<optimized out>, parent_transid=1619126)
    at kernel-shared/disk-io.c:378
#2  0x00005555555735f6 in read_node_slot (fs_info=fs_info@entry=0x55555564cbc0, root=root@entry=0x5555559547d0, 
    parent=0x555559e7e7b0, slot=240) at ./kernel-shared/ctree.h:1941
#3  0x00005555555769ff in btrfs_search_slot (trans=0x5555599539a0, root=root@entry=0x5555559547d0, key=key@entry=0x7fffffffd8a0, 
    p=p@entry=0x7fffffffd760, ins_len=ins_len@entry=49, cow=cow@entry=1) at kernel-shared/ctree.c:1429
#4  0x0000555555576e91 in btrfs_insert_empty_items (trans=trans@entry=0x5555599539a0, root=root@entry=0x5555559547d0, 
    path=path@entry=0x7fffffffd760, cpu_key=cpu_key@entry=0x7fffffffd8a0, data_size=data_size@entry=0x7fffffffd75c, nr=nr@entry=1)
    at kernel-shared/ctree.c:2824
#5  0x00005555555dfb9d in btrfs_insert_empty_item (data_size=<optimized out>, key=0x7fffffffd8a0, path=0x7fffffffd760, 
    root=0x5555559547d0, trans=0x5555599539a0) at ./kernel-shared/ctree.h:2780
#6  insert_empty_extent (trans=trans@entry=0x5555599539a0, key=key@entry=0x7fffffffd8a0, generation=generation@entry=1591667, 
    flags=flags@entry=1) at cmds/rescue-init-extent-tree.c:566
#7  0x00005555555dfe62 in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=eb@entry=0x555559092800, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:645
#8  0x00005555555dffed in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=eb@entry=0x5555590aaf80, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734
#9  0x00005555555dffed in process_eb (trans=trans@entry=0x5555599539a0, root=root@entry=0x555562939390, eb=0x5555590822a0, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734

#10 0x00005555555e0258 in record_root (root=root@entry=0x555562939390) at cmds/rescue-init-extent-tree.c:805 #11 0x00005555555e10e4 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #12 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #13 0x00005555555d7a2e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)

    at cmds/rescue.c:65
#14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#133 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 88 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

I'm going to scream. Somehow the root pointer for 11223 got messed up in all of this, do rescue tree-recover again so it can unfuck 11223, and then init-extent-tree. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x5588db830bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5588db830bc0
Checking root 2 bytenr 15645018570752
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018554368
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823668736
Repairing root 11223 bad_blocks 9 update 1
deleting slot 54 in block 11652138631168
deleting slot 54 in block 11652138631168
deleting slot 64 in block 11652138631168
deleting slot 86 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
deleting slot 87 in block 11652138631168
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781491265536
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781492101120
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781492756480
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781495197696
Checking root 163303 bytenr 15645365993472
Checking root 163316 bytenr 6781496393728
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11971021275136
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970667446272
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823799808
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1634844672 of 1635549184 possible bytesadding a bytenr that overlaps our thing, dumping paths for [1834097, 108, 1835008]
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules
doing an insert of the bytenr
doing an insert that overlaps our bytenr 3700677820416 53248
processed 1635319808 of 1635549184 possible bytes
Quoted 8 lines
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesFailed to find [10467695652864, 168, 8675328]

Yayyyy progress, I just updated the debugging for the new problem bytenr so we can figure out who to delete, run init-extent-tree again please. Thanks,

Josef

#134 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Yayyyy progress, I just updated the debugging for the new problem bytenr so we can figure out who to delete, run init-extent-tree again please. Thanks,

sure:
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

We're making progress, thanks. Is there a way to add a BUG_ON instead of the subsequent SEGV?

I'm also not too sure, how to you convert [10467695652864, 168, 8675328] into something like

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "4088,108,0" -r 1 /dev/mapper/dshelf1

It seems like the straight numbers don't work:

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "10467695652864,168,8675328" -r 1 /dev/mapper/dshelf1
FS_INFO IS 0x55e92004b600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55e92004b600
Error searching to node -2

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#135 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 41 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Yayyyy progress, I just updated the debugging for the new problem bytenr so we can figure out who to delete, run init-extent-tree again please. Thanks,

sure:
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Quoted 7 lines

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

We're making progress, thanks. Is there a way to add a BUG_ON instead of the subsequent SEGV?

Yup done. Also it's because I'm looking for the key in "dumping paths" higher up. This one is weird because we only got one output, so for this one you want

-d "76300,108,0" -r 160494

Thanks,

Josef

#136 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik
Quoted 11 lines

processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

Quoted 8 lines
Quoted 2 lines

We're making progress, thanks. Is there a way to add a BUG_ON instead of the subsequent SEGV?

Yup done. Also it's because I'm looking for the key in "dumping paths" higher up. This one is weird because we only got one output, so for this one you want

-d "76300,108,0" -r 160494

Not sure how you got those numbers, but sure. I think it worked, but it doesn't give any output saying so

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 160494 /dev/mapper/dshelf1
FS_INFO IS 0x5563243d9600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5563243d9600

not sure it worked:
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes
Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 81920 of 109019136 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x5555611a6d20, parent_transid=parent_transid@entry=1619156)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#137 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 57 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 11 lines

processed 999424 of 109035520 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555565529a60, parent_transid=parent_transid@entry=1619139)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

Quoted 8 lines
Quoted 2 lines

We're making progress, thanks. Is there a way to add a BUG_ON instead of the subsequent SEGV?

Yup done. Also it's because I'm looking for the key in "dumping paths" higher up. This one is weird because we only got one output, so for this one you want

-d "76300,108,0" -r 160494

Not sure how you got those numbers, but sure. I think it worked, but it doesn't give any output saying so

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 160494 /dev/mapper/dshelf1
FS_INFO IS 0x5563243d9600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5563243d9600

not sure it worked:
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes
Recording extents for root 159785
processed 108855296 of 108855296 possible bytes
Recording extents for root 159787
processed 49152 of 49479680 possible bytes
Recording extents for root 160494
processed 1179648 of 109035520 possible bytes

Quoted 8 lines

Recording extents for root 160496
processed 49152 of 49479680 possible bytes
Recording extents for root 161197
processed 81920 of 109019136 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

It did, now we're on a different root, you can do

btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>

And keep it rolling. Thanks,

Josef

#138 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

It did, now we're on a different root, you can do

btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>

So, it looks like this is going to be long manual process, I went through iterations and keep getting new roots. When I clear one, it goes to the next.
Any way to automate this?

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 161197 /dev/mapper/dshelf1
FS_INFO IS 0x55be3db38600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55be3db38600

Still breaking on the same file,
(...)
Recording extents for root 162632 <------------------------------------------------------------------------- processed 1523712 of 109314048 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x555562149b30, parent_transid=parent_transid@entry=1619175)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 162632 /dev/mapper/dshelf1
FS_INFO IS 0x55a45a034600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55a45a034600

And then same thing again next round:
processed 49152 of 49479680 possible bytes
Recording extents for root 163302 <--------------------------------------------------------------------------- processed 32768 of 109314048 possible bytesWTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x55555fbab9e0, parent_transid=parent_transid@entry=1619197)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1
FS_INFO IS 0x564c9f705600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564c9f705600

processed 81920 of 75792384 possible bytes
Recording extents for root 163316 <--------------------------------------------------------------------------- WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
misc/add0/new/tv1/ST/testfile.avi
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
0x000055555557bc5d in btrfs_buffer_uptodate (buf=buf@entry=0x5555608f4060, parent_transid=parent_transid@entry=1619222)

    at kernel-shared/disk-io.c:2235

2235 ret = verify_parent_transid(&buf->fs_info->extent_cache, buf,

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#139 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

It did, now we're on a different root, you can do

btrfs-corrupt-block -d "76300,108,0" -r 161197 <device>

So, it looks like this is going to be long manual process, I went through iterations and keep getting new roots. When I clear one, it goes to the next.
Any way to automate this?

I'm afraid it'll be longer/less safe for me to work out the kinks than to continue manually removing stuff. If you have to do it say 10 more times let me know and I can try and rig up something that can dump root ids that you can just feed into btrfs-corrupt-block to clear everything out at once. Thanks,

Josef

#140 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Josef Bacik

I'm afraid it'll be longer/less safe for me to work out the kinks than to continue manually removing stuff. If you have to do it say 10 more times let me know and I can try and rig up something that can dump root ids that you can just feed into btrfs-corrupt-block to clear everything out at once. Thanks,

Took a while:
./btrfs-corrupt-block -d "76300,108,0" -r 163302 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163316 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163920 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 164624 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165098 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165198 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165298 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 16384 /dev/mapper/dshelf1

Now I'm stuck here:

processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823734272 len 16384 Init extent tree failed
[Inferior 1 (process 31377) exited with code 0377]

./btrfs-corrupt-block -d "76300,108,0" -r 18446744073709551607 /dev/mapper/dshelf1

Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823734272 len 16384 Init extent tree failed

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#141 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 22 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik

I'm afraid it'll be longer/less safe for me to work out the kinks than to continue manually removing stuff. If you have to do it say 10 more times let me know and I can try and rig up something that can dump root ids that you can just feed into btrfs-corrupt-block to clear everything out at once. Thanks,

Took a while:
./btrfs-corrupt-block -d "76300,108,0" -r 163302 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163303 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163316 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 163920 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 164624 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165098 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165198 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 165298 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "76300,108,0" -r 16384 /dev/mapper/dshelf1

Quoted 2 lines

Now I'm stuck here:

Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,

Josef

#142 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,

Great

FYI:
cmds/rescue-init-extent-tree.c:417:64: warning: passing argument 1 of ‘PTR_ERR’ makes pointer from integer without a cast [-Wint-conversion]

  417 |   fprintf(stderr, "Error reading data reloc tree %d\n",PTR_ERR(ret));
      |                                                                ^~~
      |                                                                |
      |                                                                int
In file included from cmds/rescue-init-extent-tree.c:3:
./kerncompat.h:270:40: note: expected ‘const void *’ but argument is of type ‘int’
  270 | static inline long PTR_ERR(const void *ptr)
      |                            ~~~~~~~~~~~~^~~
cmds/rescue-init-extent-tree.c:417:51: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
  417 |   fprintf(stderr, "Error reading data reloc tree %d\n",PTR_ERR(ret));
      |                                                  ~^    ~~~~~~~~~~~~
      |                                                   |    |
      |                                                   int  long int
      |                                                  %ld

(..)
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
DID THE DATA RELOC TREE INIT?!
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221
processed 16384 of 255983616 possible bytes Recording extents for root 11222
processed 49479680 of 49479680 possible bytes Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224
processed 75792384 of 75792384 possible bytes Recording extents for root 159785
processed 108855296 of 108855296 possible bytes Recording extents for root 159787
processed 49152 of 49479680 possible bytes Recording extents for root 160494
processed 1179648 of 109035520 possible bytes Recording extents for root 160496
processed 49152 of 49479680 possible bytes Recording extents for root 161197
processed 147456 of 109019136 possible bytes Recording extents for root 161199
processed 49152 of 49479680 possible bytes Recording extents for root 162628
processed 49152 of 49479680 possible bytes Recording extents for root 162632
processed 2129920 of 109314048 possible bytes Recording extents for root 162645
processed 49152 of 75792384 possible bytes Recording extents for root 163298
processed 49152 of 49479680 possible bytes Recording extents for root 163302
processed 147456 of 109314048 possible bytes Recording extents for root 163303
processed 81920 of 75792384 possible bytes Recording extents for root 163316
processed 49152 of 109314048 possible bytes Recording extents for root 163318
processed 16384 of 49479680 possible bytes Recording extents for root 163916
processed 49152 of 49479680 possible bytes Recording extents for root 163920
processed 81920 of 109314048 possible bytes Recording extents for root 163921
processed 49152 of 75792384 possible bytes Recording extents for root 164620
processed 49152 of 49479680 possible bytes Recording extents for root 164624
processed 491520 of 109445120 possible bytes Recording extents for root 164633
processed 49152 of 75792384 possible bytes Recording extents for root 165098
processed 212992 of 109445120 possible bytes Recording extents for root 165100
processed 16384 of 49479680 possible bytes Recording extents for root 165198
processed 49152 of 109445120 possible bytes Recording extents for root 165200
processed 16384 of 49479680 possible bytes Recording extents for root 165294
processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823652352 len 16384 Init extent tree failed
[Inferior 1 (process 9144) exited with code 0377] (gdb)

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#143 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,

Great

Ok this is easier, the transaction commit stuff doesn't work quite right in progs, I've fixed the code and it should succeed this time. Thanks,

Josef

#144 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,

Quoted 6 lines
Quoted 2 lines

Great

Ok this is easier, the transaction commit stuff doesn't work quite right in progs, I've fixed the code and it should succeed this time. Thanks,

ainserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221
processed 16384 of 255983616 possible bytes Recording extents for root 11222
processed 49479680 of 49479680 possible bytes Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224
processed 75792384 of 75792384 possible bytes Recording extents for root 159785
processed 108855296 of 108855296 possible bytes Recording extents for root 159787
processed 49152 of 49479680 possible bytes Recording extents for root 160494
processed 1179648 of 109035520 possible bytes Recording extents for root 160496
processed 49152 of 49479680 possible bytes Recording extents for root 161197
processed 147456 of 109019136 possible bytes Recording extents for root 161199
processed 49152 of 49479680 possible bytes Recording extents for root 162628
processed 49152 of 49479680 possible bytes Recording extents for root 162632
processed 2129920 of 109314048 possible bytes Recording extents for root 162645
processed 49152 of 75792384 possible bytes Recording extents for root 163298
processed 49152 of 49479680 possible bytes Recording extents for root 163302
processed 147456 of 109314048 possible bytes Recording extents for root 163303
processed 81920 of 75792384 possible bytes Recording extents for root 163316
processed 49152 of 109314048 possible bytes Recording extents for root 163318
processed 16384 of 49479680 possible bytes Recording extents for root 163916
processed 49152 of 49479680 possible bytes Recording extents for root 163920
processed 81920 of 109314048 possible bytes Recording extents for root 163921
processed 49152 of 75792384 possible bytes Recording extents for root 164620
processed 49152 of 49479680 possible bytes Recording extents for root 164624
processed 491520 of 109445120 possible bytes Recording extents for root 164633
processed 49152 of 75792384 possible bytes Recording extents for root 165098
processed 212992 of 109445120 possible bytes Recording extents for root 165100
processed 16384 of 49479680 possible bytes Recording extents for root 165198
processed 49152 of 109445120 possible bytes Recording extents for root 165200
processed 16384 of 49479680 possible bytes Recording extents for root 165294
processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823832576 len 16384 Init extent tree failed
[Inferior 1 (process 28313) exited with code 0377]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#145 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 89 lines · Marc MERLIN
Quoted 7 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Hooray we're at the data reloc root. It should have been cleared tho, so I've fixed it up to see if it's doing the right thing, it should clear it this time, if it doesn't let me know. Thanks,

Quoted 6 lines
Quoted 2 lines

Great

Ok this is easier, the transaction commit stuff doesn't work quite right in progs, I've fixed the code and it should succeed this time. Thanks,

ainserting block group 15837217947648 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Recording extents for root 11223 processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224 processed 75792384 of 75792384 possible bytes Recording extents for root 159785 processed 108855296 of 108855296 possible bytes Recording extents for root 159787 processed 49152 of 49479680 possible bytes Recording extents for root 160494 processed 1179648 of 109035520 possible bytes Recording extents for root 160496 processed 49152 of 49479680 possible bytes Recording extents for root 161197 processed 147456 of 109019136 possible bytes Recording extents for root 161199 processed 49152 of 49479680 possible bytes Recording extents for root 162628 processed 49152 of 49479680 possible bytes Recording extents for root 162632 processed 2129920 of 109314048 possible bytes Recording extents for root 162645 processed 49152 of 75792384 possible bytes Recording extents for root 163298 processed 49152 of 49479680 possible bytes Recording extents for root 163302 processed 147456 of 109314048 possible bytes Recording extents for root 163303 processed 81920 of 75792384 possible bytes Recording extents for root 163316 processed 49152 of 109314048 possible bytes Recording extents for root 163318 processed 16384 of 49479680 possible bytes Recording extents for root 163916 processed 49152 of 49479680 possible bytes Recording extents for root 163920 processed 81920 of 109314048 possible bytes Recording extents for root 163921 processed 49152 of 75792384 possible bytes Recording extents for root 164620 processed 49152 of 49479680 possible bytes Recording extents for root 164624 processed 491520 of 109445120 possible bytes Recording extents for root 164633 processed 49152 of 75792384 possible bytes Recording extents for root 165098 processed 212992 of 109445120 possible bytes Recording extents for root 165100 processed 16384 of 49479680 possible bytes Recording extents for root 165198 processed 49152 of 109445120 possible bytes Recording extents for root 165200 processed 16384 of 49479680 possible bytes Recording extents for root 165294

Quoted 8 lines

processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction

Well it looks like it finished, but I don't see my "start transaction failed" messages which should be printing, I've added some extra debugging to figure out wherever this is failing. We're literally done and of course it's failing somewhere at the end, hopefully this'll be quicker to nail down. Thanks,

Josef

#146 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Josef Bacik
Quoted 5 lines

Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction

Well it looks like it finished, but I don't see my "start transaction failed" messages which should be printing, I've added some extra debugging to figure out wherever this is failing. We're literally done and of course it's failing somewhere at the end, hopefully this'll be quicker to nail down. Thanks,

Good news :)
Here is the new one:
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221
processed 16384 of 255983616 possible bytes Recording extents for root 11222
processed 49479680 of 49479680 possible bytes Recording extents for root 11223
processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224
processed 75792384 of 75792384 possible bytes Recording extents for root 159785
processed 108855296 of 108855296 possible bytes Recording extents for root 159787
processed 49152 of 49479680 possible bytes Recording extents for root 160494
processed 1179648 of 109035520 possible bytes Recording extents for root 160496
processed 49152 of 49479680 possible bytes Recording extents for root 161197
processed 147456 of 109019136 possible bytes Recording extents for root 161199
processed 49152 of 49479680 possible bytes Recording extents for root 162628
processed 49152 of 49479680 possible bytes Recording extents for root 162632
processed 2129920 of 109314048 possible bytes Recording extents for root 162645
processed 49152 of 75792384 possible bytes Recording extents for root 163298
processed 49152 of 49479680 possible bytes Recording extents for root 163302
processed 147456 of 109314048 possible bytes Recording extents for root 163303
processed 81920 of 75792384 possible bytes Recording extents for root 163316
processed 49152 of 109314048 possible bytes Recording extents for root 163318
processed 16384 of 49479680 possible bytes Recording extents for root 163916
processed 49152 of 49479680 possible bytes Recording extents for root 163920
processed 81920 of 109314048 possible bytes Recording extents for root 163921
processed 49152 of 75792384 possible bytes Recording extents for root 164620
processed 49152 of 49479680 possible bytes Recording extents for root 164624
processed 491520 of 109445120 possible bytes Recording extents for root 164633
processed 49152 of 75792384 possible bytes Recording extents for root 165098
processed 212992 of 109445120 possible bytes Recording extents for root 165100
processed 16384 of 49479680 possible bytes Recording extents for root 165198
processed 49152 of 109445120 possible bytes Recording extents for root 165200
processed 16384 of 49479680 possible bytes Recording extents for root 165294
processed 16384 of 49479680 possible bytes Recording extents for root 165298
processed 81920 of 109445120 possible bytes Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823750656 len 16384 Init extent tree failed
[Inferior 1 (process 8777) exited with code 0377]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#147 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 83 lines · Marc MERLIN
Quoted 11 lines · Josef Bacik
Quoted 5 lines

Recording extents for root 165299
processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
ERROR: commit_root already set when starting transaction

Well it looks like it finished, but I don't see my "start transaction failed" messages which should be printing, I've added some extra debugging to figure out wherever this is failing. We're literally done and of course it's failing somewhere at the end, hopefully this'll be quicker to nail down. Thanks,

Good news :)
Here is the new one:
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Recording extents for root 11223 processed 1635319808 of 1635549184 possible bytes Recording extents for root 11224 processed 75792384 of 75792384 possible bytes Recording extents for root 159785 processed 108855296 of 108855296 possible bytes Recording extents for root 159787 processed 49152 of 49479680 possible bytes Recording extents for root 160494 processed 1179648 of 109035520 possible bytes Recording extents for root 160496 processed 49152 of 49479680 possible bytes Recording extents for root 161197 processed 147456 of 109019136 possible bytes Recording extents for root 161199 processed 49152 of 49479680 possible bytes Recording extents for root 162628 processed 49152 of 49479680 possible bytes Recording extents for root 162632 processed 2129920 of 109314048 possible bytes Recording extents for root 162645 processed 49152 of 75792384 possible bytes Recording extents for root 163298 processed 49152 of 49479680 possible bytes Recording extents for root 163302 processed 147456 of 109314048 possible bytes Recording extents for root 163303 processed 81920 of 75792384 possible bytes Recording extents for root 163316 processed 49152 of 109314048 possible bytes Recording extents for root 163318 processed 16384 of 49479680 possible bytes Recording extents for root 163916 processed 49152 of 49479680 possible bytes Recording extents for root 163920 processed 81920 of 109314048 possible bytes Recording extents for root 163921 processed 49152 of 75792384 possible bytes Recording extents for root 164620 processed 49152 of 49479680 possible bytes Recording extents for root 164624 processed 491520 of 109445120 possible bytes Recording extents for root 164633 processed 49152 of 75792384 possible bytes Recording extents for root 165098 processed 212992 of 109445120 possible bytes Recording extents for root 165100 processed 16384 of 49479680 possible bytes Recording extents for root 165198 processed 49152 of 109445120 possible bytes Recording extents for root 165200 processed 16384 of 49479680 possible bytes Recording extents for root 165294 processed 16384 of 49479680 possible bytes

Quoted 8 lines

Recording extents for root 165298 processed 81920 of 109445120 possible bytes Recording extents for root 165299 processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes doing block accounting
doing close???

Ok must be in the block accounting stuff which has 0 prints, fixed that up. Thanks,

Josef

#148 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Josef Bacik
Quoted 7 lines

processed 81920 of 109445120 possible bytes Recording extents for root 165299 processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes doing block accounting
doing close???

Ok must be in the block accounting stuff which has 0 prints, fixed that up. Thanks,

Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#149 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 7 lines

processed 81920 of 109445120 possible bytes Recording extents for root 165299 processed 16384 of 75792384 possible bytes Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes doing block accounting
doing close???

Ok must be in the block accounting stuff which has 0 prints, fixed that up. Thanks,

Quoted 7 lines

Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed
FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

Sorry was on airplanes, pushed some more debugging. Thanks,

Josef

#150 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 1 line · Josef Bacik

Sorry was on airplanes, pushed some more debugging. Thanks,

no worries

Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed
[Inferior 1 (process 23375) exited with code 0377]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#151 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Sorry was on airplanes, pushed some more debugging. Thanks,

no worries

Quoted 13 lines

Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

More debugging, this really shouldn't happen because we wouldn't be able to map the bytenr. Thanks,

Josef

#152 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 21 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Sorry was on airplanes, pushed some more debugging. Thanks,

no worries

Quoted 13 lines

Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

Quoted 2 lines

More debugging, this really shouldn't happen because we wouldn't be able to map the bytenr. Thanks,

Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#153 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 37 lines · Marc MERLIN
Quoted 21 lines · Josef Bacik
Quoted 6 lines · Marc MERLIN
Quoted 1 line · Josef Bacik

Sorry was on airplanes, pushed some more debugging. Thanks,

no worries

Quoted 13 lines

Recording extents for root 165294
processed 16384 of 49479680 possible bytes
Recording extents for root 165298
processed 81920 of 109445120 possible bytes
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607 processed 16384 of 16384 possible bytes
doing block accounting
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

Quoted 2 lines

More debugging, this really shouldn't happen because we wouldn't be able to map the bytenr. Thanks,

Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

Quoted 5 lines

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed

Wtf, that's definitely in the range of block groups that I've seen printed. I added some more stuff, hoping the tree search is just broken. Thanks,

Josef

#154 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 17 lines · Josef Bacik
Quoted 11 lines

Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

Quoted 5 lines

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed

Quoted 3 lines

Wtf, that's definitely in the range of block groups that I've seen printed. I added some more stuff, hoping the tree search is just broken. Thanks,

Recording extents for root 163303
processed 81920 of 75792384 possible bytes
inserting block group 15731991248896
inserting block group 15733064990720
inserting block group 15734138732544
inserting block group 15735212474368
inserting block group 15736286216192
inserting block group 15737359958016
inserting block group 15738433699840
inserting block group 15739507441664
inserting block group 15740581183488
inserting block group 15741654925312
inserting block group 15742728667136
inserting block group 15743802408960
inserting block group 15744876150784
inserting block group 15745949892608
inserting block group 15747023634432
inserting block group 15748097376256
inserting block group 15749171118080
inserting block group 15750244859904
inserting block group 15751318601728
inserting block group 15752392343552
inserting block group 15753466085376
inserting block group 15754539827200
inserting block group 15755613569024
inserting block group 15756687310848
inserting block group 15757761052672
inserting block group 15758834794496
inserting block group 15759908536320
inserting block group 15760982278144
inserting block group 15762056019968
inserting block group 15763129761792
inserting block group 15764203503616
inserting block group 15765277245440
inserting block group 15766350987264
inserting block group 15767424729088
inserting block group 15768498470912
inserting block group 15769572212736
inserting block group 15770645954560
inserting block group 15771719696384
inserting block group 15778162147328
inserting block group 15779235889152
inserting block group 15780309630976
inserting block group 15781383372800
inserting block group 15782457114624
inserting block group 15783530856448
inserting block group 15784604598272
inserting block group 15785678340096
inserting block group 15786752081920
inserting block group 15787825823744
inserting block group 15788899565568
inserting block group 15789973307392
inserting block group 15791047049216
inserting block group 15792120791040
inserting block group 15793194532864
inserting block group 15794268274688
inserting block group 15795342016512
inserting block group 15796415758336
inserting block group 15797489500160
inserting block group 15798563241984
inserting block group 15799636983808
inserting block group 15800710725632
inserting block group 15801784467456
inserting block group 15802858209280
inserting block group 15803931951104
inserting block group 15809300660224
inserting block group 15810374402048
inserting block group 15811448143872
inserting block group 15812521885696
inserting block group 15813595627520
inserting block group 15814669369344
inserting block group 15815743111168
inserting block group 15816816852992
inserting block group 15817890594816
inserting block group 15818964336640
inserting block group 15820038078464
inserting block group 15821111820288
inserting block group 15822185562112
cache 15738433699840 15739507441664 in range no
cache 15739507441664 15740581183488 in range no
cache 15740581183488 15741654925312 in range no
cache 15741654925312 15742728667136 in range no
cache 15742728667136 15743802408960 in range no
cache 15743802408960 15744876150784 in range no
cache 15744876150784 15745949892608 in range no
cache 15745949892608 15747023634432 in range no
cache 15747023634432 15748097376256 in range no
cache 15748097376256 15749171118080 in range no
cache 15749171118080 15750244859904 in range no
cache 15750244859904 15751318601728 in range no
cache 15751318601728 15752392343552 in range no
cache 15752392343552 15753466085376 in range no
cache 15753466085376 15754539827200 in range no
cache 15754539827200 15755613569024 in range no
cache 15755613569024 15756687310848 in range no
cache 15756687310848 15757761052672 in range no
cache 15757761052672 15758834794496 in range no
cache 15758834794496 15759908536320 in range no
cache 15759908536320 15760982278144 in range no
cache 15760982278144 15762056019968 in range no
cache 15762056019968 15763129761792 in range no
cache 15763129761792 15764203503616 in range no
cache 15764203503616 15765277245440 in range no
cache 15765277245440 15766350987264 in range no
cache 15766350987264 15767424729088 in range no
cache 15767424729088 15768498470912 in range no
cache 15768498470912 15769572212736 in range no
cache 15769572212736 15770645954560 in range no
cache 15770645954560 15771719696384 in range no
cache 15771719696384 15772793438208 in range no
cache 15778162147328 15779235889152 in range no
cache 15779235889152 15780309630976 in range no
cache 15780309630976 15781383372800 in range no
cache 15781383372800 15782457114624 in range no
cache 15782457114624 15783530856448 in range no
cache 15783530856448 15784604598272 in range no
cache 15784604598272 15785678340096 in range no
cache 15785678340096 15786752081920 in range no
cache 15786752081920 15787825823744 in range no
cache 15787825823744 15788899565568 in range no
cache 15788899565568 15789973307392 in range no
cache 15789973307392 15791047049216 in range no
cache 15791047049216 15792120791040 in range no
cache 15792120791040 15793194532864 in range no
cache 15793194532864 15794268274688 in range no
cache 15794268274688 15795342016512 in range no
cache 15795342016512 15796415758336 in range no
cache 15796415758336 15797489500160 in range no
cache 15797489500160 15798563241984 in range no
cache 15798563241984 15799636983808 in range no
cache 15799636983808 15800710725632 in range no
cache 15800710725632 15801784467456 in range no
cache 15801784467456 15802858209280 in range no
cache 15802858209280 15803931951104 in range no
cache 15803931951104 15805005692928 in range no
cache 15809300660224 15810374402048 in range no
cache 15810374402048 15811448143872 in range no
cache 15811448143872 15812521885696 in range no
cache 15812521885696 15813595627520 in range no
cache 15813595627520 15814669369344 in range no
cache 15814669369344 15815743111168 in range no
cache 15815743111168 15816816852992 in range no
cache 15816816852992 15817890594816 in range no
cache 15817890594816 15818964336640 in range no
cache 15818964336640 15820038078464 in range no
cache 15820038078464 15821111820288 in range no
cache 15821111820288 15822185562112 in range no
cache 15822185562112 15823259303936 in range no
cache 15823259303936 15824333045760 in range no
cache 15824333045760 15825406787584 in range no
cache 15825406787584 15826480529408 in range no
cache 15826480529408 15827554271232 in range no
cache 15827554271232 15828628013056 in range no
cache 15828628013056 15829701754880 in range no
cache 15829701754880 15830775496704 in range no
cache 15830775496704 15831849238528 in range no
cache 15831849238528 15832922980352 in range no
cache 15832922980352 15833996722176 in range no
cache 15833996722176 15835070464000 in range no
cache 15835070464000 15836144205824 in range no
cache 15836144205824 15837217947648 in range no
cache 15837217947648 15838291689472 in range no
cache 15838291689472 15839365431296 in range no
cache 15839365431296 15840439173120 in range no
cache 15840439173120 15841512914944 in range no
cache 15842586656768 15843660398592 in range no
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823816192 len 16384 Init extent tree failed

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#155 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 19 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 16 lines

Sure thing:
Recording extents for root 165299
processed 16384 of 75792384 possible bytes
Recording extents for root 18446744073709551607
processed 16384 of 16384 possible bytes
doing block accounting
couldn't find a block group at bytenr 15847955365888 total left 1073217536
ERROR: update block group failed 15847955365888 1073217536 ret -1 FIX BLOCK ACCOUNTING FAILED -1
ERROR: The commit failed???? -1

doing close???
ERROR: commit_root already set when starting transaction extent buffer leak: start 13576823767040 len 16384 Init extent tree failed

Quoted 4 lines
Quoted 3 lines

Wtf, that's definitely in the range of block groups that I've seen printed. I added some more stuff, hoping the tree search is just broken. Thanks,

Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,

Josef

#156 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,

That worked. How do I delete this one?

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0027_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1
./btrfs(+0x8b9fc)[0x5575fbec99fc]
./btrfs(+0x8c0df)[0x5575fbeca0df]
./btrfs(+0x8c22d)[0x5575fbeca22d]
./btrfs(+0x8c22d)[0x5575fbeca22d]
./btrfs(+0x8c49b)[0x5575fbeca49b]
./btrfs(btrfs_init_extent_tree+0xe03)[0x5575fbecb37b]
./btrfs(+0x83b7d)[0x5575fbec1b7d]
./btrfs(handle_command_group+0x49)[0x5575fbe5617b]
./btrfs(main+0x94)[0x5575fbe56275]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7fa2455a77fd]
./btrfs(_start+0x2a)[0x5575fbe55e1a]
Aborted

Thanks
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#157 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,

That worked. How do I delete this one?

Quoted 15 lines

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223

btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>

Thanks,

Josef

#158 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 24 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,

That worked. How do I delete this one?

Quoted 15 lines

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223

Quoted 1 line

btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>

Whoops, it was right there in the text, sorry

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x564360382600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564360382600

Didn't help?
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0026_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#159 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 37 lines · Marc MERLIN
Quoted 24 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ok I've fixed it to yell about what file has this weird extent so you can delete it and we can carry on. Thanks,

That worked. How do I delete this one?

Quoted 15 lines

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223

Quoted 1 line

btrfs-corrupt-block -d "1819130,108,0" -r 11223 <device>

Whoops, it was right there in the text, sorry

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x564360382600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x564360382600
Quoted 16 lines

Didn't help?
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223

It's a different inode number, 1819131 instead of 1819130. This is going to be the frustrating delete shit until it works thing again. Thanks,

Josef

#160 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

It's a different inode number, 1819131 instead of 1819130. This is going to be the frustrating delete shit until it works thing again.

Indeed, I missed that.

Now I'm stuck here:
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55ca68c6d600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55ca68c6d600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
couldn't find root 11223

Mmmh, that's not great looking....

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x56267a388600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56267a388600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
couldn't find root 11223
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#161 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 24 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

It's a different inode number, 1819131 instead of 1819130. This is going to be the frustrating delete shit until it works thing again.

Indeed, I missed that.

Now I'm stuck here:
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x55ca68c6d600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55ca68c6d600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
couldn't find root 11223
Quoted 2 lines

Mmmh, that's not great looking....

Ugh IDK why that happens every once and a while. I pushed a fix for btrfs-corrupt-block, it should work now. Thanks,

Josef

#162 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Ugh IDK why that happens every once and a while. I pushed a fix for btrfs-corrupt-block, it should work now. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x562d920a7600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x562d920a7600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
FS_INFO IS 0x5649e173dbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5649e173dbc0
Walking all our trees and pinning down the currently accessible blocks
(..,)

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,134217728 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1

Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819135,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/ise_0007_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1 
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1 

Ok, now we're down to another one:
(...)
item 143 key (13577065349120 METADATA_ITEM 0) itemoff 11801 itemsize 33
refs 1 gen 1451459 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 144 key (13577069330432 METADATA_ITEM 0) itemoff 11768 itemsize 33
refs 1 gen 1538467 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 145 key (13577073082368 METADATA_ITEM 0) itemoff 11735 itemsize 33
refs 1 gen 1451996 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 146 key (13577075130368 METADATA_ITEM 0) itemoff 11702 itemsize 33
refs 1 gen 1452003 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 147 key (13577080455168 METADATA_ITEM 0) itemoff 11669 itemsize 33
refs 1 gen 1452018 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 148 key (13577081552896 METADATA_ITEM 0) itemoff 11636 itemsize 33
refs 1 gen 1452010 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 149 key (13577082077184 METADATA_ITEM 0) itemoff 11603 itemsize 33
refs 1 gen 1538321 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 150 key (13577082208256 METADATA_ITEM 0) itemoff 11570 itemsize 33
refs 1 gen 1538467 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 151 key (13577082552320 METADATA_ITEM 0) itemoff 11537 itemsize 33
refs 1 gen 1452024 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 152 key (13577085927424 METADATA_ITEM 0) itemoff 11504 itemsize 33
refs 1 gen 1452031 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 153 key (13577088073728 METADATA_ITEM 0) itemoff 11471 itemsize 33
refs 1 gen 1443442 flags TREE_BLOCK
tree block skinny level 0
tree block backref root DEV_TREE
item 154 key (13577088696320 METADATA_ITEM 0) itemoff 11438 itemsize 33
refs 1 gen 1452038 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 155 key (13577090547712 METADATA_ITEM 0) itemoff 11405 itemsize 33
refs 1 gen 1590216 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 11223
item 156 key (13577090596864 METADATA_ITEM 0) itemoff 11372 itemsize 33
refs 1 gen 1590216 flags TREE_BLOCK
tree block skinny level 0
tree block backref root 11223
item 157 key (13577091399680 METADATA_ITEM 0) itemoff 11339 itemsize 33
refs 1 gen 1538745 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 158 key (13577093103616 METADATA_ITEM 0) itemoff 11306 itemsize 33
refs 1 gen 1452046 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 159 key (13577095495680 METADATA_ITEM 0) itemoff 11273 itemsize 33
refs 1 gen 1452053 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
item 160 key (13577095544832 METADATA_ITEM 0) itemoff 11240 itemsize 33
refs 1 gen 1538650 flags TREE_BLOCK|FULL_BACKREF
tree block skinny level 0
shared block backref parent 15645094871040
item 161 key (13577099837440 METADATA_ITEM 0) itemoff 11207 itemsize 33
refs 1 gen 1452063 flags TREE_BLOCK
tree block skinny level 0
tree block backref root FS_TREE
failed to find block number 13576823652352 kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON 1 triggered, value 1 ./btrfs(+0x297a4)[0x56509942c7a4]
./btrfs(+0x297f4)[0x56509942c7f4] ./btrfs(btrfs_set_block_flags+0x1fb)[0x56509942f623] ./btrfs(+0x8be0e)[0x56509948ee0e]
./btrfs(+0x8c22d)[0x56509948f22d]
./btrfs(+0x8c22d)[0x56509948f22d]
./btrfs(+0x8c49b)[0x56509948f49b] ./btrfs(btrfs_init_extent_tree+0xe03)[0x56509949037b] ./btrfs(+0x83b7d)[0x565099486b7d] ./btrfs(handle_command_group+0x49)[0x56509941b17b] ./btrfs(main+0x94)[0x56509941b275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7feced8517fd] ./btrfs(_start+0x2a)[0x56509941ae1a]
Aborted

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#163 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 50 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh IDK why that happens every once and a while. I pushed a fix for btrfs-corrupt-block, it should work now. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1
FS_INFO IS 0x562d920a7600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x562d920a7600
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
parent transid verify failed on 13576823635968 wanted 1619791 found 1619802
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1
FS_INFO IS 0x5649e173dbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5649e173dbc0
Walking all our trees and pinning down the currently accessible blocks
(..,)

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,134217728 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1

Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819135,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/ise_0007_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1

Quoted 5 lines
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1

Ok, now we're down to another one:

Ok I fixed the debugging to not be so noisy so I can see what's going on, lets give that a try,

Josef

#164 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Ok I fixed the debugging to not be so noisy so I can see what's going on, lets give that a try,

(...)
inserting block group 13570381185024
inserting block group 13571454926848
inserting block group 13572528668672
inserting block group 13573602410496
inserting block group 13574676152320
inserting block group 13575749894144
inserting block group 13576823635968
inserting block group 13577897377792
inserting block group 13585413570560
(...)
inserting block group 15768498470912
inserting block group 15769572212736
inserting block group 15770645954560
inserting block group 15771719696384
inserting block group 15778162147328
inserting block group 15779235889152
inserting block group 15780309630976
inserting block group 15781383372800
inserting block group 15782457114624
inserting block group 15783530856448
inserting block group 15784604598272
inserting block group 15785678340096
inserting block group 15786752081920
inserting block group 15787825823744
inserting block group 15788899565568
inserting block group 15789973307392
inserting block group 15791047049216
inserting block group 15792120791040
inserting block group 15793194532864
inserting block group 15794268274688
inserting block group 15795342016512
inserting block group 15796415758336
inserting block group 15797489500160
inserting block group 15798563241984
inserting block group 15799636983808
inserting block group 15800710725632
inserting block group 15801784467456
inserting block group 15802858209280
inserting block group 15803931951104
inserting block group 15809300660224
inserting block group 15810374402048
inserting block group 15811448143872
inserting block group 15812521885696
inserting block group 15813595627520
inserting block group 15814669369344
inserting block group 15815743111168
inserting block group 15816816852992
inserting block group 15817890594816
inserting block group 15818964336640
inserting block group 15820038078464
inserting block group 15821111820288
inserting block group 15822185562112
inserting block group 15823259303936
inserting block group 15824333045760
inserting block group 15825406787584
inserting block group 15826480529408
inserting block group 15827554271232
inserting block group 15828628013056
inserting block group 15829701754880
inserting block group 15830775496704
inserting block group 15831849238528
inserting block group 15832922980352
inserting block group 15833996722176
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
processed 1556480 of 0 possible bytes
processed 1474560 of 0 possible bytes
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes
Recording extents for root 5
processed 10960896 of 10977280 possible bytes
Recording extents for root 7
processed 16384 of 16545742848 possible bytes
Recording extents for root 9
processed 16384 of 16384 possible bytes
Recording extents for root 11221
processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure failed to find block number 13576823652352 kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON 1 triggered, value 1

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#165 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 88 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ok I fixed the debugging to not be so noisy so I can see what's going on, lets give that a try,

(...)
inserting block group 13570381185024 inserting block group 13571454926848 inserting block group 13572528668672 inserting block group 13573602410496 inserting block group 13574676152320 inserting block group 13575749894144 inserting block group 13576823635968 inserting block group 13577897377792 inserting block group 13585413570560 (...)
inserting block group 15768498470912 inserting block group 15769572212736 inserting block group 15770645954560 inserting block group 15771719696384 inserting block group 15778162147328 inserting block group 15779235889152 inserting block group 15780309630976 inserting block group 15781383372800 inserting block group 15782457114624 inserting block group 15783530856448 inserting block group 15784604598272 inserting block group 15785678340096 inserting block group 15786752081920 inserting block group 15787825823744 inserting block group 15788899565568 inserting block group 15789973307392 inserting block group 15791047049216 inserting block group 15792120791040 inserting block group 15793194532864 inserting block group 15794268274688 inserting block group 15795342016512 inserting block group 15796415758336 inserting block group 15797489500160 inserting block group 15798563241984 inserting block group 15799636983808 inserting block group 15800710725632 inserting block group 15801784467456 inserting block group 15802858209280 inserting block group 15803931951104 inserting block group 15809300660224 inserting block group 15810374402048 inserting block group 15811448143872 inserting block group 15812521885696 inserting block group 15813595627520 inserting block group 15814669369344 inserting block group 15815743111168 inserting block group 15816816852992 inserting block group 15817890594816 inserting block group 15818964336640 inserting block group 15820038078464 inserting block group 15821111820288 inserting block group 15822185562112 inserting block group 15823259303936 inserting block group 15824333045760 inserting block group 15825406787584 inserting block group 15826480529408 inserting block group 15827554271232 inserting block group 15828628013056 inserting block group 15829701754880 inserting block group 15830775496704 inserting block group 15831849238528 inserting block group 15832922980352 inserting block group 15833996722176 inserting block group 15835070464000 inserting block group 15836144205824 inserting block group 15837217947648 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes Recording extents for root 5
processed 10960896 of 10977280 possible bytes Recording extents for root 7
processed 16384 of 16545742848 possible bytes Recording extents for root 9
processed 16384 of 16384 possible bytes Recording extents for root 11221

Quoted 9 lines

processed 16384 of 255983616 possible bytes
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes
Ignoring transid failure
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure failed to find block number 13576823652352 kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON 1 triggered, value 1

Fucking hell, do the tree-recover again and then the init-extent-tree. Once I'm done with this conference I'll work out why this keeps happening. Thanks,

Josef

#166 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik
Quoted 4 lines

Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure failed to find block number 13576823652352 kernel-shared/extent-tree.c:1432: btrfs_set_block_flags: BUG_ON 1 triggered, value 1

Fucking hell, do the tree-recover again and then the init-extent-tree. Once I'm done with this conference I'll work out why this keeps happening. Thanks,

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x561ce6c0cbc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x561ce6c0cbc0
Checking root 2 bytenr 15645018603520
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Repairing root 11223 bad_blocks 1 update 1
Repairing root 163303 bad_blocks 1 update 0
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
updating slot 11 in block 15645018505216
deleting slot 11 in block 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645018161152
Repairing root 164624 bad_blocks 1 update 1
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
updating slot 24 in block 15645502210048
deleting slot 24 in block 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Repairing root 165098 bad_blocks 1 update 0
updating slot 425 in block 11821926875136
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511369756672
Repairing root 165198 bad_blocks 1 update 1
updating slot 425 in block 11822225473536
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13576823635968
Repairing root 165298 bad_blocks 0 update 1
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 29540352
Tree recovery finished, you can run check now
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55ec18f10bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55ec18f10bc0
Checking root 2 bytenr 15645018603520
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781245882368
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781245865984
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781245898752
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781245915136
Checking root 163303 bytenr 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 29540352
Tree recovery finished, you can run check now

Good, now running repair

gargamel:/var/local/src/btrfs-progs-josefbacik# .//btrfs check --repair /dev/mapper/dshelf1
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x5584836fafd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x5584836fafd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
Error: could not find extent items for root 11223
ERROR: failed to repair root items: No such file or directory

doing the longer one, will report back

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#167 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Marc MERLIN

doing the longer one, will report back

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1

Still running, it would be lovely to have a percentage of completion on this one too :)

Repaired extent references for 331055104
ref mismatch on [331071488 16384] extent item 0, found 1
tree backref 331071488 parent 353419264 not found in extent tree backpointer mismatch on [331071488 16384]
adding new tree backref on start 331071488 len 16384 parent 353419264 root 353419264 Repaired extent references for 331071488
ref mismatch on [331087872 16384] extent item 0, found 1
tree backref 331087872 parent 353419264 not found in extent tree backpointer mismatch on [331087872 16384]
adding new tree backref on start 331087872 len 16384 parent 353419264 root 353419264

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#168 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Marc MERLIN
Quoted 2 lines · Marc MERLIN

doing the longer one, will report back

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair --init-extent-tree /dev/mapper/dshelf1

Still running, it would be lovely to have a percentage of completion on this one too :)

Quoted 11 lines

Repaired extent references for 331055104
ref mismatch on [331071488 16384] extent item 0, found 1
tree backref 331071488 parent 353419264 not found in extent tree backpointer mismatch on [331071488 16384]
adding new tree backref on start 331071488 len 16384 parent 353419264 root 353419264 Repaired extent references for 331071488
ref mismatch on [331087872 16384] extent item 0, found 1
tree backref 331087872 parent 353419264 not found in extent tree backpointer mismatch on [331087872 16384]
adding new tree backref on start 331087872 len 16384 parent 353419264 root 353419264

Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do

btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>

There may be some things left to delete but we're just about there. Thanks,

Josef

#169 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 8 lines · Josef Bacik

Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do

btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>

Got it. Note that check --repair faile

./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55fbfe283bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fbfe283bc0
Checking root 2 bytenr 29442048
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781245882368
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781245865984
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781245898752
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781245915136
Checking root 163303 bytenr 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823717888
Tree recovery finished, you can run check now

The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819130,108,0 on root 11223
inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0027_2020.1_0602_1208.xz
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1 

Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819131,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0026_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819131,108,0" -r 11223 /dev/mapper/dshelf1 
FS_INFO IS 0x561c57fe8600
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x561c57fe8600
parent transid verify failed on 13576823635968 wanted 1619857 found 1638917
parent transid verify failed on 13576823635968 wanted 1619857 found 1638917
parent transid verify failed on 13576823635968 wanted 1619857 found 1638917

processed 1619902464 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819133,108,0 on root 11223 inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0
Xilinx_Unified_2020.1_0602_1208/payload/rdi_0410_2020.1_0602_1208.xz
cmds/rescue-init-extent-tree.c:654: process_eb: BUG_ON 1 triggered, value 1

Ok, so it's the same problem as last time. My corrupt blocks have been lost/uncorrupted. So I re-ran the same commands as last time:
./btrfs-corrupt-block -d "1819130,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819131,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819133,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819135,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819137,108,0" -r 11223 /dev/mapper/dshelf1

Ok, more still, new ones I didn't have to run last time, still all on this new file: rdi_0059_2020.1_0602_1208.xz
processed 1619902464 of 1635549184 possible bytesIgnoring transid failure
processed 1619918848 of 1635549184 possible bytesWe're tyring to add a data extent that we don't have a block group for, delete 1819139,108,0 on root 11223 inode ref info failed???

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819139,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819142,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819143,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819144,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819150,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819153,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819154,108,0" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819160,108,16318464" -r 11223 /dev/mapper/dshelf1
gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "1819161,108,0" -r 11223 /dev/mapper/dshelf1

ok, took hours and I ran out of time, more tomorrow
./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819161,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819164,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819165,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819167,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819168,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819169,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819169,108,134217728" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819170,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819175,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819176,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819177,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819179,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819183,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819184,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819187,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819189,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819189,108,134217728" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819191,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819193,108,0" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819193,108,24248320" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 ./btrfs-corrupt-block -d "1819193,108,55640064" -r 11223 /dev/mapper/dshelf1 ./btrfs rescue init-extent-tree /dev/mapper/dshelf1

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#170 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 58 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik

Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do

btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>

Got it. Note that check --repair faile

./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55fbfe283bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fbfe283bc0
Checking root 2 bytenr 29442048
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781245882368
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781245865984
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781245898752
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781245915136
Checking root 163303 bytenr 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823717888
Tree recovery finished, you can run check now
Quoted 1 line

The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)

Agreed, I'm going to wire something up now. Thanks,

Josef

#171 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 60 lines · Josef Bacik
Quoted 58 lines · Marc MERLIN
Quoted 8 lines · Josef Bacik

Sorry Marc I was busy with the conference and completely misread what you did. Cancel the btrfs check now, and then do

btrfs rescue tree-recover <device> // This should succeed without doing anything but just in case
btrfs rescue init-extent-tree <device> // I'm hoping this will succeed this time, if not of course tell me
btrfs check --repair <device>

Got it. Note that check --repair faile

./gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1
FS_INFO IS 0x55fbfe283bc0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x55fbfe283bc0
Checking root 2 bytenr 29442048
Checking root 4 bytenr 15645196861440
Checking root 5 bytenr 13577660252160
Checking root 7 bytenr 15645018521600
Checking root 9 bytenr 15645878108160
Checking root 11221 bytenr 13577562996736
Checking root 11222 bytenr 15645261905920
Checking root 11223 bytenr 13576823635968
Checking root 11224 bytenr 13577126182912
Checking root 159785 bytenr 6781490577408
Checking root 159787 bytenr 15645908385792
Checking root 160494 bytenr 6781245882368
Checking root 160496 bytenr 11822309965824
Checking root 161197 bytenr 6781245865984
Checking root 161199 bytenr 13576850833408
Checking root 162628 bytenr 15645764812800
Checking root 162632 bytenr 6781245898752
Checking root 162645 bytenr 5809981095936
Checking root 163298 bytenr 15645124263936
Checking root 163302 bytenr 6781245915136
Checking root 163303 bytenr 15645018505216
Checking root 163316 bytenr 6781245931520
Checking root 163318 bytenr 15645980491776
Checking root 163916 bytenr 11822437826560
Checking root 163920 bytenr 11970640084992
Checking root 163921 bytenr 11971073802240
Checking root 164620 bytenr 15645434036224
Checking root 164624 bytenr 15645502210048
Checking root 164633 bytenr 15645526884352
Checking root 165098 bytenr 11970640101376
Checking root 165100 bytenr 11970733621248
Checking root 165198 bytenr 12511656394752
Checking root 165200 bytenr 12511677972480
Checking root 165294 bytenr 13576901328896
Checking root 165298 bytenr 13577133326336
Checking root 165299 bytenr 13577191505920
Checking root 18446744073709551607 bytenr 13576823717888
Tree recovery finished, you can run check now
Quoted 4 lines
Quoted 1 line

The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)

Agreed, I'm going to wire something up now. Thanks,

Ok I pushed something, hopefully it works and you don't have to touch it anymore? Thanks,

Josef

#172 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Josef Bacik
Quoted 4 lines
Quoted 1 line

The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)

Agreed, I'm going to wire something up now. Thanks,

Ok I pushed something, hopefully it works and you don't have to touch it anymore? Thanks,

Thanks. That seems to almost work and I like the progress percentage, thanks :)

But now it seems to loop on the same one? I got over 50 of those

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting

searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99% processed 1620017152 of 1635549184 possible bytes, 99% Found an extent we don't have a block group for in the file inode ref info failed???
elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/payload/rdi_0422_2020.1_0602_1208.xz Deleting

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#173 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Marc MERLIN
Quoted 5 lines · Josef Bacik
Quoted 4 lines
Quoted 1 line

The delete block bit should be automated somehow, it's quite slow and painful to do by hand (hours again here)

Agreed, I'm going to wire something up now. Thanks,

Quoted 8 lines
Quoted 2 lines

Ok I pushed something, hopefully it works and you don't have to touch it anymore? Thanks,

Thanks. That seems to almost work and I like the progress percentage, thanks :)

But now it seems to loop on the same one? I got over 50 of those

Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,

Josef

#174 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,

Ok, the debugging helps, now I can see that it's deleting different blocks for the same filename.
Before I couldn't quite tell if it was making progress, but now I see it's deleting new locations.
Looks like I have 100s of deletions coming up, thanks for automating this.

Will report back, it looks like it may run for a few hours

Afer that just do a
check --repair
or some other command options?

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#175 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,

Ok, the debugging helps, now I can see that it's deleting different blocks for the same filename.
Before I couldn't quite tell if it was making progress, but now I see it's deleting new locations.
Looks like I have 100s of deletions coming up, thanks for automating this.

Quoted 5 lines

Will report back, it looks like it may run for a few hours

Afer that just do a
check --repair
or some other command options?

Ok, so I got 33GB of output, so glad the tool is automated and the tool died eventually.
I'll re-run under gdb, at least it did a lot of work.

searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 111673344] root 13576824389632 path top 13576824389632 top slot 24 leaf 11821927333888 slot 52

searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 228950016] root 13576824406016 path top 13576824406016 top slot 24 leaf 11821927350272 slot 52

searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception

I'll re-run and hopefully it will finish without dying or I'll get a backtrace.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#176 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 39 lines · Marc MERLIN
Quoted 13 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Did any of the previous ones succeed? I hope so and we just have one misbehaving thing. I've pushed more debugging, maybe it's a large file that has a lot of broken extents, in either case it'll tell us what's going on so I can narrow down the problem. Thanks,

Ok, the debugging helps, now I can see that it's deleting different blocks for the same filename.
Before I couldn't quite tell if it was making progress, but now I see it's deleting new locations.
Looks like I have 100s of deletions coming up, thanks for automating this.

Quoted 5 lines

Will report back, it looks like it may run for a few hours

Afer that just do a
check --repair
or some other command options?

Ok, so I got 33GB of output, so glad the tool is automated and the tool died eventually.
I'll re-run under gdb, at least it did a lot of work.

searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 111673344] root 13576824389632 path top 13576824389632 top slot 24 leaf 11821927333888 slot 52

searching 165298 for bad extents
processed 108707840 of 108756992 possible bytes, 99%
Found an extent we don't have a block group for in the file
inode ref info failed???
History/Clubbing/20220318_Pure_Markus_Schulz.mp4
Deleting [76600, 108, 228950016] root 13576824406016 path top 13576824406016 top slot 24 leaf 11821927350272 slot 52

Quoted 9 lines

searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception

Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,

Josef

#177 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Josef Bacik
Quoted 8 lines

searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception

Quoted 3 lines

Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,

Mmmh, got worse? Failed on the first file to delete
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 inode ref info failed???
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48

#1 0x000055555559b1a5 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645169139712)
at common/extent-cache.c:179
#2 0x0000555555584d33 in set_extent_bits (tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f66 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x0000555555585949 in set_extent_buffer_dirty (eb=eb@entry=0x55555ebf5bd0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc36 in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555ebf5bd0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef58 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825108188384, parent=0,
    iref=0xffffffffffffffe3, path=0x55555c3e9ce0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825108188384, parent=0,
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555c3e9ce0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, bytenr=<optimized out>,
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e05e5 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=eb@entry=0x5555632213b0,
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:869
#10 0x00005555555e0750 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=eb@entry=0x55556321d330,
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:944
#11 0x00005555555e0750 in process_eb (trans=trans@entry=0x55555ca62310, root=root@entry=0x555555c50fa0, eb=0x555556497960,
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:944

#12 0x00005555555e0a00 in record_root (root=0x555555c50fa0) at cmds/rescue-init-extent-tree.c:1020 #13 0x00005555555dfebf in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e08fb <record_root>)
at cmds/rescue-init-extent-tree.c:87
#14 0x00005555555e0d5b in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")

    at cmds/rescue-init-extent-tree.c:1099
#15 0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#178 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 26 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 8 lines

searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception

Quoted 3 lines

Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,

Mmmh, got worse? Failed on the first file to delete
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 inode ref info failed???
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
Quoted 4 lines

#1 0x000055555559b1a5 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645169139712)
at common/extent-cache.c:179
#2 0x0000555555584d33 in set_extent_bits (tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
at kernel-shared/extent_io.c:380

I tried the old code, and here is the output

processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3

Program received signal SIGFPE, Arithmetic exception.
0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,

    current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
112       return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
(gdb) bt
#0  0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10, 
    current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
#1  0x00005555555e0745 in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=0x555555650b90, 
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:941
#2  0x00005555555e09ea in record_root (root=0x55555564e030) at cmds/rescue-init-extent-tree.c:1014
#3  0x00005555555e0d11 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")
    at cmds/rescue-init-extent-tree.c:1086
#4  0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#179 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 53 lines · Marc MERLIN
Quoted 26 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 8 lines

searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3
Floating point exception

Quoted 3 lines

Oops that's probably a divide by 0 for my fancy pct thing, I fixed it up and pushed. Thanks,

Mmmh, got worse? Failed on the first file to delete
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 inode ref info failed???
misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
48              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559afa1 <cache_tree_comp_range>,
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:48
Quoted 4 lines

#1 0x000055555559b1a5 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645169139712)
at common/extent-cache.c:179
#2 0x0000555555584d33 in set_extent_bits (tree=0x100000060, start=15645169139712, end=15645169139711, bits=bits@entry=1)
at kernel-shared/extent_io.c:380

I tried the old code, and here is the output

processed 108756992 of 108756992 possible bytes, 100% searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100% searching 18446744073709551607 for bad extents processed 16384 of 16384 possible bytes, 100% Recording extents for root 3

Program received signal SIGFPE, Arithmetic exception.
0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,

    current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
112       return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
(gdb) bt
#0  0x00005555555e08ad in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=eb@entry=0x555555654c10,
    current=current@entry=0x7fffffffdae8) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:112
#1  0x00005555555e0745 in process_eb (trans=trans@entry=0x55555930f1a0, root=root@entry=0x55555564e030, eb=0x555555650b90,
    current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:941

#2 0x00005555555e09ea in record_root (root=0x55555564e030) at cmds/rescue-init-extent-tree.c:1014 #3 0x00005555555e0d11 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")

Quoted 9 lines
    at cmds/rescue-init-extent-tree.c:1086
#4  0x00005555555d7b65 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#5  0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125
#6  handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#7  0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125
#8  main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,

Josef

#180 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 4 lines · Josef Bacik

Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,

inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%WTF IT DIDN'T DELETE IT!! processed 19365888 of 49479680 possible bytes, 39%WTF IT DIDN'T DELETE IT!! processed 25067520 of 49479680 possible bytes, 50%WTF IT DIDN'T DELETE IT!! processed 42336256 of 49479680 possible bytes, 85%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 254902272 of 1635549184 possible bytes, 15%WTF IT DIDN'T DELETE IT!! processed 311050240 of 1635549184 possible bytes, 19%WTF IT DIDN'T DELETE IT!! processed 340901888 of 1635549184 possible bytes, 20%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358498304 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358514688 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358563840 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358662144 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358711296 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! processed 358727680 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358744064 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358776832 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! processed 358793216 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 358842368 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358875136 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
processed 358891520 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! processed 358907904 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!! WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
WTF IT DIDN'T DELETE IT!!
processed 359022592 of 1635549184 possible bytes, 21%WTF IT DIDN'T DELETE IT!!

many many lines from the beginning, that I wasn't getting before, so I stopped it

Keep going?

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#181 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,

Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,

Josef

#182 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 9 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,

Quoted 4 lines

Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,

Thanks for the heads up

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#183 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 11 lines · Marc MERLIN
Quoted 9 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,

Quoted 7 lines
Quoted 4 lines

Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,

Thanks for the heads up

Ok I've pushed some code, but I'm sitting in a dealership so testing was light. I've added a 10 second pause before doing deletions from the new code to give you time to spot check if the numbers look insane. It'll only do that 5 times, so if everything looks good it'll just yolo delete stuff as it goes after 5 pauses. Let me know if you have trouble, thanks,

Josef

#184 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 17 lines · Josef Bacik
Quoted 16 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,

Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,

Quoted 8 lines
Quoted 1 line

Thanks for the heads up

Ok I've pushed some code, but I'm sitting in a dealership so testing was light. I've added a 10 second pause before doing deletions from the new code to give you time to spot check if the numbers look insane. It'll only do that 5 times, so if everything looks good it'll just yolo delete stuff as it goes after 5 pauses. Let me know if you have trouble, thanks,

Thanks.
It worked for a while, and then failed in a new way?

inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032674304 123032702976] current [123032702976 123032731648]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anythin g1
2
0
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 151552] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 92

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032731648 123032760320] current [123032760320 123032788992]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 413696] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 93

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128877895680 128877924352] current [128877924352 128877948928]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 675840] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 94

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128877948928 128877977600] current [128877977600 128878002176]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 937984] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 95

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878002176 128878030848] current [128878030848 128878055424]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1200128] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 96

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878055424 128878080000] current [128878080000 128878108672] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1462272] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 97

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706290176 147706318848] current [147706318848 147706343424] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 1855488] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 99

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706343424 147706372096] current [147706372096 147706400768] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2125824] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 101

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706400768 147706429440] current [147706429440 147706458112] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2387968] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 102

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [128878108672 128878133248] current [128878133248 128878157824] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2650112] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 103

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [147706458112 147706482688] current [147706482688 147706507264] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 2752512] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 103

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839411200 193839439872] current [193839439872 193839468544] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3145728] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 105

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839468544 193839497216] current [193839497216 193839529984] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3407872] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 106

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839529984 193839570944] current [193839570944 193839611904] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3670016] root 13576823898112 path top 13576823898112 top slot 0 leaf 13576824553472 slot 107

searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [193839611904 193839656960] current [193839656960 193839665152] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 3932160] root 13576823717888 path top 13576823717888 top slot 0 leaf 13576824340480 slot 108

(...)

searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [109405609984 109405650944] current [109405650944 109405659136] The newly found extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 16121856] root 13576823717888 path top 13576823717888 top slot 0 leaf 11970640084992 slot 25

searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [153036664832 153036668928] current [153036668928 153036689408] The original extent is older, deleting it
inode ref info failed???
DS1
Deleting [1003, 108, 15069184] root 13576823898112 path top 13576823898112 top slot 0 leaf 11970640101376 slot 20

searching 5 for bad extents
processed 32768 of 10977280 possible bytes, 0%
Found an overlapping extent orig [153036664832 153036668928] current [153036668928 153036689408] The original extent is older, deleting it
inode ref info failed???
DS1
ERROR: error searching for key?? -2

wtf
it failed?? -2
ERROR: failed to clear bad extents
doing close???
ERROR: attempt to start transaction over already running one
WARNING: reserved space leaked, flag=0x4 bytes_reserved=49152 extent buffer leak: start 13576823717888 len 16384
extent buffer leak: start 13576823717888 len 16384
WARNING: dirty eb leak (aborted trans): start 13576823717888 len 16384 extent buffer leak: start 13576823750656 len 16384
extent buffer leak: start 13576823750656 len 16384
WARNING: dirty eb leak (aborted trans): start 13576823750656 len 16384 extent buffer leak: start 11970640084992 len 16384
extent buffer leak: start 11970640084992 len 16384
WARNING: dirty eb leak (aborted trans): start 11970640084992 len 16384 Init extent tree failed
[Inferior 1 (process 19271) exited with code 0376]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#185 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 31 lines · Marc MERLIN
Quoted 17 lines · Josef Bacik
Quoted 16 lines · Marc MERLIN
Quoted 13 lines · Josef Bacik
Quoted 7 lines · Marc MERLIN
Quoted 4 lines · Josef Bacik

Yeah this is the divide by 0, the error you posted earlier is likely because of the code refactor I did to make the delete thing work. I've added some more debugging to see if we're not deleting this problem bytenr during the search for bad extents. Thanks,

Ooooh right this is the other problem, overlapping extents. This is going to be trickier to work out, I'll start writing it up, but I want to make it automatic as well, so probably won't have anything until the morning. Thanks,

Quoted 8 lines
Quoted 1 line

Thanks for the heads up

Ok I've pushed some code, but I'm sitting in a dealership so testing was light. I've added a 10 second pause before doing deletions from the new code to give you time to spot check if the numbers look insane. It'll only do that 5 times, so if everything looks good it'll just yolo delete stuff as it goes after 5 pauses. Let me know if you have trouble, thanks,

Thanks.
It worked for a while, and then failed in a new way?

Quoted 11 lines

inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 16384 of 10977280 possible bytes, 0%
Found an overlapping extent orig [123032674304 123032702976] current [123032702976 123032731648]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Josef

#186 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Looks better this time. Went pretty far, until here:
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55              struct rb_node *n = root->rb_node;
(gdb) bt 
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55

#1 0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645171073024)
at common/extent-cache.c:179
#2 0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645171073024, end=15645171073023, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556ce60d60) at kernel-shared/extent_io.c:976
#5  0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556ce60d60) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93824992431962, parent=0, 
    iref=0xffffffffffffffe3, path=0x5555703388f0, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93824992431962, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x5555703388f0, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e086b in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=eb@entry=0x555571271c40, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:986
#10 0x00005555555e09d6 in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=eb@entry=0x55557126dbc0, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#11 0x00005555555e09d6 in process_eb (trans=trans@entry=0x5555685c9d90, root=root@entry=0x5555563d6bf0, eb=0x55555649b860, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061

#12 0x00005555555e0c82 in record_root (root=0x5555563d6bf0) at cmds/rescue-init-extent-tree.c:1137 #13 0x00005555555dfd9f in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b7d <record_root>)
at cmds/rescue-init-extent-tree.c:151
#14 0x00005555555e0ff4 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")

    at cmds/rescue-init-extent-tree.c:1216
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#187 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Josef

#188 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#189 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Quoted 7 lines

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Sorry, long day, try it again. Thanks,

Josef

#190 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Quoted 8 lines
Quoted 6 lines

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Sorry, long day, try it again. Thanks,

processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55

#1 0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645363896320)
at common/extent-cache.c:179
#2 0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645363896320, end=15645363896319, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556fe993b0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556fe993b0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825336391296, parent=0, 
    iref=0xffffffffffffffe3, path=0x555569d8b680, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825336391296, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555569d8b680, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e087d in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=eb@entry=0x55557411e240, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:986
#10 0x00005555555e09e8 in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=eb@entry=0x55557411a1c0, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061
#11 0x00005555555e09e8 in process_eb (trans=trans@entry=0x55557058a2f0, root=root@entry=0x555559154170, eb=0x555559218660, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1061

#12 0x00005555555e0c98 in record_root (root=0x555559154170) at cmds/rescue-init-extent-tree.c:1137 #13 0x00005555555dfda6 in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b93 <record_root>)
at cmds/rescue-init-extent-tree.c:151
#14 0x00005555555e100a in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")

    at cmds/rescue-init-extent-tree.c:1216
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#191 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 25 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Quoted 8 lines
Quoted 6 lines

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Sorry, long day, try it again. Thanks,

Quoted 9 lines

processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Ugh such a pain, lets try this again,

Josef

#192 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 36 lines · Josef Bacik
Quoted 25 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Quoted 8 lines
Quoted 6 lines

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Sorry, long day, try it again. Thanks,

Quoted 9 lines

processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Quoted 1 line

Ugh such a pain, lets try this again,

Looks the same?

gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -6
commit 8e14dcb48ad9fd60c595821fc4ebb5c6a1cfb13f
Author: Josef Bacik <[email protected]>
Date:   Mon May 9 22:02:12 2022 -0400

add bytes

processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
Failed to find [10467695652864, 168, 8675328]

Program received signal SIGSEGV, Segmentation fault.
rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>,

    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55
55              struct rb_node *n = root->rb_node;
(gdb) bt
#0  rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd540, comp=comp@entry=0x55555559af95 <cache_tree_comp_range>, 
    next_ret=next_ret@entry=0x7fffffffd558) at common/rbtree-utils.c:55

#1 0x000055555559b199 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645181853696)
at common/extent-cache.c:179
#2 0x0000555555584d27 in set_extent_bits (tree=0x100000060, start=15645181853696, end=15645181853695, bits=bits@entry=1)
at kernel-shared/extent_io.c:380
#3 0x0000555555584f5a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>)
at kernel-shared/extent_io.c:486

#4  0x000055555558593d in set_extent_buffer_dirty (eb=eb@entry=0x55556fbb6da0) at kernel-shared/extent_io.c:976
#5  0x000055555557bc2a in btrfs_mark_buffer_dirty (eb=eb@entry=0x55556fbb6da0) at kernel-shared/disk-io.c:2224
#6  0x000055555557ef4c in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825437307456, parent=0, 
    iref=0xffffffffffffffe3, path=0x55556fdc9240, root=<optimized out>) at kernel-shared/extent-tree.c:1085
#7  insert_inline_extent_backref (refs_to_add=1, offset=0, owner=76300, root_objectid=93825437307456, parent=0, 
    num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55556fdc9240, root=<optimized out>, trans=<optimized out>)
    at kernel-shared/extent-tree.c:1197
#8  btrfs_inc_extent_ref (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, bytenr=<optimized out>, 
    num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=165198, owner=76300, offset=0)
    at kernel-shared/extent-tree.c:1262
#9  0x00005555555e087a in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=eb@entry=0x5555741a4ff0, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:987
#10 0x00005555555e09e5 in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=eb@entry=0x5555741a0f70, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1062
#11 0x00005555555e09e5 in process_eb (trans=trans@entry=0x555570857380, root=root@entry=0x55555915f5d0, eb=0x555559223b80, 
    current=current@entry=0x7fffffffd9e8) at cmds/rescue-init-extent-tree.c:1062

#12 0x00005555555e0c95 in record_root (root=0x55555915f5d0) at cmds/rescue-init-extent-tree.c:1138 #13 0x00005555555dfda6 in foreach_root (fs_info=fs_info@entry=0x55555564dbc0, cb=cb@entry=0x5555555e0b90 <record_root>)
at cmds/rescue-init-extent-tree.c:152
#14 0x00005555555e1007 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1ce "/dev/mapper/dshelf1")

    at cmds/rescue-init-extent-tree.c:1217
#15 0x00005555555d7b70 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>)
    at cmds/rescue.c:65
#16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555643d40 <cmd_struct_rescue_init_extent_tree>)
    at cmds/commands.h:125

#17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152
#18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555644cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#193 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 38 lines · Marc MERLIN
Quoted 36 lines · Josef Bacik
Quoted 34 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Quoted 8 lines
Quoted 6 lines

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Sorry, long day, try it again. Thanks,

processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Quoted 5 lines
Quoted 1 line

Ugh such a pain, lets try this again,

Looks the same?

There's no other debug printing before this? Can I get the full output from the run? If there isn't something really wonky is going on and I'm confused. Thanks,

Josef

#194 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 39 lines · Josef Bacik
Quoted 38 lines · Marc MERLIN
Quoted 36 lines · Josef Bacik
Quoted 34 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Quoted 8 lines
Quoted 6 lines

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Sorry, long day, try it again. Thanks,

processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Ugh such a pain, lets try this again,

Looks the same?

There's no other debug printing before this? Can I get the full output from the run? If there isn't something really wonky is going on and I'm confused. Thanks,

looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode

fuller output:
inserting block group 15831849238528
inserting block group 15832922980352
inserting block group 15833996722176
inserting block group 15835070464000
inserting block group 15836144205824
inserting block group 15837217947648
inserting block group 15838291689472
inserting block group 15839365431296
inserting block group 15840439173120
inserting block group 15842586656768
searching 4 for bad extents
processed 1032192 of 1064960 possible bytes, 96%
searching 5 for bad extents
processed 10960896 of 10977280 possible bytes, 99%
searching 7 for bad extents
processed 16384 of 16545742848 possible bytes, 0%
searching 9 for bad extents
processed 16384 of 16384 possible bytes, 100%
searching 11221 for bad extents
processed 16384 of 255983616 possible bytes, 0%
searching 11222 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 11223 for bad extents
prIgnoring transid failure35549184 possible bytes, 99%
processed 1635319808 of 1635549184 possible bytes, 99%
searching 11224 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 159785 for bad extents
processed 108429312 of 108429312 possible bytes, 100%
searching 159787 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 160494 for bad extents
processed 108560384 of 108560384 possible bytes, 100%
searching 160496 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 161197 for bad extents
processed 108544000 of 108544000 possible bytes, 100%
searching 161199 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 162628 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 162632 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 162645 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 163298 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163302 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163303 for bad extents
processed 71745536 of 75792384 possible bytes, 94%
searching 163316 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163318 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163916 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 163920 for bad extents
processed 108691456 of 108691456 possible bytes, 100%
searching 163921 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165200 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165294 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165298 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165299 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 18446744073709551607 for bad extents
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 3
processed 1556480 of 0 possible bytes, 0%
Recording extents for root 1
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode
doing an insert that overlaps our bytenr 10467701948416 262144
processed 1474560 of 0 possible bytes, 0%
doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467 695652864 8675328
misc/file
Failed to find [10467695652864, 168, 8675328]
Program received signal SIGSEGV, Segmentation fault.

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#195 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 46 lines · Marc MERLIN
Quoted 44 lines · Josef Bacik
Quoted 42 lines · Marc MERLIN
Quoted 36 lines · Josef Bacik
Quoted 25 lines · Marc MERLIN
Quoted 14 lines · Josef Bacik
Quoted 13 lines · Marc MERLIN
Quoted 10 lines · Josef Bacik
Quoted 5 lines · Marc MERLIN
Quoted 2 lines · Josef Bacik

Ugh shit, I had an off by one error, that's not great. I've fixed that up and adjusted the debugging, lets see how that goes. Thanks,

Sorry my laptop battery died while I was at the dealership, and of course that took allllll day. Anyway pushed some debugging, am confused, hopefully won't be confused long. Thanks,

Quoted 8 lines
Quoted 6 lines

Sorry :-/
Yeah, I bring my power supply in such cases :)

Did you upload?

sauron:/var/local/src/btrfs-progs-josefbacik# git pull
Already up to date.

Sorry, long day, try it again. Thanks,

Quoted 9 lines

processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/add0/new/file
Failed to find [10467695652864, 168, 8675328]

Quoted 1 line

Ugh such a pain, lets try this again,

Looks the same?

Quoted 8 lines
Quoted 3 lines

There's no other debug printing before this? Can I get the full output from the run? If there isn't something really wonky is going on and I'm confused. Thanks,

looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode

Yup that was it, now it makes sense, I've fixed it hopefully. Thanks,

Josef

#196 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Josef Bacik
Quoted 4 lines

looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode

Yup that was it, now it makes sense, I've fixed it hopefully. Thanks,

processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
Deleting [4075, 108, 0] root 15645018177536 path top 15645018177536 top slot 3 leaf 6781245898752 slot 66

searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
ERROR: error searching for key?? -2

wtf
it failed?? -2
ERROR: failed to clear bad extents
doing close???
ERROR: commit_root already set when starting transaction
WARNING: reserved space leaked, flag=0x4 bytes_reserved=32768 extent buffer leak: start 15645018177536 len 16384
extent buffer leak: start 15645018783744 len 16384
extent buffer leak: start 15645018783744 len 16384
WARNING: dirty eb leak (aborted trans): start 15645018783744 len 16384 extent buffer leak: start 6781245915136 len 16384
extent buffer leak: start 6781245915136 len 16384
WARNING: dirty eb leak (aborted trans): start 6781245915136 len 16384 Init extent tree failed
[Inferior 1 (process 2234) exited with code 0376]

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#197 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 37 lines · Marc MERLIN
Quoted 6 lines · Josef Bacik
Quoted 4 lines

looking for this?
processed 49152 of 0 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [4075, 108, 0] Couldn't find any paths for this inode

Yup that was it, now it makes sense, I've fixed it hopefully. Thanks,

processed 75792384 of 75792384 possible bytes, 100%
searching 164620 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 164624 for bad extents
processed 102318080 of 109445120 possible bytes, 93%
searching 164633 for bad extents
processed 75792384 of 75792384 possible bytes, 100%
searching 165098 for bad extents
processed 108756992 of 108756992 possible bytes, 100%
searching 165100 for bad extents
processed 49479680 of 49479680 possible bytes, 100%
searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1 2
3
4
5
6
7
8
9
10
The original extent is older, deleting it
Couldn't find any paths for this inode
Deleting [4075, 108, 0] root 15645018177536 path top 15645018177536 top slot 3 leaf 6781245898752 slot 66

Quoted 4 lines

searching 165198 for bad extents
processed 108249088 of 108756992 possible bytes, 99%
Found an overlapping extent orig [10467701948416 10467702210560] current [10467695652864 10467704328192]
I'm going to give you 10 seconds to bail if that doesn't look right, I'll only ask 5 times before I just assume I didn't break anything1

Ah duh, we delete it the first time, but then we find another overlapping extent and we try to delete it again and it's not there. I've fixed this up, try again please. Thanks,

Josef

#198 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 3 lines · Josef Bacik

Ah duh, we delete it the first time, but then we find another overlapping extent and we try to delete it again and it's not there. I've fixed this up, try again please. Thanks,

good news, it worked! :)

I started a simple ./btrfs check --repair /dev/mapper/dshelf1 did you want me to add options to it or run as is?

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0] misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
processed 1015808 of 108756992 possible bytes, 0%
doing block accounting
doing close???
Init extent tree finished, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. Eg.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x56119a61efd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56119a61efd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
(...)
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 385889992704 parent 11970901524480 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385889992704 parent 11970901524480 owner 0 offset 0 found 1 wanted 0 back 0x5611a97a3110 incorrect local backref count on 385889992704 root 11222 owner 54869 offset 0 found 0 wanted 1 back 0x5611a6e42180 backref disk bytenr does not match extent record, bytenr=385889992704, ref bytenr=0
backpointer mismatch on [385889992704 4096]
repair deleting extent record: key [385889992704,168,4096]
adding new data backref on 385889992704 parent 11970901524480 owner 0 offset 0 found 1
Repaired extent references for 385889992704
data backref 385900392448 parent 1429184937984 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385900392448 parent 1429184937984 owner 0 offset 0 found 1 wanted 0 back 0x5611a659e210 incorrect local backref count on 385900392448 root 11222 owner 55229 offset 524288 found 0 wanted 1 back 0x5611cd078e60 backref disk bytenr does not match extent record, bytenr=385900392448, ref bytenr=0
backpointer mismatch on [385900392448 4096]
repair deleting extent record: key [385900392448,168,4096]
adding new data backref on 385900392448 parent 1429184937984 owner 0 offset 0 found 1
Repaired extent references for 385900392448
data backref 385900478464 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385900478464 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e720 incorrect local backref count on 385900478464 root 11222 owner 55240 offset 0 found 0 wanted 1 back 0x5611cd079880 backref disk bytenr does not match extent record, bytenr=385900478464, ref bytenr=0
backpointer mismatch on [385900478464 4096]
repair deleting extent record: key [385900478464,168,4096]
adding new data backref on 385900478464 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385900478464
data backref 385907994624 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385907994624 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6ab0 incorrect local backref count on 385907994624 root 11222 owner 55204 offset 131072 found 0 wanted 1 back 0x5611cd0d05c0 backref disk bytenr does not match extent record, bytenr=385907994624, ref bytenr=0
backpointer mismatch on [385907994624 4096]
repair deleting extent record: key [385907994624,168,4096]
adding new data backref on 385907994624 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385907994624
data backref 385908449280 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385908449280 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3690 incorrect local backref count on 385908449280 root 11222 owner 55063 offset 524288 found 0 wanted 1 back 0x5611cd0d21d0 backref disk bytenr does not match extent record, bytenr=385908449280, ref bytenr=0
backpointer mismatch on [385908449280 8192]
repair deleting extent record: key [385908449280,168,8192]
adding new data backref on 385908449280 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908449280
data backref 385908580352 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385908580352 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a36f0 incorrect local backref count on 385908580352 root 11222 owner 55063 offset 655360 found 0 wanted 1 back 0x5611cd0d2db0 backref disk bytenr does not match extent record, bytenr=385908580352, ref bytenr=0
backpointer mismatch on [385908580352 8192]
repair deleting extent record: key [385908580352,168,8192]
adding new data backref on 385908580352 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908580352
data backref 385909239808 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385909239808 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6b10 incorrect local backref count on 385909239808 root 11222 owner 55204 offset 262144 found 0 wanted 1 back 0x5611cd0c32f0 backref disk bytenr does not match extent record, bytenr=385909239808, ref bytenr=0
backpointer mismatch on [385909239808 4096]
repair deleting extent record: key [385909239808,168,4096]
adding new data backref on 385909239808 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385909239808
data backref 385913561088 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385913561088 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3750 incorrect local backref count on 385913561088 root 11222 owner 55063 offset 786432 found 0 wanted 1 back 0x5611cd1196b0 backref disk bytenr does not match extent record, bytenr=385913561088, ref bytenr=0
backpointer mismatch on [385913561088 8192]
repair deleting extent record: key [385913561088,168,8192]
adding new data backref on 385913561088 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385913561088
data backref 385913696256 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385913696256 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e780 incorrect local backref count on 385913696256 root 11222 owner 55240 offset 131072 found 0 wanted 1 back 0x5611cd11a3d0 backref disk bytenr does not match extent record, bytenr=385913696256, ref bytenr=0
backpointer mismatch on [385913696256 4096]
repair deleting extent record: key [385913696256,168,4096]
adding new data backref on 385913696256 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385913696256
data backref 385914368000 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385914368000 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6a50 incorrect local backref count on 385914368000 root 11222 owner 55204 offset 0 found 0 wanted 1 back 0x5611cd11d660 backref disk bytenr does not match extent record, bytenr=385914368000, ref bytenr=0
backpointer mismatch on [385914368000 8192]
repair deleting extent record: key [385914368000,168,8192]
adding new data backref on 385914368000 parent 13577102983168 owner 0 offset 0 found 1

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#199 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

I have
./btrfs check --repair /dev/mapper/dshelf1
on hold (^Z), waiting for your ok to proceed.

All good to continue?

Quoted 225 lines · Marc MERLIN
Quoted 3 lines · Josef Bacik

Ah duh, we delete it the first time, but then we find another overlapping extent and we try to delete it again and it's not there. I've fixed this up, try again please. Thanks,

good news, it worked! :)

I started a simple ./btrfs check --repair /dev/mapper/dshelf1 did you want me to add options to it or run as is?

doing roots
Recording extents for root 4
processed 1032192 of 1064960 possible bytes, 96%
Recording extents for root 5
processed 10960896 of 10977280 possible bytes, 99%
Recording extents for root 7
processed 16384 of 16545742848 possible bytes, 0%
Recording extents for root 9
processed 16384 of 16384 possible bytes, 100%
Recording extents for root 11221
processed 16384 of 255983616 possible bytes, 0%
Recording extents for root 11222
processed 49479680 of 49479680 possible bytes, 100%
Recording extents for root 11223
processed 1619902464 of 1635549184 possible bytes, 99%Ignoring transid failure
processed 1635319808 of 1635549184 possible bytes, 99%
Recording extents for root 11224
processed 75792384 of 75792384 possible bytes, 100%
Recording extents for root 159785
processed 108429312 of 108429312 possible bytes, 100%
Recording extents for root 159787
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 160494
processed 1425408 of 108560384 possible bytes, 1%
Recording extents for root 160496
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 161197
processed 770048 of 108544000 possible bytes, 0%
Recording extents for root 161199
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162628
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 162632
processed 2441216 of 108691456 possible bytes, 2%
Recording extents for root 162645
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163298
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163302
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163303
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 163316
processed 933888 of 108691456 possible bytes, 0%
Recording extents for root 163318
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 163916
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 163920
processed 966656 of 108691456 possible bytes, 0%
Recording extents for root 163921
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 164620
processed 49152 of 49479680 possible bytes, 0%
Recording extents for root 164624
processed 98304 of 109445120 possible bytes, 0%
Recording extents for root 164633
processed 49152 of 75792384 possible bytes, 0%
Recording extents for root 165098
processed 1015808 of 108756992 possible bytes, 0%
Recording extents for root 165100
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165198
processed 491520 of 108756992 possible bytes, 0%adding a bytenr that overlaps our thing, dumping paths for [76300, 108, 0] misc/file
doing an insert of the bytenr
doing an insert that overlaps our bytenr 10467695652864 8675328
processed 983040 of 108756992 possible bytes, 0%
Recording extents for root 165200
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165294
processed 16384 of 49479680 possible bytes, 0%
Recording extents for root 165298
processed 524288 of 108756992 possible bytes, 0%WTF???? we think we already inserted this bytenr?? [76300, 108, 0] dumping paths 10467695652864 8675328 misc/file
processed 1015808 of 108756992 possible bytes, 0%
doing block accounting
doing close???
Init extent tree finished, you can run check now

gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1
enabling repair mode
WARNING:

Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. Eg.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
FS_INFO IS 0x56119a61efd0
JOSEF: root 9
Couldn't find the last root for 8
FS_INFO AFTER IS 0x56119a61efd0
Checking filesystem on /dev/mapper/dshelf1
UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e
[1/7] checking root items
checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde
checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9
checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d
checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25
checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457
checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1
checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69
checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6
checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846
checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf
checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16
checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391
checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93
Fixed 0 roots.
[2/7] checking extents
checksum verify failed on 364637798400 wanted 0xc3ba5144 found 0x30a5fff1
checksum verify failed on 364637962240 wanted 0x91d33e8a found 0x00890be7
checksum verify failed on 11970891186176 wanted 0x6b46adfb found 0xe40e4cbe
checksum verify failed on 12511729680384 wanted 0x41be9fa6 found 0x01f06e9c
checksum verify failed on 12512010420224 wanted 0x365027e7 found 0xba73f602
checksum verify failed on 12512303677440 wanted 0xc08fc7a0 found 0x3cb30242
checksum verify failed on 13576951021568 wanted 0x9511e976 found 0xd8fe35ea
checksum verify failed on 13577013624832 wanted 0x76bbe220 found 0xdfabbbdb
checksum verify failed on 13577013919744 wanted 0xafb37267 found 0x69504e0d
(...)
Device extent[1, 11231377227776, 1073741824] didn't find the relative chunk.
Device extent[1, 11406397145088, 1073741824] didn't find the relative chunk.
Device extent[1, 11416060821504, 1073741824] didn't find the relative chunk.
Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk.
Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk.
data backref 385889992704 parent 11970901524480 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385889992704 parent 11970901524480 owner 0 offset 0 found 1 wanted 0 back 0x5611a97a3110 incorrect local backref count on 385889992704 root 11222 owner 54869 offset 0 found 0 wanted 1 back 0x5611a6e42180 backref disk bytenr does not match extent record, bytenr=385889992704, ref bytenr=0
backpointer mismatch on [385889992704 4096]
repair deleting extent record: key [385889992704,168,4096]
adding new data backref on 385889992704 parent 11970901524480 owner 0 offset 0 found 1
Repaired extent references for 385889992704
data backref 385900392448 parent 1429184937984 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385900392448 parent 1429184937984 owner 0 offset 0 found 1 wanted 0 back 0x5611a659e210 incorrect local backref count on 385900392448 root 11222 owner 55229 offset 524288 found 0 wanted 1 back 0x5611cd078e60 backref disk bytenr does not match extent record, bytenr=385900392448, ref bytenr=0
backpointer mismatch on [385900392448 4096]
repair deleting extent record: key [385900392448,168,4096]
adding new data backref on 385900392448 parent 1429184937984 owner 0 offset 0 found 1
Repaired extent references for 385900392448
data backref 385900478464 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385900478464 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e720 incorrect local backref count on 385900478464 root 11222 owner 55240 offset 0 found 0 wanted 1 back 0x5611cd079880 backref disk bytenr does not match extent record, bytenr=385900478464, ref bytenr=0
backpointer mismatch on [385900478464 4096]
repair deleting extent record: key [385900478464,168,4096]
adding new data backref on 385900478464 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385900478464
data backref 385907994624 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385907994624 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6ab0 incorrect local backref count on 385907994624 root 11222 owner 55204 offset 131072 found 0 wanted 1 back 0x5611cd0d05c0 backref disk bytenr does not match extent record, bytenr=385907994624, ref bytenr=0
backpointer mismatch on [385907994624 4096]
repair deleting extent record: key [385907994624,168,4096]
adding new data backref on 385907994624 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385907994624
data backref 385908449280 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385908449280 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3690 incorrect local backref count on 385908449280 root 11222 owner 55063 offset 524288 found 0 wanted 1 back 0x5611cd0d21d0 backref disk bytenr does not match extent record, bytenr=385908449280, ref bytenr=0
backpointer mismatch on [385908449280 8192]
repair deleting extent record: key [385908449280,168,8192]
adding new data backref on 385908449280 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908449280
data backref 385908580352 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385908580352 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a36f0 incorrect local backref count on 385908580352 root 11222 owner 55063 offset 655360 found 0 wanted 1 back 0x5611cd0d2db0 backref disk bytenr does not match extent record, bytenr=385908580352, ref bytenr=0
backpointer mismatch on [385908580352 8192]
repair deleting extent record: key [385908580352,168,8192]
adding new data backref on 385908580352 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385908580352
data backref 385909239808 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385909239808 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6b10 incorrect local backref count on 385909239808 root 11222 owner 55204 offset 262144 found 0 wanted 1 back 0x5611cd0c32f0 backref disk bytenr does not match extent record, bytenr=385909239808, ref bytenr=0
backpointer mismatch on [385909239808 4096]
repair deleting extent record: key [385909239808,168,4096]
adding new data backref on 385909239808 parent 13577102983168 owner 0 offset 0 found 1
Repaired extent references for 385909239808
data backref 385913561088 parent 13577747120128 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385913561088 parent 13577747120128 owner 0 offset 0 found 1 wanted 0 back 0x5611c89a3750 incorrect local backref count on 385913561088 root 11222 owner 55063 offset 786432 found 0 wanted 1 back 0x5611cd1196b0 backref disk bytenr does not match extent record, bytenr=385913561088, ref bytenr=0
backpointer mismatch on [385913561088 8192]
repair deleting extent record: key [385913561088,168,8192]
adding new data backref on 385913561088 parent 13577747120128 owner 0 offset 0 found 1
Repaired extent references for 385913561088
data backref 385913696256 parent 11651792814080 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385913696256 parent 11651792814080 owner 0 offset 0 found 1 wanted 0 back 0x5611baa7e780 incorrect local backref count on 385913696256 root 11222 owner 55240 offset 131072 found 0 wanted 1 back 0x5611cd11a3d0 backref disk bytenr does not match extent record, bytenr=385913696256, ref bytenr=0
backpointer mismatch on [385913696256 4096]
repair deleting extent record: key [385913696256,168,4096]
adding new data backref on 385913696256 parent 11651792814080 owner 0 offset 0 found 1
Repaired extent references for 385913696256
data backref 385914368000 parent 13577102983168 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 385914368000 parent 13577102983168 owner 0 offset 0 found 1 wanted 0 back 0x5611a86e6a50 incorrect local backref count on 385914368000 root 11222 owner 55204 offset 0 found 0 wanted 1 back 0x5611cd11d660 backref disk bytenr does not match extent record, bytenr=385914368000, ref bytenr=0
backpointer mismatch on [385914368000 8192]
repair deleting extent record: key [385914368000,168,8192]
adding new data backref on 385914368000 parent 13577102983168 owner 0 offset 0 found 1

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#200 lore raw

Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

Quoted 6 lines · Marc MERLIN

I have
./btrfs check --repair /dev/mapper/dshelf1
on hold (^Z), waiting for your ok to proceed.

All good to continue?

Hold on I'm looking at the code, I'm very confused, we shouldn't be finding any extent tree errors at this point. Let me work out what's going on. Thanks,

Josef