recovering btrfs filesystem from synology raid5
hello,
i have here a broken btrfs filesystem that was on a synology nas with raid5 (mdadm + lvm) where one disk started to be broken and another was removed from the raid roughly two months ago (but the number of events on that disk isn't that much lower). more details about the raid can be seen here: https://lore.kernel.org/linux-raid/[email protected]/T/
i was able to assemble the raid5 from 4 disks (3 with up-to-date data and one with older data) to the extent that i can use lvm to see the logical volume and btrfs filesystem on it, though the filesystem reports to be broken.
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 dm-19[4] dm-16[3] dm-13[1] dm-10[0]
31236781824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/4]
[UU_UU]
bitmap: 0/59 pages [0KB], 65536KB chunkunused devices: <none>
# btrfs check /dev/vg1000/lv
Opening filesystem to check...
parent transid verify failed on 4330339713024 wanted 2221844 found 2221848
parent transid verify failed on 4330321559552 wanted 2221843 found 1957353
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
Ignoring transid failure
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
leaf 4330321559552 items 87 free space 11792 generation 1957359 owner EXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1
fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9f
item 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105
itemsize 178
item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040
itemsize 39
item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218
itemsize 44
refs 14078082519432455026 gen 151715012486066369 flags
|FULL_BACKREF
tree block skinny level -1444450301
ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
Ignoring transid failure
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
leaf 4330321559552 items 87 free space 11792 generation 1957359 owner EXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1
fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9f
item 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105
itemsize 178
item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040
itemsize 39
item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218
itemsize 44
refs 14078082519432455026 gen 151715012486066369 flags
|FULL_BACKREF
tree block skinny level -1444450301
ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
leaf 4330321559552 items 87 free space 11792 generation 1957359 owner EXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1
fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9f
item 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105
itemsize 178
item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040
itemsize 39
item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218
itemsize 44
refs 14078082519432455026 gen 151715012486066369 flags
|FULL_BACKREF
tree block skinny level -1444450301
ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file systemi have the disks mapped in read-only mode with overlay set up over them so that i can freely experiment with the filesystem. but so far i was not able to get the filesystem to a state where i could read files from it, even with btrfs restore.
there are some data on the filesystem that are crucial for the client so my goal is to be able to recover at least those. so, i'd like to ask whether there is any chance to resolve these issues shown above to be able to at least list the files from the filesystem and recover at least a selection of them.
thank you in advance for any hints.
miroslav
Quoted 4 lines · Miroslav Šulc
hello,
i have here a broken btrfs filesystem that was on a synology nas with raid5 (mdadm + lvm)
AFAIK synology has some patches that connect btrfs and MD raid5 so it fixes the problems that btrfs-raid5 has. But the patches are not upstream and I don't know how it works.
Quoted 5 lines
where one disk started to be broken and another was
removed from the raid roughly two months ago (but the number of events on that disk isn't that much lower). more details about the raid can be seen here:
https://lore.kernel.org/linux-raid/[email protected]/T/
This looks like the problems are on the MD level with the device removed, added back and interrupted synchronization. The analysis points out there are some old data or stats. More below.
Quoted 12 lines
i was able to assemble the raid5 from 4 disks (3 with up-to-date data and one with older data) to the extent that i can use lvm to see the logical volume and btrfs filesystem on it, though the filesystem reports to be broken.
# cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md127 : active raid5 dm-19[4] dm-16[3] dm-13[1] dm-10[0] 31236781824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/4] [UU_UU] bitmap: 0/59 pages [0KB], 65536KB chunk
Quoted 6 lines
unused devices: <none>
# btrfs check /dev/vg1000/lv Opening filesystem to check... parent transid verify failed on 4330339713024 wanted 2221844 found 2221848
This error typically means old data were read, the difference is 4 transactions.
Quoted 9 lines
parent transid verify failed on 4330321559552 wanted 2221843 found 1957353
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
Ignoring transid failure
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
This kind of error does not apper often, the 'transid verify failed' from above usually has a consistent b-tree node/leaf but from a different transaction "epoch". A wrong item end means there's a mismatch in the stored and calculated data. Reasons for that can vary, a software bug can never be ruled out (but we'd see more reports) or the recovery of the b-tree node/leaf was partial and the item descriptor (the stored part) is older than what's actually found in the leaf (the calculated part).
Quoted 6 lines
leaf 4330321559552 items 87 free space 11792 generation 1957359 owner
EXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1 fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9fitem 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105
UNKNOWN.164 must be some synology NAS specific thing
Quoted 2 lines
itemsize 179
item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040
Same.
Quoted 4 lines
itemsize 39
item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218 itemsize 44 refs 14078082519432455026 gen 151715012486066369 flags
The numbers 14078082519432455026 and 151715012486066369 are likely a complete garbage, either a block that was randomly overwritten or a random block read and interpreted.
Quoted 2 lines
|FULL_BACKREF
tree block skinny level -1444450301
Levels are 0..7
Quoted 47 lines
ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
Ignoring transid failure
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
leaf 4330321559552 items 87 free space 11792 generation 1957359 ownerEXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1
fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9fitem 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105 itemsize 178 item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040 itemsize 39 item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218 itemsize 44 refs 14078082519432455026 gen 151715012486066369 flags |FULL_BACKREF tree block skinny level -1444450301ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
leaf 4330321559552 items 87 free space 11792 generation 1957359 ownerEXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1
fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9fitem 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105 itemsize 178 item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040 itemsize 39 item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218 itemsize 44 refs 14078082519432455026 gen 151715012486066369 flags |FULL_BACKREF tree block skinny level -1444450301ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system
Quoted 4 lines
i have the disks mapped in read-only mode with overlay set up over them so that i can freely experiment with the filesystem. but so far i was not able to get the filesystem to a state where i could read files from it, even with btrfs restore.
The recovery options for btrfs are to try to read partially consistent data from past generations, this is what 'btrfs restore' does, but it seems that the block data are mixed and even inconsistent within somethign that btrfs would consider a unit.
Quoted 5 lines
there are some data on the filesystem that are crucial for the client so my goal is to be able to recover at least those. so, i'd like to ask whether there is any chance to resolve these issues shown above to be able to at least list the files from the filesystem and recover at least a selection of them.
You can try 'btrfs restore' if you haven't done that yet but this may also require to write some custom code to work around the specific issues you find. This is also from a custom kernel from synology with apparent modifications that the upstream linux kernel does not have so we can't help with that beyond the common points.
hi David, thank you for your detail reply. i tried btrfs restore but that didn't recover anything. though i did set up the raid on my desktop and not on the nas because the one i have here does not support btrfs. from what you wrote, it comes to my mind that it would make sense to set up the raid on synology nas that supports btrfs and check whether that improves the situation or not, because of the modified system on the nas.
Dne 2024-08-30 01:02, David Sterba napsal:
Quoted 164 lines · Miroslav Šulc
Quoted 4 lines
hello,
i have here a broken btrfs filesystem that was on a synology nas with raid5 (mdadm + lvm)
AFAIK synology has some patches that connect btrfs and MD raid5 so it fixes the problems that btrfs-raid5 has. But the patches are not upstream and I don't know how it works.
Quoted 5 lines
where one disk started to be broken and another was
removed from the raid roughly two months ago (but the number of events on that disk isn't that much lower). more details about the raid can be seen here:
https://lore.kernel.org/linux-raid/[email protected]/T/This looks like the problems are on the MD level with the device removed, added back and interrupted synchronization. The analysis points out there are some old data or stats. More below.
Quoted 13 lines
i was able to assemble the raid5 from 4 disks (3 with up-to-date data and one with older data) to the extent that i can use lvm to see the logical volume and btrfs filesystem on it, though the filesystem reports to be broken.
# cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md127 : active raid5 dm-19[4] dm-16[3] dm-13[1] dm-10[0] 31236781824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/4] [UU_UU] bitmap: 0/59 pages [0KB], 65536KB chunkQuoted 6 lines
unused devices: <none>
# btrfs check /dev/vg1000/lv Opening filesystem to check... parent transid verify failed on 4330339713024 wanted 2221844 found 2221848This error typically means old data were read, the difference is 4 transactions.
Quoted 9 lines
parent transid verify failed on 4330321559552 wanted 2221843 found 1957353
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
Ignoring transid failure
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105This kind of error does not apper often, the 'transid verify failed' from above usually has a consistent b-tree node/leaf but from a different transaction "epoch". A wrong item end means there's a mismatch in the stored and calculated data. Reasons for that can vary, a software bug can never be ruled out (but we'd see more reports) or the recovery of the b-tree node/leaf was partial and the item descriptor (the stored part) is older than what's actually found in the leaf (the calculated part).
Quoted 6 lines
leaf 4330321559552 items 87 free space 11792 generation 1957359 owner
EXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1 fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9fitem 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105UNKNOWN.164 must be some synology NAS specific thing
Quoted 2 lines
itemsize 179
item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040Same.
Quoted 6 lines
itemsize 39
item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218 itemsize 44 refs 14078082519432455026 gen 151715012486066369flags
The numbers 14078082519432455026 and 151715012486066369 are likely a complete garbage, either a block that was randomly overwritten or a random block read and interpreted.
Quoted 2 lines
|FULL_BACKREF
tree block skinny level -1444450301Levels are 0..7
Quoted 51 lines
ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
parent transid verify failed on 4330321559552 wanted 2221843 found 1957359
Ignoring transid failure
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
leaf 4330321559552 items 87 free space 11792 generation 1957359 ownerEXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1
fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9fitem 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105 itemsize 178 item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040 itemsize 39 item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218 itemsize 44 refs 14078082519432455026 gen 151715012486066369 flags |FULL_BACKREF tree block skinny level -1444450301ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
corrupt leaf: root=2 block=4330321559552 slot=1, unexpected item end, have 16079 expect 16105
leaf 4330321559552 items 87 free space 11792 generation 1957359 ownerEXTENT_TREE
leaf 4330321559552 flags 0x1(WRITTEN) backref revision 1
fs uuid 7138daf1-5d16-4053-b559-0eb7da7d0a23
chunk uuid 55d47533-7e1e-4dda-bc51-f5ad0006fc9fitem 0 key (142836266048 UNKNOWN.164 1478857) itemoff 16105 itemsize 178 item 1 key (142837019712 UNKNOWN.189 818382621) itemoff 16040 itemsize 39 item 2 key (142836675648 METADATA_ITEM 2850516995) itemoff 16218 itemsize 44 refs 14078082519432455026 gen 151715012486066369 flags |FULL_BACKREF tree block skinny level -1444450301ERROR: leaf 4330321559552 slot 3 pointer invalid, offset 16369 size 33 leaf data limit 16283
ERROR: skip remaining slots
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file systemQuoted 4 lines
i have the disks mapped in read-only mode with overlay set up over them so that i can freely experiment with the filesystem. but so far i was not able to get the filesystem to a state where i could read files from it, even with btrfs restore.
The recovery options for btrfs are to try to read partially consistent data from past generations, this is what 'btrfs restore' does, but it seems that the block data are mixed and even inconsistent within somethign that btrfs would consider a unit.
Quoted 5 lines
there are some data on the filesystem that are crucial for the client so my goal is to be able to recover at least those. so, i'd like to ask whether there is any chance to resolve these issues shown above to be able to at least list the files from the filesystem and recover at least a selection of them.
You can try 'btrfs restore' if you haven't done that yet but this may also require to write some custom code to work around the specific issues you find. This is also from a custom kernel from synology with apparent modifications that the upstream linux kernel does not have so we can't help with that beyond the common points.