How to remove an unremovable file and directory?
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
I've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs. What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs roots
Quoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856
So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Quoted 11 lines · Henri Hyyryläinen
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
Qu
Quoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Thank you for the reply.
Quoted 7 lines
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
I did not use those tools. I only have Fedora 42 installed on this computer.
Quoted 3 lines
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
Yes, that seems the most likely explanation. I switched my motherboard, RAM, and CPU a few weeks ago, and when building a new setup out of the old components and 3 different drives (including one brand new NVME SSD) I noticed that each time I ran a btrfs scrub on any of those I got a bunch of csum errors. I initially thought that maybe the drives or something were faulty, but when I ran memtest it discovered very quickly a faulty single RAM stick. That RAM had been in my main system in a faulty state for months probably at least. The different computer I built from the old parts immediately stopped failing scrubs when I removed the faulty RAM stick.
So yeah, I think the corruption would have been caused by that RAM stick. I have now completely different RAM in my system and I've run the same memtest multiple times and it has not found any problem, so I think I have now fully good RAM and won't have the same corruption issue again.
One potentially relevant detail is that I replaced one of my disks just before I discovered the corruption with a bigger hard drive (at this point I had already swapped my RAM so there shouldn't be memory corruption problems during the replace). I started a btrfs replace operation, but I had a scrub running which was then canceled. That cancel for some reason also canceled the replace operation. When I noticed that I then restarted the replace and then let it run to completion (though I think the replace got canceled twice as I didn't realize after the first one what canceled it, so I needed to restart the replace multiple times). After completing the replace command without an error message I shut down my computer and removed the replaced drive. Upon booting my filesystem could not mount without the degraded flag. Somehow the replace had copied the right amount of data to the new drive, but for some reason the old drive was still marked as used. I had to then do a btrfs remove on that device which took a really long time but eventually succeeded as it seemingly had to rebuild the missing drive from raid data (even though all of the necessary data should have been copied already). So it seems like the btrfs replace operation doesn't fully work correctly if it is interrupted.
That could be a relevant detail, but as my filesystem only had one corrupt folder I somewhat doubt that as I would expect a lot more things to have gone wrong if the replace messed up the data. So to me that seems only like a minor bug in the replace operation, but one that did not cause this corruption.
Quoted 1 line
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Yes, my RAM should be totally fine now.
Thanks, I will try running that repair tonight, and hoping everything is then fine.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 4 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks:
-117
BTRFS error (device sdc):open_ctreefailed: -117
Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extents
Quoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Quoted 11 lines · Henri Hyyryläinen
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 4 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks:
-117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
Thanks,
Qu
Quoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4
That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Quoted 16 lines · Qu Wenruo
Quoted 10 lines · Henri Hyyryläinen
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Quoted 4 lines
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Is there a way for me to check that without mounting the filesystem? As far as I could find, none of the balance / scrub commands allow working on an unmounted filesystem. So I couldn't find out if I had any in canceled state.
Though, I'm pretty sure I let the last scrub and balance operation I tried to fully complete before starting using the check and repair commands. But I'm not absolutely certain that I didn't try one of those a last time and didn't let it fully complete.
I'll start that "btrfs check --readonly" command now. And I'll report back once it is done (hopefully by the morning in my timezone).
- Henri Hyyryläinen
Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
Quoted 180 lines · Qu Wenruo
Quoted 16 lines · Henri Hyyryläinen
Quoted 10 lines
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Quoted 6 lines · Henri Hyyryläinen
Quoted 4 lines
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Is there a way for me to check that without mounting the filesystem? As far as I could find, none of the balance / scrub commands allow working on an unmounted filesystem. So I couldn't find out if I had any in canceled state.
# btrfs ins dump-tree -t root <device>
Quoted 188 lines
Though, I'm pretty sure I let the last scrub and balance operation I tried to fully complete before starting using the check and repair commands. But I'm not absolutely certain that I didn't try one of those a last time and didn't let it fully complete.
I'll start that "btrfs check --readonly" command now. And I'll report back once it is done (hopefully by the morning in my timezone).
- Henri Hyyryläinen
Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
Quoted 180 lines · Qu Wenruo
Quoted 16 lines · Henri Hyyryläinen
Quoted 10 lines
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Thanks for the command.
I tried reading through the data but I could not determine if there was any information related to scrub or balance status.
Here's the full output I got:
Quoted 381 lines
btrfs-progs v6.16.1
root tree
leaf 227587769614336 items 61 free space 3547 generation 2035509 ownerROOT_TREEleaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
item 0 key (EXTENT_TREEROOT_ITEM0) itemoff 15844 itemsize 439
generation 2035509root_dirid0 bytenr 227587769630720byte_limit0bytes_used3319316480last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22035509
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 1 key (DEV_TREEROOT_ITEM0) itemoff 15405 itemsize 439
generation 2025974root_dirid0 bytenr 227594196631552byte_limit0bytes_used21889024last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 2generation_v22025974
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 2 key (FS_TREEINODE_REF6) itemoff 15388 itemsize 17
index 0 namelen 7 name: default
item 3 key (FS_TREEROOT_ITEM0) itemoff 14949 itemsize 439
generation 2025975root_dirid256 bytenr 227573939142656byte_limit0bytes_used6603997184last_snapshot2025951 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025975
uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025953 otransid 0 stransid 0 rtransid 0
ctime 1759258850.523837328 (2025-09-30 22:00:50)
otime 1679586168.0 (2023-03-23 17:42:48)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 4 key (FS_TREEROOT_REF256) itemoff 14921 itemsize 28
root ref key dirid 256 sequence 6 name .snapshots
item 5 key (ROOT_TREE_DIRINODE_ITEM0) itemoff 14761 itemsize 160
generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1679586169.0 (2023-03-23 17:42:49)
ctime 1679586169.0 (2023-03-23 17:42:49)
mtime 1679586169.0 (2023-03-23 17:42:49)
otime 1679586169.0 (2023-03-23 17:42:49)
item 6 key (ROOT_TREE_DIRINODE_REF6) itemoff 14749 itemsize 12
index 0 namelen 2 name: ..
item 7 key (ROOT_TREE_DIRDIR_ITEM2378154706) itemoff 14712 itemsize 37 location key (FS_TREEROOT_ITEM18446744073709551615) type DIR
transid 0data_len0name_len7
name: default
item 8 key (CSUM_TREEROOT_ITEM0) itemoff 14273 itemsize 439
generation 2025954root_dirid0 bytenr 227588044062720byte_limit0bytes_used49791238144last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 9 key (UUID_TREEROOT_ITEM0) itemoff 13834 itemsize 439
generation 2025954root_dirid0 bytenr 227587802447872byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 10 key (256ROOT_ITEM0) itemoff 13395 itemsize 439
generation 2025954root_dirid256 bytenr 227587802316800byte_limit0bytes_used49152last_snapshot2015386 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 1generation_v22025954
uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025954 otransid 31 stransid 0 rtransid 0
ctime 1759260258.238143629 (2025-09-30 22:24:18)
otime 1679588253.995834574 (2023-03-23 18:17:33)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 11 key (256ROOT_BACKREF5) itemoff 13367 itemsize 28
root backref key dirid 256 sequence 6 name .snapshots
item 12 key (256ROOT_REF14428) itemoff 13341 itemsize 26
root ref key dirid 28589 sequence 2 name snapshot
item 13 key (256ROOT_REF14451) itemoff 13315 itemsize 26
root ref key dirid 28635 sequence 2 name snapshot
item 14 key (256ROOT_REF14475) itemoff 13289 itemsize 26
root ref key dirid 28683 sequence 2 name snapshot
item 15 key (256ROOT_REF14499) itemoff 13263 itemsize 26
root ref key dirid 28731 sequence 2 name snapshot
item 16 key (256ROOT_REF14523) itemoff 13237 itemsize 26
root ref key dirid 28779 sequence 2 name snapshot
item 17 key (256ROOT_REF14547) itemoff 13211 itemsize 26
root ref key dirid 28827 sequence 2 name snapshot
item 18 key (256ROOT_REF14559) itemoff 13185 itemsize 26
root ref key dirid 28851 sequence 2 name snapshot
item 19 key (256ROOT_REF14583) itemoff 13159 itemsize 26
root ref key dirid 28899 sequence 2 name snapshot
item 20 key (256ROOT_REF14607) itemoff 13133 itemsize 26
root ref key dirid 28947 sequence 2 name snapshot
item 21 key (256ROOT_REF14631) itemoff 13107 itemsize 26
root ref key dirid 28995 sequence 2 name snapshot
item 22 key (256ROOT_REF14640) itemoff 13081 itemsize 26
root ref key dirid 29013 sequence 2 name snapshot
item 23 key (256ROOT_REF14641) itemoff 13055 itemsize 26
root ref key dirid 29015 sequence 2 name snapshot
item 24 key (256ROOT_REF14642) itemoff 13029 itemsize 26
root ref key dirid 29017 sequence 2 name snapshot
item 25 key (256ROOT_REF14643) itemoff 13003 itemsize 26
root ref key dirid 29019 sequence 2 name snapshot
item 26 key (256ROOT_REF14644) itemoff 12977 itemsize 26
root ref key dirid 29021 sequence 2 name snapshot
item 27 key (256ROOT_REF14645) itemoff 12951 itemsize 26
root ref key dirid 29023 sequence 2 name snapshot
item 28 key (14428ROOT_ITEM1849346) itemoff 12512 itemsize 439
generation 2025978root_dirid256 bytenr 71398781173760byte_limit0bytes_used8815476736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025978
uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
ctime 1758406304.841464812 (2025-09-21 01:11:44)
otime 1758406304.840394467 (2025-09-21 01:11:44)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 29 key (14428ROOT_BACKREF256) itemoff 12486 itemsize 26
root backref key dirid 28589 sequence 2 name snapshot
item 30 key (14451ROOT_ITEM1899154) itemoff 12047 itemsize 439
generation 2025981root_dirid256 bytenr 71398809468928byte_limit0bytes_used6747799552last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025981
uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
ctime 1758484800.817458724 (2025-09-21 23:00:00)
otime 1758488400.101635440 (2025-09-22 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 31 key (14451ROOT_BACKREF256) itemoff 12021 itemsize 26
root backref key dirid 28635 sequence 2 name snapshot
item 32 key (14475ROOT_ITEM1942246) itemoff 11582 itemsize 439
generation 2025984root_dirid256 bytenr 71398831259648byte_limit0bytes_used6755549184last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025984
uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
ctime 1758570000.151997903 (2025-09-22 22:40:00)
otime 1758574800.777676964 (2025-09-23 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 33 key (14475ROOT_BACKREF256) itemoff 11556 itemsize 26
root backref key dirid 28683 sequence 2 name snapshot
item 34 key (14499ROOT_ITEM1977241) itemoff 11117 itemsize 439
generation 2025986root_dirid256 bytenr 71398873153536byte_limit0bytes_used6538182656last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025986
uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
ctime 1758661201.518446292 (2025-09-24 00:00:01)
otime 1758661204.161207119 (2025-09-24 00:00:04)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 35 key (14499ROOT_BACKREF256) itemoff 11091 itemsize 26
root backref key dirid 28731 sequence 2 name snapshot
item 36 key (14523ROOT_ITEM2001507) itemoff 10652 itemsize 439
generation 2025988root_dirid256 bytenr 71398873497600byte_limit0bytes_used6561398784last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025988
uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
ctime 1758747600.712874426 (2025-09-25 00:00:00)
otime 1758747600.840460363 (2025-09-25 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 37 key (14523ROOT_BACKREF256) itemoff 10626 itemsize 26
root backref key dirid 28779 sequence 2 name snapshot
item 38 key (14547ROOT_ITEM2017206) itemoff 10187 itemsize 439
generation 2025990root_dirid256 bytenr 71398872924160byte_limit0bytes_used6570868736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025990
uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
ctime 1758834001.88320469 (2025-09-26 00:00:01)
otime 1758834001.271433540 (2025-09-26 00:00:01)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 39 key (14547ROOT_BACKREF256) itemoff 10161 itemsize 26
root backref key dirid 28827 sequence 2 name snapshot
item 40 key (14559ROOT_ITEM2019735) itemoff 9722 itemsize 439
generation 2025992root_dirid256 bytenr 75423231606784byte_limit0bytes_used6576078848last_snapshot2019735 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025992
uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
ctime 1758919659.690576855 (2025-09-26 23:47:39)
otime 1758920400.700930626 (2025-09-27 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 41 key (14559ROOT_BACKREF256) itemoff 9696 itemsize 26
root backref key dirid 28851 sequence 2 name snapshot
item 42 key (14583ROOT_ITEM2022097) itemoff 9257 itemsize 439
generation 2025994root_dirid256 bytenr 65812067024896byte_limit0bytes_used6557646848last_snapshot2022097 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025994
uuid 860f0635-f296-a04b-9fc5-bd85d480799c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
ctime 1759006819.37527411 (2025-09-28 00:00:19)
otime 1759006820.861182212 (2025-09-28 00:00:20)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 43 key (14583ROOT_BACKREF256) itemoff 9231 itemsize 26
root backref key dirid 28899 sequence 2 name snapshot
item 44 key (14607ROOT_ITEM2024585) itemoff 8792 itemsize 439
generation 2025996root_dirid256 bytenr 227573944696832byte_limit0bytes_used6583533568last_snapshot2024585 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025996
uuid af3524d0-8a9a-894c-b5b7-705b10114326
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
ctime 1759089642.387498362 (2025-09-28 23:00:42)
otime 1759093200.412309185 (2025-09-29 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 45 key (14607ROOT_BACKREF256) itemoff 8766 itemsize 26
root backref key dirid 28947 sequence 2 name snapshot
item 46 key (14631ROOT_ITEM2025813) itemoff 8327 itemsize 439
generation 2025998root_dirid256 bytenr 178141351657472byte_limit0bytes_used6602981376last_snapshot2025813 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025998
uuid 6f085114-93b0-d146-8997-9a41ff62f8db
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
ctime 1759177902.175859896 (2025-09-29 23:31:42)
otime 1759210423.120386595 (2025-09-30 08:33:43)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 47 key (14631ROOT_BACKREF256) itemoff 8301 itemsize 26
root backref key dirid 28995 sequence 2 name snapshot
item 48 key (14640ROOT_ITEM2025865) itemoff 7862 itemsize 439
generation 2026000root_dirid256 bytenr 194611771719680byte_limit0bytes_used6603735040last_snapshot2025865 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026000
uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
ctime 1759237252.139079795 (2025-09-30 16:00:52)
otime 1759240800.56609287 (2025-09-30 17:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 49 key (14640ROOT_BACKREF256) itemoff 7836 itemsize 26
root backref key dirid 29013 sequence 2 name snapshot
item 50 key (14641ROOT_ITEM2025870) itemoff 7397 itemsize 439
generation 2026002root_dirid256 bytenr 65801342844928byte_limit0bytes_used6603292672last_snapshot2025870 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026002
uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
ctime 1759241717.101400473 (2025-09-30 17:15:17)
otime 1759244400.232224532 (2025-09-30 18:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 51 key (14641ROOT_BACKREF256) itemoff 7371 itemsize 26
root backref key dirid 29015 sequence 2 name snapshot
item 52 key (14642ROOT_ITEM2025874) itemoff 6932 itemsize 439
generation 2026004root_dirid256 bytenr 187252895498240byte_limit0bytes_used6603341824last_snapshot2025874 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026004
uuid f1761dfd-ec60-724a-9af0-dea952cc6771
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
ctime 1759244441.841998010 (2025-09-30 18:00:41)
otime 1759248000.122051877 (2025-09-30 19:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 53 key (14642ROOT_BACKREF256) itemoff 6906 itemsize 26
root backref key dirid 29017 sequence 2 name snapshot
item 54 key (14643ROOT_ITEM2025904) itemoff 6467 itemsize 439
generation 2026006root_dirid256 bytenr 227587769647104byte_limit0bytes_used6603587584last_snapshot2025904 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026006
uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
ctime 1759251597.815452999 (2025-09-30 19:59:57)
otime 1759251600.884521740 (2025-09-30 20:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 55 key (14643ROOT_BACKREF256) itemoff 6441 itemsize 26
root backref key dirid 29019 sequence 2 name snapshot
item 56 key (14644ROOT_ITEM2025922) itemoff 6002 itemsize 439
generation 2026008root_dirid256 bytenr 227594191208448byte_limit0bytes_used6603653120last_snapshot2025922 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026008
uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
ctime 1759254975.20573006 (2025-09-30 20:56:15)
otime 1759255200.196692319 (2025-09-30 21:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 57 key (14644ROOT_BACKREF256) itemoff 5976 itemsize 26
root backref key dirid 29021 sequence 2 name snapshot
item 58 key (14645ROOT_ITEM2025951) itemoff 5537 itemsize 439
generation 2026010root_dirid256 bytenr 193309358817280byte_limit0bytes_used6603964416last_snapshot2025951 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026010
uuid b46ce558-2c39-3944-91d4-b2b229f04a16
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
ctime 1759257514.988583086 (2025-09-30 21:38:34)
otime 1759258800.482340435 (2025-09-30 22:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 59 key (14645ROOT_BACKREF256) itemoff 5511 itemsize 26
root backref key dirid 29023 sequence 2 name snapshot
item 60 key (DATA_RELOC_TREEROOT_ITEM0) itemoff 5072 itemsize 439
generation 2018764root_dirid256 bytenr 71399055278080byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22018764
Quoted 8 lines
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000received_uuid00000000-0000-0000-0000-000000000000 ctransid 0 otransid 0 stransid 0 rtransid 0 ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
Quoted 200 lines · Henri Hyyryläinen
Quoted 6 lines
Quoted 4 lines
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Is there a way for me to check that without mounting the filesystem? As far as I could find, none of the balance / scrub commands allow working on an unmounted filesystem. So I couldn't find out if I had any in canceled state.
# btrfs ins dump-tree -t root <device>Quoted 188 lines
Though, I'm pretty sure I let the last scrub and balance operation I tried to fully complete before starting using the check and repair commands. But I'm not absolutely certain that I didn't try one of those a last time and didn't let it fully complete.
I'll start that "btrfs check --readonly" command now. And I'll report back once it is done (hopefully by the morning in my timezone).
- Henri Hyyryläinen
Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
Quoted 180 lines · Qu Wenruo
Quoted 16 lines · Henri Hyyryläinen
Quoted 10 lines
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs-progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Quoted 3 lines · Henri Hyyryläinen
Thanks for the command.
I tried reading through the data but I could not determine if there was any information related to scrub or balance status.
It's fine. No persistent item for balance/dev-replace. So it's not involved.
Waiting for the readonly check result.
And meanwhile, please prepare an environment to compile btrfs-progs.
If there is only one stray dev extent, I'll create a simple dirty fix so that you can mount the fs again.
Thanks,
Qu
Quoted 595 lines
Here's the full output I got:
Quoted 381 lines
btrfs-progs v6.16.1
root tree
leaf 227587769614336 items 61 free space 3547 generation 2035509 ownerROOT_TREEleaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
item 0 key (EXTENT_TREEROOT_ITEM0) itemoff 15844 itemsize 439
generation 2035509root_dirid0 bytenr 227587769630720byte_limit0bytes_used3319316480last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22035509
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 1 key (DEV_TREEROOT_ITEM0) itemoff 15405 itemsize 439
generation 2025974root_dirid0 bytenr 227594196631552byte_limit0bytes_used21889024last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 2generation_v22025974
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 2 key (FS_TREEINODE_REF6) itemoff 15388 itemsize 17
index 0 namelen 7 name: default
item 3 key (FS_TREEROOT_ITEM0) itemoff 14949 itemsize 439
generation 2025975root_dirid256 bytenr 227573939142656byte_limit0bytes_used6603997184last_snapshot2025951 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025975
uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025953 otransid 0 stransid 0 rtransid 0
ctime 1759258850.523837328 (2025-09-30 22:00:50)
otime 1679586168.0 (2023-03-23 17:42:48)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 4 key (FS_TREEROOT_REF256) itemoff 14921 itemsize 28
root ref key dirid 256 sequence 6 name .snapshots
item 5 key (ROOT_TREE_DIRINODE_ITEM0) itemoff 14761 itemsize 160
generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1679586169.0 (2023-03-23 17:42:49)
ctime 1679586169.0 (2023-03-23 17:42:49)
mtime 1679586169.0 (2023-03-23 17:42:49)
otime 1679586169.0 (2023-03-23 17:42:49)
item 6 key (ROOT_TREE_DIRINODE_REF6) itemoff 14749 itemsize 12
index 0 namelen 2 name: ..
item 7 key (ROOT_TREE_DIRDIR_ITEM2378154706) itemoff 14712 itemsize 37 location key (FS_TREEROOT_ITEM18446744073709551615) type DIR
transid 0data_len0name_len7
name: default
item 8 key (CSUM_TREEROOT_ITEM0) itemoff 14273 itemsize 439
generation 2025954root_dirid0 bytenr 227588044062720byte_limit0bytes_used49791238144last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 9 key (UUID_TREEROOT_ITEM0) itemoff 13834 itemsize 439
generation 2025954root_dirid0 bytenr 227587802447872byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 10 key (256ROOT_ITEM0) itemoff 13395 itemsize 439
generation 2025954root_dirid256 bytenr 227587802316800byte_limit0bytes_used49152last_snapshot2015386 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 1generation_v22025954
uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025954 otransid 31 stransid 0 rtransid 0
ctime 1759260258.238143629 (2025-09-30 22:24:18)
otime 1679588253.995834574 (2023-03-23 18:17:33)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 11 key (256ROOT_BACKREF5) itemoff 13367 itemsize 28
root backref key dirid 256 sequence 6 name .snapshots
item 12 key (256ROOT_REF14428) itemoff 13341 itemsize 26
root ref key dirid 28589 sequence 2 name snapshot
item 13 key (256ROOT_REF14451) itemoff 13315 itemsize 26
root ref key dirid 28635 sequence 2 name snapshot
item 14 key (256ROOT_REF14475) itemoff 13289 itemsize 26
root ref key dirid 28683 sequence 2 name snapshot
item 15 key (256ROOT_REF14499) itemoff 13263 itemsize 26
root ref key dirid 28731 sequence 2 name snapshot
item 16 key (256ROOT_REF14523) itemoff 13237 itemsize 26
root ref key dirid 28779 sequence 2 name snapshot
item 17 key (256ROOT_REF14547) itemoff 13211 itemsize 26
root ref key dirid 28827 sequence 2 name snapshot
item 18 key (256ROOT_REF14559) itemoff 13185 itemsize 26
root ref key dirid 28851 sequence 2 name snapshot
item 19 key (256ROOT_REF14583) itemoff 13159 itemsize 26
root ref key dirid 28899 sequence 2 name snapshot
item 20 key (256ROOT_REF14607) itemoff 13133 itemsize 26
root ref key dirid 28947 sequence 2 name snapshot
item 21 key (256ROOT_REF14631) itemoff 13107 itemsize 26
root ref key dirid 28995 sequence 2 name snapshot
item 22 key (256ROOT_REF14640) itemoff 13081 itemsize 26
root ref key dirid 29013 sequence 2 name snapshot
item 23 key (256ROOT_REF14641) itemoff 13055 itemsize 26
root ref key dirid 29015 sequence 2 name snapshot
item 24 key (256ROOT_REF14642) itemoff 13029 itemsize 26
root ref key dirid 29017 sequence 2 name snapshot
item 25 key (256ROOT_REF14643) itemoff 13003 itemsize 26
root ref key dirid 29019 sequence 2 name snapshot
item 26 key (256ROOT_REF14644) itemoff 12977 itemsize 26
root ref key dirid 29021 sequence 2 name snapshot
item 27 key (256ROOT_REF14645) itemoff 12951 itemsize 26
root ref key dirid 29023 sequence 2 name snapshot
item 28 key (14428ROOT_ITEM1849346) itemoff 12512 itemsize 439
generation 2025978root_dirid256 bytenr 71398781173760byte_limit0bytes_used8815476736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025978
uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
ctime 1758406304.841464812 (2025-09-21 01:11:44)
otime 1758406304.840394467 (2025-09-21 01:11:44)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 29 key (14428ROOT_BACKREF256) itemoff 12486 itemsize 26
root backref key dirid 28589 sequence 2 name snapshot
item 30 key (14451ROOT_ITEM1899154) itemoff 12047 itemsize 439
generation 2025981root_dirid256 bytenr 71398809468928byte_limit0bytes_used6747799552last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025981
uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
ctime 1758484800.817458724 (2025-09-21 23:00:00)
otime 1758488400.101635440 (2025-09-22 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 31 key (14451ROOT_BACKREF256) itemoff 12021 itemsize 26
root backref key dirid 28635 sequence 2 name snapshot
item 32 key (14475ROOT_ITEM1942246) itemoff 11582 itemsize 439
generation 2025984root_dirid256 bytenr 71398831259648byte_limit0bytes_used6755549184last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025984
uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
ctime 1758570000.151997903 (2025-09-22 22:40:00)
otime 1758574800.777676964 (2025-09-23 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 33 key (14475ROOT_BACKREF256) itemoff 11556 itemsize 26
root backref key dirid 28683 sequence 2 name snapshot
item 34 key (14499ROOT_ITEM1977241) itemoff 11117 itemsize 439
generation 2025986root_dirid256 bytenr 71398873153536byte_limit0bytes_used6538182656last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025986
uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
ctime 1758661201.518446292 (2025-09-24 00:00:01)
otime 1758661204.161207119 (2025-09-24 00:00:04)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 35 key (14499ROOT_BACKREF256) itemoff 11091 itemsize 26
root backref key dirid 28731 sequence 2 name snapshot
item 36 key (14523ROOT_ITEM2001507) itemoff 10652 itemsize 439
generation 2025988root_dirid256 bytenr 71398873497600byte_limit0bytes_used6561398784last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025988
uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
ctime 1758747600.712874426 (2025-09-25 00:00:00)
otime 1758747600.840460363 (2025-09-25 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 37 key (14523ROOT_BACKREF256) itemoff 10626 itemsize 26
root backref key dirid 28779 sequence 2 name snapshot
item 38 key (14547ROOT_ITEM2017206) itemoff 10187 itemsize 439
generation 2025990root_dirid256 bytenr 71398872924160byte_limit0bytes_used6570868736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025990
uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
ctime 1758834001.88320469 (2025-09-26 00:00:01)
otime 1758834001.271433540 (2025-09-26 00:00:01)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 39 key (14547ROOT_BACKREF256) itemoff 10161 itemsize 26
root backref key dirid 28827 sequence 2 name snapshot
item 40 key (14559ROOT_ITEM2019735) itemoff 9722 itemsize 439
generation 2025992root_dirid256 bytenr 75423231606784byte_limit0bytes_used6576078848last_snapshot2019735 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025992
uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
ctime 1758919659.690576855 (2025-09-26 23:47:39)
otime 1758920400.700930626 (2025-09-27 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 41 key (14559ROOT_BACKREF256) itemoff 9696 itemsize 26
root backref key dirid 28851 sequence 2 name snapshot
item 42 key (14583ROOT_ITEM2022097) itemoff 9257 itemsize 439
generation 2025994root_dirid256 bytenr 65812067024896byte_limit0bytes_used6557646848last_snapshot2022097 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025994
uuid 860f0635-f296-a04b-9fc5-bd85d480799c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
ctime 1759006819.37527411 (2025-09-28 00:00:19)
otime 1759006820.861182212 (2025-09-28 00:00:20)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 43 key (14583ROOT_BACKREF256) itemoff 9231 itemsize 26
root backref key dirid 28899 sequence 2 name snapshot
item 44 key (14607ROOT_ITEM2024585) itemoff 8792 itemsize 439
generation 2025996root_dirid256 bytenr 227573944696832byte_limit0bytes_used6583533568last_snapshot2024585 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025996
uuid af3524d0-8a9a-894c-b5b7-705b10114326
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
ctime 1759089642.387498362 (2025-09-28 23:00:42)
otime 1759093200.412309185 (2025-09-29 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 45 key (14607ROOT_BACKREF256) itemoff 8766 itemsize 26
root backref key dirid 28947 sequence 2 name snapshot
item 46 key (14631ROOT_ITEM2025813) itemoff 8327 itemsize 439
generation 2025998root_dirid256 bytenr 178141351657472byte_limit0bytes_used6602981376last_snapshot2025813 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025998
uuid 6f085114-93b0-d146-8997-9a41ff62f8db
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
ctime 1759177902.175859896 (2025-09-29 23:31:42)
otime 1759210423.120386595 (2025-09-30 08:33:43)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 47 key (14631ROOT_BACKREF256) itemoff 8301 itemsize 26
root backref key dirid 28995 sequence 2 name snapshot
item 48 key (14640ROOT_ITEM2025865) itemoff 7862 itemsize 439
generation 2026000root_dirid256 bytenr 194611771719680byte_limit0bytes_used6603735040last_snapshot2025865 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026000
uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
ctime 1759237252.139079795 (2025-09-30 16:00:52)
otime 1759240800.56609287 (2025-09-30 17:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 49 key (14640ROOT_BACKREF256) itemoff 7836 itemsize 26
root backref key dirid 29013 sequence 2 name snapshot
item 50 key (14641ROOT_ITEM2025870) itemoff 7397 itemsize 439
generation 2026002root_dirid256 bytenr 65801342844928byte_limit0bytes_used6603292672last_snapshot2025870 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026002
uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
ctime 1759241717.101400473 (2025-09-30 17:15:17)
otime 1759244400.232224532 (2025-09-30 18:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 51 key (14641ROOT_BACKREF256) itemoff 7371 itemsize 26
root backref key dirid 29015 sequence 2 name snapshot
item 52 key (14642ROOT_ITEM2025874) itemoff 6932 itemsize 439
generation 2026004root_dirid256 bytenr 187252895498240byte_limit0bytes_used6603341824last_snapshot2025874 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026004
uuid f1761dfd-ec60-724a-9af0-dea952cc6771
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
ctime 1759244441.841998010 (2025-09-30 18:00:41)
otime 1759248000.122051877 (2025-09-30 19:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 53 key (14642ROOT_BACKREF256) itemoff 6906 itemsize 26
root backref key dirid 29017 sequence 2 name snapshot
item 54 key (14643ROOT_ITEM2025904) itemoff 6467 itemsize 439
generation 2026006root_dirid256 bytenr 227587769647104byte_limit0bytes_used6603587584last_snapshot2025904 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026006
uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
ctime 1759251597.815452999 (2025-09-30 19:59:57)
otime 1759251600.884521740 (2025-09-30 20:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 55 key (14643ROOT_BACKREF256) itemoff 6441 itemsize 26
root backref key dirid 29019 sequence 2 name snapshot
item 56 key (14644ROOT_ITEM2025922) itemoff 6002 itemsize 439
generation 2026008root_dirid256 bytenr 227594191208448byte_limit0bytes_used6603653120last_snapshot2025922 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026008
uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
ctime 1759254975.20573006 (2025-09-30 20:56:15)
otime 1759255200.196692319 (2025-09-30 21:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 57 key (14644ROOT_BACKREF256) itemoff 5976 itemsize 26
root backref key dirid 29021 sequence 2 name snapshot
item 58 key (14645ROOT_ITEM2025951) itemoff 5537 itemsize 439
generation 2026010root_dirid256 bytenr 193309358817280byte_limit0bytes_used6603964416last_snapshot2025951 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026010
uuid b46ce558-2c39-3944-91d4-b2b229f04a16
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
ctime 1759257514.988583086 (2025-09-30 21:38:34)
otime 1759258800.482340435 (2025-09-30 22:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 59 key (14645ROOT_BACKREF256) itemoff 5511 itemsize 26
root backref key dirid 29023 sequence 2 name snapshot
item 60 key (DATA_RELOC_TREEROOT_ITEM0) itemoff 5072 itemsize 439
generation 2018764root_dirid256 bytenr 71399055278080byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22018764Quoted 8 lines
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000received_uuid00000000-0000-0000-0000-000000000000 ctransid 0 otransid 0 stransid 0 rtransid 0 ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
Quoted 200 lines · Henri Hyyryläinen
Quoted 6 lines
Quoted 4 lines
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Is there a way for me to check that without mounting the filesystem? As far as I could find, none of the balance / scrub commands allow working on an unmounted filesystem. So I couldn't find out if I had any in canceled state.
# btrfs ins dump-tree -t root <device>Quoted 188 lines
Though, I'm pretty sure I let the last scrub and balance operation I tried to fully complete before starting using the check and repair commands. But I'm not absolutely certain that I didn't try one of those a last time and didn't let it fully complete.
I'll start that "btrfs check --readonly" command now. And I'll report back once it is done (hopefully by the morning in my timezone).
- Henri Hyyryläinen
Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
Quoted 180 lines · Qu Wenruo
Quoted 16 lines · Henri Hyyryläinen
Quoted 10 lines
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs- progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Hello,
The check has completed and, probably thanks to the second repair, it did not find much. Here's the full output:
Quoted 12 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. [4/8] checking free space cache
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS) Opening filesystem to check...
Checking filesystem on/dev/sdb
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
Quoted 8 lines
found 49454062784512 bytes used, no error found
total csum bytes: 48225005724
total tree bytes: 72333164544
total fs tree bytes: 19189743616
total extent tree bytes: 3319316480
btree space waste bytes: 5998261217
file data blocks allocated: 119737771229184
referenced 72125752061952
- Henri Hyyryläinen
Qu Wenruo kirjoitti 5.10.2025 klo 1.41:
Quoted 618 lines · Henri Hyyryläinen
Quoted 3 lines
Thanks for the command.
I tried reading through the data but I could not determine if there was any information related to scrub or balance status.
It's fine. No persistent item for balance/dev-replace. So it's not involved.
Waiting for the readonly check result.
And meanwhile, please prepare an environment to compile btrfs-progs.
If there is only one stray dev extent, I'll create a simple dirty fix so that you can mount the fs again.
Thanks,
QuQuoted 599 lines
Here's the full output I got:
Quoted 381 lines
btrfs-progs v6.16.1
root tree
leaf 227587769614336 items 61 free space 3547 generation 2035509 ownerROOT_TREEleaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
item 0 key (EXTENT_TREEROOT_ITEM0) itemoff 15844 itemsize 439
generation 2035509root_dirid0 bytenr 227587769630720byte_limit0bytes_used3319316480last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22035509
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 1 key (DEV_TREEROOT_ITEM0) itemoff 15405 itemsize 439
generation 2025974root_dirid0 bytenr 227594196631552byte_limit0bytes_used21889024last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 2generation_v22025974
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 2 key (FS_TREEINODE_REF6) itemoff 15388 itemsize 17
index 0 namelen 7 name: default
item 3 key (FS_TREEROOT_ITEM0) itemoff 14949 itemsize 439
generation 2025975root_dirid256 bytenr 227573939142656byte_limit0bytes_used6603997184last_snapshot2025951 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025975
uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025953 otransid 0 stransid 0 rtransid 0
ctime 1759258850.523837328 (2025-09-30 22:00:50)
otime 1679586168.0 (2023-03-23 17:42:48)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 4 key (FS_TREEROOT_REF256) itemoff 14921 itemsize 28
root ref key dirid 256 sequence 6 name .snapshots
item 5 key (ROOT_TREE_DIRINODE_ITEM0) itemoff 14761 itemsize 160
generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1679586169.0 (2023-03-23 17:42:49)
ctime 1679586169.0 (2023-03-23 17:42:49)
mtime 1679586169.0 (2023-03-23 17:42:49)
otime 1679586169.0 (2023-03-23 17:42:49)
item 6 key (ROOT_TREE_DIRINODE_REF6) itemoff 14749 itemsize 12
index 0 namelen 2 name: ..
item 7 key (ROOT_TREE_DIRDIR_ITEM2378154706) itemoff 14712 itemsize 37 location key (FS_TREEROOT_ITEM18446744073709551615) type DIR
transid 0data_len0name_len7
name: default
item 8 key (CSUM_TREEROOT_ITEM0) itemoff 14273 itemsize 439
generation 2025954root_dirid0 bytenr 227588044062720byte_limit0bytes_used49791238144last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 9 key (UUID_TREEROOT_ITEM0) itemoff 13834 itemsize 439
generation 2025954root_dirid0 bytenr 227587802447872byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 10 key (256ROOT_ITEM0) itemoff 13395 itemsize 439
generation 2025954root_dirid256 bytenr 227587802316800byte_limit0bytes_used49152last_snapshot2015386 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 1generation_v22025954
uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025954 otransid 31 stransid 0 rtransid 0
ctime 1759260258.238143629 (2025-09-30 22:24:18)
otime 1679588253.995834574 (2023-03-23 18:17:33)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 11 key (256ROOT_BACKREF5) itemoff 13367 itemsize 28
root backref key dirid 256 sequence 6 name .snapshots
item 12 key (256ROOT_REF14428) itemoff 13341 itemsize 26
root ref key dirid 28589 sequence 2 name snapshot
item 13 key (256ROOT_REF14451) itemoff 13315 itemsize 26
root ref key dirid 28635 sequence 2 name snapshot
item 14 key (256ROOT_REF14475) itemoff 13289 itemsize 26
root ref key dirid 28683 sequence 2 name snapshot
item 15 key (256ROOT_REF14499) itemoff 13263 itemsize 26
root ref key dirid 28731 sequence 2 name snapshot
item 16 key (256ROOT_REF14523) itemoff 13237 itemsize 26
root ref key dirid 28779 sequence 2 name snapshot
item 17 key (256ROOT_REF14547) itemoff 13211 itemsize 26
root ref key dirid 28827 sequence 2 name snapshot
item 18 key (256ROOT_REF14559) itemoff 13185 itemsize 26
root ref key dirid 28851 sequence 2 name snapshot
item 19 key (256ROOT_REF14583) itemoff 13159 itemsize 26
root ref key dirid 28899 sequence 2 name snapshot
item 20 key (256ROOT_REF14607) itemoff 13133 itemsize 26
root ref key dirid 28947 sequence 2 name snapshot
item 21 key (256ROOT_REF14631) itemoff 13107 itemsize 26
root ref key dirid 28995 sequence 2 name snapshot
item 22 key (256ROOT_REF14640) itemoff 13081 itemsize 26
root ref key dirid 29013 sequence 2 name snapshot
item 23 key (256ROOT_REF14641) itemoff 13055 itemsize 26
root ref key dirid 29015 sequence 2 name snapshot
item 24 key (256ROOT_REF14642) itemoff 13029 itemsize 26
root ref key dirid 29017 sequence 2 name snapshot
item 25 key (256ROOT_REF14643) itemoff 13003 itemsize 26
root ref key dirid 29019 sequence 2 name snapshot
item 26 key (256ROOT_REF14644) itemoff 12977 itemsize 26
root ref key dirid 29021 sequence 2 name snapshot
item 27 key (256ROOT_REF14645) itemoff 12951 itemsize 26
root ref key dirid 29023 sequence 2 name snapshot
item 28 key (14428ROOT_ITEM1849346) itemoff 12512 itemsize 439
generation 2025978root_dirid256 bytenr 71398781173760byte_limit0bytes_used8815476736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025978
uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
ctime 1758406304.841464812 (2025-09-21 01:11:44)
otime 1758406304.840394467 (2025-09-21 01:11:44)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 29 key (14428ROOT_BACKREF256) itemoff 12486 itemsize 26
root backref key dirid 28589 sequence 2 name snapshot
item 30 key (14451ROOT_ITEM1899154) itemoff 12047 itemsize 439
generation 2025981root_dirid256 bytenr 71398809468928byte_limit0bytes_used6747799552last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025981
uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
ctime 1758484800.817458724 (2025-09-21 23:00:00)
otime 1758488400.101635440 (2025-09-22 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 31 key (14451ROOT_BACKREF256) itemoff 12021 itemsize 26
root backref key dirid 28635 sequence 2 name snapshot
item 32 key (14475ROOT_ITEM1942246) itemoff 11582 itemsize 439
generation 2025984root_dirid256 bytenr 71398831259648byte_limit0bytes_used6755549184last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025984
uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
ctime 1758570000.151997903 (2025-09-22 22:40:00)
otime 1758574800.777676964 (2025-09-23 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 33 key (14475ROOT_BACKREF256) itemoff 11556 itemsize 26
root backref key dirid 28683 sequence 2 name snapshot
item 34 key (14499ROOT_ITEM1977241) itemoff 11117 itemsize 439
generation 2025986root_dirid256 bytenr 71398873153536byte_limit0bytes_used6538182656last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025986
uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
ctime 1758661201.518446292 (2025-09-24 00:00:01)
otime 1758661204.161207119 (2025-09-24 00:00:04)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 35 key (14499ROOT_BACKREF256) itemoff 11091 itemsize 26
root backref key dirid 28731 sequence 2 name snapshot
item 36 key (14523ROOT_ITEM2001507) itemoff 10652 itemsize 439
generation 2025988root_dirid256 bytenr 71398873497600byte_limit0bytes_used6561398784last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025988
uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
ctime 1758747600.712874426 (2025-09-25 00:00:00)
otime 1758747600.840460363 (2025-09-25 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 37 key (14523ROOT_BACKREF256) itemoff 10626 itemsize 26
root backref key dirid 28779 sequence 2 name snapshot
item 38 key (14547ROOT_ITEM2017206) itemoff 10187 itemsize 439
generation 2025990root_dirid256 bytenr 71398872924160byte_limit0bytes_used6570868736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025990
uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
ctime 1758834001.88320469 (2025-09-26 00:00:01)
otime 1758834001.271433540 (2025-09-26 00:00:01)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 39 key (14547ROOT_BACKREF256) itemoff 10161 itemsize 26
root backref key dirid 28827 sequence 2 name snapshot
item 40 key (14559ROOT_ITEM2019735) itemoff 9722 itemsize 439
generation 2025992root_dirid256 bytenr 75423231606784byte_limit0bytes_used6576078848last_snapshot2019735 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025992
uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
ctime 1758919659.690576855 (2025-09-26 23:47:39)
otime 1758920400.700930626 (2025-09-27 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 41 key (14559ROOT_BACKREF256) itemoff 9696 itemsize 26
root backref key dirid 28851 sequence 2 name snapshot
item 42 key (14583ROOT_ITEM2022097) itemoff 9257 itemsize 439
generation 2025994root_dirid256 bytenr 65812067024896byte_limit0bytes_used6557646848last_snapshot2022097 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025994
uuid 860f0635-f296-a04b-9fc5-bd85d480799c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
ctime 1759006819.37527411 (2025-09-28 00:00:19)
otime 1759006820.861182212 (2025-09-28 00:00:20)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 43 key (14583ROOT_BACKREF256) itemoff 9231 itemsize 26
root backref key dirid 28899 sequence 2 name snapshot
item 44 key (14607ROOT_ITEM2024585) itemoff 8792 itemsize 439
generation 2025996root_dirid256 bytenr 227573944696832byte_limit0bytes_used6583533568last_snapshot2024585 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025996
uuid af3524d0-8a9a-894c-b5b7-705b10114326
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
ctime 1759089642.387498362 (2025-09-28 23:00:42)
otime 1759093200.412309185 (2025-09-29 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 45 key (14607ROOT_BACKREF256) itemoff 8766 itemsize 26
root backref key dirid 28947 sequence 2 name snapshot
item 46 key (14631ROOT_ITEM2025813) itemoff 8327 itemsize 439
generation 2025998root_dirid256 bytenr 178141351657472byte_limit0bytes_used6602981376last_snapshot2025813 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025998
uuid 6f085114-93b0-d146-8997-9a41ff62f8db
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
ctime 1759177902.175859896 (2025-09-29 23:31:42)
otime 1759210423.120386595 (2025-09-30 08:33:43)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 47 key (14631ROOT_BACKREF256) itemoff 8301 itemsize 26
root backref key dirid 28995 sequence 2 name snapshot
item 48 key (14640ROOT_ITEM2025865) itemoff 7862 itemsize 439
generation 2026000root_dirid256 bytenr 194611771719680byte_limit0bytes_used6603735040last_snapshot2025865 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026000
uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
ctime 1759237252.139079795 (2025-09-30 16:00:52)
otime 1759240800.56609287 (2025-09-30 17:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 49 key (14640ROOT_BACKREF256) itemoff 7836 itemsize 26
root backref key dirid 29013 sequence 2 name snapshot
item 50 key (14641ROOT_ITEM2025870) itemoff 7397 itemsize 439
generation 2026002root_dirid256 bytenr 65801342844928byte_limit0bytes_used6603292672last_snapshot2025870 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026002
uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
ctime 1759241717.101400473 (2025-09-30 17:15:17)
otime 1759244400.232224532 (2025-09-30 18:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 51 key (14641ROOT_BACKREF256) itemoff 7371 itemsize 26
root backref key dirid 29015 sequence 2 name snapshot
item 52 key (14642ROOT_ITEM2025874) itemoff 6932 itemsize 439
generation 2026004root_dirid256 bytenr 187252895498240byte_limit0bytes_used6603341824last_snapshot2025874 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026004
uuid f1761dfd-ec60-724a-9af0-dea952cc6771
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
ctime 1759244441.841998010 (2025-09-30 18:00:41)
otime 1759248000.122051877 (2025-09-30 19:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 53 key (14642ROOT_BACKREF256) itemoff 6906 itemsize 26
root backref key dirid 29017 sequence 2 name snapshot
item 54 key (14643ROOT_ITEM2025904) itemoff 6467 itemsize 439
generation 2026006root_dirid256 bytenr 227587769647104byte_limit0bytes_used6603587584last_snapshot2025904 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026006
uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
ctime 1759251597.815452999 (2025-09-30 19:59:57)
otime 1759251600.884521740 (2025-09-30 20:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 55 key (14643ROOT_BACKREF256) itemoff 6441 itemsize 26
root backref key dirid 29019 sequence 2 name snapshot
item 56 key (14644ROOT_ITEM2025922) itemoff 6002 itemsize 439
generation 2026008root_dirid256 bytenr 227594191208448byte_limit0bytes_used6603653120last_snapshot2025922 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026008
uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
ctime 1759254975.20573006 (2025-09-30 20:56:15)
otime 1759255200.196692319 (2025-09-30 21:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 57 key (14644ROOT_BACKREF256) itemoff 5976 itemsize 26
root backref key dirid 29021 sequence 2 name snapshot
item 58 key (14645ROOT_ITEM2025951) itemoff 5537 itemsize 439
generation 2026010root_dirid256 bytenr 193309358817280byte_limit0bytes_used6603964416last_snapshot2025951 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026010
uuid b46ce558-2c39-3944-91d4-b2b229f04a16
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
ctime 1759257514.988583086 (2025-09-30 21:38:34)
otime 1759258800.482340435 (2025-09-30 22:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 59 key (14645ROOT_BACKREF256) itemoff 5511 itemsize 26
root backref key dirid 29023 sequence 2 name snapshot
item 60 key (DATA_RELOC_TREEROOT_ITEM0) itemoff 5072 itemsize 439
generation 2018764root_dirid256 bytenr 71399055278080byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22018764Quoted 8 lines
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000received_uuid00000000-0000-0000-0000-000000000000 ctransid 0 otransid 0 stransid 0 rtransid 0 ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
Quoted 204 lines · Henri Hyyryläinen
Quoted 8 lines
Quoted 6 lines
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Is there a way for me to check that without mounting the filesystem? As far as I could find, none of the balance / scrub commands allow working on an unmounted filesystem. So I couldn't find out if I had any in canceled state.
# btrfs ins dump-tree -t root <device>Quoted 190 lines
Though, I'm pretty sure I let the last scrub and balance operation I tried to fully complete before starting using the check and repair commands. But I'm not absolutely certain that I didn't try one of those a last time and didn't let it fully complete.
I'll start that "btrfs check --readonly" command now. And I'll report back once it is done (hopefully by the morning in my timezone).
- Henri Hyyryläinen
Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
Quoted 182 lines · Qu Wenruo
Quoted 16 lines · Henri Hyyryläinen
Quoted 10 lines
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check --repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines · Henri Hyyryläinen
Quoted 11 lines
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs- progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Quoted 8 lines · Henri Hyyryläinen
Hello,
The check has completed and, probably thanks to the second repair, it did not find much. Here's the full output:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk.
Great news, we only need to delete that item.
Try btrfs-corrupt-block (I know this sounds dangerous) command, which may not be included by a lot of distros, thus recommended to build a btrfs-progs and run it (without installing)
# ./btrfs-corrupt-block -d 4,204,19977638903808 -r 4 <device>
If it worked, there should be no message output. Otherwise something went wrong.
Normally even if the command failed, it shouldn't cause extra damages since everything else in your fs is still fine.
Thanks,
Qu
Quoted 635 lines
Quoted 8 lines
[4/8] checking free space cache
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data) [7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS) Opening filesystem to check...
Checking filesystem on/dev/sdb
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98bQuoted 8 lines
found 49454062784512 bytes used, no error found
total csum bytes: 48225005724
total tree bytes: 72333164544
total fs tree bytes: 19189743616
total extent tree bytes: 3319316480
btree space waste bytes: 5998261217
file data blocks allocated: 119737771229184
referenced 72125752061952- Henri Hyyryläinen
Qu Wenruo kirjoitti 5.10.2025 klo 1.41:
Quoted 614 lines · Henri Hyyryläinen
Quoted 3 lines
Thanks for the command.
I tried reading through the data but I could not determine if there was any information related to scrub or balance status.
It's fine. No persistent item for balance/dev-replace. So it's not involved.
Waiting for the readonly check result.
And meanwhile, please prepare an environment to compile btrfs-progs.
If there is only one stray dev extent, I'll create a simple dirty fix so that you can mount the fs again.
Thanks,
QuQuoted 595 lines
Here's the full output I got:
Quoted 381 lines
btrfs-progs v6.16.1
root tree
leaf 227587769614336 items 61 free space 3547 generation 2035509 ownerROOT_TREEleaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
item 0 key (EXTENT_TREEROOT_ITEM0) itemoff 15844 itemsize 439
generation 2035509root_dirid0 bytenr 227587769630720byte_limit0bytes_used3319316480last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22035509
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 1 key (DEV_TREEROOT_ITEM0) itemoff 15405 itemsize 439
generation 2025974root_dirid0 bytenr 227594196631552byte_limit0bytes_used21889024last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 2generation_v22025974
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 2 key (FS_TREEINODE_REF6) itemoff 15388 itemsize 17
index 0 namelen 7 name: default
item 3 key (FS_TREEROOT_ITEM0) itemoff 14949 itemsize 439
generation 2025975root_dirid256 bytenr 227573939142656byte_limit0bytes_used6603997184last_snapshot2025951 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025975
uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025953 otransid 0 stransid 0 rtransid 0
ctime 1759258850.523837328 (2025-09-30 22:00:50)
otime 1679586168.0 (2023-03-23 17:42:48)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 4 key (FS_TREEROOT_REF256) itemoff 14921 itemsize 28
root ref key dirid 256 sequence 6 name .snapshots
item 5 key (ROOT_TREE_DIRINODE_ITEM0) itemoff 14761 itemsize 160
generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1679586169.0 (2023-03-23 17:42:49)
ctime 1679586169.0 (2023-03-23 17:42:49)
mtime 1679586169.0 (2023-03-23 17:42:49)
otime 1679586169.0 (2023-03-23 17:42:49)
item 6 key (ROOT_TREE_DIRINODE_REF6) itemoff 14749 itemsize 12
index 0 namelen 2 name: ..
item 7 key (ROOT_TREE_DIRDIR_ITEM2378154706) itemoff 14712 itemsize 37 location key (FS_TREEROOT_ITEM18446744073709551615) type DIR
transid 0data_len0name_len7
name: default
item 8 key (CSUM_TREEROOT_ITEM0) itemoff 14273 itemsize 439
generation 2025954root_dirid0 bytenr 227588044062720byte_limit0bytes_used49791238144last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 9 key (UUID_TREEROOT_ITEM0) itemoff 13834 itemsize 439
generation 2025954root_dirid0 bytenr 227587802447872byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 10 key (256ROOT_ITEM0) itemoff 13395 itemsize 439
generation 2025954root_dirid256 bytenr 227587802316800byte_limit0bytes_used49152last_snapshot2015386 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 1generation_v22025954
uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025954 otransid 31 stransid 0 rtransid 0
ctime 1759260258.238143629 (2025-09-30 22:24:18)
otime 1679588253.995834574 (2023-03-23 18:17:33)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 11 key (256ROOT_BACKREF5) itemoff 13367 itemsize 28
root backref key dirid 256 sequence 6 name .snapshots
item 12 key (256ROOT_REF14428) itemoff 13341 itemsize 26
root ref key dirid 28589 sequence 2 name snapshot
item 13 key (256ROOT_REF14451) itemoff 13315 itemsize 26
root ref key dirid 28635 sequence 2 name snapshot
item 14 key (256ROOT_REF14475) itemoff 13289 itemsize 26
root ref key dirid 28683 sequence 2 name snapshot
item 15 key (256ROOT_REF14499) itemoff 13263 itemsize 26
root ref key dirid 28731 sequence 2 name snapshot
item 16 key (256ROOT_REF14523) itemoff 13237 itemsize 26
root ref key dirid 28779 sequence 2 name snapshot
item 17 key (256ROOT_REF14547) itemoff 13211 itemsize 26
root ref key dirid 28827 sequence 2 name snapshot
item 18 key (256ROOT_REF14559) itemoff 13185 itemsize 26
root ref key dirid 28851 sequence 2 name snapshot
item 19 key (256ROOT_REF14583) itemoff 13159 itemsize 26
root ref key dirid 28899 sequence 2 name snapshot
item 20 key (256ROOT_REF14607) itemoff 13133 itemsize 26
root ref key dirid 28947 sequence 2 name snapshot
item 21 key (256ROOT_REF14631) itemoff 13107 itemsize 26
root ref key dirid 28995 sequence 2 name snapshot
item 22 key (256ROOT_REF14640) itemoff 13081 itemsize 26
root ref key dirid 29013 sequence 2 name snapshot
item 23 key (256ROOT_REF14641) itemoff 13055 itemsize 26
root ref key dirid 29015 sequence 2 name snapshot
item 24 key (256ROOT_REF14642) itemoff 13029 itemsize 26
root ref key dirid 29017 sequence 2 name snapshot
item 25 key (256ROOT_REF14643) itemoff 13003 itemsize 26
root ref key dirid 29019 sequence 2 name snapshot
item 26 key (256ROOT_REF14644) itemoff 12977 itemsize 26
root ref key dirid 29021 sequence 2 name snapshot
item 27 key (256ROOT_REF14645) itemoff 12951 itemsize 26
root ref key dirid 29023 sequence 2 name snapshot
item 28 key (14428ROOT_ITEM1849346) itemoff 12512 itemsize 439
generation 2025978root_dirid256 bytenr 71398781173760byte_limit0bytes_used8815476736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025978
uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
ctime 1758406304.841464812 (2025-09-21 01:11:44)
otime 1758406304.840394467 (2025-09-21 01:11:44)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 29 key (14428ROOT_BACKREF256) itemoff 12486 itemsize 26
root backref key dirid 28589 sequence 2 name snapshot
item 30 key (14451ROOT_ITEM1899154) itemoff 12047 itemsize 439
generation 2025981root_dirid256 bytenr 71398809468928byte_limit0bytes_used6747799552last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025981
uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
ctime 1758484800.817458724 (2025-09-21 23:00:00)
otime 1758488400.101635440 (2025-09-22 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 31 key (14451ROOT_BACKREF256) itemoff 12021 itemsize 26
root backref key dirid 28635 sequence 2 name snapshot
item 32 key (14475ROOT_ITEM1942246) itemoff 11582 itemsize 439
generation 2025984root_dirid256 bytenr 71398831259648byte_limit0bytes_used6755549184last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025984
uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
ctime 1758570000.151997903 (2025-09-22 22:40:00)
otime 1758574800.777676964 (2025-09-23 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 33 key (14475ROOT_BACKREF256) itemoff 11556 itemsize 26
root backref key dirid 28683 sequence 2 name snapshot
item 34 key (14499ROOT_ITEM1977241) itemoff 11117 itemsize 439
generation 2025986root_dirid256 bytenr 71398873153536byte_limit0bytes_used6538182656last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025986
uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
ctime 1758661201.518446292 (2025-09-24 00:00:01)
otime 1758661204.161207119 (2025-09-24 00:00:04)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 35 key (14499ROOT_BACKREF256) itemoff 11091 itemsize 26
root backref key dirid 28731 sequence 2 name snapshot
item 36 key (14523ROOT_ITEM2001507) itemoff 10652 itemsize 439
generation 2025988root_dirid256 bytenr 71398873497600byte_limit0bytes_used6561398784last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025988
uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
ctime 1758747600.712874426 (2025-09-25 00:00:00)
otime 1758747600.840460363 (2025-09-25 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 37 key (14523ROOT_BACKREF256) itemoff 10626 itemsize 26
root backref key dirid 28779 sequence 2 name snapshot
item 38 key (14547ROOT_ITEM2017206) itemoff 10187 itemsize 439
generation 2025990root_dirid256 bytenr 71398872924160byte_limit0bytes_used6570868736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025990
uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
ctime 1758834001.88320469 (2025-09-26 00:00:01)
otime 1758834001.271433540 (2025-09-26 00:00:01)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 39 key (14547ROOT_BACKREF256) itemoff 10161 itemsize 26
root backref key dirid 28827 sequence 2 name snapshot
item 40 key (14559ROOT_ITEM2019735) itemoff 9722 itemsize 439
generation 2025992root_dirid256 bytenr 75423231606784byte_limit0bytes_used6576078848last_snapshot2019735 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025992
uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
ctime 1758919659.690576855 (2025-09-26 23:47:39)
otime 1758920400.700930626 (2025-09-27 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 41 key (14559ROOT_BACKREF256) itemoff 9696 itemsize 26
root backref key dirid 28851 sequence 2 name snapshot
item 42 key (14583ROOT_ITEM2022097) itemoff 9257 itemsize 439
generation 2025994root_dirid256 bytenr 65812067024896byte_limit0bytes_used6557646848last_snapshot2022097 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025994
uuid 860f0635-f296-a04b-9fc5-bd85d480799c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
ctime 1759006819.37527411 (2025-09-28 00:00:19)
otime 1759006820.861182212 (2025-09-28 00:00:20)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 43 key (14583ROOT_BACKREF256) itemoff 9231 itemsize 26
root backref key dirid 28899 sequence 2 name snapshot
item 44 key (14607ROOT_ITEM2024585) itemoff 8792 itemsize 439
generation 2025996root_dirid256 bytenr 227573944696832byte_limit0bytes_used6583533568last_snapshot2024585 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025996
uuid af3524d0-8a9a-894c-b5b7-705b10114326
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
ctime 1759089642.387498362 (2025-09-28 23:00:42)
otime 1759093200.412309185 (2025-09-29 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 45 key (14607ROOT_BACKREF256) itemoff 8766 itemsize 26
root backref key dirid 28947 sequence 2 name snapshot
item 46 key (14631ROOT_ITEM2025813) itemoff 8327 itemsize 439
generation 2025998root_dirid256 bytenr 178141351657472byte_limit0bytes_used6602981376last_snapshot2025813 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025998
uuid 6f085114-93b0-d146-8997-9a41ff62f8db
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
ctime 1759177902.175859896 (2025-09-29 23:31:42)
otime 1759210423.120386595 (2025-09-30 08:33:43)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 47 key (14631ROOT_BACKREF256) itemoff 8301 itemsize 26
root backref key dirid 28995 sequence 2 name snapshot
item 48 key (14640ROOT_ITEM2025865) itemoff 7862 itemsize 439
generation 2026000root_dirid256 bytenr 194611771719680byte_limit0bytes_used6603735040last_snapshot2025865 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026000
uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
ctime 1759237252.139079795 (2025-09-30 16:00:52)
otime 1759240800.56609287 (2025-09-30 17:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 49 key (14640ROOT_BACKREF256) itemoff 7836 itemsize 26
root backref key dirid 29013 sequence 2 name snapshot
item 50 key (14641ROOT_ITEM2025870) itemoff 7397 itemsize 439
generation 2026002root_dirid256 bytenr 65801342844928byte_limit0bytes_used6603292672last_snapshot2025870 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026002
uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
ctime 1759241717.101400473 (2025-09-30 17:15:17)
otime 1759244400.232224532 (2025-09-30 18:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 51 key (14641ROOT_BACKREF256) itemoff 7371 itemsize 26
root backref key dirid 29015 sequence 2 name snapshot
item 52 key (14642ROOT_ITEM2025874) itemoff 6932 itemsize 439
generation 2026004root_dirid256 bytenr 187252895498240byte_limit0bytes_used6603341824last_snapshot2025874 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026004
uuid f1761dfd-ec60-724a-9af0-dea952cc6771
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
ctime 1759244441.841998010 (2025-09-30 18:00:41)
otime 1759248000.122051877 (2025-09-30 19:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 53 key (14642ROOT_BACKREF256) itemoff 6906 itemsize 26
root backref key dirid 29017 sequence 2 name snapshot
item 54 key (14643ROOT_ITEM2025904) itemoff 6467 itemsize 439
generation 2026006root_dirid256 bytenr 227587769647104byte_limit0bytes_used6603587584last_snapshot2025904 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026006
uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
ctime 1759251597.815452999 (2025-09-30 19:59:57)
otime 1759251600.884521740 (2025-09-30 20:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 55 key (14643ROOT_BACKREF256) itemoff 6441 itemsize 26
root backref key dirid 29019 sequence 2 name snapshot
item 56 key (14644ROOT_ITEM2025922) itemoff 6002 itemsize 439
generation 2026008root_dirid256 bytenr 227594191208448byte_limit0bytes_used6603653120last_snapshot2025922 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026008
uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
ctime 1759254975.20573006 (2025-09-30 20:56:15)
otime 1759255200.196692319 (2025-09-30 21:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 57 key (14644ROOT_BACKREF256) itemoff 5976 itemsize 26
root backref key dirid 29021 sequence 2 name snapshot
item 58 key (14645ROOT_ITEM2025951) itemoff 5537 itemsize 439
generation 2026010root_dirid256 bytenr 193309358817280byte_limit0bytes_used6603964416last_snapshot2025951 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026010
uuid b46ce558-2c39-3944-91d4-b2b229f04a16
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
ctime 1759257514.988583086 (2025-09-30 21:38:34)
otime 1759258800.482340435 (2025-09-30 22:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 59 key (14645ROOT_BACKREF256) itemoff 5511 itemsize 26
root backref key dirid 29023 sequence 2 name snapshot
item 60 key (DATA_RELOC_TREEROOT_ITEM0) itemoff 5072 itemsize 439
generation 2018764root_dirid256 bytenr 71399055278080byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22018764Quoted 8 lines
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000received_uuid00000000-0000-0000-0000-000000000000 ctransid 0 otransid 0 stransid 0 rtransid 0 ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
Quoted 200 lines · Henri Hyyryläinen
Quoted 6 lines
Quoted 4 lines
One extra thing, at the time of the initial "btrfs check -- repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Is there a way for me to check that without mounting the filesystem? As far as I could find, none of the balance / scrub commands allow working on an unmounted filesystem. So I couldn't find out if I had any in canceled state.
# btrfs ins dump-tree -t root <device>Quoted 188 lines
Though, I'm pretty sure I let the last scrub and balance operation I tried to fully complete before starting using the check and repair commands. But I'm not absolutely certain that I didn't try one of those a last time and didn't let it fully complete.
I'll start that "btrfs check --readonly" command now. And I'll report back once it is done (hopefully by the morning in my timezone).
- Henri Hyyryläinen
Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
Quoted 180 lines · Qu Wenruo
Quoted 16 lines · Henri Hyyryläinen
Quoted 10 lines
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check -- repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines
Quoted 11 lines · Henri Hyyryläinen
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs- progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen
Thank you so much. After compiling that program and running it like that, I can now mount my filesystem.
It took a little bit to mount due to free space cache rebuild, but as far as I can tell from a quick look, everything is fine and all the used disk sizes are correct.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 5.10.2025 klo 11.37:
Quoted 662 lines · Henri Hyyryläinen
Quoted 8 lines
Hello,
The check has completed and, probably thanks to the second repair, it did not find much. Here's the full output:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk.Great news, we only need to delete that item.
Try btrfs-corrupt-block (I know this sounds dangerous) command, which may not be included by a lot of distros, thus recommended to build a btrfs-progs and run it (without installing)
# ./btrfs-corrupt-block -d 4,204,19977638903808 -r 4 <device>If it worked, there should be no message output. Otherwise something went wrong.
Normally even if the command failed, it shouldn't cause extra damages since everything else in your fs is still fine.
Thanks,
QuQuoted 635 lines
Quoted 8 lines
[4/8] checking free space cache
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data) [7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS) Opening filesystem to check...
Checking filesystem on/dev/sdb
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98bQuoted 8 lines
found 49454062784512 bytes used, no error found
total csum bytes: 48225005724
total tree bytes: 72333164544
total fs tree bytes: 19189743616
total extent tree bytes: 3319316480
btree space waste bytes: 5998261217
file data blocks allocated: 119737771229184
referenced 72125752061952- Henri Hyyryläinen
Qu Wenruo kirjoitti 5.10.2025 klo 1.41:
Quoted 614 lines · Henri Hyyryläinen
Quoted 3 lines
Thanks for the command.
I tried reading through the data but I could not determine if there was any information related to scrub or balance status.
It's fine. No persistent item for balance/dev-replace. So it's not involved.
Waiting for the readonly check result.
And meanwhile, please prepare an environment to compile btrfs-progs.
If there is only one stray dev extent, I'll create a simple dirty fix so that you can mount the fs again.
Thanks,
QuQuoted 595 lines
Here's the full output I got:
Quoted 381 lines
btrfs-progs v6.16.1
root tree
leaf 227587769614336 items 61 free space 3547 generation 2035509 ownerROOT_TREEleaf 227587769614336 flags 0x1(WRITTEN) backref revision 1
fs uuid 2b4ad16d-e456-4adf-960b-dca43560b98b
chunk uuid 0e317678-73cc-4485-ab6b-987285e19681
item 0 key (EXTENT_TREEROOT_ITEM0) itemoff 15844 itemsize 439
generation 2035509root_dirid0 bytenr 227587769630720byte_limit0bytes_used3319316480last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22035509
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 1 key (DEV_TREEROOT_ITEM0) itemoff 15405 itemsize 439
generation 2025974root_dirid0 bytenr 227594196631552byte_limit0bytes_used21889024last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 2generation_v22025974
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 2 key (FS_TREEINODE_REF6) itemoff 15388 itemsize 17
index 0 namelen 7 name: default
item 3 key (FS_TREEROOT_ITEM0) itemoff 14949 itemsize 439
generation 2025975root_dirid256 bytenr 227573939142656byte_limit0bytes_used6603997184last_snapshot2025951 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025975
uuid eea49a45-adcf-4bff-a203-0f600f2db6a8
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025953 otransid 0 stransid 0 rtransid 0
ctime 1759258850.523837328 (2025-09-30 22:00:50)
otime 1679586168.0 (2023-03-23 17:42:48)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 4 key (FS_TREEROOT_REF256) itemoff 14921 itemsize 28
root ref key dirid 256 sequence 6 name .snapshots
item 5 key (ROOT_TREE_DIRINODE_ITEM0) itemoff 14761 itemsize 160
generation 3 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1679586169.0 (2023-03-23 17:42:49)
ctime 1679586169.0 (2023-03-23 17:42:49)
mtime 1679586169.0 (2023-03-23 17:42:49)
otime 1679586169.0 (2023-03-23 17:42:49)
item 6 key (ROOT_TREE_DIRINODE_REF6) itemoff 14749 itemsize 12
index 0 namelen 2 name: ..
item 7 key (ROOT_TREE_DIRDIR_ITEM2378154706) itemoff 14712 itemsize 37 location key (FS_TREEROOT_ITEM18446744073709551615) type DIR
transid 0data_len0name_len7
name: default
item 8 key (CSUM_TREEROOT_ITEM0) itemoff 14273 itemsize 439
generation 2025954root_dirid0 bytenr 227588044062720byte_limit0bytes_used49791238144last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 9 key (UUID_TREEROOT_ITEM0) itemoff 13834 itemsize 439
generation 2025954root_dirid0 bytenr 227587802447872byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22025954
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 0 otransid 0 stransid 0 rtransid 0
ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 10 key (256ROOT_ITEM0) itemoff 13395 itemsize 439
generation 2025954root_dirid256 bytenr 227587802316800byte_limit0bytes_used49152last_snapshot2015386 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 1generation_v22025954
uuid 56f85f76-1eb8-1245-a35b-8a9307d5c68f
parent_uuid00000000-0000-0000-0000-000000000000
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025954 otransid 31 stransid 0 rtransid 0
ctime 1759260258.238143629 (2025-09-30 22:24:18)
otime 1679588253.995834574 (2023-03-23 18:17:33)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 11 key (256ROOT_BACKREF5) itemoff 13367 itemsize 28
root backref key dirid 256 sequence 6 name .snapshots
item 12 key (256ROOT_REF14428) itemoff 13341 itemsize 26
root ref key dirid 28589 sequence 2 name snapshot
item 13 key (256ROOT_REF14451) itemoff 13315 itemsize 26
root ref key dirid 28635 sequence 2 name snapshot
item 14 key (256ROOT_REF14475) itemoff 13289 itemsize 26
root ref key dirid 28683 sequence 2 name snapshot
item 15 key (256ROOT_REF14499) itemoff 13263 itemsize 26
root ref key dirid 28731 sequence 2 name snapshot
item 16 key (256ROOT_REF14523) itemoff 13237 itemsize 26
root ref key dirid 28779 sequence 2 name snapshot
item 17 key (256ROOT_REF14547) itemoff 13211 itemsize 26
root ref key dirid 28827 sequence 2 name snapshot
item 18 key (256ROOT_REF14559) itemoff 13185 itemsize 26
root ref key dirid 28851 sequence 2 name snapshot
item 19 key (256ROOT_REF14583) itemoff 13159 itemsize 26
root ref key dirid 28899 sequence 2 name snapshot
item 20 key (256ROOT_REF14607) itemoff 13133 itemsize 26
root ref key dirid 28947 sequence 2 name snapshot
item 21 key (256ROOT_REF14631) itemoff 13107 itemsize 26
root ref key dirid 28995 sequence 2 name snapshot
item 22 key (256ROOT_REF14640) itemoff 13081 itemsize 26
root ref key dirid 29013 sequence 2 name snapshot
item 23 key (256ROOT_REF14641) itemoff 13055 itemsize 26
root ref key dirid 29015 sequence 2 name snapshot
item 24 key (256ROOT_REF14642) itemoff 13029 itemsize 26
root ref key dirid 29017 sequence 2 name snapshot
item 25 key (256ROOT_REF14643) itemoff 13003 itemsize 26
root ref key dirid 29019 sequence 2 name snapshot
item 26 key (256ROOT_REF14644) itemoff 12977 itemsize 26
root ref key dirid 29021 sequence 2 name snapshot
item 27 key (256ROOT_REF14645) itemoff 12951 itemsize 26
root ref key dirid 29023 sequence 2 name snapshot
item 28 key (14428ROOT_ITEM1849346) itemoff 12512 itemsize 439
generation 2025978root_dirid256 bytenr 71398781173760byte_limit0bytes_used8815476736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025978
uuid 97cd122d-c519-8746-bdfe-13479bfefb9d
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1849346 otransid 1849346 stransid 0 rtransid 0
ctime 1758406304.841464812 (2025-09-21 01:11:44)
otime 1758406304.840394467 (2025-09-21 01:11:44)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 29 key (14428ROOT_BACKREF256) itemoff 12486 itemsize 26
root backref key dirid 28589 sequence 2 name snapshot
item 30 key (14451ROOT_ITEM1899154) itemoff 12047 itemsize 439
generation 2025981root_dirid256 bytenr 71398809468928byte_limit0bytes_used6747799552last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025981
uuid c5e2ba38-c982-8a4f-99e5-bb9ff8c5f7cd
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1897416 otransid 1899154 stransid 0 rtransid 0
ctime 1758484800.817458724 (2025-09-21 23:00:00)
otime 1758488400.101635440 (2025-09-22 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 31 key (14451ROOT_BACKREF256) itemoff 12021 itemsize 26
root backref key dirid 28635 sequence 2 name snapshot
item 32 key (14475ROOT_ITEM1942246) itemoff 11582 itemsize 439
generation 2025984root_dirid256 bytenr 71398831259648byte_limit0bytes_used6755549184last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025984
uuid 38b9ea09-54ef-5140-bddc-83dc884c31b3
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1940067 otransid 1942246 stransid 0 rtransid 0
ctime 1758570000.151997903 (2025-09-22 22:40:00)
otime 1758574800.777676964 (2025-09-23 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 33 key (14475ROOT_BACKREF256) itemoff 11556 itemsize 26
root backref key dirid 28683 sequence 2 name snapshot
item 34 key (14499ROOT_ITEM1977241) itemoff 11117 itemsize 439
generation 2025986root_dirid256 bytenr 71398873153536byte_limit0bytes_used6538182656last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025986
uuid 703f0298-831b-1e4d-b1f1-fc37ff582770
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 1977241 otransid 1977241 stransid 0 rtransid 0
ctime 1758661201.518446292 (2025-09-24 00:00:01)
otime 1758661204.161207119 (2025-09-24 00:00:04)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 35 key (14499ROOT_BACKREF256) itemoff 11091 itemsize 26
root backref key dirid 28731 sequence 2 name snapshot
item 36 key (14523ROOT_ITEM2001507) itemoff 10652 itemsize 439
generation 2025988root_dirid256 bytenr 71398873497600byte_limit0bytes_used6561398784last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025988
uuid 9672abd0-7e7b-ab4f-9e0f-7183e58bb6c4
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2001507 otransid 2001507 stransid 0 rtransid 0
ctime 1758747600.712874426 (2025-09-25 00:00:00)
otime 1758747600.840460363 (2025-09-25 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 37 key (14523ROOT_BACKREF256) itemoff 10626 itemsize 26
root backref key dirid 28779 sequence 2 name snapshot
item 38 key (14547ROOT_ITEM2017206) itemoff 10187 itemsize 439
generation 2025990root_dirid256 bytenr 71398872924160byte_limit0bytes_used6570868736last_snapshot2018745 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025990
uuid 671b4b4f-fd99-e140-8535-fce0a16babfb
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2017206 otransid 2017206 stransid 0 rtransid 0
ctime 1758834001.88320469 (2025-09-26 00:00:01)
otime 1758834001.271433540 (2025-09-26 00:00:01)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 39 key (14547ROOT_BACKREF256) itemoff 10161 itemsize 26
root backref key dirid 28827 sequence 2 name snapshot
item 40 key (14559ROOT_ITEM2019735) itemoff 9722 itemsize 439
generation 2025992root_dirid256 bytenr 75423231606784byte_limit0bytes_used6576078848last_snapshot2019735 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025992
uuid 3ff52125-57ee-6140-87a5-4fb848ec4638
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2019711 otransid 2019735 stransid 0 rtransid 0
ctime 1758919659.690576855 (2025-09-26 23:47:39)
otime 1758920400.700930626 (2025-09-27 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 41 key (14559ROOT_BACKREF256) itemoff 9696 itemsize 26
root backref key dirid 28851 sequence 2 name snapshot
item 42 key (14583ROOT_ITEM2022097) itemoff 9257 itemsize 439
generation 2025994root_dirid256 bytenr 65812067024896byte_limit0bytes_used6557646848last_snapshot2022097 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025994
uuid 860f0635-f296-a04b-9fc5-bd85d480799c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2022097 otransid 2022097 stransid 0 rtransid 0
ctime 1759006819.37527411 (2025-09-28 00:00:19)
otime 1759006820.861182212 (2025-09-28 00:00:20)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 43 key (14583ROOT_BACKREF256) itemoff 9231 itemsize 26
root backref key dirid 28899 sequence 2 name snapshot
item 44 key (14607ROOT_ITEM2024585) itemoff 8792 itemsize 439
generation 2025996root_dirid256 bytenr 227573944696832byte_limit0bytes_used6583533568last_snapshot2024585 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025996
uuid af3524d0-8a9a-894c-b5b7-705b10114326
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2024474 otransid 2024585 stransid 0 rtransid 0
ctime 1759089642.387498362 (2025-09-28 23:00:42)
otime 1759093200.412309185 (2025-09-29 00:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 45 key (14607ROOT_BACKREF256) itemoff 8766 itemsize 26
root backref key dirid 28947 sequence 2 name snapshot
item 46 key (14631ROOT_ITEM2025813) itemoff 8327 itemsize 439
generation 2025998root_dirid256 bytenr 178141351657472byte_limit0bytes_used6602981376last_snapshot2025813 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22025998
uuid 6f085114-93b0-d146-8997-9a41ff62f8db
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025812 otransid 2025813 stransid 0 rtransid 0
ctime 1759177902.175859896 (2025-09-29 23:31:42)
otime 1759210423.120386595 (2025-09-30 08:33:43)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 47 key (14631ROOT_BACKREF256) itemoff 8301 itemsize 26
root backref key dirid 28995 sequence 2 name snapshot
item 48 key (14640ROOT_ITEM2025865) itemoff 7862 itemsize 439
generation 2026000root_dirid256 bytenr 194611771719680byte_limit0bytes_used6603735040last_snapshot2025865 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026000
uuid a48f03d9-4847-654a-af1d-5e0eaf6ecd0c
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025863 otransid 2025865 stransid 0 rtransid 0
ctime 1759237252.139079795 (2025-09-30 16:00:52)
otime 1759240800.56609287 (2025-09-30 17:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 49 key (14640ROOT_BACKREF256) itemoff 7836 itemsize 26
root backref key dirid 29013 sequence 2 name snapshot
item 50 key (14641ROOT_ITEM2025870) itemoff 7397 itemsize 439
generation 2026002root_dirid256 bytenr 65801342844928byte_limit0bytes_used6603292672last_snapshot2025870 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026002
uuid 56e7aa4e-e166-504d-9a55-8c8b074c0fd9
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025868 otransid 2025870 stransid 0 rtransid 0
ctime 1759241717.101400473 (2025-09-30 17:15:17)
otime 1759244400.232224532 (2025-09-30 18:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 51 key (14641ROOT_BACKREF256) itemoff 7371 itemsize 26
root backref key dirid 29015 sequence 2 name snapshot
item 52 key (14642ROOT_ITEM2025874) itemoff 6932 itemsize 439
generation 2026004root_dirid256 bytenr 187252895498240byte_limit0bytes_used6603341824last_snapshot2025874 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026004
uuid f1761dfd-ec60-724a-9af0-dea952cc6771
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025872 otransid 2025874 stransid 0 rtransid 0
ctime 1759244441.841998010 (2025-09-30 18:00:41)
otime 1759248000.122051877 (2025-09-30 19:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 53 key (14642ROOT_BACKREF256) itemoff 6906 itemsize 26
root backref key dirid 29017 sequence 2 name snapshot
item 54 key (14643ROOT_ITEM2025904) itemoff 6467 itemsize 439
generation 2026006root_dirid256 bytenr 227587769647104byte_limit0bytes_used6603587584last_snapshot2025904 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026006
uuid 68fd5037-6c8a-c74e-b08f-cff4cdccf752
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025904 otransid 2025904 stransid 0 rtransid 0
ctime 1759251597.815452999 (2025-09-30 19:59:57)
otime 1759251600.884521740 (2025-09-30 20:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 55 key (14643ROOT_BACKREF256) itemoff 6441 itemsize 26
root backref key dirid 29019 sequence 2 name snapshot
item 56 key (14644ROOT_ITEM2025922) itemoff 6002 itemsize 439
generation 2026008root_dirid256 bytenr 227594191208448byte_limit0bytes_used6603653120last_snapshot2025922 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026008
uuid 07ccd2ae-f51d-2c46-82df-05e16964b2e5
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025921 otransid 2025922 stransid 0 rtransid 0
ctime 1759254975.20573006 (2025-09-30 20:56:15)
otime 1759255200.196692319 (2025-09-30 21:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 57 key (14644ROOT_BACKREF256) itemoff 5976 itemsize 26
root backref key dirid 29021 sequence 2 name snapshot
item 58 key (14645ROOT_ITEM2025951) itemoff 5537 itemsize 439
generation 2026010root_dirid256 bytenr 193309358817280byte_limit0bytes_used6603964416last_snapshot2025951 flags 0x1(RDONLY) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 3generation_v22026010
uuid b46ce558-2c39-3944-91d4-b2b229f04a16
parent_uuideea49a45-adcf-4bff-a203-0f600f2db6a8
received_uuid00000000-0000-0000-0000-000000000000
ctransid 2025950 otransid 2025951 stransid 0 rtransid 0
ctime 1759257514.988583086 (2025-09-30 21:38:34)
otime 1759258800.482340435 (2025-09-30 22:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)
item 59 key (14645ROOT_BACKREF256) itemoff 5511 itemsize 26
root backref key dirid 29023 sequence 2 name snapshot
item 60 key (DATA_RELOC_TREEROOT_ITEM0) itemoff 5072 itemsize 439
generation 2018764root_dirid256 bytenr 71399055278080byte_limit0bytes_used16384last_snapshot0 flags 0x0(none) refs 1
drop_progresskey (0 UNKNOWN.0 0)drop_level0
level 0generation_v22018764Quoted 8 lines
uuid 00000000-0000-0000-0000-000000000000
parent_uuid00000000-0000-0000-0000-000000000000received_uuid00000000-0000-0000-0000-000000000000 ctransid 0 otransid 0 stransid 0 rtransid 0 ctime 0.0 (1970-01-01 02:00:00)
otime 0.0 (1970-01-01 02:00:00)
stime 0.0 (1970-01-01 02:00:00)
rtime 0.0 (1970-01-01 02:00:00)Qu Wenruo kirjoitti 5.10.2025 klo 1.07:
Quoted 200 lines · Henri Hyyryläinen
Quoted 6 lines
Quoted 4 lines
One extra thing, at the time of the initial "btrfs check -- repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Is there a way for me to check that without mounting the filesystem? As far as I could find, none of the balance / scrub commands allow working on an unmounted filesystem. So I couldn't find out if I had any in canceled state.
# btrfs ins dump-tree -t root <device>Quoted 188 lines
Though, I'm pretty sure I let the last scrub and balance operation I tried to fully complete before starting using the check and repair commands. But I'm not absolutely certain that I didn't try one of those a last time and didn't let it fully complete.
I'll start that "btrfs check --readonly" command now. And I'll report back once it is done (hopefully by the morning in my timezone).
- Henri Hyyryläinen
Qu Wenruo kirjoitti 4.10.2025 klo 23.55:
Quoted 180 lines · Qu Wenruo
Quoted 16 lines · Henri Hyyryläinen
Quoted 10 lines
Hello again.
It took over 3 days, but the btrfs check --repair has now completed seemingly successfully. I mostly saw output about the file being placed in lost+found and and directory size being corrected. However, there were some messages about mismatch of used bytes.
Unfortunately it seems like the situation has gotten worse since the repair, because now I cannot mount the filesystem at all. Instead I get an error like this:
Quoted 3 lines
BTRFS error (device sdc): dev extent physical offset 19977638903808 on devid 4 doesn't have corresponding chunk BTRFS error (device sdc): failed to verify dev extents against chunks: -117
BTRFS error (device sdc):open_ctreefailed: -117Even if I remove that one problematic device physically from my computer, the filesystem still refuses to mount with the same error. Maybe the problems with the device replace are again showing up with the actual size of the hard drive not being used correctly? I cannot try to remove the device slack as I cannot mount the filesystem.
Nope, this is a different problem, and not related to dev replace.
Quoted 7 lines
Unfortunately btrfs check has not implemented any repair for that.
Overall if the dev extent is found but not corresponding chunk, it should still be fine but some space unavailable.
But the kernel is overly cautious on chunk tree, as it's a very important and basic functionality.
Please provide the full "btrfs check --readonly" output so that we can evaluate and add the missing repair functionality.
One extra thing, at the time of the initial "btrfs check -- repair", there is no running dev-replace/balance or whatever running?
Although those operations will be automatically paused by unmount, btrfs check is not going to be able to handle some of those paused operations.
Quoted 148 lines
Thanks,
QuQuoted 10 lines
I did try to run a repair again, and this time I got a bunch of messages like:
Quoted 7 lines
repair deleting extent record: key [65795546775552,169,0]
adding new tree backref on start 65795546775552 len 16384 parent 65811674234880 root 65811674234880 adding new tree backref on start 65795546775552 len 16384 parent 0 root 14499 adding new tree backref on start 65795546775552 len 16384 parent 65791012274176 root 65791012274176 adding new tree backref on start 65795546775552 len 16384 parent 65806385807360 root 65806385807360 Repaired extent references for 65795546775552
ref mismatch on [65795548233728 16384] extent item 5, found 4That's fixing some backref mismatch, which you can ignore unless "btrfs check --reaonly" later reports new problems.
Quoted 129 lines
But the filesystem still refuses to mount with the exact same error. I did not let the repair run entirely as it would have likely taken another 3 days. What should I do? This time I'm not finding any good information on what to do. For now, I've started the repair again, but it doesn't exactly sound like it is even fixing anything now. Still, I'll let it continue. The output so far is:
Quoted 4 lines
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extentsQuoted 8 lines
Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454738989056 mismatches actual used 49454738923520 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738956288 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739021824 mismatches actual used 49454738972672 Device extent[4, 19977638903808, 1073741824] didn't find the relative chunk. super bytes used 49454739005440 mismatches actual used 49454738972672
If I was able to somehow remove that one logically corrupt devid from the filesystem, or somehow correct the size, that would hopefully allow me to rebuild from the raid10 data then, but I can't do those with the unmountable filesystem.
- Henri Hyyryläinen
Qu Wenruo kirjoitti 30.9.2025 klo 0.52:
Quoted 107 lines
Quoted 11 lines · Henri Hyyryläinen
Hello,
I hope this is the right place to ask about a filesystem problem. Really shortly put, I have a file that both exists and doesn't and prevents the containing directory from being deleted. No matter what variant of rm and inode based deletion I try I get an error about the file not existing, and I also cannot try to read the file, but if I try to delete the directory I get an error that it is not empty (so the file kind of exists). Trying to ls the directory also gives a file doesn't exist error.
Here's what btrfs check found, which I hope does better in illustrating the problem:
Quoted 3 lines
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir itemI've tried everything I've found suggested including a full scrub, balance with -dusage=75 -musage=75, resetting file attributes, deleting through the find command, and even some repair mount flags that don't seem to exist for btrfs.
The fs is corrupted, thus none of those will help. I'm more interested in how the corruption happened.
Did you use some tools other than btrfs kernel module and btrfs- progs? Like ntfs2btrfs or winbtrfs?
IIRC certain versions have some bugs related to extent tree, but should not cause this problem.
The other possibility is hardware memory bitflip, which is more common than you thought (almostly one report per month)
In that case, a full memtest is always recommended, or you will hit all kinds of weird corruptions in the future anyway.
With a full memtest proving the memory hardware is fine, then "btrfs check --repair" should be able to fix it.
Thanks,
QuQuoted 71 lines
What I haven't tried is a full rebalance with no filters, but I did not try that yet as it would take quite a long time and if it only moves data blocks around without recomputing directory items, it doesn't seem like the right tool to fix my problem. So I'm pretty much stuck and to me it seems like my only option is to run btrfs check with the repair flag, but as that has big warnings on it I thought I would try asking here first (sorry if this is not the right experts group to ask). So is there still something I can try or am I finally "allowed" to use the repair command? Here's the full output I got from btrfs check:
Quoted 49 lines
Opening filesystem to check...
Checking filesystem on/dev/sdc
UUID: 2b4ad16d-e456-4adf-960b-dca43560b98b
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
root 5 inode 25953213 errors 200, dir isize wrong
root 5 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14428 inode 25953213 errors 200, dir isize wrong
root 14428 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14451 inode 25953213 errors 200, dir isize wrong
root 14451 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14475 inode 25953213 errors 200, dir isize wrong
root 14475 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14499 inode 25953213 errors 200, dir isize wrong
root 14499 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14523 inode 25953213 errors 200, dir isize wrong
root 14523 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14544 inode 25953213 errors 200, dir isize wrong
root 14544 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14545 inode 25953213 errors 200, dir isize wrong
root 14545 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14546 inode 25953213 errors 200, dir isize wrong
root 14546 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14547 inode 25953213 errors 200, dir isize wrong
root 14547 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14548 inode 25953213 errors 200, dir isize wrong
root 14548 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14549 inode 25953213 errors 200, dir isize wrong
root 14549 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item root 14550 inode 25953213 errors 200, dir isize wrong
root 14550 inode 27166085 errors 2000, link count wrong
unresolved ref dir 25953213 index 7 namelen 33 name Microsoft.AspNetCore.Metadata.dll filetype 1 errors 1, no dir item
ERROR: errors found in fs rootsQuoted 8 lines
found 49400296812544 bytes used, error(s) found total csum bytes: 48179330432
total tree bytes: 65067483136
total fs tree bytes: 12107431936
total extent tree bytes: 3194437632
btree space waste bytes: 4558984171
file data blocks allocated: 76487982252032 referenced 60030799097856So hopefully if I'm reading things right, running a repair would delete just that one file and directory (which itself is a backup so I will not miss that file at all)?
I do not have enough disk space to copy off the entire filesystem and rebuild from scratch, without doing something like rebalancing all data from raid10 to single and then removing half the disks, but I assume that would take at least 4 weeks to process (as I just replaced a disk which took like a week).
As to what originally caused the corruption, I think it was probably faulty RAM, because up to to like 3 weeks ago I had one really bad RAM stick in my computer where a certain memory region always had incorrectly reading bytes. I had seen intermittent quite high csum errors in monthly scrubs pretty randomly, which thankfully could almost always be corrected so I didn't have any major problems even though I had like totally broken RAM in my computer for who knows how long. So btrfs was able to protect my data quite impressively from bad RAM.
Sorry for getting a bit sidetracked there, but what should I do in this situation?
- Henri Hyyryläinen