BTRFS FS corruption
Kernel: Debian 5.10.70-1~bpo10+1 (2021-10-10)
# ls -al /mnt/var/lib/postgresql/12/main/pg_stat_tmp/ ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory total 80 drwx------ 1 1001 1001 94 Dec 29 13:52 . drwx------ 1 1001 1001 552 Dec 29 13:51 .. -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 1864 Dec 29 13:52 db_16391.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp
This is a dev/test vm so we can do intrusive operations on it...
Should I run btrfsck?
btrfs scrub did not show any errors/fixes and it did not fix the issue.
Quoted 73 lines · Mark Murawski
Kernel: Debian 5.10.70-1~bpo10+1 (2021-10-10)
# ls -al /mnt/var/lib/postgresql/12/main/pg_stat_tmp/ ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory total 80 drwx------ 1 1001 1001 94 Dec 29 13:52 . drwx------ 1 1001 1001 552 Dec 29 13:51 ..(quoted diff) +0
-38-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 1864 Dec 29 13:52 db_16391.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp
Quoted 1 line
This is a dev/test vm so we can do intrusive operations on it...
Dmesg please.
Quoted 2 lines
Should I run btrfsck?
Definitely.
Quoted 3 lines
btrfs scrub did not show any errors/fixes and it did not fix the issue.
Scrub won't check cross-ref, unlike btrfs-check.
Thanks,
Qu
Quoted 1 line · Mark Murawski
Kernel: Debian 5.10.70-1~bpo10+1 (2021-10-10)
I've been trying to reproduce the "ghost dirents" bug on 5.10 without success. It's much easier to hit on 5.14 and 5.15, but apparently not impossible on 5.10.
Quoted 50 lines
# ls -al /mnt/var/lib/postgresql/12/main/pg_stat_tmp/ ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory total 80 drwx------ 1 1001 1001 94 Dec 29 13:52 . drwx------ 1 1001 1001 552 Dec 29 13:51 ..(quoted diff) +0
-28-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 0 Dec 29 13:52 db_106964.stat -rw------- 1 1001 1001 1864 Dec 29 13:52 db_16391.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -rw------- 1 1001 1001 840 Dec 29 13:52 global.stat
Review
Quoted 10 lines
(quoted diff) +0
-10-rw------- 1 1001 1001 840 Dec 29 13:52 global.stat -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp -????????? ? ? ? ? ? global.tmp
This looks like the "ghost dirents" bug I've been chasing for a while.
Some prior discussion at
Quoted 2 lines
This is a dev/test vm so we can do intrusive operations on it...
If you can reproduce this reliably, can you apply a kernel patch:
v5.16-rc1: 9a35fc9542fa btrfs: change error handling for btrfs_delete_*_in_log
There's a link to the commit in the kernel.org thread.
Quoted 1 line
Should I run btrfsck?
Ghost dirents can be removed by the normal 'rm -rf' tool (though it will complain with errors). You can run btrfs check to confirm that this is the only problem on the filesystem.
Quoted 1 line
btrfs scrub did not show any errors/fixes and it did not fix the issue.
Scrub verifies metadata block csums, and there will be no errors in csums from the ghost dirent bug case, so scrub will not detect or report ghost dirents.