Skip to contentSkip to conversation

BTRFS FS corruption

DISCUSSION Mark Murawski 3 participants 3 messages Dec 29, 2021 – Dec 30, 2021 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Kernel: Debian 5.10.70-1~bpo10+1 (2021-10-10)

# ls -al /mnt/var/lib/postgresql/12/main/pg_stat_tmp/
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file or directory
total 80
drwx------ 1 1001 1001   94 Dec 29 13:52 .
drwx------ 1 1001 1001  552 Dec 29 13:51 ..
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001 1864 Dec 29 13:52 db_16391.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp

This is a dev/test vm so we can do intrusive operations on it...

Should I run btrfsck?

btrfs scrub did not show any errors/fixes and it did not fix the issue.

#1 lore raw
Quoted 73 lines · Mark Murawski

Kernel: Debian 5.10.70-1~bpo10+1 (2021-10-10)

# ls -al /mnt/var/lib/postgresql/12/main/pg_stat_tmp/
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
ls: cannot access
'/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp': No such file
or directory
total 80
drwx------ 1 1001 1001   94 Dec 29 13:52 .
drwx------ 1 1001 1001  552 Dec 29 13:51 ..
(quoted diff) +0 -38
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001 1864 Dec 29 13:52 db_16391.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
Quoted 1 line

This is a dev/test vm so we can do intrusive operations on it...

Dmesg please.

Quoted 2 lines

Should I run btrfsck?

Definitely.

Quoted 3 lines

btrfs scrub did not show any errors/fixes and it did not fix the issue.

Scrub won't check cross-ref, unlike btrfs-check.

Thanks,
Qu

#2 lore raw
In reply to Mark Murawski
Quoted 1 line · Mark Murawski

Kernel: Debian 5.10.70-1~bpo10+1 (2021-10-10)

I've been trying to reproduce the "ghost dirents" bug on 5.10 without success. It's much easier to hit on 5.14 and 5.15, but apparently not impossible on 5.10.

Quoted 50 lines
# ls -al /mnt/var/lib/postgresql/12/main/pg_stat_tmp/
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
ls: cannot access '/mnt/var/lib/postgresql/12/main/pg_stat_tmp/global.tmp':
No such file or directory
total 80
drwx------ 1 1001 1001   94 Dec 29 13:52 .
drwx------ 1 1001 1001  552 Dec 29 13:51 ..
(quoted diff) +0 -28
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001 1526 Dec 29 13:52 db_0.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001    0 Dec 29 13:52 db_106964.stat
-rw------- 1 1001 1001 1864 Dec 29 13:52 db_16391.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat

Review

Quoted 10 lines
(quoted diff) +0 -10
-rw------- 1 1001 1001  840 Dec 29 13:52 global.stat
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp
-????????? ? ?    ?       ?            ? global.tmp

This looks like the "ghost dirents" bug I've been chasing for a while.

Some prior discussion at

https://lore.kernel.org/linux-btrfs/CAL3q7H4CYtaW_aEQSEZ_KxZ_ba3u=FmPT8VtXH+OE6FTR8oxOQ@mail.gmail.com/

Quoted 2 lines

This is a dev/test vm so we can do intrusive operations on it...

If you can reproduce this reliably, can you apply a kernel patch:

	v5.16-rc1: 9a35fc9542fa btrfs: change error handling for btrfs_delete_*_in_log

There's a link to the commit in the kernel.org thread.

Quoted 1 line

Should I run btrfsck?

Ghost dirents can be removed by the normal 'rm -rf' tool (though it will complain with errors). You can run btrfs check to confirm that this is the only problem on the filesystem.

Quoted 1 line

btrfs scrub did not show any errors/fixes and it did not fix the issue.

Scrub verifies metadata block csums, and there will be no errors in csums from the ghost dirent bug case, so scrub will not detect or report ghost dirents.