Skip to contentSkip to conversation

3.12 vfs weirdness under load

BUG Jeff King 2 participants 4 messages Nov 13, 2013 – Nov 14, 2013 VFS

lore
Share & tools
lore

Jeff King

author
#0 lore raw

On v3.12, running "make -j16 test" on a git.git checkout produces random failures in the test suite, whereas it works fine on v3.11. Git's test suite puts a lot of load on the VFS; re-running failed tests individually when the system is unloaded lets them pass. I haven't been able to get a specific syscall strace of what's going on, but some of the failures I investigated had obvious oddness, like "grep foo bar" returning non-zero, but then looking in "bar" manually shows that it contains the expected content.

I bisected this to 232d2d6 (dcache: Translating dentry into pathname without taking rename_lock, 2013-09-09), and can reliably reproduce the problem (and running with the parent ac83871 reliably works).

I see the problem on my ext4-over-dmcrypt disk, but also when running the tests on a tmpfs disk. So I don't think it's filesystem-specific.

Reading the patch for 232d2d6, it seems plausible that it's a locking problem in the new code, but I don't see anything obviously wrong. I'd expect others can reproduce by running git's test suite, but if not, I'm happy to test patches.

-Peff

author
#1 lore raw
Quoted 7 lines · Jeff King

On v3.12, running "make -j16 test" on a git.git checkout produces random failures in the test suite, whereas it works fine on v3.11. [...]
Reading the patch for 232d2d6, it seems plausible that it's a locking problem in the new code, but I don't see anything obviously wrong. I'd expect others can reproduce by running git's test suite, but if not, I'm happy to test patches.

A word of warning to anyone who wants to reproduce this: I believe it caused some data loss during my bisect. Git's test suite does a lot of:

mkdir trash.XXXXX
... do some tests in trash.XXXXX
rm -rf trash.XXXXX

Quoted 1 line

From reading the patch, I'm guessing the bug is causing wrong dir

entries to be returned under load. Random bits of my home directory are missing this morning after my testing; my theory is that when we are unlucky enough to hit the race on the "rm", it can end up deleting random things.

-Peff

maintainer
#2 lore raw
Quoted 17 lines · Jeff King
Quoted 7 lines · Jeff King

On v3.12, running "make -j16 test" on a git.git checkout produces random failures in the test suite, whereas it works fine on v3.11. [...]
Reading the patch for 232d2d6, it seems plausible that it's a locking problem in the new code, but I don't see anything obviously wrong. I'd expect others can reproduce by running git's test suite, but if not, I'm happy to test patches.

A word of warning to anyone who wants to reproduce this: I believe it caused some data loss during my bisect. Git's test suite does a lot of:

mkdir trash.XXXXX
... do some tests in trash.XXXXX
rm -rf trash.XXXXX

Quoted 5 lines
Quoted 1 line

From reading the patch, I'm guessing the bug is causing wrong dir

entries to be returned under load. Random bits of my home directory are missing this morning after my testing; my theory is that when we are unlucky enough to hit the race on the "rm", it can end up deleting random things.

Move initialization of dentry/vfsmount/mnt to just before the loop in prepend_path() and see if that fixes what you are seeing...

author
#3 lore raw
Quoted 8 lines · Al Viro
Quoted 4 lines
Quoted 4 lines

Reading the patch for 232d2d6, it seems plausible that it's a locking problem in the new code, but I don't see anything obviously wrong. I'd expect others can reproduce by running git's test suite, but if not, I'm happy to test patches.

[...]

Move initialization of dentry/vfsmount/mnt to just before the loop in prepend_path() and see if that fixes what you are seeing...

Yep, that did the trick. Looks like your fix is already in Linus's tree, too. Thanks!

-Peff