btrfs failing fsx-linux
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.
This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.
Thanks,
Nick
Quoted 5 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.
Quoted 9 lines
This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Nick,
I'll take a look at this later today and see what I can come up with. Thanks for reporting it.
Josef
Quoted 16 lines · Josef Bacik
Quoted 5 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.Quoted 9 lines
This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Quoted 6 lines
Nick,
I'll take a look at this later today and see what I can come up with. Thanks for reporting it.
Josef
Thanks. After spending a good chunk of my life tracking down fsx-linux problems in fsblock and mm code, I can't say I envy you :) It's a great test to run though (especially syncing and dropping caches because that hits writeback and bypasses caches far far more than fsx ever does on its own).
Let me know if you need me to test any patches.
Quoted 16 lines · Josef Bacik
Quoted 5 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.Quoted 9 lines
This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Quoted 4 lines
Nick,
I'll take a look at this later today and see what I can come up with. Thanks for reporting it.
Any progress with this? Were you able to reproduce?
Quoted 17 lines · Nick Piggin
Quoted 16 lines · Josef Bacik
Quoted 14 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Quoted 7 lines
Quoted 4 lines
Nick,
I'll take a look at this later today and see what I can come up with. Thanks for reporting it.
Any progress with this? Were you able to reproduce?
Err sorry no, I got distracted with my -ENOSPC stuff, it's taking longer than I expected it to. I hope to wrap it up today and take a look at this next. I was able to reproduce the problem on the experimental branch of Chris's tree, so it's not something that has been fixed yet. Thanks,
Josef
Quoted 17 lines · Nick Piggin
Quoted 16 lines · Josef Bacik
Quoted 14 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Quoted 7 lines
Quoted 4 lines
Nick,
I'll take a look at this later today and see what I can come up with. Thanks for reporting it.
Any progress with this? Were you able to reproduce?
I thought this was where the rb tree patch came from, sorry. I'll try to reproduce.
-chris
Quoted 10 lines · Chris Mason
Quoted 6 lines · Nick Piggin
Quoted 2 lines · Josef Bacik
I'll take a look at this later today and see what I can come up with. Thanks for reporting it.
Any progress with this? Were you able to reproduce?
I thought this was where the rb tree patch came from, sorry. I'll try to reproduce.
Oh, I must not have been clear: I first hit the rbtree bug, then when that was fixed, the same workload was eventually showing up fsx-linux failures (but no more kernel crashes).
Quoted 5 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.
Quoted 8 lines
This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Just an update, I think I've tracked this down. Our fixup code for when set_page_dirty was called without page_mkwrite wasn't catching every case. I'm hitting other problems because I'm testing on my new performance code, but hopefully this will all be nailed down soon.
-chris
Quoted 15 lines · Chris Mason
Quoted 5 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.Quoted 8 lines
This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Quoted 4 lines
Just an update, I think I've tracked this down. Our fixup code for when
set_page_dirtywas called withoutpage_mkwritewasn't catching every case. I'm hitting other problems because I'm testing on my new performance code, but hopefully this will all be nailed down soon.
Which cases do set_page_dirty happen without page_mkwrite? For fsx-linux I hope there shouldn't be any... but one issue with btrfs's page_mkwrite is that it still unlocks the page before page_mkwrite returns so if it is anything like other filesystems, the page might get written out between the page_mkwrite and the page fault's subsequent set_page_dirty.
So if that race is hitting, it might appear like set_page_dirty is being called without a page_mkwrite.
Quoted 16 lines · Nick Piggin
Quoted 15 lines · Chris Mason
Quoted 13 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.
Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run
while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; doneat the same time.
Quoted 7 lines
Quoted 4 lines
Just an update, I think I've tracked this down. Our fixup code for when
set_page_dirtywas called withoutpage_mkwritewasn't catching every case. I'm hitting other problems because I'm testing on my new performance code, but hopefully this will all be nailed down soon.Which cases do
set_page_dirtyhappen withoutpage_mkwrite? For fsx-linux I hope there shouldn't be any...
I've been assuming its the zap_pte_range part, but now I have enough code around it to make stack traces.
Quoted 7 lines
but one issue with btrfs's
page_mkwrite
is that it still unlocks the page beforepage_mkwritereturns so if it is anything like other filesystems, the page might get written out between thepage_mkwriteand the page fault's subsequentset_page_dirty.So if that race is hitting, it might appear like
set_page_dirtyis being called without apage_mkwrite.
Good point, I'll make sure.
-chris
Quoted 6 lines · Chris Mason
Quoted 2 lines · Nick Piggin
Which cases do
set_page_dirtyhappen withoutpage_mkwrite? For fsx-linux I hope there shouldn't be any...I've been assuming its the
zap_pte_rangepart, but now I have enough code around it to make stack traces.
OK, because AFAIK we're _supposed_ to be able to close all that up since the last round of page_mkwrite fixes here, so pte state will not be out of state with page state.
Yes set_page_dirty still gets called there, for fses without page_mkwrite or which don't try hard to keep in sync. But if you return with the page locked from page_mkwrite, this set_page_diryt call should always find the page dirty (I hope).
Quoted 9 lines
Quoted 7 lines
but one issue with btrfs's
page_mkwrite
is that it still unlocks the page beforepage_mkwritereturns so if it is anything like other filesystems, the page might get written out between thepage_mkwriteand the page fault's subsequentset_page_dirty.So if that race is hitting, it might appear like
set_page_dirtyis being called without apage_mkwrite.Good point, I'll make sure.
Though if it has exposed a bug, that's a good thing too; you still want to handle that case (for now) due to the get_uesr_pages problem.
Quoted 23 lines · Nick Piggin
Quoted 6 lines · Chris Mason
Quoted 2 lines · Nick Piggin
Which cases do
set_page_dirtyhappen withoutpage_mkwrite? For fsx-linux I hope there shouldn't be any...I've been assuming its the
zap_pte_rangepart, but now I have enough code around it to make stack traces.OK, because AFAIK we're _supposed_ to be able to close all that up since the last round of
page_mkwritefixes here, so pte state will not be out of state with page state.Yes
set_page_dirtystill gets called there, for fses withoutpage_mkwriteor which don't try hard to keep in sync. But if you return with the page locked frompage_mkwrite, thisset_page_dirytcall should always find the page dirty (I hope).Quoted 5 lines
Quoted 5 lines
but one issue with btrfs's
page_mkwrite
is that it still unlocks the page beforepage_mkwritereturns so if it is anything like other filesystems, the page might get written out between thepage_mkwriteand the page fault's subsequentset_page_dirty.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
So if that race is hitting, it might appear like
set_page_dirtyis being called without apage_mkwrite.Good point, I'll make sure.
Though if it has exposed a bug, that's a good thing too; you still want to handle that case (for now) due to the
get_uesr_pagesproblem.
Yeah, I'll hammer on it until the handler works and then I'll fixup page_mkwrite to leave things locked.
-chris
Quoted 4 lines · Nick Piggin
Hi Chris,
Don't know if this is a known issue, but I have btrfs (after my previous
inode_treefixup patch) failing fsx-linux in Linus's current git tree.
Thanks for your help on this Nick, I've got my current fixes for fsx in the master branch of the btrfs-unstable repo:
git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git
If you're curious, the 3 most relevant commits are here:
But these fixes are built on top of my performance work (which was sure to break fsx on its own).
-chris