Skip to contentSkip to conversation

btrfs failing fsx-linux

DISCUSSION Nick Piggin 3 participants 13 messages Aug 18, 2009 – Sep 11, 2009 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Thanks,
Nick

#1 lore raw
Quoted 5 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

Quoted 9 lines

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Nick,

I'll take a look at this later today and see what I can come up with. Thanks for reporting it.

Josef

#2 lore raw
Quoted 16 lines · Josef Bacik
Quoted 5 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

Quoted 9 lines

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Quoted 6 lines

Nick,

I'll take a look at this later today and see what I can come up with. Thanks for reporting it.

Josef

Thanks. After spending a good chunk of my life tracking down fsx-linux problems in fsblock and mm code, I can't say I envy you :) It's a great test to run though (especially syncing and dropping caches because that hits writeback and bypasses caches far far more than fsx ever does on its own).

Let me know if you need me to test any patches.

#3 lore raw
In reply to Josef Bacik
Quoted 16 lines · Josef Bacik
Quoted 5 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

Quoted 9 lines

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Quoted 4 lines

Nick,

I'll take a look at this later today and see what I can come up with. Thanks for reporting it.

Any progress with this? Were you able to reproduce?

#4 lore raw
Quoted 17 lines · Nick Piggin
Quoted 16 lines · Josef Bacik
Quoted 14 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Quoted 7 lines
Quoted 4 lines

Nick,

I'll take a look at this later today and see what I can come up with. Thanks for reporting it.

Any progress with this? Were you able to reproduce?

Err sorry no, I got distracted with my -ENOSPC stuff, it's taking longer than I expected it to. I hope to wrap it up today and take a look at this next. I was able to reproduce the problem on the experimental branch of Chris's tree, so it's not something that has been fixed yet. Thanks,

Josef

#5 lore raw
In reply to Nick Piggin
Quoted 17 lines · Nick Piggin
Quoted 16 lines · Josef Bacik
Quoted 14 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Quoted 7 lines
Quoted 4 lines

Nick,

I'll take a look at this later today and see what I can come up with. Thanks for reporting it.

Any progress with this? Were you able to reproduce?

I thought this was where the rb tree patch came from, sorry. I'll try to reproduce.

-chris

#6 lore raw
Quoted 10 lines · Chris Mason
Quoted 6 lines · Nick Piggin
Quoted 2 lines · Josef Bacik

I'll take a look at this later today and see what I can come up with. Thanks for reporting it.

Any progress with this? Were you able to reproduce?

I thought this was where the rb tree patch came from, sorry. I'll try to reproduce.

Oh, I must not have been clear: I first hit the rbtree bug, then when that was fixed, the same workload was eventually showing up fsx-linux failures (but no more kernel crashes).

#7 lore raw
In reply to Nick Piggin
Quoted 5 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

Quoted 8 lines

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Just an update, I think I've tracked this down. Our fixup code for when set_page_dirty was called without page_mkwrite wasn't catching every case. I'm hitting other problems because I'm testing on my new performance code, but hopefully this will all be nailed down soon.

-chris

#8 lore raw
Quoted 15 lines · Chris Mason
Quoted 5 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

Quoted 8 lines

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Quoted 4 lines

Just an update, I think I've tracked this down. Our fixup code for when set_page_dirty was called without page_mkwrite wasn't catching every case. I'm hitting other problems because I'm testing on my new performance code, but hopefully this will all be nailed down soon.

Which cases do set_page_dirty happen without page_mkwrite? For fsx-linux I hope there shouldn't be any... but one issue with btrfs's page_mkwrite is that it still unlocks the page before page_mkwrite returns so if it is anything like other filesystems, the page might get written out between the page_mkwrite and the page fault's subsequent set_page_dirty.

So if that race is hitting, it might appear like set_page_dirty is being called without a page_mkwrite.

#9 lore raw
Quoted 16 lines · Nick Piggin
Quoted 15 lines · Chris Mason
Quoted 13 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

This makes it a bit hard for me to test my btrfs truncate conversion patch unfortunately, though it does seem pretty stable so I will probably just send it out anyway.

Anyway, just a head's up. Oh, the way I reproduce is to create btrfs on 1GB rd, run 4 instances of fsx-linux on different files in the root directory, and run while true ; do sync ; echo 3 > /proc/sys/vm/drop_caches; done at the same time.

Quoted 7 lines
Quoted 4 lines

Just an update, I think I've tracked this down. Our fixup code for when set_page_dirty was called without page_mkwrite wasn't catching every case. I'm hitting other problems because I'm testing on my new performance code, but hopefully this will all be nailed down soon.

Which cases do set_page_dirty happen without page_mkwrite? For fsx-linux I hope there shouldn't be any...

I've been assuming its the zap_pte_range part, but now I have enough code around it to make stack traces.

Quoted 7 lines

but one issue with btrfs's page_mkwrite
is that it still unlocks the page before page_mkwrite returns so if it is anything like other filesystems, the page might get written out between the page_mkwrite and the page fault's subsequent set_page_dirty.

So if that race is hitting, it might appear like set_page_dirty is being called without a page_mkwrite.

Good point, I'll make sure.

-chris

#10 lore raw
Quoted 6 lines · Chris Mason
Quoted 2 lines · Nick Piggin

Which cases do set_page_dirty happen without page_mkwrite? For fsx-linux I hope there shouldn't be any...

I've been assuming its the zap_pte_range part, but now I have enough code around it to make stack traces.

OK, because AFAIK we're _supposed_ to be able to close all that up since the last round of page_mkwrite fixes here, so pte state will not be out of state with page state.

Yes set_page_dirty still gets called there, for fses without page_mkwrite or which don't try hard to keep in sync. But if you return with the page locked from page_mkwrite, this set_page_diryt call should always find the page dirty (I hope).

Quoted 9 lines
Quoted 7 lines

but one issue with btrfs's page_mkwrite
is that it still unlocks the page before page_mkwrite returns so if it is anything like other filesystems, the page might get written out between the page_mkwrite and the page fault's subsequent set_page_dirty.

So if that race is hitting, it might appear like set_page_dirty is being called without a page_mkwrite.

Good point, I'll make sure.

Though if it has exposed a bug, that's a good thing too; you still want to handle that case (for now) due to the get_uesr_pages problem.

#11 lore raw
Quoted 23 lines · Nick Piggin
Quoted 6 lines · Chris Mason
Quoted 2 lines · Nick Piggin

Which cases do set_page_dirty happen without page_mkwrite? For fsx-linux I hope there shouldn't be any...

I've been assuming its the zap_pte_range part, but now I have enough code around it to make stack traces.

OK, because AFAIK we're _supposed_ to be able to close all that up since the last round of page_mkwrite fixes here, so pte state will not be out of state with page state.

Yes set_page_dirty still gets called there, for fses without page_mkwrite or which don't try hard to keep in sync. But if you return with the page locked from page_mkwrite, this set_page_diryt call should always find the page dirty (I hope).

Quoted 5 lines
Quoted 5 lines

but one issue with btrfs's page_mkwrite
is that it still unlocks the page before page_mkwrite returns so if it is anything like other filesystems, the page might get written out between the page_mkwrite and the page fault's subsequent set_page_dirty.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

So if that race is hitting, it might appear like set_page_dirty is being called without a page_mkwrite.

Good point, I'll make sure.

Though if it has exposed a bug, that's a good thing too; you still want to handle that case (for now) due to the get_uesr_pages problem.

Yeah, I'll hammer on it until the handler works and then I'll fixup page_mkwrite to leave things locked.

-chris

#12 lore raw
In reply to Nick Piggin
Quoted 4 lines · Nick Piggin

Hi Chris,

Don't know if this is a known issue, but I have btrfs (after my previous inode_tree fixup patch) failing fsx-linux in Linus's current git tree.

Thanks for your help on this Nick, I've got my current fixes for fsx in the master branch of the btrfs-unstable repo:

git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git

If you're curious, the 3 most relevant commits are here:

http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=93c82d575055f1bd0277acae6f966bebafd80dd5

http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=50a9b214bc6c052caa05a210ebfc1bdf0d7085b2

http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=a1ed835e1ab5795f91b198d08c43e2f56848dcf3

But these fixes are built on top of my performance work (which was sure to break fsx on its own).

-chris