2.6.35 performance results
Here is the latest set of performance runs from the 2.6.35-rc5 tree. Included is a refresh of all the other filesystems with some changes for barriers on and off since this has been somewhat of a hot topic recently.
New data linked in to the history graphs here:
http://btrfs.boxacle.net/repository/raid/history/History.html
From a BTRFS performance perspective, we took a major regression on write heavy workloads. As much as a 10x hit! The problems seems to be due to this changeset:
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=5da9d01b66458b180a6bee0e637a1d0a3effc622
Btrfs: Shrink delay allocated space in a synchronized
Shrink delayed allocation space in a synchronized manner is more controllable than flushing all delay allocated space in an async thread.
This changeset introduced "btrfs_start_one_delalloc_inode" in
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=5da9d01b66458b180a6bee0e637a1d0a3effc622
In heavy write workloads this new function is now dominating the profiles:
samples % app name symbol name 8914973 65.1261 btrfs.ko btrfs_start_one_delalloc_inode 1024841 7.4867 vmlinux-2.6.35-rc5-autokern1 rb_get_reader_page 716046 5.2309 vmlinux-2.6.35-rc5-autokern1 ring_buffer_consume 315354 2.3037 oprofile.ko add_event_entry 202484 1.4792 vmlinux-2.6.35-rc5-autokern1 write_inode_now 195018 1.4247 btrfs.ko btrfs_tree_lock
Appears to be major contention on the spin lock, as this gets worse with more threads. This needs to be redone.
Steve
Quoted 5 lines · Steven Pratt
Here is the latest set of performance runs from the 2.6.35-rc5 tree. Included is a refresh of all the other filesystems with some changes for barriers on and off since this has been somewhat of a hot topic recently.
Quoted 7 lines
New data linked in to the history graphs here:
http://btrfs.boxacle.net/repository/raid/history/History.htmlFrom a BTRFS performance perspective, we took a major regression on write heavy workloads. As much as a 10x hit! The problems seems to be due to this changeset:
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=5da9d01b66458b180a6bee0e637a1d0a3effc622
Ouch! The problem is we're not being aggressive enough about allocating chunks for data, which makes the flusher come in and start data IO.
Thanks a lot for finding the regression, my machine definitely didn't show this.
I'll reproduce and fix it up.
-chris
Quoted 29 lines
Btrfs: Shrink delay allocated space in a synchronized
Shrink delayed allocation space in a synchronized manner is more controllable than flushing all delay allocated space in an async thread.
This changeset introduced "
btrfs_start_one_delalloc_inode" in
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=5da9d01b66458b180a6bee0e637a1d0a3effc622In heavy write workloads this new function is now dominating the profiles:
samples % app name symbol name 8914973 65.1261 btrfs.ko btrfs_start_one_delalloc_inode 1024841 7.4867 vmlinux-2.6.35-rc5-autokern1 rb_get_reader_page 716046 5.2309 vmlinux-2.6.35-rc5-autokern1 ring_buffer_consume 315354 2.3037 oprofile.ko add_event_entry 202484 1.4792 vmlinux-2.6.35-rc5-autokern1 write_inode_now 195018 1.4247 btrfs.ko btrfs_tree_lockAppears to be major contention on the spin lock, as this gets worse with more threads. This needs to be redone.
Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 18 lines · Steven Pratt
Here is the latest set of performance runs from the 2.6.35-rc5 tree. Included is a refresh of all the other filesystems with some changes for barriers on and off since this has been somewhat of a hot topic recently.
New data linked in to the history graphs here:
http://btrfs.boxacle.net/repository/raid/history/History.htmlFrom a BTRFS performance perspective, we took a major regression on write heavy workloads. As much as a 10x hit! The problems seems to be due to this changeset:
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=5da9d01b66458b180a6bee0e637a1d0a3effc622
Btrfs: Shrink delay allocated space in a synchronizedShrink delayed allocation space in a synchronized manner is more controllable than flushing all delay allocated space in an async thread.
Quoted 7 lines
This changeset introduced "
btrfs_start_one_delalloc_inode" in
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=5da9d01b66458b180a6bee0e637a1d0a3effc622In heavy write workloads this new function is now dominating the profiles:
samples % app name symbol name 8914973 65.1261 btrfs.ko btrfs_start_one_delalloc_inode
Hi Steve,
I think I know why this is a problem and how to fix it, but I'm having a trouble reproducing this exact setup. Which of your tests was this oprofile from?
-chris
Quoted 29 lines · Chris Mason
Quoted 18 lines · Steven Pratt
Here is the latest set of performance runs from the 2.6.35-rc5 tree. Included is a refresh of all the other filesystems with some changes for barriers on and off since this has been somewhat of a hot topic recently.
New data linked in to the history graphs here:
http://btrfs.boxacle.net/repository/raid/history/History.htmlFrom a BTRFS performance perspective, we took a major regression on write heavy workloads. As much as a 10x hit! The problems seems to be due to this changeset:
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commit;h=5da9d01b66458b180a6bee0e637a1d0a3effc622
Btrfs: Shrink delay allocated space in a synchronizedShrink delayed allocation space in a synchronized manner is more controllable than flushing all delay allocated space in an async thread.
Quoted 8 lines
This changeset introduced "
btrfs_start_one_delalloc_inode" in
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=5da9d01b66458b180a6bee0e637a1d0a3effc622In heavy write workloads this new function is now dominating the profiles:
samples % app name symbol name 8914973 65.1261 btrfs.ko btrfs_start_one_delalloc_inode
Quoted 6 lines
Hi Steve,
I think I know why this is a problem and how to fix it, but I'm having a trouble reproducing this exact setup. Which of your tests was this oprofile from?
128 thread random write. With or without nocow option.
Steve
Quoted 3 lines
-chris
Quoted 9 lines · Steven Pratt
Quoted 8 lines · Chris Mason
Quoted 7 lines
This changeset introduced "
btrfs_start_one_delalloc_inode" in
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=5da9d01b66458b180a6bee0e637a1d0a3effc622In heavy write workloads this new function is now dominating the profiles:
samples % app name symbol name 8914973 65.1261 btrfs.ko btrfs_start_one_delalloc_inode
Quoted 6 lines
Quoted 5 lines
Hi Steve,
I think I know why this is a problem and how to fix it, but I'm having a trouble reproducing this exact setup. Which of your tests was this oprofile from?
128 thread random write. With or without nocow option.
Ok, I haven't managed to reproduce your problem exactly, but this is faster for me here. Could you please give it a try:
Quoted 1 line
From
8e965331de749c39f3781d581b55d2c207de060fMon Sep 17 00:00:00 2001
From: Chris Mason <[email protected]>
Date: Wed, 18 Aug 2010 13:31:27 -0400
Subject: [PATCH] Btrfs: don't trigger delayed allocation throttling as often
We reserve metadata space based on the number of delayed allocation extents that are currently pending. As we run out of space, we start forcing writeback to turn those reservations into physical extents.
The reservations are based on some worst case math, so the sooner we turn them into real blocks, the better off we are.
But, the writeback is being forced too soon and too often. This fixes things to be less aggressive.
--- fs/btrfs/extent-tree.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
fs/btrfs/extent-tree.c +6 -1
@@ -3681,6 +3681,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode, u64 num_bytes)struct btrfs_root *root =BTRFS_I (inode)->root;struct btrfs_block_rsv *block_rsv = &root->fs_info->delalloc_block_rsv;u64 to_reserve; +u64 max_reserve;int nr_extents;int retries =0 ;int ret;
@@ -3717,7 +3718,11 @@ again:block_rsv_add_bytes (block_rsv, to_reserve,1 ); -if (block_rsv->size >512 *1024 *1024 ) +/* 10% or 2GB */ + max_reserve =min_t (u64,2ULL *1024 *1024 *1024 , +div_factor (root->fs_info->fs_devices->total_rw_bytes,1 )); + +if (block_rsv->size > max_reserve)shrink_delalloc (NULL , root, to_reserve);return 0 ;
Signature · 1 line
1.7.1.1
Quoted 18 lines · Chris Mason
Quoted 16 lines · Steven Pratt
Quoted 14 lines · Chris Mason
Quoted 8 lines
This changeset introduced "
btrfs_start_one_delalloc_inode" in
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=5da9d01b66458b180a6bee0e637a1d0a3effc622In heavy write workloads this new function is now dominating the profiles:
samples % app name symbol name 8914973 65.1261 btrfs.ko btrfs_start_one_delalloc_inodeHi Steve,
I think I know why this is a problem and how to fix it, but I'm having a trouble reproducing this exact setup. Which of your tests was this oprofile from?
Quoted 6 lines
Quoted 2 lines
128 thread random write. With or without nocow option.
Ok, I haven't managed to reproduce your problem exactly, but this is faster for me here. Could you please give it a try:
Was out on vacation. Test is running now. Should have results by uploaded by Monday.
Steve
Quoted 46 lines
Quoted 1 line
From
8e965331de749c39f3781d581b55d2c207de060fMon Sep 17 00:00:00 2001From: Chris Mason <[email protected]>
Date: Wed, 18 Aug 2010 13:31:27 -0400
Subject: [PATCH] Btrfs: don't trigger delayed allocation throttling as oftenWe reserve metadata space based on the number of delayed allocation extents that are currently pending. As we run out of space, we start forcing writeback to turn those reservations into physical extents.
The reservations are based on some worst case math, so the sooner we turn them into real blocks, the better off we are.
But, the writeback is being forced too soon and too often. This fixes things to be less aggressive.
Signed-off-by Chris Mason <[email protected]>--- fs/btrfs/extent-tree.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)fs/btrfs/extent-tree.c +6
-1@@ -3681,6 +3681,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode, u64 num_bytes)struct btrfs_root *root =BTRFS_I (inode)->root;struct btrfs_block_rsv *block_rsv = &root->fs_info->delalloc_block_rsv;u64 to_reserve; +u64 max_reserve;int nr_extents;int retries =0 ;int ret;@@ -3717,7 +3718,11 @@ again:block_rsv_add_bytes (block_rsv, to_reserve,1 ); -if (block_rsv->size >512 *1024 *1024 ) +/* 10% or 2GB */ + max_reserve =min_t (u64,2ULL *1024 *1024 *1024 , +div_factor (root->fs_info->fs_devices->total_rw_bytes,1 )); + +if (block_rsv->size > max_reserve)shrink_delalloc (NULL , root, to_reserve);return 0 ;
Quoted 76 lines · Steven Pratt
Quoted 19 lines · Chris Mason
Quoted 17 lines · Steven Pratt
Quoted 15 lines · Chris Mason
Quoted 9 lines
This changeset introduced "
btrfs_start_one_delalloc_inode" in
http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable.git;a=commitdiff;h=5da9d01b66458b180a6bee0e637a1d0a3effc622In heavy write workloads this new function is now dominating the profiles:
samples % app name symbol name 8914973 65.1261 btrfs.ko btrfs_start_one_delalloc_inodeHi Steve,
I think I know why this is a problem and how to fix it, but I'm having a trouble reproducing this exact setup. Which of your tests was this oprofile from?
Quoted 6 lines
Quoted 2 lines
128 thread random write. With or without nocow option.
Ok, I haven't managed to reproduce your problem exactly, but this is faster for me here. Could you please give it a try:
Was out on vacation. Test is running now. Should have results by uploaded by Monday.
Steve
Quoted 46 lines
Quoted 1 line
From
8e965331de749c39f3781d581b55d2c207de060fMon Sep 17 00:00:00 2001From: Chris Mason <[email protected]>
Date: Wed, 18 Aug 2010 13:31:27 -0400
Subject: [PATCH] Btrfs: don't trigger delayed allocation throttling as oftenWe reserve metadata space based on the number of delayed allocation extents that are currently pending. As we run out of space, we start forcing writeback to turn those reservations into physical extents.
The reservations are based on some worst case math, so the sooner we turn them into real blocks, the better off we are.
But, the writeback is being forced too soon and too often. This fixes things to be less aggressive.
Signed-off-by Chris Mason <[email protected]>--- fs/btrfs/extent-tree.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)fs/btrfs/extent-tree.c +1
-0@@ -3681,6 +3681,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode, u64 num_bytes)struct btrfs_root *root =BTRFS_I (inode)->root;struct btrfs_block_rsv *block_rsv = &root->fs_info->delalloc_block_rsv;u64 to_reserve; +u64 max_reserve;int nr_extents;int retries = 0; int ret;fs/btrfs/extent-tree.c +5
-1@@ -3717,7 +3718,11 @@ again:block_rsv_add_bytes (block_rsv, to_reserve,1 ); -if (block_rsv->size >512 *1024 *1024 ) +/* 10% or 2GB */ + max_reserve =min_t (u64,2ULL *1024 *1024 *1024 , +div_factor (root->fs_info->fs_devices->total_rw_bytes,1 )); + +if (block_rsv->size > max_reserve)shrink_delalloc (NULL , root, to_reserve);
Quoted 2 lines
fs/btrfs/extent-tree.c again: · line 3728
Quoted 2 lines
fs/btrfs/extent-tree.c +0
-0return 0 ;
This did not seem to help, in fact we regressed more with COW enabled.. One thing to note, the last 2 sets of runs in the history graphs were actually run by Keith and he used stock kernel trees. For my recreate, I pulled the latest btrfs-unstable which is based on a 2.6.34 tree. Should I retest this on stock 2.6.35? The high time in btrfs_start_one_delalloc_inode still exists.
Full results can be found here:
http://btrfs.boxacle.net/repository/raid/perftest/perfpatch/perfpatch.html
128 thread random write test that shows the problem:
Steve
Quoted 5 lines
Signature · 4 lines
-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Steven Pratt
This did not seem to help, in fact we regressed more with COW enabled.. One thing to note, the last 2 sets of runs in the history graphs were actually run by Keith and he used stock kernel trees. For my recreate, I pulled the latest btrfs-unstable which is based on a 2.6.34 tree. Should I retest this on stock 2.6.35? The high time in
btrfs_start_one_delalloc_inodestill exists.
btrfs-unstable or .35 are both fine.
Is this a fresh mkfs or are you reusing an existing tree?
Quoted 7 lines
Full results can be found here:
http://btrfs.boxacle.net/repository/raid/perftest/perfpatch/perfpatch.html128 thread random write test that shows the problem:
Ok, thanks, I'll try again.
-chris
Quoted 10 lines · Chris Mason
Quoted 7 lines · Steven Pratt
This did not seem to help, in fact we regressed more with COW enabled.. One thing to note, the last 2 sets of runs in the history graphs were actually run by Keith and he used stock kernel trees. For my recreate, I pulled the latest btrfs-unstable which is based on a 2.6.34 tree. Should I retest this on stock 2.6.35? The high time in
btrfs_start_one_delalloc_inodestill exists.
Quoted 3 lines
btrfs-unstable or .35 are both fine.
Ok.
Quoted 3 lines
Is this a fresh mkfs or are you reusing an existing tree?
In between. New mkfs before benchmark run, multiple tests are all then run with unmounting and remounting, but no new mkfs. The random write is preceded by sequential reads and random reads.
Quoted 10 lines
Quoted 7 lines
Full results can be found here:
http://btrfs.boxacle.net/repository/raid/perftest/perfpatch/perfpatch.html128 thread random write test that shows the problem:
Ok, thanks, I'll try again.
Ok, will probably just run the 128 thread random write next time, since I am not seeing much difference on anything else.
Steve
Quoted 2 lines
-chris