Skip to contentSkip to conversation

More performance results

DISCUSSION Steven Pratt 6 participants 11 messages Feb 2, 2009 – Mar 16, 2009 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Steve

#1 lore raw
Quoted 7 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Thanks Steve, was the mainline btrfs used for this? I'm working on the write heavy problems this week.

-chris

#2 lore raw
Quoted 11 lines · Chris Mason
Quoted 4 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Thanks Steve, was the mainline btrfs used for this? I'm working on the write heavy problems this week.

This was straight mainline Linus tree.

Steve

Quoted 8 lines

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Steven Pratt
Quoted 4 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Would you please mind explaining how bad the results are and how much more this needs to be improved for Btrfs to be perfomance wise acceptable?

I see that Btrfs almost everywhere lacks XFS and others in some cases.

Thanks,
Andev

#4 lore raw
Quoted 10 lines · debian developer
Quoted 3 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Would you please mind explaining how bad the results are and how much more this needs to be improved for Btrfs to be perfomance wise acceptable?

I see that Btrfs almost everywhere lacks XFS and others in some cases.

Nobody working on btrfs development is satisfied with the current performance. We knew before the merge that the present code would not be a benchmarking champion.

We are working on improving it. The more testing of different configurations and more feedback, the better we understand what areas need work.

For example, I'm working on really implementing O_DIRECT. Today O_DIRECT just goes through buffer cache.

"Acceptable performance" will depend on what features are important to a user. For example, we expect to use more CPU than other filesystems with btrfs doing checksumming.

jim

#5 lore raw
In reply to debian developer
Quoted 11 lines · debian developer
Quoted 4 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Would you please mind explaining how bad the results are and how much more this needs to be improved for Btrfs to be perfomance wise acceptable?

Well as I pointed out most of the write workloads seem to run into CPU/locking issues on RAID systems (especially at higher thread counts)where high levels of throughput are expected. There is lots of data out there, but a good place to look would be http://btrfs.boxacle.net/repository/raid/2.6.29-rc2/2.6.29-rc2/2.6.29-rc2.html which shows performance on the latest RC kernel. As thread counts go up, BTRFS lags more and more on write workloads. For example on 128 thread random write http://btrfs.boxacle.net/repository/raid/2.6.29-rc2/2.6.29-rc2/2.6.29-rc2_Large_file_random_writes._num_threads=128.html
BTRFS achieves about 4MB/sec where the next worse FS (XFS in this case) get 78MB/sec. So for this example BTRFS is slower by a factor of almost 20x.

Let me point out that this is not a criticism of BTRFS, this is just the normal development cycle. Most of the major function is now in, and performance can now become a focus. The point of these benchmarks is to help identify the areas that need attention and provide the debug and analysis data to help facilitate that. Chris has already stated that he hopes to start looking at write performance this week.

Quoted 2 lines

I see that Btrfs almost everywhere lacks XFS and others in some cases.

For now.

Steve

Quoted 7 lines

Thanks,
Andev
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
In reply to debian developer
Quoted 11 lines · debian developer
Quoted 4 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Would you please mind explaining how bad the results are and how much more this needs to be improved for Btrfs to be perfomance wise acceptable?

I see that Btrfs almost everywhere lacks XFS and others in some cases.

These benchmarks are great because they hammer on some of the worst cases code in btrfs. The mail-server benchmark for example isn't quite a mail server workload because it doesn't fsync the files to disk.

But what it does do is hammer on a mixed file read/write/delete workload, which hits btree concurrency and file layout. In my testing here, the big difference between ext4 and btrfs isn't writing to files, it is actually the unlinks. If I take them out of the run, btrfs is very close to ext4 times.

So, I'm working on that.

The random write workload is probably just a file allocation problem. Btrfs should be perform very well in that workload.

-chris

#7 lore raw
Quoted 12 lines · Chris Mason
Quoted 10 lines · debian developer
Quoted 4 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Would you please mind explaining how bad the results are and how much more this needs to be improved for Btrfs to be perfomance wise acceptable?

Quoted 7 lines
Quoted 2 lines

I see that Btrfs almost everywhere lacks XFS and others in some cases.

These benchmarks are great because they hammer on some of the worst cases code in btrfs. The mail-server benchmark for example isn't quite a mail server workload because it doesn't fsync the files to disk.

Actually it does. We fixed this after the first round was posted. Any results since October have fsync on the create of new files.

From the latest runs for mailserver:

	 op weights
	                 read = 0 (0.00%)
	              readall = 4 (57.14%)
	                write = 0 (0.00%)
	               create = 0 (0.00%)
	               append = 0 (0.00%)
	               delete = 1 (14.29%)
	               metaop = 0 (0.00%)
	            createdir = 0 (0.00%)
	                 stat = 0 (0.00%)
	             writeall = 0 (0.00%)
	       writeall_fsync = 0 (0.00%)
	           open_close = 0 (0.00%)
	          write_fsync = 0 (0.00%)
	         create_fsync = 2 (28.57%)
	         append_fsync = 0 (0.00%)

We should probably fsync that into the list of system calls that we track latency for.

Steve

Quoted 21 lines

But what it does do is hammer on a mixed file read/write/delete workload, which hits btree concurrency and file layout. In my testing here, the big difference between ext4 and btrfs isn't writing to files, it is actually the unlinks. If I take them out of the run, btrfs is very close to ext4 times.

So, I'm working on that.

The random write workload is probably just a file allocation problem. Btrfs should be perform very well in that workload.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw
Quoted 13 lines · Steven Pratt
Quoted 12 lines · Chris Mason
Quoted 10 lines · debian developer
Quoted 4 lines · Steven Pratt

Finally cleared out a backlog of results to upload. Main performance page is updated with all the links. (http://btrfs.boxacle.net/) Most recent results are on 2.6.29-rc2. As usual see analysis directory of results for oprofile, including call graphs.

Single disk results are not too bad. Raid still falls apart on any write heavy workload.

Would you please mind explaining how bad the results are and how much more this needs to be improved for Btrfs to be perfomance wise acceptable?

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I see that Btrfs almost everywhere lacks XFS and others in some cases.

These benchmarks are great because they hammer on some of the worst cases code in btrfs. The mail-server benchmark for example isn't quite a mail server workload because it doesn't fsync the files to disk.

Actually it does. We fixed this after the first round was posted.

Oh, sorry I thought that was put into a different run with fsync on. So, the mail server benchmark I've been tuning locally doesn't have fsync on ;) Deletes are still the slow part on that one.

Thanks for the correction though, I'll look at the fsync perf once the non-fsync perf is faster.

-chris

#9 lore raw
In reply to Chris Mason
Quoted 4 lines · Chris Mason

[...] In my testing
here, the big difference between ext4 and btrfs isn't writing to files, it is actually the unlinks. If I take them out of the run, btrfs is very close to ext4 times.

Oh man, what is it with unlinks. Nobody does them very fast.

We use "delayed delete" with Cyrus so that the majority of unlinks get saved for the weekend, and even then run them serially because the IO hit is so high. We do more IO during the cyr_expire run than even the peak of U.S. day.

A "multi-unlink" API would be seriously nice, where you could say "I want all these files to disappear, so don't bother trying to keep making the directory entries consistent in-between-times".

Especially, 'rm -rf' performance really sucks with single unlinks - you're re-creating all this directory data that's just going to be discarded in a second anyway.

Bron ( wondering how much is "it's a hard problem" and how much is
"nobody bothers to optimise it" )

#10 lore raw
In reply to Steven Pratt

=E5=9C=A8 2009-02-02=E4=B8=80=E7=9A=84 09:58 -0600=EF=BC=8CSteven Pratt= =E5=86=99=E9=81=93=EF=BC=9A

Quoted 1 line

Finally cleared out a backlog of results to upload. Main performance=

=20

Quoted 1 line

page is updated with all the links. (http://btrfs.boxacle.net/) Mos=

t=20

Quoted 8 lines

recent results are on 2.6.29-rc2. As usual see analysis directory of=20 results for oprofile, including call graphs.
=20
Single disk results are not too bad. Raid still falls apart on any=20 write heavy workload.
=20
Steve
=20

Quick link to the latest single disk IO results:

http://btrfs.boxacle.net/repository/single-disk/2.6.29-rc2/2.6.29-rc2/2=
=2E6.29-rc2.html

Steve's fs performance tests shows performance regression on ext4 compa= ring with ext3 with O_DIRECT random writes, about 10% on single disk. = =20

http://btrfs.boxacle.net/repository/single-disk/2.6.29-rc2/2.6.29-rc2/2=
=2E6.29-rc2_Large_file_random_writes_odirect._num_threads=3D32.html

And looked at the oprofile, comparing with ext3, ext4 seems mark the inode diry quit often than ext3 does.=20

62        0.0063  ext3.ko                  ext3_mark_iloc_dirty
367       0.0371  ext4.ko                  ext4_mark_iloc_dirty

This also seen in random read test (with no O_DIRECT) oprfofile data, s= o
it doesn't seem not a O_DIRECT issue.

Any thoughts?

More detail about the ext3/4 random odirect writes oprofile...

ext3:

:~/ext4$ grep ext3 oprofile-ext3-random-write-odirect.001=20
554       0.0567  ext3.ko                  ext3_get_branch
346       0.0354  ext3.ko                  ext3_file_write
336       0.0344  ext3.ko                  ext3_direct_IO
215       0.0220  ext3.ko                  ext3_get_blocks_handle
125       0.0128  ext3.ko                  ext3_block_to_path
112       0.0115  ext3.ko                  verify_chain
76        0.0078  ext3.ko                  ext3_get_block
62        0.0063  ext3.ko                  ext3_mark_iloc_dirty
39        0.0040  ext3.ko                  __ext3_get_inode_loc
17        0.0017  ext3.ko                  ext3_getblk
14        0.0014  ext3.ko                  ext3_get_group_desc
13        0.0013  ext3.ko
__ext3_journal_get_write_access
13        0.0013  ext3.ko                  ext3_find_entry
13        0.0013  ext3.ko                  ext3_new_blocks
13        0.0013  ext3.ko                  ext3_reserve_inode_write
=2E...
$ grep ext4 oprofile-ext4-random-write-odirect.001=20
warning: could not check that the binary
file /lib/modules/2.6.29-rc2/kernel/fs/ext4/ext4.ko has not been
modified since the profile was taken. Results may be inaccurate.
420       0.0425  ext4.ko                  ext4_direct_IO
411       0.0416  ext4.ko                  ext4_file_write
403       0.0408  ext4.ko                  ext4_ext_find_extent
374       0.0378  ext4.ko                  __ext4_get_inode_loc
367       0.0371  ext4.ko                  ext4_mark_iloc_dirty
202       0.0204  ext4.ko                  ext4_ext_get_blocks
194       0.0196  ext4.ko                  ext4_get_blocks_wrap
179       0.0181  ext4.ko                  __ext4_ext_check_header
178       0.0180  ext4.ko                  ext4_journal_start_sb
117       0.0118  ext4.ko                  ext4_get_block
109       0.0110  ext4.ko                  ext4_get_group_desc
89        0.0090  ext4.ko                  ext4_mark_inode_dirty
84        0.0085  ext4.ko                  ext4_dirty_inode
74        0.0075  ext4.ko                  __ext4_journal_stop
=2E....
:~/ext4$ grep jbd2 oprofile-ext4-random-write-odirect.001=20
450       0.0455  jbd2.ko                  start_this_handle
288       0.0291  jbd2.ko                  do_get_write_access
287       0.0290  jbd2.ko                  jbd2_journal_stop
244       0.0247  jbd2.ko                  jbd2_journal_start
235       0.0238  jbd2.ko                  jbd2_journal_dirty_metadata
214       0.0216  jbd2.ko                  jbd2_journal_add_journal_hea=
d

~/ext4$ grep jbd oprofile-ext3-random-write-odirect.001=20

116       0.0119  jbd.ko                   journal_add_journal_head
94        0.0096  jbd.ko                   do_get_write_access
73        0.0075  jbd.ko                   start_this_handle
68        0.0070  jbd.ko                   journal_clean_one_cp_list
58        0.0059  jbd.ko                   journal_commit_transaction
48        0.0049  jbd.ko                   journal_dirty_metadata
45        0.0046  jbd.ko                   __journal_file_buffer
42        0.0043  jbd.ko                   __journal_temp_unlink_buffer
42        0.0043  jbd.ko                   journal_put_journal_head
Quoted 2 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs=

" in

Quoted 2 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" i= n
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html