observed significant performance improvement using "delaylog" in a real-world application
Hi all,
we use XFS for a very I/O-intensive, in-house developed real-time database application, and whenever we see new or significantly changed file-systems becoming available, we run a benchmark using this application on a conserved, fixed real-world data set.
I'm pleased to state that using the experimental "delaylog" mount option (in vanilla linux-2.6.35) we measured a 17% performance increase for our benchmark scenario. (Other mount-options in use both before
and after the "delaylog" option: noatime,nodiratime,nobarrier)
That's a lot given that XFS was the fastest performing file-system for this application already.
It's also a promising result regarding stability, as several other tests (using e.g. reiser4 or ceph) in the past led to crashes in the same benchmark scenario.
So thanks to all contributing developers for this significant optimization!
Regards,
Peter Niemayer
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Peter Niemayer
Hi all,
we use XFS for a very I/O-intensive, in-house developed real-time database application, and whenever we see new or significantly changed file-systems becoming available, we run a benchmark using this application on a conserved, fixed real-world data set.
Quoted 5 lines
I'm pleased to state that using the experimental "delaylog" mount option (in vanilla linux-2.6.35) we measured a 17% performance increase
for our benchmark scenario. (Other mount-options in use both before and after the "delaylog" option: noatime,nodiratime,nobarrier)
That's great to hear. One thing that you might want to try to further improve performance is the logbsize=262144 option as well. That will help flush log IO faster by doing less IOs.
Also, if your workload is doing lots of fsync calls, then the optimisations I posted a few days ago should also help improve delaylog throughput.
Quoted 6 lines
That's a lot given that XFS was the fastest performing file-system for this application already.
It's also a promising result regarding stability, as several other tests (using e.g. reiser4 or ceph) in the past led to crashes in the same benchmark scenario.
That's definitely encouraging. ;)
Quoted 1 line
So thanks to all contributing developers for this significant optimization!
And thanks for the feedback.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Dave Chinner
One thing that you might want to try to
further improve performance is the logbsize=262144 option as well. That will help flush log IO faster by doing less IOs.
Tried that - but this did not have a positive effect. (In fact a slightly negative one, but that could be within the measurement accuracy).
Quoted 3 lines
Also, if your workload is doing lots of fsync calls, then the optimisations I posted a few days ago should also help improve delaylog throughput.
We don't to any fsync's.
Regards,
Peter Niemayer
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 1 line · On Dienstag, 10. August 2010 Peter Niemayer wrote:
17% performance increase for our benchmark scenario
Can you say what kind of data accesses you have? Database, fileserver, virtualization platform with VM Images on XFS... there are so many. Just to get an idea in which area delaylog helps that much.
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Aktuelles Radiointerview! ****** http://www.it-podcast.at/aktuelle-sendung.html // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Michael Monnerie
Quoted 1 line · On Dienstag, 10. August 2010 Peter Niemayer wrote:
17% performance increase for our benchmark scenario
Can you say what kind of data accesses you have?
The application basically writes a large data set to ~ 100,000 files,
which consists of many small messages. Each message consists of a primary key (an integer in the range of 0 to approx. 1,000,000) and an (almost random) number of data bytes (length in the range from 10 to
~1000 byte).
For each message, the application opens the file that is determined by the primary key with O_APPEND, write()s the data bytes to the file. Then it closes the file. (There are usually a few to many messages per primary key / file).
The application runs 4 threads in parallel to spread the above action over 4 CPU cores, each thread processes a quarter of the primary keys (primary_key & 0x03).
This description is, of course, somewhat simplified, but it should get you an idea on the kind of I/O-operations.
Regards,
Peter Niemayer
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · On Mittwoch, 11. August 2010 Peter Niemayer wrote:
This description is, of course, somewhat simplified, but it should get you an idea on the kind of I/O-operations.
Thank you. Are those files located within one dir or do you use a hash structure like squid cache does?
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Aktuelles Radiointerview! ****** http://www.it-podcast.at/aktuelle-sendung.html // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Michael Monnerie
Thank you. Are those files located within one dir or do you use a hash structure like squid cache does?
There's only a shallow hierarchy (for functional, not for distribution reasons), so the relevant directories have thousands of files in them.
I think after the "ext2"-age no serious file system ever had a real problem dealing with lots of files in one directory - or do you have contradicting information?
Regards,
Peter Niemayer
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · On Mittwoch, 11. August 2010 Peter Niemayer wrote:
I think after the "ext2"-age no serious file system ever had a real problem dealing with lots of files in one directory - or do you have contradicting information?
IIRC 2010 there was someone on this list reporting problems when a certain amount of files were created in a dir. He had to do with audio/video recording, where delays are important (not to have). At one point file creating time increased a lot, but I don't remember if it was a filesystem problem or something else. But as it was the only report I saw, I'd say you are right and a big number of files per dir doesn't matter anymore.
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Aktuelles Radiointerview! ****** http://www.it-podcast.at/aktuelle-sendung.html // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Peter Niemayer
Quoted 2 lines · Michael Monnerie
Thank you. Are those files located within one dir or do you use a hash structure like squid cache does?
There's only a shallow hierarchy (for functional, not for distribution reasons), so the relevant directories have thousands of files in them.
I think after the "ext2"-age no serious file system ever had a real problem dealing with lots of files in one directory - or do you have contradicting information?
Define "lots of files". :)
Quoted 1 line
From my numbers, ext3/4 still fall way behind XFS and btrfs when it
comes to handling directories with tens of thousands of entries or larger. Especially on cold-cache random lookups.
XFS also has quite sophisticated internal directory readahead, so under the cold cache directory performance of XFS is far better than ext3/4 can acheive, even for relatively small directories. IIRC this difference in directory lookup performance was one of the prime reasons kernel.org switched from ext3 to XFS a couple of years back...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs