Skip to contentSkip to conversation

ext3 throughput woes on certain (possibly heavily fragmented) files

DISCUSSION Aaron Lehmann 7 participants 10 messages Sep 3, 2002 – Sep 17, 2002

lore
Share & tools
lore
#0 lore raw

This pretty much sums it up:

[aaronl@vitelus:~]$ time cat mail/debian-legal > /dev/null
cat mail/debian-legal > /dev/null  0.00s user 0.02s system 0% cpu 5.565 total
[aaronl@vitelus:~]$ ls -l mail/debian-legal
-rw-------    1 aaronl   mail      7893525 Sep  3 00:42 mail/debian-legal
[aaronl@vitelus:~]$ time cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null
cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null  0.00s user 0.10s system 16% cpu 0.616 total
[aaronl@vitelus:~]$ ls -l /usr/src/linux-2.4.18.tar.bz2 
-rw-r--r--    1 aaronl   aaronl   24161675 Apr 14 11:53

Both files were AFAIK not in any cache, and they are on the same partition.

My current uninformed theory is that this is caused by fragmentation, since the linux tarball was downloaded all at once but the mailbox I'm comparing it to has 1695 messages, each of which having been appended seperately to the file. All of my mailboxes exhibit similarly awful performance.

Do any other filesystems handle this type of thing more gracefully? Is there room for improvement in ext3? Is there any way I can test my theory by seeing how fragmented a certain inode is? What can I do to avoid extensive fragmentation, if it is truely the cause of my issue?

I'm running 2.4.20-pre5, but this is not a recently-introduced problem.

The disk is IDE - nothing fancy, WDC WD200BB-18CAA0. IDE controller is ServerWorks CSB5. However, I've had this problem consistantly on previous hardware.

#1 lore raw

Hi,

Quoted 9 lines · Aaron Lehmann
[aaronl@vitelus:~]$ time cat mail/debian-legal > /dev/null
cat mail/debian-legal > /dev/null  0.00s user 0.02s system 0% cpu 5.565 total
[aaronl@vitelus:~]$ ls -l mail/debian-legal
-rw-------    1 aaronl   mail      7893525 Sep  3 00:42 mail/debian-legal
[aaronl@vitelus:~]$ time cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null
cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null  0.00s user 0.10s system 16% cpu 0.616 total
[aaronl@vitelus:~]$ ls -l /usr/src/linux-2.4.18.tar.bz2 
-rw-r--r--    1 aaronl   aaronl   24161675 Apr 14 11:53
Quoted 8 lines

Both files were AFAIK not in any cache, and they are on the same partition.

My current uninformed theory is that this is caused by fragmentation, since the linux tarball was downloaded all at once but the mailbox I'm comparing it to has 1695 messages, each of which having been appended seperately to the file. All of my mailboxes exhibit similarly awful performance.

Yep, both ext2 and ext3 can get badly fragmented by files which are closed, reopened and appended to frequently like that.

Quoted 1 line

Do any other filesystems handle this type of thing more gracefully?

There are some ideas from recent FFS changes. One thing they now do is to defragment things automatically as a file grows by effectively deleting and then reallocating the last 16 blocks of the file. Fragmentation will still occur, but less so, if we do that.

Cheers,
Stephen

#2 lore raw
Quoted 25 lines · Stephen C. Tweedie

Hi,

Quoted 9 lines · Aaron Lehmann
[aaronl@vitelus:~]$ time cat mail/debian-legal > /dev/null
cat mail/debian-legal > /dev/null  0.00s user 0.02s system 0% cpu 5.565 total
[aaronl@vitelus:~]$ ls -l mail/debian-legal
-rw-------    1 aaronl   mail      7893525 Sep  3 00:42 mail/debian-legal
[aaronl@vitelus:~]$ time cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null
cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null  0.00s user 0.10s system 16% cpu 0.616 total
[aaronl@vitelus:~]$ ls -l /usr/src/linux-2.4.18.tar.bz2 
-rw-r--r--    1 aaronl   aaronl   24161675 Apr 14 11:53
Quoted 8 lines

Both files were AFAIK not in any cache, and they are on the same partition.

My current uninformed theory is that this is caused by fragmentation, since the linux tarball was downloaded all at once but the mailbox I'm comparing it to has 1695 messages, each of which having been appended seperately to the file. All of my mailboxes exhibit similarly awful performance.

Yep, both ext2 and ext3 can get badly fragmented by files which are closed, reopened and appended to frequently like that.

Quoted 7 lines
Quoted 1 line

Do any other filesystems handle this type of thing more gracefully?

There are some ideas from recent FFS changes. One thing they now do is to defragment things automatically as a file grows by effectively deleting and then reallocating the last 16 blocks of the file. Fragmentation will still occur, but less so, if we do that.

Another possible solution is to try to "defer" allocation. For example, in reiser4 (and XFS, I believe) extents are allocated on the transaction commit and as a result, if file was created by several writes, it will still be allocated as one extent.

Quoted 3 lines

Cheers,
Stephen

Nikita.

#3 lore raw
Quoted 43 lines · Nikita Danilov
Quoted 25 lines · Stephen C. Tweedie

Hi,

Quoted 9 lines · Aaron Lehmann
[aaronl@vitelus:~]$ time cat mail/debian-legal > /dev/null
cat mail/debian-legal > /dev/null  0.00s user 0.02s system 0% cpu 5.565 total
[aaronl@vitelus:~]$ ls -l mail/debian-legal
-rw-------    1 aaronl   mail      7893525 Sep  3 00:42 mail/debian-legal
[aaronl@vitelus:~]$ time cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null
cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null  0.00s user 0.10s system 16% cpu 0.616 total
[aaronl@vitelus:~]$ ls -l /usr/src/linux-2.4.18.tar.bz2 
-rw-r--r--    1 aaronl   aaronl   24161675 Apr 14 11:53
Quoted 8 lines

Both files were AFAIK not in any cache, and they are on the same partition.

My current uninformed theory is that this is caused by fragmentation, since the linux tarball was downloaded all at once but the mailbox I'm comparing it to has 1695 messages, each of which having been appended seperately to the file. All of my mailboxes exhibit similarly awful performance.

Yep, both ext2 and ext3 can get badly fragmented by files which are closed, reopened and appended to frequently like that.

Quoted 7 lines
Quoted 1 line

Do any other filesystems handle this type of thing more gracefully?

There are some ideas from recent FFS changes. One thing they now do is to defragment things automatically as a file grows by effectively deleting and then reallocating the last 16 blocks of the file. Fragmentation will still occur, but less so, if we do that.

Another possible solution is to try to "defer" allocation. For example, in reiser4 (and XFS, I believe) extents are allocated on the transaction commit and as a result, if file was created by several writes, it will still be allocated as one extent.

Quoted 3 lines

Cheers,
Stephen

Quoted 10 lines

Nikita.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

The FFS approach has an advantage for the case where the file grows too slowly for allocation to be delayed.

I think I prefer that we implement a repacker for reiser4 though, as that, combined with delayed allocation, will be a balanced and thorough solution.

Hans

#4 lore raw
Quoted 3 lines · Hans Reiser

I think I prefer that we implement a repacker for reiser4 though, as that, combined with delayed allocation, will be a balanced and thorough solution.

How does current ReiserFS fare against extreme fragmentation? What about XFS? Without trying to risk a flamewar, what Linux filesystems are the most preventive of fragmentation?

The filesystem could make a huge difference on a machine like a mail server...

#5 lore raw
Quoted 7 lines · Aaron Lehmann
Quoted 3 lines · Hans Reiser

I think I prefer that we implement a repacker for reiser4 though, as that, combined with delayed allocation, will be a balanced and thorough solution.

Quoted 10 lines

How does current ReiserFS fare against extreme fragmentation? What about XFS? Without trying to risk a flamewar, what Linux filesystems are the most preventive of fragmentation?

The filesystem could make a huge difference on a machine like a mail server...

Sometimes it is best to confess that one does not have the expertise appropriate for answering a question. Someone on our mailing list studied it carefully though. Perhaps they can comment.

Hans

#6 lore raw
In reply to Nikita Danilov

Hi,

Quoted 4 lines · Nikita Danilov

Another possible solution is to try to "defer" allocation. For example, in reiser4 (and XFS, I believe) extents are allocated on the transaction commit and as a result, if file was created by several writes, it will still be allocated as one extent.

Ext2 has a preallocation mechanism so that if you have multiple writes, they get dealt with to some extent as a single allocation. However, that doesn't work over close(): the preallocated blocks are discarded wheneven we close the file.

The problem with mail files, though, is that they tend to grow quite slowly, so the writes span very many transactions and we don't have that opportunity for coalescing the writes. Actively defragmenting on writes is an alternative in that case.

Cheers,
Stephen

#7 lore raw
Quoted 9 lines · Stephen C. Tweedie

Ext2 has a preallocation mechanism so that if you have multiple writes, they get dealt with to some extent as a single allocation. However, that doesn't work over close(): the preallocated blocks are discarded wheneven we close the file.

The problem with mail files, though, is that they tend to grow quite slowly, so the writes span very many transactions and we don't have that opportunity for coalescing the writes. Actively defragmenting on writes is an alternative in that case.

We recently switched a large mail spool from ext2 to ext3 with default journalling, and we are now having huge problems with disk I/O load.

We have fsync and friends disabled for performance reasons. With ext2, the machine would happily hum along with an average load of 0.2 and a usual 400 kB - 800 kB write every 5 seconds, with about 10 kB/sec read in every second.

Now with ext3, the machine has a load average of about 15 and writing happens almost all of the time. "vmstat 1" output:

procs memory swap io system cpu

 r  b  w  swpd  free   buff  cache  si so  bi   bo   in    cs  us sy id
 0 42  2 79368 47196 100456 1080348  0  0   0 3036 2514  2077  18 21 60
 0 76  2 79368 44264 100456 1080348  0  0   0 1776 1266   823   4  3 92
 0 111 3 79368 41248 100456 1080348  0  0   0 1952 1176   722   4  5 91
 0 132 2 79368 39432 100460 1080348  0  0   0 1368 1007   612   1  3 96
 0 67  3 79368 34412 100460 1080628  0  0   0 2884 1968  1246  18 13 69
 0 41  2 79368 36572 100468 1080828  0  0  24 4020 2661  1530  16 21 64
 0 32  3 79368 31736 100500 1081456  0  0   0 3688 2696  2061  26 22 52
 0 39  3 79368 24588 100528 1082164  0  0   4 3800 2636  2643  30 21 50
 0 32  4 79368 21500 100536 1082832  0  0  24 3216 2404  2419  32 15 54
 5 28  2 79368 18160 100536 1083360  0  0   0 3416 2372  2164  24 19 57
 0 25  4 79368 19748 100552 1082896  0  0   4 4120 2544  2421  17 21 62
 4 16  4 79368 18216 100560 1083284  0  0   0 3532 2115  2361  20 17 63
 0 37  2 79368 17240 100568 1083456  0  0  16 2376 1817  1691   8 12 80
 1 67  3 79368 15112 100568 1083456  4  0   4 1644 1051   723   6  4 90
 1 88  3 79368 12028 100572 1083464  0  0   8 1884 1102   684   6  3 91
 0 108 3 79368 10132 100572 1083468  0  0   0 1716  924   503   3  3 94
15  0  2 79368 14460 100548 1081996  0  0  12 3852 2609  2000  17 25 59
 0 39  3 79368 13252 100576 1082220  0  0  52 4288 2740  2095  19 19 62

This box is primarily running a POP3 server (written in-house to cache mbox offsets, so that it can handle a huge volume of mail), and also exports the mail spool via NFS to other servers which run exim (-fsync). nfsd is exported async. Everything is mounted noatime, nodiratime. No applications should be calling sync/fsync/fdatasync or using O_SYNC. It's a mail server, so everything is fragmented.

We're using dotlocking. Would this cause metadata journalling? We had to hash the mail spool a long time ago do to system time eating all CPU (the ext2 linear directory scan to find a slot available in the spool directory to add the dotlock file). I estimate about 200 - 300 dotlock files are created per second, but these should all be asynchronous. Would switching to fctnl() locking (if this works over NFS) solve the problem?

A "ps -eo pid,stat,args,wchan | grep simpopd | grep ' D '" shows POP3 processes stuck in either "down" or in "do_get_write_access", which appears to be a journal function.

We notice there are some ext3 updates included as a patch to vanilla 2.4.18 in the newest Red Hat kernel, including changes to the do_get_write_access function. Have improvements in this area been made?

Thanks!

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#8 lore raw
Quoted 8 lines · Simon Kirby

We recently switched a large mail spool from ext2 to ext3 with default journalling, and we are now having huge problems with disk I/O load.

We have fsync and friends disabled for performance reasons. With ext2, the machine would happily hum along with an average load of 0.2 and a usual 400 kB - 800 kB write every 5 seconds, with about 10 kB/sec read in every second.

Quoted 6 lines

This box is primarily running a POP3 server (written in-house to cache mbox offsets, so that it can handle a huge volume of mail), and also exports the mail spool via NFS to other servers which run exim (-fsync). nfsd is exported async. Everything is mounted noatime, nodiratime. No applications should be calling sync/fsync/fdatasync or using O_SYNC. It's a mail server, so everything is fragmented.

Hmm, it seems strange (and rather unsafe) that you would run a mail spool without using sync I/O. Unfortunate too, because sync I/O with a large journal (and perhaps an external journal disk) would give you very fast throughput on ext3.

Quoted 3 lines

We're using dotlocking. Would this cause metadata journalling? I estimate about 200 - 300 dotlock files are created per second, but these should all be asynchronous.

Lots of it. So, that is 250 * (1 dir block + 1 inode bitmap + 1 inode table block (+ 1 block bitmap + 1 data block, if there is data in the dotlock file)) = 1250 blocks/second or so.

Quoted 2 lines

Would switching to fctnl() locking (if this works over NFS) solve the problem?

Probably (no disk I/O generated), but I don't know the state of NFS locking.

Quoted 3 lines

We had to hash the mail spool a long time ago do to system time eating all CPU (the ext2 linear directory scan to find a slot available in the spool directory to add the dotlock file).

One reason why we are adding hash-indexed directory support to ext3, so that you don't have to implement it in each application.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

#9 lore raw
In reply to Simon Kirby
Quoted 7 lines · Simon Kirby

This box is primarily running a POP3 server (written in-house to cache mbox offsets, so that it can handle a huge volume of mail), and also exports the mail spool via NFS to other servers which run exim (-fsync). nfsd is exported async. Everything is mounted noatime, nodiratime. No applications should be calling sync/fsync/fdatasync or using O_SYNC. It's a mail server, so everything is fragmented.

Quoted 7 lines

We're using dotlocking. Would this cause metadata journalling? We had to hash the mail spool a long time ago do to system time eating all CPU (the ext2 linear directory scan to find a slot available in the spool directory to add the dotlock file). I estimate about 200 - 300 dotlock files are created per second, but these should all be asynchronous. Would switching to fctnl() locking (if this works over NFS) solve the problem?

I'd absolutly go to fcntl(). As bad as dotlocking is for journaling filesystems it is even worse for NFS (when it works). Look at the lkml thread "invalidate_inode_pages in 2.5.32/3" to get an idea. Multiply the directory invalidations by the size of the directories. fcntl() is the preferred way of locking over NFS as it will even report if there is a problem with lockd.

Signature · 5 lines
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt