background on the ext3 batching performance issue

At the LSF workshop, I mentioned that we have tripped across an embarrassing performance issue in the jbd transaction code which is clearly not tuned for low latency devices.

The short summary is that we can do say 800 10k files/sec in a write/fsync/close loop with a single thread, but drop down to under 250 files/sec with 2 or more threads.

This is pretty easy to reproduce with any small file write synchronous workload (i.e., fsync() each file before close). We used my fs_mark tool to reproduce.

The core of the issue is the call in the jbd transaction code call out to schedule_timeout_uninterruptible(1) which causes us to sleep for 4ms:

        pid = current->pid;
        if (handle->h_sync && journal->j_last_sync_writer != pid) {
                journal->j_last_sync_writer = pid;
                do {
                        old_handle_count = transaction->t_handle_count;
                        schedule_timeout_uninterruptible(1);
                } while (old_handle_count != transaction->t_handle_count);
        }

This is quite topical to the concern we had with low latency devices in general, but specifically things like SSD's.

Quoted 22 lines · Ric Wheeler

At the LSF workshop, I mentioned that we have tripped across an embarrassing performance issue in the jbd transaction code which is clearly not tuned for low latency devices.

The short summary is that we can do say 800 10k files/sec in a write/fsync/close loop with a single thread, but drop down to under 250 files/sec with 2 or more threads.

This is pretty easy to reproduce with any small file write synchronous workload (i.e., fsync() each file before close). We used my fs_mark tool to reproduce.

The core of the issue is the call in the jbd transaction code call out to schedule_timeout_uninterruptible(1) which causes us to sleep for 4ms:

        pid = current->pid;
        if (handle->h_sync && journal->j_last_sync_writer != pid) {
                journal->j_last_sync_writer = pid;
                do {
                        old_handle_count = transaction->t_handle_count;
                        schedule_timeout_uninterruptible(1);
                } while (old_handle_count !=

Quoted 4 lines

transaction->t_handle_count); }

This is quite topical to the concern we had with low latency devices in general, but specifically things like SSD's.

Your testcase does in fact show a weakness in this optimization, but look at the more likely case, where you have multiple writers on the same filesystem rather than one guy doing write/fsync. If we wait we could potentially add quite a few more buffers to this transaction before flushing it, rather than flushing a buffer or two at a time. What would you propose as a solution?

Quoted 25 lines · Josef Bacik

Quoted 24 lines · Ric Wheeler

At the LSF workshop, I mentioned that we have tripped across an embarrassing performance issue in the jbd transaction code which is clearly not tuned for low latency devices.

The short summary is that we can do say 800 10k files/sec in a write/fsync/close loop with a single thread, but drop down to under 250 files/sec with 2 or more threads.

This is pretty easy to reproduce with any small file write synchronous workload (i.e., fsync() each file before close). We used my fs_mark tool to reproduce.

The core of the issue is the call in the jbd transaction code call out to schedule_timeout_uninterruptible(1) which causes us to sleep for 4ms:

        pid = current->pid;
        if (handle->h_sync && journal->j_last_sync_writer != pid) {
                journal->j_last_sync_writer = pid;
                do {
                        old_handle_count = transaction->t_handle_count;
                        schedule_timeout_uninterruptible(1);
                } while (old_handle_count !=

transaction->t_handle_count); }

Quoted 9 lines

Quoted 2 lines

This is quite topical to the concern we had with low latency devices in general, but specifically things like SSD's.

Your testcase does in fact show a weakness in this optimization, but look at the more likely case, where you have multiple writers on the same filesystem rather than one guy doing write/fsync. If we wait we could potentially add quite a few more buffers to this transaction before flushing it, rather than flushing a buffer or two at a time. What would you propose as a solution?

Forgive me, I said that badly, now that I've had my morning coffee let me try again. You are ping-ponging the j_last_sync_writer back and forth between the two threads, so you don't get the speedup you would get with one thread where we would just bypass the next sleep since we know we've got one thread doing write/sync. So this brings up the question, should we try and figure out if we have the situation where we have multiple threads doing write/sync and therefore exploiting the weakness in this optimization, and if we should, how would we do this properly? The only thing I can think to do is to track sync writers on a transaction, and if its more than one bypass this little snippet. In fact I think I'll go ahead and do that and see what fs_mark comes up with. Thank you,

Josef

Quoted 25 lines · Josef Bacik

Quoted 24 lines · Ric Wheeler

At the LSF workshop, I mentioned that we have tripped across an embarrassing performance issue in the jbd transaction code which is clearly not tuned for low latency devices.

The short summary is that we can do say 800 10k files/sec in a write/fsync/close loop with a single thread, but drop down to under 250 files/sec with 2 or more threads.

This is pretty easy to reproduce with any small file write synchronous workload (i.e., fsync() each file before close). We used my fs_mark tool to reproduce.

The core of the issue is the call in the jbd transaction code call out to schedule_timeout_uninterruptible(1) which causes us to sleep for 4ms:

        pid = current->pid;
        if (handle->h_sync && journal->j_last_sync_writer != pid) {
                journal->j_last_sync_writer = pid;
                do {
                        old_handle_count = transaction->t_handle_count;
                        schedule_timeout_uninterruptible(1);
                } while (old_handle_count !=

transaction->t_handle_count); }

Quoted 9 lines

Quoted 2 lines

This is quite topical to the concern we had with low latency devices in general, but specifically things like SSD's.

Your testcase does in fact show a weakness in this optimization, but look at the more likely case, where you have multiple writers on the same filesystem rather than one guy doing write/fsync. If we wait we could potentially add quite a few more buffers to this transaction before flushing it, rather than flushing a buffer or two at a time. What would you propose as a solution?

Forgive me, I said that badly, now that I've had my morning coffee let me try again. You are ping-ponging the j_last_sync_writer back and forth between the two threads, so you don't get the speedup you would get with one thread where we would just bypass the next sleep since we know we've got one thread doing write/sync. So this brings up the question, should we try and figure out if we have the situation where we have multiple threads doing write/sync and therefore exploiting the weakness in this optimization, and if we should, how would we do this properly? The only thing I can think to do is to track sync writers on a transaction, and if its more than one bypass this little snippet. In fact I think I'll go ahead and do that and see what fs_mark comes up with. Thank you,

Josef

Quoted 48 lines · Josef Bacik

Quoted 25 lines · Josef Bacik

Quoted 24 lines · Ric Wheeler

At the LSF workshop, I mentioned that we have tripped across an embarrassing performance issue in the jbd transaction code which is clearly not tuned for low latency devices.

The short summary is that we can do say 800 10k files/sec in a write/fsync/close loop with a single thread, but drop down to under 250 files/sec with 2 or more threads.

This is pretty easy to reproduce with any small file write synchronous workload (i.e., fsync() each file before close). We used my fs_mark tool to reproduce.

The core of the issue is the call in the jbd transaction code call out to schedule_timeout_uninterruptible(1) which causes us to sleep for 4ms:

       pid = current->pid;
       if (handle->h_sync && journal->j_last_sync_writer != pid) {
               journal->j_last_sync_writer = pid;
               do {
                       old_handle_count = transaction->t_handle_count;
                       schedule_timeout_uninterruptible(1);
               } while (old_handle_count !=

transaction->t_handle_count); }

Quoted 9 lines

Quoted 2 lines

This is quite topical to the concern we had with low latency devices in general, but specifically things like SSD's.

Your testcase does in fact show a weakness in this optimization, but look at the more likely case, where you have multiple writers on the same filesystem rather than one guy doing write/fsync. If we wait we could potentially add quite a few more buffers to this transaction before flushing it, rather than flushing a buffer or two at a time. What would you propose as a solution?

Forgive me, I said that badly, now that I've had my morning coffee let me try again. You are ping-ponging the j_last_sync_writer back and forth between the two threads, so you don't get the speedup you would get with one thread where we would just bypass the next sleep since we know we've got one thread doing write/sync. So this brings up the question, should we try and figure out if we have the situation where we have multiple threads doing write/sync and therefore exploiting the weakness in this optimization, and if we should, how would we do this properly? The only thing I can think to do is to track sync writers on a transaction, and if its more than one bypass this little snippet. In fact I think I'll go ahead and do that and see what fs_mark comes up with. Thank you,

Quoted 2 lines

Josef

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

[ fsync batching can be slow ]

Quoted 13 lines

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

Quoted 3 lines

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

It probably makes sense to keep track of the average number of writers we are able to gather into a transcation. There are lots of similar workloads where we have a pool of procs doing fsyncs and the size of the transaction or the number of times we joined a running transaction will be fairly constant.

Quoted 21 lines · Ric Wheeler

[ fsync batching can be slow ]

Quoted 9 lines

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

Quoted 7 lines

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

Quoted 5 lines

It probably makes sense to keep track of the average number of writers we are able to gather into a transcation. There are lots of similar workloads where we have a pool of procs doing fsyncs and the size of the transaction or the number of times we joined a running transaction will be fairly constant.

I'm probably missing something, but what are you trying to say? Either we
wait for writers and the number of writes is higher, or we don't wait and the number of writes in a transaction is lower...

Quoted 27 lines · Jan Kara

Quoted 21 lines · Ric Wheeler

[ fsync batching can be slow ]

Quoted 9 lines

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

Quoted 7 lines

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

Quoted 5 lines

It probably makes sense to keep track of the average number of writers we are able to gather into a transcation. There are lots of similar workloads where we have a pool of procs doing fsyncs and the size of the transaction or the number of times we joined a running transaction will be fairly constant.

Quoted 3 lines

I'm probably missing something, but what are you trying to say? Either we
wait for writers and the number of writes is higher, or we don't wait and the number of writes in a transaction is lower...

The common workload would be N mail server threads servicing incoming requests at a fairly constant rate. Right now we sleep for a bit and wait for the number of writers to increase.

My guess is that if we record the average number of times a writer joins an existing transaction, or if we record the average size of the transactions, we'll end up with a fairly constant number.

So, we can skip the sleep if the transaction has already grown close to that number. This would avoid the latencies Ric is seeing.

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

Quoted 7 lines · Ric Wheeler

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

It's the self-tuning aspect of this problem that makes it hard. In the case of XFS, the way this tuning is done is that we look at the state of the previous log I/O buffer to check if it is still syncing to disk. If it is sync to disk, we go to sleep waiting for that log buffer I/O to complete. This holds the current buffer open to aggregate more transactions before syncing it to disk and hence allows parallel fsyncs to be issued in the one log write. The fact that it waits for the previous log I/O to complete means it self-tunes to the latency of the underlying storage medium.....

Cheers,

Dave.

Quoted 7 lines · Ric Wheeler

One more thought - what we really want here is to have a sense of the latency of the device. In the S-ATA disk case, this optimization works well for batching since we "spend" an extra 4ms worst case in the chance of combining multiple, slow 18ms operations.

With the clariion box we tested, the optimization fails badly since the cost is only 1.3 ms so we optimize by waiting 3-4 times longer than it would take to do the operation immediately.

This problem has also seemed to me to be the same problem that IO schedulers do with plugging - we want to dynamically figure out when to plug and unplug here without hard coding in device specific tunings.

If we bypass the snippet for multi-threaded writers, we would probably slow down this workload on normal S-ATA/ATA drives (or even higher performance non-RAID disks).

It's the self-tuning aspect of this problem that makes it hard. In the case of XFS, the way this tuning is done is that we look at the state of the previous log I/O buffer to check if it is still syncing to disk. If it is sync to disk, we go to sleep waiting for that log buffer I/O to complete. This holds the current buffer open to aggregate more transactions before syncing it to disk and hence allows parallel fsyncs to be issued in the one log write. The fact that it waits for the previous log I/O to complete means it self-tunes to the latency of the underlying storage medium.....

Cheers,

Dave.

After the IO/FS workshop last week, I posted some details on the slow down we see with ext3 when we have a low latency back end instead of a normal local disk (SCSI/S-ATA/etc).

As a follow up to that thread, I wanted to post some real numbers that Andy from our performance team pulled together. Andy tested various patches using three classes of storage (S-ATA, RAM disk and Clariion array).

Note that this testing was done on a SLES10/SP1 kernel, but the code in question has not changed in mainline but we should probably retest on something newer just to clear up any doubts.

The work load is generated using fs_mark
(http://sourceforge.net/projects/fsmark/) which is basically a write workload with small files, each file gets fsync'ed before close. The metric is "files/sec".

The clearest result used a ramdisk to store 4k files.

We modified ext3 and jbd to accept a new mount option: bdelay Use it like:

mount -o bdelay=n dev mountpoint

n is passed to schedule_timeout_interruptible() in the jbd code. if n == 0, it skips the whole loop. if n is "yield", then substitute the schedule...(n) with yield().

Note that the first row is the value of the delay with a 250HZ build followed by the number of concurrent threads writing 4KB files.

Ramdisk test:

bdelay  1       2       4       8       10      20
0       4640    4498    3226    1721    1436     664
yield   4640    4078    2977    1611    1136     551
1       4647     250     482     588     629     483
2       4522     149     233     422     450     389
3       4504      86     165     271     308     334
4       4425      84     128     222     253     293

Midrange clariion:

bdelay   1       2       4       8       10      20
0        778     923    1567    1424    1276     785
yield    791     931    1551    1473    1328     806
1        793     304     499     714     751     760
2        789     132     201     382     441     589
3        792     124     168     298     342     471
4        786      71     116     237     277     393

Local disk:

bdelay    1       2       4       8       10      20
0         47      51      81     135     160     234
yield     36      45      74     117     138     214
1         44      52      86     148     183     258
2         40      60     109     163     184     265
3         40      52      97     148     171     264
4         35      42      83     149     169     246

Apologies for mangling the nicely formatted tables.

Note that the justification for the batching as we have it today is basically this last local drive test case.

It would be really interesting to rerun some of these tests on xfs which Dave explained in the thread last week has a more self tuning way to batch up transactions....

Note that all of those poor users who have a synchronous write workload today are in the "1" row for each of the above tables.

Quoted 3 lines · Ric Wheeler

After the IO/FS workshop last week, I posted some details on the slow down we see with ext3 when we have a low latency back end instead of a normal local disk (SCSI/S-ATA/etc).

Quoted 6 lines

It would be really interesting to rerun some of these tests on xfs which Dave explained in the thread last week has a more self tuning way to batch up transactions....

Note that all of those poor users who have a synchronous write workload today are in the "1" row for each of the above tables.

Mind giving this a whirl? The fastest thing I've got here is an Apple X RAID and its being used for something else atm, so I've only tested this on local disk to make sure it didn't make local performance suck (which it doesn't btw). This should be equivalent with what David says XFS does. Thanks much,

Josef

fs/jbd/transaction.c +7 -25

index c6cbb6c..4596e1c 100644
@@ -1333,8 +1333,7 @@ int journal_stop(handle_t *handle)
 {
 	transaction_t *transaction = handle->h_transaction;
 	journal_t *journal = transaction->t_journal;
-	int old_handle_count, err;
-	pid_t pid;
+	int err;
 
 	J_ASSERT(journal_current_handle() == handle);
 

@@ -1353,32 +1352,22 @@ int journal_stop(handle_t *handle)
 
 	jbd_debug(4, "Handle %p going down\n", handle);
 
-	/*
-	 * Implement synchronous transaction batching.  If the handle
-	 * was synchronous, don't force a commit immediately.  Let's
-	 * yield and let another thread piggyback onto this transaction.
-	 * Keep doing that while new threads continue to arrive.
-	 * It doesn't cost much - we're about to run a commit and sleep
-	 * on IO anyway.  Speeds up many-threaded, many-dir operations
-	 * by 30x or more...
-	 *
-	 * But don't do this if this process was the most recent one to
-	 * perform a synchronous write.  We do this to detect the case where a
-	 * single process is doing a stream of sync writes.  No point in waiting
-	 * for joiners in that case.
-	 */
-	pid = current->pid;
-	if (handle->h_sync && journal->j_last_sync_writer != pid) {
-		journal->j_last_sync_writer = pid;
-		do {
-			old_handle_count = transaction->t_handle_count;
-			schedule_timeout_uninterruptible(1);
-		} while (old_handle_count != transaction->t_handle_count);
-	}
-
 	current->journal_info = NULL;
 	spin_lock(&journal->j_state_lock);
 	spin_lock(&transaction->t_handle_lock);
+
+	if (journal->j_committing_transaction && handle->h_sync) {
+		tid_t tid = journal->j_committing_transaction->t_tid;
+
+		spin_unlock(&transaction->t_handle_lock);
+		spin_unlock(&journal->j_state_lock);

Quoted 4 lines · Josef Bacik

Quoted 3 lines · Ric Wheeler

After the IO/FS workshop last week, I posted some details on the slow down we see with ext3 when we have a low latency back end instead of a normal local disk (SCSI/S-ATA/etc).

...
...
...

Quoted 65 lines

Quoted 6 lines

It would be really interesting to rerun some of these tests on xfs which Dave explained in the thread last week has a more self tuning way to batch up transactions....

Note that all of those poor users who have a synchronous write workload today are in the "1" row for each of the above tables.

Mind giving this a whirl? The fastest thing I've got here is an Apple X RAID and its being used for something else atm, so I've only tested this on local disk to make sure it didn't make local performance suck (which it doesn't btw). This should be equivalent with what David says XFS does.
Thanks much,

Josef

fs/jbd/transaction.c +1 -14

index c6cbb6c..4596e1c 100644
@@ -1333,8 +1333,7 @@ int journal_stop(handle_t *handle)
 {
 	transaction_t *transaction = handle->h_transaction;
 	journal_t *journal = transaction->t_journal;
-	int old_handle_count, err;
-	pid_t pid;
+	int err;
 
 	J_ASSERT(journal_current_handle() == handle);
 

@@ -1353,32 +1352,22 @@ int journal_stop(handle_t *handle)
 
 	jbd_debug(4, "Handle %p going down\n", handle);
 
-	/*
-	 * Implement synchronous transaction batching.  If the handle
-	 * was synchronous, don't force a commit immediately.  Let's
-	 * yield and let another thread piggyback onto this transaction.
-	 * Keep doing that while new threads continue to arrive.
-	 * It doesn't cost much - we're about to run a commit and sleep
-	 * on IO anyway.  Speeds up many-threaded, many-dir operations
-	 * by 30x or more...
-	 *
-	 * But don't do this if this process was the most recent one to
-	 * perform a synchronous write.  We do this to detect the case where a
-	 * single process is doing a stream of sync writes.  No point in

waiting - * for joiners in that case.

fs/jbd/transaction.c +6 -10

-	 */
-	pid = current->pid;
-	if (handle->h_sync && journal->j_last_sync_writer != pid) {
-		journal->j_last_sync_writer = pid;
-		do {
-			old_handle_count = transaction->t_handle_count;
-			schedule_timeout_uninterruptible(1);
-		} while (old_handle_count != transaction->t_handle_count);
-	}
-
 	current->journal_info = NULL;
 	spin_lock(&journal->j_state_lock);
 	spin_lock(&transaction->t_handle_lock);
+
+	if (journal->j_committing_transaction && handle->h_sync) {
+		tid_t tid = journal->j_committing_transaction->t_tid;
+
+		spin_unlock(&transaction->t_handle_lock);
+		spin_unlock(&journal->j_state_lock);

fs/jbd/transaction.c int journal_stop(handle_t *handle) · line 1373

Quoted 10 lines

fs/jbd/transaction.c +7 -0

+
+		err = log_wait_commit(journal, tid);
+
+		spin_lock(&journal->j_state_lock);
+		spin_lock(&transaction->t_handle_lock);
+	}
+
 	transaction->t_outstanding_credits -= handle->h_buffer_credits;
 	transaction->t_updates--;
 	if (!transaction->t_updates) {

Running with Josef's patch, I was able to see a clear improvement for batching these synchronous operations on ext3 with the RAM disk and array. It is not too often that you get to do a simple change and see a 27 times improvement ;-)

On the bad side, the local disk case took as much as a 30% drop in performance. The specific disk is not one that I have a lot of experience with, I would like to retry on a disk that has been qualified
by our group (i.e., we have reasonable confidence that there are no
firmware issues, etc).

Now for the actual results.

The results are the average value of 5 runs for each number of threads.

Type Threads Baseline Josef Speedup (Josef/Baseline)

array	    1	     320.5      325.4      1.01
array	    2	     174.9      351.9      2.01
array	    4	     382.7      593.5      1.55
array	    8	     644.1      963.0      1.49
array	    10	     842.9     1038.7      1.23
array	    20	    1319.6     1432.3      1.08

RAM disk    1       5621.4     5595.1      0.99
RAM disk    2        281.5     7613.3     27.04
RAM disk    4        579.9     9111.5     15.71
RAM disk    8        891.1     9357.3     10.50
RAM disk    10      1116.3     9873.6      8.84
RAM disk    20      1952.0    10703.6      5.48

S-ATA disk  1         19.0       15.1      0.79
S-ATA disk  2         19.9       14.4      0.72
S-ATA disk  4         41.0       27.9      0.68
S-ATA disk  8         60.4       43.2      0.71
S-ATA disk  10        67.1       48.7      0.72
S-ATA disk  20       102.7       74.0      0.72

Background on the tests:

All of this is measured on three devices - a relatively old & slow array, the local (slow!) 2.5" S-ATA disk in the box and a RAM disk.

These numbers are used fs_mark to write 4096 byte files with the following commands:

fs_mark -d /home/test/t -s 4096 -n 40000 -N 50 -D 64 -t 1
...
fs_mark -d /home/test/t -s 4096 -n 20000 -N 50 -D 64 -t 2
...
fs_mark -d /home/test/t -s 4096 -n 10000 -N 50 -D 64 -t 4
...
fs_mark -d /home/test/t -s 4096 -n 5000 -N 50 -D 64 -t 8
...
fs_mark -d /home/test/t -s 4096 -n 4000 -N 50 -D 64 -t 10
...
fs_mark -d /home/test/t -s 4096 -n 2000 -N 50 -D 64 -t 20
...

Note that this spreads the files across 64 subdirectories, each thread writes 50 files and then moves on to the next in a round robin.

Quoted 141 lines · Ric Wheeler

Quoted 4 lines · Josef Bacik

Quoted 3 lines · Ric Wheeler

After the IO/FS workshop last week, I posted some details on the slow down we see with ext3 when we have a low latency back end instead of a normal local disk (SCSI/S-ATA/etc).

...
...
...

Quoted 64 lines

Quoted 6 lines

It would be really interesting to rerun some of these tests on xfs which Dave explained in the thread last week has a more self tuning way to batch up transactions....

Note that all of those poor users who have a synchronous write workload today are in the "1" row for each of the above tables.

Mind giving this a whirl? The fastest thing I've got here is an Apple X RAID and its being used for something else atm, so I've only tested this on local disk to make sure it didn't make local performance suck (which it doesn't btw). This should be equivalent with what David says XFS does.
Thanks much,

Josef

fs/jbd/transaction.c +1 -14

index c6cbb6c..4596e1c 100644
@@ -1333,8 +1333,7 @@ int journal_stop(handle_t *handle)
 {
 	transaction_t *transaction = handle->h_transaction;
 	journal_t *journal = transaction->t_journal;
-	int old_handle_count, err;
-	pid_t pid;
+	int err;
 
 	J_ASSERT(journal_current_handle() == handle);
 

@@ -1353,32 +1352,22 @@ int journal_stop(handle_t *handle)
 
 	jbd_debug(4, "Handle %p going down\n", handle);
 
-	/*
-	 * Implement synchronous transaction batching.  If the handle
-	 * was synchronous, don't force a commit immediately.  Let's
-	 * yield and let another thread piggyback onto this transaction.
-	 * Keep doing that while new threads continue to arrive.
-	 * It doesn't cost much - we're about to run a commit and sleep
-	 * on IO anyway.  Speeds up many-threaded, many-dir operations
-	 * by 30x or more...
-	 *
-	 * But don't do this if this process was the most recent one to
-	 * perform a synchronous write.  We do this to detect the case where a
-	 * single process is doing a stream of sync writes.  No point in

waiting - * for joiners in that case.

fs/jbd/transaction.c +6 -10

-	 */
-	pid = current->pid;
-	if (handle->h_sync && journal->j_last_sync_writer != pid) {
-		journal->j_last_sync_writer = pid;
-		do {
-			old_handle_count = transaction->t_handle_count;
-			schedule_timeout_uninterruptible(1);
-		} while (old_handle_count != transaction->t_handle_count);
-	}
-
 	current->journal_info = NULL;
 	spin_lock(&journal->j_state_lock);
 	spin_lock(&transaction->t_handle_lock);
+
+	if (journal->j_committing_transaction && handle->h_sync) {
+		tid_t tid = journal->j_committing_transaction->t_tid;
+
+		spin_unlock(&transaction->t_handle_lock);
+		spin_unlock(&journal->j_state_lock);

fs/jbd/transaction.c int journal_stop(handle_t *handle) · line 1373

Quoted 10 lines

fs/jbd/transaction.c +7 -0

+
+		err = log_wait_commit(journal, tid);
+
+		spin_lock(&journal->j_state_lock);
+		spin_lock(&transaction->t_handle_lock);
+	}
+
 	transaction->t_outstanding_credits -= handle->h_buffer_credits;
 	transaction->t_updates--;
 	if (!transaction->t_updates) {

Running with Josef's patch, I was able to see a clear improvement for batching these synchronous operations on ext3 with the RAM disk and array. It is not too often that you get to do a simple change and see a 27 times improvement ;-)

On the bad side, the local disk case took as much as a 30% drop in performance. The specific disk is not one that I have a lot of experience with, I would like to retry on a disk that has been qualified
by our group (i.e., we have reasonable confidence that there are no
firmware issues, etc).

Now for the actual results.

The results are the average value of 5 runs for each number of threads.

Type Threads Baseline Josef Speedup (Josef/Baseline)

array	    1	     320.5      325.4      1.01
array	    2	     174.9      351.9      2.01
array	    4	     382.7      593.5      1.55
array	    8	     644.1      963.0      1.49
array	    10	     842.9     1038.7      1.23
array	    20	    1319.6     1432.3      1.08

RAM disk    1       5621.4     5595.1      0.99
RAM disk    2        281.5     7613.3     27.04
RAM disk    4        579.9     9111.5     15.71
RAM disk    8        891.1     9357.3     10.50
RAM disk    10      1116.3     9873.6      8.84
RAM disk    20      1952.0    10703.6      5.48

S-ATA disk  1         19.0       15.1      0.79
S-ATA disk  2         19.9       14.4      0.72
S-ATA disk  4         41.0       27.9      0.68
S-ATA disk  8         60.4       43.2      0.71
S-ATA disk  10        67.1       48.7      0.72
S-ATA disk  20       102.7       74.0      0.72

Background on the tests:

All of this is measured on three devices - a relatively old & slow array, the local (slow!) 2.5" S-ATA disk in the box and a RAM disk.

These numbers are used fs_mark to write 4096 byte files with the following commands:

fs_mark -d /home/test/t -s 4096 -n 40000 -N 50 -D 64 -t 1
...
fs_mark -d /home/test/t -s 4096 -n 20000 -N 50 -D 64 -t 2
...
fs_mark -d /home/test/t -s 4096 -n 10000 -N 50 -D 64 -t 4
...
fs_mark -d /home/test/t -s 4096 -n 5000 -N 50 -D 64 -t 8
...
fs_mark -d /home/test/t -s 4096 -n 4000 -N 50 -D 64 -t 10
...
fs_mark -d /home/test/t -s 4096 -n 2000 -N 50 -D 64 -t 20
...

Quoted 3 lines

Note that this spreads the files across 64 subdirectories, each thread writes 50 files and then moves on to the next in a round robin.

I'm starting to wonder about the disks I have, because my files/second is spanking yours, and its just a local samsung 3gb/s sata drive. With those commands I'm consistently getting over 700 files/sec. I'm seeing about a 1-5% increase in speed locally with my patch. I guess I'll start looking around for some other hardware and check on there in case this box is more badass than I think it is. Thanks much,

Josef

Quoted 13 lines · Josef Bacik

Quoted 11 lines · Ric Wheeler

Quoted 4 lines · Josef Bacik

Quoted 3 lines · Ric Wheeler

After the IO/FS workshop last week, I posted some details on the slow down we see with ext3 when we have a low latency back end instead of a normal local disk (SCSI/S-ATA/etc).

...
...

Note that this spreads the files across 64 subdirectories, each thread writes 50 files and then moves on to the next in a round robin.

Quoted 9 lines

I'm starting to wonder about the disks I have, because my files/second is spanking yours, and its just a local samsung 3gb/s sata drive. With those commands I'm consistently getting over 700 files/sec. I'm seeing about a 1-5% increase in speed locally with my patch. I guess I'll start looking around for some other hardware and check on there in case this box is more badass than I think it is. Thanks much,

Josef

Sounds like you might be running with write cache on & barriers off ;-)

Make sure you have write cache & barriers enabled on the drive. With a good S-ATA drive, you should be seeing about 35-50 files/sec with a single threaded writer.

The local disk that I tested on is a relatively slow s-ata disk that is more laptop quality/performance than server.

One thought I had about the results is that we might be flipping the IO sequence with the local disk case. It is the only device of the three that I tested which is seek/head movement sensitive for small files.

Quoted 17 lines · Ric Wheeler

Quoted 9 lines · Josef Bacik

Quoted 8 lines · Ric Wheeler

Quoted 4 lines · Josef Bacik

Quoted 3 lines · Ric Wheeler

After the IO/FS workshop last week, I posted some details on the slow down we see with ext3 when we have a low latency back end instead of a normal local disk (SCSI/S-ATA/etc).

...
...

Quoted 7 lines

Quoted 3 lines

Note that this spreads the files across 64 subdirectories, each thread writes 50 files and then moves on to the next in a round robin.

I'm starting to wonder about the disks I have, because my files/second is spanking yours, and its just a local samsung 3gb/s sata drive. With those commands I'm consistently getting over 700 files/sec. I'm seeing about a 1-5% increase in speed locally with my patch. I guess I'll start looking around for some other hardware and check on there in case this box is more badass than I think it is. Thanks much,

Josef

Quoted 8 lines

Sounds like you might be running with write cache on & barriers off ;-)

Make sure you have write cache & barriers enabled on the drive. With a good S-ATA drive, you should be seeing about 35-50 files/sec with a single threaded writer.

The local disk that I tested on is a relatively slow s-ata disk that is more laptop quality/performance than server.

One thought I had about the results is that we might be flipping the IO sequence with the local disk case. It is the only device of the three that I tested which is seek/head movement sensitive for small files.

Ahh yes turning off write cache off and barriers on I get your numbers, however I'm not seeing the slowdown that you are, with and without my patch I'm seeing the same performance. Its just a plane jane intel sata controller with a samsung sata disk set at 1.5gbps. Same thing with an nvidia sata controller. I'll think about this some more and see if there is something better that could be done that may help you. Thanks much,

Josef

The work load is generated using fs_mark
(http://sourceforge.net/projects/fsmark/) which is basically a write workload with small files, each file gets fsync'ed before close. The metric is "files/sec".

It would be really interesting to rerun some of these tests on xfs which Dave explained in the thread last week has a more self tuning way to batch up transactions....