batching support for transactions
After several years of helping tune file systems for normal (ATA/S-ATA) drives, we have been doing some performance work on ext3 & reiserfs on disk arrays.
One thing that jumps out is that the way we currently batch synchronous work loads into transactions does really horrible things to performance for storage devices which have really low latency.
For example, one a mid-range clariion box, we can use a single thread to write around 750 (10240 byte) files/sec to a single directory in ext3. That gives us an average time around 1.3ms per file.
With 2 threads writing to the same directory, we instantly drop down to 234 files/sec.
The culprit seems to be the assumptions in journal_stop() which throw in a call to schedule_timeout_uninterruptible(1):
/*
* Implement synchronous transaction batching. If the handle
* was synchronous, don't force a commit immediately. Let's
* yield and let another thread piggyback onto this transaction.
* Keep doing that while new threads continue to arrive.
* It doesn't cost much - we're about to run a commit and sleep
* on IO anyway. Speeds up many-threaded, many-dir operations
* by 30x or more...
*
* But don't do this if this process was the most recent one to
* perform a synchronous write. We do this to detect the case
where a
* single process is doing a stream of sync writes. No point
in waiting
* for joiners in that case.
*/
pid = current->pid;
if (handle->h_sync && journal->j_last_sync_writer != pid) {
journal->j_last_sync_writer = pid;
do {
old_handle_count = transaction->t_handle_count;
schedule_timeout_uninterruptible(1);
} while (old_handle_count != transaction->t_handle_count);
}reiserfs and ext4 have similar if not exactly the same logic.
What seems to be needed here is either a static per file system/storage device tunable to allow us to change this timeout (maybe with "0" defaulting back to the old reiserfs trick of simply doing a yield()?) or a more dynamic, per device way to keep track of the average time it takes to commit a transaction to disk. Based on that rate, we could dynamically adjust our logic to account for lower latency devices.
A couple of last thoughts. One, if for some reason you don't have a low latency storage array handy and want to test this for yourselves, you can test the worst case by using a ram disk.
The test we used was fs_mark with 10240 bytes files, writing to one shared directory with varying the numbers of threads from 1 up to 40. In the ext3 case, it takes 8 concurrent threads to catch up to the single thread writing case.
We are continuing to play with the code and try out some ideas, but I wanted to bounce this off the broader list to see if this makes sense...
ric
Quoted 10 lines · Ric Wheeler
One thing that jumps out is that the way we currently batch synchronous work loads into transactions does really horrible things to performance for storage devices which have really low latency.
For example, one a mid-range clariion box, we can use a single thread to write around 750 (10240 byte) files/sec to a single directory in ext3. That gives us an average time around 1.3ms per file.
With 2 threads writing to the same directory, we instantly drop down to 234 files/sec.
Is this with HZ=250?
Quoted 11 lines
The culprit seems to be the assumptions in
journal_stop()which throw in a call toschedule_timeout_uninterruptible(1):pid = current->pid; if (handle->h_sync && journal->j_last_sync_writer != pid) { journal->j_last_sync_writer = pid; do { old_handle_count = transaction->t_handle_count; schedule_timeout_uninterruptible(1); } while (old_handle_count != transaction->t_handle_count); }
It would seem one of the problems is that we shouldn't really be scheduling for a fixed 1 jiffie timeout, but rather only until the other threads have a chance to run and join the existing transaction.
Quoted 3 lines
What seems to be needed here is either a static per file system/storage device tunable to allow us to change this timeout (maybe with "0" defaulting back to the old reiserfs trick of simply doing a
yield()?)
Tunables are to be avoided if possible, since they will usually not be set except by the .00001% of people who actually understand them. Using yield() seems like the right thing, but Andrew Morton added this code and my guess would be that yield() doesn't block the first thread long enough for the second one to get into the transaction (e.g. on an 2-CPU system with 2 threads, yield() will likely do nothing).
Quoted 3 lines
or a more dynamic, per device way to keep track of the average time it takes to commit a transaction to disk. Based on that rate, we could dynamically adjust our logic to account for lower latency devices.
It makes sense to track not only the time to commit a single synchronous transaction, but also the time between sync transactions to decide if the initial transaction should be held to allow later ones.
Alternately, it might be possible to check if a new thread is trying to start a sync handle when the previous one was also synchronous and had only a single handle in it, then automatically enable the delay in that case.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
Quoted 8 lines · Andreas Dilger
Quoted 5 lines · Ric Wheeler
One thing that jumps out is that the way we currently batch synchronous work loads into transactions does really horrible things to performance for storage devices which have really low latency.
For example, one a mid-range clariion box, we can use a single thread to write around 750 (10240 byte) files/sec to a single directory in ext3. That gives us an average time around 1.3ms per file.
With 2 threads writing to the same directory, we instantly drop down to 234 files/sec.
Is this with HZ=250?
Yes - I assume that with HZ=1000 the batching would start to work again since the penalty for batching would only be 1ms which would add a 0.3ms overhead while waiting for some other thread to join.
Quoted 12 lines
Quoted 11 lines
The culprit seems to be the assumptions in
journal_stop()which throw in a call toschedule_timeout_uninterruptible(1):pid = current->pid; if (handle->h_sync && journal->j_last_sync_writer != pid) { journal->j_last_sync_writer = pid; do { old_handle_count = transaction->t_handle_count; schedule_timeout_uninterruptible(1); } while (old_handle_count != transaction->t_handle_count); }
Quoted 3 lines
It would seem one of the problems is that we shouldn't really be scheduling for a fixed 1 jiffie timeout, but rather only until the other threads have a chance to run and join the existing transaction.
This is really very similar to the domain of the IO schedulers - when do you hold off an IO and/or try to combine it.
It is hard to predict the future need of threads that will be wanting to do IO, but you can dynamically measure the average time it takes a transaction to commit.
Would it work to keep this average commit time is less than say 80% of the timeout? Using the 1000HZ example, 1ms wait for the average commit time of 1.2 or 1.3 ms?
Quoted 9 lines
Quoted 2 lines
What seems to be needed here is either a static per file system/storage device tunable to allow us to change this timeout (maybe with "0" defaulting back to the old reiserfs trick of simply doing a
yield()?)Tunables are to be avoided if possible, since they will usually not be set except by the .00001% of people who actually understand them. Using
yield()seems like the right thing, but Andrew Morton added this code and my guess would be thatyield()doesn't block the first thread long enough for the second one to get into the transaction (e.g. on an 2-CPU system with 2 threads,yield()will likely do nothing).
I agree that tunables are a bad thing. It might be nice to dream about having mkfs do some test timings (issues and time the average synchronous IOs/sec) and setting this in the superblock.
Andy tried playing with yield() and it did not do well. Note this this server is a dual CPU box, so your intuition is most likely correct.
The balance is that the batching does work well for "normal" slow disks, especially when using the write barriers (giving us an average commit time closer to 20ms).
Quoted 5 lines
Quoted 1 line
or a more dynamic, per device way to keep track of the average time it takes to commit a transaction to disk. Based on that rate, we could dynamically adjust our logic to account for lower latency devices.
It makes sense to track not only the time to commit a single synchronous transaction, but also the time between sync transactions to decide if the initial transaction should be held to allow later ones.
Yes, that is what I was trying to suggest with the rate. Even if we are relatively slow, if the IO's are being synched at a low rate, we are effectively adding a potentially nasty latency for each IO.
That would give us two measurements to track per IO device - average commit time and this average IO's/sec rate. That seems very doable.
Quoted 3 lines
Alternately, it might be possible to check if a new thread is trying to start a sync handle when the previous one was also synchronous and had only a single handle in it, then automatically enable the delay in that case.
I am not sure that this avoids the problem with the current defaults at 250HZ where each wait is sufficient to do 3 fully independent transactions ;-)
ric
Quoted 8 lines · Ric Wheeler
Quoted 4 lines
Quoted 2 lines
With 2 threads writing to the same directory, we instantly drop down to 234 files/sec.
Is this with HZ=250?
Yes - I assume that with HZ=1000 the batching would start to work again since the penalty for batching would only be 1ms which would add a 0.3ms overhead while waiting for some other thread to join.
This is probably the easiest solution, but at the same time using HZ=1000 adds overhead to the server because of extra interrupts, etc.
Quoted 6 lines
Quoted 3 lines
It would seem one of the problems is that we shouldn't really be scheduling for a fixed 1 jiffie timeout, but rather only until the other threads have a chance to run and join the existing transaction.
This is really very similar to the domain of the IO schedulers - when do you hold off an IO and/or try to combine it.
I was thinking the same.
Quoted 6 lines
Quoted 3 lines
my guess would be that
yield()doesn't block the first thread long enough for the second one to get into the transaction (e.g. on an 2-CPU system with 2 threads,yield()will likely do nothing).Andy tried playing with
yield()and it did not do well. Note this this server is a dual CPU box, so your intuition is most likely correct.
How many threads did you try?
Quoted 10 lines
Quoted 3 lines
It makes sense to track not only the time to commit a single synchronous transaction, but also the time between sync transactions to decide if the initial transaction should be held to allow later ones.
Yes, that is what I was trying to suggest with the rate. Even if we are relatively slow, if the IO's are being synched at a low rate, we are effectively adding a potentially nasty latency for each IO.
That would give us two measurements to track per IO device - average commit time and this average IO's/sec rate. That seems very doable.
Agreed.
Quoted 8 lines
Quoted 4 lines
Alternately, it might be possible to check if a new thread is trying to start a sync handle when the previous one was also synchronous and had only a single handle in it, then automatically enable the delay in that case.
I am not sure that this avoids the problem with the current defaults at 250HZ where each wait is sufficient to do 3 fully independent transactions ;-)
I was trying to think if there was some way to non-busy-wait that is less than 1 jiffie.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
Quoted 7 lines · Andreas Dilger
Quoted 3 lines · Ric Wheeler
Quoted 2 lines
Quoted 1 line
With 2 threads writing to the same directory, we instantly drop down to 234 files/sec.
Is this with HZ=250?
Yes - I assume that with HZ=1000 the batching would start to work again since the penalty for batching would only be 1ms which would add a 0.3ms overhead while waiting for some other thread to join.
This is probably the easiest solution, but at the same time using HZ=1000 adds overhead to the server because of extra interrupts, etc.
We will do some testing with this in the next day or so.
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
It would seem one of the problems is that we shouldn't really be scheduling for a fixed 1 jiffie timeout, but rather only until the other threads have a chance to run and join the existing transaction.
This is really very similar to the domain of the IO schedulers - when do you hold off an IO and/or try to combine it.
Quoted 7 lines
I was thinking the same.
Quoted 4 lines
Quoted 3 lines
my guess would be that
yield()doesn't block the first thread long enough for the second one to get into the transaction (e.g. on an 2-CPU system with 2 threads,yield()will likely do nothing).Andy tried playing with
yield()and it did not do well. Note this this server is a dual CPU box, so your intuition is most likely correct.How many threads did you try?
Andy's tested 1, 2, 4, 8, 20 and 40 threads. Once we review the test and his patch, we can post the summary data.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
It makes sense to track not only the time to commit a single synchronous transaction, but also the time between sync transactions to decide if the initial transaction should be held to allow later ones.
Yes, that is what I was trying to suggest with the rate. Even if we are relatively slow, if the IO's are being synched at a low rate, we are effectively adding a potentially nasty latency for each IO.
That would give us two measurements to track per IO device - average commit time and this average IO's/sec rate. That seems very doable.
Agreed.
This would also seem to be code that would be good to share between all of the file systems for their transaction bundling.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Alternately, it might be possible to check if a new thread is trying to start a sync handle when the previous one was also synchronous and had only a single handle in it, then automatically enable the delay in that case.
I am not sure that this avoids the problem with the current defaults at 250HZ where each wait is sufficient to do 3 fully independent transactions ;-)
I was trying to think if there was some way to non-busy-wait that is less than 1 jiffie.
One other technique would be to use async IO, which could push the batching of the fsync's up to application space. For example, send down a sequence of "async fsync" requests for a series of files and then poll for completion once you have launched them.
ric