Random file I/O regressions in 2.6
Subsystems (3)
Hello!
I tried to compare random file I/O performance in 2.4 and 2.6 kernels and found some regressions that I failed to explain. I tested 2.4.25, 2.6.5-bk2 and 2.6.6-rc3 with my own utility SysBench which was written to generate workloads similar to a database under intensive load.
For 2.6.x kernels anticipatory, deadline, CFQ and noop I/O schedulers were tested with AS giving the best results for this workload, but it's still about 1.5 times worse than the results for 2.4.25 kernel.
The SysBench 'fileio' test was configured to generate the following workload:
16 worker threads are created, each running random read/write file requests in blocks of 16 KB with a read/write ratio of 1.5. All I/O operations are evenly distributed over 128 files with a total size of 3 GB. Each 100 requests, an fsync() operations is performed sequentially on each file. The total number of requests is limited by 10000.
The FS used for the test was ext3 with data=ordered.
Here are the results (values are number of seconds to complete the test):
2.4.25: 77.5377
2.6.5-bk2(noop): 165.3393
2.6.5-bk2(anticipatory): 118.7450
2.6.5-bk2(deadline): 130.3254
2.6.5-bk2(CFQ): 146.4286
2.6.6-rc3(noop): 164.9486
2.6.6-rc3(anticipatory): 125.1776
2.6.6-rc3(deadline): 131.8903
2.6.6-rc3(CFQ): 152.9280
I have published the results as well as the hardware and kernel setups at the SysBench home page: http://sysbench.sourceforge.net/results/fileio/
Any comments or suggestions would be highly appreciated.
Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Quoted 7 lines · Alexey Kopytov
Hello!
I tried to compare random file I/O performance in 2.4 and 2.6 kernels and found some regressions that I failed to explain. I tested 2.4.25, 2.6.5-bk2 and 2.6.6-rc3 with my own utility SysBench which was written to generate workloads similar to a database under intensive load.
For 2.6.x kernels anticipatory, deadline, CFQ and noop I/O schedulers were
tested with AS giving the best results for this workload, but it's still about 1.5 times worse than the results for 2.4.25 kernel.
Quoted 9 lines
The SysBench 'fileio' test was configured to generate the following workload:
16 worker threads are created, each running random read/write file requests in blocks of 16 KB with a read/write ratio of 1.5. All I/O operations are evenly distributed over 128 files with a total size of 3 GB. Each 100 requests, anfsync()operations is performed sequentially on each file. The total number of requests is limited by 10000.The FS used for the test was ext3 with data=ordered.
I am able to reproduce this here. 2.6 isn't improved by increasing nr_requests, relaxing IO scheduler deadlines, or turning off readahead. It looks like 2.6 is submitting a lot of the IO in 4KB sized requests...
Hmm, oh dear. It looks like the readahead logic shat itself and/or do_generic_mapping_read doesn't know how to handle multipage reads properly.
What ends up happening is that readahead gets turned off, then the 16K read ends up being done in 4 synchronous 4K chunks. Because they are synchronous, they have no chance of being merged with one another either.
I have attached a proof of concept hack... I think what should really happen is that page_cache_readahead should be taught about the size of the requested read, and ensures that a decent amount of reading is done while within the read request window, even if beyond-request-window-readahead has been previously unsuccessful.
Numbers with an IDE disk, 256MB ram
2.4.24: 81s
2.6.6-rc3-mm1: 126s
rc3-mm1+patch: 87s
The small remaining regression might be explained by 2.6's smaller nr_requests, IDE driver, io scheduler tuning, etc.
Quoted 14 lines
Here are the results (values are number of seconds to complete the test):
2.4.25: 77.5377
2.6.5-bk2(noop): 165.3393
2.6.5-bk2(anticipatory): 118.7450
2.6.5-bk2(deadline): 130.3254
2.6.5-bk2(CFQ): 146.42862.6.6-rc3(noop): 164.9486
2.6.6-rc3(anticipatory): 125.1776
2.6.6-rc3(deadline): 131.8903
2.6.6-rc3(CFQ): 152.9280
Quoted 5 lines
I have published the results as well as the hardware and kernel setups at the SysBench home page: http://sysbench.sourceforge.net/results/fileio/
Any comments or suggestions would be highly appreciated.
From your website:
"Another interesting fact is that AS gives the best results for this workload, though it's believed to give worse results for this kind of workloads as compared to other I/O schedulers available in 2.6.x kernels."
The anticipatory scheduler is actually in a fairly good state of tune, and can often beat deadline even for random read/write/fsync tests. The infamous database regression problem is when this sort of workload is combined with TCQ disk drives.
Nick
include/linux/mm.h | 0 linux-2.6-npiggin/mm/filemap.c | 5 ++++- 2 files changed, 4 insertions(+), 1 deletion(-)
diff -puN mm/readahead.c~read-populate mm/readahead.c
mm/filemap.c renamed from mm/filemap.c~read-populate +4 -1
@@ -627,6 +627,9 @@ void do_generic_mapping_read(struct addr index = *ppos >>PAGE_CACHE_SHIFT ; offset = *ppos & ~PAGE_CACHE_MASK ; +force_page_cache_readahead (mapping, filp, index, +max_sane_readahead (desc->count >>PAGE_CACHE_SHIFT )); +for (;;) {struct page *page;unsigned long end_index, nr, ret;
@@ -644,7 +647,7 @@ void do_generic_mapping_read(struct addr }cond_resched (); -page_cache_readahead (mapping, ra, filp, index); +page_cache_readahead (mapping, ra, filp, index + desc->count); nr = nr - offset; find_page:
diff -puN include/linux/mm.h~read-populate include/linux/mm.h
_
Quoted 5 lines · Nick Piggin
What ends up happening is that readahead gets turned off, then the 16K read ends up being done in 4 synchronous 4K chunks. Because they are synchronous, they have no chance of being merged with one another either.
yup.
Quoted 5 lines
I have attached a proof of concept hack... I think what should really happen is that
page_cache_readaheadshould be taught about the size of the requested read, and ensures that a decent amount of reading is done while within the read request window, even if beyond-request-window-readahead has been previously unsuccessful.
The "readahead turned itself off" thing is there to avoid doing lots of pagecache lookups in the very common case where the file is fully cached.
The place which needs attention is handle_ra_miss(). But first I'd like to reacquaint myself with the intent behind the lazy-readahead patch. Was never happy with the complexity and special-cases which that introduced.
mm/filemap.c void do_generic_mapping_read(struct addr · line 650
Quoted 4 lines
mm/filemap.c renamed from mm/filemap.c~read-populate +1
-1cond_resched (); -page_cache_readahead (mapping, ra, filp, index); +page_cache_readahead (mapping, ra, filp, index + desc->count);
`index' is a pagecache index and desc->count is a byte counter.
Quoted 15 lines · Andrew Morton
Quoted 5 lines · Nick Piggin
What ends up happening is that readahead gets turned off, then the 16K read ends up being done in 4 synchronous 4K chunks. Because they are synchronous, they have no chance of being merged with one another either.
yup.
Quoted 5 lines
I have attached a proof of concept hack... I think what should really happen is that
page_cache_readaheadshould be taught about the size of the requested read, and ensures that a decent amount of reading is done while within the read request window, even if beyond-request-window-readahead has been previously unsuccessful.
Quoted 6 lines
The "readahead turned itself off" thing is there to avoid doing lots of pagecache lookups in the very common case where the file is fully cached.
The place which needs attention is
handle_ra_miss(). But first I'd like to reacquaint myself with the intent behind the lazy-readahead patch. Was never happy with the complexity and special-cases which that introduced.
lazy-readahead has no role to play here. The readahead window got closed because the i/o pattern was totally random. My guess is multiple threads are generating 16k i/o on the same fd. In such a case the i/os can get interleaved and the readahead window size goes for a toss(which is expected behavior)
Well if this is infact the case: the question is
1. does the i/o pattern really has some sequentiality to deserve a readahead?
2. or should we ensure that the interleaved case be somehow handled, by including the size parameter?
I know Nick has implied option (2) but I think from the readahead's point of view it is (1),
RP
Quoted 6 lines · Ram Pai
Quoted 4 lines
The place which needs attention is
handle_ra_miss(). But first I'd like to reacquaint myself with the intent behind the lazy-readahead patch. Was never happy with the complexity and special-cases which that introduced.lazy-readahead has no role to play here.
Sure. But lazy-readahead is bolted on the side and is generally not to my liking. I'd like to find a solution to the sysbench problem which also solves the thing which lazy-readahead addressed.
Quoted 5 lines
The readahead window got closed
because the i/o pattern was totally random. My guess is multiple threads are generating 16k i/o on the same fd. In such a case the i/os can get interleaved and the readahead window size goes for a toss(which is expected behavior)
I don't think it's that. The app is doing well-aligned 16k reads and writes. If we get enough pagecache hits on the reads, readahead turns itself off (fair enough) but fails to turn itself on again.
The readahead logic _should_ be able to adapt to the fixed-sized I/Os and issue correct-sized reads immediately after each seek. I _think_ this will fix the problem which lazy-readahead addressed, but as usual we don't have a rigorous description of that problem :(
Quoted 8 lines
Well if this is infact the case: the question is
1. does the i/o pattern really has some sequentiality to deserve a readahead?
2. or should we ensure that the interleaved case be somehow handled, by including the size parameter?I know Nick has implied option (2) but I think from the readahead's point of view it is (1),
Readahead has got too complex and is getting band-aidy. I'd prefer to tear it down and rethink things.
Quoted 8 lines · Andrew Morton
Quoted 6 lines · Ram Pai
Quoted 4 lines
The place which needs attention is
handle_ra_miss(). But first I'd like to reacquaint myself with the intent behind the lazy-readahead patch. Was never happy with the complexity and special-cases which that introduced.lazy-readahead has no role to play here.
Andrew,
Could you please clarify how this things become to be dependent on read-ahead at all.
At my understanding read-ahead it to catch sequential (or other) access pattern and do some advance reading, so instead of 16K request we do 128K request, or something similar.
But how could read-ahead disabled end up in 16K request converted to several sequential synchronous 4K requests ?
It all looks pretty strange.
Signature · 2 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com
Quoted 15 lines · Peter Zaitsev
Quoted 8 lines · Andrew Morton
Quoted 6 lines · Ram Pai
Quoted 4 lines
The place which needs attention is
handle_ra_miss(). But first I'd like to reacquaint myself with the intent behind the lazy-readahead patch. Was never happy with the complexity and special-cases which that introduced.lazy-readahead has no role to play here.
Andrew,
Could you please clarify how this things become to be dependent on read-ahead at all.
Quoted 6 lines
At my understanding read-ahead it to catch sequential (or other) access pattern and do some advance reading, so instead of 16K request we do 128K request, or something similar.
But how could read-ahead disabled end up in 16K request converted to several sequential synchronous 4K requests ?
When the readahead window gets closed,the code goes into slow-read mode. In this mode, all requests are broken to page-size. Hence a 16k request gets broken into 4 4K-requests. This continues to the point where enough number of sequential i/os are requested(i.e around ra->ra_pages number of pages), at which point the readahead window gets re-activated.
Looking at it the other way, without readahead code, all requests satisfied through 4k i/os. Readahead helps in generating larger size i/os.
RP
Quoted 4 lines · Ram Pai
Looking at it the other way, without readahead code, all requests satisfied through 4k i/os. Readahead helps in generating larger size i/os.
Huh, This is kind of really strange.
If you speak about database world, Random IO is quite frequent and database page sizes are normally larger than OS page size.
Furthermore even if it is split to 4K block sizes, why are they not submitted in parallel, being merged on lower level.
Anyway we seems to all agree this is not very good behavior and it should be fixed :)
Signature · 2 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com
Quoted 8 lines · Peter Zaitsev
Quoted 6 lines · Andrew Morton
Quoted 5 lines · Ram Pai
Quoted 4 lines
The place which needs attention is
handle_ra_miss(). But first I'd like to reacquaint myself with the intent behind the lazy-readahead patch. Was never happy with the complexity and special-cases which that introduced.
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
lazy-readahead has no role to play here.
Andrew,
Could you please clarify how this things become to be dependent on read-ahead at all.
readahead is currently the only means by which we build up nice large multi-page BIOs.
Quoted 3 lines
At my understanding read-ahead it to catch sequential (or other) access pattern and do some advance reading, so instead of 16K request we do 128K request, or something similar.
That's one of its usage patterns. It's also supposed to detect the fixed-sized-reads-seeking-all-over-the-place situation. In which case it's supposed to submit correctly-sized multi-page BIOs. But it's not working right for this workload.
A naive solution would be to add special-case code which always does the fixed-size readahead after a seek. Basically that's
if (ra->next_size == -1UL) force_page_cache_readahead(...)
in filemap.c. But this means that the kernel does lots of pointless pagecache lookups when everything is in pagecache. We should detect this situation and stop doing readahead completely, until we start getting pagecache lookup misses again.
Quoted 2 lines
But how could read-ahead disabled end up in 16K request converted to several sequential synchronous 4K requests ?
Readahead got itself turned off because of pagecache hits and didn't turn itself on again.
Quoted 6 lines · Andrew Morton
Quoted 3 lines
But how could read-ahead disabled end up in 16K request converted to several sequential synchronous 4K requests ?
Readahead got itself turned off because of pagecache hits and didn't turn itself on again.
Andrew,
In the slow read path, every contiguous access increases ra->size by 1 and non-contiguous access decreases the ra->size by 1. Now in the case of 16k random request, we have 1 non-contiguous request and 3 contiguous request. As a result the ra->size should have been incremented by -1+1+1+1=2 . So at the end of 16 4K request we should have had ra->size at 32. At this point onwards the readahead should get turned on. Right?
I strongly feel the readahead got closed because of misses and not because of hits. Moreover if we are closing readahead window
because of hits, then that implies we have pretty good caching going on.Which implies i/o should rarely hit the disk and hence performance should not degrade.
Agree?
RP
Quoted 2 lines
Quoted 32 lines · Andrew Morton
Quoted 8 lines · Peter Zaitsev
Quoted 6 lines · Andrew Morton
Quoted 4 lines · Ram Pai
Quoted 3 lines
The place which needs attention is
handle_ra_miss(). But first I'd like to reacquaint myself with the intent behind the lazy-readahead patch. Was never happy with the complexity and special-cases which that introduced.Quoted 6 lines
Quoted 2 lines
Quoted 1 line
lazy-readahead has no role to play here.
Andrew,
Could you please clarify how this things become to be dependent on read-ahead at all.
readahead is currently the only means by which we build up nice large multi-page BIOs.
Quoted 3 lines
At my understanding read-ahead it to catch sequential (or other) access pattern and do some advance reading, so instead of 16K request we do 128K request, or something similar.
That's one of its usage patterns. It's also supposed to detect the fixed-sized-reads-seeking-all-over-the-place situation. In which case it's supposed to submit correctly-sized multi-page BIOs. But it's not working right for this workload.
Quoted 6 lines
A naive solution would be to add special-case code which always does the fixed-size readahead after a seek. Basically that's
if (ra->next_size == -1UL) force_page_cache_readahead(...)
I think a better solution to this case would be to ensure the readahead window is always min(size of read, some large number);
The size of the read is basically a free and accurate "hint" to the minimum size of the required readahead.
Either that or do a simple "preread" while you're still in the read request window, and run readahead when that completes.
Quoted 9 lines · Nick Piggin
Quoted 9 lines
That's one of its usage patterns. It's also supposed to detect the fixed-sized-reads-seeking-all-over-the-place situation. In which case it's supposed to submit correctly-sized multi-page BIOs. But it's not working right for this workload.
A naive solution would be to add special-case code which always does the fixed-size readahead after a seek. Basically that's
Quoted 7 lines
Quoted 3 lines
if (ra->next_size == -1UL) force_page_cache_readahead(...)I think a better solution to this case would be to ensure the readahead window is always
min(size of read, some large number);
That would cause the kernel to perform lots of pointless pagecache lookups when the file is already 100% cached.
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Nick Piggin
Quoted 11 lines
That's one of its usage patterns. It's also supposed to detect the fixed-sized-reads-seeking-all-over-the-place situation. In which case it's supposed to submit correctly-sized multi-page BIOs. But it's not working right for this workload.
A naive solution would be to add special-case code which always does the fixed-size readahead after a seek. Basically that's
if (ra->next_size == -1UL) force_page_cache_readahead(...)
Quoted 8 lines
Quoted 3 lines
I think a better solution to this case would be to ensure the readahead window is always
min(size of read, some large number);That would cause the kernel to perform lots of pointless pagecache lookups when the file is already 100% cached.
That's pretty sad. You need a "preread" or something which sends the pages back... or uses the actor itself. readahead would then have to be reworked to only run off the end of the read window, but that is what it should be doing anyway.
Quoted 25 lines · Nick Piggin
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Nick Piggin
Quoted 11 lines
That's one of its usage patterns. It's also supposed to detect the fixed-sized-reads-seeking-all-over-the-place situation. In which case it's supposed to submit correctly-sized multi-page BIOs. But it's not working right for this workload.
A naive solution would be to add special-case code which always does the fixed-size readahead after a seek. Basically that's
if (ra->next_size == -1UL) force_page_cache_readahead(...)Quoted 8 lines
Quoted 3 lines
I think a better solution to this case would be to ensure the readahead window is always
min(size of read, some large number);That would cause the kernel to perform lots of pointless pagecache lookups when the file is already 100% cached.
Quoted 4 lines
That's pretty sad. You need a "preread" or something which sends the pages back... or uses the actor itself. readahead would then have to be reworked to only run off the end of the read window, but that is what it should be doing anyway.
Sorry, If I am saying this again. I have checked the behaviour of the readahead code using my user level simulator as well as running some DSS benchmark and iozone benchmark. It generates a steady stream of large i/o for large-random-reads and should not exhibit the bad behavior that we are seeing. I feel this bad behavior is because of interleaved access by multiple thread.
To illustrate with an example:
t1 request reads from page 100 to 104
simultaneously t2 requests reads on the same fd from 200 to 204
So do_page_cache_readahead() can be called in the following pattern. 100,200,101,201,102,202,103,203,104,204.
Because of this pattern the readahaed code assumes that the read pattern is absolutely random and hence closes the readahead window.
I think I should generate a patch to validate this behavior, I will. How about having some /proc counters that keep track of number of window-closes because of cache-hits and because of cache-misses?
RP
Quoted 7 lines · Ram Pai
Sorry, If I am saying this again. I have checked the behaviour of the
readahead code using my user level simulator as well as running some DSS benchmark and iozone benchmark. It generates a steady stream of large i/o for large-random-reads and should not exhibit the bad behavior that we are seeing. I feel this bad behavior is because of interleaved access by multiple thread.
you're right - the benchmark has multiple threads issuing concurrent pread()s against the same fd. For some reason this mucks up the 2.6 readahead state more than 2.4's.
Putting a semaphore around do_generic_file_read() or maintaining the state as below fixes it up.
I wonder if we should bother fixing this? I guess as long as the app is using pread() it is a legitimate thing to be doing, so I guess we should...
mm/filemap.c renamed from mm/filemap.c~readahead-seralisation +6 -3
@@ -612,7 +612,7 @@ EXPORT_SYMBOL(grab_cache_page_nowait);* - note the struct file * is only passed for the use of readpage */ void do_generic_mapping_read(struct address_space *mapping, - struct file_ra_state *ra, + struct file_ra_state *_ra, struct file * filp, loff_t *ppos, read_descriptor_t * desc,
@@ -622,6 +622,7 @@ void do_generic_mapping_read(struct addrunsigned long index, offset;struct page *cached_page;int error; +struct file_ra_state ra = *_ra; cached_page =NULL ; index = *ppos >>PAGE_CACHE_SHIFT ;
@@ -644,13 +645,13 @@ void do_generic_mapping_read(struct addr }cond_resched (); -page_cache_readahead (mapping, ra, filp, index); +page_cache_readahead (mapping, &ra, filp, index); nr = nr - offset; find_page: page =find_get_page (mapping, index);if (unlikely (page ==NULL )) { -handle_ra_miss (mapping, ra, index); +handle_ra_miss (mapping, &ra, index); goto no_cached_page; }if (!PageUptodate (page))
@@ -752,6 +753,8 @@ no_cached_page: goto readpage; } + *_ra = ra; + *ppos = ((loff_t ) index <<PAGE_CACHE_SHIFT ) + offset;if (cached_page)page_cache_release (cached_page);
_
Quoted 9 lines · Andrew Morton
Putting a semaphore around
do_generic_file_read()or maintaining the state as below fixes it up.I wonder if we should bother fixing this? I guess as long as the app is using
pread()it is a legitimate thing to be doing, so I guess we should...
Yes this patch makes sense. I have setup sysbench on my lab machine. Let me see how much improvement the patch provides.
RP
Quoted 6 lines · Ram Pai
Quoted 4 lines · Andrew Morton
Putting a semaphore around
do_generic_file_read()or maintaining the state as below fixes it up.
Quoted 7 lines
Quoted 5 lines
I wonder if we should bother fixing this? I guess as long as the app is using
pread()it is a legitimate thing to be doing, so I guess we should...Yes this patch makes sense. I have setup sysbench on my lab machine. Let me see how much improvement the patch provides.
I ran the following command:
/root/sysbench-0.2.5/sysbench/sysbench --num-threads=256 --test=fileio --file-total-size=2800M --file-test-mode=rndrw run
Without the patch:
------------------
Operations performed: 5959 Read, 4041 Write, 10752 Other = 20752 Total Read 93Mb Written 63Mb Total Transferred 156Mb
7.549Mb/sec Transferred
483.89 Requests/sec executed
Test execution Statistics summary:
Time spent for test: 20.6661s
no of times window reset because of hits: 0 no of times window reset because of misses: 7 no of times window was shrunk because of hits: 6716 no of times the page request was non-contiguous: 5880 no of times the page request was contiguous : 19639
With the patch:
--------------
Operations performed: 5960 Read, 4040 Write, 10880 Other = 20880 Total Read 93Mb Written 63Mb Total Transferred 156Mb
7.985Mb/sec Transferred
511.85 Requests/sec executed
Test execution Statistics summary:
Time spent for test: 19.5370s
no of times window got reset because of hits: 0 no of times window got reset because of misses: 0 no of times window was shrunk because of hits: 5844 no of times the page request was non-contiguous: 5830 no of times the page request was contiguous : 20232
I have enclosed the patch that collects the hit/miss related counts.
In general I am not seeing any major difference with or without andrew's ra-copy patch; except for readahead window getting closed because of misses when run without the patch.
Would be nice if Alexey tries the patch on his machine and sees any major difference.
RP
include/linux/sysctl.h +5 -0
@@ -643,6 +643,11 @@ enumFS_XFS =17 ,/* struct: control xfs parameters */ FS_AIO_NR =18 ,/* current system-wide number of aio requests */ FS_AIO_MAX_NR =19 ,/* system-wide maximum number of aio requests */ +FS_READ_MISS_RESET =20 , +FS_READ_HIT_RESET =21 , +FS_CONTIGUOUS_CNT =22 , +FS_NON_CONTIGUOUS_CNT =22 , + FS_HIT_COUNT=23, };/* /proc/sys/fs/quota/ */
kernel/sysctl.c +47 -0
59 lines not on this page · raw message
mm/readahead.c +11 -0
@@ -316,6 +316,12 @@ int do_page_cache_readahead(struct addrereturn 0 ; } +atomic_t hit_reset=ATOMIC_INIT (0 ); +atomic_t miss_reset=ATOMIC_INIT (0 ); +atomic_t hit_count=ATOMIC_INIT (0 ); +atomic_t contiguous_cnt =ATOMIC_INIT (0 ); +atomic_t non_contiguous_cnt=ATOMIC_INIT (0 ); +/* * Check how effective readahead is being. If the amount of started IO is * less than expected then the file is partly or fully in pagecache and
@@ -331,11 +337,13 @@ check_ra_success(struct file_ra_state *rif (actual ==0 ) {if (orig_next_size >1 ) { ra->next_size = orig_next_size -1 ; +atomic_inc (&hit_count);if (ra->ahead_size) ra->ahead_size = ra->next_size; }else { ra->next_size =-1UL ; ra->size =0 ; +atomic_inc (&hit_reset); } } }
@@ -406,17 +414,20 @@ page_cache_readahead(struct address_spac* page beyond the end. Expand the next readahead size. */ ra->next_size +=2 ; +atomic_inc (&contiguous_cnt); }else {/* * A miss - lseek, pagefault, pread, etc. Shrink the readahead * window. */ ra->next_size -=2 ; +atomic_inc (&non_contiguous_cnt); }if ((long )ra->next_size > (long )max) ra->next_size = max;if ((long )ra->next_size <=0L ) { +atomic_inc (&miss_reset); ra->next_size =-1UL ; ra->size =0 ; goto out;/* Readahead is off */
Quoted 6 lines · Ram Pai
I ran the following command:
/root/sysbench-0.2.5/sysbench/sysbench--num-threads=256 --test=fileio --file-total-size=2800M --file-test-mode=rndrw run
Alexey and I have been using 16 threads.
You don't tell us how much memory your lab machine has. The above command only makes sense if it is less than 400 megabytes. Otherwise many or all of the reads are satisfied from pagecache.
I've been testing with mem=256M, --file-total-size=2G.
Quoted 11 lines · Andrew Morton
Quoted 6 lines · Ram Pai
I ran the following command:
/root/sysbench-0.2.5/sysbench/sysbench--num-threads=256 --test=fileio --file-total-size=2800M --file-test-mode=rndrw runAlexey and I have been using 16 threads.
You don't tell us how much memory your lab machine has.
It has 8GB but only 4Gb is being used. I will try with 256MB and 16 threads.
RP
Quoted 11 lines · Ram Pai
Quoted 10 lines · Andrew Morton
Quoted 6 lines · Ram Pai
I ran the following command:
/root/sysbench-0.2.5/sysbench/sysbench--num-threads=256 --test=fileio --file-total-size=2800M --file-test-mode=rndrw runAlexey and I have been using 16 threads.
/root/sysbench-0.2.5/sysbench/sysbench --num-threads=16 --test=fileio
--file-total-size=2800M --file-test-mode=rndrw run
Without the patch:
------------------
Operations performed: 6002 Read, 3998 Write, 12800 Other = 22800 Total Read 93Mb Written 62Mb Total Transferred 156Mb
1.967Mb/sec Transferred
126.11 Requests/sec executed
Test execution Statistics summary:
Time spent for test: 79.2986s
no of times window reset because of hits: 0 no of times window reset because of misses: 119 no of times window was shrunk because of hits: 417 no of times the page request was non-contiguous: 3809 no of times the page request was contiguous : 12745
With the patch:
--------------
Operations performed: 6002 Read, 3999 Write, 12672 Other = 22673 Total Read 93Mb Written 62Mb Total Transferred 156Mb
2.927Mb/sec Transferred
187.65 Requests/sec executed
Test execution Statistics summary:
Time spent for test: 53.2949s
no of times window reset because of hits: 0 no of times window reset because of misses: 0 no of times window was shrunk because of hits: 360 no of times the page request was non-contiguous: 5860 no of times the page request was contiguous : 20378
Impressive results. WOuld be nice to get a confirmation from Alexey. RP
On Tue, 2004-05-04 at 14:51, Ram Pai wrote:
memory used is ***256MB***.
Quoted 3 lines
/root/sysbench-0.2.5/sysbench/sysbench--num-threads=16 --test=fileio
--file-total-size=2800M --file-test-mode=rndrw run
RP
Quoted 14 lines · Ram Pai
Without the patch:
------------------
Time spent for test: 20.6661sno of times window reset because of hits: 0 no of times window reset because of misses: 7 no of times window was shrunk because of hits: 6716 no of times the page request was non-contiguous: 5880 no of times the page request was contiguous : 19639
With the patch:
--------------
Time spent for test: 19.5370s
Quoted 8 lines
no of times window got reset because of hits: 0 no of times window got reset because of misses: 0 no of times window was shrunk because of hits: 5844 no of times the page request was non-contiguous: 5830 no of times the page request was contiguous : 20232
Would be nice if Alexey tries the patch on his machine and sees any major difference.
Here's what I have (same hardware and test setups):
Without the patch (but with Ram's patch applied):
------------------
Time spent for test: 125.4429s
no of times window reset because of hits: 0 no of times window reset because of misses: 127 no of times window was shrunk because of hits: 1153 no of times the page request was non-contiguous: 3968 no of times the page request was contiguous : 10686
With the patch:
---------------
Time spent for test: 86.5459s
no of times window reset because of hits: 0 no of times window reset because of misses: 0 no of times window was shrunk because of hits: 1066 no of times the page request was non-contiguous: 5860 no of times the page request was contiguous : 18099
I wonder if there are some plans to further improve 2.6 behavior on this workload to match that of 2.4? Is the remaing regression a result of the different readahead handling, or it might be caused by IDE driver or I/O scheduler tuning?
Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Quoted 15 lines · Alexey Kopytov
Without the patch (but with Ram's patch applied):
------------------
Time spent for test: 125.4429sno of times window reset because of hits: 0 no of times window reset because of misses: 127 no of times window was shrunk because of hits: 1153 no of times the page request was non-contiguous: 3968 no of times the page request was contiguous : 10686
With the patch:
---------------
Time spent for test: 86.5459s
Quoted 6 lines
no of times window reset because of hits: 0 no of times window reset because of misses: 0 no of times window was shrunk because of hits: 1066 no of times the page request was non-contiguous: 5860 no of times the page request was contiguous : 18099
The patch brought my test box to the same speed as 2.4. With the deadline scheduler it was a bit faster than 2.4. I didn't do a lot of testing though. I was using ext2. Please try deadline.
Quoted 2 lines
I wonder if there are some plans to further improve 2.6 behavior on this workload to match that of 2.4?
Of course...
Tuning work is being done on the anticipatory scheduler which we hope will bring it up to deadline throughput for this sort of workload.
Quoted 3 lines
Is the remaing regression a result of the
different readahead handling, or it might be caused by IDE driver or I/O scheduler tuning?
Don't know yet. On 2.6 the test actually does about 5% fewer reads than under 2.4, so the VM page replacement is working a bit better in this case.
And 2.6 does about 40% fewer context switches for some reason. So we
should be a little bit faster - it's a matter of finding where the additional seeks or idle time are coming from.
Quoted 11 lines · Andrew Morton
Quoted 9 lines · Alexey Kopytov
With the patch:
---------------
Time spent for test: 86.5459sno of times window reset because of hits: 0 no of times window reset because of misses: 0 no of times window was shrunk because of hits: 1066 no of times the page request was non-contiguous: 5860 no of times the page request was contiguous : 18099
Quoted 4 lines
The patch brought my test box to the same speed as 2.4. With the deadline scheduler it was a bit faster than 2.4. I didn't do a lot of testing though. I was using ext2. Please try deadline.
Results with the deadline scheduler on my hardware:
Time spent for test: 92.8340s
no of times window reset because of hits: 0 no of times window reset because of misses: 0 no of times window was shrunk because of hits: 1108 no of times the page request was non-contiguous: 5860 no of times the page request was contiguous : 18091
I have updated the results on the SysBench home page with 2.6.6-rc3 with the patch applied.
Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Quoted 4 lines · Alexey Kopytov
Results with the deadline scheduler on my hardware:
Time spent for test: 92.8340s
Now we're into unreproducible results, alas. On a 256MB uniprocessor machine:
ext3:
sysbench --num-threads=16 --test=fileio --file-total-size=2G --file-test-mode=rndrw run
2.6.6-rc3-mm2, deadline:
Time spent for test: 66.7536s Time spent for test: 67.9000s 0.04s user 6.41s system 4% cpu 2:14.74 total
2.6.6-rc2-mm2, as:
Time spent for test: 66.7576s 0.07s user 6.68s system 5% cpu 2:14.18 total Time spent for test: 66.3216s 0.06s user 6.28s system 4% cpu 2:12.25 total
2.4.27-pre2:
Time spent for test: 64.9766s 0.09s user 11.57s system 8% cpu 2:17.43 total Time spent for test: 64.2852s 0.11s user 11.18s system 8% cpu 2:14.63 total
so 2.6 is a shade slower. 2.6 has tons less system CPU time, probably due to ext3 improvements.
The reason for the difference appears to be the thing which Ram added to readahead which causes it to usually read one page too many. With this exciting patch:
mm/readahead.c renamed from mm/readahead.c~a +1 -1
@@ -475,7 +475,7 @@ do_io: ra->ahead_start =0 ;/* Invalidate these */ ra->ahead_size =0 ; actual =do_page_cache_readahead (mapping, filp, offset, - ra->size); + ra->size ==5 ?4 : ra->size);if (!first_access) {/* * do not adjust the readahead window size the first
_
I get:
Time spent for test: 63.9435s 0.07s user 6.69s system 5% cpu 2:11.02 total
which is a good result.
Ram, can you take a look at fixing that up please? Something clean, not more hacks ;) I'd also be interested in an explanation of what the extra page is for. The little comment in there doesn't really help.
One thing I note about this test is that it generates a huge number of inode writes. atime updates from the reads and mtime updates from the writes. Suppressing them doesn't actually make a lot of performance difference, but that is with writeback caching enabled. I expect that with a writethrough cache these will really hurt.
The test uses 128 files, which seems excessive. I assume that four or eight files is a more likely real-life setup, and in theis case the atime/mtime update volume will be proportionately less.
Alexey, I do not know why you're seeing such a disparity. I assume that IDE DMA is enabled - the difference seems too small for that to be an explanation, but please check it.
Quoted 6 lines · Andrew Morton
One thing I note about this test is that it generates a huge number of inode writes. atime updates from the reads and mtime updates from the writes. Suppressing them doesn't actually make a lot of performance difference, but that is with writeback caching enabled. I expect that with a writethrough cache these will really hurt.
Perhaps. By the way is there a way to disable update time modification as well ? It would make quite a good sense for partition used for Database needs - you do not need last modification time in most cases.
Quoted 4 lines
The test uses 128 files, which seems excessive. I assume that four or eight files is a more likely real-life setup, and in theis case the atime/mtime update volume will be proportionately less.
Actually both single (or very few) files and large amount of files are practical setup. In MySQL 4.1 we have the option to store each Innodb table in its own file, which will mean scattered random IO to many files for OLTP workloads.
You might think 128 actively used tables are still to much, but practically we see even larger numbers - some customers partition data, creating huge number of tables with same structure, for example table-per customer.
Signature · 2 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com
Quoted 4 lines · Peter Zaitsev
Quoted 1 line · Andrew Morton
Quoted 8 lines
Quoted 5 lines
One thing I note about this test is that it generates a huge number of inode writes. atime updates from the reads and mtime updates from the writes. Suppressing them doesn't actually make a lot of performance difference, but that is with writeback caching enabled. I expect that with a writethrough cache these will really hurt.
Perhaps. By the way is there a way to disable update time modification as well ?
No, there is not.
Quoted 2 lines
It would make quite a good sense for partition used for Database needs - you do not need last modification time in most cases.
First up, one needs to remove the inode_update_time() call from generic_file_aio_write_nolock() and run the tests. If this (and noatime) indeed makes a significant difference (probably on writethrough-caching disks) then yup, we should do something.
`nomtime' would be simple enough. But another option would be to arrange for a/m/ctime dirtiness to not cause an inode writeout in fsync(). Instead, only sync the a/m/ctime-dirty inodes via sync, umount and pdflush.
That way, the inodes get written every thirty seconds rather than once per second.
It's probably not standards-compliant, but shoot me. Who cares if the mtimes come up 30 seconds out of date after a system crash?
`nomtime' would be simpler and safer to implement, but not as nice.
But we need those numbers first. I'll take a look.
Quoted 33 lines · Andrew Morton
Quoted 11 lines · Peter Zaitsev
Quoted 5 lines · Andrew Morton
One thing I note about this test is that it generates a huge number of inode writes. atime updates from the reads and mtime updates from the writes. Suppressing them doesn't actually make a lot of performance difference, but that is with writeback caching enabled. I expect that with a writethrough cache these will really hurt.
Perhaps. By the way is there a way to disable update time modification as well ?
No, there is not.
Quoted 2 lines
It would make quite a good sense for partition used for Database needs - you do not need last modification time in most cases.
First up, one needs to remove the
inode_update_time()call fromgeneric_file_aio_write_nolock()and run the tests. If this (and noatime) indeed makes a significant difference (probably on writethrough-caching disks) then yup, we should do something.`nomtime' would be simple enough. But another option would be to arrange
for a/m/ctime dirtiness to not cause an inode writeout infsync(). Instead, only sync the a/m/ctime-dirty inodes via sync, umount and pdflush.That way, the inodes get written every thirty seconds rather than once per second.
Quoted 7 lines
It's probably not standards-compliant, but shoot me. Who cares if the mtimes come up 30 seconds out of date after a system crash?
`nomtime' would be simpler and safer to implement, but not as nice.
But we need those numbers first. I'll take a look.
Can they use fdatasync? Does it do the right thing on Linux?
Quoted 7 lines · Nick Piggin
Quoted 5 lines
`nomtime' would be simpler and safer to implement, but not as nice.
But we need those numbers first. I'll take a look.
Can they use fdatasync? Does it do the right thing on Linux?
Nick,
You're right. fdatasync suppose to be solution in this case and actually test supports this mode as well as MySQL does :)
On other hand if you rather use O_DSYNC it does not seems to work being mapped to O_SYNC.
But the thing I'm mostly interested in is O_DIRECT. It seems to be the best solution for many database needs, especially used together with asynchronous IO. There is however no matching option which should not flush MetaData.
Signature · 2 lines
Peter Zaitsev, Senior Support Engineer MySQL AB, www.mysql.com
On Thu, 2004-05-06 at 01:43, Andrew Morton wrote:
sorry, I am out for 10days and hence replies are late.
Quoted 21 lines
The reason for the difference appears to be the thing which Ram added to readahead which causes it to usually read one page too many. With this exciting patch:
mm/readahead.c renamed from mm/readahead.c~a +1
-1@@ -475,7 +475,7 @@ do_io: ra->ahead_start =0 ;/* Invalidate these */ ra->ahead_size =0 ; actual =do_page_cache_readahead (mapping, filp, offset, - ra->size); + ra->size ==5 ?4 : ra->size);if (!first_access) {/* * do not adjust the readahead window size the first _
I get:
Quoted 8 lines
Time spent for test: 63.9435s 0.07s user 6.69s system 5% cpu 2:11.02 totalwhich is a good result.
Ram, can you take a look at fixing that up please? Something clean, not more hacks ;) I'd also be interested in an explanation of what the extra page is for. The little comment in there doesn't really help.
The reason for the extra page read is as follows:
Consider 16k random reads i/os. Reads are generated 4pages at a time.
the readahead is triggered when the 4th page in the 'current-window' is touched. However the data which is read-in through the 'readahead window' gets thrown away because the next 16k read-io will not access anything read in the readahead window. As a result I put in that optimization which handles this wasted readahead-pages.
The idea is, when we miss the current-window, read one more page than the number of pages accessed in the current-window.
Here is a example scenario of random 16k i/os and with Andrew's code
actual = do_page_cache_readahead(mapping, filp, offset,
- ra->size);
+ ra->size == 5 ? 4 : ra->size);
Consider that the application access page {1,2,3,4} {100,101,102,103} {200,201,202,203}
Consider that the current-window holds 4 pages. i.e page 1,2,3,4
when the application asks for {1,2,3,4} we happily satisfy them through the current-window. However when the application touches page 4, the lazy-readahead triggers in and brings in pages {5,6,7,8,9,10,11,12} but now the application wants to access {100,101,102,103}. This waste of effort is probably bearable as long as we dont commit the same mistake in the future. When the application tries to access {100,101,102,103} the code then scraps both the current-window and the readahead-window and reads in a new current-window of size 4 i.e {100,101,102,103}. However when
the application touches page 103, the lazy-readahead gets triggered and brings in 8 more pages {104,105,106,107,108,109,110,111} and as always all these pages go wasted. This wastage continues for ever.
My Optimization [ I mean hack ;) ] was meant to avoid this bad behavior. Instead of reading in 'number of pages accessed in the current-window', I read in 'one more page than the number of pages accessed in the current-window'. With this optimization the behavior changes to as follows:
when the application asks for {1,2,3,4} we happily satisfy them through the current-window. However when the application touches page 4, the lazy-readahead triggers and brings in pages {5,6,7,8,9,10,11,12} but now the application wants to access {100,101,102,103}. This bad behavior is probably ok since the optimization ensures that it does not commit the same mistake in the future. When the application tries to access {100,101,102,103} the code then scraps both the current window and the readahead window and reads in a new current window of size 4+1 i.e {100,101,102,103,104}.
However since the application does not touch page 104 and hence lazy-readahead does not get triggered we do not waste effort bringing in pages. And this nice behavior continues for ever.
Probably we may see marginal degradation of this optimization with 16k i/o but the amount of wastage avoided by this optimization (hack) is great when random i/o is of larger size. I think it was 4% better performance on DSS workload with 64k random reads.
Do you still think its a hack?
Also I think with sysbench workload and Andrew's ra-copy patch, we might be loosing some benefits of some of the optimization because if two threads simulteously work with copies of the same ra structure and update it, the optimization effect reflected in one of the ra-structure is lost depending on which ra structure gets copied back last.
RP
Quoted 6 lines · Ram Pai
Quoted 4 lines
Ram, can you take a look at fixing that up please? Something clean, not more hacks ;) I'd also be interested in an explanation of what the extra page is for. The little comment in there doesn't really help.
Quoted 6 lines
The reason for the extra page read is as follows:
Consider 16k random reads i/os. Reads are generated 4pages at a time.
the readahead is triggered when the 4th page in the 'current-window' is touched.
Right. We've added two whole unsigned longs to the file_struct to track the access patterns. That should be sufficient for us to detect when the access pattern is random, and to then not perform readahead due to a current-window miss *at all*.
So that extra page can go away, and:
mm/readahead.c renamed from mm/readahead.c~a +2 -6
@@ -492,21 +492,17 @@ do_io:*/ if (ra->ahead_start ==0 ) {/* -* if the average io-size is less than maximum +* If the average io-size is less than maximum * readahead size of the file the io pattern is * sequential. Hence bring in the readahead window * immediately. -* Else the i/o pattern is random. Bring -* in the readahead window only if the last page of -* the current window is accessed (lazy readahead). */ unsigned long average = ra->average;if (ra->serial_cnt > average) average = (ra->serial_cnt + ra->average) /2 ; -if ((average >= max) || (offset == (ra->start + - ra->size -1 ))) { +if (average >= max) { ra->ahead_start = ra->start + ra->size; ra->ahead_size = ra->next_size; actual = do_page_cache_readahead(mapping, filp,
_
That way, we read the correct amount of data, and we only start I/O when we know the application is going to actually use the data.
This may cause problems when the application transitions from seeky-access to linear-access.
Does it sound feasible?
Quoted 5 lines
Probably we may see marginal degradation of this optimization with 16k i/o but the amount of wastage avoided by this optimization (hack) is great when random i/o is of larger size. I think it was 4% better performance on DSS workload with 64k random reads.
64k sounds unusually large. We need top performance at 8k too.
Quoted 1 line
Do you still think its a hack?
yup ;)
Quoted 6 lines
Also I think with sysbench workload and Andrew's ra-copy patch, we might be loosing some benefits of some of the optimization because if two threads simulteously work with copies of the same ra structure and update it, the optimization effect reflected in one of the ra-structure is lost depending on which ra structure gets copied back last.
hm, maybe. That only makes a difference if two threads are accessing the same fd at the same time, and it was really bad before the patch. The IO patterns seemed OK to me with the patch. Except it's reading one page too many.
Quoted 51 lines · Andrew Morton
Quoted 6 lines · Ram Pai
Quoted 4 lines
Ram, can you take a look at fixing that up please? Something clean, not more hacks ;) I'd also be interested in an explanation of what the extra page is for. The little comment in there doesn't really help.
Quoted 6 lines
The reason for the extra page read is as follows:
Consider 16k random reads i/os. Reads are generated 4pages at a time.
the readahead is triggered when the 4th page in the 'current-window' is touched.
Right. We've added two whole unsigned longs to the
file_structto track the access patterns. That should be sufficient for us to detect when the access pattern is random, and to then not perform readahead due to a current-window miss *at all*.So that extra page can go away, and:
mm/readahead.c renamed from mm/readahead.c~a +2
-6@@ -492,21 +492,17 @@ do_io:*/ if (ra->ahead_start ==0 ) {/* -* if the average io-size is less than maximum +* If the average io-size is less than maximum * readahead size of the file the io pattern is * sequential. Hence bring in the readahead window * immediately. -* Else the i/o pattern is random. Bring -* in the readahead window only if the last page of -* the current window is accessed (lazy readahead). */ unsigned long average = ra->average;if (ra->serial_cnt > average) average = (ra->serial_cnt + ra->average) /2 ; -if ((average >= max) || (offset == (ra->start + - ra->size -1 ))) { +if (average >= max) { ra->ahead_start = ra->start + ra->size; ra->ahead_size = ra->next_size; actual = do_page_cache_readahead(mapping, filp,_
Quoted 7 lines
That way, we read the correct amount of data, and we only start I/O when we know the application is going to actually use the data.
This may cause problems when the application transitions from seeky-access to linear-access.
Does it sound feasible?
I am nervous about this change. You are totally getting rid of lazy-readahead and that was the optimization which gave the best possible boost in performance.
Let me see how this patch does with a DSS benchmark.
Quoted 7 lines
Quoted 6 lines
Probably we may see marginal degradation of this optimization with 16k i/o but the amount of wastage avoided by this optimization (hack) is great when random i/o is of larger size. I think it was 4% better performance on DSS workload with 64k random reads.
Quoted 6 lines
64k sounds unusually large. We need top performance at 8k too.
Quoted 1 line
Do you still think its a hack?
yup ;)
:-(
Quoted 11 lines
Quoted 6 lines
Also I think with sysbench workload and Andrew's ra-copy patch, we might be loosing some benefits of some of the optimization because if two threads simulteously work with copies of the same ra structure and update it, the optimization effect reflected in one of the ra-structure is lost depending on which ra structure gets copied back last.
hm, maybe. That only makes a difference if two threads are accessing the same fd at the same time, and it was really bad before the patch. The IO patterns seemed OK to me with the patch. Except it's reading one page too many.
In the normal large random workload this extra page would have compesated for all the wasted readaheads. However in the case of sysbench with Andrew's ra-copy patch the readahead calculation is not happening quiet right. Is it worth trying to get a marginal gain with sysbench at the cost of getting a big hit on DSS benchmarks, aio-tests,iozone and probably others. Or am I making an unsubstantiated claim? I will get back with results.
RP
Quoted 4 lines · Ram Pai
I am nervous about this change. You are totally getting rid of lazy-readahead and that was the optimization which gave the best possible boost in performance.
Because it disabled the large readahead outside the area which the app is reading. But it's still reading too much.
Quoted 1 line
Let me see how this patch does with a DSS benchmark.
That was not a real patch. More work is surely needed to get that right.
Quoted 2 lines
In the normal large random workload this extra page would have compesated for all the wasted readaheads.
I disagree that 64k is "normal"!
Quoted 6 lines
However in the case of
sysbench with Andrew's ra-copy patch the readahead calculation is not happening quiet right. Is it worth trying to get a marginal gain with sysbench at the cost of getting a big hit on DSS benchmarks, aio-tests,iozone and probably others. Or am I making an unsubstantiated claim? I will get back with results.
It shouldn't hurt at all - the app does a seek, we perform the correctly-sized read.
As I say, my main concern is that we correctly transition from seeky access to linear access and resume readahead.
Quoted 25 lines · Andrew Morton
Quoted 4 lines · Ram Pai
I am nervous about this change. You are totally getting rid of lazy-readahead and that was the optimization which gave the best possible boost in performance.
Because it disabled the large readahead outside the area which the app is reading. But it's still reading too much.
Quoted 1 line
Let me see how this patch does with a DSS benchmark.
That was not a real patch. More work is surely needed to get that right.
Quoted 2 lines
In the normal large random workload this extra page would have compesated for all the wasted readaheads.
I disagree that 64k is "normal"!
Quoted 6 lines
However in the case of
sysbench with Andrew's ra-copy patch the readahead calculation is not happening quiet right. Is it worth trying to get a marginal gain with sysbench at the cost of getting a big hit on DSS benchmarks, aio-tests,iozone and probably others. Or am I making an unsubstantiated claim? I will get back with results.
Quoted 2 lines
It shouldn't hurt at all - the app does a seek, we perform the correctly-sized read.
Looks like you are right on all counts! I did some modifications to your patch and did a preliminary run with my user-level simulator. With these changes I am able to get rid of that extra page. Also code looks much simpler and adapts well to sequential and random patterns.
However I have to run this under some benchmarks and see how it fares. Its a pre-alpha level patch.
Can you take a quick look at the changes and see if you like it? I am sure you won't consider these changes a hack ;)
RP
mm/readahead.c +25 -24
@@ -353,7 +353,7 @@ page_cache_readahead(struct address_spacunsigned orig_next_size;unsigned actual;int first_access=0 ; -unsigned long preoffset=0 ; +unsigned long average=0 ;/* * Here we detect the case where the application is performing
@@ -394,10 +394,17 @@ page_cache_readahead(struct address_spacif (ra->serial_cnt <= (max *2 )) ra->serial_cnt++; }else { - ra->average = (ra->average + ra->serial_cnt) /2 ; +/* to avoid rounding errors, ensure that 'average' +* tends towards the value of ra->serial_cnt. +*/ +if (ra->average > ra->serial_cnt) { + average = ra->average -1 ; + }else { + average = ra->average +1 ; + } + ra->average = (average + ra->serial_cnt) /2 ; ra->serial_cnt =1 ; } - preoffset = ra->prev_page; ra->prev_page = offset;if (offset >= ra->start && offset <= (ra->start + ra->size)) {
@@ -457,18 +464,14 @@ do_io:* ahead window and get some I/O underway for the new * current window. */ -if (!first_access && preoffset >= ra->start && - preoffset < (ra->start + ra->size)) { -/* Heuristic: If 'n' pages were -* accessed in the current window, there -* is a high probability that around 'n' pages -* shall be used in the next current window. -* -* To minimize lazy-readahead triggered -* in the next current window, read in -* an extra page. +if (!first_access) { +/* Heuristic: there is a high probability +* that around ra->average number of +* pages shall be accessed in the next +* current window. */ - ra->next_size = preoffset - ra->start +2 ; + ra->next_size = (ra->average > max ? + max : ra->average); } ra->start = offset; ra->size = ra->next_size;
@@ -492,21 +495,19 @@ do_io:*/ if (ra->ahead_start ==0 ) {/* -* if the average io-size is less than maximum +* If the average io-size is more than maximum * readahead size of the file the io pattern is * sequential. Hence bring in the readahead window -* immediately. -* Else the i/o pattern is random. Bring -* in the readahead window only if the last page of -* the current window is accessed (lazy readahead). +* immediately. +* If the average io-size is less than maximum +* readahead size of the file the io pattern is +* random. Hence don't bother to readahead. */ -unsigned long average = ra->average; - + average = ra->average;if (ra->serial_cnt > average) - average = (ra->serial_cnt + ra->average) /2 ; + average = (ra->serial_cnt + ra->average +1 ) /2 ; -if ((average >= max) || (offset == (ra->start + - ra->size -1 ))) { +if (average > max) { ra->ahead_start = ra->start + ra->size; ra->ahead_size = ra->next_size; actual = do_page_cache_readahead(mapping, filp,
Quoted 2 lines · Ram Pai
Looks like you are right on all counts!
It's a probabilistic thing.
Quoted 4 lines
I did some modifications to your
patch and did a preliminary run with my user-level simulator. With these changes I am able to get rid of that extra page. Also code looks much simpler and adapts well to sequential and random patterns.
That is good news.
Quoted 2 lines
However I have to run this under some benchmarks and see how it fares. Its a pre-alpha level patch.
It is nicer, thanks. I'll add it to -mm and hopefully Meredith and co will include it in regular performance testing.
Quoted 2 lines
Can you take a quick look at the changes and see if you like it? I am sure you won't consider these changes a hack ;)
Couple of minor things:
mm/readahead.c page_cache_readahead(struct address_spac · line 356
Quoted 1 line
mm/readahead.c +0
-1-unsigned long preoffset=0 ;
yay!
mm/readahead.c page_cache_readahead(struct address_spac · line 356
Quoted 1 line
mm/readahead.c +1
-0+unsigned long average=0 ;
Please add spaces around '='. But I don't think this needs to be initialised at all.
mm/readahead.c page_cache_readahead(struct address_spac · line 399
Quoted 10 lines
mm/readahead.c +3
-1/* * Here we detect the case where the application is performing @@ -394,10 +394,17 @@ page_cache_readahead(struct address_spacif (ra->serial_cnt <= (max *2 )) ra->serial_cnt++; }else { - ra->average = (ra->average + ra->serial_cnt) /2 ; +/* to avoid rounding errors, ensure that 'average' +* tends towards the value of ra->serial_cnt. +*/
multiline comment layout:
/* * To avoid rounding errors, ensure that 'average' tends * towards the value of ra->serial_cnt. */
(I said "minor").
I can't say that I immediately understand what is the issue here with rounding errors?
mm/readahead.c page_cache_readahead(struct address_spac · line 400
Quoted 1 line
mm/readahead.c +1
-0+if (ra->average > ra->serial_cnt) {
space between "if" and "("
mm/readahead.c do_io: · line 474
Quoted 2 lines
mm/readahead.c +2
-0+ ra->next_size = (ra->average > max ? + max : ra->average);
min(max, ra->average) ?
Quoted 1 line · Andrew Morton
Ram Pai <[email protected]> wrote:
I am yet to get my machine fully set up to run a DSS benchmark. But thought I will update you on the following comment.
Quoted 8 lines
multiline comment layout:
/* * To avoid rounding errors, ensure that 'average' tends * towards the value of ra->serial_cnt. */
Quoted 4 lines
(I said "minor").
I can't say that I immediately understand what is the issue here with rounding errors?
Say the i/o size is 20 pages.
Our algorithm starts by a initial average i/o size of 'ra_pages/2' which is mostly say 16.
Now every time we take a average, the 'average' progresses as follows (16+20)/2=18
(18+20)/2=19
(19+20)/2=19
(19+20)/2=19.....
and the rounding error makes it never touch 20
However the code can be further optimized to :
/*
* to avoid rounding errors, ensure that 'average'
* tends towards the value of ra->serial_cnt.
*/
if (ra->average < ra->serial_cnt) {
average = ra->average + 1;
}I will send a updated patch with all your comments incorporated as soon as I see good benchmark numbers.(probably by tomorrow).
RP
Quoted 13 lines
Quoted 3 lines
+
if(ra->average > ra->serial_cnt){space between "if" and "("
mm/readahead.c do_io: · line 474
Quoted 2 lines
mm/readahead.c +2
-0+ ra->next_size = (ra->average > max ? + max : ra->average);min(max, ra->average) ?
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 5 lines · Ram Pai
Quoted 1 line · Andrew Morton
Ram Pai <[email protected]> wrote:
I am yet to get my machine fully set up to run a DSS benchmark. But thought I will update you on the following comment.
Attached the cleaned up patch and the performance results of the patch.
Overall Observation:
1.Small improvement with iozone with the patch, and overall
much better performance than 2.4
2.Small/neglegible improvement with DSS workload. 3.Negligible impact with sysbench, but results worser than
2.4 kernels
RP
Results of iozone,sysbench and DSS workload with the
seeky-readahead-speedups.patch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Overall Observation:
1.Small improvement with iozone with the patch, and overall
much better performance than 2.4
2.Small/neglegible improvement with DSS workload. 3.Negligible impact with sysbench, but results worser than
2.4 kernels
The cleaned-up patch is included towards the end of this report.
Details:
**********************************************************************
IOZONE
run on a nfs mounted filesystem:
client machine 2proc, 733MHz, 2GB memory
server machine 8proc, 700Mhz, 8GB memory
./iozone -c -t1 -s 4096m -r 128k
---------------------------------------------------------
| | throughput | throughput | throughput | | | KB/sec | KB/sec | KB/sec | | | 266 | 266+patch | 2.4.20 | --------------------------------------------------------- |sequential read| 11697.55 | 11700.98 | 10846.87 | | | | | | |re-read | 11698.39 | 11691.84 | 10865.39 | | | | | | |reverse read | 20002.71 | 20099.86 | 10340.34 | | | | | | |stride read | 13813.01 | 13850.28 | 10193.87 | | | | | | |random read | 19705.06 | 19978.00 | 10839.57 | | | | | | |random mix | 28465.68 | 29964.38 | 10779.17 | | | | | | |pread | 11692.95 | 11697.29 | 10863.56 |
---------------------------------------------------------
**************************************************************
SYSBENCH
run on machine 2proc, 733MHz, 256MB memory
---------------------------------------------------------
| | 266 | 266+patch | 2.4.21 |
---------------------------------------------------------
|time spent | 79.6253 | 79.8176 | 73.2605sec | | | | | | |Mb/sec | 1.959Mb.sec| 1.954Mb/sec| 2.129Mb/sec| | | | | | |requests/sec | 125.59 | 125.29 | 136.54 | | | | | | |no of Reads | 6001 | 6001 | 6008 | | | | | | |no of Writes | 3999 | 3999 | 3995 | | | | | |
---------------------------------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
266 sysbench output:
Operations performed: 6001 Read, 3999 Write, 12800 Other = 22800 Total Read 93Mb Written 62Mb Total Transferred 156Mb
1.959Mb/sec Transferred
125.59 Requests/sec executed
Test execution Statistics summary:
Time spent for test: 79.6253s
Per Request statistics:
Min: 0.0000s Avg: 0.0467s Max: 0.9802s Events tracked: 10000
Total time taken by event execution: 467.1493s
Threads fairness: 87.41/94.20 distribution, 88.68/94.45 execution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
266+patch sysbench output:
Operations performed: 6001 Read, 3999 Write, 12800 Other = 22800 Total Read 93Mb Written 62Mb Total Transferred 156Mb
1.954Mb/sec Transferred
125.29 Requests/sec executed
Test execution Statistics summary:
Time spent for test: 79.8176s
Per Request statistics:
Min: 0.0000s Avg: 0.0482s Max: 0.8481s Events tracked: 10000
Total time taken by event execution: 481.7572s
Threads fairness: 85.27/93.25 distribution, 85.15/94.91 execution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2.4.21 sysbench output:
Operations performed: 6008 Read, 3995 Write, 12800 Other = 22803 Total Read 93Mb Written 62Mb Total Transferred 156Mb
2.129Mb/sec Transferred
136.54 Requests/sec executed
Test execution Statistics summary:
Time spent for test: 73.2605s
Per Request statistics:
Min: 0.0000s Avg: 0.0380s Max: 0.3712s Events tracked: 10003
Total time taken by event execution: 380.4081s
Threads fairness: 79.04/91.95 distribution, 82.52/92.44 execution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**************************************************************
DSS WORKLOAD
Got 1% improvement with the patch
**************************************************************
mm/readahead.c +24 -24
@@ -353,7 +353,7 @@ page_cache_readahead(struct address_spacunsigned orig_next_size;unsigned actual;int first_access=0 ; -unsigned long preoffset=0 ; +unsigned long average;/* * Here we detect the case where the application is performing
@@ -394,10 +394,17 @@ page_cache_readahead(struct address_spacif (ra->serial_cnt <= (max *2 )) ra->serial_cnt++; }else { - ra->average = (ra->average + ra->serial_cnt) /2 ; +/* +* to avoid rounding errors, ensure that 'average' +* tends towards the value of ra->serial_cnt. +*/ + average = ra->average; +if (average < ra->serial_cnt) { + average++; + } + ra->average = (average + ra->serial_cnt) /2 ; ra->serial_cnt =1 ; } - preoffset = ra->prev_page; ra->prev_page = offset;if (offset >= ra->start && offset <= (ra->start + ra->size)) {
@@ -457,18 +464,13 @@ do_io:* ahead window and get some I/O underway for the new * current window. */ -if (!first_access && preoffset >= ra->start && - preoffset < (ra->start + ra->size)) { -/* Heuristic: If 'n' pages were -* accessed in the current window, there -* is a high probability that around 'n' pages -* shall be used in the next current window. -* -* To minimize lazy-readahead triggered -* in the next current window, read in -* an extra page. +if (!first_access) { +/* Heuristic: there is a high probability +* that around ra->average number of +* pages shall be accessed in the next +* current window. */ - ra->next_size = preoffset - ra->start +2 ; + ra->next_size =min (ra->average , (unsigned long )max); } ra->start = offset; ra->size = ra->next_size;
@@ -492,21 +494,19 @@ do_io:*/ if (ra->ahead_start ==0 ) {/* -* if the average io-size is less than maximum +* If the average io-size is more than maximum * readahead size of the file the io pattern is * sequential. Hence bring in the readahead window -* immediately. -* Else the i/o pattern is random. Bring -* in the readahead window only if the last page of -* the current window is accessed (lazy readahead). +* immediately. +* If the average io-size is less than maximum +* readahead size of the file the io pattern is +* random. Hence don't bother to readahead. */ -unsigned long average = ra->average; - + average = ra->average;if (ra->serial_cnt > average) - average = (ra->serial_cnt + ra->average) /2 ; + average = (ra->serial_cnt + ra->average +1 ) /2 ; -if ((average >= max) || (offset == (ra->start + - ra->size -1 ))) { +if (average > max) { ra->ahead_start = ra->start + ra->size; ra->ahead_size = ra->next_size; actual = do_page_cache_readahead(mapping, filp,
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 8 lines · Ram Pai
Attached the cleaned up patch and the performance results of the patch.
Overall Observation:
1.Small improvement with iozone with the patch, and overall
much better performance than 2.4
2.Small/neglegible improvement with DSS workload. 3.Negligible impact with sysbench, but results worser than
2.4 kernels
Ram, can you clarify the status of this patch please?
I ran the same sysbench test on my hardware with patched 2.6.6 and got 122.2348s execution time, i.e. almost the same results as in the original tests. Is this patch an intermediate step to improve the sysbench workload on 2.6, or it just addresses another problem?
Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 11 lines · Alexey Kopytov
Quoted 8 lines · Ram Pai
Attached the cleaned up patch and the performance results of the patch.
Overall Observation:
1.Small improvement with iozone with the patch, and overall
much better performance than 2.4
2.Small/neglegible improvement with DSS workload. 3.Negligible impact with sysbench, but results worser than
2.4 kernels
Quoted 6 lines
Ram, can you clarify the status of this patch please?
I ran the same sysbench test on my hardware with patched 2.6.6 and got 122.2348s execution time, i.e. almost the same results as in the original tests. Is this patch an intermediate step to improve the sysbench workload on 2.6, or it just addresses another problem?
this patch by itself does not address your problem. Your problem is better addressed by Andrew's 'readahead-private' patch.
However; this patch applied on top of Andrew's 'readahead-private' patch may get you some extra performance.
Can you confirm this please?
RP
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 8 lines · Ram Pai
Quoted 6 lines · Alexey Kopytov
Ram, can you clarify the status of this patch please?
I ran the same sysbench test on my hardware with patched 2.6.6 and got 122.2348s execution time, i.e. almost the same results as in the original tests. Is this patch an intermediate step to improve the sysbench workload on 2.6, or it just addresses another problem?
Quoted 7 lines
this patch by itself does not address your problem. Your problem is better addressed by Andrew's 'readahead-private' patch.
However; this patch applied on top of Andrew's 'readahead-private' patch may get you some extra performance.
Can you confirm this please?
Yes.
2.6.6-rc3 + Andrew's patch:
Time spent for test: 86.5459s
2.6.6-bk:
Time spent for test: 83.1929s
Thanks for clarifying!
Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Re: Random file I/O regressions in 2.6 [patch+results]
In reply to Alexey Kopytov(Resend due to osdl<->vger smtp bunfight)
Quoted 5 lines · Alexey Kopytov
Quoted 2 lines · Ram Pai
Attached the cleaned up patch and the performance results of the patch.
Quoted 8 lines
Quoted 6 lines
Overall Observation:
1.Small improvement with iozone with the patch, and overall
much better performance than 2.4
2.Small/neglegible improvement with DSS workload. 3.Negligible impact with sysbench, but results worser than
2.4 kernelsRam, can you clarify the status of this patch please?
Everything we have is now in Linus's tree. And in 2.6.6-mm4.
Quoted 4 lines
I ran the same sysbench test on my hardware with patched 2.6.6 and got 122.2348s execution time, i.e. almost the same results as in the original tests. Is this patch an intermediate step to improve the sysbench workload on 2.6, or it just addresses another problem?
The patches in Linus's tree improve sysbench significantly here. It's a 256MB 2-way with IDE disks, writeback caching enabled:
sysbench --num-threads=16 --test=fileio --file-total-size=2G --file-test-mode=rndrw run
2.4.27-pre2, ext2:
Time spent for test: 61.0240s 0.06s user 6.03s system 4% cpu 2:05.95 total Time spent for test: 60.8456s 0.11s user 5.49s system 4% cpu 2:04.94 total
2.6.6, CFQ, ext2:
Time spent for test: 85.6614s 0.05s user 5.66s system 3% cpu 2:26.75 total Time spent for test: 85.2090s 0.06s user 5.32s system 3% cpu 2:24.75 total
2.6.6-bk, CFQ, ext2:
Time spent for test: 66.7717s 0.04s user 5.54s system 4% cpu 2:06.19 total Time spent for test: 67.5666s 0.04s user 5.10s system 4% cpu 2:06.72 total
2.6.6, as, ext2:
Time spent for test: 83.8358s 0.07s user 5.89s system 4% cpu 2:22.92 total Time spent for test: 83.8068s 0.06s user 5.34s system 3% cpu 2:21.33 total
2.6.6-bk, AS, ext2:
Time spent for test: 62.5316s 0.05s user 5.27s system 4% cpu 2:01.28 total Time spent for test: 62.7401s 0.04s user 5.17s system 4% cpu 2:00.50 total
2.6.6, deadline, ext2:
Time spent for test: 103.0084s 0.06s user 5.76s system 3% cpu 2:40.74 total Time spent for test: 101.9648s 0.07s user 5.35s system 3% cpu 2:38.83 total
2.6.6-bk, deadline, ext2:
Time spent for test: 63.3405s 0.03s user 5.49s system 4% cpu 2:01.05 total Time spent for test: 63.5288s 0.03s user 5.05s system 4% cpu 2:00.78 total
There's still something wrong here. 2.6.6-bk+deadline is pretty equivalent to 2.4 from an IO scheduler point of view in this test. Yet it's a couple of percent slower.
I don't know why you're still seeing significant discrepancies.
What sort of disk+controller system are you using? If scsi, what is the tag queue depth set to? Is writeback caching enabled on the disk?
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 9 lines · Andrew Morton
There's still something wrong here. 2.6.6-bk+deadline is pretty equivalent to 2.4 from an IO scheduler point of view in this test. Yet it's a couple of percent slower.
I don't know why you're still seeing significant discrepancies.
What sort of disk+controller system are you using? If scsi, what is the tag queue depth set to? Is writeback caching enabled on the disk?
If the 2.4 and 2.6 disk accounting statitics are to be believed, they show something interesting.
Workload is one run of
sysbench --num-threads=16 --test=fileio --file-total-size=2G --file-test-mode=rndrw run
on ext2.
2.4.27-pre2:
rio: 5549 (Read requests issued)
rblk: 259680 (Total sectors read)
wio: 42398 (Write requests issued)
wblk: 4368056 (Total sectors written)
2.6.6-bk, as:
reads: 5983
readsectors: 201192
writes: 22548
writesectors: 4343184
- Note that 2.6 read 20% less data from the disk. We observed this before. It appears that 2.6 page replacements decisions are working better for this workload.
- Despite that, 2.6 issued *more* read requests. So it is submitting more, and smaller I/O's
- Both kernels wrote basically the same amount of data. 2.6 a little less, perhaps because of fsync() optimisations.
- But 2.6 issued far fewer write requests. Half as many as 2.4 - a huge difference. There are a number of reasons why this could happen but frankly, I don't have a clue what's going on in there.
Given that 2.6 is issuing less IO requests it should be performing faster than 2.4. The reason that the two kernels are achieving about the same throughput despite this is that the disk is performing writeback caching and is absorbing 2.4's smaller write requests.
I set the IDE disk to do writethrough (hdparm -W0):
2.6.6-bk, as:
Time spent for test: 89.9427s 0.04s user 5.24s system 1% cpu 4:51.62 total
2.4.27-pre2:
Time spent for test: 107.8293s 0.04s user 6.00s system 1% cpu 7:26.47 total
as expected.
Open questions are:
a) Why is 2.6 write coalescing so superior to 2.4?
b) Why is 2.6 issuing more read requests, for less data?
c) Why is Alexey seeing dissimilar results?
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 8 lines · Andrew Morton
Open questions are:
a) Why is 2.6 write coalescing so superior to 2.4?
b) Why is 2.6 issuing more read requests, for less data?
c) Why is Alexey seeing dissimilar results?
Interesting. I am not too familiar with 2.4's IO scheduler, but 2.6's have pretty comprehensive merging systems. Could that be helping, Jens? Or is 2.4 pretty equivalent?
What about things like maximum request size for 2.4 vs 2.6 for example? This is another thing that can have an impact, especially for writes.
I'll take a guess at b, and say it could be as-iosched.c. Another thing might be that 2.6 has smaller nr_requests than 2.4, although you are unlikely to hid the read side limit with only 16 threads if they are doing sync IO.
As for question c, has Alexey confirmed that it is indeed 2.6-bk which has problems?
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 8 lines · Nick Piggin
Quoted 6 lines · Andrew Morton
Open questions are:
a) Why is 2.6 write coalescing so superior to 2.4?
b) Why is 2.6 issuing more read requests, for less data?
Quoted 7 lines
Quoted 2 lines
c) Why is Alexey seeing dissimilar results?
Interesting. I am not too familiar with 2.4's IO scheduler, but 2.6's have pretty comprehensive merging systems. Could that be helping, Jens? Or is 2.4 pretty equivalent?
2.4 will give up merging faster than 2.6, elevator_linus will stop looking for a merge point if the sequence drops to zero. 2.6 will always merge. So that could explain the fewer writes.
Quoted 3 lines
What about things like maximum request size for 2.4 vs 2.6 for example? This is another thing that can have an impact, especially for writes.
I think that's pretty similar. Andrew didn't say what device he was testing on, but 2.4 ide defaults to max 64k where 2.6 defaults to 128k.
Quoted 4 lines
I'll take a guess at b, and say it could be
as-iosched.c. Another thing might be that 2.6 has smallernr_requeststhan 2.4, although you are unlikely to hid the read side limit with only 16 threads if they are doing sync IO.
Andrew, you did numbers for deadline previously as well, but no rq statistics there? As for nr_requests that's true, would be worth a shot to bump available requests in 2.6.
Signature · 1 line
Jens Axboe
Spam: Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 3 lines · Jens Axboe
I think that's pretty similar. Andrew didn't say what device he was
testing on, but 2.4 ide defaults to max 64k where 2.6 defaults to 128k.
IDE.
I was being silly, sorry. Those I/O stats include the (huge linear) initial write of the "database" files, so the larger IDE request size will be dominating.
What I need is a way of getting sysbench to create and remove the database files in separate invokations, but the syntax for that is defeating me at present.
Quoted 8 lines
Quoted 4 lines
I'll take a guess at b, and say it could be
as-iosched.c. Another thing might be that 2.6 has smallernr_requeststhan 2.4, although you are unlikely to hid the read side limit with only 16 threads if they are doing sync IO.Andrew, you did numbers for deadline previously as well, but no rq statistics there? As for
nr_requeststhat's true, would be worth a shot to bump available requests in 2.6.
Doubling the request queue size makes no difference.
Re: Spam: Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 5 lines · Andrew Morton
What I need is a way of getting sysbench to create and remove the database files in separate invokations, but the syntax for that is defeating me at present.
I have changed the syntax to allow creating/removing test files and test running in separate stages:
sysbench --test=fileio --file-total-size=3G prepare
sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndrw run
sysbench --test=fileio cleanup
The updated version is available from the SysBench page at
http://sourceforge.net/projects/sysbench/
Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Re: Random file I/O regressions in 2.6 [patch+results]
In reply to Jens AxboeQuoted 15 lines · Jens Axboe
Quoted 13 lines · Nick Piggin
Quoted 8 lines · Andrew Morton
Open questions are:
a) Why is 2.6 write coalescing so superior to 2.4?
b) Why is 2.6 issuing more read requests, for less data?
c) Why is Alexey seeing dissimilar results?
Quoted 9 lines
Quoted 3 lines
Interesting. I am not too familiar with 2.4's IO scheduler, but 2.6's have pretty comprehensive merging systems. Could that be helping, Jens? Or is 2.4 pretty equivalent?
2.4 will give up merging faster than 2.6,
elevator_linuswill stop looking for a merge point if the sequence drops to zero. 2.6 will always merge. So that could explain the fewer writes.
Yep OK, that could be one thing.
Quoted 9 lines
Quoted 4 lines
What about things like maximum request size for 2.4 vs 2.6 for example? This is another thing that can have an impact, especially for writes.
I think that's pretty similar. Andrew didn't say what device he was testing on, but 2.4 ide defaults to max 64k where 2.6 defaults to 128k.
This could be another. If Andrew's using IDE, this alone could make up the entire difference *if* writes are nicely sequential. I guess they probably aren't, but it could still help.
Re: Random file I/O regressions in 2.6 [patch+results]
In reply to Andrew MortonQuoted 13 lines · Andrew Morton
The patches in Linus's tree improve sysbench significantly here. It's a 256MB 2-way with IDE disks, writeback caching enabled:
sysbench --num-threads=16 --test=fileio --file-total-size=2G --file-test-mode=rndrw run
2.4.27-pre2, ext2:
Time spent for test: 61.0240s 0.06s user 6.03s system 4% cpu 2:05.95 total Time spent for test: 60.8456s 0.11s user 5.49s system 4% cpu 2:04.94 total
Quoted 6 lines
2.6.6-bk, AS, ext2:
Time spent for test: 62.5316s 0.05s user 5.27s system 4% cpu 2:01.28 total Time spent for test: 62.7401s 0.04s user 5.17s system 4% cpu 2:00.50 total
I ran the tests with a configuration as close to yours as possible. Here are the results for mem=256M, 2G total file size (ext3):
2.4.25:
Time spent for test: 79.4146s
0.20user 16.08system 3:20.29elapsed 8%CPU
Time spent for test: 78.9797s
0.11user 15.84system 3:19.76elapsed 7%CPU 2.6.6-bk, AS:
Time spent for test: 81.2208s
0.13user 17.97system 3:13.30elapsed 9%CPU
Time spent for test: 82.5538s
0.14user 18.00system 3:14.88elapsed 9%CPUThis correlates very well your results. But when I returned back to my original configuration (mem=640M, 3G total file size), I got the following:
2.4.25:
Time spent for test: 77.5377s
2.6.6-bk, AS:
Time spent for test: 83.1929s
It seems like the smaller file size just hides the regression, but I have to run some more tests to ensure this.
Quoted 4 lines
I don't know why you're still seeing significant discrepancies.
What sort of disk+controller system are you using? If scsi, what is the tag queue depth set to? Is writeback caching enabled on the disk?
It's IDE disk without TCQ support with writeback caching enabled.
Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Re: Random file I/O regressions in 2.6 [patch+results]
Quoted 18 lines · Alexey Kopytov
I ran the tests with a configuration as close to yours as possible. Here are the results for mem=256M, 2G total file size (ext3):
2.4.25:
Time spent for test: 79.4146s 0.20user 16.08system 3:20.29elapsed 8%CPU Time spent for test: 78.9797s 0.11user 15.84system 3:19.76elapsed 7%CPU2.6.6-bk, AS:
Time spent for test: 81.2208s 0.13user 17.97system 3:13.30elapsed 9%CPU Time spent for test: 82.5538s 0.14user 18.00system 3:14.88elapsed 9%CPUThis correlates very well your results. But when I returned back to my original configuration (mem=640M, 3G total file size), I got the following:
Quoted 9 lines
2.4.25:
Time spent for test: 77.5377s2.6.6-bk, AS:
Time spent for test: 83.1929sIt seems like the smaller file size just hides the regression, but I have to run some more tests to ensure this.
The assumption appears to be true. I tried to vary the total file size and got the following results (tests were done on another IDE disk):
2.4.27-pre3:
2 GB: 58.2707s
4 GB: 72.3313s
8 GB: 83.082s2.6.7-rc1, AS:
2 GB: 60.6792s
4 GB: 82.8023s
8 GB: 99.4398sVarying the number of files while keeping the total file size constant also gives some interesting results:
2.4.27-pre3, 4 GB total file size:
1 file: 71.7288s
128 files: 72.3313s
256 files: 73.92682.6.7-rc1, AS, 4 GB total file size:
1 file: 76.443
128 files: 82.8023
256 files: 81.9618Signature · 4 lines
Alexey Kopytov, Software Developer MySQL AB, www.mysql.com Are you MySQL certified? www.mysql.com/certification
Re: Random file I/O regressions in 2.6 [patch+results]
In reply to Alexey KopytovQuoted 5 lines · Alexey Kopytov
Quoted 2 lines · Ram Pai
Attached the cleaned up patch and the performance results of the patch.
Quoted 8 lines
Quoted 6 lines
Overall Observation:
1.Small improvement with iozone with the patch, and overall
much better performance than 2.4
2.Small/neglegible improvement with DSS workload. 3.Negligible impact with sysbench, but results worser than
2.4 kernelsRam, can you clarify the status of this patch please?
Everything we have is now in Linus's tree. And in 2.6.6-mm4.
Quoted 4 lines
I ran the same sysbench test on my hardware with patched 2.6.6 and got 122.2348s execution time, i.e. almost the same results as in the original tests. Is this patch an intermediate step to improve the sysbench workload on 2.6, or it just addresses another problem?
The patches in Linus's tree improve sysbench significantly here. It's a 256MB 2-way with IDE disks, writeback caching enabled:
sysbench --num-threads=16 --test=fileio --file-total-size=2G --file-test-mode=rndrw run
2.4.27-pre2, ext2:
Time spent for test: 61.0240s 0.06s user 6.03s system 4% cpu 2:05.95 total Time spent for test: 60.8456s 0.11s user 5.49s system 4% cpu 2:04.94 total
2.6.6, CFQ, ext2:
Time spent for test: 85.6614s 0.05s user 5.66s system 3% cpu 2:26.75 total Time spent for test: 85.2090s 0.06s user 5.32s system 3% cpu 2:24.75 total
2.6.6-bk, CFQ, ext2:
Time spent for test: 66.7717s 0.04s user 5.54s system 4% cpu 2:06.19 total Time spent for test: 67.5666s 0.04s user 5.10s system 4% cpu 2:06.72 total
2.6.6, as, ext2:
Time spent for test: 83.8358s 0.07s user 5.89s system 4% cpu 2:22.92 total Time spent for test: 83.8068s 0.06s user 5.34s system 3% cpu 2:21.33 total
2.6.6-bk, AS, ext2:
Time spent for test: 62.5316s 0.05s user 5.27s system 4% cpu 2:01.28 total Time spent for test: 62.7401s 0.04s user 5.17s system 4% cpu 2:00.50 total
2.6.6, deadline, ext2:
Time spent for test: 103.0084s 0.06s user 5.76s system 3% cpu 2:40.74 total Time spent for test: 101.9648s 0.07s user 5.35s system 3% cpu 2:38.83 total
2.6.6-bk, deadline, ext2:
Time spent for test: 63.3405s 0.03s user 5.49s system 4% cpu 2:01.05 total Time spent for test: 63.5288s 0.03s user 5.05s system 4% cpu 2:00.78 total
There's still something wrong here. 2.6.6-bk+deadline is pretty equivalent to 2.4 from an IO scheduler point of view in this test. Yet it's a couple of percent slower.
I don't know why you're still seeing significant discrepancies.
What sort of disk+controller system are you using? If scsi, what is the tag queue depth set to? Is writeback caching enabled on the disk?
Quoted 31 lines · Andrew Morton
Quoted 3 lines · Ram Pai
I am nervous about this change. You are totally getting rid of lazy-readahead and that was the optimization which gave the best possible boost in performance.
Because it disabled the large readahead outside the area which the app is reading. But it's still reading too much.
Quoted 1 line
Let me see how this patch does with a DSS benchmark.
That was not a real patch. More work is surely needed to get that right.
Quoted 2 lines
In the normal large random workload this extra page would have compesated for all the wasted readaheads.
I disagree that 64k is "normal"!
Quoted 5 lines
However in the case of
sysbench with Andrew's ra-copy patch the readahead calculation is not
happening quiet right. Is it worth trying to get a marginal gain with sysbench at the cost of getting a big hit on DSS benchmarks, aio-tests,iozone and probably others. Or am I making an unsubstantiated
claim? I will get back with results.
Quoted 5 lines
It shouldn't hurt at all - the app does a seek, we perform the correctly-sized read.
As I say, my main concern is that we correctly transition from seeky access to linear access and resume readahead.
One real problem is that you are trying to do in the kernel what would be best done in the application and better done in glibc... Because the benefit of readahead varies based on fd rather than device. Consider a program reading data from a file and putting it in a database. The benefit of readahead for the sequential access data file is higher than seek-read combinations. The library could do readahead based on the bytes read since the last seek on a by-file basis, something the kernel can't.
This is not to say the kernel work hasn't been a benefit, but note that with all the patches 2.4 still seems to outperform 2.6. And that's a problem since other parts of 2.6 scale so well. I do see that 2.4 seems to outperform 2.6 for usenet news, where you have small reads against a modest database, a few TB or so, and 400-2000 processes doing random reads against the data. Settings and schedulers seem to have only modest effect there.
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 26 lines · Nick Piggin
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Nick Piggin
Quoted 11 lines
That's one of its usage patterns. It's also supposed to detect the fixed-sized-reads-seeking-all-over-the-place situation. In which case it's supposed to submit correctly-sized multi-page BIOs. But it's not working right for this workload.
A naive solution would be to add special-case code which always does the fixed-size readahead after a seek. Basically that's
if (ra->next_size == -1UL) force_page_cache_readahead(...)Quoted 8 lines
Quoted 3 lines
I think a better solution to this case would be to ensure the readahead window is always
min(size of read, some large number);That would cause the kernel to perform lots of pointless pagecache lookups when the file is already 100% cached.
Quoted 4 lines
That's pretty sad. You need a "preread" or something which sends the pages back... or uses the actor itself. readahead would then have to be reworked to only run off the end of the read window, but that is what it should be doing anyway.
Sorry, I do not understand that paragraph at all.
All forms or pagecache population need to examine the pagecache to find out if the page is already there. This involves pagecache lookups. We want the read code to "learn" that the requested pages are all coming from cache and to stop doing those lookups altogether.
Quoted 8 lines · Andrew Morton
Sorry, I do not understand that paragraph at all.
All forms or pagecache population need to examine the pagecache to find out if the page is already there. This involves pagecache lookups. We want the read code to "learn" that the requested pages are all coming from cache and to stop doing those lookups altogether.
Yeah I think I have an idea of what the basic problems are, but I'd have to understand things better before I know if I am really on the right track.
My idea would probably also involve redoing some of the code code too, so at the moment I don't think I have time. If your simple fix works though, then that sounds good.
Quoted 4 lines
That would cause the kernel to perform lots of pointless pagecache lookups when the file is already 100% cached.
well surely the read itself will do those AGAIN anyway, so in the fully cached case this is just warming up the cpu cache ;) (and thus really cheap as nett cost I suspect)
Quoted 9 lines · Arjan van de Ven
Quoted 5 lines
That would cause the kernel to perform lots of pointless pagecache lookups when the file is already 100% cached.
well surely the read itself will do those AGAIN anyway, so in the fully cached case this is just warming up the cpu cache ;) (and thus really cheap as nett cost I suspect)
Probably true for x86, but the cost is noticeable on ppc64, for example. Anton fixed some things in there shortly after it went in, but it's still apparent on profiles.
We could perhaps speed things up a little bit by using gang lookup in both __do_page_cache_readahead() and in do_generic_file_read().
Quoted 7 lines · Andrew Morton
Quoted 6 lines · Arjan van de Ven
Quoted 5 lines
That would cause the kernel to perform lots of pointless pagecache lookups when the file is already 100% cached.
Quoted 7 lines
Quoted 3 lines
well surely the read itself will do those AGAIN anyway, so in the fully cached case this is just warming up the cpu cache ;) (and thus really cheap as nett cost I suspect)
Probably true for x86, but the cost is noticeable on ppc64, for example. Anton fixed some things in there shortly after it went in, but it's still apparent on profiles.
well do the profiles also show that the actual later lookup becomes near free due to a warm cpu cache?
Quoted 3 lines
We could perhaps speed things up a little bit by using gang lookup in both
__do_page_cache_readahead()and indo_generic_file_read().
or go into the readahead path only when the first miss occurs; for the fully cached case you can then avoid the cost while when you're doing IO, well, a few premature cache misses...