Direct io on block device has performance regression on 2.6.x kernel
I don't know where to start, but let me start with the bombshell:
Direct I/O on block device running 2.6.X kernel is a lot SLOWER than running on a 2.4 Kernel!
Processing a direct I/O request on 2.6 is taking a lot more cpu time compare to the same I/O request running on a 2.4 kernel.
The proof: easy. I started off by having a pseudo disk, a software disk that has zero access latency. By hooking this pseudo disk into the block layer API, I can effectively stress the entire I/O stack above the block level. Combined with user level test programs that simply submit direct I/O in a simple while loop, I can measure how fast kernel can process these I/O requests. The performance metric can be either throughput (# of I/O per second) or per unit of work (processing time per I/O). For the data presented below, I'm using throughput metric (meaning larger number is better performance). Pay attention to relative percentage as absolute number depends on platform/CPU that test suite runs on.
synchronous I/O AIO
(pread/pwrite/read/write) io_submit
2.4.21 based
(RHEL3) 265,122 229,810
2.6.9 218,565 206,917
2.6.10 213,041 205,891
2.6.11 212,284 201,124
Quoted 1 line
From the above chart, you can see that 2.6 kernel is at least 18%
slower in processing direct I/O (on block device) in the synchronous path and 10% slower in the AIO path compare to a distributor's 2.4 kernel. What's worse, with each advance of kernel version, the I/O path is becoming slower and slower.
Most of the performance regression for 2.6.9 came from dio layer (I still have to find where the regression came from with 2.6.10 and 2.6.11). DIO is just overloaded with too many areas to cover. I think it's better to break things up a little bit.
For example, by having a set of dedicated functions that do direct I/O on block device improves the performance dramatically:
synchronous I/O AIO
(pread/pwrite/read/write) io_submit
2.4.21 based
(RHEL3) 265,122 229,810
2.6.9 218,565 206,917
2.6.9+patches 323,016 268,484
See, we can be actually 22% faster in synchronous path and 17% faster In the AIO path, if we do it right!
Kernel patch and test suite to follow in the next couple postings.
- Ken
Quoted 4 lines · Kenneth W
Direct I/O on block device running 2.6.X kernel is a lot SLOWER
than running on a 2.4 Kernel!
A little bit slower, it appears. It used to be faster.
Review
Quoted 10 lines
...
(quoted diff) +0
-0synchronous I/O AIO (pread/pwrite/read/write) io_submit 2.4.21 based (RHEL3) 265,122 229,810 2.6.9 218,565 206,917 2.6.10 213,041 205,891 2.6.11 212,284 201,124
What sort of CPU?
What speed CPU?
What size requests?
Reads or writes?
At 5 usecs per request I figure that's 3% CPU utilisation for 16k requests at 100 MB/sec.
Once you bolt this onto a real device driver the proportional difference will fall, due to addition of the constant factor.
Once you bolt all this onto a real disk controller all the numbers will get worse (but in a strictly proportional manner) due to the disk transfers depriving the CPU of memory bandwidth.
The raw driver is deprecated and we'd like to remove it. The preferred way of doing direct-IO against a blockdev is by opening it with O_DIRECT.
Your patches don't address blockdevs opened with O_DIRECT. What you should do is to make def_blk_aops.direct_IO point at a new function. That will then work correctly with both raw and with open(/dev/hdX, O_DIRECT).
But before doing anything else, please bench this on real hardware, see if it is worth pursuing. And gather an oprofile trace of the existing code.
Andrew Morton wrote on Tuesday, March 08, 2005 10:28 PM
Quoted 2 lines
But before doing anything else, please bench this on real hardware, see if it is worth pursuing.
Let me answer the questions in reverse order. We started with running industry standard transaction processing database benchmark on 2.6 kernel, on real hardware (4P smp, 64 GB memory, 450 disks) running industry standard db application. What we measured is that with best tuning done to the system, 2.6 kernel has a huge performance regression relative to its predecessor 2.4 kernel (a kernel from RHEL3, 2.4.21 based).
Ever since we had that measurement, people kick my butt everyday and asking "after you telling us how great 2.6 kernel is, why is my workload running significantly slower on this shinny 2.6 kernel?". It hurts, It hurts like a sledge hammer nailed right in the middle of my head.
This is all real: real benchmark running on real hardware, with real result showing large performance regression. Nothing synthetic here.
And yes, it is all worth pursuing, the two patches on raw device recuperate 1/3 of the total benchmark performance regression.
The reason I posted the pseudo disk driver is for people to see the effect easier without shelling out a couple of million dollar to buy all that equipment.
Quoted 7 lines
Once you bolt this onto a real device driver the proportional difference will fall, due to addition of the constant factor.
Once you bolt all this onto a real disk controller all the numbers will get worse (but in a strictly proportional manner) due to the disk transfers depriving the CPU of memory bandwidth.
That's not how I would interpret the number. Kernel utilization went up for 2.6 kernel running the same db workload. One reason is I/O stack is taxing a little bit on each I/O call (or I should say less efficient), even with minuscule amount, given the shear amount of I/O rate, it will be amplified very quickly. One cpu cycle spend in the kernel means one less cpu cycle for the application. My mean point is with less efficient I/O stack, kernel is actually taking away valuable compute resources from application to crunch SQL transaction. And that leads to lower performance.
One can extrapolate it the other way: make kernel more efficient in processing these I/O requests, kernel utilization goes down, cycle saved will transfer to application to crunch more SQL transaction, and performance goes up. I hope everyone is following me here.
Quoted 2 lines
At 5 usecs per request I figure that's 3% CPU utilisation for 16k requests at 100 MB/sec.
Our smallest setup has 450 disks, and the workload will generate about 50,000 I/O per second. Larger setup will have more I/O rate.
Quoted 8 lines
What sort of CPU?
What speed CPU?
What size requests?
Reads or writes?
1.6 GHz Itanium2, 9M L3
I/O requests are mixture of 2KB and 16KB, occasionally some large size in burst. Both read/write, about 50/50 split on rw.
- Ken
Quoted 11 lines · Kenneth W
Andrew Morton wrote on Tuesday, March 08, 2005 10:28 PM
Quoted 2 lines
But before doing anything else, please bench this on real hardware, see if it is worth pursuing.
Let me answer the questions in reverse order. We started with running industry standard transaction processing database benchmark on 2.6 kernel, on real hardware (4P smp, 64 GB memory, 450 disks) running industry standard db application. What we measured is that with best tuning done to the system, 2.6 kernel has a huge performance regression relative to its predecessor 2.4 kernel (a kernel from RHEL3, 2.4.21 based).
That's news to me. I thought we were doing OK with big database stuff. Surely lots of people have been testing such things.
Quoted 2 lines
And yes, it is all worth pursuing, the two patches on raw device recuperate 1/3 of the total benchmark performance regression.
On a real disk driver? hm, I'm wrong then.
Did you generate a kernel profile?
Andrew Morton wrote on Wednesday, March 09, 2005 12:05 PM
Quoted 10 lines
Quoted 6 lines · Kenneth W
Let me answer the questions in reverse order. We started with running industry standard transaction processing database benchmark on 2.6 kernel, on real hardware (4P smp, 64 GB memory, 450 disks) running industry standard db application. What we measured is that with best tuning done to the system, 2.6 kernel has a huge performance regression relative to its predecessor 2.4 kernel (a kernel from RHEL3, 2.4.21 based).
That's news to me. I thought we were doing OK with big database stuff. Surely lots of people have been testing such things.
There are different level of "big" stuff. We used to work on 32-way numa box, but other show stopper issues popping up before we get to the I/O stack. The good thing came out of that work is the removal of global unplug lock.
Quoted 5 lines
Quoted 2 lines
And yes, it is all worth pursuing, the two patches on raw device recuperate 1/3 of the total benchmark performance regression.
On a real disk driver? hm, I'm wrong then.
Yes, on real disk driver (qlogic fiber channel) and with real 15K rpm disks.
Quoted 1 line
Did you generate a kernel profile?
Top 40 kernel hot functions, percentage is normalized to kernel utilization.
_spin_unlock_irqrestore 23.54%_spin_unlock_irq 19.27%__blockdev_direct_IO 3.57%follow_hugetlb_page 1.84%e1000_clean 1.38%kmem_cache_alloc 1.31%put_page 1.29%__generic_file_aio_read 1.18%e1000_intr 1.07%
schedule 1.01%dio_bio_complete 0.97%mempool_alloc 0.96%kmem_cache_free 0.90%__end_that_request_first 0.88%__copy_user 0.82%
kfree 0.77%generic_make_request 0.73%_spin_lock 0.73%kref_put 0.73%vfs_read 0.68%update_atime 0.68%scsi_dispatch_cmd 0.67%fget_light 0.66%put_io_context 0.60%_spin_lock_irqsave 0.58%scsi_finish_command 0.58%generic_file_aio_write_nolock 0.57%inode_times_differ 0.55%break_fault 0.53%__do_softirq 0.48%aio_read_evt 0.48%try_atomic_semop 0.44%sys_pread64 0.43%__bio_add_page 0.43%__mod_timer 0.42%bio_alloc 0.41%scsi_decide_disposition 0.40%e1000_clean_rx_irq 0.39%find_vma 0.38%dnotify_parent 0.38%
Profile with spin lock inlined, so that it is easier to see functions that has the lock contention, again top 40 hot functions:
scsi_request_fn 7.54%finish_task_switch 6.25%__blockdev_direct_IO 4.97%__make_request 3.87%scsi_end_request 3.54%dio_bio_end_io 2.70%follow_hugetlb_page 2.39%__wake_up 2.37%aio_complete 1.82%kmem_cache_alloc 1.68%__mod_timer 1.63%e1000_clean 1.57%__generic_file_aio_read 1.42%mempool_alloc 1.37%put_page 1.35%e1000_intr 1.31%
schedule 1.25%dio_bio_complete 1.20%scsi_device_unbusy 1.07%kmem_cache_free 1.06%__copy_user 1.04%scsi_dispatch_cmd 1.04%
__end_that_request_first1.04%generic_make_request 1.02%
kfree 0.94%__aio_get_req 0.93%sys_pread64 0.83%get_request 0.79%put_io_context 0.76%dnotify_parent 0.73%vfs_read 0.73%update_atime 0.73%finished_one_bio 0.63%generic_file_aio_write_nolock 0.63%scsi_put_command 0.62%break_fault 0.62%e1000_xmit_frame 0.62%aio_read_evt 0.59%scsi_io_completion 0.59%inode_times_differ 0.58%
Quoted 7 lines · Kenneth W
Quoted 2 lines
Did you generate a kernel profile?
Top 40 kernel hot functions, percentage is normalized to kernel utilization.
_spin_unlock_irqrestore23.54%
_spin_unlock_irq19.27%
Cripes.
Is that with CONFIG_PREEMPT? If so, and if you disable CONFIG_PREEMPT, this cost should be accounting the the spin_unlock() caller and we can see
who the culprit is. Perhaps dio->bio_lock.
Andrew Morton wrote on Wednesday, March 09, 2005 2:45 PM
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Did you generate a kernel profile?
Top 40 kernel hot functions, percentage is normalized to kernel utilization.
Quoted 8 lines
Quoted 2 lines
_spin_unlock_irqrestore23.54%
_spin_unlock_irq19.27%Cripes.
Is that with
CONFIG_PREEMPT? If so, and if you disableCONFIG_PREEMPT, this cost should be accounting the thespin_unlock()caller and we can see
who the culprit is. Perhapsdio->bio_lock.
CONFIG_PREEMPT is off.
Sorry for all the confusion, I probably shouldn't post the first profile to confuse people. See 2nd profile that I posted earlier (copied here again).
scsi_request_fn 7.54%finish_task_switch 6.25%__blockdev_direct_IO 4.97%__make_request 3.87%scsi_end_request 3.54%dio_bio_end_io 2.70%follow_hugetlb_page 2.39%__wake_up 2.37%aio_complete 1.82%kmem_cache_alloc 1.68%__mod_timer 1.63%e1000_clean 1.57%__generic_file_aio_read 1.42%mempool_alloc 1.37%put_page 1.35%e1000_intr 1.31%
schedule 1.25%dio_bio_complete 1.20%scsi_device_unbusy 1.07%kmem_cache_free 1.06%__copy_user 1.04%scsi_dispatch_cmd 1.04%
__end_that_request_first1.04%generic_make_request 1.02%
kfree 0.94%__aio_get_req 0.93%sys_pread64 0.83%get_request 0.79%put_io_context 0.76%dnotify_parent 0.73%vfs_read 0.73%update_atime 0.73%finished_one_bio 0.63%generic_file_aio_write_nolock 0.63%scsi_put_command 0.62%break_fault 0.62%e1000_xmit_frame 0.62%aio_read_evt 0.59%scsi_io_completion 0.59%inode_times_differ 0.58%
Quoted 21 lines · Kenneth W
Andrew Morton wrote on Wednesday, March 09, 2005 2:45 PM
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Did you generate a kernel profile?
Top 40 kernel hot functions, percentage is normalized to kernel utilization.
Quoted 8 lines
Quoted 2 lines
_spin_unlock_irqrestore23.54%
_spin_unlock_irq19.27%Cripes.
Is that with
CONFIG_PREEMPT? If so, and if you disableCONFIG_PREEMPT, this cost should be accounting the thespin_unlock()caller and we can see
who the culprit is. Perhapsdio->bio_lock.
CONFIG_PREEMPTis off.Sorry for all the confusion, I probably shouldn't post the first profile to confuse people. See 2nd profile that I posted earlier (copied here again).
Quoted 9 lines
scsi_request_fn7.54%
finish_task_switch6.25%
__blockdev_direct_IO4.97%
__make_request3.87%
scsi_end_request3.54%
dio_bio_end_io2.70%
follow_hugetlb_page2.39%
__wake_up2.37%
aio_complete1.82%
What are these percentages? Total CPU time? The direct-io stuff doesn't look too bad. It's surprising that tweaking the direct-io submission code makes much difference.
hm. __blockdev_direct_IO() doesn't actually do much. I assume your damn compiler went and inlined direct_io_worker() on us.
Andrew Morton wrote on Wednesday, March 09, 2005 5:34 PM
Quoted 3 lines
What are these percentages? Total CPU time? The direct-io stuff doesn't look too bad. It's surprising that tweaking the direct-io submission code makes much difference.
Percentage is relative to total kernel time. There are three DIO functions showed up in the profile:
__blockdev_direct_IO 4.97%dio_bio_end_io 2.70%dio_bio_complete 1.20%
Quoted 2 lines
hm.
__blockdev_direct_IO()doesn't actually do much. I assume your damn compiler went and inlineddirect_io_worker()on us.
We are using gcc version 3.4.3. I suppose I can finger point gcc ? :-P
- Ken
Quoted 4 lines · Kenneth W
This is all real: real benchmark running on real hardware, with real
result showing large performance regression. Nothing synthetic here.
Ken, could you *please* be more complete, more organized and more specific?
What does "1/3 of the total benchmark performance regression" mean? One third of 0.1% isn't very impressive. You haven't told us anything at all about the magnitude of this regression.
Where does the rest of the regression come from?
How much system time? User time? All that stuff.
Quoted 2 lines
And yes, it is all worth pursuing, the two patches on raw device recuperate 1/3 of the total benchmark performance regression.
The patch needs a fair bit of work, and if it still provides useful gains when it's complete I guess could make sense as some database special-case.
But the first thing to do is to work out where the cycles are going to.
Also, I'm rather peeved that we're hearing about this regression now rather than two years ago. And mystified as to why yours is the only group which has reported it.
Andrew Morton wrote Wednesday, March 09, 2005 6:26 PM
Quoted 3 lines
What does "1/3 of the total benchmark performance regression" mean? One third of 0.1% isn't very impressive. You haven't told us anything at all about the magnitude of this regression.
2.6.9 kernel is 6% slower compare to distributor's 2.4 kernel (RHEL3). Roughly 2% came from storage driver (I'm not allowed to say anything beyond that, there is a fix though).
2% came from DIO.
The rest of 2% is still unaccounted for. We don't know where.
Quoted 1 line
How much system time? User time? All that stuff.
20.5% in the kernel, 79.5% in user space.
Quoted 1 line
But the first thing to do is to work out where the cycles are going to.
You've seen the profile. That's where all the cycle went.
Quoted 3 lines
Also, I'm rather peeved that we're hearing about this regression now rather than two years ago. And mystified as to why yours is the only group which has reported it.
2.6.X kernel has never been faster than the 2.4 kernel (RHEL3). At one point of time, around 2.6.2, the gap is pretty close, at around 1%, but still slower. Around 2.6.5, we found global plug list is causing huge lock contention on 32-way numa box. That got fixed in 2.6.7. Then comes 2.6.8 which took a big dip at close to 20% regression. Then we fixed 17% regression in the scheduler (fixed with cache_decay_tick). 2.6.9 is the last one we measured and it is 6% slower. It's a constant moving target, a wild goose to chase.
I don't know why other people have not reported the problem, perhaps they haven't got a chance to run transaction processing db workload on 2.6 kernel. Perhaps they have not compared, perhaps they are working on the same problem. I just don't know.
- Ken
Quoted 12 lines · Kenneth W
Quoted 3 lines
Also, I'm rather peeved that we're hearing about this regression now rather than two years ago. And mystified as to why yours is the only group which has reported it.
2.6.X kernel has never been faster than the 2.4 kernel (RHEL3). At one point of time, around 2.6.2, the gap is pretty close, at around 1%, but still slower. Around 2.6.5, we found global plug list is causing huge lock contention on 32-way numa box. That got fixed in 2.6.7. Then comes 2.6.8 which took a big dip at close to 20% regression. Then we fixed 17% regression in the scheduler (fixed with
cache_decay_tick). 2.6.9 is the last one we measured and it is 6% slower. It's a constant moving target, a wild goose to chase.
Quoted 4 lines
I don't know why other people have not reported the problem, perhaps they haven't got a chance to run transaction processing db workload on 2.6 kernel. Perhaps they have not compared, perhaps they are working on the same problem. I just don't know.
Also the 2.6 kernel is Soo much better in the case where you have many threads (even if they are all completely idle) that that improvement may be masking the regression that Ken is reporting (I've seen a 50% performance hit on 2.4 with just a thousand or two threads compared to 2.6). let's face it, a typical linux box today starts up a LOT of stuff that will never get used, but will count as an idle thread.
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 4 lines · David Lang
(I've seen a 50%
performance hit on 2.4 with just a thousand or two threads compared to 2.6)
Was that 2.4 kernel a vendor kernel with the O(1) scheduler?
Re: Direct io on block device has performance regression on 2.6.xkernel
Quoted 7 lines · Andrew Morton
Quoted 4 lines · David Lang
(I've seen a 50%
performance hit on 2.4 with just a thousand or two threads compared to 2.6)Was that 2.4 kernel a vendor kernel with the O(1) scheduler?
no, a kernel.org kernel. the 2.6 kernel is so much faster for this workload that I switched for this app and never looked back. I found that if I had 5000 or so idle tasks 2.4 performcane would drop to about a quarter of 2.6 (with the CPU system time being the limiting factor)
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare
Quoted 9 lines · Kenneth W
Andrew Morton wrote Wednesday, March 09, 2005 6:26 PM
Quoted 3 lines
What does "1/3 of the total benchmark performance regression" mean? One third of 0.1% isn't very impressive. You haven't told us anything at all about the magnitude of this regression.
2.6.9 kernel is 6% slower compare to distributor's 2.4 kernel (RHEL3). Roughly 2% came from storage driver (I'm not allowed to say anything beyond that, there is a fix though).
The codepaths are indeed longer in 2.6.
Quoted 1 line
2% came from DIO.
hm, that's not a lot.
Once you redo that patch to use aops and to work with O_DIRECT, the paths will get a little deeper, but not much. We really should do this so that O_DIRECT works, and in case someone has gone and mmapped the blockdev.
Fine-grained alignment is probably too hard, and it should fall back to __blockdev_direct_IO().
Does it do the right thing with a request which is non-page-aligned, but 512-byte aligned?
readv and writev?
2% is pretty thin :(
Quoted 1 line
The rest of 2% is still unaccounted for. We don't know where.
General cache replacement, perhaps. 9MB is a big cache though.
Quoted 7 lines
...
Around 2.6.5, we found global plug list is causing huge lock contention on 32-way numa box. That got fixed in 2.6.7. Then comes 2.6.8 which took a big dip at close to 20% regression. Then we fixed 17% regression in the scheduler (fixed withcache_decay_tick). 2.6.9 is the last one we measured and it is 6% slower. It's a constant moving target, a wild goose to chase.
OK. Seems that the 2.4 O(1) scheduler got it right for that machine.
Quoted 3 lines
haven't got a chance to run transaction processing db workload on 2.6 kernel. Perhaps they have not compared, perhaps they are working on the same problem. I just don't know.
Maybe there are other factors which drown these little things out:
architecture improvements, choice of architecture, driver changes, etc.
Andrew Morton wrote on Wednesday, March 09, 2005 8:10 PM
Quoted 5 lines
Quoted 3 lines
2.6.9 kernel is 6% slower compare to distributor's 2.4 kernel (RHEL3). Roughly 2% came from storage driver (I'm not allowed to say anything beyond that, there is a fix though).
The codepaths are indeed longer in 2.6.
Thank you for acknowledging this.
Quoted 5 lines
Quoted 1 line
2% came from DIO.
hm, that's not a lot.
....
2% is pretty thin :(
This is the exact reason that I did not want to put these numbers out in the first place. Because most people usually underestimate the magnitude of these percentage point.
Now I have to give a speech on "performance optimization 101". Take a look at this page: http://www.suse.de/~aj/SPEC/CINT/d-permanent/index.html
This page tracks the development of gcc and measures the performance of gcc with SPECint2000. Study the last chart, take out your calculator and calculate how much performance gain gcc made over the course of 3.5 years of development. Also please factor in the kind of man power that went into the compiler development.
Until people understand the kind of scale to expect when evaluating a complex piece of software, then we can talk about database transaction processing benchmark. This benchmark goes one step further. It bench the entire software stack (kernel/library/application/compiler), it bench the entire hardware platform (cpu/memory/IO/chipset) and on the grand scale, it bench system integration: storage, network, interconnect, mid- tier app server, front end clients, etc etc. Any specific function/ component only represent a small portion of the entire system, essential but small. For example, the hottest function in the kernel is 7.5%, out of 20% kernel time. If we throw away that function entirely, there will be only 1.5% direct impact on total cpu cycles.
So what's the point? The point is when judging a number whether it is thin or thick, it has to be judged against the complexity of SUT. It has to be judged against a relevant scale for that particular workload. And the scale has to be laid out correctly that represents the weight of each component.
Losing 6% just from Linux kernel is a huge deal for this type of benchmark. People work for days to implement features which might give sub percentage gain. Making Software run faster is not easy, but making software run slower apparently is a fairly easy task.
Quoted 8 lines
Fine-grained alignment is probably too hard, and it should fall back to
__blockdev_direct_IO().Does it do the right thing with a request which is non-page-aligned, but 512-byte aligned?
readv and writev?
That's why direct_io_worker() is slower. It does everything and handles every possible usage scenarios out there. I hope making the function fatter is not in the plan.
- Ken
Quoted 7 lines · Kenneth W
Losing 6% just from Linux kernel is a huge deal for this type of benchmark.
People work for days to implement features which might give sub percentage gain. Making Software run faster is not easy, but making software run slower apparently is a fairly easy task.
heh
Quoted 7 lines
Quoted 7 lines
Fine-grained alignment is probably too hard, and it should fall back to
__blockdev_direct_IO().Does it do the right thing with a request which is non-page-aligned, but 512-byte aligned?
Quoted 6 lines
Quoted 2 lines
readv and writev?
That's why
direct_io_worker()is slower. It does everything and handles every possible usage scenarios out there. I hope making the function fatter is not in the plan.
We just cannot make a change like this if it does not support readv and writev well, and if it does not support down-to-512-byte size and alignment. It will break applications.
Andrew Morton wrote on Thursday, March 10, 2005 12:31 PM
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Fine-grained alignment is probably too hard, and it should fall back to
__blockdev_direct_IO().Does it do the right thing with a request which is non-page-aligned, but 512-byte aligned?
readv and writev?
Quoted 7 lines
Quoted 3 lines
That's why
direct_io_worker()is slower. It does everything and handles every possible usage scenarios out there. I hope making the function fatter is not in the plan.We just cannot make a change like this if it does not support readv and writev well, and if it does not support down-to-512-byte size and alignment. It will break applications.
I must misread your mail. Yes it does support 512-byte size and alignment. Let me work on the readv/writev support (unless someone beat me to it).
- Ken
Quoted 2 lines · Kenneth W
Let me work on the readv/writev support (unless someone beat me to it).
Please also move it to the address_space_operations level. Yes, there are performance benefits from simply omitting the LFS checks, the mmap consistency fixes, etc. But they're there for a reason.
Quoted 9 lines · Kenneth W
Andrew Morton wrote Wednesday, March 09, 2005 6:26 PM
Quoted 3 lines
What does "1/3 of the total benchmark performance regression" mean? One third of 0.1% isn't very impressive. You haven't told us anything at all about the magnitude of this regression.
2.6.9 kernel is 6% slower compare to distributor's 2.4 kernel (RHEL3). Roughly 2% came from storage driver (I'm not allowed to say anything beyond that, there is a fix though).
Ok now, that statement piqued my interest -- since looking through a previous email it seems you are using the qla2xxx driver. Care to elaborate?
Regards,
Andrew Vasquez