changes to kiobuf support in 2.4.(?)4
I'm curious about the changes made to kiobuf support, apparently in order to improve raw I/O performance. I was reading through some old posts circa early April, where I saw reference to a rawio-bench program. Can someone explain what it does?
I have a driver that uses kiobufs to perform I/O. Prior to these changes, the kiobuf allocation and manipulation was very quick and efficient. It is now very slow.
The kiobuf part of the I/O request is as follows:
alloc_kiovec()
map_user_kiobuf()
... do I/O, using kiobuf to store mappings ... kiobuf_wait_for_io()
free_kiovec()
Now that the kiobuf is several KB in size, and 1024 buffer heads are allocated, the alloc_kiovec part goes from a percent or so of CPU usage to do 13000 requests per second to around 90% CPU usage to do 3000 per second.
It looks as though the raw driver allocates one kiobuf at open time (rather than on a per-request basis), but if two or more requests are issued to a single raw device, it too devolves into the allocate on every request strategy.
Before I go further, I'd appreciate if someone could confirm my hypotheses and also explain rawio-bench (and maybe point me to some source, if available).
thanks
jeremy
Quoted 22 lines · Jeremy Higdon
I'm curious about the changes made to kiobuf support, apparently in order to improve raw I/O performance. I was reading through some old posts circa early April, where I saw reference to a rawio-bench program. Can someone explain what it does?
I have a driver that uses kiobufs to perform I/O. Prior to these changes, the kiobuf allocation and manipulation was very quick and efficient. It is now very slow.
The kiobuf part of the I/O request is as follows:
alloc_kiovec()
map_user_kiobuf()
... do I/O, using kiobuf to store mappings ...kiobuf_wait_for_io()
free_kiovec()Now that the kiobuf is several KB in size, and 1024 buffer heads are allocated, the
alloc_kiovecpart goes from a percent or so of CPU usage to do 13000 requests per second to around 90% CPU usage to do 3000 per second.
Quoted 8 lines
It looks as though the raw driver allocates one kiobuf at open time (rather than on a per-request basis), but if two or more requests are issued to a single raw device, it too devolves into the allocate on every request strategy.
Before I go further, I'd appreciate if someone could confirm my hypotheses and also explain rawio-bench (and maybe point me to some source, if available).
That's all right, no one kiobuf should be allocated in any fast path, if you follow this rule there's no problem.
The kiobuf in 2.4 is mostly used for rawio which can allocate the kiobuf at open time. With the O_DIRECT patch I added one kiobuf per each filp (of course allocated only when you open the file with O_DIRECT), the rawio should simply switch to use the per-file kiobuf so you can open the raw device multiple times without overhead of allocating the kiobuf during simultaneous read/writes (if anybody wants to implement this improvement on top of O_DIRECT please let me know so we don't duplicate efforts).
The reason of the large allocation and to put the bh inside the kiobuf is that if we do a small allocation then we end with a zillion of allocations of the bh and freeing of the bh at every I/O!! (not even at every read/write syscall, much more frequently) The flood of allocation and freeing of the 512byte large bh at every I/O was just an obvious huge amount of
wasted cpu time and that's why allocating them statically in the kiobuf makes thing much faster (and also it avoids to run out of memory during I/O making the rawio reliable), but of course now it's proibitive to allocate any kiobuf in any fast path (even more since I switched to vmalloc/vfree to allocate the kiobuf since as said above in 2.4 nobody should ever allocate a kiobuf in any fast path anyways because of its size). If you want to make kiobuf a remotely light thing you should make the kiobuf become the I/O entity instead of the bh (which I'm not saying it's a good thing here). If you don't make the kiobuf the I/O entity understood at the very low level from the device drivers, then you'd better keep having the bh in the kiobuf.
OTOH I'm a little biased in the above reasoning since I use the kiobuf only for doing direct I/O (I always end calling brw_kiovec somehow, otherwise I wouldn't be using the kiobufs at all). If you are using the kiobufs for framebuffers and other drivers that never ends calling brw_kiovec I think you should be using the mmap callback and remap_page_range instead as most drivers correctly do to avoid the overhead of the kiobufs. But ok if you really want to use the kiobuf for non I/O things instead of remap_page_range (dunno why) we could split off the bh-array allocation from the kiobuf to make it a bit lighter so you could use it for non-IO operations without the overhead of the bhs, but still we should adapt rawio to preallocate the bh at open/close time (separately from the kiovec).
Andrea
Quoted 8 lines · Jeremy Higdon
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines
At 13000 IOPS, when allocating and freeing on every I/O request, the allocate/free overhead was approximately .6% on a 2 CPU system, where the total overhead was about 25%. So I would theoretically gain 3% (maybe a little better since there is locking involved) if I could avoid the alloc/free.
Quoted 8 lines
Quoted 3 lines
Ok good.
Andrea
So one more question for now:
Where do I get the
O_DIRECTpatch?
I will upload a new one today in my ftp area on kernel.org against 2.4.8pre3. (the current one has a silly bug spotted by Ken Preslan while testing O_DIRECT on GFS)
Quoted 1 line
Oh, and is there a plan to get it into 2.4.X?
The one I will upload today is stable enough to be ok for inclusion and it generates good numbers. Infact the O_DIRECT way on top of the fs will be the prefered way for storing DB files compared to rawio on lvm that people uses today (except when using the shared storage but even for it I just had feedback from the GFS folks that just verified GFS to work correctly with the O_DIRECT patch). The only detail left for some DB that wants to use 2k I/O granularity also on a 4k blocksized-fs is to relax the granularity of the I/O from the softblocksize to the hardblocksize but it will be tricky to do that without losing any performance (however I don't see that as a requirement to make use of O_DIRECT, infact other people is using it without seeing the softblocksize constraint as a problem) but we'll have to relax that constraint eventually to make everybody happy.
Andrea
Quoted 2 lines · Jeremy Higdon
By "mmap callback", you're referring to the mmap entry in the
file_operationsstructure?
yes.
Quoted 11 lines
I am doing direct I/O. I'm using the kiobuf to hold the page addresses of the user's data buffer, but I'm calling directly into my driver after doing the
map_user_kiobuf()(I have a read/write request, a file offset, a byte count, and a set of pages to DMA into/out of, and that gets directly translated into a SCSI command).It turns out that the old
kmem_cache_allocwas very lightweight, so I could get away with doing it once per I/O request, so I would indeed profit by going back to a light weight kiobuf, or at least an optional allocation of the bh and blocks arrays (perhaps turn them into pointers to arrays?).
I see your problem and it's a valid point indeed. But could you actually allocate the kiobuf in the file->f_iobuf pointer? I mean: could you allocate it at open/close too? That would be the way I prefer since you would need to allocate the bh anyways later (but with a flood of alloc/free). So if you could move the kiobufs allocation out of the fast path you would get a benefit too I believe.
Andrea
Quoted 1 line · Jeremy Higdon
The first is that I don't need the bh's or block's in my implementation.
So the very latest layer of the scsi driver understand the kiobufs natively for you right? (I assume you ported this functionality only to a few scsi drivers, right?)
Quoted 1 line
Everything I need is in the old-style kiobuf or is passed as an argument.
Ok.
Quoted 2 lines
The second is I don't see a
file->f_iobufpointer in my source tree, which is 2.4.8-pre3, I believe. In fact, the kiobuf pointer is stored in the
It's in the O_DIRECT patch.
Quoted 2 lines
raw_devicesarray in my version ofraw.c, and there is only one per raw device.
This is why I said also rawio should start using the f_iobuf to have one kiobuf per-file like with O_DIRECT, infact if you open("/dev/hda", O_DIRECT) instead of using the rawio API you will just get the kiobuf per file.
Quoted 3 lines
Assuming I'm out of date, and there is some way to store a kiobuf pointer into the file data structure, and I'll never see two requests outstanding at the same time to the same file, then I could do as you suggest. I'd
There's the possibility of two requests outstanding on the same file still if you share the same file with multiple filedescriptors but I don't think that's an interesting case to optimize, however I still need a test_and_set_bit and a slow path allocation of the kiovec to handle the multiple fd pointing to the same filp case correctly (but I think that's ok).
Quoted 2 lines
be wasting about 16KB per open file (assuming 512KB and 64 bit) and adding unneeded CPU overhead at open time, but I could live with that.
If you don't need the bh and blocks part of the kiobuf and we split it off (which I would be fine to change if the lowlevel driver would understand the kiobuf as an I/O entity, that again I'm not saying it's a good thing or not here) you should still be faster by avoiding allocating the kiobuf in the fast path and you won't have the 16KB overhead per open device once/if the kiobuf will shrink in size (so it should still better than allocating a smaller kiobuf in a fast path).
What do you think?
Andrea
Quoted 4 lines
The reason of the large allocation and to put the bh inside the kiobuf is that if we do a small allocation then we end with a zillion of allocations of the bh and freeing of the bh at every I/O!! (not even at every read/write syscall, much more frequently)
That is true for block device I/O only. Current bttv versions are using kiobufs to lock down user pages for DMA. But I don't need the bh's to transfer the video frames ...
Gerd
Signature · 1 line
Damn lot people confuse usability and eye-candy.
Quoted 4 lines · Jeremy Higdon
My understanding is that databases like to have multiple outstanding requests to the same file, which I believe falls into the the multiple file descriptors, one file case. So for us, it is interesting. Or do I misunderstand what you wrote?
the databases I know open the file without sharing the fd, so they're fine even if we share the same iobuf from multiple fd. Infact even with threads you can open the same file multiple times.
Quoted 9 lines
Actually, I want to be clear on this . . .
If I do dd if=/dev/raw1 . . . & dd if=/dev/raw1 . . . & waitwith the
O_DIRECTpatch, do I get some slow path allocations?
That will work fine with the per-file iobuf ala O_DIRECT, no slow path allocations will ever happen.
Even if you do:
clone()
parent:
fd = open("/dev/raw1")
write(fd)
read(fd)
child:
fd = open("/dev/raw1")
write(fd)
read(fd)you won't run into slow path allocations.
Quoted 5 lines
At 13000 IOPS, when allocating and freeing on every I/O request, the allocate/free overhead was approximately .6% on a 2 CPU system, where the total overhead was about 25%. So I would theoretically gain 3% (maybe a little better since there is locking involved) if I could avoid the alloc/free.
Ok good.
Andrea
Quoted 8 lines · Gerd Knorr
Quoted 4 lines
The reason of the large allocation and to put the bh inside the kiobuf is that if we do a small allocation then we end with a zillion of allocations of the bh and freeing of the bh at every I/O!! (not even at every read/write syscall, much more frequently)
That is true for block device I/O only. Current bttv versions are using kiobufs to lock down user pages for DMA. But I don't need the bh's to transfer the video frames ...
This is another problem. We miss a whole layer of fast streaming and chunking IO, which is not meant for block devices.
The most practical example would be the sg-driver, but there are other things which require this kind of semantic:
Transmit Buffer via DMA to device (and programming it for operations on that buffer) and possible receiving sth. back.
We need no reordering here, but would like to do the DMA directly from user space buffers.
Examples:
- GiMP plugin, which renders some complex algorithm on a DSP.
- Crypto (Co-)processors, which encrypt data streams
- Screengrabbers, which can grab a specific area.
- (intelligent, multi) data aquisistion devices (large sensor arrays)
...
I would like to have a thing like the zero copy IO in the network layer, but not done with NICs only, but with ANY device.
For the DSP case (which might be the most complex one) I did some research already, but I don't like to see this work duplicated over and over again.
I built transfer structs similar to BHs but more simply and have a queue of that per device and allocate transfers from a slab.
The blocking and unblocking is done in the driver anyway. I only try to do zero copy IO, if I have complete pages, which simplifies it a "little" ;-)
Without the coalescing and reordering it's pretty simple.
What do the gurus think here?
Regards
Ingo Oeser
Signature · 3 lines
You mean a kill file only kills email!!!! DAMN!!!! All these years
I thought I was doing the gene pool some good...
--- "Clint Wolff" <[email protected]>Quoted 8 lines · Gerd Knorr
Quoted 4 lines
The reason of the large allocation and to put the bh inside the kiobuf is that if we do a small allocation then we end with a zillion of allocations of the bh and freeing of the bh at every I/O!! (not even at every read/write syscall, much more frequently)
That is true for block device I/O only. Current bttv versions are using kiobufs to lock down user pages for DMA. But I don't need the bh's to transfer the video frames ...
I guess you use map_user_kiobuf to provide zerocopy to read/write too (not just to user-mapped ram allocated by bttv), right?
If you allocate the kiobuf not in any fast path the vmalloc and big allocation won't be a real issue even now, however I agree it's ok to split the bh/block array allocation out of the kiobuf to make it lighter (but still it won't be a light thing).
Andrea