Zero Copy IO
Quoted 8 lines · Alex Q Chen
I am trying to find a way to pin down user space memory from kernel, so that these user space buffer can be used for direct IO transfer or otherwise known as "zero copying IO". Searching through the Internet and reading comments on various news groups, it would appear that most developers including Linus himself doesn't believe in the benefit of "zero copying IO". Most of the discussion however was based
Quoted 8 lines
on network card drivers. For certain other drivers such as SCSI Tape driver, which need to handle great deal of data transfer, it would seemed still be more advantageous to enable zero copy IO than
copy_from_user()andcopy_to_user()all the data. Other OS such as AIX and OS2 have kernel functions that can be used to accomplish such a task. Has any ground work been done in Linux 2.4 to enable "zero copying IO"?
Alex,
The kiobufs mechanism in the 2.4 series is the appropriate tool for avoiding copy_from_user() and copy_to_user(). The definitive driver is in drivers/char/raw.c which does synchronous IO to block devices such as disks (but is probably not appropriate for tapes).
The SCSI generic (sg) driver supports direct IO. The driver in lk 2.4.3 has the direct IO code commented out while a version that I'm currently testing (sg 3.1.18 at www.torque.net/sg) has its direct IO code activated. I have a web page comparing throughput times and CPU utilizations at http://www.torque.net/sg/rbuf_tbl.html . My testing indicates that the kiobufs mechanism is now working quite well. For various reasons I still think that it is best to default to indirect IO and let speed hungry users enable dio (which is done in sg via procfs). Even when the user selects direct IO is should be possible to fall back to indirect IO. [Sg does this when a SCSI adapter can't support direct IO (e.g. an ISA adapter).]
Since the SCSI tape (st) driver is structurally similar to sg, it should be possible to add direct IO support to st.
One thing to note is that when you let the user provide the buffer for direct IO (e.g. with malloc) then on the i386 it won't be contiguous from a bus address POV. This means large scatter gather lists (typically with each element 4 KB on i386) which can be time consuming to load on some SCSI adapters. One way around this would be for a driver to provide "malloc/free" like ioctls.
Doug Gilbert
[QUESTIONS] Transision from pcmcia-cs to 2.4 built-in PCMCIA
I have a 3c595 CardBus card and a Lucent Orinoco card
under pcmcia-cs I used the following drivers:
pcmcia_core, ds, and (I think) cs for low level stuff, i82365 for the controller,cb_enabler and 3c595_cb for the 3c595,
and wvlan_cs for the Orinoco
under the kernel pcmcia support, I use:
pcmcia_core and ds for low level stuff,yenta_socket for the controller,
3c59x (same driver as PCI) for the 3c595,
and hermes and orinoco_cs for the Orinoco
Now after awhile I figured out what new drivers I should start using, but I have a few things about the change that still confuse me...
First, why have I stopped needing cs and cb_enabler?
Second, why is yenta_socket only compiled if I enable CardBus support in the kernel? I'm running an Orinoco card on another machine, and since I don't think it's CardBus (am I wrong?), I didn't enable CB in the kernel. The i82365 driver is the only one compiled, but it seems to work fine on that machine. Should I enable CardBus support and use yenta_socket instead?
Third, on the first machine with both cards, neither card works if I use i82365 instead of yenta_socket, why? The Orinoco gets Tx timeouts on every packet, and inserting the 3c595 causes the controller (socket) to time out waiting for reset and it doesn't recognize the 3c595.
Despite the confusion of changing systems, I must say that the orinoco driver works much better than wvlan_cs for me, as the two Orinoco cards in IBSS Ad-Hoc mode would get intermittent Tx timeouts with the wvlan_cs driver. It's also nice not to have to rebuild pcmcia-cs when I upgrade my kernel anymore. Keep up the good work!
Ryan
Re: [QUESTIONS] Transision from pcmcia-cs to 2.4 built-in PCMCIA
Quoted 1 line · [email protected] said:
First, why have I stopped needing cs and
cb_enabler?
cs is built into pcmcia_core.o, cb_enabler should still be there though. It's feasible that you only need cb_enabler for the old CardBus drivers, though - I'm not sure.
Quoted 6 lines
Second, why is
yenta_socketonly compiled if I enable CardBus support in the kernel? I'm running an Orinoco card on another machine, and since I don't think it's CardBus (am I wrong?), I didn't enable CB in the kernel. The i82365 driver is the only one compiled, but it seems to work fine on that machine. Should I enable CardBus support and useyenta_socketinstead?
yenta_socket is the driver for CardBus i82365-compatible sockets. i82365 no longer drives CardBus sockets, only PCMCIA.
Quoted 5 lines
Third, on the first machine with both cards, neither card works if I use i82365 instead of
yenta_socket, why? The Orinoco gets Tx timeouts on every packet, and inserting the 3c595 causes the controller (socket) to time out waiting for reset and it doesn't recognize the 3c595.
The PCMCIA card ought to work. It's probably screwed up the IRQ routing - it no longer knows about some of the differences between CardBus and PCMCIA bridges. What exactly is the bridge in this machine?
--
dwmw2
Quoted 44 lines · Douglas Gilbert
Quoted 8 lines · Alex Q Chen
I am trying to find a way to pin down user space memory from kernel, so that these user space buffer can be used for direct IO transfer or otherwise known as "zero copying IO". Searching through the Internet and reading comments on various news groups, it would appear that most developers including Linus himself doesn't believe in the benefit of "zero copying IO". Most of the discussion however was based
Quoted 8 lines
on network card drivers. For certain other drivers such as SCSI Tape driver, which need to handle great deal of data transfer, it would seemed still be more advantageous to enable zero copy IO than
copy_from_user()andcopy_to_user()all the data. Other OS such as AIX and OS2 have kernel functions that can be used to accomplish such a task. Has any ground work been done in Linux 2.4 to enable "zero copying IO"?Alex,
The kiobufs mechanism in the 2.4 series is the appropriate tool for avoidingcopy_from_user()andcopy_to_user(). The definitive driver is indrivers/char/raw.cwhich does synchronous IO to block devices such as disks (but is probably not appropriate for tapes).The SCSI generic (sg) driver supports direct IO. The driver in lk 2.4.3 has the direct IO code commented out while a version that I'm currently testing (sg 3.1.18 at www.torque.net/sg) has its direct IO code activated. I have a web page comparing throughput times and CPU utilizations at http://www.torque.net/sg/rbuf_tbl.html . My testing indicates that the kiobufs mechanism is now working quite well. For various reasons I still think that it is best to default to indirect IO and let speed hungry users enable dio (which is done in sg via procfs). Even when the user selects direct IO is should be possible to fall back to indirect IO. [Sg does this when a SCSI adapter can't support direct IO (e.g. an ISA adapter).]
Since the SCSI tape (st) driver is structurally similar to sg, it should be possible to add direct IO support to st.
Quoted 7 lines
One thing to note is that when you let the user provide the buffer for direct IO (e.g. with malloc) then on the i386 it won't be contiguous from a bus address POV. This means large scatter gather lists (typically with each element 4 KB on i386) which can be time consuming to load on some SCSI adapters. One way around this would be for a driver to provide "malloc/free" like ioctls.
I'm not very knowledgeable, but doesn't the sound driver use mmap() to do this? Either way, AGP GART is basically a paged-MMU allowing non-contiguous phys-mem to be made to look contiguous from the AGP *and* from PCI (on most chipsets?). perhaps this would be helpful.
Large contiguous phys-mem seems to be difficult currently, but would be nice as it would allow use of larger MMU pages with many cpus. Someone mentioned reverse page table support would be required first...
Quoted 7 lines
Doug Gilbert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/