is CONFIG_PACKET_MMAP always a win?
Newbie gig-ethernet question here.
I'm helping somebody implement a program that needs to process raw packets as close to wire rate as is possible. We're using kernel 2.4.16 or so.
What's the best way to retrieve raw packets from the kernel?
a) use libpcap
Overhead: a little bit worse than the best of any of the other options?
b) use af_packet, and call recvfrom or recvmsg myself for each packet
Overhead: one full memcpy of the packet body and one system call per packet.
c) enable CONFIG_PACKET_MMAP, use PACKET_RX_RING, and read packets from an mmap'd ring buffer
Overhead: kernel does a full memcpy of the packet body to get it
into the ring buffer, and my program does another to get it out.
If I understand it right, b costs one memcpy and one recv, and c costs two memcpys. Which one wins?
I guess I should benchmark these alternatives myself, but before I do, does anyone know of a good place to look for this info? Maybe I'm reinventing the wheel here.
Thanks,
Dan
Quoted 3 lines
c) enable
CONFIG_PACKET_MMAP, usePACKET_RX_RING, and read packets from an mmap'd ring buffer
Overhead: kernel does a full memcpy of the packet body to get it
into the ring buffer, and my program does another to get it out.
Why are you copying it out of the ring not processing it in place ?
From: Dan Kegel <[email protected]>
Date: Thu, 21 Feb 2002 17:51:20 -0800
What's the best way to retrieve raw packets from the kernel?
a) use libpcap
...
b) use af_packet
...
c) enable CONFIG_PACKET_MMAP, use PACKET_RX_RING
If I understand it right, b costs one memcpy and one recv, and c costs two memcpys. Which one wins?
"a" should be doing "c" when it is available in the kernel. If not, get a newer copy of the libpcap sources, preferably from Alexey's site:
ftp.inr.ac.ru:/ip-routing/
Quoted 11 lines · David S. Miller
From: Dan Kegel <[email protected]>
Date: Thu, 21 Feb 2002 17:51:20 -0800What's the best way to retrieve raw packets from the kernel?
a) use libpcap
...
b) useaf_packet
...
c) enableCONFIG_PACKET_MMAP, usePACKET_RX_RING
Quoted 8 lines
If I understand it right, b costs one memcpy and one recv, and c costs two memcpys. Which one wins?
"a" should be doing "c" when it is available in the kernel. If not, get a newer copy of the libpcap sources, preferably from Alexey's site:
ftp.inr.ac.ru:/ip-routing/
And if you can figure out how to do c, and feel like sharing, please do let me know! Documentation is a bit sparse..at least wherever I've been looking.
Enjoy,
Ben
Signature · 3 lines
Ben Greear <[email protected]> <Ben_Greear AT excite.com> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Quoted 5 lines · David S. Miller
...
If not, get a newer copy of the libpcap sources, preferably from Alexey's site:ftp.inr.ac.ru:/ip-routing/
The important files are a bit buried. The important ones seem to be
ftp://ftp.inr.ac.ru/ip-routing/lbl-tools/README ftp://ftp.inr.ac.ru/ip-routing/lbl-tools/libpcap-0.4-ss991029.dif.gz ftp://ftp.inr.ac.ru/ip-routing/lbl-tools/libpcap-0.4.tar.gz
The .dif file contains the first example I've seen of how to use socket option PACKET_RX_RING.
- Dan
Quoted 22 lines · Ben Greear
Quoted 18 lines · David S. Miller
From: Dan Kegel <[email protected]>
Date: Thu, 21 Feb 2002 17:51:20 -0800What's the best way to retrieve raw packets from the kernel?
a) use libpcap
...
b) useaf_packet
...
c) enableCONFIG_PACKET_MMAP, usePACKET_RX_RINGIf I understand it right, b costs one memcpy and one recv, and c costs two memcpys. Which one wins?
"a" should be doing "c" when it is available in the kernel. If not, get a newer copy of the libpcap sources, preferably from Alexey's site:
Quoted 6 lines
Quoted 1 line
ftp.inr.ac.ru:/ip-routing/
And if you can figure out how to do c, and feel like sharing, please do let me know! Documentation is a bit sparse..at least wherever I've been looking.
Yeah I found it a bit lacking too, I got there in the end though. Check
out: http://www.scaramanga.co.uk/code-fu/lincap.c
Signature · 2 lines
// Gianni Tedesco <[email protected]> 80% of all email is a figment of procmails imagination.
Quoted 8 lines · Dan Kegel
The important files are a bit buried. The important ones seem to be
ftp://ftp.inr.ac.ru/ip-routing/lbl-tools/README ftp://ftp.inr.ac.ru/ip-routing/lbl-tools/libpcap-0.4-ss991029.dif.gz ftp://ftp.inr.ac.ru/ip-routing/lbl-tools/libpcap-0.4.tar.gz
The .dif file contains the first example I've seen of how to use socket option
PACKET_RX_RING.
Too bad the changes did not get integrated -- libpcap 0.7.1 doesn't know
anything about PACKET_RX_RING...
Signature · 3 lines
+ Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: [email protected], PGP key available +
Quoted 3 lines · Dan Kegel
c) enable
CONFIG_PACKET_MMAP, usePACKET_RX_RING, and read packets from an mmap'd ring buffer
Overhead: kernel does a full memcpy of the packet body to get it
into the ring buffer, and my program does another to get it out.
I had a look at this about a year ago, and it seems there is no method provided to read the packets without copying them, if you need them in user space.
Probably the fastest way to process packets in user space is to use a special protocol handler of your own that mmaps the area where packets are already DMAd from the driver. I have been known to suggest simply mapping all 1GB of low kernel memory into user space for this :-) I haven't tried this, or writing the protocol handler, though.
cheers,
-- Jamie
Quoted 6 lines
Quoted 2 lines
Overhead: kernel does a full memcpy of the packet body to get it into the ring buffer, and my program does another to get it out.
I had a look at this about a year ago, and it seems there is no method provided to read the packets without copying them, if you need them in user space.
You can process them in the ring buffer. If you can't keep up then you are screwed any way you look at it 8)
Quoted 9 lines · Alan Cox
Quoted 6 lines
Quoted 2 lines
Overhead: kernel does a full memcpy of the packet body to get it into the ring buffer, and my program does another to get it out.
I had a look at this about a year ago, and it seems there is no method provided to read the packets without copying them, if you need them in user space.
You can process them in the ring buffer. If you can't keep up then you are screwed any way you look at it 8)
That still doesn't avoid copying: af_packet copies the whole packet (if you want the whole packet) from the original skbuff to the ring buffer.
-- Jamie
Quoted 5 lines
Quoted 2 lines
You can process them in the ring buffer. If you can't keep up then you are screwed any way you look at it 8)
That still doesn't avoid copying:
af_packetcopies the whole packet (if you want the whole packet) from the original skbuff to the ring buffer.
I'd make a handwaved claim that the first copy of the packet from a DMA receiving source is free. Its certainly pretty close to free because the overhead of sucking it into L1 cache will dominate and you need to do that anyway.
Zero copy is sometimes a false friend.
Quoted 11 lines · Alan Cox
Quoted 5 lines
Quoted 2 lines
You can process them in the ring buffer. If you can't keep up then you are screwed any way you look at it 8)
That still doesn't avoid copying:
af_packetcopies the whole packet (if you want the whole packet) from the original skbuff to the ring buffer.I'd make a handwaved claim that the first copy of the packet from a DMA receiving source is free. Its certainly pretty close to free because the overhead of sucking it into L1 cache will dominate and you need to do that anyway.
Doesn't DMA access system memory directly and leave processor caches alone? If so, then the fewer copies that have to pollute the L1/2 caches the better.
Even if it does for UP, I'd immagine that it doesn't for SMP...
Mike
Quoted 5 lines
Quoted 4 lines
receiving source is free. Its certainly pretty close to free because the overhead of sucking it into L1 cache will dominate and you need to do that anyway.
Doesn't DMA access system memory directly and leave processor caches alone?
It accesses system memory. That means the copy you have in cache is stale so you need to get rid of the copy in the cache - be that software or hardware.