Skip to contentSkip to conversation

Ethernet over Kernel Sockets

DISCUSSION Lukas Razik 2 participants 3 messages Feb 29, 2008 – Mar 3, 2008

lore
Share & tools
lore
#0 lore raw

Hello all!

As you know, some network cards doesn't have a 'eth' interface under linux.
Because of that I'm developing a net_device based driver which doesn't transmit and receive directly through a real network card but through UDP kernel sockets.
That means:
If my net_device->hard_start_xmit function gets an packet to transmit (in interrupt context), there will be a work struct queued into a workqueue and the packet will be processed by the workqueue later on (in process context). On the receiver side I have a kernel thread which blocks on sock_recvmsg() and if there comes a UDP message, then it will be processed and a sk_buff will be passed to the kernel.
The current state is that all works stable but I've bad transmission rates and bad ping times.

For example I've to systems and each has one Gigabit-Ethernet card:
System 1: ifconfig eth0 192.168.0.1
System 2: ifconfig eth0 192.168.0.2

If I load my driver then I get an additional eth interface (for example):
System 1: ifconfig eth1 192.168.1.1
System 2: ifconfig eth1 192.168.1.2

So the eth1 interfaces are based on sockets which use the eth0 (Gigabit) interfaces for the communication.
Now, if I measure the transmission between 192.168.0.1 and 192.168.0.2 then I get transmission rates that are normal for Gigabit-Ethernet (~25µsec (PingPong) and ~900Mbit/s). If I measure the transmission through the eth1 interfaces between 192.168.1.1 and 192.168.1.2 then I only get ~1msec (PingPong) and only about ~400Mbit/s.

The interesting thing is, that I always get PingPongs of 1ms (= one jiffie). Although if I measure with 100MBit- or other non-Ethernet cards. Maybe it's because I run the socket_recvmsg() function in a kernel thread which blocks on the function and must wait to be executed by the scheduler.

Now I don't know how I could solve this problem:
How can I force the kernel to process a received message immediately when it comes in through an UDP kernel socket?

I hope that someone of you can help me....
The source code you can find here:
http://net.razik.de/ethos.tar.gz

Regards and Many Thanks for any help!
Lukas

#1 lore raw
Quoted 7 lines · Lukas Razik

Hello all!

As you know, some network cards doesn't have a 'eth' interface under linux.
Because of that I'm developing a net_device based driver which doesn't transmit and receive directly through a real network card but through UDP kernel sockets.

What is it good for?

Quoted 10 lines

So the eth1 interfaces are based on sockets which use the eth0 (Gigabit) interfaces for the communication. Now, if I measure the transmission between 192.168.0.1 and 192.168.0.2 then I get transmission rates that are normal for Gigabit-Ethernet (~25?sec (PingPong) and ~900Mbit/s).
If I measure the transmission through the eth1 interfaces between 192.168.1.1 and 192.168.1.2 then I only get ~1msec (PingPong) and only about ~400Mbit/s.

Quoted 3 lines

The interesting thing is, that I always get PingPongs of 1ms (= one jiffie). Although if I measure with 100MBit- or other non-Ethernet cards.

Try increasing HZ :-)... or maybe it is a gigabit interrupt mitigation issue?

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#2 lore raw

Hello Pavel!

Thanks for your answer!

Quoted 4 lines
Quoted 2 lines

As you know, some network cards doesn't have a 'eth' interface under linux.
Because of that I'm developing a net_device based driver which doesn't transmit and receive directly through a real network card but through UDP kernel sockets.

What is it good for?

It's good for network cards which have a socket implementation but no 'eth' interface which is needed by some applications...

Quoted 4 lines
Quoted 1 line

The interesting thing is, that I always get PingPongs of 1ms (= one jiffie). Although if I measure with 100MBit- or other non-Ethernet cards.

Try increasing HZ :-)... or maybe it is a gigabit interrupt mitigation issue?

Yes, that would have been a nice solution for an OS-freak (like me and sure like you) but there are also other people :-/ who want to run more than the really nice linux kernel on their machines... ;-)))

The performance brake was the kernel thread for receiving packets. My solution are socket callbacks.
The idea I've got from someone of the kernelnewbies mailing list and some other people. Now I've PingPong times of ~30µsec (with Ethernet over Sockets over Ethernet, which natively has ~23µsec).

For interested people:
http://mail.nl.linux.org/kernelnewbies/2008-03/msg00005.html

Regards,
Lukas