tun/tap network throughput problem
Hi,
i was running an expriment to measure network throughput of UML. physical machine A and B are connected with gigabit network. and UML runs on machine A. i'm sending bulk UDP packets from UML to machine B using ttcp network throughput benchmark program.
i'm using 2.4.26 for both UML and host. (skas + sysemu patched)
i modified UML kernel code to get high performance a little bit, and i got some gain with TCP and UDP throughput. but the problem arises when i send large UDP packets with high sending rate. when i send 16K-sized UDP packets with rate of 500Mbits/s, receiving rate at machine B was about the same (~500Mbits/s), i mean, no problem. but when i send 16K-sized UDP packets with higher rate of more than 700Mbits/s, the receiving rate drops to 150Mbits/s. (MTU = 1500)
i'm using tun/tap + bridge. so, i checked tap0 and eth0 device status, but no packet drop reported (using ifconfig). i also checked the number of packets received and sent. tap0 at host side received around 60007 packets, and eth0 at machine A sends 505536 packets during that experiment. eth0 at machine B receives 505533 packets. so, it looks like between tap0 and eth0, packets are being dropped somehow. (the reason of dropping rapidly from 700 to 150 is the entire UDP packet will be lost even if one of the fragmented packets is lost, i think)
i used tcpdump to check any trouble with UDP checksum because i saw some posting about bug of bad UDP checksum when a UDP packet is fragmented, but no checksum failure reported. (and it works well with the lower sending rate)
i found other postings discussing queues of tun/tap devices when packets are coming to UML kernel and between two UML machines. but i couldn't find posting about packets from UML to other machine, even though they are all related, it looks.
anybody has any idea where packets are being dropped and how i can solve it?
thank you,
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 7 lines · Young Koh
i found other postings discussing queues of tun/tap devices when packets are coming to UML kernel and between two UML machines. but i couldn't find posting about packets from UML to other machine, even though they are all related, it looks.
anybody has any idea where packets are being dropped and how i can solve it?
Remember to check the packet drop readings from inside the UML kernel as well as on all interfaces on the host side.
Since you are sending large packets, I would suggest making sure that all involved interfaces have large enough txqueues.
You can monitor queue buildup by:
watch -n1 tc -d -s qdisc show
There should be no places for silent drops in the packet handling things - go around and try to find where the packets are dropping (or being queued forever).
-- Naked
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Hi,
thank you for your reply.
Quoted 2 lines
Remember to check the packet drop readings from inside the UML kernel as well as on all interfaces on the host side.
i checked the packet drop reading from UML kernel shell, but it still shows no packet dropping.
Quoted 1 line
watch -n1 tc -d -s qdisc show
so, i ran tc in the host machine as you suggested, and it reported some packets are being dropped in eth0. (even though the stat from /sbin/ifconfig still didn't report any packet drops)
$watch -n1 tc -d -s qdisc show
qdisc pfifo_fast 0: dev eth0 [Unknown qdisc, optlen=20]
Sent 18300080724 bytes 13121666 pkts (dropped 1328688, overlimits 0)
qdisc pfifo_fast 0: dev tap0 [Unknown qdisc, optlen=20] Sent 7664312 bytes 17650 pkts (dropped 0, overlimits 0)
---
i saw around 50,000 packet droppings out of around 240,000 packets. then, i tried larger txqueues in eth0. as i configured with the larger txqueue, the less packet droppings i got.
so, when i finally i assigned 50,000 for txqueuelen in the eth0. there was no packet dropping. and the final rates are, sending rate = ~800Mbits, and receiving rate(in another machine) = ~650Mbits
it seems to me that UML was sending too fast for eth0 to handle, so there were packet droppings because of the rate difference between tap0 and eth0. hrmm.. is there any flow control between tap0 and eth0? (i guess not, from the result, though)
even in host Linux kernel, how is flow control managed between native Linux and eth0? i mean, what happens if (host) Linux kernel sends too many packets(or socket buffers) to eth0 device?
thank you very much,
-Young
Quoted 14 lines
Since you are sending large packets, I would suggest making sure that all involved interfaces have large enough txqueues.
You can monitor queue buildup by:
watch -n1 tc -d -s qdisc show
There should be no places for silent drops in the packet handling things - go around and try to find where the packets are dropping (or being queued forever).
-- Naked
Quoted 10 lines
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 4 lines · Young Koh
it seems to me that UML was sending too fast for eth0 to handle, so there were packet droppings because of the rate difference between tap0 and eth0. hrmm.. is there any flow control between tap0 and eth0? (i guess not, from the result, though)
Yes, that seems to be the correct assumption - eth0 is dropping the packets. However, there is no 'rate difference' between tap0 and eth0. Packets are queued from inside the UML kernel, to the internal eth0 interface there (there's a packet queue there). Then they pop up from tap0 on the host side and are synchronously queued to eth0 queue. From there on, they are queued onwards by eth0 hard_start_xmit, which gets triggered from the interrupts. So there are two queues at work here, the inside the UML kernel eth0 queue, and the host kernel eth0 queue - of which the queue inside the UML kernel is faster and apparently never drops packets.
And you are correct - there's nothing to limit the speed of packets queued from inside the UML kernel to the speed that host side eth0 might accept them.
Quoted 3 lines
even in host Linux kernel, how is flow control managed between native Linux and eth0? i mean, what happens if (host) Linux kernel sends too many packets(or socket buffers) to eth0 device?
All sent packets are added to eth0 queue. Eth0 queue is depleted by hard_start_xmits triggered by interrupts (or other things, depending on the driver). If too many packets are enqueued to eth0 queue, the packets are being dropped.
If Linux tc is involved in limiting the queue speed at eth0, it merely throttles the speed of packets received by the network card - all packets are still queued at the same speed to eth0 queue.
In general, there's nothing to limit how many packets get queued to the interface queue - if something sends a million packets, that is how many packets are queued to the interface. However, for TCP, there are two things at work. First of all is ofcourse the fact that TCP will automatically throttle itself to the network speed. This one works with UML as well. Second issue is that the kernel will throttle TCP queues if they start taking too much memory, even before any network device queues are filled. This will only work if the TCP connection originates from the host machine - eg. not with UML.
With UDP, the userland program has no way to throttle itself or know how fast the network can send UDP packets. In there, saying sendto for the UDP packets, might return ENOBUFS, but from the manpage of sendto():
ENOBUFS
The output queue for a network interface was full. This gener- ally indicates that the interface has stopped sending, but may
be caused by transient congestion. (Normally, this does not
occur in Linux. Packets are just silently dropped when a device queue overflows.)
So, UDP send performance measures with packet drop are not really meaningful, since it all depends on how much time the userland process gets time and if that is more than what the network device can dish out.
-- Naked
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Hi,
Quoted 3 lines
In general, there's nothing to limit how many packets get queued to the interface queue - if something sends a million packets, that is how many packets are queued to the interface. However, for TCP, there
interesting. then, the UDP sending rate should be solely determined by system performance regardless of what kind of the network interface used, cause the rate will be determined by how fast the application enqueues packets to the device queue, which is an internal data structure in the kernel, right? when i measured UDP sending rate in the host machine, it was about 650Mbits/s. that means the throughput of data transfer from an application to the kernel internal structure is only 650Mbits/s? but, when i measured UDP sending rate with loopback device (sending packets to 127.0.0.1), it was about 5,000Mbits/s, which is a lot faster. should they, i mean, the sending rates for eth0 and lo, be the same? (maybe lo uses special queuing mechanism?)
thank you very much!
-Young
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 5 lines · Young Koh
Quoted 4 lines
In general, there's nothing to limit how many packets get queued to the interface queue - if something sends a million packets, that is how many packets are queued to the interface. However, for TCP, there
Quoted 12 lines
interesting. then, the UDP sending rate should be solely determined by system performance regardless of what kind of the network interface used, cause the rate will be determined by how fast the application enqueues packets to the device queue, which is an internal data structure in the kernel, right? when i measured UDP sending rate in the host machine, it was about 650Mbits/s. that means the throughput of data transfer from an application to the kernel internal structure is only 650Mbits/s? but, when i measured UDP sending rate with loopback device (sending packets to 127.0.0.1), it was about 5,000Mbits/s, which is a lot faster. should they, i mean, the sending rates for eth0 and lo, be the same? (maybe lo uses special queuing mechanism?)
Almost, but not quite. When packets are queued to the lo interface, they are handled immediately, synchronously even. When packets are queued to the eth0 interface, they get queued and the hardware interrupts trigger their processing. This means that the userland process is constantly being interrupted by the hardware sending packets, where as in the lo case it can just fire away as it wants.
However, the speed difference seems too large to be simply caused by that - so I am wondering a bit about the test tool, if it might manage to do something special. Perhaps there is a way to see if the queue is full from an UDP socket somehow, perhaps select does not return write availability or something. Or perhaps it simply asks how much data is in buffers - but I thought that only worked for receiving and TCP. I would need to look into this a bit further.
-- Naked
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 8 lines · Nuutti Kotivuori
Quoted 7 lines · Young Koh
when i measured UDP sending rate in the host machine, it was about 650Mbits/s. that means the throughput of data transfer from an application to the kernel internal structure is only 650Mbits/s? but, when i measured UDP sending rate with loopback device (sending packets to 127.0.0.1), it was about 5,000Mbits/s, which is a lot faster. should they, i mean, the sending rates for eth0 and lo, be the same? (maybe lo uses special queuing mechanism?)
[...]
Quoted 7 lines
However, the speed difference seems too large to be simply caused by that - so I am wondering a bit about the test tool, if it might manage to do something special. Perhaps there is a way to see if the queue is full from an UDP socket somehow, perhaps select does not return write availability or something. Or perhaps it simply asks how much data is in buffers - but I thought that only worked for receiving and TCP. I would need to look into this a bit further.
I just made a trivial test program in ocaml:
,----[ ocamlopt -o udptest unix.cmxa udptest.ml ]
| let data = String.make 15000 ' ';;
| try
| let dst = Unix.inet_addr_of_string Sys.argv.(1) in
| let port = int_of_string Sys.argv.(2) in
| let addr = Unix.ADDR_INET(dst, port) in
| let sock = Unix.socket Unix.PF_INET Unix.SOCK_DGRAM 0 in
| let _ = Unix.connect sock addr in
| let total = ref 0 in
| let starttime = Unix.gettimeofday () in
| for i = 1 to 10000 do
| let ret = Unix.send sock data 0 (String.length data) [] in
| total := !total + ret
| done;
| let endtime = Unix.gettimeofday () in
| print_endline ("Time: " ^ (string_of_float (endtime -. starttime))
| ^ " Total: " ^ (string_of_int !total))
| with Unix.Unix_error(e, f, a) ->
| print_endline ("Error: " ^ (Unix.error_message e))`----
Then ran it against localhost and an external host (with packets being dropped at input, so we do not get connection refused back):
,----
| $ ./a.out 127.0.0.1 1024 | Time: 0.352205991745 Total: 150000000 | | $ ./a.out 172.20.0.1 1024 | Time: 0.478994131088 Total: 150000000
`----
So in the localhost case, the throughput is over 3 gigabits per second, and in the case of the remote host, it is over 2 gigabits per second. This seems more in line of what it should be. The network here is a mere 100Mbit network - so the dropped counters in eth0 (seen with tc) fly up during the test.
So, it is probable that your test software either manages to throttle itself in case of network congestion, or simply only counts packets that are not dropped, somehow.
-- Naked
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 12 lines · Nuutti Kotivuori
With UDP, the userland program has no way to throttle itself or know how fast the network can send UDP packets. In there, saying sendto for the UDP packets, might return ENOBUFS, but from the manpage of
sendto():ENOBUFS
The output queue for a network interface was full. This gener- ally indicates that the interface has stopped sending, but may
be caused by transient congestion. (Normally, this does not
occur in Linux. Packets are just silently dropped when a device queue overflows.)
Quoted 5 lines
So, UDP send performance measures with packet drop are not really meaningful, since it all depends on how much time the userland process gets time and if that is more than what the network device can dish out.
-- Naked
The logical thing to return for a temporary nonblocking congestion condition is -EAGAIN to indicate the packet was dropped, but retransmitting it should work. (Client programs are free to ignore this, and if broken clients didn't ignore it and it screwed up existing implementations, possibly an extra open flag would be necessary to trigger it.) Putting this in UML itself wouldn't help, but if the host kernel did this UML (and TCP in general) could throttle more sanely.
However, nobody's ever bothered to fix this because UDP is not a guaranteed protocol over the internet or hub-based ethernet. Now that we've got switched ethernet (synchronous in the case of gigabit), packets are virtually never dropped over a LAN and it's possible that some infrastructure to not drop packets without remarking on it in the kernel's own packet handling layers is now worth the bother...
But I'm unlikely to personally code it up any time soon. :)
Rob
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 10 lines · Young Koh
Hi,
i was running an expriment to measure network throughput of UML. physical machine A and B are connected with gigabit network. and UML runs on machine A. i'm sending bulk UDP packets from UML to machine B using ttcp network throughput benchmark program.
i'm using 2.4.26 for both UML and host. (skas + sysemu patched)i modified UML kernel code to get high performance a little bit, and i got some gain with TCP and UDP throughput.
Hmm, can you share your changes with us, please? We'd be very grateful...
Signature · 16 lines
Paolo Giarrusso, aka Blaisorblade Linux registered user n. 292729 http://www.user-mode-linux.org/~blaisorblade ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
sure, allow me to check if everything looks fine (at least) to me...
Quoted 9 lines · Blaisorblade
Quoted 8 lines · Young Koh
Hi,
i was running an expriment to measure network throughput of UML. physical machine A and B are connected with gigabit network. and UML runs on machine A. i'm sending bulk UDP packets from UML to machine B using ttcp network throughput benchmark program.
i'm using 2.4.26 for both UML and host. (skas + sysemu patched)
Quoted 9 lines
Quoted 2 lines
i modified UML kernel code to get high performance a little bit, and i got some gain with TCP and UDP throughput.
Hmm, can you share your changes with us, please? We'd be very grateful... --
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel