udp weirdness

I'm observing some UDP weirdness, or I'd better say some UDP behaviour that I can't explain.

Two machines: one sending a UDP flow (using sendto) and another receiving this UDP flow (using bind + recv).

When the dgram length is lower that 357 Bytes I observe strange results at the send side. My home-made udp_tx program gives the following:

$./udp_tx -h 192.168.4.1 -m 357
357 1312621 357.518

357 is the dgram length (in B), 1312621 the number of dgrams sent and 357.518 the perceived thruput (in Mbits/s). The weirdness is that I get 357.518 Mbits/s whereas the underlying network is 10Mbits/s!

At the receive side the results are consistent (obviously):

$./udp_rx -m 357
357 29519 8.00884

<netstat -s --udp> on the send machine before and after the run also gives me such a large amount of sent packets (~1312700), whereas </sbin/ifconfig> confirms that about 29519 packets have been sent out.

Below 357 Bytes, the same kind of results are observed. Above 357 Bytes, the results make more sense to me:

$./udp_tx -h 192.168.4.1 -m 358
358 29505 8.04393

$./udp_rx -m 358
358 29468 8.0179

Does anybody know where I lose packets? And why do I lose them only when the dgram length is below 357 Bytes?

BTW, I'm running 2.4.18-vanilla w/ the 3c59x driver.

I figured out that packets can be dropped in pfifo_fast_enqueue() [the default qdisc's enqueue func], even though the driver/kernel flow control has triggered.

And sendto does not notify the user when packet gets dropped because the output queue overflows (as indicated in sendto manpage).

Why doesn't the kernel just put the process into sleep instead of dropping packets?

Quoted 3 lines · Eric Lemoine

I figured out that packets can be dropped in pfifo_fast_enqueue() [the default qdisc's enqueue func], even though the driver/kernel flow control has triggered.

What trigger do you suggest to wake up the process again? A better idea maybe to return something to the socket so it can manage things instead -- not sure what to return though that wouldnt break some standard;

cheers,
jamal

Quoted 5 lines · jamal

Quoted 3 lines · Eric Lemoine

I figured out that packets can be dropped in pfifo_fast_enqueue() [the default qdisc's enqueue func], even though the driver/kernel flow control has triggered.

...

Quoted 4 lines

What trigger do you suggest to wake up the process again? A better idea maybe to return something to the socket so it can manage things instead -- not sure what to return though that wouldnt break some standard;

"man sendto" error return codes:

ENOBUFS
The output queue for a network interface was full. This generally indicates that the interface has stopped sending, but may be caused by transient congestion. (This cannot occur in Linux, packets are just silently dropped when a device queue over� flows.)

Quoted 2 lines

cheers,
jamal

/Matti Aarnio

Ok, understood. Actually we already seem to have enobufs being returned.

Eric,
Does the attached patch fix it? Not tested or even compiled. Someone going to change the manpages?

net/ipv4/ip_output.c +10 -4

@@ -603,8 +603,11 @@
 		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, 
 			      skb->dst->dev, output_maybe_reroute);
 		if (err) {
-			if (err > 0)
-				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+			if (err > 0) {
+				err = net_xmit_errno(err);
+				if (err && sk->protinfo.af_inet.recverr)
+				       sk->protinfo.af_inet.recverr = err;
+			}
 			if (err)
 				goto error;
 		}

@@ -713,8 +716,11 @@
 
 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
 		      output_maybe_reroute);
-	if (err > 0)
-		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+	if (err > 0) {
+		err = net_xmit_errno(err);
+		if (err && sk->protinfo.af_inet.recverr)
+		       	 sk->protinfo.af_inet.recverr = err;
+	}
 	if (err)
 		goto error;
 out:

Quoted 5 lines

Ok, understood. Actually we already seem to have enobufs being returned.

Eric,
Does the attached patch fix it? Not tested or even compiled. Someone going to change the manpages?

protinfo.af_inet.recverr is a flag (1-bit variable) to enable the "extended reliable error message passing"; I dont see any reason for modifying it here. Setting this flag is the user's responsability, isn't it?

Quoted 5 lines

Ok, understood. Actually we already seem to have enobufs being returned.

Eric,
Does the attached patch fix it? Not tested or even compiled. Someone going to change the manpages?

protinfo.af_inet.recverr is a flag (1-bit variable) to enable the "extended reliable error message passing"; I dont see any reason for modifying it here. Setting this flag is the user's responsability, isn't it?

was to use sk->protinfo.af_inet.recverr only in the case of all is clean from the device layer.

Eric,

you are right about sk->protinfo.af_inet.recverr ..
I checked what other OSes do and i am convinced that the patch below at least makes us behave like the BSDs.
Alexey/Dave, sk->protinfo.af_inet.recverr is not needed for enobufs to be propagated back to the socket level; please review and probably apply:

ip_output.c +10 -4

@@ -603,8 +603,11 @@
 		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
 			      skb->dst->dev, output_maybe_reroute);
 		if (err) {
-			if (err > 0)
-				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+			if (err > 0) {
+				err = net_xmit_errno(err);
+				if (!err && sk->protinfo.af_inet.recverr)
+				       err = sk->protinfo.af_inet.recverr;
+			}
 			if (err)
 				goto error;
 		}

@@ -713,8 +716,11 @@
 
 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
 		      output_maybe_reroute);
-	if (err > 0)
-		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+	if (err > 0) {
+		err = net_xmit_errno(err);
+		if (!err && sk->protinfo.af_inet.recverr)
+		       	 err = sk->protinfo.af_inet.recverr;
+	}
 	if (err)
 		goto error;
 out:

Jamal,

Have you posted the right patch?

I see that sk->protinfo.af_inet.recverr is
still around. Here follows the patch that I think is the correct one. Please confirm. Thx.

be propagated back to the socket level; please review and probably apply:

Hello!

Quoted 2 lines

be propagated back to the socket level; please review and probably apply:

That was tried and failed miserably ages ago. Applications become insane when getting this error from logging tons of useless messages to abort()ing and cpu hogging. From the other hand silent reaction to loss is almost exactly desired behaviour in presence of congestion in the most of cases.

I have no idea, why that OSes work. Probably, they are so great that never lose packets. We do.

Probably, the world has changed since that time, but I have no desire to repeat the experiment to be honest.

Alexey

That was tried and failed miserably ages ago. Applications become insane when getting this error from logging tons of useless messages to abort()ing and cpu hogging. From the other hand silent reaction to loss is almost exactly desired behaviour in presence of congestion in the most of cases.

I have no idea, why that OSes work. Probably, they are so great that never lose packets. We do.

If I'm looping on a sendto() with a blocking socket, I would expect the syscall to block until the packet has been handed off to the device driver. This may mean blocking until local congestion backs off.

The way we do it now means that we can chew up massive amounts of cpu creating packets in userspace and throwing them away in the kernel, with no way of knowing from userspace that it is happening.

Feel free to implement. :-)

What? If your applications is enough clever to handle ENOBUFS right, set IP_RECVERR and live in peace.

Quoted 1 line

Feel free to implement. :-)

I may have to poke around...if nothing else I'll learn more about the networking code...

dropped when there is congestion. If IP_RECVERR is turned on, would sendto() then return -1 so I know to try and read the error messages?

I'm assuming I get ENOBUFS back in the ee_code field? Or can I get away with reading errno and ignoring the error queue?

there doesn't seem to be a lot of
documentation about it.

But not all the errors are queued,
only those which are supported asynchronously. ENOBUFS is still not. F.e. when packet is dropped lately (some qdiscs do this, dropping already queued packets to give place for another ones) ENOBUFS is not sent back.

Quoted 3 lines

there doesn't seem to be a lot of
documentation about it.

Damn! (I'm sorry) It is documented _very_ well, thanks to Andi. I really hate this mode when people do not worrying even to look to manpages before pronouncing such statements.

man recvmsg
man ip

Hmm...so even with IP_RECVERR I may not be notified if the packet is dropped?

either a) got out onto the wire, or b) my app got an error message back explaining why the message didn't get onto the wire so that it can decide how to proceed.

Quoted 2 lines

Hmm...so even with IP_RECVERR I may not be notified if the packet is dropped?

Do you not jest occasionally? :-) "Even" sounds interesting, because this kind of errors cannot be reported _principially_ without IP_RECVERR, providing asynchronous error reporting.

Packet can be dropped, damaged _after_ they are successfully queued to the device. And this need not to be emphasized in documentation on IP_RECVERR, except for a section "FUTURE DEVELOPMENT".

Hello!

Quoted 3 lines

Quoted 1 line

Feel free to implement. :-)

I may have to poke around...if nothing else I'll learn more about the networking code...

It is difficult task, if possible at all.

The main obstacle is that we must not block after select() succeeded, otherwise applications will lockup. Taking into account nature of datagram services (and generally of networking services, where routes change et al.) you do not know at time of select(), where the datagram will go. So, blocking can be made only based on a criterium not depending on this.

We use sndbuf (like all the OSes). Actually, the problem with silent losses is solved by tuning SO_SNDBUF. Though it is not a complete solution (failing whith lots of senders), it solves all those bullshit problems with silent losses. People just do not care about this, so they get the thing which they deserve.

Quoted 4 lines

dropped when there is congestion. If IP_RECVERR is turned on, would sendto() then return -1 so I know to try and read the error messages?

Yes.

Quoted 1 line

I'm assuming I get ENOBUFS back in the ee_code field? Or can I get away with reading errno and ignoring the error queue?

No, error queue should be read. But not all the errors are queued, only those which are supported asynchronously. ENOBUFS is still not. F.e. when packet is dropped lately (some qdiscs do this, dropping already queued packets to give place for another ones) ENOBUFS is not sent back.

Quoted 2 lines

there doesn't seem to be a lot of
documentation about it.

Damn! (I'm sorry) It is documented _very_ well, thanks to Andi. I really hate this mode when people do not worrying even to look to manpages before pronouncing such statements.

man recvmsg
man ip

Alexey

I cannot see any logical reason that we cannot ensure that a packet is not at least given to the driver when accepted by sendto. As soon as I get my pktgen and send-to-self code cleaned up, I am planning to start working on making UDP reliably send packets, or return an error to the calling code. I will, of course, keep you informed if I actually get something working...

get my pktgen and send-to-self code cleaned up, I am planning to start working on making UDP reliably send packets, or return an error to the calling code. I will, of course, keep you informed if I actually get something working...

Quoted 5 lines · Ben Greear

get my pktgen and send-to-self code cleaned up, I am planning to start working on making UDP reliably send packets, or return an error to the calling code. I will, of course, keep you informed if I actually get something working...

If you want realibility then thats what TCP is for.

I am curious why you would even want to retransmit a voice packet or why a local drop should be treated any different from a remote/network drop in a voice application ...

When you fail in sendto/sendmsg, errno is set to ENOBUFS as long as you set IP_RECVERR in the socket options; you can also receive ICMP errors as described in the manpages (use a msg_control buffer and call recvmsg with MSG_ERRQUEUE).

BTW, a good sample of an app that makes good use of ENOBUFS to do congestion control, IP_RECVERR and MSG_ERRQUEUE is the ping app in Alexeys iputils package. Why did i not remember all this before chasing the phantom with Eric is an indication i need to increase my cafeine consumption

to be silently dropped by the kernel because userspace is sending faster than they can get onto the wire during that tight loop.

Okay, so with IP_RECVERR set the case that Eric saw will not happen? I mean that sendto() will return with -1 and errno set to ENOBUFS?

Quoted 3 lines · Chris Friesen

to be silently dropped by the kernel because userspace is sending faster than they can get onto the wire during that tight loop.

So what happens when you find packets being dropped?
AFAIK, a dropped voice packet is as good as dead whether local or remote.

Quoted 4 lines

Okay, so with IP_RECVERR set the case that Eric saw will not happen? I mean that sendto() will return with -1 and errno set to ENOBUFS?

yes

cheers,
jamal

Quoted 2 lines · Chris Friesen

to be silently dropped by the kernel because userspace is sending faster than they can get onto the wire during that tight loop.

So what happens when you find packets being dropped? AFAIK, a dropped voice packet is as good as dead whether local or remote.

Quoted 4 lines · Ben Greear

get my pktgen and send-to-self code cleaned up, I am planning to start working on making UDP reliably send packets, or return an error to the calling code. I will, of course, keep you informed if I actually get something working...

If you want realibility then thats what TCP is for.
I am curious why you would even want to retransmit a voice packet or why a local drop should be treated any different from a remote/network drop in a voice application ...

When you fail in sendto/sendmsg, errno is set to ENOBUFS as long as you set IP_RECVERR in the socket options; you can also receive ICMP errors as described in the manpages (use a msg_control buffer and call recvmsg with MSG_ERRQUEUE).

BTW, a good sample of an app that makes good use of ENOBUFS to do congestion control, IP_RECVERR and MSG_ERRQUEUE is the ping app in Alexeys iputils package. Why did i not remember all this before chasing the phantom with Eric is an indication i need to increase my cafeine consumption.

cheers,
jamal

Quoted 2 lines

I may have to poke around...if nothing else I'll learn more about the networking code...

It is difficult task, if possible at all.

The main obstacle is that we must not block after select() succeeded, otherwise applications will lockup. Taking into account nature of datagram services (and generally of networking services, where routes change et al.) you do not know at time of select(), where the datagram will go. So, blocking can be made only based on a criterium not depending on this. problems with silent losses. People just do not care about this, so they get the thing which they deserve.

Would you mind explaining a bit more why apps will lockup if we block after select() succeeded. Or anyone?

Quoted 11 lines

Quoted 1 line

I may have to poke around...if nothing else I'll learn more about the networking code...

It is difficult task, if possible at all.

The main obstacle is that we must not block after select() succeeded, otherwise applications will lockup. Taking into account nature of datagram services (and generally of networking services, where routes change et al.) you do not know at time of select(), where the datagram will go. So, blocking can be made only based on a criterium not depending on this. problems with silent losses. People just do not care about this, so they get the thing which they deserve.

Alexey,

Would you mind explaining a bit more why apps will lockup if we block after select() succeeded. Or anyone?

Thx.

Eric.

Quoted 13 lines

Quoted 10 lines

It is difficult task, if possible at all.

The main obstacle is that we must not block after select() succeeded, otherwise applications will lockup. Taking into account nature of datagram services (and generally of networking services, where routes change et al.)
you do not know at time of select(), where the datagram will go. So, blocking can be made only based on a criterium not depending on this. problems with silent losses. People just do not care about this, so they get the thing which they deserve.

Alexey,

Quoted 2 lines

Would you mind explaining a bit more why apps will lockup if we block after select() succeeded. Or anyone?

Actually, I'm more interested to know why we would **need** to block after select has succeeded. It would seem to me that select is busted in this case. For the case of a very large UDP packet and a small send buffer, select gets confused, but at least when the send buffer is > 128k, it should be right...

Quoted 2 lines

Quoted 2 lines

Would you mind explaining a bit more why apps will lockup if we block after select() succeeded. Or anyone?

Quoted 7 lines · Eric Lemoine

I figured out that packets can be dropped in pfifo_fast_enqueue() [the default qdisc's enqueue func], even though the driver/kernel flow control has triggered.

And sendto does not notify the user when packet gets dropped because the output queue overflows (as indicated in sendto manpage).

Quoted 3 lines

Why doesn't the kernel just put the process into sleep instead of dropping packets?

What trigger do you suggest to wake up the process again? A better idea maybe to return something to the socket so it can manage things instead -- not sure what to return though that wouldnt break some standard;

Quoted 4 lines

I figured out that packets can be dropped in pfifo_fast_enqueue() [the default qdisc's enqueue func], even though the driver/kernel flow control has triggered.

Quoted 6 lines

And sendto does not notify the user when packet gets dropped because the output queue overflows (as indicated in sendto manpage).

Why doesn't the kernel just put the process into sleep instead of dropping packets?

What trigger do you suggest to wake up the process again?

A better idea maybe to return something to the socket so it can manage things instead -- not sure what to return though that wouldnt break some standard;

cheers,
jamal