Skip to contentSkip to conversation

udp: UDP socket send queue repair

PATCH In review Bui Quang Minh 5 participants 9 messages Aug 11, 2021 – Aug 17, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 udp: UDP socket send queue repair

In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.

A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.

While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.

---
 include/linux/udp.h      |  3 +-
 include/net/udp.h        |  2 +
 include/uapi/linux/udp.h |  1 +
 net/ipv4/udp.c           | 94 +++++++++++++++++++++++++++++++++++++++-
 net/ipv6/udp.c           | 56 +++++++++++++++++++++++-
 5 files changed, 151 insertions(+), 5 deletions(-)
include/linux/udp.h +2 -1
index ae66dadd8543..63df0753966e 100644
@@ -70,7 +70,8 @@ struct udp_sock {
 #define UDPLITE_SEND_CC  0x2  		/* set via udplite setsockopt         */
 #define UDPLITE_RECV_CC  0x4		/* set via udplite setsocktopt        */
 	__u8		 pcflag;        /* marks socket as UDP-Lite if > 0    */
-	__u8		 unused[3];
+	__u8		 repair;
+	__u8		 unused[2];
 	/*
 	 * For encapsulation sockets.
 	 */
include/net/udp.h +2 -0
index 360df454356c..4550e72b9f2a 100644
@@ -331,6 +331,8 @@ struct sock *udp6_lib_lookup_skb(const struct sk_buff *skb,
 				 __be16 sport, __be16 dport);
 int udp_read_sock(struct sock *sk, read_descriptor_t *desc,
 		  sk_read_actor_t recv_actor);
+int udp_peek_sndq(struct sock *sk, struct msghdr *msg,
+		  size_t len);
 
 /* UDP uses skb->dev_scratch to cache as much information as possible and avoid
  * possibly multiple cache miss on dequeue()
include/uapi/linux/udp.h +1 -0
index 4828794efcf8..255d056403da 100644
@@ -29,6 +29,7 @@ struct udphdr {
 
 /* UDP socket options */
 #define UDP_CORK	1	/* Never send partially complete segments */
+#define UDP_REPAIR	2	/* UDP sock is under repair right now */
 #define UDP_ENCAP	100	/* Set the socket to accept encapsulated packets */
 #define UDP_NO_CHECK6_TX 101	/* Disable sending checksum for UDP6X */
 #define UDP_NO_CHECK6_RX 102	/* Disable accpeting checksum for UDP6 */
net/ipv4/udp.c +92 -2

134 lines not on this page · raw message

net/ipv6/udp.c +54 -2

86 lines not on this page · raw message

Signature · 1 line
2.17.1
#1 lore raw
Quoted 9 lines · Bui Quang Minh

In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.

A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.

While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.

What is the use case for this feature, adding a test in UDP fast path ?

CORKed UDP is hardly used nowadays.

IMO, TCP_REPAIR hijacking standard system calls was a design error, we should have added new system calls.

#2 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 7 lines · Bui Quang Minh

In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.

A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.

Quoted 5 lines
Quoted 2 lines

While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.

What is the use case for this feature, adding a test in UDP fast path ?

This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.

Quoted 2 lines

IMO, TCP_REPAIR hijacking standard system calls was a design error, we should have added new system calls.

You are right that adding new system calls is a better approach. What do you think about adding a new option in getsockopt approach?

Thanks,
Quang Minh.

#3 lore raw
Quoted 13 lines · Bui Quang Minh
Quoted 10 lines · Eric Dumazet
Quoted 7 lines · Bui Quang Minh

In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.

A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.

What is the use case for this feature, adding a test in UDP fast path ?

This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.

UDP is not reliable.

I find a bit strange we add so many lines of code for a feature trying very hard to to drop _one_ packet.

I think a much better changelog would be welcomed.

Quoted 8 lines
Quoted 3 lines

IMO, TCP_REPAIR hijacking standard system calls was a design error, we should have added new system calls.

You are right that adding new system calls is a better approach. What do you think about adding a new option in getsockopt approach?

Thanks,
Quang Minh.

#4 lore raw
Quoted 22 lines · Eric Dumazet
Quoted 19 lines · Bui Quang Minh
Quoted 15 lines · Eric Dumazet
Quoted 9 lines · Bui Quang Minh

In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.

A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.

While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.

What is the use case for this feature, adding a test in UDP fast path ?

Quoted 8 lines
Quoted 1 line

This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.

UDP is not reliable.

I find a bit strange we add so many lines of code for a feature trying very hard to to drop _one_ packet.

I think a much better changelog would be welcomed.

The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via UDP_CORK socket or with MSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.

Thanks,
Quang Minh.

#5 lore raw

From: Bui Quang Minh

Quoted 1 line

Sent: 13 August 2021 12:08

...

Quoted 8 lines

The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via UDP_CORK socket or with MSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.

Patient: It hurts if I do xxx.
Doctor: Don't do xxx then.

It has to be more efficient to buffer partial UDP packets in userspace and only send when all the packet is available.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#6 lore raw
Quoted 12 lines · David Laight

From: Bui Quang Minh

Quoted 1 line

Sent: 13 August 2021 12:08

...

Quoted 8 lines

The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via UDP_CORK socket or with MSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.

Quoted 5 lines

Patient: It hurts if I do xxx.
Doctor: Don't do xxx then.

It has to be more efficient to buffer partial UDP packets in userspace and only send when all the packet is available.

You are right. It can be more efficient, but we don't have controls over user-space processes, and they can do whatever the kernel allows them to do.

Quoted 6 lines

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

maintainer
#7 lore raw
In reply to Bui Quang Minh
Quoted 35 lines · Bui Quang Minh
Quoted 22 lines · Eric Dumazet
Quoted 19 lines · Bui Quang Minh
Quoted 15 lines · Eric Dumazet
Quoted 9 lines · Bui Quang Minh

In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.

A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.

While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.

What is the use case for this feature, adding a test in UDP fast path ?

Quoted 8 lines
Quoted 1 line

This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.

UDP is not reliable.

I find a bit strange we add so many lines of code for a feature trying very hard to to drop _one_ packet.

I think a much better changelog would be welcomed.

Quoted 8 lines

The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via UDP_CORK socket or with MSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.

Instead of checkpointing the state, how about making the kernel drop the next packet.

For instance by setting up->pending to something else than AF_UNSPEC, AF_INET, AF_INET6 from a new setsockopt and testing for this case in the udp_sendmsg up->pending slowpath.

udp_sendmsg already calls udp_v6_flush_pending_frames on error when appending to a pending packet, so returning an error on the next call after restore and have that imply a flush is acceptable. I would introduce a new error code.

The state can perhaps be inferred in other ways, e.g., from up->pending && !up->len or up->pending && !skb_peek_tail(queue). But an explicit up->pending mode will be easier to grasp.

maintainer
#8 lore raw
In reply to Bui Quang Minh
Quoted 7 lines · Bui Quang Minh

In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.

A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.

[ intended to include in my previous response ]

What other related information? Fields like transmit_time and gso_size?

This would be another reason to prefer dropping the packet over trying to restore it incompletely.