udp: UDP socket send queue repair
Subsystems (3)
PATCH v2 1/2 udp: UDP socket send queue repair
In this patch, I implement UDP_REPAIR sockoption and a new path in udp_recvmsg for dumping the corked packet in UDP socket's send queue.
A userspace program can use recvmsg syscall to get the packet's data and the msg_name information of the packet. Currently, other related information in inet_cork that are set in cmsg are not dumped.
While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.
--- include/linux/udp.h | 3 +- include/net/udp.h | 2 + include/uapi/linux/udp.h | 1 + net/ipv4/udp.c | 94 +++++++++++++++++++++++++++++++++++++++- net/ipv6/udp.c | 56 +++++++++++++++++++++++- 5 files changed, 151 insertions(+), 5 deletions(-)
include/linux/udp.h +2 -1
@@ -70,7 +70,8 @@ struct udp_sock {#define UDPLITE_SEND_CC 0x2/* set via udplite setsockopt */ #define UDPLITE_RECV_CC 0x4/* set via udplite setsocktopt */ __u8 pcflag;/* marks socket as UDP-Lite if > 0 */ -__u8 unused[3 ]; +__u8 repair; +__u8 unused[2 ];/* * For encapsulation sockets. */
include/net/udp.h +2 -0
@@ -331,6 +331,8 @@ struct sock *udp6_lib_lookup_skb(const struct sk_buff *skb, __be16 sport, __be16 dport);int udp_read_sock (struct sock *sk,read_descriptor_t *desc,sk_read_actor_t recv_actor); +int udp_peek_sndq (struct sock *sk,struct msghdr *msg, +size_t len);/* UDP uses skb->dev_scratch to cache as much information as possible and avoid * possibly multiple cache miss on dequeue()
include/uapi/linux/udp.h +1 -0
@@ -29,6 +29,7 @@ struct udphdr {/* UDP socket options */ #define UDP_CORK 1/* Never send partially complete segments */ +#define UDP_REPAIR 2/* UDP sock is under repair right now */ #define UDP_ENCAP 100/* Set the socket to accept encapsulated packets */ #define UDP_NO_CHECK6_TX 101/* Disable sending checksum for UDP6X */ #define UDP_NO_CHECK6_RX 102/* Disable accpeting checksum for UDP6 */
net/ipv4/udp.c +92 -2
134 lines not on this page · raw message
net/ipv6/udp.c +54 -2
86 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 9 lines · Bui Quang Minh
In this patch, I implement
UDP_REPAIRsockoption and a new path inudp_recvmsgfor dumping the corked packet in UDP socket's send queue.A userspace program can use recvmsg syscall to get the packet's data and the
msg_nameinformation of the packet. Currently, other related information ininet_corkthat are set in cmsg are not dumped.While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.
What is the use case for this feature, adding a test in UDP fast path ?
CORKed UDP is hardly used nowadays.
IMO, TCP_REPAIR hijacking standard system calls was a design error, we should have added new system calls.
Quoted 10 lines · Eric Dumazet
Quoted 7 lines · Bui Quang Minh
In this patch, I implement
UDP_REPAIRsockoption and a new path inudp_recvmsgfor dumping the corked packet in UDP socket's send queue.A userspace program can use recvmsg syscall to get the packet's data and the
msg_nameinformation of the packet. Currently, other related information ininet_corkthat are set in cmsg are not dumped.
Quoted 5 lines
Quoted 2 lines
While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.
What is the use case for this feature, adding a test in UDP fast path ?
This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.
Quoted 2 lines
IMO,
TCP_REPAIRhijacking standard system calls was a design error, we should have added new system calls.
You are right that adding new system calls is a better approach. What do you think about adding a new option in getsockopt approach?
Thanks,
Quang Minh.
Quoted 13 lines · Bui Quang Minh
Quoted 10 lines · Eric Dumazet
Quoted 7 lines · Bui Quang Minh
In this patch, I implement
UDP_REPAIRsockoption and a new path inudp_recvmsgfor dumping the corked packet in UDP socket's send queue.A userspace program can use recvmsg syscall to get the packet's data and the
msg_nameinformation of the packet. Currently, other related information ininet_corkthat are set in cmsg are not dumped.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.
What is the use case for this feature, adding a test in UDP fast path ?
This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.
UDP is not reliable.
I find a bit strange we add so many lines of code for a feature trying very hard to to drop _one_ packet.
I think a much better changelog would be welcomed.
Quoted 8 lines
Quoted 3 lines
IMO,
TCP_REPAIRhijacking standard system calls was a design error, we should have added new system calls.You are right that adding new system calls is a better approach. What do you think about adding a new option in getsockopt approach?
Thanks,
Quang Minh.
Quoted 22 lines · Eric Dumazet
Quoted 19 lines · Bui Quang Minh
Quoted 15 lines · Eric Dumazet
Quoted 9 lines · Bui Quang Minh
In this patch, I implement
UDP_REPAIRsockoption and a new path inudp_recvmsgfor dumping the corked packet in UDP socket's send queue.A userspace program can use recvmsg syscall to get the packet's data and the
msg_nameinformation of the packet. Currently, other related information ininet_corkthat are set in cmsg are not dumped.While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.
What is the use case for this feature, adding a test in UDP fast path ?
Quoted 8 lines
Quoted 1 line
This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.
UDP is not reliable.
I find a bit strange we add so many lines of code for a feature trying very hard to to drop _one_ packet.
I think a much better changelog would be welcomed.
The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via UDP_CORK socket or with MSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.
Thanks,
Quang Minh.
From: Bui Quang Minh
Quoted 1 line
Sent: 13 August 2021 12:08
...
Quoted 8 lines
The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via
UDP_CORKsocket or withMSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.
Patient: It hurts if I do xxx.
Doctor: Don't do xxx then.
It has to be more efficient to buffer partial UDP packets in userspace and only send when all the packet is available.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 12 lines · David Laight
From: Bui Quang Minh
Quoted 1 line
Sent: 13 August 2021 12:08
...
Quoted 8 lines
The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via
UDP_CORKsocket or withMSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.
Quoted 5 lines
Patient: It hurts if I do xxx.
Doctor: Don't do xxx then.It has to be more efficient to buffer partial UDP packets in userspace and only send when all the packet is available.
You are right. It can be more efficient, but we don't have controls over user-space processes, and they can do whatever the kernel allows them to do.
Quoted 6 lines
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Willem de Bruijn
maintainerQuoted 35 lines · Bui Quang Minh
Quoted 22 lines · Eric Dumazet
Quoted 19 lines · Bui Quang Minh
Quoted 15 lines · Eric Dumazet
Quoted 9 lines · Bui Quang Minh
In this patch, I implement
UDP_REPAIRsockoption and a new path inudp_recvmsgfor dumping the corked packet in UDP socket's send queue.A userspace program can use recvmsg syscall to get the packet's data and the
msg_nameinformation of the packet. Currently, other related information ininet_corkthat are set in cmsg are not dumped.While working on this, I was aware of Lese Doru Calin's patch and got some ideas from it.
What is the use case for this feature, adding a test in UDP fast path ?
Quoted 8 lines
Quoted 1 line
This feature is used to help CRIU to dump CORKed UDP packet in send queue. I'm sorry for being not aware of the performance perspective here.
UDP is not reliable.
I find a bit strange we add so many lines of code for a feature trying very hard to to drop _one_ packet.
I think a much better changelog would be welcomed.
Quoted 8 lines
The reason we want to dump the packet in send queue is to make to state of the application consistent. The scenario is that when an application sends UDP packets via
UDP_CORKsocket or withMSG_MORE, CRIU comes and checkpoints the application. If we drop the data in send queue, when application restores, it sends some more data then turns off the cork and actually sends a packet. The receiving side may get that packet but it's unusual that the first part of that packet is missing because we drop it. So we try to solve this problem with some help from the Linux kernel.
Instead of checkpointing the state, how about making the kernel drop the next packet.
For instance by setting up->pending to something else than AF_UNSPEC, AF_INET, AF_INET6 from a new setsockopt and testing for this case in the udp_sendmsg up->pending slowpath.
udp_sendmsg already calls udp_v6_flush_pending_frames on error when appending to a pending packet, so returning an error on the next call after restore and have that imply a flush is acceptable. I would introduce a new error code.
The state can perhaps be inferred in other ways, e.g., from up->pending && !up->len or up->pending && !skb_peek_tail(queue). But an explicit up->pending mode will be easier to grasp.
Willem de Bruijn
maintainerQuoted 7 lines · Bui Quang Minh
In this patch, I implement
UDP_REPAIRsockoption and a new path inudp_recvmsgfor dumping the corked packet in UDP socket's send queue.A userspace program can use recvmsg syscall to get the packet's data and the
msg_nameinformation of the packet. Currently, other related information ininet_corkthat are set in cmsg are not dumped.
[ intended to include in my previous response ]
What other related information? Fields like transmit_time and gso_size?
This would be another reason to prefer dropping the packet over trying to restore it incompletely.