Skip to contentSkip to conversation

vsock/virtio: fix potential unbounded skb queue

PATCH Applied Eric Dumazet 8 participants 29 messages Apr 30 – May 25

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vsock/virtio: fix potential unbounded skb queue

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>
---
 net/vmw_vsock/virtio_transport_common.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +3 -1
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
 					u32 len)
 {
-	if (vvs->buf_used + len > vvs->buf_alloc)
+	u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+	if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
 		return false;
 
 	vvs->rx_bytes += len;
Signature · 1 line
2.54.0.545.g6539524ca2-goog
#1 lore raw
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Thu, 30 Apr 2026 12:26:52 +0000 you wrote:

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

[...]

Here is the summary with links:
- [net] vsock/virtio: fix potential unbounded skb queue

    https://git.kernel.org/netdev/net/c/059b7dbd20a6

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#2 lore raw
In reply to Eric Dumazet
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
 					u32 len)
 {
-	if (vvs->buf_used + len > vvs->buf_alloc)
+	u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+	if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
 		return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Thanks,
Stefano

#3 lore raw
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

#4 lore raw
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Thanks,
Stefano

#5 lore raw
Quoted 78 lines · Stefano Garzarella
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

Quoted 8 lines

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Thanks,
Stefano

Indeed the old pre-fix skb code would have the same issue.

I can't think of any way around this without extending the spec.

Best,
Bobby

#6 lore raw
Quoted 62 lines · Bobby Eshleman
Quoted 61 lines · Stefano Garzarella
Quoted 59 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

Quoted 5 lines
Quoted 5 lines

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Thanks!

Quoted 16 lines
Quoted 16 lines
Quoted 8 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Quoted 8 lines
Quoted 4 lines

Thanks,
Stefano

Indeed the old pre-fix skb code would have the same issue.

I can't think of any way around this without extending the spec.

Hi, thanks, agree with Bobby, that accounting metadata (e.g. skb size here) was not implemented "by design" in credit logic - another side of data exchange knows nothing about that. Also the same situation was before skb implementation was added by Bobby. So looks like need to update spec may be.

Thanks!

Quoted 3 lines

Best,
Bobby

#7 lore raw
Quoted 94 lines · Arseniy Krasnov
Quoted 62 lines · Bobby Eshleman
Quoted 61 lines · Stefano Garzarella
Quoted 59 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

Quoted 5 lines
Quoted 5 lines

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Thanks!

Quoted 21 lines
Quoted 20 lines
Quoted 8 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Thanks,
Stefano

Quoted 8 lines
Quoted 3 lines

Indeed the old pre-fix skb code would have the same issue.

I can't think of any way around this without extending the spec.

Hi, thanks, agree with Bobby, that accounting metadata (e.g. skb size here) was not implemented "by design" in credit logic - another side of data exchange knows nothing about that. Also the same situation was before skb implementation was added by Bobby. So looks like need to update spec may be.

Even if we change the specifications, we still need to work with older devices, so we should find a solution for this as well.

My main concern is data loss, so I'm considering the following options:

1. Notify the other peer of a smaller buf_alloc from the start, leave some room for overhead, and when it's running out, notify them that buf_alloc = 0. This way, the peer realizes it can’t send anything else.

2. Or update buf_alloc each time by removing the overhead, similar to what’s currently done in virtio_transport_inc_rx_pkt(), but also do it in virtio_transport_inc_tx_pkt().

As I said, IMO this patch alone is incomplete; we need to communicate with the peer somehow regarding space. I don’t think including the overhead in fwd_cnt is spec compliant, since the other peer has no idea how much overhead is needed, but reducing buf_alloc should be okay, even though I’m concerned about packets in flight.

As a quick fix, I think option 2 might be the easiest; I’ll run some tests and send over a patch.

But in the long run, I think we absolutely need to improve memory management in vsock, perhaps by avoiding custom solutions.

Thanks,
Stefano

#8 lore raw
In reply to Stefano Garzarella
Quoted 78 lines · Stefano Garzarella
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

Quoted 6 lines

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Thanks,
Stefano

Why do we need the metadata? Just don't keep it around if you begin running low on memory.

Signature · 1 line
MST
#9 lore raw
Quoted 83 lines · Michael S. Tsirkin
Quoted 82 lines · Stefano Garzarella
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Quoted 5 lines
Quoted 2 lines

Thanks,
Stefano

Why do we need the metadata? Just don't keep it around if you begin running low on memory.

I don't think removing the skuffs will be easy; we added them for ebpf, zero-copy, and seqpacket as well. For now, we're already doing something: merging the skuffs if they don't have EOM set.

As a quick fix, I'm thinking of reducing the buf_alloc value to account for the overhead and notifying the other peer, at least until we find a better solution.

Stefano

#10 lore raw
Quoted 84 lines · Stefano Garzarella
Quoted 83 lines · Michael S. Tsirkin
Quoted 82 lines · Stefano Garzarella
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Stefano

Why do we need the metadata? Just don't keep it around if you begin running low on memory.

I don't think removing the skuffs will be easy; we added them for ebpf, zero-copy, and seqpacket as well.

You do not need to remove them completely.

Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

Quoted 5 lines

As a quick fix, I'm thinking of reducing the buf_alloc value to account for the overhead and notifying the other peer, at least until we find a better solution.

Stefano

well if you want to support pathological cases such as 1 byte messages that would mean like 100x reduction no?

#11 lore raw
Quoted 94 lines · Michael S. Tsirkin
Quoted 84 lines · Stefano Garzarella
Quoted 83 lines · Michael S. Tsirkin
Quoted 82 lines · Stefano Garzarella
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Stefano

Why do we need the metadata? Just don't keep it around if you begin running low on memory.

I don't think removing the skuffs will be easy; we added them for ebpf, zero-copy, and seqpacket as well.

Quoted 8 lines

You do not need to remove them completely.

Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

Quoted 10 lines
Quoted 6 lines

As a quick fix, I'm thinking of reducing the buf_alloc value to account for the overhead and notifying the other peer, at least until we find a better solution.

Stefano

well if you want to support pathological cases such as 1 byte messages that would mean like 100x reduction no?

Yep, but since this patch is already merged, IMHO that is better than losing data in those pathological cases.

Thanks,
Stefano

maintainer
#12 lore raw
Quoted 5 lines · On Thu, 7 May 2026 14:59:13 +0200 Stefano Garzarella wrote:
Quoted 2 lines

well if you want to support pathological cases such as 1 byte messages that would mean like 100x reduction no?

Yep, but since this patch is already merged, IMHO that is better than losing data in those pathological cases.

We can revert if you think that the risk of regression is high.. Please LMK soon, we can do it before patch reaches Linus.

#13 lore raw
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Thu, 7 May 2026 14:59:13 +0200 Stefano Garzarella wrote:
Quoted 2 lines

well if you want to support pathological cases such as 1 byte messages that would mean like 100x reduction no?

Yep, but since this patch is already merged, IMHO that is better than losing data in those pathological cases.

We can revert if you think that the risk of regression is high.. Please LMK soon, we can do it before patch reaches Linus.

Some tests in tools/testing/vsock/vsock_test.c are failing with this patch applied.

Test 18 are failing sometime in this way (I guess because we are dropping packets):

18 - SOCK_STREAM MSG_ZEROCOPY...hash mismatch

Test 22 is failing 100% in this way:

22 - SOCK_STREAM virtio credit update + SO_RCVLOWAT...send failed: Resource temporarily unavailable

With my followup patch adding also advertisement to the other peer (still draft locally, waiting for Michael proposal) I saw 22 failing, because tests expects that can use the entire buf_alloc, but now we are reducing it. So IMO we should do like in __sock_set_rcvbuf() and double the buffer size, or at least digest an overhead equal to the buffer size set by the user via SO_VM_SOCKETS_BUFFER_SIZE (yeah, AF_VSOCK has it owns sockopt since the beginning :-().

With that approach tests are passing, but I'd like to stress a bit more that patch. I'll send it tomorrow as fixup of this patch, or if you prefer to revert, I'll send as standalone.

Thanks,
Stefano

#14 lore raw
Quoted 35 lines · Stefano Garzarella
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Thu, 7 May 2026 14:59:13 +0200 Stefano Garzarella wrote:
Quoted 2 lines

well if you want to support pathological cases such as 1 byte messages that would mean like 100x reduction no?

Yep, but since this patch is already merged, IMHO that is better than losing data in those pathological cases.

We can revert if you think that the risk of regression is high.. Please LMK soon, we can do it before patch reaches Linus.

Some tests in tools/testing/vsock/vsock_test.c are failing with this patch applied.

Test 18 are failing sometime in this way (I guess because we are dropping packets):

18 - SOCK_STREAM MSG_ZEROCOPY...hash mismatch

Test 22 is failing 100% in this way:

22 - SOCK_STREAM virtio credit update + SO_RCVLOWAT...send failed:
Resource temporarily unavailable

With my followup patch adding also advertisement to the other peer (still draft locally, waiting for Michael proposal) I saw 22 failing, because tests expects that can use the entire buf_alloc, but now we are reducing it. So IMO we should do like in __sock_set_rcvbuf() and double the buffer size, or at least digest an overhead equal to the buffer size set by the user via SO_VM_SOCKETS_BUFFER_SIZE (yeah, AF_VSOCK has it owns sockopt since the beginning :-().

Quoted 4 lines

With that approach tests are passing, but I'd like to stress a bit more that patch. I'll send it tomorrow as fixup of this patch, or if you prefer to revert, I'll send as standalone.

A plain revert is a big issue, now users now how to crash hypervisors.

This vulnerability allows a compromised guest (controlling virtio_vsock_hdr fields)
to continuously flood the host's vsock receive queue without triggering any memory
accounting limits or reader wakeups, resulting in unbounded host
kernel memory consumption (Host DoS via OOM).

A vulnerability where a KVM guest can crash or deadlock its host is classified as a KVM DoS.

Am I missing something?

maintainer
#15 lore raw
Quoted 40 lines · On Thu, 7 May 2026 09:32:24 -0700 Eric Dumazet wrote:
Quoted 26 lines · Stefano Garzarella
Quoted 2 lines · Jakub Kicinski

We can revert if you think that the risk of regression is high.. Please LMK soon, we can do it before patch reaches Linus.

Some tests in tools/testing/vsock/vsock_test.c are failing with this patch applied.

Test 18 are failing sometime in this way (I guess because we are dropping packets):

18 - SOCK_STREAM MSG_ZEROCOPY...hash mismatch

Test 22 is failing 100% in this way:

22 - SOCK_STREAM virtio credit update + SO_RCVLOWAT...send failed:
Resource temporarily unavailable

With my followup patch adding also advertisement to the other peer (still draft locally, waiting for Michael proposal) I saw 22 failing, because tests expects that can use the entire buf_alloc, but now we are reducing it. So IMO we should do like in __sock_set_rcvbuf() and double the buffer size, or at least digest an overhead equal to the buffer size set by the user via SO_VM_SOCKETS_BUFFER_SIZE (yeah, AF_VSOCK has it owns sockopt since the beginning :-().

Quoted 3 lines

With that approach tests are passing, but I'd like to stress a bit more that patch. I'll send it tomorrow as fixup of this patch, or if you prefer to revert, I'll send as standalone.

A plain revert is a big issue, now users now how to crash hypervisors.

This vulnerability allows a compromised guest (controlling virtio_vsock_hdr fields)
to continuously flood the host's vsock receive queue without triggering any memory
accounting limits or reader wakeups, resulting in unbounded host
kernel memory consumption (Host DoS via OOM).

Quoted 4 lines

A vulnerability where a KVM guest can crash or deadlock its host is classified as a KVM DoS.

Am I missing something?

Alright, let's leave it.

#16 lore raw
Quoted 41 lines · Jakub Kicinski
Quoted 40 lines · On Thu, 7 May 2026 09:32:24 -0700 Eric Dumazet wrote:
Quoted 26 lines · Stefano Garzarella
Quoted 2 lines · Jakub Kicinski

We can revert if you think that the risk of regression is high.. Please LMK soon, we can do it before patch reaches Linus.

Some tests in tools/testing/vsock/vsock_test.c are failing with this patch applied.

Test 18 are failing sometime in this way (I guess because we are dropping packets):

18 - SOCK_STREAM MSG_ZEROCOPY...hash mismatch

Test 22 is failing 100% in this way:

22 - SOCK_STREAM virtio credit update + SO_RCVLOWAT...send failed:
Resource temporarily unavailable

With my followup patch adding also advertisement to the other peer (still draft locally, waiting for Michael proposal) I saw 22 failing, because tests expects that can use the entire buf_alloc, but now we are reducing it. So IMO we should do like in __sock_set_rcvbuf() and double the buffer size, or at least digest an overhead equal to the buffer size set by the user via SO_VM_SOCKETS_BUFFER_SIZE (yeah, AF_VSOCK has it owns sockopt since the beginning :-().

Quoted 3 lines

With that approach tests are passing, but I'd like to stress a bit more that patch. I'll send it tomorrow as fixup of this patch, or if you prefer to revert, I'll send as standalone.

A plain revert is a big issue, now users now how to crash hypervisors.

This vulnerability allows a compromised guest (controlling virtio_vsock_hdr fields)
to continuously flood the host's vsock receive queue without triggering any memory
accounting limits or reader wakeups, resulting in unbounded host
kernel memory consumption (Host DoS via OOM).

Quoted 7 lines
Quoted 4 lines

A vulnerability where a KVM guest can crash or deadlock its host is classified as a KVM DoS.

Am I missing something?

Alright, let's leave it.

I posted a potential fixup: https://lore.kernel.org/netdev/[email protected]/

Thanks,
Stefano

#17 lore raw
In reply to Stefano Garzarella
Quoted 101 lines · Stefano Garzarella
Quoted 100 lines · Michael S. Tsirkin
Quoted 84 lines · Stefano Garzarella
Quoted 83 lines · Michael S. Tsirkin
Quoted 82 lines · Stefano Garzarella
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Stefano

Why do we need the metadata? Just don't keep it around if you begin running low on memory.

I don't think removing the skuffs will be easy; we added them for ebpf, zero-copy, and seqpacket as well.

You do not need to remove them completely.

Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Quoted 6 lines
Quoted 2 lines

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

I'll send something shortly just to give you an idea.

Quoted 16 lines
Quoted 10 lines
Quoted 6 lines

As a quick fix, I'm thinking of reducing the buf_alloc value to account for the overhead and notifying the other peer, at least until we find a better solution.

Stefano

well if you want to support pathological cases such as 1 byte messages that would mean like 100x reduction no?

Yep, but since this patch is already merged, IMHO that is better than losing data in those pathological cases.

Thanks,
Stefano

#18 lore raw
In reply to Stefano Garzarella
Quoted 101 lines · Stefano Garzarella
Quoted 100 lines · Michael S. Tsirkin
Quoted 84 lines · Stefano Garzarella
Quoted 83 lines · Michael S. Tsirkin
Quoted 82 lines · Stefano Garzarella
Quoted 56 lines · Eric Dumazet
Quoted 48 lines · Stefano Garzarella
Quoted 33 lines · Eric Dumazet

virtio_transport_inc_rx_pkt() checks vvs->rx_bytes + len > vvs->buf_alloc.

virtio_transport_recv_enqueue() skips coalescing for packets with VIRTIO_VSOCK_SEQ_EOM.

If fed with packets with len == 0 and VIRTIO_VSOCK_SEQ_EOM, a very large number of packets can be queued because vvs->rx_bytes stays at 0.

Fix this by estimating the skb metadata size:

(Number of skbs in the queue) * SKB_TRUESIZE(0)

Fixes 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Cc Arseniy Krasnov <[email protected]>
Cc Stefan Hajnoczi <[email protected]>
Cc Stefano Garzarella <[email protected]>
Cc "Michael S. Tsirkin" <[email protected]>
Cc Jason Wang <[email protected]>
Cc Xuan Zhuo <[email protected]>
Cc "Eugenio Pérez" <[email protected]>

---
net/vmw_vsock/virtio_transport_common.c | 4 +++-

1 file changed, 3 insertions(+), 1 deletion(-)
net/vmw_vsock/virtio_transport_common.c +0 -0
index 416d533f493d7b07e9c77c43f741d28cfcd0953e..9b8014516f4fb1130ae184635fbba4dfee58bd64 100644

net/vmw_vsock/virtio_transport_common.c static int virtio_transport_send_pkt_info(struct vsock_sock *vsk, · line 453

Quoted 8 lines
net/vmw_vsock/virtio_transport_common.c +3 -1
@@ -447,7 +447,9 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
 static bool virtio_transport_inc_rx_pkt(struct virtio_vsock_sock *vvs,
                                      u32 len)
 {
-      if (vvs->buf_used + len > vvs->buf_alloc)
+      u64 skb_overhead = (skb_queue_len(&vvs->rx_queue) + 1) * SKB_TRUESIZE(0);
+
+      if (skb_overhead + vvs->buf_used + len > vvs->buf_alloc)
              return false;

I'm not sure about this fix, I mean that maybe this is incomplete. In virtio-vsock, there is a credit mechanism between the two peers:
https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-4850003

Quoted 6 lines

This takes only the payload into account, so it’s true that this problem exists; however, perhaps we should also inform the other peer of a lower credit balance, otherwise the other peer will believe it has much more credit than it actually does, send a large payload, and then the packet will be discarded and the data lost (there are no retransmissions, etc.).

Quoted 3 lines

I dunno, perhaps revert 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
and find a better fix then?

IIRC the same issue was there before the commit fixed by that one (commit 71dc9ec9ac7d ("virtio/vsock: replace virtio_vsock_pkt with sk_buff")), so not sure about reverting it TBH.

CCing Arseniy and Bobby.

Quoted 7 lines

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

This kind of contract is broken.

Yep, I agree, but before we start discarding data (and losing it), IMHO we should at least inform the other peer that we're out of space.

@Stefan, @Michael, do you think we can do something in the spec to avoid this issue and in some way take into account also the metadata in the credit. I mean to avoid the 1-byte packets flooding.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Stefano

Why do we need the metadata? Just don't keep it around if you begin running low on memory.

I don't think removing the skuffs will be easy; we added them for ebpf, zero-copy, and seqpacket as well.

You do not need to remove them completely.

Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Quoted 6 lines
Quoted 2 lines

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.
Yet it seems to work fine in light testing.

-->

vsock/virtio: use DWARF ULEB128 to record EOM boundaries, enable cross-EOM skb coalescing

virtio_transport_recv_enqueue() currently refuses to coalesce an incoming skb with the previous one when the previous skb carries VIRTIO_VSOCK_SEQ_EOM. This forces one skb per seqpacket message. For workloads with many small or zero-byte messages the per-skb overhead (~960 bytes) dominates, causing unbounded memory growth.

Decouple message boundary tracking from the skb structure: store boundary offsets in a compact side buffer using DWARF ULEB128 encoding with the EOR flag folded into the low bit, then allow the data of multiple complete messages to be coalesced into a single skb.

Cross-EOM coalescing fires only when:
- both the tail skb and the incoming packet carry EOM (complete msgs)
- the incoming packet fits in the tail skb's tailroom
- no BPF psock is attached (read_skb expects one msg per skb)

On allocation failure the code falls back to separate skbs (existing behaviour). Credit accounting is unchanged; the boundary buffer is capped at PAGE_SIZE.

include/linux/virtio_vsock.h +7 -0
index f91704731057..e36b9ab28372 100644
@@ -12,6 +12,7 @@
 struct virtio_vsock_skb_cb {
 	bool reply;
 	bool tap_delivered;
+	bool has_boundary_entries;
 	u32 offset;
 };
 
@@ -167,6 +168,12 @@ struct virtio_vsock_sock {
 	u32 buf_used;
 	struct sk_buff_head rx_queue;
 	u32 msg_count;
+
+	/* ULEB128-encoded seqpacket message boundary buffer */
+	u8 *boundary_buf;
+	u32 boundary_len;
+	u32 boundary_alloc;
+	u32 boundary_off;
 };
 
 struct virtio_vsock_pkt_info {
net/vmw_vsock/virtio_transport_common.c +275 -63

420 lines not on this page · raw message

#19 lore raw
Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines · Stefano Garzarella
Quoted 1 line · Michael S. Tsirkin

On Thu, May 07, 2026 at 11:09:47AM +0200, Stefano Garzarella wrote:

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

Quoted 7 lines
Quoted 3 lines

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.

Okay, I somewhat understand, but clearly this isn't net material, so for now I think the best thing to do is to merge the fixup I sent (or something similar): https://lore.kernel.org/netdev/[email protected]/

This is a major change that should be merged with more caution. Could this have too much of an impact on performance?

Thanks,
Stefano

Quoted 488 lines

Yet it seems to work fine in light testing.

-->

vsock/virtio: use DWARF ULEB128 to record EOM boundaries, enable cross-EOM skb coalescing

virtio_transport_recv_enqueue() currently refuses to coalesce an incoming skb with the previous one when the previous skb carries VIRTIO_VSOCK_SEQ_EOM. This forces one skb per seqpacket message. For workloads with many small or zero-byte messages the per-skb overhead (~960 bytes) dominates, causing unbounded memory growth.

Decouple message boundary tracking from the skb structure: store boundary offsets in a compact side buffer using DWARF ULEB128 encoding with the EOR flag folded into the low bit, then allow the data of multiple complete messages to be coalesced into a single skb.

Cross-EOM coalescing fires only when:
- both the tail skb and the incoming packet carry EOM (complete msgs)
- the incoming packet fits in the tail skb's tailroom
- no BPF psock is attached (read_skb expects one msg per skb)

On allocation failure the code falls back to separate skbs (existing behaviour). Credit accounting is unchanged; the boundary buffer is capped at PAGE_SIZE.

include/linux/virtio_vsock.h +7 -0
index f91704731057..e36b9ab28372 100644
@@ -12,6 +12,7 @@
 struct virtio_vsock_skb_cb {
 	bool reply;
 	bool tap_delivered;
+	bool has_boundary_entries;
 	u32 offset;
 };
 
@@ -167,6 +168,12 @@ struct virtio_vsock_sock {
 	u32 buf_used;
 	struct sk_buff_head rx_queue;
 	u32 msg_count;
+
+	/* ULEB128-encoded seqpacket message boundary buffer */
+	u8 *boundary_buf;
+	u32 boundary_len;
+	u32 boundary_alloc;
+	u32 boundary_off;
 };
 
 struct virtio_vsock_pkt_info {
net/vmw_vsock/virtio_transport_common.c +275 -63
index 416d533f493d..81654f70f72c 100644
@@ -11,6 +11,7 @@
 #include <linux/sched/signal.h>
 #include <linux/ctype.h>
 #include <linux/list.h>
+#include <linux/skmsg.h>
 #include <linux/virtio_vsock.h>
 #include <uapi/linux/vsockmon.h>
 
@@ -26,6 +27,91 @@
 /* Threshold for detecting small packets to copy */
 #define GOOD_COPY_LEN  128
 
+#define VSOCK_BOUNDARY_BUF_INIT	64
+#define VSOCK_BOUNDARY_BUF_MAX	PAGE_SIZE
+
+/* ULEB128 boundary encoding: value = (msg_len << 1) | eor.
+ * Each byte carries 7 data bits; bit 7 is set on all but the last byte.
+ * Max 5 bytes for a u32 msg_len (33 bits with eor shift).
+ */
+static int vsock_uleb_encode_boundary(u8 *buf, u32 msg_len, bool eor)
+{
+	u64 val = ((u64)msg_len << 1) | eor;
+	int n = 0;
+
+	do {
+		buf[n] = val & 0x7f;
+		val >>= 7;
+		if (val)
+			buf[n] |= 0x80;
+		n++;
+	} while (val);
+
+	return n;
+}
+
+static int vsock_uleb_decode_boundary(const u8 *buf, u32 avail,
+				      u32 *msg_len, bool *eor)
+{
+	u64 val = 0;
+	int shift = 0;
+	int n = 0;
+
+	do {
+		if (n >= avail || shift >= 35)
+			return -EINVAL;
+		val |= (u64)(buf[n] & 0x7f) << shift;
+		shift += 7;
+	} while (buf[n++] & 0x80);
+
+	*eor = val & 1;
+	*msg_len = val >> 1;
+	return n;
+}
+
+static void vsock_boundary_buf_compact(struct virtio_vsock_sock *vvs)
+{
+	if (vvs->boundary_off == 0)
+		return;
+
+	vvs->boundary_len -= vvs->boundary_off;
+	memmove(vvs->boundary_buf, vvs->boundary_buf + vvs->boundary_off,
+		vvs->boundary_len);
+	vvs->boundary_off = 0;
+}
+
+static int vsock_boundary_buf_ensure(struct virtio_vsock_sock *vvs, u32 needed)
+{
+	u32 new_alloc;
+	u8 *new_buf;
+
+	if (vvs->boundary_alloc >= needed)
+		return 0;
+
+	/* Reclaim consumed space before growing */
+	if (vvs->boundary_off) {
+		needed -= vvs->boundary_off;
+		vsock_boundary_buf_compact(vvs);
+		if (vvs->boundary_alloc >= needed)
+			return 0;
+	}
+
+	new_alloc = max(needed, vvs->boundary_alloc ? vvs->boundary_alloc * 2
+						    : VSOCK_BOUNDARY_BUF_INIT);
+	if (new_alloc > VSOCK_BOUNDARY_BUF_MAX)
+		new_alloc = VSOCK_BOUNDARY_BUF_MAX;
+	if (new_alloc < needed)
+		return -ENOMEM;
+
+	new_buf = krealloc(vvs->boundary_buf, new_alloc, GFP_ATOMIC);
+	if (!new_buf)
+		return -ENOMEM;
+
+	vvs->boundary_buf = new_buf;
+	vvs->boundary_alloc = new_alloc;
+	return 0;
+}
+
 static void virtio_transport_cancel_close_work(struct vsock_sock *vsk,
 					       bool cancel_timeout);
 static s64 virtio_transport_has_space(struct virtio_vsock_sock *vvs);
@@ -682,41 +768,74 @@ virtio_transport_seqpacket_do_peek(struct vsock_sock *vsk,
 	total = 0;
 	len = msg_data_left(msg);
 
-	skb_queue_walk(&vvs->rx_queue, skb) {
-		struct virtio_vsock_hdr *hdr;
+	skb = skb_peek(&vvs->rx_queue);
+	if (skb && VIRTIO_VSOCK_SKB_CB(skb)->has_boundary_entries) {
+		u32 msg_len, offset;
+		size_t bytes;
+		bool eor;
+		int ret;
 
-		if (total < len) {
-			size_t bytes;
+		ret = vsock_uleb_decode_boundary(
+			vvs->boundary_buf + vvs->boundary_off,
+			vvs->boundary_len - vvs->boundary_off,
+			&msg_len, &eor);
+		if (ret < 0)
+			goto unlock;
+
+		offset = VIRTIO_VSOCK_SKB_CB(skb)->offset;
+		bytes = min(len, (size_t)msg_len);
+
+		if (bytes) {
 			int err;
 
-			bytes = len - total;
-			if (bytes > skb->len)
-				bytes = skb->len;
-
 			spin_unlock_bh(&vvs->rx_lock);
-
-			/* sk_lock is held by caller so no one else can dequeue.
-			 * Unlock rx_lock since skb_copy_datagram_iter() may sleep.
-			 */
-			err = skb_copy_datagram_iter(skb, VIRTIO_VSOCK_SKB_CB(skb)->offset,
+			err = skb_copy_datagram_iter(skb, offset,
 						     &msg->msg_iter, bytes);
 			if (err)
 				return err;
-
 			spin_lock_bh(&vvs->rx_lock);
 		}
 
-		total += skb->len;
-		hdr = virtio_vsock_hdr(skb);
+		total = msg_len;
+		if (eor)
+			msg->msg_flags |= MSG_EOR;
+	} else {
+		skb_queue_walk(&vvs->rx_queue, skb) {
+			struct virtio_vsock_hdr *hdr;
 
-		if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
-			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOR)
-				msg->msg_flags |= MSG_EOR;
+			if (total < len) {
+				size_t bytes;
+				int err;
 
-			break;
+				bytes = len - total;
+				if (bytes > skb->len)
+					bytes = skb->len;
+
+				spin_unlock_bh(&vvs->rx_lock);
+
+				err = skb_copy_datagram_iter(
+					skb,
+					VIRTIO_VSOCK_SKB_CB(skb)->offset,
+					&msg->msg_iter, bytes);
+				if (err)
+					return err;
+
+				spin_lock_bh(&vvs->rx_lock);
+			}
+
+			total += skb->len;
+			hdr = virtio_vsock_hdr(skb);
+
+			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+				if (le32_to_cpu(hdr->flags) &
+				    VIRTIO_VSOCK_SEQ_EOR)
+					msg->msg_flags |= MSG_EOR;
+				break;
+			}
 		}
 	}
 
+unlock:
 	spin_unlock_bh(&vvs->rx_lock);
 
 	return total;
@@ -740,57 +859,105 @@ static int virtio_transport_seqpacket_do_dequeue(struct vsock_sock *vsk,
 	}
 
 	while (!msg_ready) {
-		struct virtio_vsock_hdr *hdr;
-		size_t pkt_len;
-
-		skb = __skb_dequeue(&vvs->rx_queue);
+		skb = skb_peek(&vvs->rx_queue);
 		if (!skb)
 			break;
-		hdr = virtio_vsock_hdr(skb);
-		pkt_len = (size_t)le32_to_cpu(hdr->len);
 
-		if (dequeued_len >= 0) {
+		if (VIRTIO_VSOCK_SKB_CB(skb)->has_boundary_entries) {
 			size_t bytes_to_copy;
+			u32 msg_len, offset;
+			bool eor;
+			int ret;
 
-			bytes_to_copy = min(user_buf_len, pkt_len);
+			ret = vsock_uleb_decode_boundary(
+				vvs->boundary_buf + vvs->boundary_off,
+				vvs->boundary_len - vvs->boundary_off,
+				&msg_len, &eor);
+			if (ret < 0)
+				break;
+			vvs->boundary_off += ret;
 
-			if (bytes_to_copy) {
+			offset = VIRTIO_VSOCK_SKB_CB(skb)->offset;
+			bytes_to_copy = min(user_buf_len, (size_t)msg_len);
+
+			if (bytes_to_copy && dequeued_len >= 0) {
 				int err;
 
-				/* sk_lock is held by caller so no one else can dequeue.
-				 * Unlock rx_lock since skb_copy_datagram_iter() may sleep.
-				 */
 				spin_unlock_bh(&vvs->rx_lock);
-
-				err = skb_copy_datagram_iter(skb, 0,
+				err = skb_copy_datagram_iter(skb, offset,
 							     &msg->msg_iter,
 							     bytes_to_copy);
-				if (err) {
-					/* Copy of message failed. Rest of
-					 * fragments will be freed without copy.
-					 */
-					dequeued_len = err;
-				} else {
-					user_buf_len -= bytes_to_copy;
-				}
-
 				spin_lock_bh(&vvs->rx_lock);
+				if (err)
+					dequeued_len = err;
+				else
+					user_buf_len -= bytes_to_copy;
 			}
 
 			if (dequeued_len >= 0)
-				dequeued_len += pkt_len;
-		}
+				dequeued_len += msg_len;
 
-		if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+			VIRTIO_VSOCK_SKB_CB(skb)->offset += msg_len;
 			msg_ready = true;
 			vvs->msg_count--;
 
-			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOR)
+			if (eor)
 				msg->msg_flags |= MSG_EOR;
-		}
 
-		virtio_transport_dec_rx_pkt(vvs, pkt_len, pkt_len);
-		kfree_skb(skb);
+			virtio_transport_dec_rx_pkt(vvs, msg_len, msg_len);
+
+			if (VIRTIO_VSOCK_SKB_CB(skb)->offset >= skb->len) {
+				__skb_unlink(skb, &vvs->rx_queue);
+				kfree_skb(skb);
+			}
+
+			if (vvs->boundary_off >= vvs->boundary_len / 2)
+				vsock_boundary_buf_compact(vvs);
+		} else {
+			struct virtio_vsock_hdr *hdr;
+			size_t pkt_len;
+
+			skb = __skb_dequeue(&vvs->rx_queue);
+			if (!skb)
+				break;
+			hdr = virtio_vsock_hdr(skb);
+			pkt_len = (size_t)le32_to_cpu(hdr->len);
+
+			if (dequeued_len >= 0) {
+				size_t bytes_to_copy;
+
+				bytes_to_copy = min(user_buf_len, pkt_len);
+
+				if (bytes_to_copy) {
+					int err;
+
+					spin_unlock_bh(&vvs->rx_lock);
+					err = skb_copy_datagram_iter(
+						skb, 0, &msg->msg_iter,
+						bytes_to_copy);
+					if (err)
+						dequeued_len = err;
+					else
+						user_buf_len -= bytes_to_copy;
+					spin_lock_bh(&vvs->rx_lock);
+				}
+
+				if (dequeued_len >= 0)
+					dequeued_len += pkt_len;
+			}
+
+			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+				msg_ready = true;
+				vvs->msg_count--;
+
+				if (le32_to_cpu(hdr->flags) &
+				    VIRTIO_VSOCK_SEQ_EOR)
+					msg->msg_flags |= MSG_EOR;
+			}
+
+			virtio_transport_dec_rx_pkt(vvs, pkt_len, pkt_len);
+			kfree_skb(skb);
+		}
 	}
 
 	spin_unlock_bh(&vvs->rx_lock);
@@ -1132,6 +1299,7 @@ void virtio_transport_destruct(struct vsock_sock *vsk)
 
 	virtio_transport_cancel_close_work(vsk, true);
 
+	kfree(vvs->boundary_buf);
 	kfree(vvs);
 	vsk->trans = NULL;
 }
@@ -1224,6 +1392,11 @@ static void virtio_transport_remove_sock(struct vsock_sock *vsk)
 	 * removing it.
 	 */
 	__skb_queue_purge(&vvs->rx_queue);
+	kfree(vvs->boundary_buf);
+	vvs->boundary_buf = NULL;
+	vvs->boundary_len = 0;
+	vvs->boundary_alloc = 0;
+	vvs->boundary_off = 0;
 	vsock_remove_sock(vsk);
 }
 
@@ -1395,23 +1568,62 @@ virtio_transport_recv_enqueue(struct vsock_sock *vsk,
 	    !skb_is_nonlinear(skb)) {
 		struct virtio_vsock_hdr *last_hdr;
 		struct sk_buff *last_skb;
+		bool last_has_eom;
+		bool has_eom;
 
 		last_skb = skb_peek_tail(&vvs->rx_queue);
 		last_hdr = virtio_vsock_hdr(last_skb);
+		last_has_eom = le32_to_cpu(last_hdr->flags) & VIRTIO_VSOCK_SEQ_EOM;
+		has_eom = le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM;
 
-		/* If there is space in the last packet queued, we copy the
-		 * new packet in its buffer. We avoid this if the last packet
-		 * queued has VIRTIO_VSOCK_SEQ_EOM set, because this is
-		 * delimiter of SEQPACKET message, so 'pkt' is the first packet
-		 * of a new message.
-		 */
-		if (skb->len < skb_tailroom(last_skb) &&
-		    !(le32_to_cpu(last_hdr->flags) & VIRTIO_VSOCK_SEQ_EOM)) {
-			memcpy(skb_put(last_skb, skb->len), skb->data, skb->len);
-			free_pkt = true;
-			last_hdr->flags |= hdr->flags;
-			le32_add_cpu(&last_hdr->len, len);
-			goto out;
+		if (skb->len < skb_tailroom(last_skb)) {
+			if (!last_has_eom) {
+				/* Same-message coalescing (existing path) */
+				memcpy(skb_put(last_skb, skb->len),
+				       skb->data, skb->len);
+				free_pkt = true;
+				last_hdr->flags |= hdr->flags;
+				le32_add_cpu(&last_hdr->len, len);
+				goto out;
+			}
+
+			/* Cross-EOM: coalesce complete messages into one skb,
+			 * recording message boundaries in a compact BER buffer.
+			 * Only when incoming packet also has EOM (complete msg).
+			 */
+			if (has_eom && !sk_psock(sk_vsock(vsk))) {
+				bool prev_eor, cur_eor;
+				u8 tmp[12];
+				int n = 0;
+
+				cur_eor = le32_to_cpu(hdr->flags) &
+					  VIRTIO_VSOCK_SEQ_EOR;
+
+				if (!VIRTIO_VSOCK_SKB_CB(last_skb)->has_boundary_entries) {
+					u32 prev_len = le32_to_cpu(last_hdr->len);
+
+					prev_eor = le32_to_cpu(last_hdr->flags) &
+						   VIRTIO_VSOCK_SEQ_EOR;
+					n += vsock_uleb_encode_boundary(
+						tmp + n, prev_len, prev_eor);
+				}
+				n += vsock_uleb_encode_boundary(
+					tmp + n, len, cur_eor);
+
+				if (!vsock_boundary_buf_ensure(
+					    vvs, vvs->boundary_len + n)) {
+					memcpy(vvs->boundary_buf +
+					       vvs->boundary_len, tmp, n);
+					vvs->boundary_len += n;
+					VIRTIO_VSOCK_SKB_CB(last_skb)->has_boundary_entries = true;
+					memcpy(skb_put(last_skb, skb->len),
+					       skb->data, skb->len);
+					free_pkt = true;
+					last_hdr->flags |= hdr->flags;
+					le32_add_cpu(&last_hdr->len, len);
+					goto out;
+				}
+			}
 		}
 	}
 
#20 lore raw
Quoted 18 lines · Stefano Garzarella
Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines · Stefano Garzarella
Quoted 1 line · Michael S. Tsirkin

On Thu, May 07, 2026 at 11:09:47AM +0200, Stefano Garzarella wrote:

[...]

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

Quoted 8 lines
Quoted 3 lines

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.

Okay, I somewhat understand, but clearly this isn't net material, so for now I think the best thing to do is to merge the fixup I sent (or something similar):
https://lore.kernel.org/netdev/[email protected]/

will respond there.

Quoted 5 lines

This is a major change that should be merged with more caution. Could this have too much of an impact on performance?

Thanks,
Stefano

Upstream is so broken now, I'd not worry about perf even.

Quoted 488 lines
Quoted 488 lines

Yet it seems to work fine in light testing.

-->

vsock/virtio: use DWARF ULEB128 to record EOM boundaries, enable cross-EOM skb coalescing

virtio_transport_recv_enqueue() currently refuses to coalesce an incoming skb with the previous one when the previous skb carries VIRTIO_VSOCK_SEQ_EOM. This forces one skb per seqpacket message. For workloads with many small or zero-byte messages the per-skb overhead (~960 bytes) dominates, causing unbounded memory growth.

Decouple message boundary tracking from the skb structure: store boundary offsets in a compact side buffer using DWARF ULEB128 encoding with the EOR flag folded into the low bit, then allow the data of multiple complete messages to be coalesced into a single skb.

Cross-EOM coalescing fires only when:
- both the tail skb and the incoming packet carry EOM (complete msgs)
- the incoming packet fits in the tail skb's tailroom
- no BPF psock is attached (read_skb expects one msg per skb)

On allocation failure the code falls back to separate skbs (existing behaviour). Credit accounting is unchanged; the boundary buffer is capped at PAGE_SIZE.

include/linux/virtio_vsock.h +7 -0
index f91704731057..e36b9ab28372 100644
@@ -12,6 +12,7 @@
 struct virtio_vsock_skb_cb {
 	bool reply;
 	bool tap_delivered;
+	bool has_boundary_entries;
 	u32 offset;
 };
 
@@ -167,6 +168,12 @@ struct virtio_vsock_sock {
 	u32 buf_used;
 	struct sk_buff_head rx_queue;
 	u32 msg_count;
+
+	/* ULEB128-encoded seqpacket message boundary buffer */
+	u8 *boundary_buf;
+	u32 boundary_len;
+	u32 boundary_alloc;
+	u32 boundary_off;
 };
 
 struct virtio_vsock_pkt_info {
net/vmw_vsock/virtio_transport_common.c +275 -63
index 416d533f493d..81654f70f72c 100644
@@ -11,6 +11,7 @@
 #include <linux/sched/signal.h>
 #include <linux/ctype.h>
 #include <linux/list.h>
+#include <linux/skmsg.h>
 #include <linux/virtio_vsock.h>
 #include <uapi/linux/vsockmon.h>
 
@@ -26,6 +27,91 @@
 /* Threshold for detecting small packets to copy */
 #define GOOD_COPY_LEN  128
 
+#define VSOCK_BOUNDARY_BUF_INIT	64
+#define VSOCK_BOUNDARY_BUF_MAX	PAGE_SIZE
+
+/* ULEB128 boundary encoding: value = (msg_len << 1) | eor.
+ * Each byte carries 7 data bits; bit 7 is set on all but the last byte.
+ * Max 5 bytes for a u32 msg_len (33 bits with eor shift).
+ */
+static int vsock_uleb_encode_boundary(u8 *buf, u32 msg_len, bool eor)
+{
+	u64 val = ((u64)msg_len << 1) | eor;
+	int n = 0;
+
+	do {
+		buf[n] = val & 0x7f;
+		val >>= 7;
+		if (val)
+			buf[n] |= 0x80;
+		n++;
+	} while (val);
+
+	return n;
+}
+
+static int vsock_uleb_decode_boundary(const u8 *buf, u32 avail,
+				      u32 *msg_len, bool *eor)
+{
+	u64 val = 0;
+	int shift = 0;
+	int n = 0;
+
+	do {
+		if (n >= avail || shift >= 35)
+			return -EINVAL;
+		val |= (u64)(buf[n] & 0x7f) << shift;
+		shift += 7;
+	} while (buf[n++] & 0x80);
+
+	*eor = val & 1;
+	*msg_len = val >> 1;
+	return n;
+}
+
+static void vsock_boundary_buf_compact(struct virtio_vsock_sock *vvs)
+{
+	if (vvs->boundary_off == 0)
+		return;
+
+	vvs->boundary_len -= vvs->boundary_off;
+	memmove(vvs->boundary_buf, vvs->boundary_buf + vvs->boundary_off,
+		vvs->boundary_len);
+	vvs->boundary_off = 0;
+}
+
+static int vsock_boundary_buf_ensure(struct virtio_vsock_sock *vvs, u32 needed)
+{
+	u32 new_alloc;
+	u8 *new_buf;
+
+	if (vvs->boundary_alloc >= needed)
+		return 0;
+
+	/* Reclaim consumed space before growing */
+	if (vvs->boundary_off) {
+		needed -= vvs->boundary_off;
+		vsock_boundary_buf_compact(vvs);
+		if (vvs->boundary_alloc >= needed)
+			return 0;
+	}
+
+	new_alloc = max(needed, vvs->boundary_alloc ? vvs->boundary_alloc * 2
+						    : VSOCK_BOUNDARY_BUF_INIT);
+	if (new_alloc > VSOCK_BOUNDARY_BUF_MAX)
+		new_alloc = VSOCK_BOUNDARY_BUF_MAX;
+	if (new_alloc < needed)
+		return -ENOMEM;
+
+	new_buf = krealloc(vvs->boundary_buf, new_alloc, GFP_ATOMIC);
+	if (!new_buf)
+		return -ENOMEM;
+
+	vvs->boundary_buf = new_buf;
+	vvs->boundary_alloc = new_alloc;
+	return 0;
+}
+
 static void virtio_transport_cancel_close_work(struct vsock_sock *vsk,
 					       bool cancel_timeout);
 static s64 virtio_transport_has_space(struct virtio_vsock_sock *vvs);
@@ -682,41 +768,74 @@ virtio_transport_seqpacket_do_peek(struct vsock_sock *vsk,
 	total = 0;
 	len = msg_data_left(msg);
 
-	skb_queue_walk(&vvs->rx_queue, skb) {
-		struct virtio_vsock_hdr *hdr;
+	skb = skb_peek(&vvs->rx_queue);
+	if (skb && VIRTIO_VSOCK_SKB_CB(skb)->has_boundary_entries) {
+		u32 msg_len, offset;
+		size_t bytes;
+		bool eor;
+		int ret;
 
-		if (total < len) {
-			size_t bytes;
+		ret = vsock_uleb_decode_boundary(
+			vvs->boundary_buf + vvs->boundary_off,
+			vvs->boundary_len - vvs->boundary_off,
+			&msg_len, &eor);
+		if (ret < 0)
+			goto unlock;
+
+		offset = VIRTIO_VSOCK_SKB_CB(skb)->offset;
+		bytes = min(len, (size_t)msg_len);
+
+		if (bytes) {
 			int err;
 
-			bytes = len - total;
-			if (bytes > skb->len)
-				bytes = skb->len;
-
 			spin_unlock_bh(&vvs->rx_lock);
-
-			/* sk_lock is held by caller so no one else can dequeue.
-			 * Unlock rx_lock since skb_copy_datagram_iter() may sleep.
-			 */
-			err = skb_copy_datagram_iter(skb, VIRTIO_VSOCK_SKB_CB(skb)->offset,
+			err = skb_copy_datagram_iter(skb, offset,
 						     &msg->msg_iter, bytes);
 			if (err)
 				return err;
-
 			spin_lock_bh(&vvs->rx_lock);
 		}
 
-		total += skb->len;
-		hdr = virtio_vsock_hdr(skb);
+		total = msg_len;
+		if (eor)
+			msg->msg_flags |= MSG_EOR;
+	} else {
+		skb_queue_walk(&vvs->rx_queue, skb) {
+			struct virtio_vsock_hdr *hdr;
 
-		if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
-			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOR)
-				msg->msg_flags |= MSG_EOR;
+			if (total < len) {
+				size_t bytes;
+				int err;
 
-			break;
+				bytes = len - total;
+				if (bytes > skb->len)
+					bytes = skb->len;
+
+				spin_unlock_bh(&vvs->rx_lock);
+
+				err = skb_copy_datagram_iter(
+					skb,
+					VIRTIO_VSOCK_SKB_CB(skb)->offset,
+					&msg->msg_iter, bytes);
+				if (err)
+					return err;
+
+				spin_lock_bh(&vvs->rx_lock);
+			}
+
+			total += skb->len;
+			hdr = virtio_vsock_hdr(skb);
+
+			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+				if (le32_to_cpu(hdr->flags) &
+				    VIRTIO_VSOCK_SEQ_EOR)
+					msg->msg_flags |= MSG_EOR;
+				break;
+			}
 		}
 	}
 
+unlock:
 	spin_unlock_bh(&vvs->rx_lock);
 
 	return total;
@@ -740,57 +859,105 @@ static int virtio_transport_seqpacket_do_dequeue(struct vsock_sock *vsk,
 	}
 
 	while (!msg_ready) {
-		struct virtio_vsock_hdr *hdr;
-		size_t pkt_len;
-
-		skb = __skb_dequeue(&vvs->rx_queue);
+		skb = skb_peek(&vvs->rx_queue);
 		if (!skb)
 			break;
-		hdr = virtio_vsock_hdr(skb);
-		pkt_len = (size_t)le32_to_cpu(hdr->len);
 
-		if (dequeued_len >= 0) {
+		if (VIRTIO_VSOCK_SKB_CB(skb)->has_boundary_entries) {
 			size_t bytes_to_copy;
+			u32 msg_len, offset;
+			bool eor;
+			int ret;
 
-			bytes_to_copy = min(user_buf_len, pkt_len);
+			ret = vsock_uleb_decode_boundary(
+				vvs->boundary_buf + vvs->boundary_off,
+				vvs->boundary_len - vvs->boundary_off,
+				&msg_len, &eor);
+			if (ret < 0)
+				break;
+			vvs->boundary_off += ret;
 
-			if (bytes_to_copy) {
+			offset = VIRTIO_VSOCK_SKB_CB(skb)->offset;
+			bytes_to_copy = min(user_buf_len, (size_t)msg_len);
+
+			if (bytes_to_copy && dequeued_len >= 0) {
 				int err;
 
-				/* sk_lock is held by caller so no one else can dequeue.
-				 * Unlock rx_lock since skb_copy_datagram_iter() may sleep.
-				 */
 				spin_unlock_bh(&vvs->rx_lock);
-
-				err = skb_copy_datagram_iter(skb, 0,
+				err = skb_copy_datagram_iter(skb, offset,
 							     &msg->msg_iter,
 							     bytes_to_copy);
-				if (err) {
-					/* Copy of message failed. Rest of
-					 * fragments will be freed without copy.
-					 */
-					dequeued_len = err;
-				} else {
-					user_buf_len -= bytes_to_copy;
-				}
-
 				spin_lock_bh(&vvs->rx_lock);
+				if (err)
+					dequeued_len = err;
+				else
+					user_buf_len -= bytes_to_copy;
 			}
 
 			if (dequeued_len >= 0)
-				dequeued_len += pkt_len;
-		}
+				dequeued_len += msg_len;
 
-		if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+			VIRTIO_VSOCK_SKB_CB(skb)->offset += msg_len;
 			msg_ready = true;
 			vvs->msg_count--;
 
-			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOR)
+			if (eor)
 				msg->msg_flags |= MSG_EOR;
-		}
 
-		virtio_transport_dec_rx_pkt(vvs, pkt_len, pkt_len);
-		kfree_skb(skb);
+			virtio_transport_dec_rx_pkt(vvs, msg_len, msg_len);
+
+			if (VIRTIO_VSOCK_SKB_CB(skb)->offset >= skb->len) {
+				__skb_unlink(skb, &vvs->rx_queue);
+				kfree_skb(skb);
+			}
+
+			if (vvs->boundary_off >= vvs->boundary_len / 2)
+				vsock_boundary_buf_compact(vvs);
+		} else {
+			struct virtio_vsock_hdr *hdr;
+			size_t pkt_len;
+
+			skb = __skb_dequeue(&vvs->rx_queue);
+			if (!skb)
+				break;
+			hdr = virtio_vsock_hdr(skb);
+			pkt_len = (size_t)le32_to_cpu(hdr->len);
+
+			if (dequeued_len >= 0) {
+				size_t bytes_to_copy;
+
+				bytes_to_copy = min(user_buf_len, pkt_len);
+
+				if (bytes_to_copy) {
+					int err;
+
+					spin_unlock_bh(&vvs->rx_lock);
+					err = skb_copy_datagram_iter(
+						skb, 0, &msg->msg_iter,
+						bytes_to_copy);
+					if (err)
+						dequeued_len = err;
+					else
+						user_buf_len -= bytes_to_copy;
+					spin_lock_bh(&vvs->rx_lock);
+				}
+
+				if (dequeued_len >= 0)
+					dequeued_len += pkt_len;
+			}
+
+			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+				msg_ready = true;
+				vvs->msg_count--;
+
+				if (le32_to_cpu(hdr->flags) &
+				    VIRTIO_VSOCK_SEQ_EOR)
+					msg->msg_flags |= MSG_EOR;
+			}
+
+			virtio_transport_dec_rx_pkt(vvs, pkt_len, pkt_len);
+			kfree_skb(skb);
+		}
 	}
 
 	spin_unlock_bh(&vvs->rx_lock);
@@ -1132,6 +1299,7 @@ void virtio_transport_destruct(struct vsock_sock *vsk)
 
 	virtio_transport_cancel_close_work(vsk, true);
 
+	kfree(vvs->boundary_buf);
 	kfree(vvs);
 	vsk->trans = NULL;
 }
@@ -1224,6 +1392,11 @@ static void virtio_transport_remove_sock(struct vsock_sock *vsk)
 	 * removing it.
 	 */
 	__skb_queue_purge(&vvs->rx_queue);
+	kfree(vvs->boundary_buf);
+	vvs->boundary_buf = NULL;
+	vvs->boundary_len = 0;
+	vvs->boundary_alloc = 0;
+	vvs->boundary_off = 0;
 	vsock_remove_sock(vsk);
 }
 
@@ -1395,23 +1568,62 @@ virtio_transport_recv_enqueue(struct vsock_sock *vsk,
 	    !skb_is_nonlinear(skb)) {
 		struct virtio_vsock_hdr *last_hdr;
 		struct sk_buff *last_skb;
+		bool last_has_eom;
+		bool has_eom;
 
 		last_skb = skb_peek_tail(&vvs->rx_queue);
 		last_hdr = virtio_vsock_hdr(last_skb);
+		last_has_eom = le32_to_cpu(last_hdr->flags) & VIRTIO_VSOCK_SEQ_EOM;
+		has_eom = le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM;
 
-		/* If there is space in the last packet queued, we copy the
-		 * new packet in its buffer. We avoid this if the last packet
-		 * queued has VIRTIO_VSOCK_SEQ_EOM set, because this is
-		 * delimiter of SEQPACKET message, so 'pkt' is the first packet
-		 * of a new message.
-		 */
-		if (skb->len < skb_tailroom(last_skb) &&
-		    !(le32_to_cpu(last_hdr->flags) & VIRTIO_VSOCK_SEQ_EOM)) {
-			memcpy(skb_put(last_skb, skb->len), skb->data, skb->len);
-			free_pkt = true;
-			last_hdr->flags |= hdr->flags;
-			le32_add_cpu(&last_hdr->len, len);
-			goto out;
+		if (skb->len < skb_tailroom(last_skb)) {
+			if (!last_has_eom) {
+				/* Same-message coalescing (existing path) */
+				memcpy(skb_put(last_skb, skb->len),
+				       skb->data, skb->len);
+				free_pkt = true;
+				last_hdr->flags |= hdr->flags;
+				le32_add_cpu(&last_hdr->len, len);
+				goto out;
+			}
+
+			/* Cross-EOM: coalesce complete messages into one skb,
+			 * recording message boundaries in a compact BER buffer.
+			 * Only when incoming packet also has EOM (complete msg).
+			 */
+			if (has_eom && !sk_psock(sk_vsock(vsk))) {
+				bool prev_eor, cur_eor;
+				u8 tmp[12];
+				int n = 0;
+
+				cur_eor = le32_to_cpu(hdr->flags) &
+					  VIRTIO_VSOCK_SEQ_EOR;
+
+				if (!VIRTIO_VSOCK_SKB_CB(last_skb)->has_boundary_entries) {
+					u32 prev_len = le32_to_cpu(last_hdr->len);
+
+					prev_eor = le32_to_cpu(last_hdr->flags) &
+						   VIRTIO_VSOCK_SEQ_EOR;
+					n += vsock_uleb_encode_boundary(
+						tmp + n, prev_len, prev_eor);
+				}
+				n += vsock_uleb_encode_boundary(
+					tmp + n, len, cur_eor);
+
+				if (!vsock_boundary_buf_ensure(
+					    vvs, vvs->boundary_len + n)) {
+					memcpy(vvs->boundary_buf +
+					       vvs->boundary_len, tmp, n);
+					vvs->boundary_len += n;
+					VIRTIO_VSOCK_SKB_CB(last_skb)->has_boundary_entries = true;
+					memcpy(skb_put(last_skb, skb->len),
+					       skb->data, skb->len);
+					free_pkt = true;
+					last_hdr->flags |= hdr->flags;
+					le32_add_cpu(&last_hdr->len, len);
+					goto out;
+				}
+			}
 		}
 	}
 
#21 lore raw
In reply to Stefano Garzarella
Quoted 16 lines · Stefano Garzarella
Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines · Stefano Garzarella
Quoted 1 line · Michael S. Tsirkin

On Thu, May 07, 2026 at 11:09:47AM +0200, Stefano Garzarella wrote:

[...]

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Do you have something in mind?

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.

Okay, I somewhat understand, but clearly this isn't net material

I doubt we have many other options given reverting the regression was ruled out.

Quoted 4 lines

so for now
I think the best thing to do is to merge the fixup I sent (or something similar):
https://lore.kernel.org/netdev/[email protected]/

I reviewed that one, problem is it's a spec violation/change that we'll have to support forever.

Quoted 5 lines

This is a major change that should be merged with more caution. Could this have too much of an impact on performance?

Thanks,
Stefano

It's really a POC, real patch is left as an excersise for the reader :). The correct approach IMHO is to only start using this
when we wasted a lot of memory on small packets.

For example, if sum(truesize) >= buf size.

then we'll not see a perf impact unless it's already pathological.

Quoted 488 lines
Quoted 488 lines

Yet it seems to work fine in light testing.

-->

vsock/virtio: use DWARF ULEB128 to record EOM boundaries, enable cross-EOM skb coalescing

virtio_transport_recv_enqueue() currently refuses to coalesce an incoming skb with the previous one when the previous skb carries VIRTIO_VSOCK_SEQ_EOM. This forces one skb per seqpacket message. For workloads with many small or zero-byte messages the per-skb overhead (~960 bytes) dominates, causing unbounded memory growth.

Decouple message boundary tracking from the skb structure: store boundary offsets in a compact side buffer using DWARF ULEB128 encoding with the EOR flag folded into the low bit, then allow the data of multiple complete messages to be coalesced into a single skb.

Cross-EOM coalescing fires only when:
- both the tail skb and the incoming packet carry EOM (complete msgs)
- the incoming packet fits in the tail skb's tailroom
- no BPF psock is attached (read_skb expects one msg per skb)

On allocation failure the code falls back to separate skbs (existing behaviour). Credit accounting is unchanged; the boundary buffer is capped at PAGE_SIZE.

include/linux/virtio_vsock.h +7 -0
index f91704731057..e36b9ab28372 100644
@@ -12,6 +12,7 @@
 struct virtio_vsock_skb_cb {
 	bool reply;
 	bool tap_delivered;
+	bool has_boundary_entries;
 	u32 offset;
 };
 
@@ -167,6 +168,12 @@ struct virtio_vsock_sock {
 	u32 buf_used;
 	struct sk_buff_head rx_queue;
 	u32 msg_count;
+
+	/* ULEB128-encoded seqpacket message boundary buffer */
+	u8 *boundary_buf;
+	u32 boundary_len;
+	u32 boundary_alloc;
+	u32 boundary_off;
 };
 
 struct virtio_vsock_pkt_info {
net/vmw_vsock/virtio_transport_common.c +275 -63
index 416d533f493d..81654f70f72c 100644
@@ -11,6 +11,7 @@
 #include <linux/sched/signal.h>
 #include <linux/ctype.h>
 #include <linux/list.h>
+#include <linux/skmsg.h>
 #include <linux/virtio_vsock.h>
 #include <uapi/linux/vsockmon.h>
 
@@ -26,6 +27,91 @@
 /* Threshold for detecting small packets to copy */
 #define GOOD_COPY_LEN  128
 
+#define VSOCK_BOUNDARY_BUF_INIT	64
+#define VSOCK_BOUNDARY_BUF_MAX	PAGE_SIZE
+
+/* ULEB128 boundary encoding: value = (msg_len << 1) | eor.
+ * Each byte carries 7 data bits; bit 7 is set on all but the last byte.
+ * Max 5 bytes for a u32 msg_len (33 bits with eor shift).
+ */
+static int vsock_uleb_encode_boundary(u8 *buf, u32 msg_len, bool eor)
+{
+	u64 val = ((u64)msg_len << 1) | eor;
+	int n = 0;
+
+	do {
+		buf[n] = val & 0x7f;
+		val >>= 7;
+		if (val)
+			buf[n] |= 0x80;
+		n++;
+	} while (val);
+
+	return n;
+}
+
+static int vsock_uleb_decode_boundary(const u8 *buf, u32 avail,
+				      u32 *msg_len, bool *eor)
+{
+	u64 val = 0;
+	int shift = 0;
+	int n = 0;
+
+	do {
+		if (n >= avail || shift >= 35)
+			return -EINVAL;
+		val |= (u64)(buf[n] & 0x7f) << shift;
+		shift += 7;
+	} while (buf[n++] & 0x80);
+
+	*eor = val & 1;
+	*msg_len = val >> 1;
+	return n;
+}
+
+static void vsock_boundary_buf_compact(struct virtio_vsock_sock *vvs)
+{
+	if (vvs->boundary_off == 0)
+		return;
+
+	vvs->boundary_len -= vvs->boundary_off;
+	memmove(vvs->boundary_buf, vvs->boundary_buf + vvs->boundary_off,
+		vvs->boundary_len);
+	vvs->boundary_off = 0;
+}
+
+static int vsock_boundary_buf_ensure(struct virtio_vsock_sock *vvs, u32 needed)
+{
+	u32 new_alloc;
+	u8 *new_buf;
+
+	if (vvs->boundary_alloc >= needed)
+		return 0;
+
+	/* Reclaim consumed space before growing */
+	if (vvs->boundary_off) {
+		needed -= vvs->boundary_off;
+		vsock_boundary_buf_compact(vvs);
+		if (vvs->boundary_alloc >= needed)
+			return 0;
+	}
+
+	new_alloc = max(needed, vvs->boundary_alloc ? vvs->boundary_alloc * 2
+						    : VSOCK_BOUNDARY_BUF_INIT);
+	if (new_alloc > VSOCK_BOUNDARY_BUF_MAX)
+		new_alloc = VSOCK_BOUNDARY_BUF_MAX;
+	if (new_alloc < needed)
+		return -ENOMEM;
+
+	new_buf = krealloc(vvs->boundary_buf, new_alloc, GFP_ATOMIC);
+	if (!new_buf)
+		return -ENOMEM;
+
+	vvs->boundary_buf = new_buf;
+	vvs->boundary_alloc = new_alloc;
+	return 0;
+}
+
 static void virtio_transport_cancel_close_work(struct vsock_sock *vsk,
 					       bool cancel_timeout);
 static s64 virtio_transport_has_space(struct virtio_vsock_sock *vvs);
@@ -682,41 +768,74 @@ virtio_transport_seqpacket_do_peek(struct vsock_sock *vsk,
 	total = 0;
 	len = msg_data_left(msg);
 
-	skb_queue_walk(&vvs->rx_queue, skb) {
-		struct virtio_vsock_hdr *hdr;
+	skb = skb_peek(&vvs->rx_queue);
+	if (skb && VIRTIO_VSOCK_SKB_CB(skb)->has_boundary_entries) {
+		u32 msg_len, offset;
+		size_t bytes;
+		bool eor;
+		int ret;
 
-		if (total < len) {
-			size_t bytes;
+		ret = vsock_uleb_decode_boundary(
+			vvs->boundary_buf + vvs->boundary_off,
+			vvs->boundary_len - vvs->boundary_off,
+			&msg_len, &eor);
+		if (ret < 0)
+			goto unlock;
+
+		offset = VIRTIO_VSOCK_SKB_CB(skb)->offset;
+		bytes = min(len, (size_t)msg_len);
+
+		if (bytes) {
 			int err;
 
-			bytes = len - total;
-			if (bytes > skb->len)
-				bytes = skb->len;
-
 			spin_unlock_bh(&vvs->rx_lock);
-
-			/* sk_lock is held by caller so no one else can dequeue.
-			 * Unlock rx_lock since skb_copy_datagram_iter() may sleep.
-			 */
-			err = skb_copy_datagram_iter(skb, VIRTIO_VSOCK_SKB_CB(skb)->offset,
+			err = skb_copy_datagram_iter(skb, offset,
 						     &msg->msg_iter, bytes);
 			if (err)
 				return err;
-
 			spin_lock_bh(&vvs->rx_lock);
 		}
 
-		total += skb->len;
-		hdr = virtio_vsock_hdr(skb);
+		total = msg_len;
+		if (eor)
+			msg->msg_flags |= MSG_EOR;
+	} else {
+		skb_queue_walk(&vvs->rx_queue, skb) {
+			struct virtio_vsock_hdr *hdr;
 
-		if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
-			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOR)
-				msg->msg_flags |= MSG_EOR;
+			if (total < len) {
+				size_t bytes;
+				int err;
 
-			break;
+				bytes = len - total;
+				if (bytes > skb->len)
+					bytes = skb->len;
+
+				spin_unlock_bh(&vvs->rx_lock);
+
+				err = skb_copy_datagram_iter(
+					skb,
+					VIRTIO_VSOCK_SKB_CB(skb)->offset,
+					&msg->msg_iter, bytes);
+				if (err)
+					return err;
+
+				spin_lock_bh(&vvs->rx_lock);
+			}
+
+			total += skb->len;
+			hdr = virtio_vsock_hdr(skb);
+
+			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+				if (le32_to_cpu(hdr->flags) &
+				    VIRTIO_VSOCK_SEQ_EOR)
+					msg->msg_flags |= MSG_EOR;
+				break;
+			}
 		}
 	}
 
+unlock:
 	spin_unlock_bh(&vvs->rx_lock);
 
 	return total;
@@ -740,57 +859,105 @@ static int virtio_transport_seqpacket_do_dequeue(struct vsock_sock *vsk,
 	}
 
 	while (!msg_ready) {
-		struct virtio_vsock_hdr *hdr;
-		size_t pkt_len;
-
-		skb = __skb_dequeue(&vvs->rx_queue);
+		skb = skb_peek(&vvs->rx_queue);
 		if (!skb)
 			break;
-		hdr = virtio_vsock_hdr(skb);
-		pkt_len = (size_t)le32_to_cpu(hdr->len);
 
-		if (dequeued_len >= 0) {
+		if (VIRTIO_VSOCK_SKB_CB(skb)->has_boundary_entries) {
 			size_t bytes_to_copy;
+			u32 msg_len, offset;
+			bool eor;
+			int ret;
 
-			bytes_to_copy = min(user_buf_len, pkt_len);
+			ret = vsock_uleb_decode_boundary(
+				vvs->boundary_buf + vvs->boundary_off,
+				vvs->boundary_len - vvs->boundary_off,
+				&msg_len, &eor);
+			if (ret < 0)
+				break;
+			vvs->boundary_off += ret;
 
-			if (bytes_to_copy) {
+			offset = VIRTIO_VSOCK_SKB_CB(skb)->offset;
+			bytes_to_copy = min(user_buf_len, (size_t)msg_len);
+
+			if (bytes_to_copy && dequeued_len >= 0) {
 				int err;
 
-				/* sk_lock is held by caller so no one else can dequeue.
-				 * Unlock rx_lock since skb_copy_datagram_iter() may sleep.
-				 */
 				spin_unlock_bh(&vvs->rx_lock);
-
-				err = skb_copy_datagram_iter(skb, 0,
+				err = skb_copy_datagram_iter(skb, offset,
 							     &msg->msg_iter,
 							     bytes_to_copy);
-				if (err) {
-					/* Copy of message failed. Rest of
-					 * fragments will be freed without copy.
-					 */
-					dequeued_len = err;
-				} else {
-					user_buf_len -= bytes_to_copy;
-				}
-
 				spin_lock_bh(&vvs->rx_lock);
+				if (err)
+					dequeued_len = err;
+				else
+					user_buf_len -= bytes_to_copy;
 			}
 
 			if (dequeued_len >= 0)
-				dequeued_len += pkt_len;
-		}
+				dequeued_len += msg_len;
 
-		if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+			VIRTIO_VSOCK_SKB_CB(skb)->offset += msg_len;
 			msg_ready = true;
 			vvs->msg_count--;
 
-			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOR)
+			if (eor)
 				msg->msg_flags |= MSG_EOR;
-		}
 
-		virtio_transport_dec_rx_pkt(vvs, pkt_len, pkt_len);
-		kfree_skb(skb);
+			virtio_transport_dec_rx_pkt(vvs, msg_len, msg_len);
+
+			if (VIRTIO_VSOCK_SKB_CB(skb)->offset >= skb->len) {
+				__skb_unlink(skb, &vvs->rx_queue);
+				kfree_skb(skb);
+			}
+
+			if (vvs->boundary_off >= vvs->boundary_len / 2)
+				vsock_boundary_buf_compact(vvs);
+		} else {
+			struct virtio_vsock_hdr *hdr;
+			size_t pkt_len;
+
+			skb = __skb_dequeue(&vvs->rx_queue);
+			if (!skb)
+				break;
+			hdr = virtio_vsock_hdr(skb);
+			pkt_len = (size_t)le32_to_cpu(hdr->len);
+
+			if (dequeued_len >= 0) {
+				size_t bytes_to_copy;
+
+				bytes_to_copy = min(user_buf_len, pkt_len);
+
+				if (bytes_to_copy) {
+					int err;
+
+					spin_unlock_bh(&vvs->rx_lock);
+					err = skb_copy_datagram_iter(
+						skb, 0, &msg->msg_iter,
+						bytes_to_copy);
+					if (err)
+						dequeued_len = err;
+					else
+						user_buf_len -= bytes_to_copy;
+					spin_lock_bh(&vvs->rx_lock);
+				}
+
+				if (dequeued_len >= 0)
+					dequeued_len += pkt_len;
+			}
+
+			if (le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM) {
+				msg_ready = true;
+				vvs->msg_count--;
+
+				if (le32_to_cpu(hdr->flags) &
+				    VIRTIO_VSOCK_SEQ_EOR)
+					msg->msg_flags |= MSG_EOR;
+			}
+
+			virtio_transport_dec_rx_pkt(vvs, pkt_len, pkt_len);
+			kfree_skb(skb);
+		}
 	}
 
 	spin_unlock_bh(&vvs->rx_lock);
@@ -1132,6 +1299,7 @@ void virtio_transport_destruct(struct vsock_sock *vsk)
 
 	virtio_transport_cancel_close_work(vsk, true);
 
+	kfree(vvs->boundary_buf);
 	kfree(vvs);
 	vsk->trans = NULL;
 }
@@ -1224,6 +1392,11 @@ static void virtio_transport_remove_sock(struct vsock_sock *vsk)
 	 * removing it.
 	 */
 	__skb_queue_purge(&vvs->rx_queue);
+	kfree(vvs->boundary_buf);
+	vvs->boundary_buf = NULL;
+	vvs->boundary_len = 0;
+	vvs->boundary_alloc = 0;
+	vvs->boundary_off = 0;
 	vsock_remove_sock(vsk);
 }
 
@@ -1395,23 +1568,62 @@ virtio_transport_recv_enqueue(struct vsock_sock *vsk,
 	    !skb_is_nonlinear(skb)) {
 		struct virtio_vsock_hdr *last_hdr;
 		struct sk_buff *last_skb;
+		bool last_has_eom;
+		bool has_eom;
 
 		last_skb = skb_peek_tail(&vvs->rx_queue);
 		last_hdr = virtio_vsock_hdr(last_skb);
+		last_has_eom = le32_to_cpu(last_hdr->flags) & VIRTIO_VSOCK_SEQ_EOM;
+		has_eom = le32_to_cpu(hdr->flags) & VIRTIO_VSOCK_SEQ_EOM;
 
-		/* If there is space in the last packet queued, we copy the
-		 * new packet in its buffer. We avoid this if the last packet
-		 * queued has VIRTIO_VSOCK_SEQ_EOM set, because this is
-		 * delimiter of SEQPACKET message, so 'pkt' is the first packet
-		 * of a new message.
-		 */
-		if (skb->len < skb_tailroom(last_skb) &&
-		    !(le32_to_cpu(last_hdr->flags) & VIRTIO_VSOCK_SEQ_EOM)) {
-			memcpy(skb_put(last_skb, skb->len), skb->data, skb->len);
-			free_pkt = true;
-			last_hdr->flags |= hdr->flags;
-			le32_add_cpu(&last_hdr->len, len);
-			goto out;
+		if (skb->len < skb_tailroom(last_skb)) {
+			if (!last_has_eom) {
+				/* Same-message coalescing (existing path) */
+				memcpy(skb_put(last_skb, skb->len),
+				       skb->data, skb->len);
+				free_pkt = true;
+				last_hdr->flags |= hdr->flags;
+				le32_add_cpu(&last_hdr->len, len);
+				goto out;
+			}
+
+			/* Cross-EOM: coalesce complete messages into one skb,
+			 * recording message boundaries in a compact BER buffer.
+			 * Only when incoming packet also has EOM (complete msg).
+			 */
+			if (has_eom && !sk_psock(sk_vsock(vsk))) {
+				bool prev_eor, cur_eor;
+				u8 tmp[12];
+				int n = 0;
+
+				cur_eor = le32_to_cpu(hdr->flags) &
+					  VIRTIO_VSOCK_SEQ_EOR;
+
+				if (!VIRTIO_VSOCK_SKB_CB(last_skb)->has_boundary_entries) {
+					u32 prev_len = le32_to_cpu(last_hdr->len);
+
+					prev_eor = le32_to_cpu(last_hdr->flags) &
+						   VIRTIO_VSOCK_SEQ_EOR;
+					n += vsock_uleb_encode_boundary(
+						tmp + n, prev_len, prev_eor);
+				}
+				n += vsock_uleb_encode_boundary(
+					tmp + n, len, cur_eor);
+
+				if (!vsock_boundary_buf_ensure(
+					    vvs, vvs->boundary_len + n)) {
+					memcpy(vvs->boundary_buf +
+					       vvs->boundary_len, tmp, n);
+					vvs->boundary_len += n;
+					VIRTIO_VSOCK_SKB_CB(last_skb)->has_boundary_entries = true;
+					memcpy(skb_put(last_skb, skb->len),
+					       skb->data, skb->len);
+					free_pkt = true;
+					last_hdr->flags |= hdr->flags;
+					le32_add_cpu(&last_hdr->len, len);
+					goto out;
+				}
+			}
 		}
 	}
 
#22 lore raw
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Stefano Garzarella
Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines · Stefano Garzarella
Quoted 1 line · Michael S. Tsirkin

On Thu, May 07, 2026 at 11:09:47AM +0200, Stefano Garzarella wrote:

[...]

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.

Okay, I somewhat understand, but clearly this isn't net material

I doubt we have many other options given reverting the regression was ruled out.

As Eric pointed out, we can't revert it.

Quoted 9 lines
Quoted 6 lines

so for now
I think the best thing to do is to merge the fixup I sent (or something similar):
https://lore.kernel.org/netdev/[email protected]/

I reviewed that one, problem is it's a spec violation/change that we'll have to support forever.

I have a few points to make on this, but let's discuss them there.

Quoted 8 lines
Quoted 6 lines

This is a major change that should be merged with more caution. Could this have too much of an impact on performance?

Thanks,
Stefano

It's really a POC, real patch is left as an excersise for the reader :).

eheh, I see, but honestly, this overcomplication scares me. I'll try to think it over.

Quoted 6 lines

The correct approach IMHO is to only start using this when we wasted a lot of memory on small packets.

For example, if sum(truesize) >= buf size.

then we'll not see a perf impact unless it's already pathological.

Agree on this, which is similar to what I'm doing in that patch. Reducing the advertised buf_alloc only in pathological cases (e.g. overhead > buf_alloc).

Stefano

#23 lore raw
Quoted 24 lines · Stefano Garzarella
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Stefano Garzarella
Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines · Stefano Garzarella
Quoted 1 line · Michael S. Tsirkin

On Thu, May 07, 2026 at 11:09:47AM +0200, Stefano Garzarella wrote:

[...]

Quoted 14 lines
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Okay, I somewhat understand, but clearly this isn't net material

I doubt we have many other options given reverting the regression was ruled out.

As Eric pointed out, we can't revert it.

Could there be an option to disable the mitigation in guests, for the situation where the host is trusted? There are VMMs that implement AF_VSOCK in userspace with a backing AF_UNIX socket.

Quoted 26 lines
Quoted 7 lines
Quoted 4 lines

so for now
I think the best thing to do is to merge the fixup I sent (or something similar):
https://lore.kernel.org/netdev/[email protected]/

I reviewed that one, problem is it's a spec violation/change that we'll have to support forever.

I have a few points to make on this, but let's discuss them there.

Quoted 9 lines
Quoted 6 lines

This is a major change that should be merged with more caution. Could this have too much of an impact on performance?

Thanks,
Stefano

It's really a POC, real patch is left as an excersise for the reader :).

eheh, I see, but honestly, this overcomplication scares me. I'll try to think it over.

Quoted 3 lines

The correct approach IMHO is to only start using this when we wasted a lot of memory on small packets.

Quoted 7 lines
Quoted 3 lines

For example, if sum(truesize) >= buf size.

then we'll not see a perf impact unless it's already pathological.

Agree on this, which is similar to what I'm doing in that patch. Reducing the advertised buf_alloc only in pathological cases (e.g. overhead > buf_alloc).

This isn't enough to prevent data loss due to race conditions.

I'm CCing the virtio-comment list and a few others.

Right now, any application that needs to send massive amount of data over a vsock is simply broken. This isn't just theoretical. It's causing real-world problems for users of Waypipe. Waypipe forwards Wayland protocol messages over AF_VSOCK, so it can send a large amount of traffic over the socket. See https://gitlab.freedesktop.org/mstoceckl/waypipe/work_items/165.

If one is willing to mutate the ring buffer in-place, or to maintain an auxiliary counter, it's possible to store all messages with bounded (in practice) overhead. Specifically:

- If the first byte of a block of data is nonzero, it's a variable-length length. 1 byte for messages less than 128 bytes.

- If the first byte of a block of data is zero, the subsequent bytes are a variable-length counter that stores the number of consecutive zero-byte messages.

That adds a lot of complexity, which is very unfortunate for something that needs to be backported to stable kernels. I also suspect it requires all access to the ring buffer to take a lock rather than being lock-free. But it's the only approach that I can think of that can work with the current spec.

Could there at least be a normative note stating that drivers and devices should treat each message as consuming 1024 bytes + the size of the message itself, and warning that anything that doesn't is going to be broken in practice?

I'm CCing Val Packet (of Invisible Things Lab) and Alyssa Ross (of Spectrum) because both of them are working on systems that rely critically on vsock.

Signature · 2 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
#24 lore raw
Quoted 98 lines · Demi Marie Obenour
Quoted 24 lines · Stefano Garzarella
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Stefano Garzarella
Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines · Stefano Garzarella
Quoted 1 line · Michael S. Tsirkin

On Thu, May 07, 2026 at 11:09:47AM +0200, Stefano Garzarella wrote:

[...]

Quoted 14 lines
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Okay, I somewhat understand, but clearly this isn't net material

I doubt we have many other options given reverting the regression was ruled out.

As Eric pointed out, we can't revert it.

Could there be an option to disable the mitigation in guests, for the situation where the host is trusted? There are VMMs that implement AF_VSOCK in userspace with a backing AF_UNIX socket.

Quoted 26 lines
Quoted 7 lines
Quoted 4 lines

so for now
I think the best thing to do is to merge the fixup I sent (or something similar):
https://lore.kernel.org/netdev/[email protected]/

I reviewed that one, problem is it's a spec violation/change that we'll have to support forever.

I have a few points to make on this, but let's discuss them there.

Quoted 9 lines
Quoted 6 lines

This is a major change that should be merged with more caution. Could this have too much of an impact on performance?

Thanks,
Stefano

It's really a POC, real patch is left as an excersise for the reader :).

eheh, I see, but honestly, this overcomplication scares me. I'll try to think it over.

Quoted 3 lines

The correct approach IMHO is to only start using this when we wasted a lot of memory on small packets.

Quoted 7 lines
Quoted 3 lines

For example, if sum(truesize) >= buf size.

then we'll not see a perf impact unless it's already pathological.

Agree on this, which is similar to what I'm doing in that patch. Reducing the advertised buf_alloc only in pathological cases (e.g. overhead > buf_alloc).

This isn't enough to prevent data loss due to race conditions.

I'm CCing the virtio-comment list and a few others.

Right now, any application that needs to send massive amount of data over a vsock is simply broken. This isn't just theoretical. It's causing real-world problems for users of Waypipe. Waypipe forwards Wayland protocol messages over AF_VSOCK, so it can send a large amount of traffic over the socket. See https://gitlab.freedesktop.org/mstoceckl/waypipe/work_items/165.

If one is willing to mutate the ring buffer in-place, or to maintain an auxiliary counter, it's possible to store all messages with bounded (in practice) overhead. Specifically:

- If the first byte of a block of data is nonzero, it's a variable-length length. 1 byte for messages less than 128 bytes.

- If the first byte of a block of data is zero, the subsequent bytes are a variable-length counter that stores the number of consecutive zero-byte messages.

That adds a lot of complexity, which is very unfortunate for something that needs to be backported to stable kernels. I also suspect it requires all access to the ring buffer to take a lock rather than being lock-free. But it's the only approach that I can think of that can work with the current spec.

Quoted 8 lines

Could there at least be a normative note stating that drivers and devices should treat each message as consuming 1024 bytes + the size of the message itself, and warning that anything that doesn't is going to be broken in practice?

I'm CCing Val Packet (of Invisible Things Lab) and Alyssa Ross (of Spectrum) because both of them are working on systems that rely critically on vsock.

Update: I see that patches have been upstreamed (with CC: stable) that reset the connection instead of data loss.

Signature · 2 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
#25 lore raw
Quoted 29 lines · Demi Marie Obenour
Quoted 27 lines · Demi Marie Obenour
Quoted 26 lines · Stefano Garzarella
Quoted 25 lines · Michael S. Tsirkin
Quoted 23 lines · Stefano Garzarella
Quoted 3 lines · Michael S. Tsirkin
Quoted 2 lines · Stefano Garzarella
Quoted 1 line · Michael S. Tsirkin

On Thu, May 07, 2026 at 11:09:47AM +0200, Stefano Garzarella wrote:

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

For now, we're already doing something:
merging the skuffs if they don't have EOM set.

Right that's good. You could go further and merge with EOM too if you stick the info about message boundaries somewhere else.

Quoted 7 lines
Quoted 3 lines

This adds a lot of complexity IMO, but we can try.

Do you have something in mind?

BER is clearly overkill but here's a POC that claude made for me, just to give u an idea. It's clearly has a ton of issues, for example I dislike how GFP_ATOMIC is handled.

Okay, I somewhat understand, but clearly this isn't net material

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

I doubt we have many other options given reverting the regression was ruled out.

As Eric pointed out, we can't revert it.

Could there be an option to disable the mitigation in guests, for the situation where the host is trusted? There are VMMs that implement AF_VSOCK in userspace with a backing AF_UNIX socket.

Here the main problem was an attack to the host from the guest.

Can the application set the max (IIRC 4G) of SO_VM_SOCKETS_BUFFER_SIZE ?

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 8 lines
Quoted 5 lines

so for now
I think the best thing to do is to merge the fixup I sent (or something similar):
https://lore.kernel.org/netdev/[email protected]/

I reviewed that one, problem is it's a spec violation/change that we'll have to support forever.

I have a few points to make on this, but let's discuss them there.

Quoted 9 lines
Quoted 6 lines

This is a major change that should be merged with more caution. Could this have too much of an impact on performance?

Thanks,
Stefano

It's really a POC, real patch is left as an excersise for the reader :).

eheh, I see, but honestly, this overcomplication scares me. I'll try to think it over.

Quoted 5 lines

The correct approach IMHO is to only start using this when we wasted a lot of memory on small packets.

For example, if sum(truesize) >= buf size.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

then we'll not see a perf impact unless it's already pathological.

Agree on this, which is similar to what I'm doing in that patch. Reducing the advertised buf_alloc only in pathological cases (e.g. overhead > buf_alloc).

This isn't enough to prevent data loss due to race conditions.

Yep, we didn't go in that direction.

Quoted 21 lines
Quoted 21 lines

I'm CCing the virtio-comment list and a few others.

Right now, any application that needs to send massive amount of data over a vsock is simply broken. This isn't just theoretical. It's causing real-world problems for users of Waypipe. Waypipe forwards Wayland protocol messages over AF_VSOCK, so it can send a large amount of traffic over the socket. See https://gitlab.freedesktop.org/mstoceckl/waypipe/work_items/165.

If one is willing to mutate the ring buffer in-place, or to maintain an auxiliary counter, it's possible to store all messages with bounded (in practice) overhead. Specifically:

- If the first byte of a block of data is nonzero, it's a variable-length length. 1 byte for messages less than 128 bytes.

- If the first byte of a block of data is zero, the subsequent bytes are a variable-length counter that stores the number of consecutive zero-byte messages.

Quoted 5 lines
Quoted 5 lines

That adds a lot of complexity, which is very unfortunate for something that needs to be backported to stable kernels. I also suspect it requires all access to the ring buffer to take a lock rather than being lock-free. But it's the only approach that I can think of that can work with the current spec.

Yep, we were discussing something like that in this thread, but we are still working on that.
That said, any help on preparing patches for this would be more than welcome.

Quoted 5 lines
Quoted 5 lines

Could there at least be a normative note stating that drivers and devices should treat each message as consuming 1024 bytes + the size of the message itself, and warning that anything that doesn't is going to be broken in practice?

We would like to fix the issue. This is a transitive situation.

Quoted 7 lines
Quoted 4 lines

I'm CCing Val Packet (of Invisible Things Lab) and Alyssa Ross (of Spectrum) because both of them are working on systems that rely critically on vsock.

Update: I see that patches have been upstreamed (with CC: stable) that reset the connection instead of data loss.

Yep, hope that helps the situation. But again, can setting the max SO_VM_SOCKETS_BUFFER_SIZE helps those applications?

Thanks,
Stefano

#26 lore raw
In reply to Eric Dumazet
Quoted 3 lines · Eric Dumazet

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

We can if we copy.

Quoted 1 line

This kind of contract is broken.

#27 lore raw
Quoted 7 lines · Michael S. Tsirkin
Quoted 3 lines · Eric Dumazet

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

We can if we copy.

You mean, ignore VIRTIO_VSOCK_SEQ_EOM?

#28 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 7 lines · Michael S. Tsirkin
Quoted 3 lines · Eric Dumazet

There is always a discrepancy between skb->len and skb->truesize. You will not be able to announce a 1MB window, and accept one milliion skb of 1-byte each.

We can if we copy.

You mean, ignore VIRTIO_VSOCK_SEQ_EOM?

No I mean saving it in a compact form.
If packet boundaries are the only concern, the overhead needn't exceed 50% even for 1 byte messages. asn.1 ber, for an over-engineered example?