Skip to contentSkip to conversation

fixes for vhost_net

PATCH Superseded wangyunjian 4 participants 25 messages Dec 15, 2020 – Dec 23, 2020

Subsystems (4)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover fixes for vhost_net

From: Yunjian Wang <[email protected]>

This series include two fixes patches for vhost_net.

Yunjian Wang (2):
vhost_net: fix ubuf refcount incorrectly when sendmsg fails
vhost_net: fix high cpu load when sendmsg fails

 drivers/vhost/net.c | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/2 vhost_net: fix ubuf refcount incorrectly when sendmsg fails

From: Yunjian Wang <[email protected]>

Currently the vhost_zerocopy_callback() maybe be called to decrease the refcount when sendmsg fails in tun. The error handling in vhost handle_tx_zerocopy() will try to decrease the same refcount again. This is wrong. To fix this issue, we only call vhost_net_ubuf_put() when vq->heads[nvq->desc].len == VHOST_DMA_IN_PROGRESS.

Fixes 4477138fa0ae ("tun: properly test for IFF_UP")
Fixes 90e33d459407 ("tun: enable napi_gro_frags() for TUN/TAP driver")
---
 drivers/vhost/net.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/vhost/net.c +3 -3
index 531a00d703cd..c8784dfafdd7 100644
@@ -863,6 +863,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 	size_t len, total_len = 0;
 	int err;
 	struct vhost_net_ubuf_ref *ubufs;
+	struct ubuf_info *ubuf;
 	bool zcopy_used;
 	int sent_pkts = 0;
 
@@ -895,9 +896,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 
 		/* use msg_control to pass vhost zerocopy ubuf info to skb */
 		if (zcopy_used) {
-			struct ubuf_info *ubuf;
 			ubuf = nvq->ubuf_info + nvq->upend_idx;
-
 			vq->heads[nvq->upend_idx].id = cpu_to_vhost32(vq, head);
 			vq->heads[nvq->upend_idx].len = VHOST_DMA_IN_PROGRESS;
 			ubuf->callback = vhost_zerocopy_callback;
@@ -927,7 +926,8 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
-				vhost_net_ubuf_put(ubufs);
+				if (vq->heads[ubuf->desc].len == VHOST_DMA_IN_PROGRESS)
+					vhost_net_ubuf_put(ubufs);
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
Signature · 1 line
2.23.0
#2 lore raw

Re: [PATCH net 1/2] vhost_net: fix ubuf refcount incorrectly when sendmsg fails

Quoted 9 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently the vhost_zerocopy_callback() maybe be called to decrease the refcount when sendmsg fails in tun. The error handling in vhost handle_tx_zerocopy() will try to decrease the same refcount again. This is wrong. To fix this issue, we only call vhost_net_ubuf_put() when vq->heads[nvq->desc].len == VHOST_DMA_IN_PROGRESS.

Quoted 4 lines
Fixes 4477138fa0ae ("tun: properly test for IFF_UP")
Fixes 90e33d459407 ("tun: enable napi_gro_frags() for TUN/TAP driver")

Patch looks good to me. Thanks.

But I think the right Fixes tag would be

Fixes 0690899b4d45 ("tun: experimental zero copy tx support")

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#3 lore raw

RE: [PATCH net 1/2] vhost_net: fix ubuf refcount incorrectly when sendmsg fails

Quoted 28 lines

-----Original Message-----
From: Willem de Bruijn [mailto:[email protected]]
Sent: Tuesday, December 15, 2020 10:46 AM
To: wangyunjian <[email protected]>
Cc: Network Development <[email protected]>; Michael S. Tsirkin <[email protected]>; Jason Wang <[email protected]>; Willem de Bruijn <[email protected]>; [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net 1/2] vhost_net: fix ubuf refcount incorrectly when sendmsg fails

Quoted 13 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently the vhost_zerocopy_callback() maybe be called to decrease the refcount when sendmsg fails in tun. The error handling in vhost handle_tx_zerocopy() will try to decrease the same refcount again. This is wrong. To fix this issue, we only call vhost_net_ubuf_put() when vq->heads[nvq->desc].len == VHOST_DMA_IN_PROGRESS.

Fixes 4477138fa0ae ("tun: properly test for IFF_UP")
Fixes 90e33d459407 ("tun: enable napi_gro_frags() for TUN/TAP

driver")

Quoted 7 lines
Quoted 1 line

Patch looks good to me. Thanks.

But I think the right Fixes tag would be

Fixes 0690899b4d45 ("tun: experimental zero copy tx support")

OK, thanks for the suggestion. I will fix it in next version.

#4 lore raw

PATCH 2/2 vhost_net: fix high cpu load when sendmsg fails

In reply to wangyunjian

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

---
 drivers/vhost/net.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c +9 -12
index c8784dfafdd7..f966592d8900 100644
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
 				msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
 			vhost_discard_vq_desc(vq, 1);
 			vhost_net_enable_vq(net, vq);
 			break;
-		}
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err < 0 || err != len))
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
 done:
 		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
 		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 			msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
 		}
 		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
 		if (!zcopy_used)
 			vhost_add_used_and_signal(&net->dev, vq, head, 0);
 		else
Signature · 1 line
2.23.0
#5 lore raw

Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 20 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

---
  drivers/vhost/net.c | 21 +++++++++------------
  1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c +0 -0
index c8784dfafdd7..f966592d8900 100644
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
  				msg.msg_flags &= ~MSG_MORE;

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 836

Quoted 10 lines
drivers/vhost/net.c +1 -3
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
  		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
  			vhost_discard_vq_desc(vq, 1);
  			vhost_net_enable_vq(net, vq);
  			break;
-		}

As I've pointed out in last version. If you don't discard descriptor, you probably need to add the head to used ring. Otherwise this descriptor will be always inflight that may confuse drivers.

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 835

Quoted 4 lines
drivers/vhost/net.c +1 -3
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err < 0 || err != len))

It looks to me err != len covers err < 0.

Thanks

Quoted 22 lines
drivers/vhost/net.c +4 -4
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
  done:
  		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
  		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
  			msg.msg_flags &= ~MSG_MORE;
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
  		err = sock->ops->sendmsg(sock, &msg, len);
  		if (unlikely(err < 0)) {
  			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
  					% UIO_MAXIOV;
  			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) · line 940

Quoted 10 lines
drivers/vhost/net.c +3 -2
+				break;
+			}
  		}
  		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
  		if (!zcopy_used)
  			vhost_add_used_and_signal(&net->dev, vq, head, 0);
  		else

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#6 lore raw

RE: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 38 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Tuesday, December 15, 2020 12:10 PM
To: wangyunjian <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net,

Quoted 19 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

---
  drivers/vhost/net.c | 21 +++++++++------------
  1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c index renamed from drivers/vhost/net.c +0 -0

c8784dfafdd7..f966592d8900 100644

drivers/vhost/net.c +0 -0
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,

struct socket *sock)

Quoted 3 lines
drivers/vhost/net.c +0 -0
  				msg.msg_flags &= ~MSG_MORE;
  		}
 
Quoted 13 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 836

Quoted 8 lines
drivers/vhost/net.c +1 -3
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
  		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
  			vhost_discard_vq_desc(vq, 1);
  			vhost_net_enable_vq(net, vq);
  			break;
-		}

As I've pointed out in last version. If you don't discard descriptor, you probably need to add the head to used ring. Otherwise this descriptor will be always inflight that may confuse drivers.

Sorry for missing the comment.

After deleting discard descriptor and break, the next processing will be the same as the normal success of sendmsg(), and vhost_zerocopy_signal_used() or vhost_add_used_and_signal() method will be called to add the head to used ring.

Thanks

Quoted 9 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 835

Quoted 6 lines
drivers/vhost/net.c +1 -3
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err < 0 || err != len))

It looks to me err != len covers err < 0.

OK

Quoted 40 lines

Thanks

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 6 lines
drivers/vhost/net.c +2 -0
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err,
+len);
  done:
  		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
  		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 8 lines
drivers/vhost/net.c +0 -1
  			msg.msg_flags &= ~MSG_MORE;
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
  		err = sock->ops->sendmsg(sock, &msg, len);
  		if (unlikely(err < 0)) {
  			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

Quoted 20 lines
drivers/vhost/net.c +7 -5
  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
  					% UIO_MAXIOV;
  			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
  		}
  		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err,
+len);
  		if (!zcopy_used)
  			vhost_add_used_and_signal(&net->dev, vq, head, 0);
  		else
#7 lore raw

Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

----- Original Message -----

Quoted 54 lines
Quoted 39 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Tuesday, December 15, 2020 12:10 PM
To: wangyunjian <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net,

Quoted 19 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

---
  drivers/vhost/net.c | 21 +++++++++------------
  1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c index renamed from drivers/vhost/net.c +0 -0

c8784dfafdd7..f966592d8900 100644

drivers/vhost/net.c +0 -0
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,

struct socket *sock)

Quoted 3 lines
drivers/vhost/net.c +0 -0
  				msg.msg_flags &= ~MSG_MORE;
  		}
 
Quoted 14 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 836

Quoted 8 lines
drivers/vhost/net.c +1 -3
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
  		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
  			vhost_discard_vq_desc(vq, 1);
  			vhost_net_enable_vq(net, vq);
  			break;
-		}

As I've pointed out in last version. If you don't discard descriptor, you probably
need to add the head to used ring. Otherwise this descriptor will be always inflight that may confuse drivers.

Quoted 7 lines

Sorry for missing the comment.

After deleting discard descriptor and break, the next processing will be the same
as the normal success of sendmsg(), and vhost_zerocopy_signal_used() or vhost_add_used_and_signal() method will be called to add the head to used ring.

It's the next head not the one that contains the buggy packet?

Thanks

Quoted 46 lines

Thanks

Quoted 9 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 835

Quoted 6 lines
drivers/vhost/net.c +1 -3
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err < 0 || err != len))

It looks to me err != len covers err < 0.

OK

Quoted 32 lines

Thanks

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 6 lines
drivers/vhost/net.c +2 -0
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err,
+len);
  done:
  		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
  		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 8 lines
drivers/vhost/net.c +0 -1
  			msg.msg_flags &= ~MSG_MORE;
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
  		err = sock->ops->sendmsg(sock, &msg, len);
  		if (unlikely(err < 0)) {
  			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

Quoted 12 lines
drivers/vhost/net.c +5 -3
  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
  					% UIO_MAXIOV;
  			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
  		}
Quoted 10 lines
Quoted 8 lines

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) · line 940

Quoted 8 lines
drivers/vhost/net.c +2 -2
  		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err,
+len);
  		if (!zcopy_used)
  			vhost_add_used_and_signal(&net->dev, vq, head, 0);
  		else

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#8 lore raw

RE: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 73 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, December 16, 2020 1:57 PM
To: wangyunjian <[email protected]>
Cc: [email protected]; [email protected]; willemdebruijn kernel <[email protected]>; [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

----- Original Message -----

Quoted 15 lines
Quoted 15 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Tuesday, December 15, 2020 12:10 PM
To: wangyunjian <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 4 lines
Quoted 4 lines · wangyunjian
Quoted 3 lines

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when

sendmsg

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

---
  drivers/vhost/net.c | 21 +++++++++------------
  1 file changed, 9 insertions(+), 12 deletions(-)
Quoted 5 lines
Quoted 5 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net

Quoted 5 lines
drivers/vhost/net.c index renamed from drivers/vhost/net.c +0 -0

c8784dfafdd7..f966592d8900 100644

drivers/vhost/net.c +0 -0
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net

*net,

Quoted 5 lines
Quoted 5 lines

struct socket *sock)

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock)

Quoted 4 lines
drivers/vhost/net.c +0 -1
  				msg.msg_flags &= ~MSG_MORE;
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS?

*/

Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
drivers/vhost/net.c +1 -1
  		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
  			vhost_discard_vq_desc(vq, 1);
  			vhost_net_enable_vq(net, vq);
Quoted 8 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 836

Quoted 2 lines
drivers/vhost/net.c +0 -1
  			break;
-		}

As I've pointed out in last version. If you don't discard descriptor, you probably
need to add the head to used ring. Otherwise this descriptor will be always inflight that may confuse drivers.

Sorry for missing the comment.

Quoted 7 lines
Quoted 5 lines

After deleting discard descriptor and break, the next processing will be the same
as the normal success of sendmsg(), and vhost_zerocopy_signal_used() or vhost_add_used_and_signal() method will be called to add the head to used ring.

It's the next head not the one that contains the buggy packet?

In the modified code logic, the head added to used ring is exectly the one that contains the buggy packet.

Thanks

Quoted 63 lines

Thanks

Quoted 12 lines

Thanks

Quoted 9 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 835

Quoted 6 lines
drivers/vhost/net.c +1 -3
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err < 0 || err != len))

It looks to me err != len covers err < 0.

Quoted 7 lines

OK

Quoted 5 lines

Thanks

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net · line 836

Quoted 1 line
drivers/vhost/net.c +1 -0
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n",

err,

Quoted 10 lines
Quoted 10 lines

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 5 lines
drivers/vhost/net.c +1 -0
+len);
  done:
  		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
  		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock)

Quoted 4 lines
drivers/vhost/net.c +0 -1
  			msg.msg_flags &= ~MSG_MORE;
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS?

*/

Quoted 4 lines
Quoted 4 lines

drivers/vhost/net.c static void handle_tx_zerocopy(struct

Quoted 4 lines
drivers/vhost/net.c +0 -0
  		err = sock->ops->sendmsg(sock, &msg, len);
  		if (unlikely(err < 0)) {
  			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct

vhost_net

Quoted 17 lines
Quoted 17 lines

*net, struct socket *sock)

Quoted 6 lines
drivers/vhost/net.c +0 -3
  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
  					% UIO_MAXIOV;
  			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)

Quoted 10 lines
drivers/vhost/net.c +6 -2
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
  		}
  		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n",

err,

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
drivers/vhost/net.c +1 -0
+len);
  		if (!zcopy_used)
  			vhost_add_used_and_signal(&net->dev, vq, head, 0);
  		else
#9 lore raw

Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

----- Original Message -----

Quoted 86 lines
Quoted 86 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, December 16, 2020 1:57 PM
To: wangyunjian <[email protected]>
Cc: [email protected]; [email protected]; willemdebruijn kernel <[email protected]>; [email protected];
Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

----- Original Message -----

Quoted 15 lines
Quoted 15 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Tuesday, December 15, 2020 12:10 PM
To: wangyunjian <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 4 lines
Quoted 4 lines · wangyunjian
Quoted 3 lines

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when

sendmsg

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

fails. When the worker wakes up again, we'll meet the same error. This
will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description
and don't drop packet.

---
  drivers/vhost/net.c | 21 +++++++++------------
  1 file changed, 9 insertions(+), 12 deletions(-)
Quoted 5 lines
Quoted 5 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net

Quoted 5 lines
drivers/vhost/net.c index renamed from drivers/vhost/net.c +0 -0

c8784dfafdd7..f966592d8900 100644

drivers/vhost/net.c +0 -0
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net

*net,

Quoted 5 lines
Quoted 5 lines

struct socket *sock)

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock)

Quoted 4 lines
drivers/vhost/net.c +0 -1
  				msg.msg_flags &= ~MSG_MORE;
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS?

*/

Quoted 24 lines
Quoted 9 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 836

Quoted 7 lines
drivers/vhost/net.c +1 -2
  		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
  			vhost_discard_vq_desc(vq, 1);
  			vhost_net_enable_vq(net, vq);
  			break;
-		}
Quoted 6 lines

As I've pointed out in last version. If you don't discard descriptor, you
probably
need to add the head to used ring. Otherwise this descriptor will be always
inflight that may confuse drivers.

Sorry for missing the comment.

After deleting discard descriptor and break, the next processing will be the
same
as the normal success of sendmsg(), and vhost_zerocopy_signal_used() or vhost_add_used_and_signal() method will be called to add the head to used ring.

Quoted 4 lines
Quoted 1 line

It's the next head not the one that contains the buggy packet?

In the modified code logic, the head added to used ring is exectly the one that contains the buggy packet.

-ENOTEA :( You're right, I misread the code.

Thanks

Quoted 68 lines

Thanks

Quoted 63 lines

Thanks

Quoted 12 lines

Thanks

Quoted 9 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 835

Quoted 6 lines
drivers/vhost/net.c +1 -3
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err < 0 || err != len))

It looks to me err != len covers err < 0.

Quoted 7 lines

OK

Quoted 5 lines

Thanks

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net · line 836

Quoted 1 line
drivers/vhost/net.c +1 -0
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n",

err,

Quoted 10 lines
Quoted 10 lines

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 5 lines
drivers/vhost/net.c +1 -0
+len);
  done:
  		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
  		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock)

Quoted 4 lines
drivers/vhost/net.c +0 -1
  			msg.msg_flags &= ~MSG_MORE;
  		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS?

*/

Quoted 4 lines
Quoted 4 lines

drivers/vhost/net.c static void handle_tx_zerocopy(struct

Quoted 4 lines
drivers/vhost/net.c +0 -0
  		err = sock->ops->sendmsg(sock, &msg, len);
  		if (unlikely(err < 0)) {
  			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct

vhost_net

Quoted 17 lines
Quoted 17 lines

*net, struct socket *sock)

Quoted 6 lines
drivers/vhost/net.c +0 -3
  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
  					% UIO_MAXIOV;
  			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)

Quoted 10 lines
drivers/vhost/net.c +6 -2
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
  		}
  		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n",

err,

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
drivers/vhost/net.c +1 -0
+len);
  		if (!zcopy_used)
  			vhost_add_used_and_signal(&net->dev, vq, head, 0);
  		else

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#10 lore raw

v2 cover fixes for vhost_net

In reply to wangyunjian

From: Yunjian Wang <[email protected]>

This series include two fixes patches for vhost_net.

---
v2:
* update patch 1/2 Fixes tag suggested by Willem de Bruijn
* update patch 2/2 code styles suggested by Jason Wang

Yunjian Wang (2):
vhost_net: fix ubuf refcount incorrectly when sendmsg fails
vhost_net: fix high cpu load when sendmsg fails

 drivers/vhost/net.c | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)
Signature · 1 line
2.23.0
#11 lore raw

PATCH v2 1/2 vhost_net: fix ubuf refcount incorrectly when sendmsg fails

From: Yunjian Wang <[email protected]>

Currently the vhost_zerocopy_callback() maybe be called to decrease the refcount when sendmsg fails in tun. The error handling in vhost handle_tx_zerocopy() will try to decrease the same refcount again. This is wrong. To fix this issue, we only call vhost_net_ubuf_put() when vq->heads[nvq->desc].len == VHOST_DMA_IN_PROGRESS.

Fixes 0690899b4d45 ("tun: experimental zero copy tx support")
---
 drivers/vhost/net.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/vhost/net.c +3 -3
index 531a00d703cd..c8784dfafdd7 100644
@@ -863,6 +863,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 	size_t len, total_len = 0;
 	int err;
 	struct vhost_net_ubuf_ref *ubufs;
+	struct ubuf_info *ubuf;
 	bool zcopy_used;
 	int sent_pkts = 0;
 
@@ -895,9 +896,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 
 		/* use msg_control to pass vhost zerocopy ubuf info to skb */
 		if (zcopy_used) {
-			struct ubuf_info *ubuf;
 			ubuf = nvq->ubuf_info + nvq->upend_idx;
-
 			vq->heads[nvq->upend_idx].id = cpu_to_vhost32(vq, head);
 			vq->heads[nvq->upend_idx].len = VHOST_DMA_IN_PROGRESS;
 			ubuf->callback = vhost_zerocopy_callback;
@@ -927,7 +926,8 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
-				vhost_net_ubuf_put(ubufs);
+				if (vq->heads[ubuf->desc].len == VHOST_DMA_IN_PROGRESS)
+					vhost_net_ubuf_put(ubufs);
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
Signature · 1 line
2.23.0
Acked-by on v2 1/2
#12 lore raw

Re: [PATCH net v2 1/2] vhost_net: fix ubuf refcount incorrectly when sendmsg fails

Quoted 9 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently the vhost_zerocopy_callback() maybe be called to decrease the refcount when sendmsg fails in tun. The error handling in vhost handle_tx_zerocopy() will try to decrease the same refcount again. This is wrong. To fix this issue, we only call vhost_net_ubuf_put() when vq->heads[nvq->desc].len == VHOST_DMA_IN_PROGRESS.

Quoted 3 lines
Fixes 0690899b4d45 ("tun: experimental zero copy tx support")

for next time: it's not customary to have an empty line between Fixes and Signed-off-by
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

maintainer Acked-by on v2 1/2
#13 lore raw

Re: [PATCH net v2 1/2] vhost_net: fix ubuf refcount incorrectly when sendmsg fails

In reply to wangyunjian
Quoted 11 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently the vhost_zerocopy_callback() maybe be called to decrease the refcount when sendmsg fails in tun. The error handling in vhost handle_tx_zerocopy() will try to decrease the same refcount again. This is wrong. To fix this issue, we only call vhost_net_ubuf_put() when vq->heads[nvq->desc].len == VHOST_DMA_IN_PROGRESS.

Fixes 0690899b4d45 ("tun: experimental zero copy tx support")
Quoted 38 lines
---
 drivers/vhost/net.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/vhost/net.c +3 -3
index 531a00d703cd..c8784dfafdd7 100644
@@ -863,6 +863,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 	size_t len, total_len = 0;
 	int err;
 	struct vhost_net_ubuf_ref *ubufs;
+	struct ubuf_info *ubuf;
 	bool zcopy_used;
 	int sent_pkts = 0;
 
@@ -895,9 +896,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 
 		/* use msg_control to pass vhost zerocopy ubuf info to skb */
 		if (zcopy_used) {
-			struct ubuf_info *ubuf;
 			ubuf = nvq->ubuf_info + nvq->upend_idx;
-
 			vq->heads[nvq->upend_idx].id = cpu_to_vhost32(vq, head);
 			vq->heads[nvq->upend_idx].len = VHOST_DMA_IN_PROGRESS;
 			ubuf->callback = vhost_zerocopy_callback;
@@ -927,7 +926,8 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
-				vhost_net_ubuf_put(ubufs);
+				if (vq->heads[ubuf->desc].len == VHOST_DMA_IN_PROGRESS)
+					vhost_net_ubuf_put(ubufs);
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
Signature · 1 line
2.23.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#14 lore raw

PATCH v2 2/2 vhost_net: fix high cpu load when sendmsg fails

In reply to wangyunjian

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

---
 drivers/vhost/net.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c +9 -12
index c8784dfafdd7..3d33f3183abe 100644
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
 				msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
 			vhost_discard_vq_desc(vq, 1);
 			vhost_net_enable_vq(net, vq);
 			break;
-		}
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err != len))
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
 done:
 		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
 		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 			msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
 		}
 		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
 		if (!zcopy_used)
 			vhost_add_used_and_signal(&net->dev, vq, head, 0);
 		else
Signature · 1 line
2.23.0
#15 lore raw

Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 8 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

Question: with this patch, what happens if sendmsg is interrupted by a signal?

Quoted 45 lines
---
 drivers/vhost/net.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c +3 -10
index c8784dfafdd7..3d33f3183abe 100644
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
 				msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
 			vhost_discard_vq_desc(vq, 1);
 			vhost_net_enable_vq(net, vq);
 			break;
-		}
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err != len))
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
 done:
 		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
 		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 			msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) · line 937

Quoted 10 lines
drivers/vhost/net.c +6 -2
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
 		}
 		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);

I'd rather make the pr_debug -> vq_err a separate change, with proper commit log describing motivation.

Quoted 5 lines
drivers/vhost/net.c +0 -0
 		if (!zcopy_used)
 			vhost_add_used_and_signal(&net->dev, vq, head, 0);
 		else
Signature · 1 line
2.23.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#16 lore raw

RE: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 14 lines

-----Original Message-----
From: Michael S. Tsirkin [mailto:[email protected]]
Sent: Wednesday, December 16, 2020 5:23 PM
To: wangyunjian <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 2 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Quoted 8 lines
Quoted 6 lines

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

Question: with this patch, what happens if sendmsg is interrupted by a signal?

The descriptors are consumed as normal. However, the packet is discarded. Could you explain the specific scenario?

Quoted 57 lines

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net,

Quoted 13 lines
---
 drivers/vhost/net.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c index renamed from drivers/vhost/net.c +0 -0

c8784dfafdd7..3d33f3183abe 100644

drivers/vhost/net.c +0 -0
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,

struct socket *sock)

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 21 lines
drivers/vhost/net.c +4 -6
 				msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN)) {
 			vhost_discard_vq_desc(vq, 1);
 			vhost_net_enable_vq(net, vq);
 			break;
-		}
-		if (err != len)
-			pr_debug("Truncated TX packet: len %d != %zd\n",
-				 err, len);
+		} else if (unlikely(err != len))
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err,
+len);
 done:
 		vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
 		vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net

Quoted 8 lines
drivers/vhost/net.c +0 -1
 			msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net

*net, struct socket *sock)

Quoted 12 lines
drivers/vhost/net.c +5 -3
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;
+			if (err == -EAGAIN) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
 		}
Quoted 8 lines

drivers/vhost/net.c static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)

Quoted 5 lines
drivers/vhost/net.c +2 -2
 		if (err != len)
-			pr_debug("Truncated TX packet: "
-				 " len %d != %zd\n", err, len);
+			vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err,
+len);

I'd rather make the pr_debug -> vq_err a separate change, with proper commit log describing motivation.

This log was originally triggered when packets were truncated. But after the modification of this patch, other error scenarios will also trigger this log. That's why I modified the content and level of this log together. Now, should I just change the content of this patch?

Thanks

Quoted 7 lines
Quoted 7 lines
drivers/vhost/net.c +0 -1
 		if (!zcopy_used)
 			vhost_add_used_and_signal(&net->dev, vq, head, 0);
 		else
--

2.23.0

#17 lore raw

Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

In reply to Michael S. Tsirkin
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · wangyunjian

From: Yunjian Wang<[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error. This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

Question: with this patch, what happens if sendmsg is interrupted by a signal?

Since we use MSG_DONTWAIT, we don't need to care about signal I think.

Thanks

Quoted 2 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#18 lore raw

Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

In reply to wangyunjian
Quoted 6 lines · wangyunjian

From: Yunjian Wang <[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error.

The patch is based on the assumption that such error cases always return EAGAIN. Can it not also be ENOMEM, such as from tun_build_skb?

Quoted 4 lines

This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

the -> that

here and above: description -> descriptor

Perhaps slightly revise to more explicitly state that

1. in the case of persistent failure (i.e., bad packet), the driver drops the packet
2. in the case of transient failure (e.g,. memory pressure) the driver schedules the worker to try again later

Quoted 16 lines
---
 drivers/vhost/net.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
drivers/vhost/net.c +0 -2
index c8784dfafdd7..3d33f3183abe 100644
@@ -827,16 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
                                msg.msg_flags &= ~MSG_MORE;
                }
 
-               /* TODO: Check specific error and bomb out unless ENOBUFS? */
                err = sock->ops->sendmsg(sock, &msg, len);
-               if (unlikely(err < 0)) {

drivers/vhost/net.c static void handle_tx_copy(struct vhost_net *net, struct socket *sock) · line 836

Quoted 10 lines
drivers/vhost/net.c +3 -4
+               if (unlikely(err == -EAGAIN)) {
                        vhost_discard_vq_desc(vq, 1);
                        vhost_net_enable_vq(net, vq);
                        break;
-               }
-               if (err != len)
-                       pr_debug("Truncated TX packet: len %d != %zd\n",
-                                err, len);
+               } else if (unlikely(err != len))
+                       vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);

sending -> send

Even though vq_err is a wrapper around pr_debug, I agree with Michael that such a change should be a separate patch to net-next, does not belong in a fix.

More importantly, the error message is now the same for persistent errors and for truncated packets. But on truncation the packet was sent, so that is not entirely correct.

Quoted 32 lines
drivers/vhost/net.c +6 -7
 done:
                vq->heads[nvq->done_idx].id = cpu_to_vhost32(vq, head);
                vq->heads[nvq->done_idx].len = 0;
@@ -922,7 +919,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
                        msg.msg_flags &= ~MSG_MORE;
                }
 
-               /* TODO: Check specific error and bomb out unless ENOBUFS? */
                err = sock->ops->sendmsg(sock, &msg, len);
                if (unlikely(err < 0)) {
                        if (zcopy_used) {
@@ -931,13 +927,14 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
                                nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
                                        % UIO_MAXIOV;
                        }
-                       vhost_discard_vq_desc(vq, 1);
-                       vhost_net_enable_vq(net, vq);
-                       break;
+                       if (err == -EAGAIN) {
+                               vhost_discard_vq_desc(vq, 1);
+                               vhost_net_enable_vq(net, vq);
+                               break;
+                       }
                }
                if (err != len)
-                       pr_debug("Truncated TX packet: "
-                                " len %d != %zd\n", err, len);
+                       vq_err(vq, "Fail to sending packets err : %d, len : %zd\n", err, len);
                if (!zcopy_used)
                        vhost_add_used_and_signal(&net->dev, vq, head, 0);
                else
--
Quoted 2 lines

2.23.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#19 lore raw

Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 15 lines · Willem de Bruijn
Quoted 5 lines · wangyunjian

From: Yunjian Wang<[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error.

The patch is based on the assumption that such error cases always return EAGAIN. Can it not also be ENOMEM, such as from tun_build_skb?

Quoted 4 lines

This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

the -> that

Quoted 8 lines

here and above: description -> descriptor

Perhaps slightly revise to more explicitly state that

1. in the case of persistent failure (i.e., bad packet), the driver drops the packet
2. in the case of transient failure (e.g,. memory pressure) the driver schedules the worker to try again later

If we want to go with this way, we need a better time to wakeup the worker. Otherwise it just produces more stress on the cpu that is what this patch tries to avoid.

Thanks

Quoted 2 lines

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#20 lore raw

Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 28 lines · Jason Wang
Quoted 15 lines · Willem de Bruijn
Quoted 5 lines · wangyunjian

From: Yunjian Wang<[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error.

The patch is based on the assumption that such error cases always return EAGAIN. Can it not also be ENOMEM, such as from tun_build_skb?

Quoted 4 lines

This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

the -> that

Quoted 8 lines

here and above: description -> descriptor

Perhaps slightly revise to more explicitly state that

1. in the case of persistent failure (i.e., bad packet), the driver drops the packet
2. in the case of transient failure (e.g,. memory pressure) the driver schedules the worker to try again later

Quoted 3 lines

If we want to go with this way, we need a better time to wakeup the worker. Otherwise it just produces more stress on the cpu that is what this patch tries to avoid.

Perhaps I misunderstood the purpose of the patch: is it to drop everything, regardless of transient or persistent failure, until the ring runs out of descriptors?

I can understand both a blocking and drop strategy during memory pressure. But partial drop strategy until exceeding ring capacity seems like a peculiar hybrid?
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#21 lore raw

Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 27 lines · Willem de Bruijn
Quoted 26 lines · Jason Wang
Quoted 23 lines · Willem de Bruijn
Quoted 5 lines · wangyunjian

From: Yunjian Wang<[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error.

The patch is based on the assumption that such error cases always return EAGAIN. Can it not also be ENOMEM, such as from tun_build_skb?

Quoted 4 lines

This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

the -> that

here and above: description -> descriptor

Perhaps slightly revise to more explicitly state that

1. in the case of persistent failure (i.e., bad packet), the driver drops the packet
2. in the case of transient failure (e.g,. memory pressure) the driver schedules the worker to try again later

Quoted 6 lines
Quoted 3 lines

If we want to go with this way, we need a better time to wakeup the worker. Otherwise it just produces more stress on the cpu that is what this patch tries to avoid.

Perhaps I misunderstood the purpose of the patch: is it to drop everything, regardless of transient or persistent failure, until the ring runs out of descriptors?

My understanding is that the main motivation is to avoid high cpu utilization when sendmsg() fail due to guest reason (e.g bad packet).

Quoted 4 lines

I can understand both a blocking and drop strategy during memory pressure. But partial drop strategy until exceeding ring capacity seems like a peculiar hybrid?

Yes. So I wonder if we want to be do better when we are in the memory pressure. E.g can we let socket wake up us instead of rescheduling the workers here? At least in this case we know some memory might be freed?

Thanks

Quoted 1 line

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#22 lore raw

RE: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 45 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, December 23, 2020 10:54 AM
To: Willem de Bruijn <[email protected]>
Cc: wangyunjian <[email protected]>; Network Development <[email protected]>; Michael S. Tsirkin <[email protected]>; [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 1 line · Willem de Bruijn

On Mon, Dec 21, 2020 at 11:41 PM Jason Wang <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines · Willem de Bruijn
Quoted 1 line

On Wed, Dec 16, 2020 at 3:20 AM wangyunjian<[email protected]>

Quoted 25 lines · wrote:
Quoted 25 lines
Quoted 14 lines
Quoted 5 lines

From: Yunjian Wang<[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error.

The patch is based on the assumption that such error cases always return EAGAIN. Can it not also be ENOMEM, such as from tun_build_skb?

Quoted 4 lines

This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

the -> that

Quoted 7 lines

here and above: description -> descriptor

Perhaps slightly revise to more explicitly state that

1. in the case of persistent failure (i.e., bad packet), the driver drops the packet 2. in the case of transient failure (e.g,. memory pressure) the driver schedules the worker to try again later

If we want to go with this way, we need a better time to wakeup the worker. Otherwise it just produces more stress on the cpu that is what this patch tries to avoid.

Quoted 8 lines
Quoted 3 lines

Perhaps I misunderstood the purpose of the patch: is it to drop everything, regardless of transient or persistent failure, until the ring runs out of descriptors?

My understanding is that the main motivation is to avoid high cpu utilization when sendmsg() fail due to guest reason (e.g bad packet).

My main motivation is to avoid the tx queue stuck.

Should I describe it like this:
Currently the driver don't drop a packet which can't be send by tun (e.g bad packet). In this case, the driver will always process the same packet lead to the tx queue stuck.

To fix this issue:
1. in the case of persistent failure (e.g bad packet), the driver can skip this descriptior by ignoring the error.
2. in the case of transient failure (e.g -EAGAIN and -ENOMEM), the driver schedules the worker to try again.

Thanks

Quoted 15 lines
Quoted 5 lines

I can understand both a blocking and drop strategy during memory pressure. But partial drop strategy until exceeding ring capacity seems like a peculiar hybrid?

Yes. So I wonder if we want to be do better when we are in the memory pressure. E.g can we let socket wake up us instead of rescheduling the workers here? At least in this case we know some memory might be freed?

Thanks

Quoted 1 line
#23 lore raw

Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 62 lines · wangyunjian
Quoted 46 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, December 23, 2020 10:54 AM
To: Willem de Bruijn <[email protected]>
Cc: wangyunjian <[email protected]>; Network Development <[email protected]>; Michael S. Tsirkin <[email protected]>; [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 1 line · Willem de Bruijn

On Mon, Dec 21, 2020 at 11:41 PM Jason Wang <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines · Willem de Bruijn
Quoted 1 line

On Wed, Dec 16, 2020 at 3:20 AM wangyunjian<[email protected]>

Quoted 25 lines · wrote:
Quoted 25 lines
Quoted 14 lines
Quoted 5 lines

From: Yunjian Wang<[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error.

The patch is based on the assumption that such error cases always return EAGAIN. Can it not also be ENOMEM, such as from tun_build_skb?

Quoted 4 lines

This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

the -> that

Quoted 7 lines

here and above: description -> descriptor

Perhaps slightly revise to more explicitly state that

1. in the case of persistent failure (i.e., bad packet), the driver drops the packet 2. in the case of transient failure (e.g,. memory pressure) the driver schedules the worker to try again later

If we want to go with this way, we need a better time to wakeup the worker. Otherwise it just produces more stress on the cpu that is what this patch tries to avoid.

Quoted 8 lines
Quoted 3 lines

Perhaps I misunderstood the purpose of the patch: is it to drop everything, regardless of transient or persistent failure, until the ring runs out of descriptors?

My understanding is that the main motivation is to avoid high cpu utilization when sendmsg() fail due to guest reason (e.g bad packet).

My main motivation is to avoid the tx queue stuck.

Should I describe it like this:
Currently the driver don't drop a packet which can't be send by tun (e.g bad packet). In this case, the driver will always process the same packet lead to the tx queue stuck.

Quoted 5 lines

To fix this issue:
1. in the case of persistent failure (e.g bad packet), the driver can skip this descriptior by ignoring the error.
2. in the case of transient failure (e.g -EAGAIN and -ENOMEM), the driver schedules the worker to try again.

That sounds good to me, thanks.

Quoted 12 lines

Thanks

Quoted 10 lines
Quoted 5 lines

I can understand both a blocking and drop strategy during memory pressure. But partial drop strategy until exceeding ring capacity seems like a peculiar hybrid?

Yes. So I wonder if we want to be do better when we are in the memory pressure. E.g can we let socket wake up us instead of rescheduling the workers here? At least in this case we know some memory might be freed?

I don't know whether a blocking or drop strategy is the better choice. Either way, it probably deserves to be handled separately.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#24 lore raw

RE: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

In reply to Jason Wang
Quoted 35 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Tuesday, December 22, 2020 12:41 PM
To: Willem de Bruijn <[email protected]>; wangyunjian <[email protected]>
Cc: Network Development <[email protected]>; Michael S. Tsirkin <[email protected]>; [email protected]; Lilijun (Jerry) <[email protected]>; chenchanghu <[email protected]>; xudingke <[email protected]>; huangbin (J) <[email protected]>
Subject: Re: [PATCH net v2 2/2] vhost_net: fix high cpu load when sendmsg fails

Quoted 1 line · Willem de Bruijn

On Wed, Dec 16, 2020 at 3:20 AM wangyunjian<[email protected]>

Quoted 17 lines · wrote:
Quoted 4 lines

From: Yunjian Wang<[email protected]>

Currently we break the loop and wake up the vhost_worker when sendmsg fails. When the worker wakes up again, we'll meet the same error.

The patch is based on the assumption that such error cases always return EAGAIN. Can it not also be ENOMEM, such as from tun_build_skb?

Quoted 4 lines

This will cause high CPU load. To fix this issue, we can skip this description by ignoring the error. When we exceeds sndbuf, the return value of sendmsg is -EAGAIN. In the case we don't skip the description and don't drop packet.

the -> that

here and above: description -> descriptor

Perhaps slightly revise to more explicitly state that

Quoted 8 lines
Quoted 3 lines

1. in the case of persistent failure (i.e., bad packet), the driver drops the packet 2. in the case of transient failure (e.g,. memory pressure) the driver schedules the worker to try again later

If we want to go with this way, we need a better time to wakeup the worker. Otherwise it just produces more stress on the cpu that is what this patch tries to avoid.

The problem was initially discovered when a VM sent an abnormal packet, which causing the VM can't send packets anymore. After this patch "feb8892cb441c7 vhost_net: conditionally enable tx polling", there have also been high CPU consumption issues.

It is the first problem that I am actually more concerned with and want to solve.

Thanks

Quoted 6 lines

Thanks

Quoted 2 lines