net/9p: Fix a deadlock case in the virtio transport

When client has multiple threads that issue io requests all the time, and the server has a very good performance, it may cause cpu is running in the irq context for a long time because it can check virtqueue has buf in the *while* loop.

So we should keep chan->lock in the whole loop.

Signed-off-by Yiwen Jiang <[email protected]>

---
 net/9p/trans_virtio.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

net/9p/trans_virtio.c +6 -11

index 05006cb..e5fea8b 100644
@@ -148,20 +148,15 @@ static void req_done(struct virtqueue *vq)
 
 	p9_debug(P9_DEBUG_TRANS, ": request done\n");
 
-	while (1) {
-		spin_lock_irqsave(&chan->lock, flags);
-		req = virtqueue_get_buf(chan->vq, &len);
-		if (req == NULL) {
-			spin_unlock_irqrestore(&chan->lock, flags);
-			break;
-		}
-		chan->ring_bufs_avail = 1;
-		spin_unlock_irqrestore(&chan->lock, flags);
-		/* Wakeup if anyone waiting for VirtIO ring space. */
-		wake_up(chan->vc_wq);
+	spin_lock_irqsave(&chan->lock, flags);
+	while ((req = virtqueue_get_buf(chan->vq, &len)) != NULL) {
 		if (len)
 			p9_client_cb(chan->client, req, REQ_STATUS_RCVD);
 	}
+	chan->ring_bufs_avail = 1;
+	spin_unlock_irqrestore(&chan->lock, flags);
+	/* Wakeup if anyone waiting for VirtIO ring space. */
+	wake_up(chan->vc_wq);
 }
 
 /**

Subject: net/9p: Fix a deadlock case in the virtio transport

When client has multiple threads that issue io requests all the time, and the server has a very good performance, it may cause cpu is running in the irq context for a long time because it can check virtqueue has buf in the *while* loop.

So we should keep chan->lock in the whole loop.

Signed-off-by Yiwen Jiang <[email protected]>

---
 net/9p/trans_virtio.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

net/9p/trans_virtio.c +0 -7

index 05006cb..e5fea8b 100644
@@ -148,20 +148,15 @@ static void req_done(struct virtqueue *vq)
 
 	p9_debug(P9_DEBUG_TRANS, ": request done\n");
 
-	while (1) {
-		spin_lock_irqsave(&chan->lock, flags);
-		req = virtqueue_get_buf(chan->vq, &len);
-		if (req == NULL) {
-			spin_unlock_irqrestore(&chan->lock, flags);
-			break;
-		}

Quoted 2 lines

Subject: net/9p: Fix a deadlock case in the virtio transport

I hadn't noticed in the v1, but how is that a deadlock fix? The previous code doesn't look like it deadlocks to me, the commit message is more correct.

jiangyiwen wrote on Tue, Jul 17, 2018:

Quoted 28 lines

When client has multiple threads that issue io requests all the time, and the server has a very good performance, it may cause cpu is running in the irq context for a long time because it can check virtqueue has buf in the *while* loop.

So we should keep chan->lock in the whole loop.

Signed-off-by Yiwen Jiang <[email protected]>

---
 net/9p/trans_virtio.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

net/9p/trans_virtio.c +0 -7

index 05006cb..e5fea8b 100644
@@ -148,20 +148,15 @@ static void req_done(struct virtqueue *vq)
 
 	p9_debug(P9_DEBUG_TRANS, ": request done\n");
 
-	while (1) {
-		spin_lock_irqsave(&chan->lock, flags);
-		req = virtqueue_get_buf(chan->vq, &len);
-		if (req == NULL) {
-			spin_unlock_irqrestore(&chan->lock, flags);
-			break;
-		}

net/9p/trans_virtio.c static void req_done(struct virtqueue *vq) · line 156

Quoted 10 lines

net/9p/trans_virtio.c +3 -4

-		chan->ring_bufs_avail = 1;
-		spin_unlock_irqrestore(&chan->lock, flags);
-		/* Wakeup if anyone waiting for VirtIO ring space. */
-		wake_up(chan->vc_wq);
+	spin_lock_irqsave(&chan->lock, flags);
+	while ((req = virtqueue_get_buf(chan->vq, &len)) != NULL) {
 		if (len)
 			p9_client_cb(chan->client, req, REQ_STATUS_RCVD);
 	}
+	chan->ring_bufs_avail = 1;

Do we have a guarantee that req_done is only called if there is at least one buf to read?
For example, that there isn't two threads queueing the same callback but the first one reads everything and the second has nothing to read?

If virtblk_done takes care of setting up a "req_done" bool to only notify waiters if something has been done I'd rather have a reason to do differently, even if you can argue that nothing bad will happen in case of a gratuitous wake_up

net/9p/trans_virtio.c static void req_done(struct virtqueue *vq) · line 160

Quoted 4 lines

net/9p/trans_virtio.c +3 -0

+	spin_unlock_irqrestore(&chan->lock, flags);
+	/* Wakeup if anyone waiting for VirtIO ring space. */
+	wake_up(chan->vc_wq);
 }

Thanks,

Quoted 7 lines

Quoted 2 lines

Subject: net/9p: Fix a deadlock case in the virtio transport

I hadn't noticed in the v1, but how is that a deadlock fix? The previous code doesn't look like it deadlocks to me, the commit message is more correct.

If cpu is running in the irq context for a long time, NMI watchdog will detect the hard lockup in the cpu, and then it will cause kernel panic. So I use this subject to underline the scenario.

Quoted 10 lines

Do we have a guarantee that req_done is only called if there is at least one buf to read?
For example, that there isn't two threads queueing the same callback but the first one reads everything and the second has nothing to read?

If virtblk_done takes care of setting up a "req_done" bool to only notify waiters if something has been done I'd rather have a reason to do differently, even if you can argue that nothing bad will happen in case of a gratuitous wake_up

Sorry, I don't fully understand what your mean.
I think even if the ring buffer don't have the data, wakeup operation will not cause any other problem, and the loss of performance can be ignored.

jiangyiwen wrote on Tue, Jul 17, 2018:

Quoted 9 lines · Dominique Martinet

Quoted 7 lines

Quoted 2 lines

Subject: net/9p: Fix a deadlock case in the virtio transport

I hadn't noticed in the v1, but how is that a deadlock fix? The previous code doesn't look like it deadlocks to me, the commit message is more correct.

Quoted 4 lines

If cpu is running in the irq context for a long time, NMI watchdog will detect the hard lockup in the cpu, and then it will cause kernel panic. So I use this subject to underline the scenario.

That's still not a deadlock - fix lockup would be more appropriate?

Quoted 11 lines

Quoted 10 lines

Do we have a guarantee that req_done is only called if there is at least one buf to read?
For example, that there isn't two threads queueing the same callback but the first one reads everything and the second has nothing to read?

If virtblk_done takes care of setting up a "req_done" bool to only notify waiters if something has been done I'd rather have a reason to do differently, even if you can argue that nothing bad will happen in case of a gratuitous wake_up

Quoted 4 lines

Sorry, I don't fully understand what your mean.
I think even if the ring buffer don't have the data, wakeup operation will not cause any other problem, and the loss of performance can be ignored.

I just mean "others do check, why not us?". It's almost free to check if we had something to read, but if there are many pending read/writes waiting for a buffer they will all wake up and spin uselessly.

I've checked other callers of virtqueue_get_buf() and out of 9 that loop around in a callback then wake another thread up, 6 do check before waking up, two check that something happened just to print a debug statement if not (virtio_test and virtgpu) and one doesn't check (virtio_input); so I guess we wouldn't be the first ones, just not following the trend.

But yes, nothing bad will happen, so let's agree to disagree and I'll defer to others opinion on this

Thanks,