vhost: use try_lock in rte_vhost_vring_call
PATCH 1/1 vhost: use try_lock in rte_vhost_vring_call
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using try_lock here, users can have another try in case of VQ's access lock is held by vhost-events thread.
--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
lib/vhost/vhost.c +5 -1
@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);
Signature · 1 line
2.21.3
Quoted 15 lines · On Tue, 6 Sep 2022 10:22:25 +0800 Changpeng Liu <[email protected]>…
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1337
Quoted 10 lines
lib/vhost/vhost.c +5
-1@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }
If it is a race, logging a message is not a good idea; the log will fill with this noise.
Instead make it statistic that can be seen by xstats.
Quoted 42 lines
-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Wednesday, September 7, 2022 5:16 AM
To: Liu, Changpeng <[email protected]>
Cc: [email protected]; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 15 lines · On Tue, 6 Sep 2022 10:22:25 +0800 Changpeng Liu <[email protected]>…
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1337
Quoted 10 lines
lib/vhost/vhost.c +5
-1@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }If it is a race, logging a message is not a good idea; the log will fill with this noise.
Quoted 1 line
Instead make it statistic that can be seen by xstats.
It's a DEBUG log, users can't see it in practice.
Quoted 42 lines · Changpeng
Quoted 42 lines
-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Wednesday, September 7, 2022 5:16 AM
To: Liu, Changpeng <[email protected]>
Cc: [email protected]; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 15 lines · On Tue, 6 Sep 2022 10:22:25 +0800 Changpeng Liu <[email protected]>…
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1337
Quoted 10 lines
lib/vhost/vhost.c +5
-1@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }If it is a race, logging a message is not a good idea; the log will fill with this noise.
Quoted 3 lines
Quoted 1 line
Instead make it statistic that can be seen by xstats.
It's a DEBUG log, users can't see it in practice.
Having an xstat would enable live debugging & post-mortem analysis. You can have both the stats and the debug log.
Regards,
Maxime
Hi Changpeng,
Quoted 8 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 6, 2022 10:22 AM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: [PATCH] vhost: usetry_lockinrte_vhost_vring_call
Quoted 4 lines
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.
Better to describe the issue this patch wants to fix and how does it fix.
I remember it's a bz issue, do you want to backport? And it has some bz ID, we need to add it in commit message.
Quoted 11 lines
Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1333
Quoted 6 lines
lib/vhost/vhost.c +2
-1@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { + VHOST_LOG_CONFIG(dev->ifname, DEBUG,
Should use VHOST_LOG_DATA
Thanks,
Chenbo
Quoted 8 lines
lib/vhost/vhost.c +3
-1+ "failed to kick guest, virtqueue busy.\n"); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq); --2.21.3
Hi Bo,
Quoted 22 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, September 20, 2022 10:25 AM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Changpeng,
Quoted 8 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 6, 2022 10:22 AM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 4 lines
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.
Quoted 5 lines
Better to describe the issue this patch wants to fix and how does it fix.
I remember it's a bz issue, do you want to backport? And it has some bz ID, we need to add it in commit message.
Actually it's my intention not to add bz ID, as I think for this bz ID, It's better not to lock all VQ's access lock for KICK/CALLFD messages, What do you think? If this is identified as a fix, I can backport it to 22.05.
Quoted 15 lines
Quoted 15 lines
Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;
Quoted 5 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1333
Quoted 3 lines
lib/vhost/vhost.c +2
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { + VHOST_LOG_CONFIG(dev->ifname, DEBUG,Should use
VHOST_LOG_DATA
OK.
Quoted 12 lines
Thanks,
ChenboQuoted 8 lines
lib/vhost/vhost.c +3
-1+ "failed to kick guest, virtqueue busy.\n"); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq); --2.21.3
Quoted 31 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 20, 2022 10:34 AM
To: Xia, Chenbo <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Bo,
Quoted 22 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, September 20, 2022 10:25 AM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Changpeng,
Quoted 12 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 6, 2022 10:22 AM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: [PATCH] vhost: usetry_lockinrte_vhost_vring_callNote that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.
Quoted 7 lines
Quoted 5 lines
Better to describe the issue this patch wants to fix and how does it fix.
I remember it's a bz issue, do you want to backport? And it has some bz ID, we need to add it in commit message.
Actually it's my intention not to add bz ID, as I think for this bz ID, It's better not to lock all VQ's access lock for KICK/CALLFD messages,
Do you plan to add this change? I think that may be an improvement to current locking implementation.
Maxime, what do you think of this idea about only locking specific queue when handling vring related message (not global config like mem table)?
Quoted 2 lines
What do you think? If this is identified as a fix, I can backport it to 22.05.
You can decide, if this is planned to be the fix, just backport. I am just thinking if this is not the fix for the bz, do we still need this?
Thanks,
Chenbo
Quoted 34 lines
Quoted 7 lines
Quoted 7 lines
Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)Quoted 5 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t
Quoted 5 lines
lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t
vring_idx)Quoted 8 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1333
Quoted 6 lines
lib/vhost/vhost.c +2
-1if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { + VHOST_LOG_CONFIG(dev->ifname, DEBUG,Should use
VHOST_LOG_DATAOK.
Quoted 12 lines
Thanks,
ChenboQuoted 8 lines
lib/vhost/vhost.c +3
-1+ "failed to kick guest, virtqueue busy.\n"); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq); --2.21.3
Quoted 43 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, September 20, 2022 10:54 AM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 34 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 20, 2022 10:34 AM
To: Xia, Chenbo <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Bo,
Quoted 25 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, September 20, 2022 10:25 AM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Changpeng,
Quoted 8 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 6, 2022 10:22 AM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 4 lines
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Better to describe the issue this patch wants to fix and how does it fix.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I remember it's a bz issue, do you want to backport? And it has some bz ID, we need to add it in commit message.
Actually it's my intention not to add bz ID, as I think for this bz ID, It's better not to lock all VQ's access lock for KICK/CALLFD messages,
Do you plan to add this change? I think that may be an improvement to current locking implementation.
No, I don't have such a plan.
Quoted 9 lines
Maxime, what do you think of this idea about only locking specific queue when handling vring related message (not global config like mem table)?
Quoted 2 lines
What do you think? If this is identified as a fix, I can backport it to 22.05.
You can decide, if this is planned to be the fix, just backport. I am just thinking if this is not the fix for the bz, do we still need this?
Adding the bz ID is OK to me. From SPDK's view indeed it's a fix. Will send V2 later. Thanks.
Quoted 38 lines
Thanks,
ChenboQuoted 34 lines
Quoted 7 lines
Quoted 7 lines
Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)Quoted 5 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t
Quoted 5 lines
lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t
vring_idx)Quoted 8 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1333
Quoted 6 lines
lib/vhost/vhost.c +2
-1if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { + VHOST_LOG_CONFIG(dev->ifname, DEBUG,Should use
VHOST_LOG_DATAOK.
Quoted 12 lines
Thanks,
ChenboQuoted 8 lines
lib/vhost/vhost.c +3
-1+ "failed to kick guest, virtqueue busy.\n"); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq); --2.21.3
Quoted 36 lines · Chenbo
Quoted 36 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 20, 2022 10:34 AM
To: Xia, Chenbo <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Bo,
Quoted 27 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, September 20, 2022 10:25 AM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Changpeng,
Quoted 8 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 6, 2022 10:22 AM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 4 lines
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Better to describe the issue this patch wants to fix and how does it fix.
I remember it's a bz issue, do you want to backport? And it has some bz ID, we need to add it in commit message.
Quoted 8 lines
Quoted 2 lines
Actually it's my intention not to add bz ID, as I think for this bz ID, It's better not to lock all VQ's access lock for KICK/CALLFD messages,
Do you plan to add this change? I think that may be an improvement to current locking implementation.
Maxime, what do you think of this idea about only locking specific queue when handling vring related message (not global config like mem table)?
I think this is not a good idea.
For example SET_VRING_KICK can currently call translate_ring_addresses(), which itself can call numa_realloc().
numa_realloc() may reallocate the dev, so you don't want it to be used by other queues while it happens.
Quoted 44 lines
Quoted 2 lines
What do you think? If this is identified as a fix, I can backport it to 22.05.
You can decide, if this is planned to be the fix, just backport. I am just thinking if this is not the fix for the bz, do we still need this?
Thanks,
ChenboQuoted 34 lines
Quoted 7 lines
Quoted 7 lines
Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)Quoted 5 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t
Quoted 5 lines
lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t
vring_idx)Quoted 8 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1333
Quoted 6 lines
lib/vhost/vhost.c +2
-1if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { + VHOST_LOG_CONFIG(dev->ifname, DEBUG,Should use
VHOST_LOG_DATAOK.
Quoted 12 lines
Thanks,
ChenboQuoted 8 lines
lib/vhost/vhost.c +3
-1+ "failed to kick guest, virtqueue busy.\n"); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq); --2.21.3
Quoted 48 lines · Maxime Coquelin
Quoted 36 lines · Chenbo
Quoted 36 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 20, 2022 10:34 AM
To: Xia, Chenbo <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Bo,
Quoted 27 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, September 20, 2022 10:25 AM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>
Subject: RE: [PATCH] vhost: usetry_lockinrte_vhost_vring_callHi Changpeng,
Quoted 8 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Tuesday, September 6, 2022 10:22 AM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 4 lines
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Better to describe the issue this patch wants to fix and how does it fix.
I remember it's a bz issue, do you want to backport? And it has some bz ID, we need to add it in commit message.
Quoted 8 lines
Quoted 2 lines
Actually it's my intention not to add bz ID, as I think for this bz ID, It's better not to lock all VQ's access lock for KICK/CALLFD messages,
Do you plan to add this change? I think that may be an improvement to current locking implementation.
Maxime, what do you think of this idea about only locking specific queue when handling vring related message (not global config like mem table)?
Quoted 6 lines
I think this is not a good idea.
For exampleSET_VRING_KICKcan currently calltranslate_ring_addresses(), which itself can callnuma_realloc().
numa_realloc()may reallocate the dev, so you don't want it to be used by other queues while it happens.
Hmm, actually that may be possible because numa_realloc() reallocs the dev only if it is not running.
So maybe you can propose something, but you will have to test it carefully with use-cases involving NUMA reallocation.
Quoted 44 lines
Quoted 44 lines
Quoted 2 lines
What do you think? If this is identified as a fix, I can backport it to 22.05.
You can decide, if this is planned to be the fix, just backport. I am just thinking if this is not the fix for the bz, do we still need this?
Thanks,
ChenboQuoted 34 lines
Quoted 7 lines
Quoted 7 lines
Signed-off-by Changpeng Liu <[email protected]>---
lib/vhost/vhost.c| 6 +++++-1 file changed, 5 insertions(+), 1 deletion(-)Quoted 5 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t
Quoted 5 lines
lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t
vring_idx)Quoted 8 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1333
Quoted 6 lines
lib/vhost/vhost.c +2
-1if (!vq) return -1; - rte_spinlock_lock(&vq->access_lock); + if (!rte_spinlock_trylock(&vq->access_lock)) { + VHOST_LOG_CONFIG(dev->ifname, DEBUG,Should use
VHOST_LOG_DATAOK.
Quoted 12 lines
Thanks,
ChenboQuoted 8 lines
lib/vhost/vhost.c +3
-0+ "failed to kick guest, virtqueue busy.\n"); + return -1; + } if (vq_is_packed(dev)) vhost_vring_call_packed(dev, vq);Signature · 1 line
2.21.3
Quoted 17 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1339
Quoted 10 lines
lib/vhost/vhost.c +5
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);
I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Just some internal DPDK usage of this API:./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid, qid);./examples/vhost/virtio_net.c:236: rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost/virtio_net.c:446: rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost_blk/vhost_blk.c:99: rte_vhost_vring_call(task->ctrlr->vid, vq->id); ./examples/vhost_blk/vhost_blk.c:134: rte_vhost_vring_call(task->ctrlr->vid, vq->id);
This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Regards,
Maxime
Hi Maxime,
Quoted 51 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 17 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1339
Quoted 10 lines
lib/vhost/vhost.c +5
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost/virtio_net.c:446:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);
Quoted 5 lines
This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK project is blocked for a while which can't be used with DPDK 22.05 or newer.
Vhost-blk and scsi devices are not same with vhost-net, we need to cover SeaBIOS and VM cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
Quoted 55 lines · Changpeng
Hi Maxime,
Quoted 53 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 17 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1339
Quoted 10 lines
lib/vhost/vhost.c +5
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost/virtio_net.c:446:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);This change will break all the above uses.
Quoted 7 lines
Quoted 3 lines
And that's not counting external projects.
ou should better introduce a new API that does not block.
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK project is blocked for a while which can't be used with DPDK 22.05 or newer.
DPDK v22.05?
What is the commit introducing the regression?
Note that if we introduce a new API, it won't be backported to stable branches.
Quoted 6 lines
Vhost-blk and scsi devices are not same with vhost-net, we need to cover SeaBIOS and VM cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
Quoted 70 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:35 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 58 lines · Changpeng
Hi Maxime,
Quoted 56 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 17 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1339
Quoted 10 lines
lib/vhost/vhost.c +5
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost/virtio_net.c:446:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Quoted 8 lines
Quoted 3 lines
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK project
is blocked for
Quoted 1 line
a while which can't be used with DPDK 22.05 or newer.
DPDK v22.05?
What is the commit introducing the regression?
Here is the commit introducing this issue c5736998305d ("vhost: fix missing virtqueue lock protection") Bugzilla ID: 1015
Quoted 3 lines
Note that if we introduce a new API, it won't be backported to stable branches.
I understand, but do we have better idea in short time? we're planning to release SPDK 22.09 recently.
Quoted 9 lines
Quoted 3 lines
Vhost-blk and scsi devices are not same with vhost-net, we need to cover
SeaBIOS and VM
Quoted 5 lines
cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
Quoted 73 lines · Changpeng
Quoted 73 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:35 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 61 lines · Changpeng
Hi Maxime,
Quoted 51 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 17 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1339
Quoted 10 lines
lib/vhost/vhost.c +5
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost/virtio_net.c:446:rte_vhost_vring_call(dev->vid, queue_id);./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);Quoted 5 lines
This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK project
Quoted 8 lines
Quoted 5 lines
is blocked for
Quoted 1 line
a while which can't be used with DPDK 22.05 or newer.
DPDK v22.05?
What is the commit introducing the regression?Here is the commit introducing this issue
c5736998305d("vhost: fix missing virtqueue lock protection") Bugzilla ID: 1015
Ok, it cannot be reverted, as it prevents some undefined behaviors/crashes.
Quoted 5 lines
Quoted 3 lines
Note that if we introduce a new API, it won't be backported to stable branches.
I understand, but do we have better idea in short time? we're planning to release SPDK 22.09 recently.
You can have another thread that sends the call?
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Vhost-blk and scsi devices are not same with vhost-net, we need to cover
SeaBIOS and VM
Quoted 5 lines
cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
Quoted 101 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 4:13 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 55 lines · Changpeng
Quoted 55 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:35 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 43 lines · Changpeng
Hi Maxime,
Quoted 41 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 17 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1339
Quoted 10 lines
lib/vhost/vhost.c +5
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
./examples/vhost/virtio_net.c:446: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK
project
Quoted 8 lines
Quoted 5 lines
is blocked for
Quoted 1 line
a while which can't be used with DPDK 22.05 or newer.
DPDK v22.05?
What is the commit introducing the regression?Here is the commit introducing this issue
c5736998305d("vhost: fix missing virtqueue lock protection") Bugzilla ID: 1015Ok, it cannot be reverted, as it prevents some undefined behaviors/crashes.
Quoted 1 line
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Note that if we introduce a new API, it won't be backported to stable branches.
I understand, but do we have better idea in short time? we're planning to release SPDK 22.09 recently.
You can have another thread that sends the call?
We already use two threads to do this. Here is the example for existing code in SPDK:
DPDK vhost-events thread SPDK thread
SET_VRING_KICK VQ1 ----> Start polling VQ1
Reply to DPDK <---- Done
SET_VRING_KICK VQ2 ----> thread is blocked on VQ's access lock, SPDK thread can't provide reply message For example, we can just return for SET_VRING_KICK VQ2 message without checking SPDK thread, but this leave uncertain replies to VM.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Vhost-blk and scsi devices are not same with vhost-net, we need to cover
SeaBIOS and VM
Quoted 5 lines
cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
Quoted 109 lines · Changpeng
Quoted 101 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 4:13 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 55 lines · Changpeng
Quoted 55 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:35 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 43 lines · Changpeng
Hi Maxime,
Quoted 41 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 17 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)if (!vq)return -1 ;lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1339
Quoted 10 lines
lib/vhost/vhost.c +5
-1-rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
./examples/vhost/virtio_net.c:446: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK
project
Quoted 8 lines
Quoted 5 lines
is blocked for
Quoted 1 line
a while which can't be used with DPDK 22.05 or newer.
DPDK v22.05?
What is the commit introducing the regression?Here is the commit introducing this issue
c5736998305d("vhost: fix missing virtqueue lock protection") Bugzilla ID: 1015Ok, it cannot be reverted, as it prevents some undefined behaviors/crashes.
Quoted 1 line
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Note that if we introduce a new API, it won't be backported to stable branches.
I understand, but do we have better idea in short time? we're planning to release SPDK 22.09 recently.
You can have another thread that sends the call?
We already use two threads to do this. Here is the example for existing code in SPDK:
Quoted 8 lines
DPDK vhost-events thread SPDK thread
SET_VRING_KICK VQ1 ----> Start polling VQ1 Reply to DPDK <---- Done SET_VRING_KICK VQ2 ----> thread is blocked on VQ's access lock, SPDK thread can't provide reply messageFor example, we can just return for
SET_VRING_KICKVQ2 message without checking SPDK thread, but this leave uncertain replies to VM.
I'm sorry but you will have to find a workaround while v22.11 is out and you can consume it. We can neither backport new API nor we can break all the other applications not handling locking failure.
Regarding the new API for v22.11, I should be named something like rte_vhost_vring_call_nonblock(), and ideally should return some like -EAGAIN instead of -1 o that the applications can distinguish between a real failure and a need for retry.
Regards,
Maxime
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Vhost-blk and scsi devices are not same with vhost-net, we need to cover
SeaBIOS and VM
Quoted 5 lines
cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
Quoted 130 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Wednesday, September 21, 2022 5:41 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 46 lines · Changpeng
Quoted 46 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 4:13 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 34 lines · Changpeng
Quoted 34 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:35 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 22 lines · Changpeng
Hi Maxime,
Quoted 20 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 10 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t
Quoted 5 lines
lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t
vring_idx)Quoted 50 lines
Quoted 50 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t · line 1339
Quoted 12 lines
lib/vhost/vhost.c +5
-1if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
./examples/vhost/virtio_net.c:446: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK
project
Quoted 8 lines
Quoted 5 lines
is blocked for
Quoted 1 line
a while which can't be used with DPDK 22.05 or newer.
DPDK v22.05?
What is the commit introducing the regression?Here is the commit introducing this issue
c5736998305d("vhost: fix missing virtqueue lock protection") Bugzilla ID: 1015Ok, it cannot be reverted, as it prevents some undefined behaviors/crashes.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Note that if we introduce a new API, it won't be backported to stable branches.
I understand, but do we have better idea in short time? we're planning to release SPDK 22.09 recently.
You can have another thread that sends the call?
We already use two threads to do this. Here is the example for existing code in
SPDK:
Quoted 6 lines
DPDK vhost-events thread SPDK thread
SET_VRING_KICK VQ1 ----> Start polling VQ1 Reply to DPDK <---- Done SET_VRING_KICK VQ2 ----> thread is blocked on VQ's access lock,SPDK thread can't provide reply message
Quoted 1 line
Quoted 7 lines
Quoted 1 line
For example, we can just return for
SET_VRING_KICKVQ2 message withoutchecking SPDK thread, but this leave
Quoted 1 line
uncertain replies to VM.
I'm sorry but you will have to find a workaround while v22.11 is out and you can consume it. We can neither backport new API nor we can break all the other applications not handling locking failure.
By processing vhost-user message in asynchronous way in SPDK can be a workaround now, we can backport the workaround to SPDK earlier version so that it can work with distro DPDK releases.
Quoted 5 lines
Regarding the new API for v22.11, I should be named something like
rte_vhost_vring_call_nonblock(), and ideally should return some like-EAGAINinstead of -1 o that the applications can distinguish between a real failure and a need for retry.
Agreed, then we can switch to the new API finally.
Quoted 16 lines
Regards,
MaximeQuoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Vhost-blk and scsi devices are not same with vhost-net, we need to cover
SeaBIOS and VM
Quoted 5 lines
cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
Hi Changpeng,
Quoted 141 lines · Changpeng
Quoted 130 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Wednesday, September 21, 2022 5:41 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 46 lines · Changpeng
Quoted 46 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 4:13 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 34 lines · Changpeng
Quoted 34 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:35 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 22 lines · Changpeng
Hi Maxime,
Quoted 20 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, September 20, 2022 3:19 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>
Subject: Re: [PATCH] vhost: usetry_lockinrte_vhost_vring_callQuoted 10 lines · Changpeng Liu
Note that this function is in data path, so the thread context may not same as socket messages processing context, by using
try_lockhere, users can have another try in case of VQ's access lock is held byvhost-eventsthread.Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t
Quoted 5 lines
lib/vhost/vhost.c +0
-0@@ -1329,7 +1329,11 @@ rte_vhost_vring_call(int vid, uint16_t
vring_idx)Quoted 50 lines
Quoted 50 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t · line 1339
Quoted 12 lines
lib/vhost/vhost.c +5
-1if (!vq)return -1 ; -rte_spinlock_lock (&vq->access_lock); +if (!rte_spinlock_trylock (&vq->access_lock)) { +VHOST_LOG_CONFIG (dev->ifname,DEBUG , +"failed to kick guest, virtqueue busy.\n" ); +return -1 ; + }if (vq_is_packed (dev))vhost_vring_call_packed (dev, vq);I think that's problematic, because it will break other applications that currently rely on the API to block until the call is done.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Just some internal DPDK usage of this API:
./drivers/vdpa/ifc/ifcvf_vdpa.c:871: rte_vhost_vring_call(internal->vid,
qid);./examples/vhost/virtio_net.c:236: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
./examples/vhost/virtio_net.c:446: rte_vhost_vring_call(dev->vid,
queue_id);Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
./examples/vhost_blk/vhost_blk.c:99:rte_vhost_vring_call(task->ctrlr->vid, vq->id);./examples/vhost_blk/vhost_blk.c:134:rte_vhost_vring_call(task->ctrlr->vid, vq->id);Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
This change will break all the above uses.
And that's not counting external projects.
ou should better introduce a new API that does not block.
Could you add a new API to do this?
I think we can use the new API in SPDK as a workaround, note that SPDK
project
Quoted 8 lines
Quoted 5 lines
is blocked for
Quoted 1 line
a while which can't be used with DPDK 22.05 or newer.
DPDK v22.05?
What is the commit introducing the regression?Here is the commit introducing this issue
c5736998305d("vhost: fix missing virtqueue lock protection") Bugzilla ID: 1015Ok, it cannot be reverted, as it prevents some undefined behaviors/crashes.
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Note that if we introduce a new API, it won't be backported to stable branches.
I understand, but do we have better idea in short time? we're planning to release SPDK 22.09 recently.
You can have another thread that sends the call?
We already use two threads to do this. Here is the example for existing code in
SPDK:
Quoted 6 lines
DPDK vhost-events thread SPDK thread
SET_VRING_KICK VQ1 ----> Start polling VQ1 Reply to DPDK <---- Done SET_VRING_KICK VQ2 ----> thread is blocked on VQ's access lock,SPDK thread can't provide reply message
Quoted 1 line
Quoted 7 lines
Quoted 1 line
For example, we can just return for
SET_VRING_KICKVQ2 message withoutchecking SPDK thread, but this leave
Quoted 1 line
uncertain replies to VM.
I'm sorry but you will have to find a workaround while v22.11 is out and you can consume it. We can neither backport new API nor we can break all the other applications not handling locking failure.
By processing vhost-user message in asynchronous way in SPDK can be a workaround now, we can backport the workaround to SPDK earlier version so that it can work with distro DPDK releases.
Quoted 1 line
Quoted 5 lines
Quoted 4 lines
Regarding the new API for v22.11, I should be named something like
rte_vhost_vring_call_nonblock(), and ideally should return some like-EAGAINinstead of -1 o that the applications can distinguish between a real failure and a need for retry.Agreed, then we can switch to the new API finally.
Just a reminder that -rc2 is in ~ two weeks, have you prepared the patch adding the new API?
Regards,
Maxime
Quoted 16 lines
Quoted 15 lines
Regards,
MaximeQuoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Vhost-blk and scsi devices are not same with vhost-net, we need to cover
SeaBIOS and VM
Quoted 5 lines
cases, so we need to start processing vrings after 1 vring is ready.
Quoted 3 lines
Regards,
Maxime
PATCH v2 1/1 vhost: add new `rte_vhost_vring_call_nonblock` API
In reply to Changpeng LiuVhost-user library locks all VQ's access lock when processing vring based messages, such as SET_VRING_KICK and SET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.
Bugzilla ID: 1015
--- lib/vhost/rte_vhost.h | 15 +++++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+)
lib/vhost/rte_vhost.h +15 -0
@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_t vring_idx,*/ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This +* function acts as a memory barrier. This function will return -EAGAIN when +* vq's access lock is held by other thread, user should try again later. +* +* @param vid +* vhost device ID +* @param vring_idx +* vring index +* @return +* 0 on success, -1 on failure, -EAGAIN for another retry +*/ +__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. *
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {
lib/vhost/vhost.c +30 -0
@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) {
Signature · 1 line
2.21.3
Re: [PATCH v2] vhost: add new `rte_vhost_vring_call_nonblock` API
Hi Changpeng,
Quoted 89 lines · Changpeng Liu
Vhost-user library locks all VQ's access lock when processing vring based messages, such as
SET_VRING_KICKandSET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.Bugzilla ID: 1015
Fixes c5736998305d ("vhost: fix missing virtqueue lock protection")Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/rte_vhost.h | 15 +++++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+)lib/vhost/rte_vhost.h +15
-0@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_t vring_idx,*/ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This +* function acts as a memory barrier. This function will return -EAGAIN when +* vq's access lock is held by other thread, user should try again later. +* +* @param vid +* vhost device ID +* @param vring_idx +* vring index +* @return +* 0 on success, -1 on failure, -EAGAIN for another retry +*/ +__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. * lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {lib/vhost/vhost.c +23
-0@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else
lib/vhost/vhost.c rte_vhost_vring_call(int vid, uint16_t vring_idx) · line 1352
Quoted 10 lines
lib/vhost/vhost.c +7
-0+vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) {
Thanks,
Maxime
RE: [PATCH v2] vhost: add new `rte_vhost_vring_call_nonblock` API
In reply to Changpeng LiuHi Changpeng,
Quoted 18 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Wednesday, October 12, 2022 2:40 PM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; David Marchand <[email protected]>
Subject: [PATCH v2] vhost: add newrte_vhost_vring_call_nonblockAPIVhost-user library locks all VQ's access lock when processing vring based messages, such as
SET_VRING_KICKandSET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.Bugzilla ID: 1015
Quoted 8 lines
Fixes c5736998305d ("vhost: fix missing virtqueue lock protection")Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/rte_vhost.h | 15 +++++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+)
For new API, we need to update release_22_11.rst and vhost_lib.rst.
You can refer to http://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Thanks,
Chenbo
Quoted 87 lines
lib/vhost/rte_vhost.h +2
-0@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_tvring_idx, */ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This
+ * function acts as a memory barrier. This function will return-EAGAINwhen
+ * vq's access lock is held by other thread, user should try again later.
+ *
+ * @param vid
+ * vhost device ID
+ * @paramvring_idx
+ * vring index
+ * @return
+ * 0 on success, -1 on failure,-EAGAINfor another retry
+ */lib/vhost/rte_vhost.h +3
-0+__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. * lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {lib/vhost/vhost.c +30
-1@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) { --2.21.3
RE: [PATCH v2] vhost: add new `rte_vhost_vring_call_nonblock` API
Quoted 32 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, October 17, 2022 2:47 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>; David Marchand <[email protected]>
Subject: RE: [PATCH v2] vhost: add newrte_vhost_vring_call_nonblockAPIHi Changpeng,
Quoted 20 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Wednesday, October 12, 2022 2:40 PM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; David Marchand <[email protected]>
Subject: [PATCH v2] vhost: add newrte_vhost_vring_call_nonblockAPIVhost-user library locks all VQ's access lock when processing vring based messages, such as
SET_VRING_KICKandSET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.Bugzilla ID: 1015
Fixes c5736998305d ("vhost: fix missing virtqueue lock protection")
Quoted 8 lines
Quoted 6 lines
Signed-off-by Changpeng Liu <[email protected]>--- lib/vhost/rte_vhost.h | 15 +++++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+)For new API, we need to update
release_22_11.rstandvhost_lib.rst.
Thanks Chenbo, a new v3 is sent for review.
Quoted 95 lines
You can refer to
http://patchwork.dpdk.org/project/dpdk/patch/20221013092708.4922-2-
[email protected]/Thanks,
ChenboQuoted 87 lines
lib/vhost/rte_vhost.h +2
-0@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_tvring_idx, */ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This
+ * function acts as a memory barrier. This function will return-EAGAINwhen
+ * vq's access lock is held by other thread, user should try again later.
+ *
+ * @param vid
+ * vhost device ID
+ * @paramvring_idx
+ * vring index
+ * @return
+ * 0 on success, -1 on failure,-EAGAINfor another retry
+ */lib/vhost/rte_vhost.h +3
-0+__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. * lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {lib/vhost/vhost.c +30
-1@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) { --2.21.3
PATCH v3 1/1 vhost: add new `rte_vhost_vring_call_nonblock` API
In reply to Changpeng LiuVhost-user library locks all VQ's access lock when processing vring based messages, such as SET_VRING_KICK and SET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.
Bugzilla ID: 1015
--- doc/guides/prog_guide/vhost_lib.rst | 6 ++++++ doc/guides/rel_notes/release_22_11.rst | 6 ++++++ lib/vhost/rte_vhost.h | 15 +++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++ 5 files changed, 60 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +6 -0
@@ -297,6 +297,12 @@ The following is an overview of some key Vhost API functions: Clear in-flight packets which are submitted to async channel in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` + + Notify the guest that used descriptors have been added to the vring. This function + will return -EAGAIN when vq's access lock is held by other thread, user should try + again later. + * ``rte_vhost_vring_stats_get_names(int vid, uint16_t queue_id, struct rte_vhost_stat_name *names, unsigned int size)`` This function returns the names of the queue statistics. It requires
doc/guides/rel_notes/release_22_11.rst +6 -0
@@ -236,6 +236,12 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added non-block notify API to the vhost library.** + + Added API to notify the guest that used descriptors have been added + to the vring in non-block way, user should check the return value of + this API. + Removed Items -------------
lib/vhost/rte_vhost.h +15 -0
@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_t vring_idx,*/ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This +* function acts as a memory barrier. This function will return -EAGAIN when +* vq's access lock is held by other thread, user should try again later. +* +* @param vid +* vhost device ID +* @param vring_idx +* vring index +* @return +* 0 on success, -1 on failure, -EAGAIN for another retry +*/ +__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. *
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {
lib/vhost/vhost.c +30 -0
@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) {
Signature · 1 line
2.21.3
RE: [PATCH v3] vhost: add new `rte_vhost_vring_call_nonblock` API
Hi Changpeng,
Quoted 58 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Monday, October 17, 2022 3:15 PM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; David Marchand <[email protected]>
Subject: [PATCH v3] vhost: add newrte_vhost_vring_call_nonblockAPIVhost-user library locks all VQ's access lock when processing vring based messages, such as
SET_VRING_KICKandSET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.Bugzilla ID: 1015
Fixes c5736998305d ("vhost: fix missing virtqueue lock protection")Signed-off-by Changpeng Liu <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 6 ++++++ doc/guides/rel_notes/release_22_11.rst | 6 ++++++ lib/vhost/rte_vhost.h | 15 +++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++ 5 files changed, 60 insertions(+)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -297,6 +297,12 @@ The following is an overview of some key Vhost APIfunctions:
Clear in-flight packets which are submitted to async channel in vhost
async data
path. Completed packets are returned to applications through ``pkts``.doc/guides/prog_guide/vhost_lib.rst +3
-0+* ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` + + Notify the guest that used descriptors have been added to the vring.This function
+ will return-EAGAINwhen vq's access lock is held by other thread, user should trydoc/guides/prog_guide/vhost_lib.rst +2
-0+ again later. + * ``rte_vhost_vring_stats_get_names(int vid, uint16_t queue_id, struct
rte_vhost_stat_name*names, unsigned int size)``This function returns the names of the queue statistics. It requires
doc/guides/rel_notes/release_22_11.rst +0
-0
doc/guides/rel_notes/release_22_11.rst New Features · line 244
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +6
-0@@ -236,6 +236,12 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added non-block notify API to the vhost library.** + + Added API to notify the guest that used descriptors have been added + to the vring in non-block way, user should check the return value of + this API. +
This may be better:
* **Added non-blocking notify API to the vhost library.**
Added ``rte_vhost_vring_call_nonblock`` API to notify the guest that used descriptors have been added to the vring in non-blocking way. User should check the return value of this API and try again if needed.
And for new features, library should come before scripts, so you need to move this before ' Rewritten pmdinfo script. ' item.
Thanks,
Chenbo
Quoted 89 lines
doc/guides/rel_notes/release_22_11.rst +0
-0Removed Items -------------lib/vhost/rte_vhost.h +2
-0@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_tvring_idx, */ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This
+ * function acts as a memory barrier. This function will return-EAGAINwhen
+ * vq's access lock is held by other thread, user should try again later.
+ *
+ * @param vid
+ * vhost device ID
+ * @paramvring_idx
+ * vring index
+ * @return
+ * 0 on success, -1 on failure,-EAGAINfor another retry
+ */lib/vhost/rte_vhost.h +3
-0+__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. * lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {lib/vhost/vhost.c +30
-1@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) { --2.21.3
RE: [PATCH v3] vhost: add new `rte_vhost_vring_call_nonblock` API
Quoted 82 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, October 17, 2022 3:40 PM
To: Liu, Changpeng <[email protected]>; [email protected]
Cc: Maxime Coquelin <[email protected]>; David Marchand <[email protected]>
Subject: RE: [PATCH v3] vhost: add newrte_vhost_vring_call_nonblockAPIHi Changpeng,
Quoted 58 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Monday, October 17, 2022 3:15 PM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; David Marchand <[email protected]>
Subject: [PATCH v3] vhost: add newrte_vhost_vring_call_nonblockAPIVhost-user library locks all VQ's access lock when processing vring based messages, such as
SET_VRING_KICKandSET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.Bugzilla ID: 1015
Fixes c5736998305d ("vhost: fix missing virtqueue lock protection")Signed-off-by Changpeng Liu <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 6 ++++++ doc/guides/rel_notes/release_22_11.rst | 6 ++++++ lib/vhost/rte_vhost.h | 15 +++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++ 5 files changed, 60 insertions(+)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -297,6 +297,12 @@ The following is an overview of some key Vhost APIfunctions:
Clear in-flight packets which are submitted to async channel in vhost
async data
path. Completed packets are returned to applications through ``pkts``.doc/guides/prog_guide/vhost_lib.rst +3
-0+* ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` + + Notify the guest that used descriptors have been added to the vring.This function
+ will return-EAGAINwhen vq's access lock is held by other thread, user should trydoc/guides/prog_guide/vhost_lib.rst +2
-0+ again later. + * ``rte_vhost_vring_stats_get_names(int vid, uint16_t queue_id, struct
rte_vhost_stat_name*names, unsigned int size)``This function returns the names of the queue statistics. It requires
doc/guides/rel_notes/release_22_11.rst +0
-0doc/guides/rel_notes/release_22_11.rst New Features · line 244
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +6
-0@@ -236,6 +236,12 @@ New Features strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' +* **Added non-block notify API to the vhost library.** + + Added API to notify the guest that used descriptors have been added + to the vring in non-block way, user should check the return value of + this API. +This may be better:
Quoted 8 lines
* **Added non-blocking notify API to the vhost library.**
Added ``rte_vhost_vring_call_nonblock`` API to notify the guest that used descriptors have been added to the vring in non-blocking way. User should check the return value of this API and try again if needed.
And for new features, library should come before scripts, so you need to move this before ' Rewritten pmdinfo script. ' item.
Thanks Chenbo, applied them with v4.
Quoted 93 lines
Thanks,
ChenboQuoted 89 lines
doc/guides/rel_notes/release_22_11.rst +0
-0Removed Items -------------lib/vhost/rte_vhost.h +2
-0@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_tvring_idx, */ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This
+ * function acts as a memory barrier. This function will return-EAGAINwhen
+ * vq's access lock is held by other thread, user should try again later.
+ *
+ * @param vid
+ * vhost device ID
+ * @paramvring_idx
+ * vring index
+ * @return
+ * 0 on success, -1 on failure,-EAGAINfor another retry
+ */lib/vhost/rte_vhost.h +3
-0+__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. * lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {lib/vhost/vhost.c +30
-1@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) { --2.21.3
PATCH v4 1/1 vhost: add new `rte_vhost_vring_call_nonblock` API
In reply to Changpeng LiuVhost-user library locks all VQ's access lock when processing vring based messages, such as SET_VRING_KICK and SET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.
Bugzilla ID: 1015
--- doc/guides/prog_guide/vhost_lib.rst | 6 ++++++ doc/guides/rel_notes/release_22_11.rst | 6 ++++++ lib/vhost/rte_vhost.h | 15 +++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++ 5 files changed, 60 insertions(+)
doc/guides/prog_guide/vhost_lib.rst +6 -0
@@ -297,6 +297,12 @@ The following is an overview of some key Vhost API functions: Clear in-flight packets which are submitted to async channel in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` + + Notify the guest that used descriptors have been added to the vring. This function + will return -EAGAIN when vq's access lock is held by other thread, user should try + again later. + * ``rte_vhost_vring_stats_get_names(int vid, uint16_t queue_id, struct rte_vhost_stat_name *names, unsigned int size)`` This function returns the names of the queue statistics. It requires
doc/guides/rel_notes/release_22_11.rst +6 -0
@@ -225,6 +225,12 @@ New Features sysfs entries to adjust the minimum and maximum uncore frequency values, which works on Linux with Intel hardware only. +* **Added non-blocking notify API to the vhost library.** + + Added ``rte_vhost_vring_call_nonblock`` API to notify the guest that + used descriptors have been added to the vring in non-blocking way. User + should check the return value of this API and try again if needed. + * **Rewritten pmdinfo script.** The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
lib/vhost/rte_vhost.h +15 -0
@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_t vring_idx,*/ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This +* function acts as a memory barrier. This function will return -EAGAIN when +* vq's access lock is held by other thread, user should try again later. +* +* @param vid +* vhost device ID +* @param vring_idx +* vring index +* @return +* 0 on success, -1 on failure, -EAGAIN for another retry +*/ +__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. *
lib/vhost/version.map +3 -0
@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {
lib/vhost/vhost.c +30 -0
@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) {
Signature · 1 line
2.21.3
RE: [PATCH v4] vhost: add new `rte_vhost_vring_call_nonblock` API
Quoted 155 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Monday, October 17, 2022 3:48 PM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; David Marchand <[email protected]>
Subject: [PATCH v4] vhost: add newrte_vhost_vring_call_nonblockAPIVhost-user library locks all VQ's access lock when processing vring based messages, such as
SET_VRING_KICKandSET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.Bugzilla ID: 1015
Fixes c5736998305d ("vhost: fix missing virtqueue lock protection")Signed-off-by Changpeng Liu <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 6 ++++++ doc/guides/rel_notes/release_22_11.rst | 6 ++++++ lib/vhost/rte_vhost.h | 15 +++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++ 5 files changed, 60 insertions(+)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -297,6 +297,12 @@ The following is an overview of some key Vhost APIfunctions:
Clear in-flight packets which are submitted to async channel in vhost
async data
path. Completed packets are returned to applications through ``pkts``.doc/guides/prog_guide/vhost_lib.rst +3
-0+* ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` + + Notify the guest that used descriptors have been added to the vring.This function
+ will return-EAGAINwhen vq's access lock is held by other thread, user should trydoc/guides/prog_guide/vhost_lib.rst +2
-0+ again later. + * ``rte_vhost_vring_stats_get_names(int vid, uint16_t queue_id, struct
rte_vhost_stat_name*names, unsigned int size)``This function returns the names of the queue statistics. It requires
doc/guides/rel_notes/release_22_11.rst +6
-0@@ -225,6 +225,12 @@ New Features sysfs entries to adjust the minimum and maximum uncore frequency values, which works on Linux with Intel hardware only. +* **Added non-blocking notify API to the vhost library.** + + Added ``rte_vhost_vring_call_nonblock`` API to notify the guest that + used descriptors have been added to the vring in non-blocking way. User + should check the return value of this API and try again if needed. + * **Rewritten pmdinfo script.** The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.lib/vhost/rte_vhost.h +2
-0@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_tvring_idx, */ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This
+ * function acts as a memory barrier. This function will return-EAGAINwhen
+ * vq's access lock is held by other thread, user should try again later.
+ *
+ * @param vid
+ * vhost device ID
+ * @paramvring_idx
+ * vring index
+ * @return
+ * 0 on success, -1 on failure,-EAGAINfor another retry
+ */lib/vhost/rte_vhost.h +3
-0+__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. * lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {lib/vhost/vhost.c +30
-1@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) { --
Quoted 1 line
2.21.3
Xia, Chenbo
AppliedRE: [PATCH v4] vhost: add new `rte_vhost_vring_call_nonblock` API
In reply to Changpeng LiuQuoted 155 lines
-----Original Message-----
From: Liu, Changpeng <[email protected]>
Sent: Monday, October 17, 2022 3:48 PM
To: [email protected]
Cc: Liu, Changpeng <[email protected]>; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; David Marchand <[email protected]>
Subject: [PATCH v4] vhost: add newrte_vhost_vring_call_nonblockAPIVhost-user library locks all VQ's access lock when processing vring based messages, such as
SET_VRING_KICKandSET_VRING_CALL, and the data processing thread may already be started, e.g: SPDK vhost-blk and vhost-scsi will start the data processing thread when one vring is ready, then deadlock may happen when SPDK is posting interrupts to VM. Here, we add a new API which allows caller to try again later for this case.Bugzilla ID: 1015
Fixes c5736998305d ("vhost: fix missing virtqueue lock protection")Signed-off-by Changpeng Liu <[email protected]>--- doc/guides/prog_guide/vhost_lib.rst | 6 ++++++ doc/guides/rel_notes/release_22_11.rst | 6 ++++++ lib/vhost/rte_vhost.h | 15 +++++++++++++ lib/vhost/version.map | 3 +++ lib/vhost/vhost.c | 30 ++++++++++++++++++++++++++ 5 files changed, 60 insertions(+)doc/guides/prog_guide/vhost_lib.rst +0
-0@@ -297,6 +297,12 @@ The following is an overview of some key Vhost APIfunctions:
Clear in-flight packets which are submitted to async channel in vhost
async data
path. Completed packets are returned to applications through ``pkts``.doc/guides/prog_guide/vhost_lib.rst +3
-0+* ``rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx)`` + + Notify the guest that used descriptors have been added to the vring.This function
+ will return-EAGAINwhen vq's access lock is held by other thread, user should trydoc/guides/prog_guide/vhost_lib.rst +2
-0+ again later. + * ``rte_vhost_vring_stats_get_names(int vid, uint16_t queue_id, struct
rte_vhost_stat_name*names, unsigned int size)``This function returns the names of the queue statistics. It requires
doc/guides/rel_notes/release_22_11.rst +6
-0@@ -225,6 +225,12 @@ New Features sysfs entries to adjust the minimum and maximum uncore frequency values, which works on Linux with Intel hardware only. +* **Added non-blocking notify API to the vhost library.** + + Added ``rte_vhost_vring_call_nonblock`` API to notify the guest that + used descriptors have been added to the vring in non-blocking way. User + should check the return value of this API and try again if needed. + * **Rewritten pmdinfo script.** The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.lib/vhost/rte_vhost.h +2
-0@@ -909,6 +909,21 @@ rte_vhost_clr_inflight_desc_packed(int vid, uint16_tvring_idx, */ int rte_vhost_vring_call (int vid,uint16_t vring_idx); +/** +* Notify the guest that used descriptors have been added to the vring. This
+ * function acts as a memory barrier. This function will return-EAGAINwhen
+ * vq's access lock is held by other thread, user should try again later.
+ *
+ * @param vid
+ * vhost device ID
+ * @paramvring_idx
+ * vring index
+ * @return
+ * 0 on success, -1 on failure,-EAGAINfor another retry
+ */lib/vhost/rte_vhost.h +3
-0+__rte_experimental +int rte_vhost_vring_call_nonblock(int vid, uint16_t vring_idx); +/** * Get vhost RX queue avail count. * lib/vhost/version.map +3
-0@@ -94,6 +94,9 @@ EXPERIMENTAL { rte_vhost_async_try_dequeue_burst; rte_vhost_driver_get_vdpa_dev_type; rte_vhost_clear_queue; + + # added in 22.11 + rte_vhost_vring_call_nonblock; }; INTERNAL {lib/vhost/vhost.c +30
-1@@ -1317,6 +1317,36 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)return 0 ; } +int +rte_vhost_vring_call_nonblock (int vid,uint16_t vring_idx) +{ +struct virtio_net *dev; +struct vhost_virtqueue *vq; + + dev =get_device (vid); +if (!dev) +return -1 ; + +if (vring_idx >=VHOST_MAX_VRING ) +return -1 ; + + vq = dev->virtqueue[vring_idx]; +if (!vq) +return -1 ; + +if (!rte_spinlock_trylock (&vq->access_lock)) +return -EAGAIN ; + +if (vq_is_packed (dev)) +vhost_vring_call_packed (dev, vq); +else +vhost_vring_call_split (dev, vq); + +rte_spinlock_unlock (&vq->access_lock); + +return 0 ; +} +uint16_t rte_vhost_avail_entries (int vid,uint16_t queue_id) { --
Quoted 1 line
2.21.3
Change title to 'vhost: add non-blocking API for posting interrupt'
Applied to next-virtio/main. thanks