kni: Fix request overwritten

PATCH In review Elad Nachman 3 participants 15 messages Sep 24, 2021 – Oct 8, 2021

Share & tools

Elad Nachman

author

2021-09-24 10:54 UTC #0 lore raw

PATCH v2 1/1 kni: Fix request overwritten

Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.

Bugzilla ID: 809

Signed-off-by Elad Nachman <[email protected]>

---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)

kernel/linux/kni/kni_net.c +9 -0

index 611719b5ee..927bf9537c 100644
@@ -110,6 +110,7 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req)
 	void *resp_va;
 	uint32_t num;
 	int ret_val;
+	struct rte_kni_request *cur_req;
 
 	ASSERT_RTNL();

@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req)
 
 	mutex_lock(&kni->sync_lock);
 
+	/* Check that existing request has been processed: */
+	cur_req = (struct rte_kni_request *)kni->sync_kva;
+	if (cur_req->req_in_progress) {
+		ret = -EAGAIN;
+		goto fail;
+	}
+
 	/* Construct data */
+	req->req_in_progress = 1;
 	memcpy(kni->sync_kva, req, sizeof(struct rte_kni_request));
 	num = kni_fifo_put(kni->req_q, &kni->sync_va, 1);
 	if (num < 1) {

lib/kni/rte_kni.c +2 -0

index d3e236005e..0599e0356a 100644
@@ -307,6 +307,7 @@ rte_kni_alloc(struct rte_mempool *pktmbuf_pool,
 	kni->sync_addr = kni->m_sync_addr->addr;
 	dev_info.sync_va = kni->m_sync_addr->addr;
 	dev_info.sync_phys = kni->m_sync_addr->iova;
+	memset(kni->sync_addr, 0, sizeof(struct rte_kni_request));
 
 	kni->pktmbuf_pool = pktmbuf_pool;
 	kni->group_id = conf->group_id;

@@ -596,6 +597,7 @@ rte_kni_handle_request(struct rte_kni *kni)
 		ret = kni_fifo_put(kni->resp_q, (void **)&req, 1);
 	else
 		ret = 1;
+	req->req_in_progress = 0;
 	if (ret != 1) {
 		RTE_LOG(ERR, KNI, "Fail to put the muf back to resp_q\n");
 		return -1; /* It is an error of can't putting the mbuf back */

lib/kni/rte_kni_common.h +1 -0

index b547ea5501..1973e467f9 100644
@@ -40,6 +40,7 @@ enum rte_kni_req_id {
  */
 struct rte_kni_request {
 	uint32_t req_id;             /**< Request id */
+	uint32_t req_in_progress;    /**< Request in progress flag */
 	RTE_STD_C11
 	union {
 		uint32_t new_mtu;    /**< New MTU */

Signature · 1 line

2.17.1

Ferruh Yigit

2021-10-04 13:05 UTC #1 lore raw

Quoted 6 lines · Elad Nachman

Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809

Hi Eric,

Can you please test this patch, if it solves the issue you reported?

Quoted 8 lines

Signed-off-by Elad Nachman <[email protected]>

---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)

<...>

kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130

Quoted 8 lines

kernel/linux/kni/kni_net.c +4 -0

@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req)
 
 	mutex_lock(&kni->sync_lock);
 
+	/* Check that existing request has been processed: */
+	cur_req = (struct rte_kni_request *)kni->sync_kva;
+	if (cur_req->req_in_progress) {
+		ret = -EAGAIN;

Overall logic in the KNI looks good to me, this helps to serialize the requests even for async ones.

But can you please clarify how it behaves in the kernel side with '-EAGAIN' return type? Will linux call the ndo again, or will it just fail.

If it just fails should we handle the re-try on '-EAGAIN' within the kni module?

Elad Nachman

author

2021-10-04 13:10 UTC #2 lore raw

Hi,

EAGAIN is propogated back to the kernel and to the caller.

We cannot retry from the kni kernel module since we hold the rtnl lock.

FYI,

Elad

בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:

Quoted 44 lines · Elad Nachman

Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130
Quoted 7 lines
kernel/linux/kni/kni_net.c +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with '-EAGAIN' return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the kni module?

Ferruh Yigit

2021-10-04 14:11 UTC #3 lore raw

Quoted 4 lines · Elad Nachman

Hi,
EAGAIN is propogated back to the kernel and to the caller.

So will the user get an error, or it will be handled by the kernel and retried?

Quoted 2 lines

We cannot retry from the kni kernel module since we hold the rtnl lock.

Why not? We are already waiting until a command time out, like 'kni_net_open()' can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we can limit the number of retry for safety.

Quoted 51 lines

FYI,
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 44 lines · Elad Nachman
Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130
Quoted 7 lines
kernel/linux/kni/kni_net.c +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with '-EAGAIN' return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the kni module?

Elad Nachman

author

2021-10-04 14:25 UTC #4 lore raw

1. Userspace will get an error
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl unlocked for interface down command causes a crash because of a race condition in the device delete/unregister list in the kernel.

FYI,

Elad.

בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:

Quoted 72 lines · Elad Nachman

Quoted 4 lines
Hi,
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel and retried?
Quoted 2 lines
We cannot retry from the kni kernel module since we hold the rtnl lock.
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we can limit the
number of retry for safety.
Quoted 45 lines
FYI,
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 38 lines · Elad Nachman
Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130
Quoted 7 lines
kernel/linux/kni/kni_net.c +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 6 lines
Quoted 6 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the kni module?

Ferruh Yigit

2021-10-04 14:51 UTC #5 lore raw

On 10/4/2021 3:25 PM, Elad Nachman wrote:

Can you please try to not top post, it will make impossible to follow this discussion later from the mail archives.

Quoted 1 line

1. Userspace will get an error

So there is nothing special with returning '-EAGAIN', user will only observe an error.
Wasn't initial intention to use '-EAGAIN' to try request again?

Quoted 4 lines

2. Waiting with rtnl locked causes a deadlock; waiting with rtnl unlocked for interface down command causes a crash because of a race condition in the device delete/unregister list in the kernel.

Why waiting with rthnl lock causes a deadlock? As said below we are already doing it, why it is different with retry logic?

I agree to not wait with rtnl unlocked.

Quoted 79 lines

FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 72 lines · Elad Nachman
Quoted 4 lines
Hi,
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel and retried?
Quoted 2 lines
We cannot retry from the kni kernel module since we hold the rtnl lock.
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we can limit the
number of retry for safety.
Quoted 45 lines
FYI,
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 38 lines · Elad Nachman
Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130
Quoted 7 lines
kernel/linux/kni/kni_net.c +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 6 lines
Quoted 6 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the kni module?

Elad Nachman

author

2021-10-04 14:58 UTC #6 lore raw

בתאריך יום ב׳, 4 באוק׳ 2021, 17:51, מאת Ferruh Yigit ‏< [email protected]>:

Quoted 110 lines · Elad Nachman

Can you please try to not top post, it will make impossible to follow this discussion later from the mail archives.
Quoted 1 line
1. Userspace will get an error
So there is nothing special with returning '-EAGAIN', user will only observe an
error.
Wasn't initial intention to use '-EAGAIN' to try request again?
To signal user-space to retry the operation.
Quoted 4 lines
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl unlocked for interface down command causes a crash because of a race condition in the device delete/unregister list in the kernel.
Why waiting with rthnl lock causes a deadlock? As said below we are already doing it, why it is different with retry logic?
Because it can be interface down request.
I agree to not wait with rtnl unlocked.
Quoted 74 lines
FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 67 lines · Elad Nachman
Quoted 4 lines
Hi,
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel and retried?
Quoted 2 lines
We cannot retry from the kni kernel module since we hold the rtnl lock.
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we can limit the
number of retry for safety.
Quoted 45 lines
FYI,
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 38 lines · Elad Nachman
Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130
Quoted 7 lines
kernel/linux/kni/kni_net.c +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 3 lines
Quoted 3 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the
kni
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
module?
Elad.

Ferruh Yigit

2021-10-04 16:02 UTC #7 lore raw

Quoted 8 lines · Elad Nachman

בתאריך יום ב׳, 4 באוק׳ 2021, 17:51, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 5 lines · Elad Nachman
Can you please try to not top post, it will make impossible to follow this discussion later from the mail archives.

Quoted 9 lines

Quoted 7 lines
Quoted 1 line
1. Userspace will get an error
So there is nothing special with returning '-EAGAIN', user will only observe an
error.
Wasn't initial intention to use '-EAGAIN' to try request again?
To signal user-space to retry the operation.

Not sure if it will reach to the end user. If user is calling "ifconfig <iface> down", it will just fail right, it won't recognize the error type.

Unless this is common usage by the Linux network drivers, having this usage in KNI won't help much. I am for handling this in the kernel side if we can.

Quoted 11 lines

Quoted 9 lines
Quoted 5 lines
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl unlocked for interface down command causes a crash because of a race condition in the device delete/unregister list in the kernel.
Why waiting with rthnl lock causes a deadlock? As said below we are already doing it, why it is different with retry logic?
Because it can be interface down request.

(sure you like short answers)

Please help me to see why "interface down" is special. Isn't it point of your patch to wait the request execution in the userspace even it is an async request?

And yet again, number of retry can be limited.

Quoted 85 lines

Quoted 85 lines
I agree to not wait with rtnl unlocked.
Quoted 74 lines
FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 67 lines · Elad Nachman
Quoted 4 lines
Hi,
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel and retried?
Quoted 2 lines
We cannot retry from the kni kernel module since we hold the rtnl lock.
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we can limit the
number of retry for safety.
Quoted 45 lines
FYI,
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 38 lines · Elad Nachman
Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
Review kni_net_process_request(struct net_device *dev, · line 129
Quoted 7 lines
(quoted diff) +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 3 lines
Quoted 3 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the
kni
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
module?
Elad.

Elad Nachman

author

2021-10-04 16:19 UTC #8 lore raw

Quoted 20 lines · Ferruh Yigit

Quoted 6 lines · Elad Nachman
בתאריך יום ב׳, 4 באוק׳ 2021, 17:51, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 3 lines · Elad Nachman
Can you please try to not top post, it will make impossible to follow
this
Quoted 11 lines
Quoted 9 lines
discussion later from the mail archives.
Quoted 1 line
1. Userspace will get an error
So there is nothing special with returning '-EAGAIN', user will only observe an
error.
Wasn't initial intention to use '-EAGAIN' to try request again?
To signal user-space to retry the operation.

Quoted 9 lines

Not sure if it will reach to the end user. If user is calling "ifconfig <iface>
down", it will just fail right, it won't recognize the error type.
Unless this is common usage by the Linux network drivers, having this usage in
KNI won't help much. I am for handling this in the kernel side if we can.

If user calls ifconfig <iface> down it will not happen. It requires some multi-core race condition only Eric can recreate.

Quoted 15 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl
unlocked
Quoted 1 line
Quoted 1 line
Quoted 1 line
for interface down command causes a crash because of a race condition
in
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
the device delete/unregister list in the kernel.
Why waiting with rthnl lock causes a deadlock? As said below we are
already
Quoted 4 lines
Quoted 2 lines
doing it, why it is different with retry logic?
Because it can be interface down request.

Quoted 10 lines

(sure you like short answers)
Please help me to see why "interface down" is special. Isn't it point of your
patch to wait the request execution in the userspace even it is an async request?
And yet again, number of retry can be limited.

No, it is not. Please look again:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/

Quoted 91 lines

Quoted 14 lines
Quoted 14 lines
I agree to not wait with rtnl unlocked.
Quoted 10 lines
FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 3 lines · Elad Nachman
Quoted 2 lines
Hi,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel and retried?
Quoted 1 line
We cannot retry from the kni kernel module since we hold the rtnl
lock.
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 1 line
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we can limit the
number of retry for safety.
Quoted 28 lines
FYI,
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 21 lines · Elad Nachman
Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<...>
Review kni_net_process_request(struct net_device
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device
*dev,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct rte_kni_request *req)
Review kni_net_process_request(struct net_device · line 129
Quoted 7 lines
(quoted diff) +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize
the
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 3 lines
Quoted 3 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the
kni
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
module?
Elad.

Eric Christian

2021-10-04 17:00 UTC #9 lore raw

I am not sure that only we can recreate the KNI request overwrite. We may be the only ones with a current use case that exposes the vulnerability.
It is possible for any KNI operation to encounter this issue with the new
async mechanism. As long as the call to kni_net_process_request() is a separate thread from rte_kni_handle_request() this has the potential to occur with the use of async requests.

All you need is one async KNI request followed closely by a second KNI request before the rte_kni_handle_request() has had a chance to process the first request.

The kernel dev driver simply returns the error value back to the caller if it is less than zero.

Eric

Quoted 164 lines · Elad Nachman

Quoted 20 lines · Ferruh Yigit
Quoted 6 lines · Elad Nachman
בתאריך יום ב׳, 4 באוק׳ 2021, 17:51, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 3 lines · Elad Nachman
Can you please try to not top post, it will make impossible to follow
this
Quoted 11 lines
Quoted 9 lines
discussion later from the mail archives.
Quoted 1 line
1. Userspace will get an error
So there is nothing special with returning '-EAGAIN', user will only observe an
error.
Wasn't initial intention to use '-EAGAIN' to try request again?
To signal user-space to retry the operation.
Quoted 9 lines
Not sure if it will reach to the end user. If user is calling "ifconfig <iface>
down", it will just fail right, it won't recognize the error type.
Unless this is common usage by the Linux network drivers, having this usage in
KNI won't help much. I am for handling this in the kernel side if we can.
If user calls ifconfig <iface> down it will not happen. It requires some multi-core race condition only Eric can recreate.
Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl
unlocked
Quoted 1 line
Quoted 1 line
Quoted 1 line
for interface down command causes a crash because of a race condition
in
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
the device delete/unregister list in the kernel.
Why waiting with rthnl lock causes a deadlock? As said below we are
already
Quoted 4 lines
Quoted 2 lines
doing it, why it is different with retry logic?
Because it can be interface down request.
Quoted 10 lines
(sure you like short answers)
Please help me to see why "interface down" is special. Isn't it point of your
patch to wait the request execution in the userspace even it is an async request?
And yet again, number of retry can be limited.
No, it is not. Please look again:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/
Quoted 96 lines
Quoted 11 lines
Quoted 11 lines
I agree to not wait with rtnl unlocked.
Quoted 7 lines
FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Elad Nachman
Quoted 4 lines
Hi,
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
retried?
Quoted 1 line
We cannot retry from the kni kernel module since we hold the rtnl
lock.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we
can
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
limit the
number of retry for safety.
Quoted 2 lines
FYI,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 2 lines · Elad Nachman
Quoted 1 line
Fix lack of multiple KNI requests handling support by introducing
a
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you
reported?
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<...>
Review kni_net_process_request(struct net_device
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device
*dev,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct rte_kni_request *req)
Review kni_net_process_request(struct net_device · line 129
Quoted 7 lines
(quoted diff) +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize
the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 3 lines
Quoted 3 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within
the
Quoted 8 lines
Quoted 8 lines
kni
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
module?
Elad.

Elad Nachman

author

2021-10-04 18:27 UTC #10 lore raw

בתאריך יום ב׳, 4 באוק׳ 2021, 20:00, מאת Eric Christian ‏<[email protected]

Quoted 13 lines

:
I am not sure that only we can recreate the KNI request overwrite. We may be the only ones with a current use case that exposes the vulnerability.
It is possible for any KNI operation to encounter this issue with the new
async mechanism. As long as the call to kni_net_process_request() is a separate thread from rte_kni_handle_request() this has the potential to occur with the use of async requests.
All you need is one async KNI request followed closely by a second KNI request before the rte_kni_handle_request() has had a chance to process the first request.

Quoted 6 lines

The kernel dev driver simply returns the error value back to the caller if it is less than zero.
Eric

What I did in order to attempt to recreate this issue was:

Create a qemu VM with four cores.
Run the KNI sample application.

Then:

1. Run ifconfig vEth0_0 down followed immediately by ifconfig vEth0_0 up

2. Same as above but in a sample userspace C application which calls ioctl twice in a row to achieve the same.

Both did not recreate the problem.

The only way to recreate it IMHO is to either:
run the ioctl from two parallel threads or perhaps assign RT fifo priority
to the application calling ioctl in a row.

Elad

Quoted 167 lines · Elad Nachman

Quoted 165 lines
Quoted 20 lines · Ferruh Yigit
Quoted 6 lines · Elad Nachman
בתאריך יום ב׳, 4 באוק׳ 2021, 17:51, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 3 lines · Elad Nachman
Can you please try to not top post, it will make impossible to follow
this
Quoted 11 lines
Quoted 9 lines
discussion later from the mail archives.
Quoted 1 line
1. Userspace will get an error
So there is nothing special with returning '-EAGAIN', user will only observe an
error.
Wasn't initial intention to use '-EAGAIN' to try request again?
To signal user-space to retry the operation.
Quoted 9 lines
Not sure if it will reach to the end user. If user is calling "ifconfig <iface>
down", it will just fail right, it won't recognize the error type.
Unless this is common usage by the Linux network drivers, having this usage in
KNI won't help much. I am for handling this in the kernel side if we can.
If user calls ifconfig <iface> down it will not happen. It requires some multi-core race condition only Eric can recreate.
Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl
unlocked
Quoted 1 line
Quoted 1 line
Quoted 1 line
for interface down command causes a crash because of a race
condition in
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
the device delete/unregister list in the kernel.
Why waiting with rthnl lock causes a deadlock? As said below we are
already
Quoted 4 lines
Quoted 2 lines
doing it, why it is different with retry logic?
Because it can be interface down request.
Quoted 10 lines
(sure you like short answers)
Please help me to see why "interface down" is special. Isn't it point of your
patch to wait the request execution in the userspace even it is an async request?
And yet again, number of retry can be limited.
No, it is not. Please look again:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/
Quoted 96 lines
Quoted 11 lines
Quoted 11 lines
I agree to not wait with rtnl unlocked.
Quoted 7 lines
FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Elad Nachman
Quoted 4 lines
Hi,
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
retried?
Quoted 1 line
We cannot retry from the kni kernel module since we hold the rtnl
lock.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we
can
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
limit the
number of retry for safety.
Quoted 2 lines
FYI,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 2 lines · Elad Nachman
Quoted 1 line
Fix lack of multiple KNI requests handling support by
introducing a
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you
reported?
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<...>
Review kni_net_process_request(struct net_device
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device
*dev,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct rte_kni_request *req)
Review kni_net_process_request(struct net_device · line 129
Quoted 7 lines
(quoted diff) +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to
serialize the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 3 lines
Quoted 3 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within
the
Quoted 8 lines
Quoted 8 lines
kni
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
module?
Elad.

Ferruh Yigit

2021-10-08 20:23 UTC #11 lore raw

Quoted 34 lines · Elad Nachman

בתאריך יום ב׳, 4 באוק׳ 2021, 20:00, מאת Eric Christian ‏<[email protected]
Quoted 13 lines
:
I am not sure that only we can recreate the KNI request overwrite. We may be the only ones with a current use case that exposes the vulnerability.
It is possible for any KNI operation to encounter this issue with the new
async mechanism. As long as the call to kni_net_process_request() is a separate thread from rte_kni_handle_request() this has the potential to occur with the use of async requests.
All you need is one async KNI request followed closely by a second KNI request before the rte_kni_handle_request() has had a chance to process the first request.
Quoted 6 lines
The kernel dev driver simply returns the error value back to the caller if it is less than zero.
Eric
What I did in order to attempt to recreate this issue was:
Create a qemu VM with four cores.
Run the KNI sample application.
Then:
1. Run ifconfig vEth0_0 down followed immediately by ifconfig vEth0_0 up
2. Same as above but in a sample userspace C application which calls ioctl twice in a row to achieve the same.

Quoted 6 lines

Both did not recreate the problem.
The only way to recreate it IMHO is to either:
run the ioctl from two parallel threads or perhaps assign RT fifo priority
to the application calling ioctl in a row.

In kni sample app 'rte_kni_handle_request()' is called in endless loop in the datapath, this may be making it hard to reproduce the issue.

But what Eric described is valid problem and can happen. Reducing the frequency of the 'rte_kni_handle_request()' call can make it easy to reproduce.

Quoted 169 lines

Quoted 169 lines · Elad Nachman
Quoted 165 lines
Quoted 20 lines · Ferruh Yigit
Quoted 6 lines · Elad Nachman
בתאריך יום ב׳, 4 באוק׳ 2021, 17:51, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 3 lines · Elad Nachman
Can you please try to not top post, it will make impossible to follow
this
Quoted 11 lines
Quoted 9 lines
discussion later from the mail archives.
Quoted 1 line
1. Userspace will get an error
So there is nothing special with returning '-EAGAIN', user will only observe an
error.
Wasn't initial intention to use '-EAGAIN' to try request again?
To signal user-space to retry the operation.
Quoted 9 lines
Not sure if it will reach to the end user. If user is calling "ifconfig <iface>
down", it will just fail right, it won't recognize the error type.
Unless this is common usage by the Linux network drivers, having this usage in
KNI won't help much. I am for handling this in the kernel side if we can.
If user calls ifconfig <iface> down it will not happen. It requires some multi-core race condition only Eric can recreate.
Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl
unlocked
Quoted 1 line
Quoted 1 line
Quoted 1 line
for interface down command causes a crash because of a race
condition in
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
the device delete/unregister list in the kernel.
Why waiting with rthnl lock causes a deadlock? As said below we are
already
Quoted 4 lines
Quoted 2 lines
doing it, why it is different with retry logic?
Because it can be interface down request.
Quoted 10 lines
(sure you like short answers)
Please help me to see why "interface down" is special. Isn't it point of your
patch to wait the request execution in the userspace even it is an async request?
And yet again, number of retry can be limited.
No, it is not. Please look again:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/
Quoted 96 lines
Quoted 11 lines
Quoted 11 lines
I agree to not wait with rtnl unlocked.
Quoted 7 lines
FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Elad Nachman
Quoted 4 lines
Hi,
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
retried?
Quoted 1 line
We cannot retry from the kni kernel module since we hold the rtnl
lock.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we
can
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
limit the
number of retry for safety.
Quoted 2 lines
FYI,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 2 lines · Elad Nachman
Quoted 1 line
Fix lack of multiple KNI requests handling support by
introducing a
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you
reported?
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Signed-off-by Elad Nachman <[email protected]>
---
  kernel/linux/kni/kni_net.c | 9 +++++++++
  lib/kni/rte_kni.c          | 2 ++
  lib/kni/rte_kni_common.h   | 1 +
  3 files changed, 12 insertions(+)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<...>
Review kni_net_process_request(struct net_device
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device
*dev,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct rte_kni_request *req)
Review kni_net_process_request(struct net_device · line 129
Quoted 7 lines
(quoted diff) +4 -0
       mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to
serialize the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 3 lines
Quoted 3 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within
the
Quoted 8 lines
Quoted 8 lines
kni
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
module?
Elad.

Ferruh Yigit

2021-10-08 21:11 UTC #12 lore raw

In reply to Elad Nachman

Quoted 56 lines · Elad Nachman

Quoted 20 lines · Ferruh Yigit
Quoted 6 lines · Elad Nachman
בתאריך יום ב׳, 4 באוק׳ 2021, 17:51, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 3 lines · Elad Nachman
Can you please try to not top post, it will make impossible to follow
this
Quoted 11 lines
Quoted 9 lines
discussion later from the mail archives.
Quoted 1 line
1. Userspace will get an error
So there is nothing special with returning '-EAGAIN', user will only observe an
error.
Wasn't initial intention to use '-EAGAIN' to try request again?
To signal user-space to retry the operation.
Quoted 9 lines
Not sure if it will reach to the end user. If user is calling "ifconfig <iface>
down", it will just fail right, it won't recognize the error type.
Unless this is common usage by the Linux network drivers, having this usage in
KNI won't help much. I am for handling this in the kernel side if we can.
If user calls ifconfig <iface> down it will not happen. It requires some multi-core race condition only Eric can recreate.
Quoted 23 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2. Waiting with rtnl locked causes a deadlock; waiting with rtnl
unlocked
Quoted 1 line
Quoted 1 line
Quoted 1 line
for interface down command causes a crash because of a race condition
in
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
the device delete/unregister list in the kernel.
Why waiting with rthnl lock causes a deadlock? As said below we are
already
Quoted 4 lines
Quoted 2 lines
doing it, why it is different with retry logic?
Because it can be interface down request.
(sure you like short answers)
Please help me to see why "interface down" is special. Isn't it point of your
patch to wait the request execution in the userspace even it is an async request?

Quoted 6 lines

Quoted 3 lines
And yet again, number of retry can be limited.
No, it is not. Please look again:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/

Still not clear why not to handle EAGAIN within KNI module.

Also another problem is kernel relying userspace to continue processing requests is error prone, we need some escape mechanism, some kind of time out etc...

Anyway, I will send a patch disable bifurcated device support by default.

Quoted 93 lines

Quoted 93 lines
Quoted 16 lines
Quoted 16 lines
I agree to not wait with rtnl unlocked.
Quoted 10 lines
FYI,
Elad.
בתאריך יום ב׳, 4 באוק׳ 2021, 17:13, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 3 lines · Elad Nachman
Quoted 2 lines
Hi,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
EAGAIN is propogated back to the kernel and to the caller.
So will the user get an error, or it will be handled by the kernel and retried?
Quoted 1 line
We cannot retry from the kni kernel module since we hold the rtnl
lock.
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 1 line
Why not? We are already waiting until a command time out, like 'kni_net_open()'
can retry if 'kni_net_process_request()' returns '-EAGAIN'. And we can limit the
number of retry for safety.
Quoted 28 lines
FYI,
Elad
בתאריך יום ב׳, 4 באוק׳ 2021, 16:05, מאת Ferruh Yigit ‏< [email protected]>:
Quoted 21 lines · Elad Nachman
Quoted 6 lines
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
  kernel/linux/kni/kni_net.c | 9 +++++++++
  lib/kni/rte_kni.c          | 2 ++
  lib/kni/rte_kni_common.h   | 1 +
  3 files changed, 12 insertions(+)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
<...>
Review kni_net_process_request(struct net_device
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device
*dev,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct rte_kni_request *req)
Review kni_net_process_request(struct net_device · line 129
Quoted 7 lines
(quoted diff) +4 -0
       mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize
the
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
requests
even for async ones.
But can you please clarify how it behaves in the kernel side with
'-EAGAIN'
Quoted 3 lines
Quoted 3 lines
return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the
kni
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
module?
Elad.

Eric Christian

2021-10-04 14:14 UTC #13 lore raw

In reply to Ferruh Yigit

Adding Sahithi.

I believe adding the -EAGAIN method puts the responsibility on the application/caller. If we take the change MAC address as an example. Most application code just does this kind of check:

        ret = ioctl(sockfd, SIOCSIFHWADDR, &ifr);

        if (ret < 0) {
                PMD_LOG_ERRNO(ERR, "ioctl(SIOCSIFHWADDR) failed");
                return -EINVAL;
        }

So the existing application code will treat the -EAGAIN as a failure and not retry. Unless it is expected that the IOCTL can return -EAGAIN and the application decides to keep retrying?

We can try this, but we have temporarily patched out the async changes in our code as it was blocking QA due to
https://bugs.dpdk.org/show_bug.cgi?id=816

Eric

Quoted 44 lines · Ferruh Yigit

Quoted 6 lines · Elad Nachman
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130
Quoted 7 lines
kernel/linux/kni/kni_net.c +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with '-EAGAIN' return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the kni module?

Ferruh Yigit

2021-10-04 15:31 UTC #14 lore raw

Quoted 6 lines · Eric Christian

Adding Sahithi.
I believe adding the -EAGAIN method puts the responsibility on the application/caller. If we take the change MAC address as an example. Most application code just does this kind of check:

Quoted 7 lines

        ret = ioctl(sockfd, SIOCSIFHWADDR, &ifr);

        if (ret < 0) {
                PMD_LOG_ERRNO(ERR, "ioctl(SIOCSIFHWADDR) failed");
                return -EINVAL;
        }

I am not sure '-EAGAIN' should be handled by the userspace code. I assumed that kernel netdev layer will try again if ndo returns '-EAGAIN' but that seems not the case, so perhaps we can retry in the KNI kernel module. So the issue can be handled without the KNI module transparent to the user application.

Quoted 65 lines

So the existing application code will treat the -EAGAIN as a failure and not retry. Unless it is expected that the IOCTL can return -EAGAIN and the application decides to keep retrying?
We can try this, but we have temporarily patched out the async changes in our code as it was blocking QA due to
https://bugs.dpdk.org/show_bug.cgi?id=816
Eric
Quoted 44 lines · Ferruh Yigit
Quoted 6 lines · Elad Nachman
Fix lack of multiple KNI requests handling support by introducing a request in progress flag which will fail additional requests with EAGAIN return code if the original request has not been processed by user-space.
Bugzilla ID: 809
Hi Eric,
Can you please test this patch, if it solves the issue you reported?
Quoted 8 lines
Signed-off-by Elad Nachman <[email protected]>
---
 kernel/linux/kni/kni_net.c | 9 +++++++++
 lib/kni/rte_kni.c          | 2 ++
 lib/kni/rte_kni_common.h   | 1 +
 3 files changed, 12 insertions(+)
<...>
Review kni_net_process_request(struct net_device *dev,
Quoted 1 line
(quoted diff) +0 -0
@@ -123,7 +124,15 @@ kni_net_process_request(struct net_device *dev,
struct rte_kni_request *req)
kernel/linux/kni/kni_net.c kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) · line 130
Quoted 7 lines
kernel/linux/kni/kni_net.c +4 -0
      mutex_lock(&kni->sync_lock);
 
+     /* Check that existing request has been processed: */
+     cur_req = (struct rte_kni_request *)kni->sync_kva;
+     if (cur_req->req_in_progress) {
+             ret = -EAGAIN;
Overall logic in the KNI looks good to me, this helps to serialize the requests
even for async ones.
But can you please clarify how it behaves in the kernel side with '-EAGAIN' return type? Will linux call the ndo again, or will it just fail.
If it just fails should we handle the re-try on '-EAGAIN' within the kni module?