Skip to contentSkip to conversation

examples/ntb: remove redundant logic for dev close

PATCH In review Junfeng Guo 4 participants 11 messages Jun 14, 2023 – Jul 3, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 examples/ntb: remove redundant logic for dev close

During EAL cleanup stage, all bus devices are cleaned up properly. Based on this, there is no need to do extra device close process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

Thus, this patch just remove the redundant logic for device close. Ports will be closed at rte_eal_cleanup() in the main loop.

Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
---
 examples/ntb/ntb_fwd.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
examples/ntb/ntb_fwd.c +3 -5
index 585aad9d70..3385b4b73c 100644
@@ -177,13 +177,11 @@ cmd_quit_parsed(__rte_unused void *parsed_result,
 	rte_eal_mp_wait_lcore();
 	in_test = 0;
 
-	/* Stop traffic and Close port. */
+	/* Stop traffic only. */
 	rte_rawdev_stop(dev_id);
-	rte_rawdev_close(dev_id);
-	if (eth_port_id < RTE_MAX_ETHPORTS && fwd_mode == IOFWD) {
+	if (eth_port_id < RTE_MAX_ETHPORTS && fwd_mode == IOFWD)
 		rte_eth_dev_stop(eth_port_id);
-		rte_eth_dev_close(eth_port_id);
-	}
+	/* Ports will be closed at rte_eal_cleanup() in the main loop. */
 
 	cmdline_quit(cl);
 }
Signature · 1 line
2.34.1
Tested-by on v1 1/1
#1 lore raw
Quoted 17 lines

-----Original Message-----
From: Junfeng Guo <[email protected]>
Sent: Wednesday, June 14, 2023 1:14 PM
To: Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Guo, Junfeng <[email protected]>
Subject: [PATCH] examples/ntb: remove redundant logic for dev close

During EAL cleanup stage, all bus devices are cleaned up properly. Based on this, there is no need to do extra device close process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL:
Error disabling MSI-X interrupts for fd xx"

Quoted 8 lines

Thus, this patch just remove the redundant logic for device close. Ports will be closed at rte_eal_cleanup() in the main loop.

Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")

---

#2 lore raw
In reply to Junfeng Guo

14/06/2023 07:14, Junfeng Guo:

Quoted 7 lines

During EAL cleanup stage, all bus devices are cleaned up properly. Based on this, there is no need to do extra device close process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

You should fix ntb to not disable interrupt twice.

#3 lore raw
Quoted 17 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Wednesday, June 28, 2023 08:12
To: Wu, Jingjing <[email protected]>; Guo, Junfeng <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH] examples/ntb: remove redundant logic for dev close

14/06/2023 07:14, Junfeng Guo:

Quoted 4 lines

During EAL cleanup stage, all bus devices are cleaned up properly. Based on this, there is no need to do extra device close process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

Quoted 5 lines
Quoted 3 lines

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

You should fix ntb to not disable interrupt twice.

Thanks for the comment!

Yes, it is exactly what this patch did.

The interrupt disabling process is part of 'dev_close' ops, and will be called twice (one in EAL cleanup, another in ntb example cleanup).

Now that fixed & covered by the EAL cleanup, there is no need to run dev_close at ntb example. Just remove that redundant logic.

Regards,
Junfeng

Quoted 1 line
#4 lore raw
Quoted 34 lines

-----Original Message-----
From: Guo, Junfeng
Sent: Wednesday, June 28, 2023 10:19
To: Thomas Monjalon <[email protected]>; Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Richardson, Bruce <[email protected]>
Subject: RE: [PATCH] examples/ntb: remove redundant logic for dev close

Quoted 9 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Wednesday, June 28, 2023 08:12
To: Wu, Jingjing <[email protected]>; Guo, Junfeng <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH] examples/ntb: remove redundant logic for dev

close

Quoted 10 lines

14/06/2023 07:14, Junfeng Guo:

Quoted 7 lines

During EAL cleanup stage, all bus devices are cleaned up properly. Based on this, there is no need to do extra device close process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

Quoted 5 lines
Quoted 1 line

You should fix ntb to not disable interrupt twice.

Thanks for the comment!

Yes, it is exactly what this patch did.

Another solution is to add extra logic in dev_close, i.e., to do the interrupt disabling process only when the interrupt handler is valid. That is, to disable interrupt only when 'intr_handle->max_intr' or 'intr_handle->nb_efd' is non-zero.

But I'm not sure if this method is a better solution. Please help give some advice, thanks!

Quoted 11 lines

The interrupt disabling process is part of 'dev_close' ops, and will be called twice (one in EAL cleanup, another in ntb example cleanup).

Now that fixed & covered by the EAL cleanup, there is no need to run dev_close at ntb example. Just remove that redundant logic.

Regards,
Junfeng

Quoted 1 line
#5 lore raw

28/06/2023 04:37, Guo, Junfeng:

Quoted 17 lines

From: Guo, Junfeng

Quoted 10 lines

From: Thomas Monjalon <[email protected]>

Quoted 9 lines

14/06/2023 07:14, Junfeng Guo:

Quoted 7 lines

During EAL cleanup stage, all bus devices are cleaned up properly. Based on this, there is no need to do extra device close process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

Quoted 5 lines
Quoted 1 line

You should fix ntb to not disable interrupt twice.

Thanks for the comment!

Yes, it is exactly what this patch did.

Quoted 7 lines

Another solution is to add extra logic in dev_close, i.e., to do the interrupt disabling process only when the interrupt handler is valid. That is, to disable interrupt only when 'intr_handle->max_intr' or 'intr_handle->nb_efd' is non-zero.

But I'm not sure if this method is a better solution. Please help give some advice, thanks!

Yes this is what I suggest.
You cannot fix all applications in the world, but you can fix the driver, so it is a better way :)

Quoted 5 lines
Quoted 5 lines

The interrupt disabling process is part of 'dev_close' ops, and will be called twice (one in EAL cleanup, another in ntb example cleanup).

Now that fixed & covered by the EAL cleanup, there is no need to run dev_close at ntb example. Just remove that redundant logic.

#6 lore raw
Quoted 35 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Wednesday, June 28, 2023 15:53
To: Wu, Jingjing <[email protected]>; Guo, Junfeng <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Richardson, Bruce <[email protected]>
Subject: Re: [PATCH] examples/ntb: remove redundant logic for dev close

28/06/2023 04:37, Guo, Junfeng:

Quoted 22 lines

From: Guo, Junfeng

Quoted 10 lines

From: Thomas Monjalon <[email protected]>

Quoted 9 lines

14/06/2023 07:14, Junfeng Guo:

Quoted 7 lines

During EAL cleanup stage, all bus devices are cleaned up properly. Based on this, there is no need to do extra device close process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

Quoted 5 lines
Quoted 1 line

You should fix ntb to not disable interrupt twice.

Thanks for the comment!

Yes, it is exactly what this patch did.

Another solution is to add extra logic in dev_close, i.e., to do the interrupt disabling process only when the interrupt handler is valid. That is, to disable interrupt only when 'intr_handle->max_intr' or 'intr_handle->nb_efd' is non-zero.

Quoted 6 lines
Quoted 2 lines

But I'm not sure if this method is a better solution. Please help give some advice, thanks!

Yes this is what I suggest.
You cannot fix all applications in the world, but you can fix the driver, so it is a better way :)

Sure! Thanks for the advice!

Will update with this solution, thanks!

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

The interrupt disabling process is part of 'dev_close' ops, and will be called twice (one in EAL cleanup, another in ntb example cleanup).

Now that fixed & covered by the EAL cleanup, there is no need to run dev_close at ntb example. Just remove that redundant logic.

#7 lore raw

PATCH v2 1/1 raw/ntb: add check for disabling interrupt in dev close ops

In reply to Junfeng Guo

During EAL cleanup stage, all bus devices are cleaned up properly. In the meantime, the ntb example app will also do the device cleanup process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

Thus, this patch added the check process for disabling interrupt in dev_close ops, to ensure that interrupt only be disabled once.

Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
---
 drivers/raw/ntb/ntb.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/raw/ntb/ntb.c +5 -0
index 76e98fe515..0ed4c14592 100644
@@ -1045,6 +1045,11 @@ ntb_dev_close(struct rte_rawdev *dev)
 	hw->queue_pairs = 0;
 
 	intr_handle = hw->pci_dev->intr_handle;
+	/* Disable interrupt only once */
+	if (!rte_intr_nb_efd_get(intr_handle) &&
+	    !rte_intr_max_intr_get(intr_handle))
+		return 0;
+
 	/* Clean datapath event and vec mapping */
 	rte_intr_efd_disable(intr_handle);
 	rte_intr_vec_list_free(intr_handle);
Signature · 1 line
2.34.1
Tested-by on v2 1/1
#8 lore raw

RE: [PATCH v2] raw/ntb: add check for disabling interrupt in dev close ops

Quoted 21 lines

-----Original Message-----
From: Junfeng Guo <[email protected]>
Sent: Wednesday, June 28, 2023 5:12 PM
To: Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Guo, Junfeng <[email protected]>
Subject: [PATCH v2] raw/ntb: add check for disabling interrupt in dev close ops

During EAL cleanup stage, all bus devices are cleaned up properly. In the meantime, the ntb example app will also do the device cleanup process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL:
Error disabling MSI-X interrupts for fd xx"

Thus, this patch added the check process for disabling interrupt in dev_close ops, to ensure that interrupt only be disabled once.

Quoted 8 lines
Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
---
 drivers/raw/ntb/ntb.c | 5 +++++
 1 file changed, 5 insertions(+)
Acked-by on v2 1/1
#9 lore raw

RE: [PATCH v2] raw/ntb: add check for disabling interrupt in dev close ops

In reply to Junfeng Guo
Quoted 18 lines

-----Original Message-----
From: Guo, Junfeng <[email protected]>
Sent: Wednesday, June 28, 2023 5:12 PM
To: Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Guo, Junfeng <[email protected]>
Subject: [PATCH v2] raw/ntb: add check for disabling interrupt in dev close ops

During EAL cleanup stage, all bus devices are cleaned up properly. In the meantime, the ntb example app will also do the device cleanup process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

Quoted 7 lines

Thus, this patch added the check process for disabling interrupt in dev_close ops, to ensure that interrupt only be disabled once.

Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
#10 lore raw

Re: [PATCH v2] raw/ntb: add check for disabling interrupt in dev close ops

03/07/2023 08:53, Wu, Jingjing:

Quoted 20 lines
Quoted 20 lines

-----Original Message-----
From: Guo, Junfeng <[email protected]>
Sent: Wednesday, June 28, 2023 5:12 PM
To: Wu, Jingjing <[email protected]>
Cc: [email protected]; [email protected]; He, Xingguang <[email protected]>; Laatz, Kevin <[email protected]>; Guo, Junfeng <[email protected]>
Subject: [PATCH v2] raw/ntb: add check for disabling interrupt in dev close ops

During EAL cleanup stage, all bus devices are cleaned up properly. In the meantime, the ntb example app will also do the device cleanup process, which may call the dev ops '*dev->dev_ops->dev_close' twice.

If this dev ops for ntb was called twice, the interrupt handle for EAL will be disabled twice and will lead to error for the seconde time. Like this: "EAL: Error disabling MSI-X interrupts for fd xx"

Thus, this patch added the check process for disabling interrupt in dev_close ops, to ensure that interrupt only be disabled once.

Quoted 6 lines
Quoted 4 lines
Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")

Applied, thanks.