Skip to contentSkip to conversation

net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

PATCH In review Thinh Tran 5 participants 11 messages Mar 7, 2022 – Mar 10, 2022 Network drivers

lore
Share & tools
lore

Thinh Tran

author
#0 lore raw

PATCH 1/1 net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

I hit a failure during ports drop queue RQ creation when my adapters
are on CPU socket ID 1 instead of socket ID 0:
....
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.0 (socket 1)
EAL: set_mempolicy failed: Invalid argument
mlx5_common: Failed to allocate memory for RQ.
mlx5_net: Port 0 drop queue RQ creation failed.
mlx5_net: Cannot create drop RX queue
mlx5_net: probe of PCI device 0020:01:00.0 aborted after encountering an error: Success
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.1 (socket 1)
EAL: set_mempolicy failed: Invalid argument
mlx5_common: Failed to allocate memory for RQ.
mlx5_net: Port 0 drop queue RQ creation failed.
mlx5_net: Cannot create drop RX queue
mlx5_net: probe of PCI device 0020:01:00.1 aborted after encountering an error: Success
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
...

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID.
Result with this patch:
......
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.0 (socket 1)
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.1 (socket 1)
TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected
......
Configuring Port 0 (socket 1)
Port 0: 0C:42:A1:ED:C1:20
Configuring Port 1 (socket 1)
Port 1: 0C:42:A1:ED:C1:21
Checking link statuses...
Done

---
 drivers/net/mlx5/mlx5_devx.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/mlx5/mlx5_devx.c +2 -0
index af106bda50..5ab092a259 100644
@@ -947,6 +947,8 @@ mlx5_rxq_devx_obj_drop_create(struct rte_eth_dev *dev)
 		rte_errno = ENOMEM;
 		goto error;
 	}
+	/* set the CPU socket ID where the rxq_ctrl was allocated */
+	rxq_ctrl->socket = socket_id;
 	rxq_obj->rxq_ctrl = rxq_ctrl;
 	rxq_ctrl->is_hairpin = false;
 	rxq_ctrl->sh = priv->sh;
Signature · 1 line
2.27.0
Acked-by on v1 1/1
#1 lore raw

From: Thinh Tran <[email protected]>

Quoted 38 lines

I hit a failure during ports drop queue RQ creation when my adapters are on CPU socket ID 1 instead of socket ID 0:
....
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.0 (socket 1)
EAL: set_mempolicy failed: Invalid argument
mlx5_common: Failed to allocate memory for RQ.
mlx5_net: Port 0 drop queue RQ creation failed.
mlx5_net: Cannot create drop RX queue
mlx5_net: probe of PCI device 0020:01:00.0 aborted after encountering an
error: Success
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.1 (socket 1)
EAL: set_mempolicy failed: Invalid argument
mlx5_common: Failed to allocate memory for RQ.
mlx5_net: Port 0 drop queue RQ creation failed.
mlx5_net: Cannot create drop RX queue
mlx5_net: probe of PCI device 0020:01:00.1 aborted after encountering an
error: Success
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
...

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID.
Result with this patch:
......
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.0 (socket 1)
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.1 (socket 1)
TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected ......
Configuring Port 0 (socket 1)
Port 0: 0C:42:A1:ED:C1:20
Configuring Port 1 (socket 1)
Port 1: 0C:42:A1:ED:C1:21
Checking link statuses...
Done

Quoted 2 lines
#2 lore raw
In reply to Thinh Tran

Hi Thinh,

Quoted 7 lines

-----Original Message-----
From: Thinh Tran <[email protected]>
Sent: Tuesday, March 8, 2022 12:53 AM
To: [email protected]
Cc: [email protected]; Thinh Tran <[email protected]>
Subject: [PATCH] net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

This is a fix, so it's better to start commit title with a fix:
Something like this maybe:
"net/mlx5: fix CPU socket ID for mlx5_rxq_ctrl"

Quoted 12 lines

I hit a failure during ports drop queue RQ creation when my adapters are on CPU socket ID 1 instead of socket ID 0:
....
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.0 (socket 1)
EAL: set_mempolicy failed: Invalid argument
mlx5_common: Failed to allocate memory for RQ.
mlx5_net: Port 0 drop queue RQ creation failed.
mlx5_net: Cannot create drop RX queue
mlx5_net: probe of PCI device 0020:01:00.0 aborted after encountering an
error: Success
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.1 (socket 1)
EAL: set_mempolicy failed: Invalid argument

Quoted 8 lines

mlx5_common: Failed to allocate memory for RQ.
mlx5_net: Port 0 drop queue RQ creation failed.
mlx5_net: Cannot create drop RX queue
mlx5_net: probe of PCI device 0020:01:00.1 aborted after encountering an
error: Success
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
...

It's better to describe the issue rather than showing an example alone, Maybe something like this:

The socket ID is used to determine the socket where to allocate memory for mlx5_rxq_ctrl, currently it's set to 0 by default which mistakenly leads to allocating the memory on socket 0 always.

Quoted 5 lines

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID.
Result with this patch:

This sets the correct CPU socket ID before memory allocation to use the correct socket ID.

Quoted 4 lines

......
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.0 (socket 1)
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0020:01:00.1 (socket 1)
TELEMETRY: No legacy callbacks, legacy socket not created

Quoted 9 lines

Interactive-mode selected
......
Configuring Port 0 (socket 1)
Port 0: 0C:42:A1:ED:C1:20
Configuring Port 1 (socket 1)
Port 1: 0C:42:A1:ED:C1:21
Checking link statuses...
Done

Missing:
Fixes tag:

Fixes 5ceb3a02b000 ("net/mlx5: move Rx queue DevX resource")

Missing Cc stable for backport.

Quoted 4 lines

---

Kindest regards,
Raslan Darawsheh

#3 lore raw

Hi Raslan,

Quoted 5 lines

Missing:
Fixes tag:

Fixes 5ceb3a02b000 ("net/mlx5: move Rx queue DevX resource")

I believe the bug originates from my earlier commit, not Xueming's one:

Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")
#4 lore raw
Quoted 14 lines

-----Original Message-----
From: Dmitry Kozlyuk <[email protected]>
Sent: Tuesday, March 8, 2022 2:23 PM
To: Raslan Darawsheh <[email protected]>
Cc: Thinh Tran <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH] net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

Hi Raslan,

Quoted 3 lines

Missing:
Fixes tag:

Quoted 6 lines
Quoted 2 lines
Fixes 5ceb3a02b000 ("net/mlx5: move Rx queue DevX resource")

I believe the bug originates from my earlier commit, not Xueming's one:

Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")

Yes I think you are correct, my mistake 😊

Kindest regards,
Raslan Darawsheh

#5 lore raw

Hi, Thinh

Thank you for the patch, the code looks OK to me, but commit message is not compliant:
- it should contain "fix" keyword in the title, like this:
"net/mlx5: fix CPU socket ID for Rx queue creation"
- could you, please, make problem description less personal and less wordy? "The default CPU socket ID was used while creating the Rx queue and this caused
creation failure in case if hardware was not resided on the default socket.

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID."
- please add tags:

Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")

With best regards,
Slava

Quoted 33 lines

-----Original Message-----
From: Raslan Darawsheh <[email protected]>
Sent: Tuesday, March 8, 2022 14:25
To: Dmitry Kozlyuk <[email protected]>
Cc: Thinh Tran <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH] net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

Quoted 14 lines

-----Original Message-----
From: Dmitry Kozlyuk <[email protected]>
Sent: Tuesday, March 8, 2022 2:23 PM
To: Raslan Darawsheh <[email protected]>
Cc: Thinh Tran <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH] net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

Hi Raslan,

Quoted 3 lines

Missing:
Fixes tag:

Quoted 6 lines
Quoted 2 lines
Fixes 5ceb3a02b000 ("net/mlx5: move Rx queue DevX resource")

I believe the bug originates from my earlier commit, not Xueming's one:

Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")

Yes I think you are correct, my mistake 😊

Kindest regards,
Raslan Darawsheh

author
#6 lore raw

Hi,

Quoted 13 lines · Slava Ovsiienko

Hi, Thinh

Thank you for the patch, the code looks OK to me, but commit message is not compliant:
- it should contain "fix" keyword in the title, like this:
"net/mlx5: fix CPU socket ID for Rx queue creation"
- could you, please, make problem description less personal and less wordy? "The default CPU socket ID was used while creating the Rx queue and this caused
creation failure in case if hardware was not resided on the default socket.

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID."
- please add tags:

Quoted 6 lines
Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")

With best regards,
Slava

I'll resubmit the patch with suggestions above.

Regards,
Thinh

Quoted 33 lines
Quoted 33 lines

-----Original Message-----
From: Raslan Darawsheh <[email protected]>
Sent: Tuesday, March 8, 2022 14:25
To: Dmitry Kozlyuk <[email protected]>
Cc: Thinh Tran <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH] net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

Quoted 14 lines

-----Original Message-----
From: Dmitry Kozlyuk <[email protected]>
Sent: Tuesday, March 8, 2022 2:23 PM
To: Raslan Darawsheh <[email protected]>
Cc: Thinh Tran <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH] net/mlx5: set correct CPU socket ID for mlx5_rxq_ctrl

Hi Raslan,

Quoted 3 lines

Missing:
Fixes tag:

Quoted 6 lines
Quoted 2 lines
Fixes 5ceb3a02b000 ("net/mlx5: move Rx queue DevX resource")

I believe the bug originates from my earlier commit, not Xueming's one:

Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")

Yes I think you are correct, my mistake 😊

Kindest regards,
Raslan Darawsheh

author
#7 lore raw

PATCH v2 1/1 net/mlx5: fix CPU socket ID for Rx queue creation

In reply to Thinh Tran

The default CPU socket ID was used while creating the Rx queue and this caused creation failure in case if hardware was not resided on the default socket.

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID.

v2: change patch description, no code change

Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")
---
 drivers/net/mlx5/mlx5_devx.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/mlx5/mlx5_devx.c +2 -0
index af106bda50..5ab092a259 100644
@@ -947,6 +947,8 @@ mlx5_rxq_devx_obj_drop_create(struct rte_eth_dev *dev)
 		rte_errno = ENOMEM;
 		goto error;
 	}
+	/* set the CPU socket ID where the rxq_ctrl was allocated */
+	rxq_ctrl->socket = socket_id;
 	rxq_obj->rxq_ctrl = rxq_ctrl;
 	rxq_ctrl->is_hairpin = false;
 	rxq_ctrl->sh = priv->sh;
Signature · 1 line
2.27.0
Acked-by on v2 1/1
#8 lore raw

RE: [PATCH v2] net/mlx5: fix CPU socket ID for Rx queue creation

Hi, Thinh

the line: "v2: change patch description, no code change" should not be in commit message body, it should be after "---" Besides this:

With best regards,
Slava

Quoted 47 lines

-----Original Message-----
From: Thinh Tran <[email protected]>
Sent: Wednesday, March 9, 2022 21:49
To: [email protected]
Cc: [email protected]; Slava Ovsiienko <[email protected]>; Raslan Darawsheh <[email protected]>; Dmitry Kozlyuk <[email protected]>; Thinh Tran <[email protected]>; [email protected]
Subject: [PATCH v2] net/mlx5: fix CPU socket ID for Rx queue creation

The default CPU socket ID was used while creating the Rx queue and this caused creation failure in case if hardware was not resided on the default socket.

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID.

v2: change patch description, no code change

Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")
---
 drivers/net/mlx5/mlx5_devx.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/mlx5/mlx5_devx.c +2 -0
index af106bda50..5ab092a259 100644
@@ -947,6 +947,8 @@ mlx5_rxq_devx_obj_drop_create(struct rte_eth_dev
 *dev)
 		rte_errno = ENOMEM;
 		goto error;
 	}
+	/* set the CPU socket ID where the rxq_ctrl was allocated */
+	rxq_ctrl->socket = socket_id;
 	rxq_obj->rxq_ctrl = rxq_ctrl;
 	rxq_ctrl->is_hairpin = false;

rxq_ctrl->sh = priv->sh;
--
2.27.0

#9 lore raw

RE: [PATCH v2] net/mlx5: fix CPU socket ID for Rx queue creation

Hi,

Quoted 8 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Thursday, March 10, 2022 11:23 AM
To: Thinh Tran <[email protected]>; [email protected]
Cc: [email protected]; Raslan Darawsheh <[email protected]>; Dmitry Kozlyuk <[email protected]>; [email protected]
Subject: RE: [PATCH v2] net/mlx5: fix CPU socket ID for Rx queue creation

Quoted 4 lines

Hi, Thinh

the line: "v2: change patch description, no code change" should not be in commit message body, it should be after "---"

I'll handle this before integration,
It's also missing the Ack from Matan from the previous version, I'll add it as well.

Quoted 6 lines

Besides this:

With best regards,
Slava

Kindest regards
Raslan Darawsheh

#10 lore raw

RE: [PATCH v2] net/mlx5: fix CPU socket ID for Rx queue creation

In reply to Thinh Tran

Hi,

Quoted 14 lines

-----Original Message-----
From: Thinh Tran <[email protected]>
Sent: Wednesday, March 9, 2022 9:49 PM
To: [email protected]
Cc: [email protected]; Slava Ovsiienko <[email protected]>; Raslan Darawsheh <[email protected]>; Dmitry Kozlyuk <[email protected]>; Thinh Tran <[email protected]>; [email protected]
Subject: [PATCH v2] net/mlx5: fix CPU socket ID for Rx queue creation

The default CPU socket ID was used while creating the Rx queue and this caused creation failure in case if hardware was not resided on the default socket.

Quoted 5 lines

The patch sets the correct CPU socket ID for the mlx5_rxq_ctrl before calling the mlx5_rxq_create_devx_rq_resources() which eventually calls mlx5_devx_rq_create() with correct CPU socket ID.

v2: change patch description, no code change

Removed this part,

Quoted 7 lines
Fixes bc5bee028ebc ("net/mlx5: create drop queue using DevX")

Added missing Ack from V1 from Matan,

Patch applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh