Skip to contentSkip to conversation

eal: prevent socket closure before MP sync

PATCH Superseded Yang Ming 6 participants 18 messages Mar 14, 2025 – Sep 22, 2025 Crypto API

lore
Share & tools
lore

Yang Ming

author
#0 lore raw

PATCH 1/1 eal: prevent socket closure before MP sync

The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.

Case situation:
eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.

Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:
ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);

Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
---
 lib/eal/linux/eal.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/eal/linux/eal.c +1 -1
index b1e63e37fc..73ea47b12d 100644
@@ -1318,11 +1318,11 @@ rte_eal_cleanup(void)
 		rte_memseg_walk(mark_freeable, NULL);
 
 	rte_service_finalize();
+	eal_bus_cleanup();
 #ifdef VFIO_PRESENT
 	vfio_mp_sync_cleanup();
 #endif
 	rte_mp_channel_cleanup();
-	eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
 	eal_mp_dev_hotplug_cleanup();
Signature · 1 line
2.34.1
Acked-by on v1 1/1
#1 lore raw
Quoted 29 lines · On Fri, 14 Mar 2025 18:36:38 +0800 Yang Ming <[email protected]>…

The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.

Case situation:
eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.

Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:
ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);

Quoted 5 lines
Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")

Looks good, could there be a test?

author
#2 lore raw
Quoted 34 lines · Stephen Hemminger

Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.

Quoted 29 lines · On Fri, 14 Mar 2025 18:36:38 +0800 Yang Ming <[email protected]>…

The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.

Case situation:
eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.

Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:
ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);

Quoted 9 lines
Quoted 5 lines
Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")

Looks good, could there be a test?

Hi Thanks Stephen,

We have some findings for this change in our product test line. I will do more investigation for this patch.

Brs,

Yang Ming

author
#3 lore raw
Quoted 45 lines · Yang Ming
Quoted 34 lines · Stephen Hemminger

Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.

Quoted 29 lines · On Fri, 14 Mar 2025 18:36:38 +0800 Yang Ming <[email protected]>…

The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.

Case situation:
eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.

Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:
ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);

Quoted 9 lines
Quoted 5 lines
Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")

Looks good, could there be a test?

Quoted 9 lines

Hi Thanks Stephen,

We have some findings for this change in our product test line. I will do more investigation for this patch.

Brs,

Yang Ming

Hi,

This patch has been tested and functions correctly under normal conditions. However, during testing, we observed some new error logs in specific cases. Upon investigation, we identified that these logs originate from a separate issue, which will be addressed in the next version of this patch. Additionally, a similar issue may affect FreeBSD, and I plan to include a fix for that as well in the upcoming patch series. Please note that the entire patch series has been thoroughly tested.

Brs,

Yang Ming

author
#4 lore raw

PATCH v2 1/2 eal: prevent socket closure before MP sync

In reply to Yang Ming

The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.

Case situation:
eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.

Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:
ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);

Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
---
 lib/eal/freebsd/eal.c | 2 +-
 lib/eal/linux/eal.c   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
lib/eal/freebsd/eal.c +1 -1
index d07cff8651..d81f12a7b1 100644
@@ -899,8 +899,8 @@ rte_eal_cleanup(void)
 	struct internal_config *internal_conf =
 		eal_get_internal_configuration();
 	rte_service_finalize();
-	rte_mp_channel_cleanup();
 	eal_bus_cleanup();
+	rte_mp_channel_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
 	rte_eal_alarm_cleanup();
lib/eal/linux/eal.c +1 -1
index b1e63e37fc..73ea47b12d 100644
@@ -1318,11 +1318,11 @@ rte_eal_cleanup(void)
 		rte_memseg_walk(mark_freeable, NULL);
 
 	rte_service_finalize();
+	eal_bus_cleanup();
 #ifdef VFIO_PRESENT
 	vfio_mp_sync_cleanup();
 #endif
 	rte_mp_channel_cleanup();
-	eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
 	eal_mp_dev_hotplug_cleanup();
Signature · 1 line
2.34.1
author
#5 lore raw

PATCH v2 2/2 crypto/ipsec_mb: fix QP release in secondary

From: myang <[email protected]>

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

---
 drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +5 -2
index 910efb1a97..50ee140ccd 100644
@@ -138,6 +138,7 @@ int
 ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
 {
 	struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+	uint16_t process_id = (uint16_t)getpid();
 
 	if (!qp)
 		return 0;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
 		rte_free(qp);
 		dev->data->queue_pairs[qp_id] = NULL;
 	} else { /* secondary process */
-		return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id,
-					NULL, 0, RTE_IPSEC_MB_MP_REQ_QP_FREE);
+		if (qp->qp_used_by_pid == process_id)
+			return ipsec_mb_secondary_qp_op(dev->data->dev_id,
+						qp_id, NULL, 0,
+						RTE_IPSEC_MB_MP_REQ_QP_FREE);
 	}
 	return 0;
 }
Signature · 1 line
2.34.1
author
#6 lore raw

Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Hi,

Quoted 35 lines · Yang Ming

From: myang <[email protected]>

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

---
  drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1 -0
index 910efb1a97..50ee140ccd 100644
@@ -138,6 +138,7 @@ int
  ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
  {
  	struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+	uint16_t process_id = (uint16_t)getpid();
 
  	if (!qp)
  		return 0;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
  		rte_free(qp);
  		dev->data->queue_pairs[qp_id] = NULL;

drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162

Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4 -2
  	} else { /* secondary process */
-		return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id,
-					NULL, 0, RTE_IPSEC_MB_MP_REQ_QP_FREE);
+		if (qp->qp_used_by_pid == process_id)
+			return ipsec_mb_secondary_qp_op(dev->data->dev_id,
+						qp_id, NULL, 0,
+						RTE_IPSEC_MB_MP_REQ_QP_FREE);
  	}
  	return 0;
  }

Hi Experts,

Is there any chance to review and accept this patch?

Brs,

Yang Ming

#7 lore raw

Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Hello,

Quoted 52 lines · Yang Ming

Hi,

Quoted 35 lines · Yang Ming

From: myang <[email protected]>

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

---
  drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1 -0
index 910efb1a97..50ee140ccd 100644
@@ -138,6 +138,7 @@ int
  ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
  {
      struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+     uint16_t process_id = (uint16_t)getpid();
 
      if (!qp)
              return 0;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
              rte_free(qp);
              dev->data->queue_pairs[qp_id] = NULL;

drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162

Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4 -2
      } else { /* secondary process */
-             return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id,
-                                     NULL, 0, RTE_IPSEC_MB_MP_REQ_QP_FREE);
+             if (qp->qp_used_by_pid == process_id)
+                     return ipsec_mb_secondary_qp_op(dev->data->dev_id,
+                                             qp_id, NULL, 0,
+                                             RTE_IPSEC_MB_MP_REQ_QP_FREE);
      }
      return 0;
  }

Hi Experts,

Quoted 1 line

Is there any chance to review and accept this patch?

Don't forget to Cc maintainers.

Signature · 1 line
David Marchand
#8 lore raw

Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

In reply to Yang Ming

Hi Yangming,

PID check is implemented here:
https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376

Can you share the steps to re-produce the error ?

Regards

Kai

________________________________
From: Yang Ming
Sent: Thursday, April 24, 2025 15:26
To: [email protected]
Subject: Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Hi,

Quoted 35 lines · Yang Ming

From: myang <[email protected]>

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

---
  drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1 -0
index 910efb1a97..50ee140ccd 100644
@@ -138,6 +138,7 @@ int
  ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
  {
       struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+     uint16_t process_id = (uint16_t)getpid();
 
       if (!qp)
               return 0;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
               rte_free(qp);
               dev->data->queue_pairs[qp_id] = NULL;

drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162

Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4 -2
       } else { /* secondary process */
-             return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id,
-                                     NULL, 0, RTE_IPSEC_MB_MP_REQ_QP_FREE);
+             if (qp->qp_used_by_pid == process_id)
+                     return ipsec_mb_secondary_qp_op(dev->data->dev_id,
+                                             qp_id, NULL, 0,
+                                             RTE_IPSEC_MB_MP_REQ_QP_FREE);
       }
       return 0;
  }

Hi Experts,

Is there any chance to review and accept this patch?

Brs,

Yang Ming

#9 lore raw

Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Quoted 69 lines · Kai

Hi Yangming,

PID check is implemented here:
https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376 <https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376>

Can you share the steps to re-produce the error ?

Regards

Kai

------------------------------------------------------------------------
*From:* Yang Ming
*Sent:* Thursday, April 24, 2025 15:26
*To:* [email protected]
*Subject:* Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Hi,

Quoted 15 lines · Yang Ming

From: myang <[email protected]>

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

Review

Quoted 6 lines

---
   drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--

   1 file changed, 5 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c

drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev,

Quoted 12 lines

index 910efb1a97..50ee140ccd 100644

drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1 -0
@@ -138,6 +138,7 @@ int
    ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
    {
         struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+     uint16_t process_id = (uint16_t)getpid();
 
         if (!qp)
                 return 0;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, 

uint16_t qp_id)

drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)

Quoted 5 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +0 -2
                 rte_free(qp);
                 dev->data->queue_pairs[qp_id] = NULL;
         } else { /* secondary process */
-             return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id,
-                                     NULL, 0, 

RTE_IPSEC_MB_MP_REQ_QP_FREE);

drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162

Quoted 7 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4 -0
+             if (qp->qp_used_by_pid == process_id)
+                     return ipsec_mb_secondary_qp_op(dev->data->dev_id,
+                                             qp_id, NULL, 0,
+ RTE_IPSEC_MB_MP_REQ_QP_FREE);
         }
         return 0;
    }
Quoted 8 lines

Hi Experts,

Is there any chance to review and accept this patch?

Brs,

Yang Ming

Hi,

David. Thanks for your feedback. I will Cc maintainers in next version.

Kai, Thanks for your feedback. Here's our test steps after applying the previous patch called "eal: prevent socket closure before MP sync" in this serious:
1. Start the primary process: Run the DPDK primary process with the IPsec MB crypto device.
2. Launch the secondary process: Start a DPDK secondary process using the same device parameters.
3. Exit the secondary normally.

On secondary exit, error logs show below as my commit log's description:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket: ipsec_mb_mp_msg

This message corresponds exactly to the PID check in the code you referenced:
if (qp->qp_used_by_pid != req_param->process_id) { CDEV_LOG_ERR("Unable to release qp_id=%d", qp_id);
    goto out;
}

In our view, this log indicates an abnormal condition: a secondary process should be able to release its own QPs without triggering an error during normal shutdown.

Thanks for your help.

Best,
Yang Ming

#10 lore raw

Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Hi Yangming,

How did you configure the queue pairs differently between the primary and secondary processes?

The application must call rte_cryptodev_queue_pair_setup() with a unique qp_id for each process. This implies that each process should receive distinct queue pair configurations at runtime. From what I can tell, it’s likely that the secondary process is still using the same queue pair as the primary process.

Regards

Kai

________________________________
From: Moses Young <[email protected]>
Sent: Monday, May 12, 2025 11:10
To: Ji, Kai <[email protected]>; Yang Ming <[email protected]>; [email protected] <[email protected]>
Subject: Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

On 5/7/2025 11:25 PM, Ji, Kai wrote:

Hi Yangming,

PID check is implemented here:
https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376

Can you share the steps to re-produce the error ?

Regards

Kai

________________________________
From: Yang Ming
Sent: Thursday, April 24, 2025 15:26
To: [email protected]<mailto:[email protected]>
Subject: Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Hi,

Quoted 35 lines · Yang Ming

From: myang <[email protected]><mailto:[email protected]>

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

---
  drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1 -0
index 910efb1a97..50ee140ccd 100644
@@ -138,6 +138,7 @@ int
  ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
  {
       struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+     uint16_t process_id = (uint16_t)getpid();
 
       if (!qp)
               return 0;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
               rte_free(qp);
               dev->data->queue_pairs[qp_id] = NULL;

drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162

Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4 -2
       } else { /* secondary process */
-             return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id,
-                                     NULL, 0, RTE_IPSEC_MB_MP_REQ_QP_FREE);
+             if (qp->qp_used_by_pid == process_id)
+                     return ipsec_mb_secondary_qp_op(dev->data->dev_id,
+                                             qp_id, NULL, 0,
+                                             RTE_IPSEC_MB_MP_REQ_QP_FREE);
       }
       return 0;
  }

Hi Experts,

Is there any chance to review and accept this patch?

Brs,

Yang Ming

Hi,

David. Thanks for your feedback. I will Cc maintainers in next version.

Kai, Thanks for your feedback. Here's our test steps after applying the previous patch called "eal: prevent socket closure before MP sync" in this serious:
1. Start the primary process: Run the DPDK primary process with the IPsec MB crypto device.
2. Launch the secondary process: Start a DPDK secondary process using the same device parameters.
3. Exit the secondary normally.

On secondary exit, error logs show below as my commit log's description:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket: ipsec_mb_mp_msg

This message corresponds exactly to the PID check in the code you referenced:

if (qp->qp_used_by_pid != req_param->process_id) {
    CDEV_LOG_ERR("Unable to release qp_id=%d", qp_id);
    goto out;
}

In our view, this log indicates an abnormal condition: a secondary process should be able to release its own QPs without triggering an error during normal shutdown.

Thanks for your help.

Best,
Yang Ming

#11 lore raw

Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

Quoted 125 lines · Kai

Hi Yangming,

How did you configure the queue pairs differently between the primary and secondary processes?

The application must call rte_cryptodev_queue_pair_setup() with a unique qp_id for each process. This implies that each process should receive distinct queue pair configurations at runtime. From what I can tell, it’s likely that the secondary process is still using the same queue pair as the primary process.

Regards

Kai

------------------------------------------------------------------------
*From:* Moses Young <[email protected]>
*Sent:* Monday, May 12, 2025 11:10
*To:* Ji, Kai <[email protected]>; Yang Ming <[email protected]>; [email protected] <[email protected]> *Subject:* Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary

On 5/7/2025 11:25 PM, Ji, Kai wrote:

    Hi Yangming,

    PID check is implemented here:
    https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376
    <https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376>

    Can you share the steps to re-produce the error ?

    Regards

    Kai

    ------------------------------------------------------------------------
    *From:* Yang Ming
    *Sent:* Thursday, April 24, 2025 15:26
    *To:* [email protected] <mailto:[email protected]>
    *Subject:* Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in
    secondary

    Hi,

    On 2025/4/7 13:25, Yang Ming wrote:
    > From: myang <[email protected]>
    <mailto:[email protected]>
    >
    > When a secondary process tries to release a queue pair (QP) that
    > does not belong to it, error logs occur:
    > CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release
    > qp_id=0
    > EAL: Message data is too long
    > EAL: Fail to handle message: ipsec_mb_mp_msg
    > EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
    > ipsec_mb_mp_msg
    >
    > This patch ensures that a secondary process only frees a QP if
    > it actually owns it, preventing conflicts and resolving the
    > issue.
    >
    > Signed-off-by: myang <[email protected]>
    <mailto:[email protected]>
    > ---
    >   drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--
    >   1 file changed, 5 insertions(+), 2 deletions(-)
    >
    > diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_ops.c
    b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c
    > index 910efb1a97..50ee140ccd 100644
    > --- a/drivers/crypto/ipsec_mb/ipsec_mb_ops.c
    > +++ b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c
    > @@ -138,6 +138,7 @@ int
    >   ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
    >   {
    >        struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
    > +     uint16_t process_id = (uint16_t)getpid();
    >
    >        if (!qp)
    >                return 0;
    > @@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev
    *dev, uint16_t qp_id)
    >                rte_free(qp);
    >                dev->data->queue_pairs[qp_id] = NULL;
    >        } else { /* secondary process */
    > -             return ipsec_mb_secondary_qp_op(dev->data->dev_id,
    qp_id,
    > -                                     NULL, 0,
    RTE_IPSEC_MB_MP_REQ_QP_FREE);
    > +             if (qp->qp_used_by_pid == process_id)
    > +                     return
    ipsec_mb_secondary_qp_op(dev->data->dev_id,
    > +                                             qp_id, NULL, 0,
    > + RTE_IPSEC_MB_MP_REQ_QP_FREE);
    >        }
    >        return 0;
    >   }

    Hi Experts,

    Is there any chance to review and accept this patch?

    Brs,

    Yang Ming

Hi,

David. Thanks for your feedback. I will Cc maintainers in next version.

Kai, Thanks for your feedback. Here's our test steps after applying the previous patch called "eal: prevent socket closure before MP sync" in this serious:
1. Start the primary process: Run the DPDK primary process with the IPsec MB crypto device.
2. Launch the secondary process: Start a DPDK secondary process using the same device parameters.
3. Exit the secondary normally.

On secondary exit, error logs show below as my commit log's description:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket: ipsec_mb_mp_msg

This message corresponds exactly to the PID check in the code you referenced:
if (qp->qp_used_by_pid != req_param->process_id) { CDEV_LOG_ERR("Unable to release qp_id=%d", qp_id);
    goto out;
}

Quoted 6 lines

In our view, this log indicates an abnormal condition: a secondary process should be able to release its own QPs without triggering an error during normal shutdown.

Thanks for your help.

Best,
Yang Ming

Hi Kai,

The queue pairs is shared between the primary and secondary processes. Even if each process calls rte_cryptodev_queue_pair_setup() with a unique qp_id, the primary and secondary still end up using the same shared queue pair structure, because cryptodev->data is shared between them.

From the code path, cryptodev->data is allocated in the primary via rte_cryptodev_data_alloc() (inside ipsec_mb_create-->rte_cryptodev_pmd_create-->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the same cryptodev->data, including nb_queue_pairs and queue_pairs[]. As a result, when the secondary process exits, ipsec_mb_remove() is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup-->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release-->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++)
    ipsec_mb_qp_release(cryptodev, qp_id);

This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned in previous mail.

Best regards,
Yang Ming

#12 lore raw

PATCH v3 1/2 eal: prevent socket closure before MP sync

In reply to Yang Ming

The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.

Case situation:
eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.

Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:
ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);

Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
---
 lib/eal/freebsd/eal.c | 2 +-
 lib/eal/linux/eal.c   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
lib/eal/freebsd/eal.c +1 -1
index c1ab8d86d2..f161ba0bac 100644
@@ -905,8 +905,8 @@ rte_eal_cleanup(void)
 	struct internal_config *internal_conf =
 		eal_get_internal_configuration();
 	rte_service_finalize();
-	rte_mp_channel_cleanup();
 	eal_bus_cleanup();
+	rte_mp_channel_cleanup();
 	rte_eal_alarm_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
lib/eal/linux/eal.c +1 -1
index 52efb8626b..d44950d563 100644
@@ -1330,11 +1330,11 @@ rte_eal_cleanup(void)
 		rte_memseg_walk(mark_freeable, NULL);
 
 	rte_service_finalize();
+	eal_bus_cleanup();
 #ifdef VFIO_PRESENT
 	vfio_mp_sync_cleanup();
 #endif
 	rte_mp_channel_cleanup();
-	eal_bus_cleanup();
 	rte_eal_alarm_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
Signature · 1 line
2.49.0.windows.1
#13 lore raw

PATCH v3 2/2 crypto/ipsec_mb: fix QP release in secondary

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

From the code path, cryptodev->data is allocated in the primary via rte_cryptodev_data_alloc() (inside ipsec_mb_create-->rte_cryptodev_pmd_create -->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the same cryptodev->data, including nb_queue_pairs and queue_pairs[].

As a result, when the secondary process exits, ipsec_mb_remove() is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup -->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release -->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++)

     ipsec_mb_qp_release(cryptodev, qp_id);

This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned above.

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

Fixes b35848bc01f6 ("crypto/ipsec_mb: add multi-process IPC request handler")
---
 drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +5 -2
index 910efb1a97..50ee140ccd 100644
@@ -138,6 +138,7 @@ int
 ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
 {
 	struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id];
+	uint16_t process_id = (uint16_t)getpid();
 
 	if (!qp)
 		return 0;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
 		rte_free(qp);
 		dev->data->queue_pairs[qp_id] = NULL;
 	} else { /* secondary process */
-		return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id,
-					NULL, 0, RTE_IPSEC_MB_MP_REQ_QP_FREE);
+		if (qp->qp_used_by_pid == process_id)
+			return ipsec_mb_secondary_qp_op(dev->data->dev_id,
+						qp_id, NULL, 0,
+						RTE_IPSEC_MB_MP_REQ_QP_FREE);
 	}
 	return 0;
 }
Signature · 1 line
2.49.0.windows.1
#14 lore raw

Re: [PATCH v3 2/2] crypto/ipsec_mb: fix QP release in secondary

Hi Experts,

is there any chance to review and accept this patch?

Brs,
Yang Ming

Acked-by on v3 2/2
#15 lore raw

Re: [PATCH v3 2/2] crypto/ipsec_mb: fix QP release in secondary

In reply to Yang Ming
Quoted 34 lines · Yang Ming

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

From the code path, cryptodev->data is allocated in the primary
via rte_cryptodev_data_alloc() (inside ipsec_mb_create-->rte_cryptodev_pmd_create -->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the same cryptodev->data, including nb_queue_pairs and queue_pairs[].

As a result, when the secondary process exits, ipsec_mb_remove() is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup -->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release -->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++)

      ipsec_mb_qp_release(cryptodev, qp_id);

This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned above.

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

Quoted 6 lines
Fixes b35848bc01f6 ("crypto/ipsec_mb: add multi-process IPC request handler")

---

Signature · 2 lines
Thanks,
Anatoly
#16 lore raw

Re: [PATCH v3 2/2] crypto/ipsec_mb: fix QP release in secondary

Quoted 36 lines · Anatoly
Quoted 34 lines · Yang Ming

When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:
ipsec_mb_mp_msg

From the code path, cryptodev->data is allocated in the primary
via rte_cryptodev_data_alloc() (inside ipsec_mb_create-->rte_cryptodev_pmd_create -->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the same cryptodev->data, including nb_queue_pairs and queue_pairs[].

As a result, when the secondary process exits, ipsec_mb_remove() is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup -->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release -->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++)

      ipsec_mb_qp_release(cryptodev, qp_id);

This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned above.

This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.

Quoted 6 lines
Quoted 5 lines
Fixes b35848bc01f6 ("crypto/ipsec_mb: add multi-process IPC request handler")

Series applied, thanks.

Signature · 1 line
David Marchand
Acked-by on v3 1/2
#17 lore raw
In reply to Yang Ming
Quoted 29 lines · Yang Ming

The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.

Case situation:
eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.

Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:
ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);

Quoted 5 lines
Fixes 1cab1a40ea9b ("bus: cleanup devices on shutdown")
Signature · 2 lines
Thanks,
Anatoly