eal: prevent socket closure before MP sync
PATCH 1/1 eal: prevent socket closure before MP sync
The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.
Case situation:eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.
Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.
Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);
--- lib/eal/linux/eal.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/eal/linux/eal.c +1 -1
@@ -1318,11 +1318,11 @@ rte_eal_cleanup(void)rte_memseg_walk (mark_freeable,NULL );rte_service_finalize (); +eal_bus_cleanup ();#ifdef VFIO_PRESENT vfio_mp_sync_cleanup ();#endif rte_mp_channel_cleanup (); -eal_bus_cleanup ();rte_trace_save ();eal_trace_fini ();eal_mp_dev_hotplug_cleanup ();
Signature · 1 line
2.34.1
Quoted 29 lines · On Fri, 14 Mar 2025 18:36:38 +0800 Yang Ming <[email protected]>…
The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.
Case situation:
eal_bus_cleanuphas been added inrte_eal_cleanup. But for the secondary process,rte_eal_cleanupfirstly performsrte_mp_channel_cleanup, which closes socket file for the MP channel, makingmp_fdinvalid. Subsequently,eal_bus_cleanuptriggersvdev_cleanup, which callsmp_request_syncto send a message via the MP channel. Sincemp_fdis invalid, error logs occur.Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request/tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.Function call trace:
1.rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync->send_msg->sendmsg(mp_fd, &msgh, 0);
Quoted 5 lines
Signed-off-by Yang Ming <[email protected]>
Looks good, could there be a test?
Quoted 34 lines · Stephen Hemminger
Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.
Quoted 29 lines · On Fri, 14 Mar 2025 18:36:38 +0800 Yang Ming <[email protected]>…
The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.
Case situation:
eal_bus_cleanuphas been added inrte_eal_cleanup. But for the secondary process,rte_eal_cleanupfirstly performsrte_mp_channel_cleanup, which closes socket file for the MP channel, makingmp_fdinvalid. Subsequently,eal_bus_cleanuptriggersvdev_cleanup, which callsmp_request_syncto send a message via the MP channel. Sincemp_fdis invalid, error logs occur.Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request/tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.Function call trace:
1.rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync->send_msg->sendmsg(mp_fd, &msgh, 0);
Quoted 9 lines
Quoted 5 lines
Signed-off-by Yang Ming <[email protected]>Looks good, could there be a test?
Hi Thanks Stephen,
We have some findings for this change in our product test line. I will do more investigation for this patch.
Brs,
Yang Ming
Quoted 45 lines · Yang Ming
Quoted 34 lines · Stephen Hemminger
Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.
Quoted 29 lines · On Fri, 14 Mar 2025 18:36:38 +0800 Yang Ming <[email protected]>…
The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.
Case situation:
eal_bus_cleanuphas been added inrte_eal_cleanup. But for the secondary process,rte_eal_cleanupfirstly performsrte_mp_channel_cleanup, which closes socket file for the MP channel, makingmp_fdinvalid. Subsequently,eal_bus_cleanuptriggersvdev_cleanup, which callsmp_request_syncto send a message via the MP channel. Sincemp_fdis invalid, error logs occur.Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request/tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.Function call trace:
1.rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync->send_msg->sendmsg(mp_fd, &msgh, 0);Quoted 9 lines
Quoted 5 lines
Signed-off-by Yang Ming <[email protected]>Looks good, could there be a test?
Quoted 9 lines
Hi Thanks Stephen,
We have some findings for this change in our product test line. I will do more investigation for this patch.
Brs,
Yang Ming
Hi,
This patch has been tested and functions correctly under normal conditions. However, during testing, we observed some new error logs in specific cases. Upon investigation, we identified that these logs originate from a separate issue, which will be addressed in the next version of this patch. Additionally, a similar issue may affect FreeBSD, and I plan to include a fix for that as well in the upcoming patch series. Please note that the entire patch series has been thoroughly tested.
Brs,
Yang Ming
PATCH v2 1/2 eal: prevent socket closure before MP sync
In reply to Yang MingThe secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.
Case situation:eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.
Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.
Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);
--- lib/eal/freebsd/eal.c | 2 +- lib/eal/linux/eal.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
lib/eal/freebsd/eal.c +1 -1
@@ -899,8 +899,8 @@ rte_eal_cleanup(void)struct internal_config *internal_conf =eal_get_internal_configuration ();rte_service_finalize (); -rte_mp_channel_cleanup ();eal_bus_cleanup (); +rte_mp_channel_cleanup ();rte_trace_save ();eal_trace_fini ();rte_eal_alarm_cleanup ();
lib/eal/linux/eal.c +1 -1
@@ -1318,11 +1318,11 @@ rte_eal_cleanup(void)rte_memseg_walk (mark_freeable,NULL );rte_service_finalize (); +eal_bus_cleanup ();#ifdef VFIO_PRESENT vfio_mp_sync_cleanup ();#endif rte_mp_channel_cleanup (); -eal_bus_cleanup ();rte_trace_save ();eal_trace_fini ();eal_mp_dev_hotplug_cleanup ();
Signature · 1 line
2.34.1
PATCH v2 2/2 crypto/ipsec_mb: fix QP release in secondary
From: myang <[email protected]>
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msg
This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
--- drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +5 -2
@@ -138,6 +138,7 @@ int ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) {struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; +uint16_t process_id = (uint16_t )getpid ();if (!qp)return 0 ;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)rte_free (qp); dev->data->queue_pairs[qp_id] =NULL ; }else {/* secondary process */ -return ipsec_mb_secondary_qp_op (dev->data->dev_id, qp_id, -NULL ,0 ,RTE_IPSEC_MB_MP_REQ_QP_FREE ); +if (qp->qp_used_by_pid == process_id) +return ipsec_mb_secondary_qp_op (dev->data->dev_id, + qp_id,NULL ,0 , +RTE_IPSEC_MB_MP_REQ_QP_FREE ); }return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
Hi,
Quoted 35 lines · Yang Ming
From: myang <[email protected]>
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgThis patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
Signed-off-by myang <[email protected]>--- drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1
-0@@ -138,6 +138,7 @@ int ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) {struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; +uint16_t process_id = (uint16_t )getpid ();if (!qp)return 0 ;@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)rte_free (qp); dev->data->queue_pairs[qp_id] =NULL ;
drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162
Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4
-2}else {/* secondary process */ -return ipsec_mb_secondary_qp_op (dev->data->dev_id, qp_id, -NULL ,0 ,RTE_IPSEC_MB_MP_REQ_QP_FREE ); +if (qp->qp_used_by_pid == process_id) +return ipsec_mb_secondary_qp_op (dev->data->dev_id, + qp_id,NULL ,0 , +RTE_IPSEC_MB_MP_REQ_QP_FREE ); }return 0 ; }
Hi Experts,
Is there any chance to review and accept this patch?
Brs,
Yang Ming
Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
Hello,
Quoted 52 lines · Yang Ming
Hi,
Quoted 35 lines · Yang Ming
From: myang <[email protected]>
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgThis patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
Signed-off-by myang <[email protected]>--- drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1
-0@@ -138,6 +138,7 @@ int ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) {struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; +uint16_t process_id = (uint16_t )getpid ();if (!qp)return 0 ;@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)rte_free (qp); dev->data->queue_pairs[qp_id] =NULL ;drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162
Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4
-2}else {/* secondary process */ -return ipsec_mb_secondary_qp_op (dev->data->dev_id, qp_id, -NULL ,0 ,RTE_IPSEC_MB_MP_REQ_QP_FREE ); +if (qp->qp_used_by_pid == process_id) +return ipsec_mb_secondary_qp_op (dev->data->dev_id, + qp_id,NULL ,0 , +RTE_IPSEC_MB_MP_REQ_QP_FREE ); }return 0 ; }Hi Experts,
Quoted 1 line
Is there any chance to review and accept this patch?
Don't forget to Cc maintainers.
Signature · 1 line
David Marchand
Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
In reply to Yang MingHi Yangming,
PID check is implemented here:
https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376
Can you share the steps to re-produce the error ?
Regards
Kai
________________________________
From: Yang Ming
Sent: Thursday, April 24, 2025 15:26
To: [email protected]
Subject: Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
Hi,
Quoted 35 lines · Yang Ming
From: myang <[email protected]>
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgThis patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
Signed-off-by myang <[email protected]>--- drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1
-0@@ -138,6 +138,7 @@ int ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) {struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; +uint16_t process_id = (uint16_t )getpid ();if (!qp)return 0 ;@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)rte_free (qp); dev->data->queue_pairs[qp_id] =NULL ;
drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162
Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4
-2}else {/* secondary process */ -return ipsec_mb_secondary_qp_op (dev->data->dev_id, qp_id, -NULL ,0 ,RTE_IPSEC_MB_MP_REQ_QP_FREE ); +if (qp->qp_used_by_pid == process_id) +return ipsec_mb_secondary_qp_op (dev->data->dev_id, + qp_id,NULL ,0 , +RTE_IPSEC_MB_MP_REQ_QP_FREE ); }return 0 ; }
Hi Experts,
Is there any chance to review and accept this patch?
Brs,
Yang Ming
Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
Quoted 69 lines · Kai
Hi Yangming,
PID check is implemented here:
https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376 <https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376>Can you share the steps to re-produce the error ?
Regards
Kai
------------------------------------------------------------------------
*From:* Yang Ming
*Sent:* Thursday, April 24, 2025 15:26
*To:* [email protected]
*Subject:* Re: [PATCH v2 2/2]crypto/ipsec_mb: fix QP release in secondaryHi,
Quoted 15 lines · Yang Ming
From: myang <[email protected]>
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgThis patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
Review
Quoted 6 lines
Signed-off-by myang <[email protected]>---
drivers/crypto/ipsec_mb/ipsec_mb_ops.c| 7 +++++--1 file changed, 5 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/crypto/ipsec_mb/ipsec_mb_ops.cdrivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev,
Quoted 12 lines
index
910efb1a97..50ee140ccd100644drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1
-0@@ -138,6 +138,7 @@ int ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) { struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; + uint16_t process_id = (uint16_t)getpid(); if (!qp) return 0;@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev,
uint16_tqp_id)drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)
Quoted 5 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +0
-2rte_free(qp); dev->data->queue_pairs[qp_id] = NULL; } else { /* secondary process */ - return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id, - NULL, 0,
RTE_IPSEC_MB_MP_REQ_QP_FREE);drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162
Quoted 7 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4
-0+ if (qp->qp_used_by_pid == process_id) + return ipsec_mb_secondary_qp_op(dev->data->dev_id, + qp_id, NULL, 0, +RTE_IPSEC_MB_MP_REQ_QP_FREE ); } return 0; }
Quoted 8 lines
Hi Experts,
Is there any chance to review and accept this patch?
Brs,
Yang Ming
Hi,
David. Thanks for your feedback. I will Cc maintainers in next version.
Kai, Thanks for your feedback. Here's our test steps after applying the previous patch called "eal: prevent socket closure before MP sync" in this serious:
1. Start the primary process: Run the DPDK primary process with the IPsec MB crypto device.
2. Launch the secondary process: Start a DPDK secondary process using the same device parameters.
3. Exit the secondary normally.
On secondary exit, error logs show below as my commit log's description:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket: ipsec_mb_mp_msg
This message corresponds exactly to the PID check in the code you referenced:
if (qp->qp_used_by_pid != req_param->process_id) { CDEV_LOG_ERR("Unable to release qp_id=%d", qp_id);
goto out;
}
In our view, this log indicates an abnormal condition: a secondary process should be able to release its own QPs without triggering an error during normal shutdown.
Thanks for your help.
Best,
Yang Ming
Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
Hi Yangming,
How did you configure the queue pairs differently between the primary and secondary processes?
The application must call rte_cryptodev_queue_pair_setup() with a unique qp_id for each process. This implies that each process should receive distinct queue pair configurations at runtime. From what I can tell, it’s likely that the secondary process is still using the same queue pair as the primary process.
Regards
Kai
________________________________
From: Moses Young <[email protected]>
Sent: Monday, May 12, 2025 11:10
To: Ji, Kai <[email protected]>; Yang Ming <[email protected]>; [email protected] <[email protected]>
Subject: Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
On 5/7/2025 11:25 PM, Ji, Kai wrote:
Hi Yangming,
PID check is implemented here:
https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376
Can you share the steps to re-produce the error ?
Regards
Kai
________________________________
From: Yang Ming
Sent: Thursday, April 24, 2025 15:26
To: [email protected]<mailto:[email protected]>
Subject: Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
Hi,
Quoted 35 lines · Yang Ming
From: myang <[email protected]><mailto:[email protected]>
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgThis patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
Signed-off-by myang <[email protected]><mailto:[email protected]>--- drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)drivers/crypto/ipsec_mb/ipsec_mb_ops.c +1
-0@@ -138,6 +138,7 @@ int ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) {struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; +uint16_t process_id = (uint16_t )getpid ();if (!qp)return 0 ;@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)rte_free (qp); dev->data->queue_pairs[qp_id] =NULL ;
drivers/crypto/ipsec_mb/ipsec_mb_ops.c ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) · line 162
Quoted 10 lines
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +4
-2}else {/* secondary process */ -return ipsec_mb_secondary_qp_op (dev->data->dev_id, qp_id, -NULL ,0 ,RTE_IPSEC_MB_MP_REQ_QP_FREE ); +if (qp->qp_used_by_pid == process_id) +return ipsec_mb_secondary_qp_op (dev->data->dev_id, + qp_id,NULL ,0 , +RTE_IPSEC_MB_MP_REQ_QP_FREE ); }return 0 ; }
Hi Experts,
Is there any chance to review and accept this patch?
Brs,
Yang Ming
Hi,
David. Thanks for your feedback. I will Cc maintainers in next version.
Kai, Thanks for your feedback. Here's our test steps after applying the previous patch called "eal: prevent socket closure before MP sync" in this serious:
1. Start the primary process: Run the DPDK primary process with the IPsec MB crypto device.
2. Launch the secondary process: Start a DPDK secondary process using the same device parameters.
3. Exit the secondary normally.
On secondary exit, error logs show below as my commit log's description:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket: ipsec_mb_mp_msg
This message corresponds exactly to the PID check in the code you referenced:
if (qp->qp_used_by_pid != req_param->process_id) {
CDEV_LOG_ERR("Unable to release qp_id=%d", qp_id);
goto out;
}In our view, this log indicates an abnormal condition: a secondary process should be able to release its own QPs without triggering an error during normal shutdown.
Thanks for your help.
Best,
Yang Ming
Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary
Quoted 125 lines · Kai
Hi Yangming,
How did you configure the queue pairs differently between the primary and secondary processes?
The application must call
rte_cryptodev_queue_pair_setup()with a uniqueqp_idfor each process. This implies that each process should receive distinct queue pair configurations at runtime. From what I can tell, it’s likely that the secondary process is still using the same queue pair as the primary process.Regards
Kai
------------------------------------------------------------------------
*From:* Moses Young <[email protected]>
*Sent:* Monday, May 12, 2025 11:10
*To:* Ji, Kai <[email protected]>; Yang Ming <[email protected]>; [email protected] <[email protected]> *Subject:* Re: [PATCH v2 2/2]crypto/ipsec_mb: fix QP release in secondaryOn 5/7/2025 11:25 PM, Ji, Kai wrote:
Hi Yangming, PID check is implemented here: https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376 <https://github.com/DPDK/dpdk/blob/75f179ebe347b6098cf3af26d3d3b7168fe3fe24/drivers/crypto/ipsec_mb/ipsec_mb_ops.c#L376> Can you share the steps to re-produce the error ? Regards Kai ------------------------------------------------------------------------ *From:* Yang Ming *Sent:* Thursday, April 24, 2025 15:26 *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [PATCH v2 2/2] crypto/ipsec_mb: fix QP release in secondary Hi, On 2025/4/7 13:25, Yang Ming wrote: > From: myang <[email protected]> <mailto:[email protected]> > > When a secondary process tries to release a queue pair (QP) that > does not belong to it, error logs occur: > CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release > qp_id=0 > EAL: Message data is too long > EAL: Fail to handle message: ipsec_mb_mp_msg > EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket: > ipsec_mb_mp_msg > > This patch ensures that a secondary process only frees a QP if > it actually owns it, preventing conflicts and resolving the > issue. > > Signed-off-by: myang <[email protected]> <mailto:[email protected]> > --- > drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > > diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_ops.c b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c > index 910efb1a97..50ee140ccd 100644 > --- a/drivers/crypto/ipsec_mb/ipsec_mb_ops.c > +++ b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c > @@ -138,6 +138,7 @@ int > ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) > { > struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; > + uint16_t process_id = (uint16_t)getpid(); > > if (!qp) > return 0; > @@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) > rte_free(qp); > dev->data->queue_pairs[qp_id] = NULL; > } else { /* secondary process */ > - return ipsec_mb_secondary_qp_op(dev->data->dev_id, qp_id, > - NULL, 0, RTE_IPSEC_MB_MP_REQ_QP_FREE); > + if (qp->qp_used_by_pid == process_id) > + return ipsec_mb_secondary_qp_op(dev->data->dev_id, > + qp_id, NULL, 0, > + RTE_IPSEC_MB_MP_REQ_QP_FREE); > } > return 0; > } Hi Experts, Is there any chance to review and accept this patch? Brs, Yang MingHi,
David. Thanks for your feedback. I will Cc maintainers in next version.
Kai, Thanks for your feedback. Here's our test steps after applying the previous patch called "eal: prevent socket closure before MP sync" in this serious:
1. Start the primary process: Run the DPDK primary process with the IPsec MB crypto device.
2. Launch the secondary process: Start a DPDK secondary process using the same device parameters.
3. Exit the secondary normally.On secondary exit, error logs show below as my commit log's description:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgThis message corresponds exactly to the PID check in the code you referenced:
if (qp->qp_used_by_pid!=req_param->process_id) {CDEV_LOG_ERR("Unable to release qp_id=%d", qp_id);
goto out;
}
Quoted 6 lines
In our view, this log indicates an abnormal condition: a secondary process should be able to release its own QPs without triggering an error during normal shutdown.
Thanks for your help.
Best,
Yang Ming
Hi Kai,
The queue pairs is shared between the primary and secondary processes. Even if each process calls rte_cryptodev_queue_pair_setup() with a unique qp_id, the primary and secondary still end up using the same shared queue pair structure, because cryptodev->data is shared between them.
From the code path, cryptodev->data is allocated in the primary via rte_cryptodev_data_alloc() (inside ipsec_mb_create-->rte_cryptodev_pmd_create-->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the same cryptodev->data, including nb_queue_pairs and queue_pairs[]. As a result, when the secondary process exits, ipsec_mb_remove() is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup-->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release-->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++)
ipsec_mb_qp_release(cryptodev, qp_id);
This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned in previous mail.
Best regards,
Yang Ming
PATCH v3 1/2 eal: prevent socket closure before MP sync
In reply to Yang MingThe secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.
Case situation:eal_bus_cleanup has been added in rte_eal_cleanup. But for the secondary process, rte_eal_cleanup firstly performs rte_mp_channel_cleanup, which closes socket file for the MP channel, making mp_fd invalid. Subsequently, eal_bus_cleanup triggers vdev_cleanup, which calls mp_request_sync to send a message via the MP channel. Since mp_fd is invalid, error logs occur.
Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.
Function call trace:
1. rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync-> send_msg->sendmsg(mp_fd, &msgh, 0);
--- lib/eal/freebsd/eal.c | 2 +- lib/eal/linux/eal.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
lib/eal/freebsd/eal.c +1 -1
@@ -905,8 +905,8 @@ rte_eal_cleanup(void)struct internal_config *internal_conf =eal_get_internal_configuration ();rte_service_finalize (); -rte_mp_channel_cleanup ();eal_bus_cleanup (); +rte_mp_channel_cleanup ();rte_eal_alarm_cleanup ();rte_trace_save ();eal_trace_fini ();
lib/eal/linux/eal.c +1 -1
@@ -1330,11 +1330,11 @@ rte_eal_cleanup(void)rte_memseg_walk (mark_freeable,NULL );rte_service_finalize (); +eal_bus_cleanup ();#ifdef VFIO_PRESENT vfio_mp_sync_cleanup ();#endif rte_mp_channel_cleanup (); -eal_bus_cleanup ();rte_eal_alarm_cleanup ();rte_trace_save ();eal_trace_fini ();
Signature · 1 line
2.49.0.windows.1
PATCH v3 2/2 crypto/ipsec_mb: fix QP release in secondary
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV: ipsec_mb_ipc_request() line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message: ipsec_mb_mp_msg
EAL: Fail to recv reply for request /tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msg
From the code path, cryptodev->data is allocated in the primary via rte_cryptodev_data_alloc() (inside ipsec_mb_create-->rte_cryptodev_pmd_create -->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the same cryptodev->data, including nb_queue_pairs and queue_pairs[].
As a result, when the secondary process exits, ipsec_mb_remove() is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup -->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release -->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++)
ipsec_mb_qp_release(cryptodev, qp_id);
This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned above.
This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
--- drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/crypto/ipsec_mb/ipsec_mb_ops.c +5 -2
@@ -138,6 +138,7 @@ int ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) {struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; +uint16_t process_id = (uint16_t )getpid ();if (!qp)return 0 ;
@@ -152,8 +153,10 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id)rte_free (qp); dev->data->queue_pairs[qp_id] =NULL ; }else {/* secondary process */ -return ipsec_mb_secondary_qp_op (dev->data->dev_id, qp_id, -NULL ,0 ,RTE_IPSEC_MB_MP_REQ_QP_FREE ); +if (qp->qp_used_by_pid == process_id) +return ipsec_mb_secondary_qp_op (dev->data->dev_id, + qp_id,NULL ,0 , +RTE_IPSEC_MB_MP_REQ_QP_FREE ); }return 0 ; }
Signature · 1 line
2.49.0.windows.1
Re: [PATCH v3 2/2] crypto/ipsec_mb: fix QP release in secondary
Hi Experts,
is there any chance to review and accept this patch?
Brs,
Yang Ming
Re: [PATCH v3 2/2] crypto/ipsec_mb: fix QP release in secondary
In reply to Yang MingQuoted 34 lines · Yang Ming
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgFrom the code path,
cryptodev->datais allocated in the primary
viarte_cryptodev_data_alloc()(inside ipsec_mb_create-->rte_cryptodev_pmd_create -->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the samecryptodev->data, includingnb_queue_pairsand queue_pairs[].As a result, when the secondary process exits,
ipsec_mb_remove()is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup -->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release -->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id= 0;qp_id<cryptodev->data->nb_queue_pairs; qp_id++)ipsec_mb_qp_release(cryptodev, qp_id);This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned above.
This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
Quoted 6 lines
Fixes b35848bc01f6 ("crypto/ipsec_mb: add multi-process IPC request handler")Signed-off-by Yang Ming <[email protected]>---
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v3 2/2] crypto/ipsec_mb: fix QP release in secondary
Quoted 36 lines · Anatoly
Quoted 34 lines · Yang Ming
When a secondary process tries to release a queue pair (QP) that does not belong to it, error logs occur:
CRYPTODEV:ipsec_mb_ipc_request()line 373: Unable to release qp_id=0
EAL: Message data is too long
EAL: Fail to handle message:ipsec_mb_mp_msg
EAL: Fail to recv reply for request/tmp/dpdk/l2hi/mp_socket:ipsec_mb_mp_msgFrom the code path,
cryptodev->datais allocated in the primary
viarte_cryptodev_data_alloc()(inside ipsec_mb_create-->rte_cryptodev_pmd_create -->rte_cryptodev_pmd_allocate-->rte_cryptodev_data_alloc). This memory is placed in a shared memzone (rte_cryptodev_data_%u), so both primary and secondary processes reference the samecryptodev->data, includingnb_queue_pairsand queue_pairs[].As a result, when the secondary process exits,
ipsec_mb_remove()is called (inside rte_eal_cleanup-->eal_bus_cleanup-->vdev_cleanup -->rte_vdev_driver-->ipsec_mb_remove-->ipsec_mb_qp_release -->ipsec_mb_secondary_qp_op) and it loops through all queue pairs using:
for (qp_id= 0;qp_id<cryptodev->data->nb_queue_pairs; qp_id++)ipsec_mb_qp_release(cryptodev, qp_id);This causes the secondary to attempt releasing queue pairs it doesn't own, triggering the error logs mentioned above.
This patch ensures that a secondary process only frees a QP if it actually owns it, preventing conflicts and resolving the issue.
Quoted 6 lines
Quoted 5 lines
Fixes b35848bc01f6 ("crypto/ipsec_mb: add multi-process IPC request handler")Signed-off-by Yang Ming <[email protected]>Acked-by Anatoly Burakov <[email protected]>
Series applied, thanks.
Signature · 1 line
David Marchand
Quoted 29 lines · Yang Ming
The secordary process should not close socket file for MP channel before performing MP request synchronization. This prevents error logs when the secondary process exits without any operation on the crypto device while the primary process starts the device.
Case situation:
eal_bus_cleanuphas been added inrte_eal_cleanup. But for the secondary process,rte_eal_cleanupfirstly performsrte_mp_channel_cleanup, which closes socket file for the MP channel, makingmp_fdinvalid. Subsequently,eal_bus_cleanuptriggersvdev_cleanup, which callsmp_request_syncto send a message via the MP channel. Sincemp_fdis invalid, error logs occur.Error logs occur as below when the secordary process exit:
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad file descriptor
EAL: Fail to send request/tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.Function call trace:
1.rte_eal_cleanup->rte_mp_channel_cleanup->close_socket_fd
2. rte_eal_cleanup->eal_bus_cleanup->vdev_cleanup-> rte_vdev_driver->ipsec_mb_remove->ipsec_mb_qp_release-> ipsec_mb_secondary_qp_op->rte_mp_request_sync->mp_request_sync->send_msg->sendmsg(mp_fd, &msgh, 0);
Quoted 5 lines
Signed-off-by Yang Ming <[email protected]>
Signature · 2 lines
Thanks, Anatoly