net/rds: handle zerocopy send cleanup before the message is queued
Subsystems (4)
PATCH 1/1 net/rds: handle zerocopy send cleanup before the message is queued
In reply to a message not in the archiveFrom: Nan Li <[email protected]>
A zerocopy send can fail after user pages have been pinned but before the message is attached to the sending socket.
The purge path currently infers zerocopy state from rm->m_rs, so an unqueued message can be cleaned up as if it owned normal payload pages. However, zerocopy ownership is really determined by the presence of op_mmp_znotifier, regardless of whether the message has reached the socket queue.
Capture op_mmp_znotifier up front in rds_message_purge() and use it as the cleanup discriminator. If the message is already associated with a socket, keep the existing completion path. Otherwise, drop the pinned page accounting directly and release the notifier before putting the payload pages.
This keeps early send failure cleanup consistent with the zerocopy lifetime rules without changing the normal queued completion path.
--- net/rds/message.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
net/rds/message.c +15 -5
@@ -131,24 +131,34 @@ static void rds_rm_zerocopy_callback(struct rds_sock *rs,*/ static void rds_message_purge (struct rds_message *rm) { +struct rds_znotifier *znotifier;unsigned long i, flags; -bool zcopy = false; +bool zcopy;if (unlikely (test_bit (RDS_MSG_PAGEVEC , &rm->m_flags)))return ;spin_lock_irqsave (&rm->m_rs_lock, flags); + znotifier = rm->data.op_mmp_znotifier; + rm->data.op_mmp_znotifier =NULL ; + zcopy = !!znotifier; +if (rm->m_rs) {struct rds_sock *rs = rm->m_rs; -if (rm->data.op_mmp_znotifier) { - zcopy = true; -rds_rm_zerocopy_callback (rs, rm->data.op_mmp_znotifier); +if (znotifier) { +rds_rm_zerocopy_callback (rs, znotifier);rds_wake_sk_sleep (rs); - rm->data.op_mmp_znotifier =NULL ; }sock_put (rds_rs_to_sk (rs)); rm->m_rs =NULL ; + }else if (znotifier) { +/* +* Zerocopy can fail before the message is queued on the +* socket, so there is no rs to carry the notification. +*/ +mm_unaccount_pinned_pages (&znotifier->z_mmp); +kfree (rds_info_from_znotifier (znotifier)); }spin_unlock_irqrestore (&rm->m_rs_lock, flags);
Signature · 1 line
2.43.0
Quoted 20 lines · Ren Wei
From: Nan Li <[email protected]>
A zerocopy send can fail after user pages have been pinned but before the message is attached to the sending socket.
The purge path currently infers zerocopy state from
rm->m_rs, so an unqueued message can be cleaned up as if it owned normal payload pages. However, zerocopy ownership is really determined by the presence ofop_mmp_znotifier, regardless of whether the message has reached the socket queue.Capture
op_mmp_znotifierup front inrds_message_purge()and use it as the cleanup discriminator. If the message is already associated with a socket, keep the existing completion path. Otherwise, drop the pinned page accounting directly and release the notifier before putting the payload pages.This keeps early send failure cleanup consistent with the zerocopy lifetime rules without changing the normal queued completion path.
Quoted 10 lines
Fixes 0cebaccef3ac ("rds: zerocopy Tx support.")Reported-by Yuan Tan <[email protected]>Reported-by Yifan Wu <[email protected]>Reported-by Juefei Pu <[email protected]>Reported-by Xin Liu <[email protected]>Co-developed-by Xiao Liu <[email protected]>Signed-off-by Xiao Liu <[email protected]>Signed-off-by Nan Li <[email protected]>Signed-off-by Ren Wei <[email protected]>
This fix looks fine to me. Thanks Ren Wei!
Allison
Quoted 48 lines
--- net/rds/message.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)net/rds/message.c +15
-5@@ -131,24 +131,34 @@ static void rds_rm_zerocopy_callback(struct rds_sock *rs,*/ static void rds_message_purge (struct rds_message *rm) { +struct rds_znotifier *znotifier;unsigned long i, flags; -bool zcopy = false; +bool zcopy;if (unlikely (test_bit (RDS_MSG_PAGEVEC , &rm->m_flags)))return ;spin_lock_irqsave (&rm->m_rs_lock, flags); + znotifier = rm->data.op_mmp_znotifier; + rm->data.op_mmp_znotifier =NULL ; + zcopy = !!znotifier; +if (rm->m_rs) {struct rds_sock *rs = rm->m_rs; -if (rm->data.op_mmp_znotifier) { - zcopy = true; -rds_rm_zerocopy_callback (rs, rm->data.op_mmp_znotifier); +if (znotifier) { +rds_rm_zerocopy_callback (rs, znotifier);rds_wake_sk_sleep (rs); - rm->data.op_mmp_znotifier =NULL ; }sock_put (rds_rs_to_sk (rs)); rm->m_rs =NULL ; + }else if (znotifier) { +/* +* Zerocopy can fail before the message is queued on the +* socket, so there is no rs to carry the notification. +*/ +mm_unaccount_pinned_pages (&znotifier->z_mmp); +kfree (rds_info_from_znotifier (znotifier)); }spin_unlock_irqrestore (&rm->m_rs_lock, flags);
Paolo Abeni
maintainerQuoted 32 lines · Allison Henderson
Quoted 20 lines · Ren Wei
From: Nan Li <[email protected]>
A zerocopy send can fail after user pages have been pinned but before the message is attached to the sending socket.
The purge path currently infers zerocopy state from
rm->m_rs, so an unqueued message can be cleaned up as if it owned normal payload pages. However, zerocopy ownership is really determined by the presence ofop_mmp_znotifier, regardless of whether the message has reached the socket queue.Capture
op_mmp_znotifierup front inrds_message_purge()and use it as the cleanup discriminator. If the message is already associated with a socket, keep the existing completion path. Otherwise, drop the pinned page accounting directly and release the notifier before putting the payload pages.This keeps early send failure cleanup consistent with the zerocopy lifetime rules without changing the normal queued completion path.
Quoted 10 lines
Fixes 0cebaccef3ac ("rds: zerocopy Tx support.")Reported-by Yuan Tan <[email protected]>Reported-by Yifan Wu <[email protected]>Reported-by Juefei Pu <[email protected]>Reported-by Xin Liu <[email protected]>Co-developed-by Xiao Liu <[email protected]>Signed-off-by Xiao Liu <[email protected]>Signed-off-by Nan Li <[email protected]>Signed-off-by Ren Wei <[email protected]>
Quoted 2 lines
This fix looks fine to me. Thanks Ren Wei!
Reviewed-by Allison Henderson <[email protected]>
Note that sashiko spotted more pre-existing problems in this area, please have a look:
/P
This patch was applied to netdev/net.git (main) by Paolo Abeni <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Paolo Abeni <[email protected]>:
Quoted 5 lines · On Fri, 1 May 2026 09:08:44 +0800 you wrote:
From: Nan Li <[email protected]>
A zerocopy send can fail after user pages have been pinned but before the message is attached to the sending socket.
Quoted 7 lines
The purge path currently infers zerocopy state from
rm->m_rs, so an unqueued message can be cleaned up as if it owned normal payload pages. However, zerocopy ownership is really determined by the presence ofop_mmp_znotifier, regardless of whether the message has reached the socket queue.[...]
Here is the summary with links:
- [net,1/1] net/rds: handle zerocopy send cleanup before the message is queued
https://git.kernel.org/netdev/net/c/44b550d88b26
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Allison Henderson
maintainerQuoted 36 lines · Paolo Abeni
Quoted 32 lines · Allison Henderson
Quoted 20 lines · Ren Wei
From: Nan Li <[email protected]>
A zerocopy send can fail after user pages have been pinned but before the message is attached to the sending socket.
The purge path currently infers zerocopy state from
rm->m_rs, so an unqueued message can be cleaned up as if it owned normal payload pages. However, zerocopy ownership is really determined by the presence ofop_mmp_znotifier, regardless of whether the message has reached the socket queue.Capture
op_mmp_znotifierup front inrds_message_purge()and use it as the cleanup discriminator. If the message is already associated with a socket, keep the existing completion path. Otherwise, drop the pinned page accounting directly and release the notifier before putting the payload pages.This keeps early send failure cleanup consistent with the zerocopy lifetime rules without changing the normal queued completion path.
Quoted 10 lines
Fixes 0cebaccef3ac ("rds: zerocopy Tx support.")Reported-by Yuan Tan <[email protected]>Reported-by Yifan Wu <[email protected]>Reported-by Juefei Pu <[email protected]>Reported-by Xin Liu <[email protected]>Co-developed-by Xiao Liu <[email protected]>Signed-off-by Xiao Liu <[email protected]>Signed-off-by Nan Li <[email protected]>Signed-off-by Ren Wei <[email protected]>Quoted 2 lines
This fix looks fine to me. Thanks Ren Wei!
Reviewed-by Allison Henderson <[email protected]>
Quoted 6 lines
Note that sashiko spotted more pre-existing problems in this area, please have a look:
/P
Thanks for pointing this out. It looks like a valid catch, I will see if I can get a fix out today or tomorrow.
Thank you!
Allison