Skip to contentSkip to conversation

RDMA/siw: add sendpage_ok() check to disable MSG_SPLICE_PAGES

PATCH Superseded Showrya M N 2 participants 3 messages Oct 3, 2024 – Oct 4, 2024

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/siw: add sendpage_ok() check to disable MSG_SPLICE_PAGES

While running ISER over SIW, the initiator machine encounters a warning from skb_splice_from_iter() indicating that a slab page is being used in send_page. To address this, it is better to add a sendpage_ok() check within the driver itself, and if it returns 0, then MSG_SPLICE_PAGES flag should be disabled before entering the network stack.

A similar issue has been discussed for NVMe in this thread:
https://lore.kernel.org/all/[email protected]/

stack trace:
...

[ 2157.532917] WARNING: CPU: 0 PID: 5342 at net/core/skbuff.c:7140 skb_splice_from_iter+0x173/0x320
Call Trace:
[ 2157.533064] Call Trace:
[ 2157.533069]  ? __warn+0x84/0x130
[ 2157.533073]  ? skb_splice_from_iter+0x173/0x320
[ 2157.533075]  ? report_bug+0xfc/0x1e0
[ 2157.533081]  ? handle_bug+0x3f/0x70
[ 2157.533085]  ? exc_invalid_op+0x17/0x70
[ 2157.533088]  ? asm_exc_invalid_op+0x1a/0x20
[ 2157.533096]  ? skb_splice_from_iter+0x173/0x320
[ 2157.533101]  tcp_sendmsg_locked+0x368/0xe40
[ 2157.533111]  siw_tx_hdt+0x695/0xa40 [siw]
[ 2157.533134]  ? sched_balance_find_src_group+0x44/0x4f0
[ 2157.533143]  ? __update_load_avg_cfs_rq+0x272/0x300
[ 2157.533152]  ? place_entity+0x19/0xf0
[ 2157.533157]  ? enqueue_entity+0xdb/0x3d0
[ 2157.533162]  ? pick_eevdf+0xe2/0x120
[ 2157.533169]  ? check_preempt_wakeup_fair+0x161/0x1f0
[ 2157.533174]  ? wakeup_preempt+0x61/0x70
[ 2157.533177]  ? ttwu_do_activate+0x5d/0x1e0
[ 2157.533183]  ? try_to_wake_up+0x78/0x610
[ 2157.533188]  ? xas_load+0xd/0xb0
[ 2157.533193]  ? xa_load+0x80/0xb0
[ 2157.533200]  siw_qp_sq_process+0x102/0xb00 [siw]
[ 2157.533213]  ? __pfx_siw_run_sq+0x10/0x10 [siw]
[ 2157.533224]  siw_sq_resume+0x39/0x110 [siw]
[ 2157.533236]  siw_run_sq+0x74/0x160 [siw]
[ 2157.533246]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 2157.533252]  kthread+0xd2/0x100
[ 2157.533257]  ? __pfx_kthread+0x10/0x10
[ 2157.533261]  ret_from_fork+0x34/0x40
[ 2157.533266]  ? __pfx_kthread+0x10/0x10
[ 2157.533269]  ret_from_fork_asm+0x1a/0x30
.
[ 2157.533301] iser: iser_qp_event_callback: qp event QP request error (2)
[ 2157.533307] iser: iser_qp_event_callback: qp event send queue drained (5)
[ 2157.533348]  connection26:0: detected conn error (1011)
---
 drivers/infiniband/sw/siw/siw_qp_tx.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/sw/siw/siw_qp_tx.c +3 -0
index 64ad9e0895bd..d777d06037db 100644
@@ -334,6 +334,9 @@ static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset,
 		bvec_set_page(&bvec, page[i], bytes, offset);
 		iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size);
 
+		if (!sendpage_ok(page[i]))
+			msg.msg_flags &= ~MSG_SPLICE_PAGES;
+
 try_page_again:
 		lock_sock(sk);
 		rv = tcp_sendmsg_locked(sk, &msg, size);
Signature · 1 line
2.39.1
#1 lore raw
Quoted 78 lines

-----Original Message-----
From: Showrya M N <[email protected]>
Sent: Thursday, October 3, 2024 2:46 PM
To: [email protected]; [email protected]; Bernard Metzler <[email protected]>
Cc: [email protected]; Showrya M N <[email protected]>; Potnuri Bharat Teja <[email protected]>
Subject: [EXTERNAL] [PATCH for-rc] RDMA/siw: add sendpage_ok() check to disable MSG_SPLICE_PAGES

While running ISER over SIW, the initiator machine encounters a warning from skb_splice_from_iter() indicating that a slab page is being used in send_page. To address this, it is better to add a sendpage_ok() check within the driver itself, and if it returns 0, then MSG_SPLICE_PAGES flag should be disabled before entering the network stack.

A similar issue has been discussed for NVMe in this thread:
INVALID URI REMOVED 3A__lore.kernel.org_all_20240530142417.146696-2D1-2Dofir.gal- 40volumez.com_&d=DwIDAg&c=BSDicqBQBDjDI9RkVyTcHQ&r=4ynb4Sj_4MUcZXbhvovE4tYS bqxyOwdSiLedP4yO55g&m=5zLkbzFXCNvwiYwzyhSLC9r5GQnIt4VKawdBIinJdVEfirE1BFqwS 9QGbhVWOOKo&s=wxM2sRuzDoq36W23_jA5pNgeGDEPfTmyPhsvPqp0_-E&e=

stack trace:
...

[ 2157.532917] WARNING: CPU: 0 PID: 5342 at net/core/skbuff.c:7140
skb_splice_from_iter+0x173/0x320
Call Trace:
[ 2157.533064] Call Trace:
[ 2157.533069]  ? __warn+0x84/0x130
[ 2157.533073]  ? skb_splice_from_iter+0x173/0x320
[ 2157.533075]  ? report_bug+0xfc/0x1e0
[ 2157.533081]  ? handle_bug+0x3f/0x70
[ 2157.533085]  ? exc_invalid_op+0x17/0x70
[ 2157.533088]  ? asm_exc_invalid_op+0x1a/0x20
[ 2157.533096]  ? skb_splice_from_iter+0x173/0x320
[ 2157.533101]  tcp_sendmsg_locked+0x368/0xe40
[ 2157.533111]  siw_tx_hdt+0x695/0xa40 [siw]
[ 2157.533134]  ? sched_balance_find_src_group+0x44/0x4f0
[ 2157.533143]  ? __update_load_avg_cfs_rq+0x272/0x300
[ 2157.533152]  ? place_entity+0x19/0xf0
[ 2157.533157]  ? enqueue_entity+0xdb/0x3d0
[ 2157.533162]  ? pick_eevdf+0xe2/0x120
[ 2157.533169]  ? check_preempt_wakeup_fair+0x161/0x1f0
[ 2157.533174]  ? wakeup_preempt+0x61/0x70
[ 2157.533177]  ? ttwu_do_activate+0x5d/0x1e0
[ 2157.533183]  ? try_to_wake_up+0x78/0x610
[ 2157.533188]  ? xas_load+0xd/0xb0
[ 2157.533193]  ? xa_load+0x80/0xb0
[ 2157.533200]  siw_qp_sq_process+0x102/0xb00 [siw]
[ 2157.533213]  ? __pfx_siw_run_sq+0x10/0x10 [siw]
[ 2157.533224]  siw_sq_resume+0x39/0x110 [siw]
[ 2157.533236]  siw_run_sq+0x74/0x160 [siw]
[ 2157.533246]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 2157.533252]  kthread+0xd2/0x100
[ 2157.533257]  ? __pfx_kthread+0x10/0x10
[ 2157.533261]  ret_from_fork+0x34/0x40
[ 2157.533266]  ? __pfx_kthread+0x10/0x10
[ 2157.533269]  ret_from_fork_asm+0x1a/0x30
.
[ 2157.533301] iser: iser_qp_event_callback: qp event QP request error (2)
[ 2157.533307] iser: iser_qp_event_callback: qp event send queue drained
(5)
[ 2157.533348]  connection26:0: detected conn error (1011)
---
 drivers/infiniband/sw/siw/siw_qp_tx.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/sw/siw/siw_qp_tx.c +0 -0
index 64ad9e0895bd..d777d06037db 100644

drivers/infiniband/sw/siw/siw_qp_tx.c static int siw_tcp_sendpages(struct socket *s, struct · line 340

Quoted 7 lines
drivers/infiniband/sw/siw/siw_qp_tx.c +3 -0
@@ -334,6 +334,9 @@ static int siw_tcp_sendpages(struct socket *s, struct
 page **page, int offset,
 		bvec_set_page(&bvec, page[i], bytes, offset);
 		iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size);
 
+		if (!sendpage_ok(page[i]))
+			msg.msg_flags &= ~MSG_SPLICE_PAGES;
+

Thanks! This looks good to me. Although, I would suggest moving this further up just before bvec_set_page(): While it is not dong anything to the page, it looks more clean to first alter the pages flags before linking the bvec with it.

Thanks! Bernard

Quoted 5 lines
drivers/infiniband/sw/siw/siw_qp_tx.c +0 -0
 try_page_again:
 		lock_sock(sk);

rv = tcp_sendmsg_locked(sk, &msg, size);
--
2.39.1

#2 lore raw

ok bernard, I will send a v2.

Thanks,
Showrya

-----Original Message-----
From: Bernard Metzler <[email protected]>
Sent: 03 October 2024 19:56
To: Showrya M N <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; Potnuri Bharat Teja <[email protected]>
Subject: RE: [PATCH for-rc] RDMA/siw: add sendpage_ok() check to disable MSG_SPLICE_PAGES

Quoted 72 lines

-----Original Message-----
From: Showrya M N <[email protected]>
Sent: Thursday, October 3, 2024 2:46 PM
To: [email protected]; [email protected]; Bernard Metzler <[email protected]>
Cc: [email protected]; Showrya M N <[email protected]>; Potnuri Bharat Teja <[email protected]>
Subject: [EXTERNAL] [PATCH for-rc] RDMA/siw: add sendpage_ok() check to disable MSG_SPLICE_PAGES

While running ISER over SIW, the initiator machine encounters a warning from skb_splice_from_iter() indicating that a slab page is being used in send_page. To address this, it is better to add a sendpage_ok() check within the driver itself, and if it returns 0, then MSG_SPLICE_PAGES flag should be disabled before entering the network stack.

A similar issue has been discussed for NVMe in this thread:
INVALID URI REMOVED 3A__lore.kernel.org_all_20240530142417.146696-2D1-2Dofir.gal- 40volumez.com_&d=DwIDAg&c=BSDicqBQBDjDI9RkVyTcHQ&r=4ynb4Sj_4MUcZXbhvov E4tYS bqxyOwdSiLedP4yO55g&m=5zLkbzFXCNvwiYwzyhSLC9r5GQnIt4VKawdBIinJdVEfirE1 BFqwS 9QGbhVWOOKo&s=wxM2sRuzDoq36W23_jA5pNgeGDEPfTmyPhsvPqp0_-E&e=

stack trace:
...

[ 2157.532917] WARNING: CPU: 0 PID: 5342 at net/core/skbuff.c:7140
skb_splice_from_iter+0x173/0x320
Call Trace:
[ 2157.533064] Call Trace:
[ 2157.533069]  ? __warn+0x84/0x130
[ 2157.533073]  ? skb_splice_from_iter+0x173/0x320 [ 2157.533075]  ? 
report_bug+0xfc/0x1e0 [ 2157.533081]  ? handle_bug+0x3f/0x70 [ 
2157.533085]  ? exc_invalid_op+0x17/0x70 [ 2157.533088]  ? 
asm_exc_invalid_op+0x1a/0x20 [ 2157.533096]  ? 
skb_splice_from_iter+0x173/0x320 [ 2157.533101]  
tcp_sendmsg_locked+0x368/0xe40 [ 2157.533111]  siw_tx_hdt+0x695/0xa40 
[siw] [ 2157.533134]  ? sched_balance_find_src_group+0x44/0x4f0
[ 2157.533143]  ? __update_load_avg_cfs_rq+0x272/0x300
[ 2157.533152]  ? place_entity+0x19/0xf0 [ 2157.533157]  ? 
enqueue_entity+0xdb/0x3d0 [ 2157.533162]  ? pick_eevdf+0xe2/0x120 [ 
2157.533169]  ? check_preempt_wakeup_fair+0x161/0x1f0
[ 2157.533174]  ? wakeup_preempt+0x61/0x70 [ 2157.533177]  ? 
ttwu_do_activate+0x5d/0x1e0 [ 2157.533183]  ? 
try_to_wake_up+0x78/0x610 [ 2157.533188]  ? xas_load+0xd/0xb0 [ 
2157.533193]  ? xa_load+0x80/0xb0 [ 2157.533200]  
siw_qp_sq_process+0x102/0xb00 [siw] [ 2157.533213]  ? 
__pfx_siw_run_sq+0x10/0x10 [siw] [ 2157.533224]  
siw_sq_resume+0x39/0x110 [siw] [ 2157.533236]  siw_run_sq+0x74/0x160 
[siw] [ 2157.533246]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 2157.533252]  kthread+0xd2/0x100
[ 2157.533257]  ? __pfx_kthread+0x10/0x10 [ 2157.533261]  
ret_from_fork+0x34/0x40 [ 2157.533266]  ? __pfx_kthread+0x10/0x10 [ 
2157.533269]  ret_from_fork_asm+0x1a/0x30 .
[ 2157.533301] iser: iser_qp_event_callback: qp event QP request error 
(2) [ 2157.533307] iser: iser_qp_event_callback: qp event send queue 
drained
(5)
[ 2157.533348]  connection26:0: detected conn error (1011)
---
 drivers/infiniband/sw/siw/siw_qp_tx.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/sw/siw/siw_qp_tx.c +0 -0
index 64ad9e0895bd..d777d06037db 100644

drivers/infiniband/sw/siw/siw_qp_tx.c static int siw_tcp_sendpages(struct socket *s, · line 340

Quoted 7 lines
drivers/infiniband/sw/siw/siw_qp_tx.c +3 -0
@@ -334,6 +334,9 @@ static int siw_tcp_sendpages(struct socket *s, 
 struct page **page, int offset,
 		bvec_set_page(&bvec, page[i], bytes, offset);
 		iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size);
 
+		if (!sendpage_ok(page[i]))
+			msg.msg_flags &= ~MSG_SPLICE_PAGES;
+

Thanks! This looks good to me. Although, I would suggest moving this further up just before bvec_set_page(): While it is not dong anything to the page, it looks more clean to first alter the pages flags before linking the bvec with it.

Thanks! Bernard

Quoted 5 lines
drivers/infiniband/sw/siw/siw_qp_tx.c +0 -0
 try_page_again:
 		lock_sock(sk);

rv = tcp_sendmsg_locked(sk, &msg, size);
--
2.39.1