Skip to contentSkip to conversation

RDMA/rtrs: Call {get,put}_cpu_ptr to silence a debug kernel warning

PATCH Superseded Guoqing Jiang 3 participants 5 messages Nov 21, 2021 – Nov 22, 2021

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/rtrs: Call {get,put}_cpu_ptr to silence a debug kernel warning

With preemption enabled (CONFIG_DEBUG_PREEMPT=y), the following appeared when rnbd client tries to map remote block device.

[ 2123.221071] BUG: using smp_processor_id() in preemptible [00000000] code: bash/1733
[ 2123.221175] caller is debug_smp_processor_id+0x17/0x20
[ 2123.221214] CPU: 0 PID: 1733 Comm: bash Not tainted 5.16.0-rc1 #5
[ 2123.221218] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
[ 2123.221229] Call Trace:
[ 2123.221231]  <TASK>
[ 2123.221235]  dump_stack_lvl+0x5d/0x78
[ 2123.221252]  dump_stack+0x10/0x12
[ 2123.221257]  check_preemption_disabled+0xe4/0xf0
[ 2123.221266]  debug_smp_processor_id+0x17/0x20
[ 2123.221271]  rtrs_clt_update_all_stats+0x3b/0x70 [rtrs_client]
[ 2123.221285]  rtrs_clt_read_req+0xc3/0x380 [rtrs_client]
[ 2123.221298]  ? rtrs_clt_init_req+0xe3/0x120 [rtrs_client]
[ 2123.221321]  rtrs_clt_request+0x1a7/0x320 [rtrs_client]
[ 2123.221340]  ? 0xffffffffc0ab1000
[ 2123.221357]  send_usr_msg+0xbf/0x160 [rnbd_client]
[ 2123.221370]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221377]  ? send_usr_msg+0x160/0x160 [rnbd_client]
[ 2123.221386]  ? sg_alloc_table+0x27/0xb0
[ 2123.221395]  ? sg_zero_buffer+0xd0/0xd0
[ 2123.221407]  send_msg_sess_info+0xe9/0x180 [rnbd_client]
[ 2123.221413]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221429]  ? blk_mq_alloc_tag_set+0x2ef/0x370
[ 2123.221447]  rnbd_clt_map_device+0xba8/0xcd0 [rnbd_client]
[ 2123.221462]  ? send_msg_open+0x200/0x200 [rnbd_client]
[ 2123.221479]  rnbd_clt_map_device_store+0x3e5/0x620 [rnbd_client

To supress the calltrace, let's call get_cpu_ptr/put_cpu_ptr pair in rtrs_clt_update_rdma_stats to disable preemption when accessing per-cpu variable.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +2 -1
index f7e459fe68be..6ff72f2b1a3a 100644
@@ -169,9 +169,10 @@ static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats,
 {
 	struct rtrs_clt_stats_pcpu *s;
 
-	s = this_cpu_ptr(stats->pcpu_stats);
+	s = get_cpu_ptr(stats->pcpu_stats);
 	s->rdma.dir[d].cnt++;
 	s->rdma.dir[d].size_total += size;
+	put_cpu_ptr(stats->pcpu_stats);
 }
 
 void rtrs_clt_update_all_stats(struct rtrs_clt_io_req *req, int dir)
Signature · 1 line
2.31.1
maintainer Acked-by on v1 1/1
#1 lore raw
Quoted 31 lines · Guoqing Jiang

With preemption enabled (CONFIG_DEBUG_PREEMPT=y), the following appeared when rnbd client tries to map remote block device.

[ 2123.221071] BUG: using smp_processor_id() in preemptible [00000000] code: bash/1733
[ 2123.221175] caller is debug_smp_processor_id+0x17/0x20
[ 2123.221214] CPU: 0 PID: 1733 Comm: bash Not tainted 5.16.0-rc1 #5
[ 2123.221218] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
[ 2123.221229] Call Trace:
[ 2123.221231]  <TASK>
[ 2123.221235]  dump_stack_lvl+0x5d/0x78
[ 2123.221252]  dump_stack+0x10/0x12
[ 2123.221257]  check_preemption_disabled+0xe4/0xf0
[ 2123.221266]  debug_smp_processor_id+0x17/0x20
[ 2123.221271]  rtrs_clt_update_all_stats+0x3b/0x70 [rtrs_client]
[ 2123.221285]  rtrs_clt_read_req+0xc3/0x380 [rtrs_client]
[ 2123.221298]  ? rtrs_clt_init_req+0xe3/0x120 [rtrs_client]
[ 2123.221321]  rtrs_clt_request+0x1a7/0x320 [rtrs_client]
[ 2123.221340]  ? 0xffffffffc0ab1000
[ 2123.221357]  send_usr_msg+0xbf/0x160 [rnbd_client]
[ 2123.221370]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221377]  ? send_usr_msg+0x160/0x160 [rnbd_client]
[ 2123.221386]  ? sg_alloc_table+0x27/0xb0
[ 2123.221395]  ? sg_zero_buffer+0xd0/0xd0
[ 2123.221407]  send_msg_sess_info+0xe9/0x180 [rnbd_client]
[ 2123.221413]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221429]  ? blk_mq_alloc_tag_set+0x2ef/0x370
[ 2123.221447]  rnbd_clt_map_device+0xba8/0xcd0 [rnbd_client]
[ 2123.221462]  ? send_msg_open+0x200/0x200 [rnbd_client]
[ 2123.221479]  rnbd_clt_map_device_store+0x3e5/0x620 [rnbd_client
Quoted 5 lines

To supress the calltrace, let's call get_cpu_ptr/put_cpu_ptr pair in rtrs_clt_update_rdma_stats to disable preemption when accessing per-cpu variable.

Looks good to me.

Thanks Guoqing.

Quoted 23 lines
---
 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +2 -2
index f7e459fe68be..6ff72f2b1a3a 100644
@@ -169,9 +169,10 @@ static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats,
 {
        struct rtrs_clt_stats_pcpu *s;
 
-       s = this_cpu_ptr(stats->pcpu_stats);
+       s = get_cpu_ptr(stats->pcpu_stats);
        s->rdma.dir[d].cnt++;
        s->rdma.dir[d].size_total += size;
+       put_cpu_ptr(stats->pcpu_stats);
 }
 
 void rtrs_clt_update_all_stats(struct rtrs_clt_io_req *req, int dir)
--

2.31.1

#2 lore raw
In reply to Guoqing Jiang
Quoted 44 lines · Guoqing Jiang

With preemption enabled (CONFIG_DEBUG_PREEMPT=y), the following appeared when rnbd client tries to map remote block device.

[ 2123.221071] BUG: using smp_processor_id() in preemptible [00000000] code: bash/1733
[ 2123.221175] caller is debug_smp_processor_id+0x17/0x20
[ 2123.221214] CPU: 0 PID: 1733 Comm: bash Not tainted 5.16.0-rc1 #5
[ 2123.221218] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
[ 2123.221229] Call Trace:
[ 2123.221231]  <TASK>
[ 2123.221235]  dump_stack_lvl+0x5d/0x78
[ 2123.221252]  dump_stack+0x10/0x12
[ 2123.221257]  check_preemption_disabled+0xe4/0xf0
[ 2123.221266]  debug_smp_processor_id+0x17/0x20
[ 2123.221271]  rtrs_clt_update_all_stats+0x3b/0x70 [rtrs_client]
[ 2123.221285]  rtrs_clt_read_req+0xc3/0x380 [rtrs_client]
[ 2123.221298]  ? rtrs_clt_init_req+0xe3/0x120 [rtrs_client]
[ 2123.221321]  rtrs_clt_request+0x1a7/0x320 [rtrs_client]
[ 2123.221340]  ? 0xffffffffc0ab1000
[ 2123.221357]  send_usr_msg+0xbf/0x160 [rnbd_client]
[ 2123.221370]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221377]  ? send_usr_msg+0x160/0x160 [rnbd_client]
[ 2123.221386]  ? sg_alloc_table+0x27/0xb0
[ 2123.221395]  ? sg_zero_buffer+0xd0/0xd0
[ 2123.221407]  send_msg_sess_info+0xe9/0x180 [rnbd_client]
[ 2123.221413]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221429]  ? blk_mq_alloc_tag_set+0x2ef/0x370
[ 2123.221447]  rnbd_clt_map_device+0xba8/0xcd0 [rnbd_client]
[ 2123.221462]  ? send_msg_open+0x200/0x200 [rnbd_client]
[ 2123.221479]  rnbd_clt_map_device_store+0x3e5/0x620 [rnbd_client

To supress the calltrace, let's call get_cpu_ptr/put_cpu_ptr pair in rtrs_clt_update_rdma_stats to disable preemption when accessing per-cpu variable.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +0 -0
index f7e459fe68be..6ff72f2b1a3a 100644

drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats, · line 175

Quoted 8 lines
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +2 -1
@@ -169,9 +169,10 @@ static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats,
 {
 	struct rtrs_clt_stats_pcpu *s;
 
-	s = this_cpu_ptr(stats->pcpu_stats);
+	s = get_cpu_ptr(stats->pcpu_stats);
 	s->rdma.dir[d].cnt++;
 	s->rdma.dir[d].size_total += size;
+	put_cpu_ptr(stats->pcpu_stats);

I see that this_cpu_ptr() is used in many other places in rtrs-clt-stats.c, why do we need to change only one place?

Thanks

Quoted 6 lines
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +0 -0
 }
 
 void rtrs_clt_update_all_stats(struct rtrs_clt_io_req *req, int dir)
Signature · 1 line
2.31.1
maintainer
#3 lore raw
Quoted 49 lines · Leon Romanovsky
Quoted 47 lines · Guoqing Jiang

With preemption enabled (CONFIG_DEBUG_PREEMPT=y), the following appeared when rnbd client tries to map remote block device.

[ 2123.221071] BUG: using smp_processor_id() in preemptible [00000000] code: bash/1733
[ 2123.221175] caller is debug_smp_processor_id+0x17/0x20
[ 2123.221214] CPU: 0 PID: 1733 Comm: bash Not tainted 5.16.0-rc1 #5
[ 2123.221218] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
[ 2123.221229] Call Trace:
[ 2123.221231]  <TASK>
[ 2123.221235]  dump_stack_lvl+0x5d/0x78
[ 2123.221252]  dump_stack+0x10/0x12
[ 2123.221257]  check_preemption_disabled+0xe4/0xf0
[ 2123.221266]  debug_smp_processor_id+0x17/0x20
[ 2123.221271]  rtrs_clt_update_all_stats+0x3b/0x70 [rtrs_client]
[ 2123.221285]  rtrs_clt_read_req+0xc3/0x380 [rtrs_client]
[ 2123.221298]  ? rtrs_clt_init_req+0xe3/0x120 [rtrs_client]
[ 2123.221321]  rtrs_clt_request+0x1a7/0x320 [rtrs_client]
[ 2123.221340]  ? 0xffffffffc0ab1000
[ 2123.221357]  send_usr_msg+0xbf/0x160 [rnbd_client]
[ 2123.221370]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221377]  ? send_usr_msg+0x160/0x160 [rnbd_client]
[ 2123.221386]  ? sg_alloc_table+0x27/0xb0
[ 2123.221395]  ? sg_zero_buffer+0xd0/0xd0
[ 2123.221407]  send_msg_sess_info+0xe9/0x180 [rnbd_client]
[ 2123.221413]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221429]  ? blk_mq_alloc_tag_set+0x2ef/0x370
[ 2123.221447]  rnbd_clt_map_device+0xba8/0xcd0 [rnbd_client]
[ 2123.221462]  ? send_msg_open+0x200/0x200 [rnbd_client]
[ 2123.221479]  rnbd_clt_map_device_store+0x3e5/0x620 [rnbd_client

To supress the calltrace, let's call get_cpu_ptr/put_cpu_ptr pair in rtrs_clt_update_rdma_stats to disable preemption when accessing per-cpu variable.

---
 drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +0 -0
index f7e459fe68be..6ff72f2b1a3a 100644
@@ -169,9 +169,10 @@ static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats,
 {
      struct rtrs_clt_stats_pcpu *s;
 
Quoted 8 lines

drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats, · line 175

Quoted 5 lines
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +2 -1
-     s = this_cpu_ptr(stats->pcpu_stats);
+     s = get_cpu_ptr(stats->pcpu_stats);
      s->rdma.dir[d].cnt++;
      s->rdma.dir[d].size_total += size;
+     put_cpu_ptr(stats->pcpu_stats);

I see that this_cpu_ptr() is used in many other places in rtrs-clt-stats.c, why do we need to change only one place?

Right, indeed. guoqing, mind to respin?

Thanks

Quoted 9 lines

Thanks

Quoted 6 lines
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +0 -1
 }
 
 void rtrs_clt_update_all_stats(struct rtrs_clt_io_req *req, int dir)
--

2.31.1

#4 lore raw
Quoted 49 lines · Jinpu Wang
Quoted 48 lines · Leon Romanovsky
Quoted 47 lines · Guoqing Jiang

With preemption enabled (CONFIG_DEBUG_PREEMPT=y), the following appeared when rnbd client tries to map remote block device.

[ 2123.221071] BUG: using smp_processor_id() in preemptible [00000000] code: bash/1733
[ 2123.221175] caller is debug_smp_processor_id+0x17/0x20
[ 2123.221214] CPU: 0 PID: 1733 Comm: bash Not tainted 5.16.0-rc1 #5
[ 2123.221218] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
[ 2123.221229] Call Trace:
[ 2123.221231]  <TASK>
[ 2123.221235]  dump_stack_lvl+0x5d/0x78
[ 2123.221252]  dump_stack+0x10/0x12
[ 2123.221257]  check_preemption_disabled+0xe4/0xf0
[ 2123.221266]  debug_smp_processor_id+0x17/0x20
[ 2123.221271]  rtrs_clt_update_all_stats+0x3b/0x70 [rtrs_client]
[ 2123.221285]  rtrs_clt_read_req+0xc3/0x380 [rtrs_client]
[ 2123.221298]  ? rtrs_clt_init_req+0xe3/0x120 [rtrs_client]
[ 2123.221321]  rtrs_clt_request+0x1a7/0x320 [rtrs_client]
[ 2123.221340]  ? 0xffffffffc0ab1000
[ 2123.221357]  send_usr_msg+0xbf/0x160 [rnbd_client]
[ 2123.221370]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221377]  ? send_usr_msg+0x160/0x160 [rnbd_client]
[ 2123.221386]  ? sg_alloc_table+0x27/0xb0
[ 2123.221395]  ? sg_zero_buffer+0xd0/0xd0
[ 2123.221407]  send_msg_sess_info+0xe9/0x180 [rnbd_client]
[ 2123.221413]  ? rnbd_clt_put_sess+0x60/0x60 [rnbd_client]
[ 2123.221429]  ? blk_mq_alloc_tag_set+0x2ef/0x370
[ 2123.221447]  rnbd_clt_map_device+0xba8/0xcd0 [rnbd_client]
[ 2123.221462]  ? send_msg_open+0x200/0x200 [rnbd_client]
[ 2123.221479]  rnbd_clt_map_device_store+0x3e5/0x620 [rnbd_client

To supress the calltrace, let's call get_cpu_ptr/put_cpu_ptr pair in rtrs_clt_update_rdma_stats to disable preemption when accessing per-cpu variable.

---
  drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +0 -0
index f7e459fe68be..6ff72f2b1a3a 100644
@@ -169,9 +169,10 @@ static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats,
  {
       struct rtrs_clt_stats_pcpu *s;
 
Quoted 8 lines
Quoted 7 lines

drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c static inline void rtrs_clt_update_rdma_stats(struct rtrs_clt_stats *stats, · line 175

Quoted 5 lines
drivers/infiniband/ulp/rtrs/rtrs-clt-stats.c +2 -1
-     s = this_cpu_ptr(stats->pcpu_stats);
+     s = get_cpu_ptr(stats->pcpu_stats);
       s->rdma.dir[d].cnt++;
       s->rdma.dir[d].size_total += size;
+     put_cpu_ptr(stats->pcpu_stats);

I see that this_cpu_ptr() is used in many other places in rtrs-clt-stats.c, why do we need to change only one place?

Right, indeed. guoqing, mind to respin?

I just see this calltrace, probably other paths are not triggered since the connection can't be established. Anyway, will send v2.

Thanks,
Guoqing