RDMA/rxe: Fix bug in rxe_net.c
Subsystems (2)
PATCH 1/1 RDMA/rxe: Fix bug in rxe_net.c
An earlier patch removed setting of tot_len in IPV4 headers because it was also set in ip_local_out. However, this change resulted in an incorrect ICRC being computed because the tot_len field is not masked out. This patch restores that line. This fixes the bug reported by Zhu Yanjun. This bug would have also affected anyone using rxe.
Reported_by: Zhu Yanjun <[email protected]>
--- drivers/infiniband/sw/rxe/rxe_net.c | 1 + 1 file changed, 1 insertion(+)
drivers/infiniband/sw/rxe/rxe_net.c +1 -0
@@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb, iph->version =IPVERSION ; iph->ihl =sizeof (struct iphdr ) >>2 ; + iph->tot_len =htons (skb->len); iph->frag_off = df; iph->protocol = proto; iph->tos = tos;
Signature · 1 line
2.30.2
Quoted 19 lines · Bob Pearson
An earlier patch removed setting of
tot_lenin IPV4 headers because it was also set inip_local_out. However, this change resulted in an incorrect ICRC being computed because thetot_lenfield is not masked out. This patch restores that line. This fixes the bug reported by Zhu Yanjun. This bug would have also affected anyone using rxe.Fixes 230bb836ee88 ("RDMA/rxe: Fix redundant call to ip_send_check")
Reported_by: Zhu Yanjun <[email protected]>Signed-off-by Bob Pearson <[email protected]>--- drivers/infiniband/sw/rxe/rxe_net.c | 1 + 1 file changed, 1 insertion(+)drivers/infiniband/sw/rxe/rxe_net.c +0
-0
drivers/infiniband/sw/rxe/rxe_net.c static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb,
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_net.c +1
-1@@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb, iph->version =IPVERSION ; iph->ihl =sizeof (struct iphdr ) >>2 ; + iph->tot_len =htons (skb->len); iph->frag_off = df; iph->protocol = proto; iph->tos = tos; --
This patch made the server crash (just like one of the other crashes I've seen and posted to the list).
The client logs:
[ 206.437839] rdma_rxe: bad ICRC from 192.168.1.92 [ 211.043978] rdma_rxe: bad ICRC from 192.168.1.92 [ 215.652973] rdma_rxe: bad ICRC from 192.168.1.92
Server crash:
[11568.440098] BUG: unable to handle page fault for address: ffffaddb21f61180 [11568.442923] #PF: supervisor write access in kernel mode [11568.444452] #PF: error_code(0x0002) - not-present page [11568.445996] PGD 1000067 P4D 1000067 PUD 11b9067 PMD 0 [11568.447527] Oops: 0002 [#1] SMP PTI [11568.448606] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 5.14.0-rc1+ #42 [11568.450911] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [11568.454072] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe] [11568.455613] Code: 8b b3 48 01 00 00 4d 8b 48 08 41 8b 48 28 49 8d b9 80 01 00 00 85 f6 0f 84 78 01 00 00 41 8b 50 34 d3 e2 48 01 fa 48 8b 4d 00 <48> 89 0a 48 8b 4d 08 48 89 4a 08 48 8b 4d 10 48 89 4a 10 48 8b 4d [11568.461093] RSP: 0018:ffffaddb004c0988 EFLAGS: 00010082 [11568.462621] RAX: 0000000000000246 RBX: ffff9c9137df1a00 RCX: 0000000000000000 [11568.464695] RDX: ffffaddb21f61180 RSI: 0000000000000001 RDI: ffffaddb05f5f180 [11568.466779] RBP: ffffaddb004c0a30 R08: ffff9c9123186c00 R09: ffffaddb05f5f000 [11568.468902] R10: 80139a1c70550000 R11: 400000005d050000 R12: 0000000000000000 [11568.470977] R13: ffff9c9137df1b40 R14: ffff9c9137d50008 R15: 000000000000000a [11568.473050] FS: 0000000000000000(0000) GS:ffff9c917be40000(0000) knlGS:0000000000000000 [11568.475395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.477090] CR2: ffffaddb21f61180 CR3: 0000000043476005 CR4: 00000000001706e0 [11568.479170] Call Trace: [11568.479966] <IRQ> [11568.480683] rxe_responder+0x612/0x2470 [rdma_rxe] [11568.482122] rxe_do_task+0x89/0x100 [rdma_rxe] [11568.483427] rxe_rcv+0x2eb/0x900 [rdma_rxe] [11568.484655] ? __udp4_lib_lookup+0x2c8/0x440 [11568.486159] rxe_udp_encap_recv+0x68/0xa0 [rdma_rxe] [11568.487721] ? rxe_enable_task+0x10/0x10 [rdma_rxe] [11568.489223] udp_queue_rcv_one_skb+0x1df/0x4e0 [11568.490528] udp_unicast_rcv_skb.isra.67+0x74/0x90 [11568.491926] __udp4_lib_rcv+0x555/0xb90 [11568.493053] ip_protocol_deliver_rcu+0xe8/0x1b0 [11568.494479] ip_local_deliver_finish+0x44/0x50 [11568.496204] ip_local_deliver+0xf1/0x100 [11568.497621] ? ip_protocol_deliver_rcu+0x1b0/0x1b0 [11568.499147] ip_rcv+0xcb/0xe0 [11568.500032] __netif_receive_skb_core+0x3a2/0x1010 [11568.501491] ? packet_rcv+0x40/0x4b0 [11568.502661] ? select_idle_sibling+0x29/0x970 [11568.504019] __netif_receive_skb_one_core+0x3c/0xa0 [11568.505455] netif_receive_skb+0x3d/0x130 [11568.506650] vmxnet3_rq_rx_complete+0x5f0/0xdc0 [vmxnet3] [11568.508808] vmxnet3_poll_rx_only+0x31/0xa0 [vmxnet3] [11568.510526] __napi_poll+0x2b/0x120 [11568.511596] net_rx_action+0xe2/0x240 [11568.512678] ? vmxnet3_msix_rx+0x4a/0x60 [vmxnet3] [11568.514084] __do_softirq+0xd9/0x2a1 [11568.515218] irq_exit_rcu+0xba/0xd0 [11568.516272] common_interrupt+0x77/0x90 [11568.517438] </IRQ> [11568.518059] asm_common_interrupt+0x1e/0x40 [11568.519291] RIP: 0010:acpi_idle_do_entry+0x4c/0x50 [11568.520680] Code: 08 48 8b 15 3a e3 94 01 ed c3 e9 5f fc ff ff 65 48 8b 04 25 00 6f 01 00 48 8b 00 a8 08 75 ea eb 07 0f 00 2d 40 41 50 00 fb f4 <fa> c3 cc cc 0f 1f 44 00 00 41 55 41 89 d5 41 54 49 89 f4 55 53 48 [11568.526026] RSP: 0018:ffffaddb0009be68 EFLAGS: 00000246 [11568.527569] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff9c917be40000 [11568.529627] RDX: 0000000000000001 RSI: ffffffff9dcc99c0 RDI: ffff9c917c03b464 [11568.531723] RBP: ffff9c9105f63400 R08: ffff9c917c03b400 R09: 000000000000b0e0 [11568.533772] R10: 0000000000001e99 R11: ffff9c917be6a984 R12: ffffffff9dcc9a40 [11568.535918] R13: ffffffff9dcc99c0 R14: 0000000000000001 R15: 0000000000000000 [11568.538558] ? sched_clock_cpu+0x9/0xa0 [11568.539706] acpi_idle_enter+0x4d/0xb0 [11568.540912] cpuidle_enter_state+0x8c/0x350 [11568.542164] cpuidle_enter+0x29/0x40 [11568.543211] do_idle+0x257/0x2a0 [11568.544303] cpu_startup_entry+0x19/0x20 [11568.545455] start_secondary+0x116/0x150 [11568.546928] secondary_startup_64_no_verify+0xc2/0xcb [11568.548479] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm
rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core fuse rfcomm xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 nft_counter nft_compat nf_tables nfnetlink tun bridge stp llc vmw_vsock_vmci_transport vsock bnep snd_seq_midi snd_seq_midi_event intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul vmw_balloon ghash_clmulni_intel joydev pcspkr btusb btrtl btbcm btintel bluetooth uvcvideo rfkill videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_ens1371 snd_ac97_codec ac97_bus snd_seq videobuf2_common snd_pcm videodev mc ecdh_generic ecc snd_timer snd_rawmidi snd_seq_device snd soundcore vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sg crc32c_intel ata_generic vmwgfx ttm serio_raw nvme drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvme_core t10_pi cec ata_piix ahci vmxnet3 libahci drm libata
[11568.575542] CR2: ffffaddb21f61180 [11568.577210] ---[ end trace 8afcc89bb91d9b85 ]--- [11568.578573] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe]
Quoted 2 lines
2.30.2
OK. For tomorrow. I need to know more about your setup. Which versions of kernel, rdma-core and what application SW you are running so I can try to reproduce your results.
Regards,
Bob Pearson
-----Original Message-----
From: Olga Kornievskaia <[email protected]>
Sent: Wednesday, July 21, 2021 7:31 PM
To: Bob Pearson <[email protected]>
Cc: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; linux-rdma <[email protected]>
Subject: Re: [PATCH for-next] RDMA/rxe: Fix bug in rxe_net.c
Quoted 28 lines · Bob Pearson
An earlier patch removed setting of
tot_lenin IPV4 headers because it was also set inip_local_out. However, this change resulted in an incorrect ICRC being computed because thetot_lenfield is not masked out. This patch restores that line. This fixes the bug reported by Zhu Yanjun. This bug would have also affected anyone using rxe.Fixes 230bb836ee88 ("RDMA/rxe: Fix redundant call to ip_send_check")
Reported_by: Zhu Yanjun <[email protected]>Signed-off-by Bob Pearson <[email protected]>--- drivers/infiniband/sw/rxe/rxe_net.c | 1 + 1 file changed, 1 insertion(+)drivers/infiniband/sw/rxe/rxe_net.c +1
-0@@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb, iph->version =IPVERSION ; iph->ihl =sizeof (struct iphdr ) >>2 ; + iph->tot_len =htons (skb->len); iph->frag_off = df; iph->protocol = proto;iph->tos = tos;
Quoted 1 line
--
This patch made the server crash (just like one of the other crashes I've seen and posted to the list).
The client logs:
[ 206.437839] rdma_rxe: bad ICRC from 192.168.1.92 [ 211.043978] rdma_rxe: bad ICRC from 192.168.1.92 [ 215.652973] rdma_rxe: bad ICRC from 192.168.1.92
Server crash:
[11568.440098] BUG: unable to handle page fault for address: ffffaddb21f61180 [11568.442923] #PF: supervisor write access in kernel mode [11568.444452] #PF: error_code(0x0002) - not-present page [11568.445996] PGD 1000067 P4D 1000067 PUD 11b9067 PMD 0 [11568.447527] Oops: 0002 [#1] SMP PTI [11568.448606] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 5.14.0-rc1+ #42 [11568.450911] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [11568.454072] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe] [11568.455613] Code: 8b b3 48 01 00 00 4d 8b 48 08 41 8b 48 28 49 8d b9 80 01 00 00 85 f6 0f 84 78 01 00 00 41 8b 50 34 d3 e2 48 01 fa 48 8b 4d 00 <48> 89 0a 48 8b 4d 08 48 89 4a 08 48 8b 4d 10 48 89 4a 10 48 8b 4d [11568.461093] RSP: 0018:ffffaddb004c0988 EFLAGS: 00010082 [11568.462621] RAX: 0000000000000246 RBX: ffff9c9137df1a00 RCX: 0000000000000000 [11568.464695] RDX: ffffaddb21f61180 RSI: 0000000000000001 RDI: ffffaddb05f5f180 [11568.466779] RBP: ffffaddb004c0a30 R08: ffff9c9123186c00 R09: ffffaddb05f5f000 [11568.468902] R10: 80139a1c70550000 R11: 400000005d050000 R12: 0000000000000000 [11568.470977] R13: ffff9c9137df1b40 R14: ffff9c9137d50008 R15: 000000000000000a [11568.473050] FS: 0000000000000000(0000) GS:ffff9c917be40000(0000) knlGS:0000000000000000 [11568.475395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.477090] CR2: ffffaddb21f61180 CR3: 0000000043476005 CR4: 00000000001706e0 [11568.479170] Call Trace: [11568.479966] <IRQ> [11568.480683] rxe_responder+0x612/0x2470 [rdma_rxe] [11568.482122] rxe_do_task+0x89/0x100 [rdma_rxe] [11568.483427] rxe_rcv+0x2eb/0x900 [rdma_rxe] [11568.484655] ? __udp4_lib_lookup+0x2c8/0x440 [11568.486159] rxe_udp_encap_recv+0x68/0xa0 [rdma_rxe] [11568.487721] ? rxe_enable_task+0x10/0x10 [rdma_rxe] [11568.489223] udp_queue_rcv_one_skb+0x1df/0x4e0 [11568.490528] udp_unicast_rcv_skb.isra.67+0x74/0x90 [11568.491926] __udp4_lib_rcv+0x555/0xb90 [11568.493053] ip_protocol_deliver_rcu+0xe8/0x1b0 [11568.494479] ip_local_deliver_finish+0x44/0x50 [11568.496204] ip_local_deliver+0xf1/0x100 [11568.497621] ? ip_protocol_deliver_rcu+0x1b0/0x1b0 [11568.499147] ip_rcv+0xcb/0xe0 [11568.500032] __netif_receive_skb_core+0x3a2/0x1010 [11568.501491] ? packet_rcv+0x40/0x4b0 [11568.502661] ? select_idle_sibling+0x29/0x970 [11568.504019] __netif_receive_skb_one_core+0x3c/0xa0 [11568.505455] netif_receive_skb+0x3d/0x130 [11568.506650] vmxnet3_rq_rx_complete+0x5f0/0xdc0 [vmxnet3] [11568.508808] vmxnet3_poll_rx_only+0x31/0xa0 [vmxnet3] [11568.510526] __napi_poll+0x2b/0x120 [11568.511596] net_rx_action+0xe2/0x240 [11568.512678] ? vmxnet3_msix_rx+0x4a/0x60 [vmxnet3] [11568.514084] __do_softirq+0xd9/0x2a1 [11568.515218] irq_exit_rcu+0xba/0xd0 [11568.516272] common_interrupt+0x77/0x90 [11568.517438] </IRQ> [11568.518059] asm_common_interrupt+0x1e/0x40 [11568.519291] RIP: 0010:acpi_idle_do_entry+0x4c/0x50 [11568.520680] Code: 08 48 8b 15 3a e3 94 01 ed c3 e9 5f fc ff ff 65 48 8b 04 25 00 6f 01 00 48 8b 00 a8 08 75 ea eb 07 0f 00 2d 40 41 50 00 fb f4 <fa> c3 cc cc 0f 1f 44 00 00 41 55 41 89 d5 41 54 49 89 f4 55 53 48 [11568.526026] RSP: 0018:ffffaddb0009be68 EFLAGS: 00000246 [11568.527569] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff9c917be40000 [11568.529627] RDX: 0000000000000001 RSI: ffffffff9dcc99c0 RDI: ffff9c917c03b464 [11568.531723] RBP: ffff9c9105f63400 R08: ffff9c917c03b400 R09: 000000000000b0e0 [11568.533772] R10: 0000000000001e99 R11: ffff9c917be6a984 R12: ffffffff9dcc9a40 [11568.535918] R13: ffffffff9dcc99c0 R14: 0000000000000001 R15: 0000000000000000 [11568.538558] ? sched_clock_cpu+0x9/0xa0 [11568.539706] acpi_idle_enter+0x4d/0xb0 [11568.540912] cpuidle_enter_state+0x8c/0x350 [11568.542164] cpuidle_enter+0x29/0x40 [11568.543211] do_idle+0x257/0x2a0 [11568.544303] cpu_startup_entry+0x19/0x20 [11568.545455] start_secondary+0x116/0x150 [11568.546928] secondary_startup_64_no_verify+0xc2/0xcb [11568.548479] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core fuse rfcomm xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT
nf_reject_ipv4 nft_counter nft_compat nf_tables nfnetlink tun bridge stp llc vmw_vsock_vmci_transport vsock bnep snd_seq_midi snd_seq_midi_event intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul vmw_balloon ghash_clmulni_intel joydev pcspkr btusb btrtl btbcm btintel bluetooth uvcvideo rfkill videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_ens1371 snd_ac97_codec ac97_bus snd_seq videobuf2_common snd_pcm videodev mc ecdh_generic ecc snd_timer snd_rawmidi snd_seq_device snd soundcore vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sg crc32c_intel ata_generic vmwgfx ttm serio_raw nvme drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvme_core t10_pi cec ata_piix ahci vmxnet3 libahci drm libata [11568.575542] CR2: ffffaddb21f61180 [11568.577210] ---[ end trace 8afcc89bb91d9b85 ]--- [11568.578573] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe]
Quoted 2 lines
2.30.2
I'm RHEL based in terms of my userland software. I work on upstream kernels using that.
Client is running kernel version 5.14-rc1 (when I started, now rc2) on a RHEL8.4 (beta, when I started) VM (RHEL8.2 VM with 5.14-rc1 kernel for server). RHEL8.4 beta that came with userland versions [aglo@localhost linux-nfs]$ rpm -qa | grep rdma rdma-core-devel-32.0-1.el8.x86_64 librdmacm-utils-32.0-1.el8.x86_64 rdma-core-32.0-1.el8.x86_64
librdmacm-32.0-1.el8.x86_64
I upgraded to RHEL8.4GA to make sure it's on an official release of the userspace. The results are the same (at the end of the mail). [root@localhost yum.repos.d]# rpm -qa | grep rdma rdma-core-32.0-4.el8.x86_64 rdma-core-devel-32.0-4.el8.x86_64 librdmacm-utils-32.0-4.el8.x86_64 librdmacm-32.0-4.el8.x86_64
Now, let's go back to NFSoRDMA so that we remove the variable of what version are the userland libraries (and if there are any interoperability issues with kernel changes and userland). Doing an NFS mount, leads to client logging continuously logging "bad ICRC" until mount fails with connection refused.
Network trace has "ConnectRequest" which gets back ConnectReject (reason 0x001c) which I'm assuming is bad ICRC?
nfs oops (that doesn't actually crash the machine which is nice) (this is a snippet and doesn't reflect the #of bad ICRC message in total):
[ 342.290895] rdma_rxe: bad ICRC from 192.168.1.92 [ 348.947562] rdma_rxe: bad ICRC from 192.168.1.92 [ 355.602913] rdma_rxe: invalid mask or state for qp [ 355.606411] rdma_rxe: invalid mask or state for qp [ 355.608928] ------------[ cut here ]------------ [ 355.610831] failed to drain recv queue: -22 [ 355.612549] WARNING: CPU: 1 PID: 516 at drivers/infiniband/core/verbs.c:2738 __ib_drain_rq+0x258/0x290 [ib_core] [ 355.616200] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver
nfs lockd grace rpcrdma rdma_ucm rdma_cm iw_cm ib_cm rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core uinput nls_utf8 isofs rfcomm xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc bnep vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock intel_rapl_msr snd_seq_midi snd_seq_midi_event intel_rapl_common crct10dif_pclmul crc32_pclmul vmw_balloon ghash_clmulni_intel rapl snd_ens1371 joydev pcspkr snd_ac97_codec ac97_bus snd_seq uvcvideo btusb snd_pcm videobuf2_vmalloc btrtl videobuf2_memops btbcm videobuf2_v4l2 btintel videobuf2_common bluetooth videodev snd_timer rfkill snd_rawmidi snd_seq_device mc ecdh_generic snd ecc soundcore vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sg ata_generic crc32c_intel vmwgfx ttm drm_kms_helper nvme ahci syscopyarea
[ 355.616399] sysfillrect libahci sysimgblt ata_piix serio_raw fb_sys_fops drm nvme_core libata vmxnet3 t10_pi dm_mirror dm_region_hash dm_log dm_mod fuse [ 355.648889] CPU: 1 PID: 516 Comm: kworker/u256:28 Tainted: G W 5.14.0-rc2+ #199 [ 355.651852] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [ 355.655245] Workqueue: xprtiod xprt_autoclose [sunrpc] [ 355.657033] RIP: 0010:__ib_drain_rq+0x258/0x290 [ib_core] [ 355.658808] Code: 00 00 00 48 89 ef e8 f7 a9 cc de 48 85 c0 74 e1 e9 f6 fe ff ff 89 c6 48 c7 c7 40 09 4d c1 c6 05 0a 60 08 00 01 e8 da 29 c6 de <0f> 0b e9 da fe ff ff 80 3d f6 5f 08 00 00 0f 85 cd fe ff ff 89 c6 [ 355.665601] RSP: 0018:ffff888008cc7b48 EFLAGS: 00010286 [ 355.667435] RAX: 0000000000000000 RBX: 1ffff11001198f69 RCX: ffffffff9f427a3e [ 355.669758] RDX: 1ffff1100b98cd35 RSI: 0000000000000008 RDI: ffff88805cc669ac [ 355.672397] RBP: ffff88801c83c058 R08: ffffed100b98df31 R09: ffffed100b98df31 [ 355.675018] R10: ffff88805cc6f987 R11: ffffed100b98df30 R12: ffff88801312f000 [ 355.677844] R13: ffff8880183ef810 R14: ffffffffc18173c0 R15: ffff888001119000 [ 355.680599] FS: 0000000000000000(0000) GS:ffff88805cc40000(0000) knlGS:0000000000000000 [ 355.683904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 355.686447] CR2: 00003bafb8832fd0 CR3: 00000000142e6002 CR4: 00000000001706e0 [ 355.689154] Call Trace: [ 355.690157] ? __ib_drain_sq+0x280/0x280 [ib_core] [ 355.692013] ? autoremove_wake_function+0x82/0xa0 [ 355.694000] ? mutex_lock+0x8e/0xe0 [ 355.695683] ? mutex_unlock+0x1d/0x40 [ 355.697821] ? cma_modify_qp_err+0xa5/0xf0 [rdma_cm] [ 355.700409] ? rdma_unlock_handler+0x20/0x20 [rdma_cm] [ 355.702602] ? __update_load_avg_cfs_rq+0x5a/0x550 [ 355.704558] ib_drain_rq+0x9f/0xb0 [ib_core] [ 355.706253] rpcrdma_xprt_disconnect+0xbe/0x4b0 [rpcrdma] [ 355.708215] xprt_rdma_close+0xe/0x50 [rpcrdma] [ 355.709785] xprt_autoclose+0x8b/0x160 [sunrpc] [ 355.711810] process_one_work+0x3ab/0x6b0 [ 355.713303] worker_thread+0x57/0x5c0 [ 355.714477] ? process_one_work+0x6b0/0x6b0 [ 355.715806] kthread+0x1bf/0x1f0 [ 355.716901] ? set_kthread_struct+0x80/0x80 [ 355.718333] ret_from_fork+0x22/0x30 [ 355.719577] ---[ end trace dc0181bd9d91f55b ]--- [ 355.721135] rdma_rxe: invalid mask or state for qp [ 355.723117] ------------[ cut here ]------------
rping oops.
[13873.255148] rdma_rxe: bad ICRC from 192.168.1.92 [13877.567475] rdma_rxe: bad ICRC from 192.168.1.92 [13882.175544] rdma_rxe: bad ICRC from 192.168.1.92 [13886.784329] rdma_rxe: bad ICRC from 192.168.1.92 [13891.391534] rdma_rxe: bad ICRC from 192.168.1.92 [13896.000084] rdma_rxe: bad ICRC from 192.168.1.92 [13900.608291] rdma_rxe: bad ICRC from 192.168.1.92 [13905.219925] rdma_rxe: bad ICRC from 192.168.1.92 [13905.222298] rdma_rxe: bad ICRC from 192.168.1.92 [13907.392305] rdma_rxe: bad ICRC from 192.168.1.92 [13909.569156] rdma_rxe: bad ICRC from 192.168.1.92 [13911.744391] rdma_rxe: bad ICRC from 192.168.1.92 [13913.921244] rdma_rxe: bad ICRC from 192.168.1.92 [13916.097423] rdma_rxe: bad ICRC from 192.168.1.92 [13918.272800] rdma_rxe: bad ICRC from 192.168.1.92 [13920.449837] BUG: unable to handle page fault for address: ffffc90103782194 [13920.453440] #PF: supervisor read access in kernel mode [13920.455627] #PF: error_code(0x0000) - not-present page [13920.457585] PGD 1000067 P4D 1000067 PUD 0 [13920.459103] Oops: 0000 [#1] SMP KASAN PTI [13920.460659] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.14.0-rc2+ #199 [13920.463284] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [13920.466820] RIP: 0010:copy_data+0x45/0x3a0 [rdma_rxe] [13920.468732] Code: 20 48 89 0c 24 44 89 4c 24 10 45 85 c0 0f 84 6c 01 00 00 48 8d 42 04 49 89 d4 45 89 c5 48 89 c7 48 89 44 24 30 e8 fb c0 6e ec <45> 8b 7c 24 04 44 89 7c 24 14 45 39 ef 0f 8c 08 03 00 00 49 8d 44 [13920.474397] RSP: 0018:ffff88805cc092e0 EFLAGS: 00010246 [13920.476010] RAX: 0000000000000000 RBX: ffff88800ed36520 RCX: ffffffffc15b4555 [13920.478234] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: ffffc90103782194 [13920.480721] RBP: ffff888042fc9a48 R08: 0000000000000010 R09: 0000000000000000 [13920.483235] R10: ffff888042fc9a55 R11: ffffed1001da6c01 R12: ffffc90103782190 [13920.485626] R13: 0000000000000010 R14: ffff8880172a536a R15: ffff88800ed36000 [13920.488373] FS: 0000000000000000(0000) GS:ffff88805cc00000(0000) knlGS:0000000000000000 [13920.491327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13920.493344] CR2: ffffc90103782194 CR3: 0000000044dac005 CR4: 00000000001706f0 [13920.495705] Call Trace: [13920.496608] <IRQ> [13920.497554] send_data_in.isra.30+0x21/0x40 [rdma_rxe] [13920.499371] rxe_responder+0x1a06/0x3e50 [rdma_rxe] [13920.500970] ? fib_info_nh_uses_dev+0x6d/0x320 [13920.502530] ? rxe_resp_queue_pkt+0x60/0x60 [rdma_rxe] [13920.504183] ? crc32_pclmul_update+0x36/0x42 [crc32_pclmul] [13920.505895] ? rxe_crc32.isra.14+0x7d/0x100 [rdma_rxe] [13920.507485] ? check_type_state.isra.8+0x150/0x150 [rdma_rxe] [13920.509248] ? find_gid+0x166/0x210 [ib_core] [13920.510978] ? _raw_spin_lock_irqsave+0x80/0xe0 [13920.512449] ? _raw_write_lock_irqsave+0xe0/0xe0 [13920.513877] ? rxe_resp_queue_pkt+0x60/0x60 [rdma_rxe] [13920.515506] rxe_do_task+0xd2/0x160 [rdma_rxe] [13920.516881] rxe_rcv+0x5a5/0xe30 [rdma_rxe] [13920.518510] ? rxe_crc32.isra.14+0x100/0x100 [rdma_rxe] [13920.520297] ? __udp4_lib_lookup+0x3fa/0x5b0 [13920.521617] ? ib_device_get_by_netdev+0x165/0x1b0 [ib_core] [13920.523403] ? ib_unregister_driver+0x170/0x170 [ib_core] [13920.525327] ? stack_access_ok+0x35/0x80 [13920.526808] rxe_udp_encap_recv+0xd0/0x120 [rdma_rxe] [13920.528541] ? rxe_enable_task+0x20/0x20 [rdma_rxe] [13920.530252] udp_queue_rcv_one_skb+0x36d/0x8a0 [13920.531985] udp_unicast_rcv_skb.isra.65+0x126/0x140 [13920.533800] __udp4_lib_rcv+0x924/0x1310 [13920.535186] ? udp_err+0x20/0x20 [13920.536190] ? is_bpf_text_address+0x13/0x20 [13920.537554] ? kernel_text_address+0x100/0x110 [13920.538944] ? __unwind_start+0x2e8/0x370 [13920.540193] ? raw_rcv+0x1a0/0x1a0 [13920.541253] ? nft_do_chain_arp+0xa0/0xa0 [nf_tables] [13920.542913] ? nft_do_chain_ipv4+0xe4/0x110 [nf_tables] [13920.544569] ? nf_nat_ipv4_fn+0x21/0xc0 [nf_nat] [13920.546109] ip_protocol_deliver_rcu+0x170/0x2c0 [13920.547907] ip_local_deliver_finish+0xae/0xc0 [13920.549598] ip_local_deliver+0x1ae/0x1c0 [13920.551031] ? ip_local_deliver_finish+0xc0/0xc0 [13920.552586] ? ip_route_input_rcu+0x421/0x4b0 [13920.554071] ? ip_protocol_deliver_rcu+0x2c0/0x2c0 [13920.555662] ? ip_sublist_rcv+0x3c0/0x3c0 [13920.556962] ? ip_sublist_rcv+0x3c0/0x3c0 [13920.558439] ip_rcv+0x159/0x160 [13920.559549] ? ip_sublist_rcv+0x3c0/0x3c0 [13920.560782] ? secondary_startup_64_no_verify+0xc2/0xcb [13920.562683] ? remove_all_stable_nodes+0x40/0x190 [13920.564674] ? ip_local_deliver+0x1c0/0x1c0 [13920.566054] ? __napi_poll+0x5d/0x1f0 [13920.567310] ? net_rx_action+0x21c/0x4a0 [13920.568616] ? __do_softirq+0xf9/0x376 [13920.569809] __netif_receive_skb_one_core+0x133/0x150 [13920.571350] ? __netif_receive_skb_core+0x1760/0x1760 [13920.572889] ? ip_finish_output+0xc0/0xc0 [13920.574123] ? _raw_spin_lock_irqsave+0x80/0xe0 [13920.575505] ? _raw_write_lock_irqsave+0xe0/0xe0 [13920.576910] ? kasan_set_track+0x1c/0x30 [13920.578205] netif_receive_skb+0x94/0x240 [13920.579667] ? __netif_receive_skb+0xa0/0xa0 [13920.581132] ? eth_type_trans+0x134/0x270 [13920.582422] ? eth_gro_receive+0x310/0x310 [13920.583679] ? __build_skb_around+0x10e/0x130 [13920.585023] ? dma_unmap_page_attrs+0x1c6/0x2d0 [13920.586439] vmxnet3_rq_rx_complete+0xa76/0x17b0 [vmxnet3] [13920.588146] vmxnet3_poll_rx_only+0x47/0xd0 [vmxnet3] [13920.589693] __napi_poll+0x5d/0x1f0 [13920.590766] net_rx_action+0x21c/0x4a0 [13920.591918] ? napi_threaded_poll+0x1c0/0x1c0 [13920.593253] ? vmxnet3_msix_tx+0x100/0x100 [vmxnet3] [13920.594792] ? note_interrupt+0xf0/0x3a0 [13920.596042] ? add_interrupt_randomness+0x15f/0x2a0 [13920.597677] ? _raw_spin_lock+0x7a/0xd0 [13920.598853] ? _raw_write_lock_bh+0xe0/0xe0 [13920.600144] __do_softirq+0xf9/0x376 [13920.601247] irq_exit_rcu+0x118/0x130 [13920.602435] common_interrupt+0x77/0x90 [13920.603712] </IRQ> [13920.604411] asm_common_interrupt+0x1e/0x40 [13920.606075] RIP: 0010:acpi_idle_do_entry+0x61/0x70 [13920.607750] Code: ef 01 00 be 08 00 00 00 48 89 df e8 89 10 54 ff 48 89 df e8 41 06 54 ff 48 8b 03 a8 08 75 0c eb 07 0f 00 2d 01 b1 73 00 fb f4 <fa> 5b c3 48 89 df 5b e9 93 f9 ff ff cc cc cc 0f 1f 44 00 00 41 57 [13920.613840] RSP: 0018:ffffffffaf407d98 EFLAGS: 00000246 [13920.615463] RAX: 0000000000004000 RBX: ffffffffaf41a400 RCX: ffffffffae76014f [13920.617651] RDX: dffffc0000000000 RSI: 0000000000000008 RDI: ffffffffaf41a400 [13920.619820] RBP: 0000000000000001 R08: fffffbfff5e83481 R09: fffffbfff5e83481 [13920.622005] R10: ffffffffaf41a407 R11: fffffbfff5e83480 R12: ffff88800953a000 [13920.624162] R13: 0000000000000001 R14: ffff88800953a004 R15: ffff8880046c1800 [13920.626335] ? acpi_idle_do_entry+0x4f/0x70 [13920.627740] ? acpi_idle_do_entry+0x4f/0x70 [13920.629020] acpi_idle_enter+0x14d/0x1c0 [13920.630295] cpuidle_enter_state+0xb2/0x590 [13920.631603] ? tick_nohz_stop_tick+0x1f0/0x2d0 [13920.632987] cpuidle_enter+0x3c/0x60 [13920.634136] do_idle+0x399/0x400 [13920.635192] ? arch_cpu_idle_exit+0x40/0x40 [13920.636471] ? do_idle+0x26d/0x400 [13920.637517] cpu_startup_entry+0x19/0x20 [13920.638716] start_kernel+0x378/0x396 [13920.639925] secondary_startup_64_no_verify+0xc2/0xcb [13920.641564] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm
rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core uinput nls_utf8 isofs rfcomm xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc bnep vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock intel_rapl_msr snd_seq_midi snd_seq_midi_event intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel vmw_balloon rapl pcspkr joydev snd_ens1371 snd_ac97_codec ac97_bus btusb snd_seq uvcvideo btrtl btbcm btintel videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common bluetooth snd_pcm videodev mc snd_timer rfkill snd_rawmidi snd_seq_device ecdh_generic snd ecc soundcore vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sg ata_generic crc32c_intel nvme vmwgfx ata_piix serio_raw nvme_core ttm ahci libahci drm_kms_helper libata syscopyarea
[13920.641737] sysfillrect sysimgblt fb_sys_fops vmxnet3 t10_pi drm dm_mirror dm_region_hash dm_log dm_mod fuse [13920.671747] CR2: ffffc90103782194 [13920.674506] ---[ end trace 6ae70b2fba32e277 ]---
Quoted 81 lines · Robert B
OK. For tomorrow. I need to know more about your setup. Which versions of kernel, rdma-core and what application SW you are running so I can try to reproduce your results.
Regards,
Bob Pearson
-----Original Message-----
From: Olga Kornievskaia <[email protected]>
Sent: Wednesday, July 21, 2021 7:31 PM
To: Bob Pearson <[email protected]>
Cc: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; linux-rdma <[email protected]>
Subject: Re: [PATCH for-next] RDMA/rxe: Fix bug inrxe_net.cQuoted 28 lines · Bob Pearson
An earlier patch removed setting of
tot_lenin IPV4 headers because it was also set inip_local_out. However, this change resulted in an incorrect ICRC being computed because thetot_lenfield is not masked out. This patch restores that line. This fixes the bug reported by Zhu Yanjun. This bug would have also affected anyone using rxe.Fixes 230bb836ee88 ("RDMA/rxe: Fix redundant call to ip_send_check")
Reported_by: Zhu Yanjun <[email protected]>Signed-off-by Bob Pearson <[email protected]>--- drivers/infiniband/sw/rxe/rxe_net.c | 1 + 1 file changed, 1 insertion(+)drivers/infiniband/sw/rxe/rxe_net.c +1
-0@@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb, iph->version =IPVERSION ; iph->ihl =sizeof (struct iphdr ) >>2 ; + iph->tot_len =htons (skb->len); iph->frag_off = df; iph->protocol = proto;iph->tos = tos;Quoted 1 line
--
This patch made the server crash (just like one of the other crashes I've seen and posted to the list).
The client logs:
[ 206.437839] rdma_rxe: bad ICRC from 192.168.1.92 [ 211.043978] rdma_rxe: bad ICRC from 192.168.1.92 [ 215.652973] rdma_rxe: bad ICRC from 192.168.1.92Server crash:
[11568.440098] BUG: unable to handle page fault for address: ffffaddb21f61180 [11568.442923] #PF: supervisor write access in kernel mode [11568.444452] #PF: error_code(0x0002) - not-present page [11568.445996] PGD 1000067 P4D 1000067 PUD 11b9067 PMD 0 [11568.447527] Oops: 0002 [#1] SMP PTI [11568.448606] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 5.14.0-rc1+ #42 [11568.450911] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [11568.454072] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe] [11568.455613] Code: 8b b3 48 01 00 00 4d 8b 48 08 41 8b 48 28 49 8d b9 80 01 00 00 85 f6 0f 84 78 01 00 00 41 8b 50 34 d3 e2 48 01 fa 48 8b 4d 00 <48> 89 0a 48 8b 4d 08 48 89 4a 08 48 8b 4d 10 48 89 4a 10 48 8b 4d [11568.461093] RSP: 0018:ffffaddb004c0988 EFLAGS: 00010082 [11568.462621] RAX: 0000000000000246 RBX: ffff9c9137df1a00 RCX: 0000000000000000 [11568.464695] RDX: ffffaddb21f61180 RSI: 0000000000000001 RDI: ffffaddb05f5f180 [11568.466779] RBP: ffffaddb004c0a30 R08: ffff9c9123186c00 R09: ffffaddb05f5f000 [11568.468902] R10: 80139a1c70550000 R11: 400000005d050000 R12: 0000000000000000 [11568.470977] R13: ffff9c9137df1b40 R14: ffff9c9137d50008 R15: 000000000000000a [11568.473050] FS: 0000000000000000(0000) GS:ffff9c917be40000(0000) knlGS:0000000000000000 [11568.475395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.477090] CR2: ffffaddb21f61180 CR3: 0000000043476005 CR4: 00000000001706e0 [11568.479170] Call Trace: [11568.479966] <IRQ> [11568.480683] rxe_responder+0x612/0x2470 [rdma_rxe] [11568.482122] rxe_do_task+0x89/0x100 [rdma_rxe] [11568.483427] rxe_rcv+0x2eb/0x900 [rdma_rxe] [11568.484655] ? __udp4_lib_lookup+0x2c8/0x440 [11568.486159] rxe_udp_encap_recv+0x68/0xa0 [rdma_rxe] [11568.487721] ? rxe_enable_task+0x10/0x10 [rdma_rxe] [11568.489223] udp_queue_rcv_one_skb+0x1df/0x4e0 [11568.490528] udp_unicast_rcv_skb.isra.67+0x74/0x90 [11568.491926] __udp4_lib_rcv+0x555/0xb90 [11568.493053] ip_protocol_deliver_rcu+0xe8/0x1b0 [11568.494479] ip_local_deliver_finish+0x44/0x50 [11568.496204] ip_local_deliver+0xf1/0x100 [11568.497621] ? ip_protocol_deliver_rcu+0x1b0/0x1b0 [11568.499147] ip_rcv+0xcb/0xe0 [11568.500032] __netif_receive_skb_core+0x3a2/0x1010 [11568.501491] ? packet_rcv+0x40/0x4b0 [11568.502661] ? select_idle_sibling+0x29/0x970 [11568.504019] __netif_receive_skb_one_core+0x3c/0xa0 [11568.505455] netif_receive_skb+0x3d/0x130 [11568.506650] vmxnet3_rq_rx_complete+0x5f0/0xdc0 [vmxnet3] [11568.508808] vmxnet3_poll_rx_only+0x31/0xa0 [vmxnet3] [11568.510526] __napi_poll+0x2b/0x120 [11568.511596] net_rx_action+0xe2/0x240 [11568.512678] ? vmxnet3_msix_rx+0x4a/0x60 [vmxnet3] [11568.514084] __do_softirq+0xd9/0x2a1 [11568.515218] irq_exit_rcu+0xba/0xd0 [11568.516272] common_interrupt+0x77/0x90 [11568.517438] </IRQ> [11568.518059] asm_common_interrupt+0x1e/0x40 [11568.519291] RIP: 0010:acpi_idle_do_entry+0x4c/0x50 [11568.520680] Code: 08 48 8b 15 3a e3 94 01 ed c3 e9 5f fc ff ff 65 48 8b 04 25 00 6f 01 00 48 8b 00 a8 08 75 ea eb 07 0f 00 2d 40 41 50 00 fb f4 <fa> c3 cc cc 0f 1f 44 00 00 41 55 41 89 d5 41 54 49 89 f4 55 53 48 [11568.526026] RSP: 0018:ffffaddb0009be68 EFLAGS: 00000246 [11568.527569] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff9c917be40000 [11568.529627] RDX: 0000000000000001 RSI: ffffffff9dcc99c0 RDI: ffff9c917c03b464 [11568.531723] RBP: ffff9c9105f63400 R08: ffff9c917c03b400 R09: 000000000000b0e0 [11568.533772] R10: 0000000000001e99 R11: ffff9c917be6a984 R12: ffffffff9dcc9a40 [11568.535918] R13: ffffffff9dcc99c0 R14: 0000000000000001 R15: 0000000000000000 [11568.538558] ? sched_clock_cpu+0x9/0xa0 [11568.539706] acpi_idle_enter+0x4d/0xb0 [11568.540912] cpuidle_enter_state+0x8c/0x350 [11568.542164] cpuidle_enter+0x29/0x40 [11568.543211] do_idle+0x257/0x2a0 [11568.544303] cpu_startup_entry+0x19/0x20 [11568.545455] start_secondary+0x116/0x150 [11568.546928] secondary_startup_64_no_verify+0xc2/0xcb [11568.548479] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core fuse rfcomm xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT
nf_reject_ipv4nft_counternft_compatnf_tablesnfnetlink tun bridge stp llcvmw_vsock_vmci_transportvsock bnepsnd_seq_midisnd_seq_midi_eventintel_rapl_msrintel_rapl_commoncrct10dif_pclmulcrc32_pclmulvmw_balloonghash_clmulni_inteljoydev pcspkr btusb btrtl btbcm btintel bluetooth uvcvideo rfkillvideobuf2_vmallocvideobuf2_memopsvideobuf2_v4l2snd_ens1371snd_ac97_codecac97_bussnd_seqvideobuf2_commonsnd_pcmvideodev mcecdh_genericeccsnd_timersnd_rawmidisnd_seq_devicesnd soundcorevmw_vmcii2c_piix4auth_rpcgsssunrpcip_tablesxfs libcrc32csr_modcdrom sgcrc32c_intelata_genericvmwgfx ttmserio_rawnvmedrm_kms_helpersyscopyarea sysfillrect sysimgbltfb_sys_fopsnvme_coret10_picecata_piixahci vmxnet3 libahci drm libata [11568.575542] CR2:ffffaddb21f61180[11568.577210] ---[ end trace8afcc89bb91d9b85]--- [11568.578573] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe]Quoted 2 lines
2.30.2
Zhu Yanjun
maintainerQuoted 3 lines · Olga Kornievskaia
I'm RHEL based in terms of my userland software. I work on upstream kernels using that.
Simply, one host with the kernel 5.14-rc1, the other host with the kernel 5.14-rc2.
Then the following errors will appear
"
...
[13873.255148] rdma_rxe: bad ICRC from 192.168.1.92 [13877.567475] rdma_rxe: bad ICRC from 192.168.1.92 [13882.175544] rdma_rxe: bad ICRC from 192.168.1.92
...
"
Correct?
Zhu Yanjun
Quoted 344 lines
Client is running kernel version 5.14-rc1 (when I started, now rc2) on a RHEL8.4 (beta, when I started) VM (RHEL8.2 VM with 5.14-rc1 kernel for server). RHEL8.4 beta that came with userland versions [aglo@localhost linux-nfs]$ rpm -qa | grep rdma rdma-core-devel-32.0-1.el8.x86_64 librdmacm-utils-32.0-1.el8.x86_64 rdma-core-32.0-1.el8.x86_64
librdmacm-32.0-1.el8.x86_64I upgraded to RHEL8.4GA to make sure it's on an official release of the userspace. The results are the same (at the end of the mail). [root@localhost yum.repos.d]# rpm -qa | grep rdma rdma-core-32.0-4.el8.x86_64 rdma-core-devel-32.0-4.el8.x86_64 librdmacm-utils-32.0-4.el8.x86_64 librdmacm-32.0-4.el8.x86_64
Now, let's go back to NFSoRDMA so that we remove the variable of what version are the userland libraries (and if there are any interoperability issues with kernel changes and userland). Doing an NFS mount, leads to client logging continuously logging "bad ICRC" until mount fails with connection refused.
Network trace has "ConnectRequest" which gets back ConnectReject (reason
0x001c) which I'm assuming is bad ICRC?nfs oops (that doesn't actually crash the machine which is nice) (this is a snippet and doesn't reflect the #of bad ICRC message in total):
[ 342.290895] rdma_rxe: bad ICRC from 192.168.1.92 [ 348.947562] rdma_rxe: bad ICRC from 192.168.1.92 [ 355.602913] rdma_rxe: invalid mask or state for qp [ 355.606411] rdma_rxe: invalid mask or state for qp [ 355.608928] ------------[ cut here ]------------ [ 355.610831] failed to drain recv queue: -22 [ 355.612549] WARNING: CPU: 1 PID: 516 at drivers/infiniband/core/verbs.c:2738 __ib_drain_rq+0x258/0x290 [ib_core] [ 355.616200] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolvernfs lockd grace rpcrdma
rdma_ucmrdma_cmiw_cmib_cmrdma_rxeip6_udp_tunneludp_tunnelib_uverbsib_coreuinputnls_utf8isofs rfcommxt_conntrackipt_REJECTnf_reject_ipv4nft_compatnft_counternft_chain_natnf_natnf_conntracknf_defrag_ipv6nf_defrag_ipv4nf_tablesnfnetlink tun bridge stp llc bnepvsock_loopbackvmw_vsock_virtio_transport_commonvmw_vsock_vmci_transportvsockintel_rapl_msrsnd_seq_midisnd_seq_midi_eventintel_rapl_commoncrct10dif_pclmulcrc32_pclmulvmw_balloonghash_clmulni_intelraplsnd_ens1371joydev pcspkrsnd_ac97_codecac97_bussnd_sequvcvideo btusbsnd_pcmvideobuf2_vmallocbtrtlvideobuf2_memopsbtbcmvideobuf2_v4l2btintelvideobuf2_commonbluetooth videodevsnd_timerrfkillsnd_rawmidisnd_seq_devicemcecdh_genericsnd ecc soundcorevmw_vmcii2c_piix4auth_rpcgsssunrpcip_tablesxfs libcrc32csr_modcdrom sgata_genericcrc32c_intelvmwgfx ttmdrm_kms_helpernvme ahci syscopyarea[ 355.616399] sysfillrect libahci sysimgblt ata_piix serio_raw fb_sys_fops drm nvme_core libata vmxnet3 t10_pi dm_mirror dm_region_hash dm_log dm_mod fuse [ 355.648889] CPU: 1 PID: 516 Comm: kworker/u256:28 Tainted: G W 5.14.0-rc2+ #199 [ 355.651852] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [ 355.655245] Workqueue: xprtiod xprt_autoclose [sunrpc] [ 355.657033] RIP: 0010:__ib_drain_rq+0x258/0x290 [ib_core] [ 355.658808] Code: 00 00 00 48 89 ef e8 f7 a9 cc de 48 85 c0 74 e1 e9 f6 fe ff ff 89 c6 48 c7 c7 40 09 4d c1 c6 05 0a 60 08 00 01 e8 da 29 c6 de <0f> 0b e9 da fe ff ff 80 3d f6 5f 08 00 00 0f 85 cd fe ff ff 89 c6 [ 355.665601] RSP: 0018:ffff888008cc7b48 EFLAGS: 00010286 [ 355.667435] RAX: 0000000000000000 RBX: 1ffff11001198f69 RCX: ffffffff9f427a3e [ 355.669758] RDX: 1ffff1100b98cd35 RSI: 0000000000000008 RDI: ffff88805cc669ac [ 355.672397] RBP: ffff88801c83c058 R08: ffffed100b98df31 R09: ffffed100b98df31 [ 355.675018] R10: ffff88805cc6f987 R11: ffffed100b98df30 R12: ffff88801312f000 [ 355.677844] R13: ffff8880183ef810 R14: ffffffffc18173c0 R15: ffff888001119000 [ 355.680599] FS: 0000000000000000(0000) GS:ffff88805cc40000(0000) knlGS:0000000000000000 [ 355.683904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 355.686447] CR2: 00003bafb8832fd0 CR3: 00000000142e6002 CR4: 00000000001706e0 [ 355.689154] Call Trace: [ 355.690157] ? __ib_drain_sq+0x280/0x280 [ib_core] [ 355.692013] ? autoremove_wake_function+0x82/0xa0 [ 355.694000] ? mutex_lock+0x8e/0xe0 [ 355.695683] ? mutex_unlock+0x1d/0x40 [ 355.697821] ? cma_modify_qp_err+0xa5/0xf0 [rdma_cm] [ 355.700409] ? rdma_unlock_handler+0x20/0x20 [rdma_cm] [ 355.702602] ? __update_load_avg_cfs_rq+0x5a/0x550 [ 355.704558] ib_drain_rq+0x9f/0xb0 [ib_core] [ 355.706253] rpcrdma_xprt_disconnect+0xbe/0x4b0 [rpcrdma] [ 355.708215] xprt_rdma_close+0xe/0x50 [rpcrdma] [ 355.709785] xprt_autoclose+0x8b/0x160 [sunrpc] [ 355.711810] process_one_work+0x3ab/0x6b0 [ 355.713303] worker_thread+0x57/0x5c0 [ 355.714477] ? process_one_work+0x6b0/0x6b0 [ 355.715806] kthread+0x1bf/0x1f0 [ 355.716901] ? set_kthread_struct+0x80/0x80 [ 355.718333] ret_from_fork+0x22/0x30 [ 355.719577] ---[ end trace dc0181bd9d91f55b ]--- [ 355.721135] rdma_rxe: invalid mask or state for qp [ 355.723117] ------------[ cut here ]------------rping oops.
[13873.255148] rdma_rxe: bad ICRC from 192.168.1.92 [13877.567475] rdma_rxe: bad ICRC from 192.168.1.92 [13882.175544] rdma_rxe: bad ICRC from 192.168.1.92 [13886.784329] rdma_rxe: bad ICRC from 192.168.1.92 [13891.391534] rdma_rxe: bad ICRC from 192.168.1.92 [13896.000084] rdma_rxe: bad ICRC from 192.168.1.92 [13900.608291] rdma_rxe: bad ICRC from 192.168.1.92 [13905.219925] rdma_rxe: bad ICRC from 192.168.1.92 [13905.222298] rdma_rxe: bad ICRC from 192.168.1.92 [13907.392305] rdma_rxe: bad ICRC from 192.168.1.92 [13909.569156] rdma_rxe: bad ICRC from 192.168.1.92 [13911.744391] rdma_rxe: bad ICRC from 192.168.1.92 [13913.921244] rdma_rxe: bad ICRC from 192.168.1.92 [13916.097423] rdma_rxe: bad ICRC from 192.168.1.92 [13918.272800] rdma_rxe: bad ICRC from 192.168.1.92 [13920.449837] BUG: unable to handle page fault for address: ffffc90103782194 [13920.453440] #PF: supervisor read access in kernel mode [13920.455627] #PF: error_code(0x0000) - not-present page [13920.457585] PGD 1000067 P4D 1000067 PUD 0 [13920.459103] Oops: 0000 [#1] SMP KASAN PTI [13920.460659] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.14.0-rc2+ #199 [13920.463284] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [13920.466820] RIP: 0010:copy_data+0x45/0x3a0 [rdma_rxe] [13920.468732] Code: 20 48 89 0c 24 44 89 4c 24 10 45 85 c0 0f 84 6c 01 00 00 48 8d 42 04 49 89 d4 45 89 c5 48 89 c7 48 89 44 24 30 e8 fb c0 6e ec <45> 8b 7c 24 04 44 89 7c 24 14 45 39 ef 0f 8c 08 03 00 00 49 8d 44 [13920.474397] RSP: 0018:ffff88805cc092e0 EFLAGS: 00010246 [13920.476010] RAX: 0000000000000000 RBX: ffff88800ed36520 RCX: ffffffffc15b4555 [13920.478234] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: ffffc90103782194 [13920.480721] RBP: ffff888042fc9a48 R08: 0000000000000010 R09: 0000000000000000 [13920.483235] R10: ffff888042fc9a55 R11: ffffed1001da6c01 R12: ffffc90103782190 [13920.485626] R13: 0000000000000010 R14: ffff8880172a536a R15: ffff88800ed36000 [13920.488373] FS: 0000000000000000(0000) GS:ffff88805cc00000(0000) knlGS:0000000000000000 [13920.491327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13920.493344] CR2: ffffc90103782194 CR3: 0000000044dac005 CR4: 00000000001706f0 [13920.495705] Call Trace: [13920.496608] <IRQ> [13920.497554] send_data_in.isra.30+0x21/0x40 [rdma_rxe] [13920.499371] rxe_responder+0x1a06/0x3e50 [rdma_rxe] [13920.500970] ? fib_info_nh_uses_dev+0x6d/0x320 [13920.502530] ? rxe_resp_queue_pkt+0x60/0x60 [rdma_rxe] [13920.504183] ? crc32_pclmul_update+0x36/0x42 [crc32_pclmul] [13920.505895] ? rxe_crc32.isra.14+0x7d/0x100 [rdma_rxe] [13920.507485] ? check_type_state.isra.8+0x150/0x150 [rdma_rxe] [13920.509248] ? find_gid+0x166/0x210 [ib_core] [13920.510978] ? _raw_spin_lock_irqsave+0x80/0xe0 [13920.512449] ? _raw_write_lock_irqsave+0xe0/0xe0 [13920.513877] ? rxe_resp_queue_pkt+0x60/0x60 [rdma_rxe] [13920.515506] rxe_do_task+0xd2/0x160 [rdma_rxe] [13920.516881] rxe_rcv+0x5a5/0xe30 [rdma_rxe] [13920.518510] ? rxe_crc32.isra.14+0x100/0x100 [rdma_rxe] [13920.520297] ? __udp4_lib_lookup+0x3fa/0x5b0 [13920.521617] ? ib_device_get_by_netdev+0x165/0x1b0 [ib_core] [13920.523403] ? ib_unregister_driver+0x170/0x170 [ib_core] [13920.525327] ? stack_access_ok+0x35/0x80 [13920.526808] rxe_udp_encap_recv+0xd0/0x120 [rdma_rxe] [13920.528541] ? rxe_enable_task+0x20/0x20 [rdma_rxe] [13920.530252] udp_queue_rcv_one_skb+0x36d/0x8a0 [13920.531985] udp_unicast_rcv_skb.isra.65+0x126/0x140 [13920.533800] __udp4_lib_rcv+0x924/0x1310 [13920.535186] ? udp_err+0x20/0x20 [13920.536190] ? is_bpf_text_address+0x13/0x20 [13920.537554] ? kernel_text_address+0x100/0x110 [13920.538944] ? __unwind_start+0x2e8/0x370 [13920.540193] ? raw_rcv+0x1a0/0x1a0 [13920.541253] ? nft_do_chain_arp+0xa0/0xa0 [nf_tables] [13920.542913] ? nft_do_chain_ipv4+0xe4/0x110 [nf_tables] [13920.544569] ? nf_nat_ipv4_fn+0x21/0xc0 [nf_nat] [13920.546109] ip_protocol_deliver_rcu+0x170/0x2c0 [13920.547907] ip_local_deliver_finish+0xae/0xc0 [13920.549598] ip_local_deliver+0x1ae/0x1c0 [13920.551031] ? ip_local_deliver_finish+0xc0/0xc0 [13920.552586] ? ip_route_input_rcu+0x421/0x4b0 [13920.554071] ? ip_protocol_deliver_rcu+0x2c0/0x2c0 [13920.555662] ? ip_sublist_rcv+0x3c0/0x3c0 [13920.556962] ? ip_sublist_rcv+0x3c0/0x3c0 [13920.558439] ip_rcv+0x159/0x160 [13920.559549] ? ip_sublist_rcv+0x3c0/0x3c0 [13920.560782] ? secondary_startup_64_no_verify+0xc2/0xcb [13920.562683] ? remove_all_stable_nodes+0x40/0x190 [13920.564674] ? ip_local_deliver+0x1c0/0x1c0 [13920.566054] ? __napi_poll+0x5d/0x1f0 [13920.567310] ? net_rx_action+0x21c/0x4a0 [13920.568616] ? __do_softirq+0xf9/0x376 [13920.569809] __netif_receive_skb_one_core+0x133/0x150 [13920.571350] ? __netif_receive_skb_core+0x1760/0x1760 [13920.572889] ? ip_finish_output+0xc0/0xc0 [13920.574123] ? _raw_spin_lock_irqsave+0x80/0xe0 [13920.575505] ? _raw_write_lock_irqsave+0xe0/0xe0 [13920.576910] ? kasan_set_track+0x1c/0x30 [13920.578205] netif_receive_skb+0x94/0x240 [13920.579667] ? __netif_receive_skb+0xa0/0xa0 [13920.581132] ? eth_type_trans+0x134/0x270 [13920.582422] ? eth_gro_receive+0x310/0x310 [13920.583679] ? __build_skb_around+0x10e/0x130 [13920.585023] ? dma_unmap_page_attrs+0x1c6/0x2d0 [13920.586439] vmxnet3_rq_rx_complete+0xa76/0x17b0 [vmxnet3] [13920.588146] vmxnet3_poll_rx_only+0x47/0xd0 [vmxnet3] [13920.589693] __napi_poll+0x5d/0x1f0 [13920.590766] net_rx_action+0x21c/0x4a0 [13920.591918] ? napi_threaded_poll+0x1c0/0x1c0 [13920.593253] ? vmxnet3_msix_tx+0x100/0x100 [vmxnet3] [13920.594792] ? note_interrupt+0xf0/0x3a0 [13920.596042] ? add_interrupt_randomness+0x15f/0x2a0 [13920.597677] ? _raw_spin_lock+0x7a/0xd0 [13920.598853] ? _raw_write_lock_bh+0xe0/0xe0 [13920.600144] __do_softirq+0xf9/0x376 [13920.601247] irq_exit_rcu+0x118/0x130 [13920.602435] common_interrupt+0x77/0x90 [13920.603712] </IRQ> [13920.604411] asm_common_interrupt+0x1e/0x40 [13920.606075] RIP: 0010:acpi_idle_do_entry+0x61/0x70 [13920.607750] Code: ef 01 00 be 08 00 00 00 48 89 df e8 89 10 54 ff 48 89 df e8 41 06 54 ff 48 8b 03 a8 08 75 0c eb 07 0f 00 2d 01 b1 73 00 fb f4 <fa> 5b c3 48 89 df 5b e9 93 f9 ff ff cc cc cc 0f 1f 44 00 00 41 57 [13920.613840] RSP: 0018:ffffffffaf407d98 EFLAGS: 00000246 [13920.615463] RAX: 0000000000004000 RBX: ffffffffaf41a400 RCX: ffffffffae76014f [13920.617651] RDX: dffffc0000000000 RSI: 0000000000000008 RDI: ffffffffaf41a400 [13920.619820] RBP: 0000000000000001 R08: fffffbfff5e83481 R09: fffffbfff5e83481 [13920.622005] R10: ffffffffaf41a407 R11: fffffbfff5e83480 R12: ffff88800953a000 [13920.624162] R13: 0000000000000001 R14: ffff88800953a004 R15: ffff8880046c1800 [13920.626335] ? acpi_idle_do_entry+0x4f/0x70 [13920.627740] ? acpi_idle_do_entry+0x4f/0x70 [13920.629020] acpi_idle_enter+0x14d/0x1c0 [13920.630295] cpuidle_enter_state+0xb2/0x590 [13920.631603] ? tick_nohz_stop_tick+0x1f0/0x2d0 [13920.632987] cpuidle_enter+0x3c/0x60 [13920.634136] do_idle+0x399/0x400 [13920.635192] ? arch_cpu_idle_exit+0x40/0x40 [13920.636471] ? do_idle+0x26d/0x400 [13920.637517] cpu_startup_entry+0x19/0x20 [13920.638716] start_kernel+0x378/0x396 [13920.639925] secondary_startup_64_no_verify+0xc2/0xcb [13920.641564] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm
rdma_rxeip6_udp_tunneludp_tunnelib_uverbsib_coreuinputnls_utf8isofs rfcommxt_conntrackipt_REJECTnf_reject_ipv4nft_compatnft_counternft_chain_natnf_natnf_conntracknf_defrag_ipv6nf_defrag_ipv4nf_tablesnfnetlink tun bridge stp llc bnepvsock_loopbackvmw_vsock_virtio_transport_commonvmw_vsock_vmci_transportvsockintel_rapl_msrsnd_seq_midisnd_seq_midi_eventintel_rapl_commoncrct10dif_pclmulcrc32_pclmulghash_clmulni_intelvmw_balloonrapl pcspkr joydevsnd_ens1371snd_ac97_codecac97_busbtusbsnd_sequvcvideo btrtl btbcm btintelvideobuf2_vmallocvideobuf2_memopsvideobuf2_v4l2videobuf2_commonbluetoothsnd_pcmvideodev mcsnd_timerrfkillsnd_rawmidisnd_seq_deviceecdh_genericsnd ecc soundcorevmw_vmcii2c_piix4auth_rpcgsssunrpcip_tablesxfs libcrc32csr_modcdrom sgata_genericcrc32c_intelnvme vmwgfxata_piixserio_rawnvme_corettm ahci libahcidrm_kms_helperlibata syscopyarea[13920.641737] sysfillrect sysimgblt fb_sys_fops vmxnet3 t10_pi drm dm_mirror dm_region_hash dm_log dm_mod fuse [13920.671747] CR2: ffffc90103782194 [13920.674506] ---[ end trace 6ae70b2fba32e277 ]---Quoted 81 lines · Robert B
OK. For tomorrow. I need to know more about your setup. Which versions of kernel, rdma-core and what application SW you are running so I can try to reproduce your results.
Regards,
Bob Pearson
-----Original Message-----
From: Olga Kornievskaia <[email protected]>
Sent: Wednesday, July 21, 2021 7:31 PM
To: Bob Pearson <[email protected]>
Cc: Jason Gunthorpe <[email protected]>; Zhu Yanjun <[email protected]>; linux-rdma <[email protected]>
Subject: Re: [PATCH for-next] RDMA/rxe: Fix bug inrxe_net.cQuoted 28 lines · Bob Pearson
An earlier patch removed setting of
tot_lenin IPV4 headers because it was also set inip_local_out. However, this change resulted in an incorrect ICRC being computed because thetot_lenfield is not masked out. This patch restores that line. This fixes the bug reported by Zhu Yanjun. This bug would have also affected anyone using rxe.Fixes 230bb836ee88 ("RDMA/rxe: Fix redundant call to ip_send_check")
Reported_by: Zhu Yanjun <[email protected]>Signed-off-by Bob Pearson <[email protected]>--- drivers/infiniband/sw/rxe/rxe_net.c | 1 + 1 file changed, 1 insertion(+)drivers/infiniband/sw/rxe/rxe_net.c +1
-0@@ -259,6 +259,7 @@ static void prepare_ipv4_hdr(struct dst_entry *dst, struct sk_buff *skb, iph->version =IPVERSION ; iph->ihl =sizeof (struct iphdr ) >>2 ; + iph->tot_len =htons (skb->len); iph->frag_off = df; iph->protocol = proto;iph->tos = tos;Quoted 1 line
--
This patch made the server crash (just like one of the other crashes I've seen and posted to the list).
The client logs:
[ 206.437839] rdma_rxe: bad ICRC from 192.168.1.92 [ 211.043978] rdma_rxe: bad ICRC from 192.168.1.92 [ 215.652973] rdma_rxe: bad ICRC from 192.168.1.92Server crash:
[11568.440098] BUG: unable to handle page fault for address: ffffaddb21f61180 [11568.442923] #PF: supervisor write access in kernel mode [11568.444452] #PF: error_code(0x0002) - not-present page [11568.445996] PGD 1000067 P4D 1000067 PUD 11b9067 PMD 0 [11568.447527] Oops: 0002 [#1] SMP PTI [11568.448606] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 5.14.0-rc1+ #42 [11568.450911] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 [11568.454072] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe] [11568.455613] Code: 8b b3 48 01 00 00 4d 8b 48 08 41 8b 48 28 49 8d b9 80 01 00 00 85 f6 0f 84 78 01 00 00 41 8b 50 34 d3 e2 48 01 fa 48 8b 4d 00 <48> 89 0a 48 8b 4d 08 48 89 4a 08 48 8b 4d 10 48 89 4a 10 48 8b 4d [11568.461093] RSP: 0018:ffffaddb004c0988 EFLAGS: 00010082 [11568.462621] RAX: 0000000000000246 RBX: ffff9c9137df1a00 RCX: 0000000000000000 [11568.464695] RDX: ffffaddb21f61180 RSI: 0000000000000001 RDI: ffffaddb05f5f180 [11568.466779] RBP: ffffaddb004c0a30 R08: ffff9c9123186c00 R09: ffffaddb05f5f000 [11568.468902] R10: 80139a1c70550000 R11: 400000005d050000 R12: 0000000000000000 [11568.470977] R13: ffff9c9137df1b40 R14: ffff9c9137d50008 R15: 000000000000000a [11568.473050] FS: 0000000000000000(0000) GS:ffff9c917be40000(0000) knlGS:0000000000000000 [11568.475395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.477090] CR2: ffffaddb21f61180 CR3: 0000000043476005 CR4: 00000000001706e0 [11568.479170] Call Trace: [11568.479966] <IRQ> [11568.480683] rxe_responder+0x612/0x2470 [rdma_rxe] [11568.482122] rxe_do_task+0x89/0x100 [rdma_rxe] [11568.483427] rxe_rcv+0x2eb/0x900 [rdma_rxe] [11568.484655] ? __udp4_lib_lookup+0x2c8/0x440 [11568.486159] rxe_udp_encap_recv+0x68/0xa0 [rdma_rxe] [11568.487721] ? rxe_enable_task+0x10/0x10 [rdma_rxe] [11568.489223] udp_queue_rcv_one_skb+0x1df/0x4e0 [11568.490528] udp_unicast_rcv_skb.isra.67+0x74/0x90 [11568.491926] __udp4_lib_rcv+0x555/0xb90 [11568.493053] ip_protocol_deliver_rcu+0xe8/0x1b0 [11568.494479] ip_local_deliver_finish+0x44/0x50 [11568.496204] ip_local_deliver+0xf1/0x100 [11568.497621] ? ip_protocol_deliver_rcu+0x1b0/0x1b0 [11568.499147] ip_rcv+0xcb/0xe0 [11568.500032] __netif_receive_skb_core+0x3a2/0x1010 [11568.501491] ? packet_rcv+0x40/0x4b0 [11568.502661] ? select_idle_sibling+0x29/0x970 [11568.504019] __netif_receive_skb_one_core+0x3c/0xa0 [11568.505455] netif_receive_skb+0x3d/0x130 [11568.506650] vmxnet3_rq_rx_complete+0x5f0/0xdc0 [vmxnet3] [11568.508808] vmxnet3_poll_rx_only+0x31/0xa0 [vmxnet3] [11568.510526] __napi_poll+0x2b/0x120 [11568.511596] net_rx_action+0xe2/0x240 [11568.512678] ? vmxnet3_msix_rx+0x4a/0x60 [vmxnet3] [11568.514084] __do_softirq+0xd9/0x2a1 [11568.515218] irq_exit_rcu+0xba/0xd0 [11568.516272] common_interrupt+0x77/0x90 [11568.517438] </IRQ> [11568.518059] asm_common_interrupt+0x1e/0x40 [11568.519291] RIP: 0010:acpi_idle_do_entry+0x4c/0x50 [11568.520680] Code: 08 48 8b 15 3a e3 94 01 ed c3 e9 5f fc ff ff 65 48 8b 04 25 00 6f 01 00 48 8b 00 a8 08 75 ea eb 07 0f 00 2d 40 41 50 00 fb f4 <fa> c3 cc cc 0f 1f 44 00 00 41 55 41 89 d5 41 54 49 89 f4 55 53 48 [11568.526026] RSP: 0018:ffffaddb0009be68 EFLAGS: 00000246 [11568.527569] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff9c917be40000 [11568.529627] RDX: 0000000000000001 RSI: ffffffff9dcc99c0 RDI: ffff9c917c03b464 [11568.531723] RBP: ffff9c9105f63400 R08: ffff9c917c03b400 R09: 000000000000b0e0 [11568.533772] R10: 0000000000001e99 R11: ffff9c917be6a984 R12: ffffffff9dcc9a40 [11568.535918] R13: ffffffff9dcc99c0 R14: 0000000000000001 R15: 0000000000000000 [11568.538558] ? sched_clock_cpu+0x9/0xa0 [11568.539706] acpi_idle_enter+0x4d/0xb0 [11568.540912] cpuidle_enter_state+0x8c/0x350 [11568.542164] cpuidle_enter+0x29/0x40 [11568.543211] do_idle+0x257/0x2a0 [11568.544303] cpu_startup_entry+0x19/0x20 [11568.545455] start_secondary+0x116/0x150 [11568.546928] secondary_startup_64_no_verify+0xc2/0xcb [11568.548479] Modules linked in: rpcrdma rdma_ucm rdma_cm iw_cm ib_cm rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core fuse rfcomm xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT
nf_reject_ipv4nft_counternft_compatnf_tablesnfnetlink tun bridge stp llcvmw_vsock_vmci_transportvsock bnepsnd_seq_midisnd_seq_midi_eventintel_rapl_msrintel_rapl_commoncrct10dif_pclmulcrc32_pclmulvmw_balloonghash_clmulni_inteljoydev pcspkr btusb btrtl btbcm btintel bluetooth uvcvideo rfkillvideobuf2_vmallocvideobuf2_memopsvideobuf2_v4l2snd_ens1371snd_ac97_codecac97_bussnd_seqvideobuf2_commonsnd_pcmvideodev mcecdh_genericeccsnd_timersnd_rawmidisnd_seq_devicesnd soundcorevmw_vmcii2c_piix4auth_rpcgsssunrpcip_tablesxfs libcrc32csr_modcdrom sgcrc32c_intelata_genericvmwgfx ttmserio_rawnvmedrm_kms_helpersyscopyarea sysfillrect sysimgbltfb_sys_fopsnvme_coret10_picecata_piixahci vmxnet3 libahci drm libata [11568.575542] CR2:ffffaddb21f61180[11568.577210] ---[ end trace8afcc89bb91d9b85]--- [11568.578573] RIP: 0010:rxe_cq_post+0x98/0x210 [rdma_rxe]Quoted 2 lines
2.30.2
-----Original Message-----
From: Zhu Yanjun <[email protected]>
Sent: Monday, July 26, 2021 2:43 AM
To: Olga Kornievskaia <[email protected]>
Cc: Pearson, Robert B <[email protected]>; Bob Pearson <[email protected]>; Jason Gunthorpe <[email protected]>; linux-rdma <[email protected]>
Subject: Re: [PATCH for-next] RDMA/rxe: Fix bug in rxe_net.c
Quoted 3 lines · Olga Kornievskaia
I'm RHEL based in terms of my userland software. I work on upstream kernels using that.
Simply, one host with the kernel 5.14-rc1, the other host with the kernel 5.14-rc2. Then the following errors will appear
"
...
[13873.255148] rdma_rxe: bad ICRC from 192.168.1.92 [13877.567475] rdma_rxe: bad ICRC from 192.168.1.92 [13882.175544] rdma_rxe: bad ICRC from 192.168.1.92 ...
"
Correct?
Zhu Yanjun
Zhu,
I was able to reproduce the error you reported a few days ago which were ICRC errors. The above referenced patch fixed those errors. Working with Olga, it also seemed to clear up the ICRC errors she was seeing but not the other problems.
Bob