New: Running ib_rdma_bw over rxe loopback triggers a kernel oops

https://bugzilla.kernel.org/show_bug.cgi?id=176901

            Bug ID: 176901
           Summary: Running ib_rdma_bw over rxe loopback triggers a kernel
                    oops
           Product: Drivers
           Version: 2.5
    Kernel Version: v4.8.0
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Infiniband/RDMA
          Assignee: drivers_infiniband-rdma-ztI5WcYan/[email protected]
          Reporter: [email protected]
        Regression: No

After I ran the following commands in a virtual machine:

modprobe rdma_rxe
rxe_cfg add eth0
modprobe ib_uverbs
ib_send_bw &
ib_send_bw 192.168.123.238

the following output appeared on the system console:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff81023060>] check_addr+0x30/0x60
PGD 39751067 PUD 30689067 PMD 0 
Oops: 0000 [#1] SMP
Modules linked in: ib_uverbs ib_srp(O) scsi_transport_srp(O) scst_local(O)

iscsi_scst(O) ib_srpt(O) scst_vdisk(O) scst(O) libcrc32c dlm configfs rdma_rxe ip6_udp_tunnel udp_tunnel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd virtio_console virtio_balloon serio_raw virtio_rng i2c_piix4 acpi_cpufreq button iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ext4 jbd2 mbcache virtio_net virtio_blk psmouse drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm virtio_pci drm floppy [last unloaded: scsi_transport_srp]

CPU: 3 PID: 2941 Comm: ib_send_bw Tainted: G           O    4.8.0-debug+ #2
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
task: ffff8800388fa7c0 task.stack: ffff88002cdd8000
RIP: 0010:[<ffffffff81023060>]  [<ffffffff81023060>] check_addr+0x30/0x60
RSP: 0018:ffff88002cddbb78  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000087654321 RCX: 0000000000001000
RDX: 00000000042a1000 RSI: ffff88003294cac8 RDI: ffffffff819b7189
RBP: ffff88002cddbba8 R08: 0000000000000000 R09: ffff88003294cac8
R10: 0000000000000001 R11: 0000000000000020 R12: 0000000000000000
R13: 0000000000000020 R14: ffff88003294cac8 R15: ffff8800393e6fc8
FS:  00007fea71eaf740(0000) GS:ffff88003fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000003bc65000 CR4: 00000000001406e0
Stack:
 ffffffff81023188 ffff880039a34050 ffff88003941f188 ffff880033b2f000
 ffffffff81c222a0 0000000000000000 ffff88002cddbc20 ffffffff814f5fca
 ffff88003294cac8 ffff8800393e6fc8 ffff880000000020 0000000000000020
Call Trace:
 [<ffffffff81023188>] ? nommu_map_sg+0x88/0xf0
 [<ffffffff814f5fca>] ib_umem_get+0x4ea/0x520
 [<ffffffffa02418e1>] rxe_mem_init_user+0x41/0x2d0 [rdma_rxe]
 [<ffffffff810acabd>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffffa023eb2c>] rxe_reg_user_mr+0x9c/0x150 [rdma_rxe]
 [<ffffffffa03463b6>] ib_uverbs_reg_mr+0x146/0x330 [ib_uverbs]
 [<ffffffffa03414fd>] ib_uverbs_write+0x1fd/0x460 [ib_uverbs]
 [<ffffffffa03413b0>] ? ib_uverbs_write+0xb0/0x460 [ib_uverbs]
 [<ffffffff81647f42>] ? _raw_spin_unlock+0x22/0x30
 [<ffffffff8115630a>] ? handle_mm_fault+0x61a/0x12e0
 [<ffffffff811ab823>] __vfs_write+0x23/0x120
 [<ffffffff811ac6f9>] ? rw_verify_area+0x49/0xb0
 [<ffffffff811ac943>] vfs_write+0xb3/0x1b0
 [<ffffffff810aca28>] ? trace_hardirqs_on_caller+0x128/0x1b0
 [<ffffffff811adc54>] SyS_write+0x44/0xa0
 [<ffffffff816486c0>] entry_SYSCALL_64_fastpath+0x23/0xc1
Code: 48 8b 86 38 03 00 00 48 85 c0 74 1f 4c 8b 00 48 8d 74 0a ff b8 01 00 00

00 4c 39 c6 76 33 b8 fe ff ff ff 49 39 c0 77 13 31 c0 c3 <4c> 8b 04 25 00 00 00 00 eb e9 b8 01 00 00 00 c3 55 48 89 fe 48
RIP [<ffffffff81023060>] check_addr+0x30/0x60
RSP <ffff88002cddbb78>
CR2: 0000000000000000

You are receiving this mail because:
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

This bug is fixed with my patch [1] and Doug ack also about merging fix.

[1] https://patchwork.kernel.org/patch/9354863/

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hello Parav,

Thank you for the feedback. Do you perhaps know which branch Doug will send to Linus? I haven't found any post-v4.8 rxe changes in Doug's for-4.9-good branch:

$ git log --format=oneline v4.8..dledford/k.o/for-4.9-good drivers/infiniband/sw include/uapi/rdma/*rxe*
60747ef4d173c2747bf7f0377fb22846cb422195 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net

Something else I was wondering about is whether or not the rdma_rxe driver should allow being attached to a bridge interface? Apparently the rdma_rxe driver allows attaching to both the non-bridged and the bridged interfaces, resulting in duplicate GIDs:

# modprobe rdma_rxe
# brctl addbr virbr0
# brctl addif virbr0 eth1
# rxe_cfg add eth1
# rxe_cfg add virbr0
# rxe_cfg
  Name    Link  Driver      Speed  NMTU  IPv4_addr      RDEV  RMTU
  eth0    yes   virtio_net         1500  192.168.1.123
  eth1    no    virtio_net         1500                 rxe0
  virbr0  no    bridge             1500                 rxe1
# (cd /sys/class/infiniband && grep -aH '' */ports/*/gids/0)
rxe0/ports/1/gids/0:fe80:0000:0000:0000:5054:00ff:fefd:2e4e
rxe1/ports/1/gids/0:fe80:0000:0000:0000:5054:00ff:fefd:2e4e

Should it ever be allowed that two different ports have the same GID?

Thanks,

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I had to rebase my k.o/for-4.9 branch. So, k.o/for-4.9 became k.o/for-4.9-bad (which has Parav's patches in it), and I made a new k.o/for-4.9-good that does not have Parav's patches in it. I then had to add Parav's patches to my k.o/for-4.9-hfi1 branch (I know, that makes no sense, but bear with me). The issue is that the k.o/for-4.9-hfi1 branch needed a very late 4.7-rc base (4.7-rc7 I think) in order for the hfi1 patches to apply cleanly. I put Parav's patches into my for-4.9 branch originally, but when I did a merge test with Linus tree, I found that one of them in particular conflicts with patches that were also late in the 4.7-rc cycle. By putting Parav's patches into the hfi1 branch, I can resolve that conflict myself and will be able to send a clean pull request of just the rxe patches to Linus. So, long story short, pull my k.o/for-4.9-hfi1 branch and test against it.

Doug Ledford <[email protected]>
    GPG Key ID: 0E572FDD

Hi Bart,

My take is it should not to avoid mis configurations.
Below fix is required, which I tested to not allow on bridge type devices. I will send formal patch.

b/drivers/infiniband/sw/rxe/rxe_sysfs.c
index cf8e778..7f4e0f3 100644

@@ -90,7 +90,11 @@ static int rxe_param_set_add(const char *val, const

struct kernel_param *kp)

                err = -EINVAL;
                goto err;
        }
-
+       if (netif_is_bridge_master(ndev)) {
+               pr_err("bridge device %s is unsupported\n", intf);
+               err = -EINVAL;
+               goto err;
+       }
        if (net_to_rxe(ndev)) {
                pr_err("rxe: already configured on %s\n", intf);
                err = -EINVAL;

Not sure. It might be allowed LAG mode (at least IP based GID). I am not upto date of recent LAG updates from Leon.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html