Crash in indirect block infra after unloading driver module
Subsystems (2)
Hi Pablo,
I've encountered a new issue with indirect offloads infrastructure. The issue is that on driver offload its indirect callbacks are not removed from blocks and any following offloads operations on block that has such callback in its offloads cb list causes call to unmapped address.
Steps to reproduce:
echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind devlink dev eswitch set pci/0000:81:00.0 mode switchdev
ip link add vxlan1 type vxlan dstport 4789 external
ip addr add 192.168.1.1 dev ens1f0
link set up dev ens1f0
ip link set up dev ens1f0
tc qdisc add dev vxlan1 ingress
tc filter add dev vxlan1 protocol ip ingress flower enc_src_ip 192.168.1.2 enc_dst_ip 192.168.1.1 enc_key_id 42 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev ens1f0_0 tc -s filter show dev vxlan1 ingress
rmmod mlx5_ib
rmmod mlx5_core
tc -s filter show dev vxlan1 ingress
Resulting dmesg:
[ 153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0 [ 153.747975] #PF: supervisor instruction fetch in kernel mode [ 153.748071] #PF: error_code(0x0010) - not-present page [ 153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0 [ 153.748328] Oops: 0010 [#1] SMP KASAN PTI [ 153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170 [ 153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017 [ 153.748638] RIP: 0010:0xffffffffc114cee0 [ 153.748709] Code: Bad RIP value. [ 153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246 [ 153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba [ 153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002 [ 153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c [ 153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0 [ 153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00 [ 153.771689] FS: 00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000 [ 153.777983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0 [ 153.790567] Call Trace: [ 153.796844] ? tc_setup_cb_call+0xd8/0x170 [ 153.803164] ? fl_hw_update_stats+0x117/0x280 [cls_flower] [ 153.809516] ? 0xffffffffc1328000 [ 153.815766] ? _find_next_bit.constprop.0+0x3e/0xf0 [ 153.822079] ? __nla_reserve+0x4c/0x60 [ 153.828283] ? memcpy+0x39/0x60 [ 153.834311] ? fl_dump+0x335/0x350 [cls_flower] [ 153.840389] ? fl_tmplt_dump+0x110/0x110 [cls_flower] [ 153.846483] ? memcpy+0x39/0x60 [ 153.852516] ? tcf_fill_node+0x2ea/0x420 [ 153.858564] ? tcf_chain_tp_delete_empty+0x170/0x170 [ 153.864558] ? tcf_node_dump+0xf9/0x110 [ 153.870481] ? fl_walk+0xf6/0x240 [cls_flower] [ 153.876357] ? fl_put+0x10/0x10 [cls_flower] [ 153.882173] ? __mutex_lock_slowpath+0x10/0x10 [ 153.887922] ? tcf_chain_dump+0x237/0x450 [ 153.893595] ? tcf_block_release+0x50/0x50 [ 153.899141] ? tfilter_notify+0x160/0x160 [ 153.904630] ? tc_dump_tfilter+0x388/0x4a0 [ 153.910024] ? tcf_chain_dump+0x450/0x450 [ 153.915390] ? __mutex_lock_slowpath+0x10/0x10 [ 153.920657] ? netlink_dump+0x2ea/0x670 [ 153.925851] ? __netlink_sendskb+0x70/0x70 [ 153.931028] ? __mutex_lock_slowpath+0x10/0x10 [ 153.936117] ? __alloc_skb+0xc3/0x310 [ 153.941199] ? __netlink_dump_start+0x2f6/0x3c0 [ 153.946247] ? rtnetlink_rcv_msg+0x383/0x510 [ 153.951222] ? tcf_chain_dump+0x450/0x450 [ 153.956124] ? rtnl_calcit.isra.0+0x1c0/0x1c0 [ 153.961010] ? kmem_cache_free+0x84/0x2a0 [ 153.965787] ? skb_free_datagram+0x12/0x60 [ 153.970576] ? netlink_recvmsg+0x281/0x690 [ 153.975256] ? tcf_chain_dump+0x450/0x450 [ 153.979861] ? netlink_compare+0x53/0x70 [ 153.984447] ? netlink_rcv_skb+0xd0/0x200 [ 153.988872] ? rtnl_calcit.isra.0+0x1c0/0x1c0 [ 153.993233] ? netlink_ack+0x460/0x460 [ 153.997573] ? __kasan_kmalloc.constprop.0+0xc2/0xd0 [ 154.001936] ? netlink_deliver_tap+0x48/0x390 [ 154.006298] ? netlink_unicast+0x2d8/0x3d0 [ 154.010553] ? netlink_attachskb+0x3d0/0x3d0 [ 154.014747] ? __virt_addr_valid+0xbb/0x130 [ 154.018886] ? netlink_sendmsg+0x3af/0x690 [ 154.022997] ? netlink_unicast+0x3d0/0x3d0 [ 154.026993] ? import_iovec+0x135/0x1e0 [ 154.030898] ? netlink_unicast+0x3d0/0x3d0 [ 154.034776] ? sock_sendmsg+0x96/0xa0 [ 154.038526] ? ____sys_sendmsg+0x388/0x420 [ 154.042282] ? kernel_sendmsg+0x30/0x30 [ 154.045980] ? __copy_msghdr_from_user+0x260/0x260 [ 154.049702] ? kernel_recvmsg+0x60/0x60 [ 154.053397] ? copy_msghdr_from_user+0xa7/0x100 [ 154.057097] ? ___sys_sendmsg+0xe0/0x140 [ 154.060807] ? sendmsg_copy_msghdr+0x30/0x30 [ 154.064517] ? ___sys_recvmsg+0xdc/0x130 [ 154.068198] ? recvmsg_copy_msghdr+0x40/0x40 [ 154.071852] ? _raw_read_lock_irqsave+0x50/0x50 [ 154.075540] ? __count_memcg_events+0x33/0x100 [ 154.079222] ? handle_mm_fault+0x66d/0x1eb0 [ 154.082884] ? __fget_light+0x9e/0xf0 [ 154.086533] ? __sys_sendmsg+0xb3/0x130 [ 154.090164] ? __sys_sendmsg_sock+0x60/0x60 [ 154.093768] ? do_syscall_64+0x4d/0x90 [ 154.097357] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 154.101006] Modules linked in: act_mirred act_tunnel_key cls_flower sch_ingress vxlan ip6_udp_tunnel udp_tunnel nfsv3 nfs_acl nfs lockd grace fscache tun bridge stp llc sunrpc rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_core intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass igb crct10dif_pclmul crc32_pclmul mlxfw crc32c_intel ipmi_ssif ptp iTCO_wdt pps_core ses ioatdma ghash_clmulni_intel mei_me iTCO_vendor_suppo rt enclosure i2c_i801 ipmi_si intel_cstate joydev mei i2c_smbus lpc_ich pcspkr intel_uncore dca wmi ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad ast i2c_algo_bit drm_vram_helper drm_kms_helper drm_ttm_helper ttm drm mpt3sas raid_class scsi_transport_sas [last unloaded: mlx5_core] [ 154.132864] CR2: ffffffffc114cee0 [ 154.137879] ---[ end trace 1c03f81270107f93 ]--- [ 154.162631] RIP: 0010:0xffffffffc114cee0 [ 154.167703] Code: Bad RIP value. [ 154.172705] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246 [ 154.177817] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba [ 154.183006] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002 [ 154.188259] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c [ 154.193585] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0 [ 154.198928] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00 [ 154.204315] FS: 00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000 [ 154.209867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 154.215417] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4:
00000000001606e0
I can come up with something to fix mlx5 but it looks like all other drivers that support indirect devices are also susceptible to similar issue.
Regards,
Vlad
Quoted 21 lines · Vlad Buslov
Hi Pablo,
I've encountered a new issue with indirect offloads infrastructure. The issue is that on driver offload its indirect callbacks are not removed from blocks and any following offloads operations on block that has such callback in its offloads cb list causes call to unmapped address.
Steps to reproduce:
echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs echo 0000:81:00.2 >
/sys/bus/pci/drivers/mlx5_core/unbinddevlink dev eswitch set pci/0000:81:00.0 mode switchdevip link add vxlan1 type vxlan dstport 4789 external
ip addr add 192.168.1.1 dev ens1f0
link set up dev ens1f0
ip link set up dev ens1f0
tc qdisc add dev vxlan1 ingress
tc filter add dev vxlan1 protocol ip ingress flowerenc_src_ip192.168.1.2enc_dst_ip192.168.1.1enc_key_id42enc_dst_port4789 actiontunnel_keyunset action mirred egress redirect devens1f0_0tc -s filter show dev vxlan1 ingress
Quoted 3 lines
rmmod
mlx5_ib
rmmodmlx5_core
tc -s filter show dev vxlan1 ingress
On module removal, the representors are gone and the ->cleanup callback should be exercised, this callback removes the flow_block and removes the rules in the driver.
Can you check if the ->cleanup callback is exercised?
Quoted 26 lines
Resulting dmesg:
[ 153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0 [ 153.747975] #PF: supervisor instruction fetch in kernel mode [ 153.748071] #PF: error_code(0x0010) - not-present page [ 153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0 [ 153.748328] Oops: 0010 [#1] SMP KASAN PTI [ 153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170 [ 153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017 [ 153.748638] RIP: 0010:0xffffffffc114cee0 [ 153.748709] Code: Bad RIP value. [ 153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246 [ 153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba [ 153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002 [ 153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c [ 153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0 [ 153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00 [ 153.771689] FS: 00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000 [ 153.777983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0 [ 153.790567] Call Trace: [ 153.796844] ? tc_setup_cb_call+0xd8/0x170 [ 153.803164] ? fl_hw_update_stats+0x117/0x280 [cls_flower] [ 153.809516] ? 0xffffffffc1328000 [ 153.815766] ? _find_next_bit.constprop.0+0x3e/0xf0 [ 153.822079] ? __nla_reserve+0x4c/0x60
[...]
Quoted 4 lines
I can come up with something to fix mlx5 but it looks like all other drivers that support indirect devices are also susceptible to similar issue.
How does the fix you have in mind look like?
Thanks.
Quoted 26 lines · Pablo Neira Ayuso
Quoted 21 lines · Vlad Buslov
Hi Pablo,
I've encountered a new issue with indirect offloads infrastructure. The issue is that on driver offload its indirect callbacks are not removed from blocks and any following offloads operations on block that has such callback in its offloads cb list causes call to unmapped address.
Steps to reproduce:
echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs echo 0000:81:00.2 >
/sys/bus/pci/drivers/mlx5_core/unbinddevlink dev eswitch set pci/0000:81:00.0 mode switchdevip link add vxlan1 type vxlan dstport 4789 external
ip addr add 192.168.1.1 dev ens1f0
link set up dev ens1f0
ip link set up dev ens1f0
tc qdisc add dev vxlan1 ingress
tc filter add dev vxlan1 protocol ip ingress flowerenc_src_ip192.168.1.2enc_dst_ip192.168.1.1enc_key_id42enc_dst_port4789 actiontunnel_keyunset action mirred egress redirect devens1f0_0tc -s filter show dev vxlan1 ingressQuoted 3 lines
rmmod
mlx5_ib
rmmodmlx5_core
tc -s filter show dev vxlan1 ingress
Quoted 5 lines
On module removal, the representors are gone and the ->cleanup callback should be exercised, this callback removes the
flow_blockand removes the rules in the driver.Can you check if the ->cleanup callback is exercised?
I added some traces. On module unload mlx5e_cleanup_rep_tx() and flow_indr_dev_unregister() are called, but not tc_block_indr_cleanup(). Maybe this is the problem that wenxu fixed in one of his patches? I'll try to reproduce on net branch.
Quoted 27 lines
Quoted 19 lines
Resulting dmesg:
[ 153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0 [ 153.747975] #PF: supervisor instruction fetch in kernel mode [ 153.748071] #PF: error_code(0x0010) - not-present page [ 153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0 [ 153.748328] Oops: 0010 [#1] SMP KASAN PTI [ 153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170 [ 153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017 [ 153.748638] RIP: 0010:0xffffffffc114cee0 [ 153.748709] Code: Bad RIP value. [ 153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246 [ 153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba [ 153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002 [ 153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c [ 153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0 [ 153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00 [ 153.771689] FS: 00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000Quoted 8 lines
[ 153.777983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0 [ 153.790567] Call Trace: [ 153.796844] ? tc_setup_cb_call+0xd8/0x170 [ 153.803164] ? fl_hw_update_stats+0x117/0x280 [cls_flower] [ 153.809516] ? 0xffffffffc1328000 [ 153.815766] ? _find_next_bit.constprop.0+0x3e/0xf0 [ 153.822079] ? __nla_reserve+0x4c/0x60
Quoted 7 lines
[...]
Quoted 4 lines
I can come up with something to fix mlx5 but it looks like all other drivers that support indirect devices are also susceptible to similar issue.
How does the fix you have in mind look like?
To call flow_indr_dev_unregister() from right path. But you are right, it is already called, so we just need to determine why it doesn't perform the proper cleanup.
Quoted 2 lines
Thanks.
Quoted 31 lines · Vlad Buslov
Quoted 30 lines · Pablo Neira Ayuso
Quoted 21 lines · Vlad Buslov
Hi Pablo,
I've encountered a new issue with indirect offloads infrastructure. The issue is that on driver offload its indirect callbacks are not removed from blocks and any following offloads operations on block that has such callback in its offloads cb list causes call to unmapped address.
Steps to reproduce:
echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs echo 0000:81:00.2 >
/sys/bus/pci/drivers/mlx5_core/unbinddevlink dev eswitch set pci/0000:81:00.0 mode switchdevip link add vxlan1 type vxlan dstport 4789 external
ip addr add 192.168.1.1 dev ens1f0
link set up dev ens1f0
ip link set up dev ens1f0
tc qdisc add dev vxlan1 ingress
tc filter add dev vxlan1 protocol ip ingress flowerenc_src_ip192.168.1.2enc_dst_ip192.168.1.1enc_key_id42enc_dst_port4789 actiontunnel_keyunset action mirred egress redirect devens1f0_0tc -s filter show dev vxlan1 ingressQuoted 3 lines
rmmod
mlx5_ib
rmmodmlx5_core
tc -s filter show dev vxlan1 ingressOn module removal, the representors are gone and the ->cleanup callback should be exercised, this callback removes the
flow_blockand removes the rules in the driver.
Quoted 6 lines
Quoted 1 line
Can you check if the ->cleanup callback is exercised?
I added some traces. On module unload
mlx5e_cleanup_rep_tx()andflow_indr_dev_unregister()are called, but nottc_block_indr_cleanup(). Maybe this is the problem that wenxu fixed in one of his patches? I'll try to reproduce on net branch.
Indeed, on net branch tc_block_indr_cleanup() is called and crash is not reproduced. It seems to be fixed by a1db217861f3 ("net: flow_offload:
fix flow_indr_dev_unregister path").
Quoted 42 lines
Quoted 28 lines
Quoted 20 lines
Resulting dmesg:
[ 153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0 [ 153.747975] #PF: supervisor instruction fetch in kernel mode [ 153.748071] #PF: error_code(0x0010) - not-present page [ 153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0 [ 153.748328] Oops: 0010 [#1] SMP KASAN PTI [ 153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170 [ 153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017 [ 153.748638] RIP: 0010:0xffffffffc114cee0 [ 153.748709] Code: Bad RIP value. [ 153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246 [ 153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba [ 153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002 [ 153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c [ 153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0 [ 153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00 [ 153.771689] FS: 00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000Quoted 8 lines
[ 153.777983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0 [ 153.790567] Call Trace: [ 153.796844] ? tc_setup_cb_call+0xd8/0x170 [ 153.803164] ? fl_hw_update_stats+0x117/0x280 [cls_flower] [ 153.809516] ? 0xffffffffc1328000 [ 153.815766] ? _find_next_bit.constprop.0+0x3e/0xf0 [ 153.822079] ? __nla_reserve+0x4c/0x60Quoted 7 lines
[...]
Quoted 4 lines
I can come up with something to fix mlx5 but it looks like all other drivers that support indirect devices are also susceptible to similar issue.
How does the fix you have in mind look like?
To call
flow_indr_dev_unregister()from right path. But you are right, it is already called, so we just need to determine why it doesn't perform the proper cleanup.Quoted 2 lines
Thanks.