rmmod mlx4_core panic 3.16-rc1

DISCUSSION Shirley Ma 4 participants 13 messages Jun 19, 2014 – Jun 22, 2014 8 messages not in the archive

Subsystems (3)

Network drivers InfiniBand Mellanox MLX4 core VPI driver

lore

Share & tools

lore

Shirley Ma

author

2014-06-19 03:33 UTC #0 lore raw

Hello Or,

Two questions here:
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?

I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it failed with
mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet

2. After mlx4_core module is being loaded with with num_vfs={} parameters, when removing mlx4_core, it consistently hits below panic. Whether this problem is being tracked?

Thanks
Shirley

<mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014)
mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
mlx4_core: Initializing 0000:40:00.0
mlx4_core 0000:40:00.0: Enabling SR-IOV with 2 VFs
pci 0000:40:00.1: [15b3:1002] type 00 class 0x0c0600
mlx4_core: Initializing 0000:40:00.1
mlx4_core 0000:40:00.1: enabling device (0000 -> 0002)
mlx4_core 0000:40:00.1: Skipping virtual function:1
pci 0000:40:00.2: [15b3:1002] type 00 class 0x0c0600
mlx4_core: Initializing 0000:40:00.2
mlx4_core 0000:40:00.2: enabling device (0000 -> 0002)
mlx4_core 0000:40:00.2: Skipping virtual function:2
mlx4_core 0000:40:00.0: Running in master mode
mlx4_core 0000:40:00.0: PCIe BW is different than device's capability
mlx4_core 0000:40:00.0: PCIe link speed is 5.0GT/s, device supports 8.0GT/s
mlx4_core 0000:40:00.0: PCIe link width is x8, device supports x8
mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet

BUG: unable to handle kernel NULL pointer dereference at 000000000000038c
IP: [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
PGD 45d3ba067 PUD 45ace8067 PMD 0 
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas scsi_transport_sas raid_class [last unloaded: mlx4_core]
CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
Hardware name: Oracle Corporation SUN FIRE X4170 M3     /ASSY,MOTHERBOARD,1U   , BIOS 17050100 08/29/2013
task: ffff880461540110 ti: ffff880465000000 task.ti: ffff880465000000
RIP: 0010:[<ffffffffa0350450>]  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
RSP: 0018:ffff880465003d88  EFLAGS: 00010296
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000026 RSI: 0000000000000292 RDI: ffff880468b8f000
RBP: ffff880465003db8 R08: 0000000000000000 R09: 0000000000000000
R10: 09f911029d74e35b R11: 09f911029d74e35b R12: 0000000000000000
R13: ffff880468b8f000 R14: ffffffffa036de40 R15: 0000000000000001
FS:  00007ff287fc2700(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000038c CR3: 000000045cfae000 CR4: 00000000000407e0
Stack:
 ffff880465003da8 ffff880468b8f000 0000000000000000 ffff880468b8f000
 ffffffffa036de40 0000000000000001 ffff880465003dd8 ffffffffa0350805
 ffff880468b8f098 ffffffffa036dd60 ffff880465003e08 ffffffff812ebaa6
Call Trace:
 [<ffffffffa0350805>] mlx4_remove_one+0x25/0x50 [mlx4_core]
 [<ffffffff812ebaa6>] pci_device_remove+0x46/0xc0
 [<ffffffff813ce08f>] __device_release_driver+0x7f/0xf0
 [<ffffffff813ce1c8>] driver_detach+0xc8/0xd0
 [<ffffffff813cced9>] bus_remove_driver+0x59/0xd0
 [<ffffffff813cef80>] driver_unregister+0x30/0x70
 [<ffffffff812ebc13>] pci_unregister_driver+0x23/0x80
 [<ffffffffa03650e4>] mlx4_cleanup+0x10/0x1e [mlx4_core]
 [<ffffffff810ceff9>] SyS_delete_module+0x189/0x210
 [<ffffffff815d2f12>] system_call_fastpath+0x16/0x1b
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 9f 58 01 00 00 49 89 fd <44> 8b b3 8c 03 00 00 45 85 f6 0f 85 41 02 00 00 f6 43 08 04 44 
RIP  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
 RSP <ffff880465003d88>

CR2: 000000000000038c
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Or Gerlitz

2014-06-20 03:34 UTC #2 lore raw

In reply to a message not in the archive

Quoted 4 lines · Shirley Ma

1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it failed with mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet

What do you mean by "port1" and "port2" -- can you give the exact command line you used?

Single ported VFs are currently supported for Ethernet only configuration, that is not for only IB nor for VPI, that is only if you use port_type_arrary=2,2

Quoted 3 lines

2. After mlx4_core module is being loaded with with num_vfs={} parameters, when removing mlx4_core, it consistently hits below panic. Whether this problem is being tracked?

what do you mean by "num_vfs={}", is it num_vfs=N or {N}, also here, please send the exact setting you used. The crash you indicated below is supposed to be fixed by the upstream commit da1de8dfff09d33d4a5345762c21b487028e25f5 "net/mlx4_core: Keep only one driver entry release" - are you sure to have this commit in the tree you are working with?

Or.

Quoted 54 lines

BUG: unable to handle kernel NULL pointer dereference at 000000000000038c
IP: [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
PGD 45d3ba067 PUD 45ace8067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas scsi_transport_sas raid_class [last unloaded: mlx4_core]
CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
Hardware name: Oracle Corporation SUN FIRE X4170 M3     /ASSY,MOTHERBOARD,1U   , BIOS 17050100 08/29/2013
task: ffff880461540110 ti: ffff880465000000 task.ti: ffff880465000000
RIP: 0010:[<ffffffffa0350450>]  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
RSP: 0018:ffff880465003d88  EFLAGS: 00010296
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000026 RSI: 0000000000000292 RDI: ffff880468b8f000
RBP: ffff880465003db8 R08: 0000000000000000 R09: 0000000000000000
R10: 09f911029d74e35b R11: 09f911029d74e35b R12: 0000000000000000
R13: ffff880468b8f000 R14: ffffffffa036de40 R15: 0000000000000001
FS:  00007ff287fc2700(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000038c CR3: 000000045cfae000 CR4: 00000000000407e0
Stack:
 ffff880465003da8 ffff880468b8f000 0000000000000000 ffff880468b8f000
 ffffffffa036de40 0000000000000001 ffff880465003dd8 ffffffffa0350805
 ffff880468b8f098 ffffffffa036dd60 ffff880465003e08 ffffffff812ebaa6
Call Trace:
 [<ffffffffa0350805>] mlx4_remove_one+0x25/0x50 [mlx4_core]
 [<ffffffff812ebaa6>] pci_device_remove+0x46/0xc0
 [<ffffffff813ce08f>] __device_release_driver+0x7f/0xf0
 [<ffffffff813ce1c8>] driver_detach+0xc8/0xd0
 [<ffffffff813cced9>] bus_remove_driver+0x59/0xd0
 [<ffffffff813cef80>] driver_unregister+0x30/0x70
 [<ffffffff812ebc13>] pci_unregister_driver+0x23/0x80
 [<ffffffffa03650e4>] mlx4_cleanup+0x10/0x1e [mlx4_core]
 [<ffffffff810ceff9>] SyS_delete_module+0x189/0x210
 [<ffffffff815d2f12>] system_call_fastpath+0x16/0x1b
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 9f 58 01 00 00 49 89 fd <44> 8b b3 8c 03 00 00 45 85 f6 0f 85 41 02 00 00 f6 43 08 04 44
RIP  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
 RSP <ffff880465003d88>

Quoted 5 lines

CR2: 000000000000038c
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Wei Yang

2014-06-20 03:51 UTC #4 lore raw

In reply to a message not in the archive

Quoted 21 lines · Or Gerlitz

Quoted 4 lines · Shirley Ma
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it failed with mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet
What do you mean by "port1" and "port2" -- can you give the exact command line you used?
Single ported VFs are currently supported for Ethernet only configuration, that is not for only IB nor for VPI, that is only if you use port_type_arrary=2,2
Quoted 3 lines
2. After mlx4_core module is being loaded with with num_vfs={} parameters, when removing mlx4_core, it consistently hits below panic. Whether this problem is being tracked?

Quoted 7 lines

what do you mean by "num_vfs={}", is it num_vfs=N or {N}, also here, please send the exact setting you used. The crash you indicated below is supposed to be fixed by the upstream commit da1de8dfff09d33d4a5345762c21b487028e25f5 "net/mlx4_core: Keep only one driver entry release" - are you sure to have this commit in the tree you are working with?

Just checked, this patch is in 3.16-rc1.

Quoted 23 lines

Or.
Quoted 18 lines
<mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014)
mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
mlx4_core: Initializing 0000:40:00.0
mlx4_core 0000:40:00.0: Enabling SR-IOV with 2 VFs
pci 0000:40:00.1: [15b3:1002] type 00 class 0x0c0600
mlx4_core: Initializing 0000:40:00.1
mlx4_core 0000:40:00.1: enabling device (0000 -> 0002)
mlx4_core 0000:40:00.1: Skipping virtual function:1
pci 0000:40:00.2: [15b3:1002] type 00 class 0x0c0600
mlx4_core: Initializing 0000:40:00.2
mlx4_core 0000:40:00.2: enabling device (0000 -> 0002)
mlx4_core 0000:40:00.2: Skipping virtual function:2
mlx4_core 0000:40:00.0: Running in master mode
mlx4_core 0000:40:00.0: PCIe BW is different than device's capability
mlx4_core 0000:40:00.0: PCIe link speed is 5.0GT/s, device supports 8.0GT/s
mlx4_core 0000:40:00.0: PCIe link width is x8, device supports x8
mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet
Quoted 2 lines
BUG: unable to handle kernel NULL pointer dereference at 000000000000038c
IP: [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]

Quoted 1 line

From this log, it happens during probe?

If not, any action after probe?

Quoted 39 lines

PGD 45d3ba067 PUD 45ace8067 PMD 0

Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas scsi_transport_sas raid_class [last unloaded: mlx4_core]
CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
Hardware name: Oracle Corporation SUN FIRE X4170 M3     /ASSY,MOTHERBOARD,1U   , BIOS 17050100 08/29/2013
task: ffff880461540110 ti: ffff880465000000 task.ti: ffff880465000000
RIP: 0010:[<ffffffffa0350450>]  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
RSP: 0018:ffff880465003d88  EFLAGS: 00010296
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000026 RSI: 0000000000000292 RDI: ffff880468b8f000
RBP: ffff880465003db8 R08: 0000000000000000 R09: 0000000000000000
R10: 09f911029d74e35b R11: 09f911029d74e35b R12: 0000000000000000
R13: ffff880468b8f000 R14: ffffffffa036de40 R15: 0000000000000001
FS:  00007ff287fc2700(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000038c CR3: 000000045cfae000 CR4: 00000000000407e0
Stack:
 ffff880465003da8 ffff880468b8f000 0000000000000000 ffff880468b8f000
 ffffffffa036de40 0000000000000001 ffff880465003dd8 ffffffffa0350805
 ffff880468b8f098 ffffffffa036dd60 ffff880465003e08 ffffffff812ebaa6
Call Trace:
 [<ffffffffa0350805>] mlx4_remove_one+0x25/0x50 [mlx4_core]
 [<ffffffff812ebaa6>] pci_device_remove+0x46/0xc0
 [<ffffffff813ce08f>] __device_release_driver+0x7f/0xf0
 [<ffffffff813ce1c8>] driver_detach+0xc8/0xd0
 [<ffffffff813cced9>] bus_remove_driver+0x59/0xd0
 [<ffffffff813cef80>] driver_unregister+0x30/0x70
 [<ffffffff812ebc13>] pci_unregister_driver+0x23/0x80
 [<ffffffffa03650e4>] mlx4_cleanup+0x10/0x1e [mlx4_core]
 [<ffffffff810ceff9>] SyS_delete_module+0x189/0x210
 [<ffffffff815d2f12>] system_call_fastpath+0x16/0x1b
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 9f 58 01 00 00 49 89 fd <44> 8b b3 8c 03 00 00 45 85 f6 0f 85 41 02 00 00 f6 43 08 04 44
RIP  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
 RSP <ffff880465003d88>

Signature · 7 lines

Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Or Gerlitz

2014-06-20 04:02 UTC #5 lore raw

Quoted 6 lines · Wei Yang

Quoted 3 lines
Quoted 3 lines
mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet
BUG: unable to handle kernel NULL pointer dereference at 000000000000038c
IP: [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
From this log, it happens during probe?
If not, any action after probe?

yep, maybe the bug still exists in the error flow of probe? you can probe with

num_vfs=1,1,1 port_type_array=1,1 and see if you hit it

Quoted 41 lines

Quoted 32 lines

PGD 45d3ba067 PUD 45ace8067 PMD 0

Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas scsi_transport_sas raid_class [last unloaded: mlx4_core]
CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
Hardware name: Oracle Corporation SUN FIRE X4170 M3     /ASSY,MOTHERBOARD,1U   , BIOS 17050100 08/29/2013
task: ffff880461540110 ti: ffff880465000000 task.ti: ffff880465000000
RIP: 0010:[<ffffffffa0350450>]  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
RSP: 0018:ffff880465003d88  EFLAGS: 00010296
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000026 RSI: 0000000000000292 RDI: ffff880468b8f000
RBP: ffff880465003db8 R08: 0000000000000000 R09: 0000000000000000
R10: 09f911029d74e35b R11: 09f911029d74e35b R12: 0000000000000000
R13: ffff880468b8f000 R14: ffffffffa036de40 R15: 0000000000000001
FS:  00007ff287fc2700(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000038c CR3: 000000045cfae000 CR4: 00000000000407e0
Stack:
 ffff880465003da8 ffff880468b8f000 0000000000000000 ffff880468b8f000
 ffffffffa036de40 0000000000000001 ffff880465003dd8 ffffffffa0350805
 ffff880468b8f098 ffffffffa036dd60 ffff880465003e08 ffffffff812ebaa6
Call Trace:
 [<ffffffffa0350805>] mlx4_remove_one+0x25/0x50 [mlx4_core]
 [<ffffffff812ebaa6>] pci_device_remove+0x46/0xc0
 [<ffffffff813ce08f>] __device_release_driver+0x7f/0xf0
 [<ffffffff813ce1c8>] driver_detach+0xc8/0xd0
 [<ffffffff813cced9>] bus_remove_driver+0x59/0xd0
 [<ffffffff813cef80>] driver_unregister+0x30/0x70
 [<ffffffff812ebc13>] pci_unregister_driver+0x23/0x80
 [<ffffffffa03650e4>] mlx4_cleanup+0x10/0x1e [mlx4_core]
 [<ffffffff810ceff9>] SyS_delete_module+0x189/0x210
 [<ffffffff815d2f12>] system_call_fastpath+0x16/0x1b

Quoted 8 lines

Quoted 8 lines
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 9f 58 01 00 00 49 89 fd <44> 8b b3 8c 03 00 00 45 85 f6 0f 85 41 02 00 00 f6 43 08 04 44
RIP  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
 RSP <ffff880465003d88>
CR2: 000000000000038c
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

--
Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Or Gerlitz

2014-06-20 06:17 UTC #13 lore raw

In reply to a message not in the archive

Quoted 10 lines · Or Gerlitz

Quoted 2 lines · Shirley Ma
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it failed with mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet
What do you mean by "port1" and "port2" -- can you give the exact command line you used?
Single ported VFs are currently supported for Ethernet only configuration, that is not for only IB nor for VPI, that is only if you use port_type_arrary=2,2

Note that you can still use dual-ported VFs, for both IB, Ethernet and VPI, that is
num_vfs=N will create N dual-ported VFs, are you on IB?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Wei Yang

2014-06-20 06:21 UTC #7 lore raw

In reply to a message not in the archive

Quoted 6 lines · Or Gerlitz

Quoted 4 lines · Wei Yang
Quoted 3 lines
Quoted 3 lines
mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet
BUG: unable to handle kernel NULL pointer dereference at 000000000000038c
IP: [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]

Quoted 7 lines

Quoted 2 lines
From this log, it happens during probe?
If not, any action after probe?
yep, maybe the bug still exists in the error flow of probe? you can probe with
num_vfs=1,1,1 port_type_array=1,1 and see if you hit it

I tried this

modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1

It looks good to me.

BTW, I didn't test 3.16-rc1, sine the SRIOV patch on power platform is not rebased to the latest kernel yet.

Quoted 46 lines

Quoted 33 lines

PGD 45d3ba067 PUD 45ace8067 PMD 0

Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas scsi_transport_sas raid_class [last unloaded: mlx4_core]
CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
Hardware name: Oracle Corporation SUN FIRE X4170 M3     /ASSY,MOTHERBOARD,1U   , BIOS 17050100 08/29/2013
task: ffff880461540110 ti: ffff880465000000 task.ti: ffff880465000000
RIP: 0010:[<ffffffffa0350450>]  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
RSP: 0018:ffff880465003d88  EFLAGS: 00010296
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000026 RSI: 0000000000000292 RDI: ffff880468b8f000
RBP: ffff880465003db8 R08: 0000000000000000 R09: 0000000000000000
R10: 09f911029d74e35b R11: 09f911029d74e35b R12: 0000000000000000
R13: ffff880468b8f000 R14: ffffffffa036de40 R15: 0000000000000001
FS:  00007ff287fc2700(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000038c CR3: 000000045cfae000 CR4: 00000000000407e0
Stack:
 ffff880465003da8 ffff880468b8f000 0000000000000000 ffff880468b8f000
 ffffffffa036de40 0000000000000001 ffff880465003dd8 ffffffffa0350805
 ffff880468b8f098 ffffffffa036dd60 ffff880465003e08 ffffffff812ebaa6
Call Trace:
 [<ffffffffa0350805>] mlx4_remove_one+0x25/0x50 [mlx4_core]
 [<ffffffff812ebaa6>] pci_device_remove+0x46/0xc0
 [<ffffffff813ce08f>] __device_release_driver+0x7f/0xf0
 [<ffffffff813ce1c8>] driver_detach+0xc8/0xd0
 [<ffffffff813cced9>] bus_remove_driver+0x59/0xd0
 [<ffffffff813cef80>] driver_unregister+0x30/0x70
 [<ffffffff812ebc13>] pci_unregister_driver+0x23/0x80
 [<ffffffffa03650e4>] mlx4_cleanup+0x10/0x1e [mlx4_core]
 [<ffffffff810ceff9>] SyS_delete_module+0x189/0x210
 [<ffffffff815d2f12>] system_call_fastpath+0x16/0x1b

Quoted 8 lines

Quoted 8 lines
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 9f 58 01 00 00 49 89 fd <44> 8b b3 8c 03 00 00 45 85 f6 0f 85 41 02 00 00 f6 43 08 04 44
RIP  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
 RSP <ffff880465003d88>
CR2: 000000000000038c
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Richard Yang
Help you, Help me

Signature · 7 lines

Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Or Gerlitz

2014-06-20 06:33 UTC #8 lore raw

Quoted 9 lines · Wei Yang

Quoted 5 lines
Quoted 2 lines
From this log, it happens during probe?
If not, any action after probe?
yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it
I tried this
modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 It looks good to me.

NO. I wanted you to hit the error flow where this bug seems to remain... so you need to try and use single ported VFs with IB e.g

$ modprobe mlx4_core num_vfs=1,1,1 port_type_array=1,1

Quoted 2 lines

BTW, I didn't test 3.16-rc1, sine the SRIOV patch on power platform is not rebased to the latest kernel yet.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Or Gerlitz

2014-06-20 06:34 UTC #10 lore raw

In reply to a message not in the archive

Quoted 8 lines · Or Gerlitz

Quoted 5 lines · Wei Yang
Quoted 5 lines
Quoted 2 lines
From this log, it happens during probe?
If not, any action after probe?
yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it

Quoted 7 lines

Quoted 3 lines
I tried this
modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 It looks good to me.
NO. I wanted you to hit the error flow where this bug seems to remain... so you need to try and use single ported VFs with IB e.g
$ modprobe mlx4_core num_vfs=1,1,1 port_type_array=1,1

$ modprobe mlx4_core num_vfs=1,1,1 probe_vf=1,1,1 rt_type_array=1,1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Shirley Ma

author

2014-06-20 17:15 UTC #16 lore raw

In reply to a message not in the archive

Quoted 7 lines · Or Gerlitz

Quoted 4 lines · Shirley Ma
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it failed with mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet

Quoted 7 lines

What do you mean by "port1" and "port2" -- can you give the exact command line you used?
Single ported VFs are currently supported for Ethernet only configuration, that is not for only IB nor for VPI, that is only if you use port_type_arrary=2,2

I tried command line with num_vfs without port_type_array=2,2.

num_vfs=2
num_vfs={1,1,2}

both failed.

Quoted 6 lines

Quoted 4 lines
2. After mlx4_core module is being loaded with with num_vfs={} parameters, when removing mlx4_core, it consistently hits below panic. Whether this problem is being tracked?

Quoted 8 lines

what do you mean by "num_vfs={}", is it num_vfs=N or {N}, also here, please send the exact setting you used. The crash you indicated below is supposed to be fixed by the upstream commit da1de8dfff09d33d4a5345762c21b487028e25f5 "net/mlx4_core: Keep only one driver entry release" - are you sure to have this commit in the tree you are working with?
Or.

Yes, I tried net-next tree with this commit a1de8dfff09d33d4a5345762c21b487028e25f5.

Quoted 51 lines

BUG: unable to handle kernel NULL pointer dereference at 000000000000038c
IP: [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
PGD 45d3ba067 PUD 45ace8067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: mlx4_core(-) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode ipmi_si ipmi_msghandler acpi_cpufreq pcspkr i2c_i801 i2c_core lpc_ich mfd_core shpchp sg ioatdma ib_sa ib_mad ib_core ib_addr ipv6 vxlan ixgbe dca ptp pps_core hwmon mdio ext3 jbd mbcache sd_mod crc_t10dif crct10dif_common usb_storage ahci libahci mpt2sas scsi_transport_sas raid_class [last unloaded: mlx4_core]
CPU: 13 PID: 7212 Comm: rmmod Not tainted 3.16.0-rc1+ #1
Hardware name: Oracle Corporation SUN FIRE X4170 M3     /ASSY,MOTHERBOARD,1U   , BIOS 17050100 08/29/2013
task: ffff880461540110 ti: ffff880465000000 task.ti: ffff880465000000
RIP: 0010:[<ffffffffa0350450>]  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
RSP: 0018:ffff880465003d88  EFLAGS: 00010296
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000026 RSI: 0000000000000292 RDI: ffff880468b8f000
RBP: ffff880465003db8 R08: 0000000000000000 R09: 0000000000000000
R10: 09f911029d74e35b R11: 09f911029d74e35b R12: 0000000000000000
R13: ffff880468b8f000 R14: ffffffffa036de40 R15: 0000000000000001
FS:  00007ff287fc2700(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000038c CR3: 000000045cfae000 CR4: 00000000000407e0
Stack:
 ffff880465003da8 ffff880468b8f000 0000000000000000 ffff880468b8f000
 ffffffffa036de40 0000000000000001 ffff880465003dd8 ffffffffa0350805
 ffff880468b8f098 ffffffffa036dd60 ffff880465003e08 ffffffff812ebaa6
Call Trace:
 [<ffffffffa0350805>] mlx4_remove_one+0x25/0x50 [mlx4_core]
 [<ffffffff812ebaa6>] pci_device_remove+0x46/0xc0
 [<ffffffff813ce08f>] __device_release_driver+0x7f/0xf0
 [<ffffffff813ce1c8>] driver_detach+0xc8/0xd0
 [<ffffffff813cced9>] bus_remove_driver+0x59/0xd0
 [<ffffffff813cef80>] driver_unregister+0x30/0x70
 [<ffffffff812ebc13>] pci_unregister_driver+0x23/0x80
 [<ffffffffa03650e4>] mlx4_cleanup+0x10/0x1e [mlx4_core]
 [<ffffffff810ceff9>] SyS_delete_module+0x189/0x210
 [<ffffffff815d2f12>] system_call_fastpath+0x16/0x1b

Quoted 8 lines

Quoted 8 lines
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 9f 58 01 00 00 49 89 fd <44> 8b b3 8c 03 00 00 45 85 f6 0f 85 41 02 00 00 f6 43 08 04 44
RIP  [<ffffffffa0350450>] __mlx4_remove_one+0x20/0x380 [mlx4_core]
 RSP <ffff880465003d88>
CR2: 000000000000038c
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Shirley Ma

author

2014-06-20 17:16 UTC #15 lore raw

In reply to a message not in the archive

Quoted 8 lines · Or Gerlitz

Quoted 6 lines · Or Gerlitz
Quoted 2 lines · Shirley Ma
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it failed with mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet
What do you mean by "port1" and "port2" -- can you give the exact command line you used?

Quoted 7 lines

Quoted 3 lines
Single ported VFs are currently supported for Ethernet only configuration, that is not for only IB nor for VPI, that is only if you use port_type_arrary=2,2
Note that you can still use dual-ported VFs, for both IB, Ethernet and VPI, that is
num_vfs=N will create N dual-ported VFs, are you on IB?

Yes, I tried num_vfs=N. It failed. I didn't combine with port_type_array=2,2. If it's required, then the code needs to check the arguments.

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Shirley Ma

author

2014-06-20 17:18 UTC #12 lore raw

In reply to a message not in the archive

When loading the modules, the proper arguments need to be checked if one depends on another. You can easily reproduce this problem by only using num_vfs = N.

Shirley

Quoted 13 lines · Or Gerlitz

Quoted 12 lines · Or Gerlitz
Quoted 9 lines · Wei Yang
Quoted 5 lines
Quoted 2 lines
From this log, it happens during probe?
If not, any action after probe?
yep, maybe the bug still exists in the error flow of probe? you can probe with num_vfs=1,1,1 port_type_array=1,1 and see if you hit it
I tried this
modprobe mlx4_core num_vfs=3 probe_vf=3 port_type_array=1,1 It looks good to me.

Quoted 8 lines

Quoted 3 lines
NO. I wanted you to hit the error flow where this bug seems to remain... so you need to try and use single ported VFs with IB e.g
$ modprobe mlx4_core num_vfs=1,1,1 port_type_array=1,1
or
$ modprobe mlx4_core num_vfs=1,1,1 probe_vf=1,1,1 rt_type_array=1,1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Or Gerlitz

2014-06-22 09:29 UTC #18 lore raw

In reply to a message not in the archive

Quoted 14 lines · Shirley Ma

Quoted 12 lines · Or Gerlitz
Quoted 3 lines · Shirley Ma
1. Whether IB VFs is supported in ConnectX-2 (mlx4 driver)?
I tried to num_vfs={port1,port2,port1+2} when loading mlx4_core module, it failed with mlx4_core 0000:40:00.0: Invalid syntax of num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet
What do you mean by "port1" and "port2" -- can you give the exact command line you used?
Single ported VFs are currently supported for Ethernet only configuration, that is not for only IB nor for VPI, that is only if you use port_type_arrary=2,2

Quoted 6 lines

I tried command line with num_vfs without port_type_array=2,2.
num_vfs=2
num_vfs={1,1,2}
both failed.

Please provide further details on what fails when you use num_vfs=2, here it works just fine, e.g send us the output of

$ modprobe -v mlx4_core

and the related dmesg

As for the crash you reported, indeed, it seems we have a bug there on the error flow of module loading when single ported VF are requested in conjunction with IB ports. I have it fixed on my system with this patch which are will review etc

From 6dec77e0dc68679e4127de19f6798151cc8b2fe6 Mon Sep 17 00:00:00 2001
From: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>
Date: Sun, 22 Jun 2014 13:46:04 +0300
Subject: [PATCH] net/mlx4_core: Return error when module is loaded with invalid VF configuration

Single ported VF are currently not supported on configurations where one or both ports are IB. When we hit this case, the module load function didn't return error, fix that.

Fixes dd41cc3 ('net/mlx4: Adapt num_vfs/probed_vf params for single port VF')

Signed-off-by Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]>

---
 drivers/net/ethernet/mellanox/mlx4/main.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

drivers/net/ethernet/mellanox/mlx4/main.c +0 -0

index 5f42f6d..4f4d48c 100644
@@ -2439,6 +2439,7 @@ slave_start:
                            (num_vfs_argc > 1 || probe_vfs_argc > 1)) {
                                mlx4_err(dev,
                                         "Invalid syntax of

num_vfs/probe_vfs with IB port - single port VFs syntax is only supported when all ports are configured as ethernet\n");

+                               err = -EINVAL;
                                goto err_close;
                        }
                        for (i = 0; i < sizeof(nvfs)/sizeof(nvfs[0]);

i++) {
--
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Or Gerlitz

2014-06-22 10:25 UTC #20 lore raw

In reply to a message not in the archive

Quoted 2 lines · Or Gerlitz

As for the crash you reported, indeed, it seems we have a bug there on the error flow of module loading when single ported VF are requested in conjunction with IB ports. I have it fixed on my system with this patch which are will review etc

Just posted a patch fixing the crash you reported to netdev
http://marc.info/?l=linux-netdev&m=140343252214146&w=2

Thanks for the report and tell us if you have further issues.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html