Failure to recreate virtual functions
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patch b7297783c2bb ("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any value
Quoted 1 line
0.
[ 223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 223.663864] pci 0000:81:00.2: enabling Extended Tags [ 223.665143] pci 0000:81:00.2: Adding to iommu group 52 [ 223.665215] pci 0000:81:00.2: Using iommu direct mapping [ 223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 224.041711] pci 0000:81:00.3: enabling Extended Tags [ 224.043660] pci 0000:81:00.3: Adding to iommu group 53 [ 224.043738] pci 0000:81:00.3: Using iommu direct mapping [ 224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846] ens1f0_0: renamed from eth0 [ 237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 237.526880] ens1f0_1: renamed from eth1 [ 248.953873] pci 0000:81:00.2: Removing from iommu group 52 [ 248.954114] pci 0000:81:00.3: Removing from iommu group 53 [ 249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 258.936258] pci 0000:81:00.2: enabling Extended Tags [ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16 [ 258.938053] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 258.938196] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 258.938229] mlx5_core 0000:81:00.2: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting [ 258.938315] mlx5_core 0000:81:00.2: init_one:1308:(pid 265): mlx5_load_one failed with error code -12 [ 258.938540] mlx5_core: probe of 0000:81:00.2 failed with error -12 [ 258.938597] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 258.938657] pci 0000:81:00.3: enabling Extended Tags [ 258.939431] pci 0000:81:00.3: Failed to add to iommu group 52: -16 [ 258.939928] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 258.940039] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 258.940071] mlx5_core 0000:81:00.3: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting [ 258.940158] mlx5_core 0000:81:00.3: init_one:1308:(pid 265): mlx5_load_one failed with error code -12 [ 258.940400] mlx5_core: probe of 0000:81:00.3 failed with error -12
On previous patch 0e31a7266508 ("iommu/vt-d: Remove startup parameter from device_def_domain_type()") in the series same sequence of actions doesn't trigger any iommu errors:
[ 164.252254] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 164.288724] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 164.394839] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 164.394938] pci 0000:81:00.2: enabling Extended Tags [ 164.396087] pci 0000:81:00.2: Adding to iommu group 52 [ 164.396154] pci 0000:81:00.2: Using iommu direct mapping [ 164.396679] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 164.396803] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 164.619320] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 164.625754] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 164.625922] mlx5_core 0000:81:00.2: Assigned random MAC address 5e:1e:9b:ca:c8:e5 [ 164.739694] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 164.774637] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 164.774709] pci 0000:81:00.3: enabling Extended Tags [ 164.775816] pci 0000:81:00.3: Adding to iommu group 53 [ 164.775886] pci 0000:81:00.3: Using iommu direct mapping [ 164.776610] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 164.776734] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 164.999360] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 165.007118] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 165.007327] mlx5_core 0000:81:00.3: Assigned random MAC address 82:4a:7a:5f:81:55 [ 165.123927] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 172.063665] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 172.103306] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 173.033033] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.091605] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.129258] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 173.129863] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.203879] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 173.204002] ens1f0_0: renamed from eth1 [ 173.289454] ens1f0_1: renamed from eth0 [ 186.720692] pci 0000:81:00.2: Removing from iommu group 52 [ 186.720994] pci 0000:81:00.3: Removing from iommu group 53 [ 187.771549] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 188.141758] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 188.394072] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 191.116400] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 191.128965] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 191.235151] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 191.235250] pci 0000:81:00.2: enabling Extended Tags [ 191.236463] pci 0000:81:00.2: Adding to iommu group 52 [ 191.236531] pci 0000:81:00.2: Using iommu direct mapping [ 191.237037] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 191.237161] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 191.457369] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 191.463355] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 191.463509] mlx5_core 0000:81:00.2: Assigned random MAC address e6:f2:0c:b4:e3:2e [ 191.572884] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 191.608592] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 191.608664] pci 0000:81:00.3: enabling Extended Tags [ 191.609434] pci 0000:81:00.3: Adding to iommu group 53 [ 191.609466] pci 0000:81:00.3: Using iommu direct mapping [ 191.609760] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 191.609862] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 191.826324] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 191.832558] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 191.832730] mlx5_core 0000:81:00.3: Assigned random MAC address a2:dc:76:30:18:6c [ 191.949625] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0
Thanks,
Vlad
Hi Vilad,
Quoted 43 lines · Vlad Buslov
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patchb7297783c2bb("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any valueQuoted 1 line
0.
[ 223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 223.663864] pci 0000:81:00.2: enabling Extended Tags [ 223.665143] pci 0000:81:00.2: Adding to iommu group 52 [ 223.665215] pci 0000:81:00.2: Using iommu direct mapping [ 223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 224.041711] pci 0000:81:00.3: enabling Extended Tags [ 224.043660] pci 0000:81:00.3: Adding to iommu group 53 [ 224.043738] pci 0000:81:00.3: Using iommu direct mapping [ 224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846] ens1f0_0: renamed from eth0 [ 237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 237.526880] ens1f0_1: renamed from eth1 [ 248.953873] pci 0000:81:00.2: Removing from iommu group 52 [ 248.954114] pci 0000:81:00.3: Removing from iommu group 53
Quoted 8 lines
[ 249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 258.936258] pci 0000:81:00.2: enabling Extended Tags [ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16
It seems that an EBUSY error returned from iommu_group_add_device(). Can you please hack some debug messages in iommu_group_add_device() so that we can know where the EBUSY returns?
Best regards,
Baolu
Quoted 78 lines
[ 258.938053] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 258.938196] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 258.938229] mlx5_core 0000:81:00.2: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting [ 258.938315] mlx5_core 0000:81:00.2: init_one:1308:(pid 265): mlx5_load_one failed with error code -12 [ 258.938540] mlx5_core: probe of 0000:81:00.2 failed with error -12 [ 258.938597] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 258.938657] pci 0000:81:00.3: enabling Extended Tags [ 258.939431] pci 0000:81:00.3: Failed to add to iommu group 52: -16 [ 258.939928] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 258.940039] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 258.940071] mlx5_core 0000:81:00.3: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting [ 258.940158] mlx5_core 0000:81:00.3: init_one:1308:(pid 265): mlx5_load_one failed with error code -12 [ 258.940400] mlx5_core: probe of 0000:81:00.3 failed with error -12On previous patch
0e31a7266508("iommu/vt-d: Remove startup parameter fromdevice_def_domain_type()") in the series same sequence of actions doesn't trigger any iommu errors:[ 164.252254] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 164.288724] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 164.394839] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 164.394938] pci 0000:81:00.2: enabling Extended Tags [ 164.396087] pci 0000:81:00.2: Adding to iommu group 52 [ 164.396154] pci 0000:81:00.2: Using iommu direct mapping [ 164.396679] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 164.396803] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 164.619320] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 164.625754] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 164.625922] mlx5_core 0000:81:00.2: Assigned random MAC address 5e:1e:9b:ca:c8:e5 [ 164.739694] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 164.774637] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 164.774709] pci 0000:81:00.3: enabling Extended Tags [ 164.775816] pci 0000:81:00.3: Adding to iommu group 53 [ 164.775886] pci 0000:81:00.3: Using iommu direct mapping [ 164.776610] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 164.776734] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 164.999360] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 165.007118] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 165.007327] mlx5_core 0000:81:00.3: Assigned random MAC address 82:4a:7a:5f:81:55 [ 165.123927] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 172.063665] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 172.103306] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 173.033033] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.091605] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.129258] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 173.129863] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.203879] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 173.204002] ens1f0_0: renamed from eth1 [ 173.289454] ens1f0_1: renamed from eth0 [ 186.720692] pci 0000:81:00.2: Removing from iommu group 52 [ 186.720994] pci 0000:81:00.3: Removing from iommu group 53 [ 187.771549] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 188.141758] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 188.394072] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 191.116400] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 191.128965] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 191.235151] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 191.235250] pci 0000:81:00.2: enabling Extended Tags [ 191.236463] pci 0000:81:00.2: Adding to iommu group 52 [ 191.236531] pci 0000:81:00.2: Using iommu direct mapping [ 191.237037] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 191.237161] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 191.457369] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 191.463355] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 191.463509] mlx5_core 0000:81:00.2: Assigned random MAC address e6:f2:0c:b4:e3:2e [ 191.572884] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 191.608592] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 191.608664] pci 0000:81:00.3: enabling Extended Tags [ 191.609434] pci 0000:81:00.3: Adding to iommu group 53 [ 191.609466] pci 0000:81:00.3: Using iommu direct mapping [ 191.609760] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 191.609862] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 191.826324] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 191.832558] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 191.832730] mlx5_core 0000:81:00.3: Assigned random MAC address a2:dc:76:30:18:6c [ 191.949625] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0
Quoted 3 lines
Thanks,
Vlad
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 55 lines · Lu Baolu
Hi Vilad,
Quoted 43 lines · Vlad Buslov
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patchb7297783c2bb("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any valueQuoted 1 line
0.
[ 223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 223.663864] pci 0000:81:00.2: enabling Extended Tags [ 223.665143] pci 0000:81:00.2: Adding to iommu group 52 [ 223.665215] pci 0000:81:00.2: Using iommu direct mapping [ 223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 224.041711] pci 0000:81:00.3: enabling Extended Tags [ 224.043660] pci 0000:81:00.3: Adding to iommu group 53 [ 224.043738] pci 0000:81:00.3: Using iommu direct mapping [ 224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846] ens1f0_0: renamed from eth0 [ 237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 237.526880] ens1f0_1: renamed from eth1 [ 248.953873] pci 0000:81:00.2: Removing from iommu group 52 [ 248.954114] pci 0000:81:00.3: Removing from iommu group 53Quoted 8 lines
[ 249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 258.936258] pci 0000:81:00.2: enabling Extended Tags [ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16
Quoted 6 lines
It seems that an EBUSY error returned from
iommu_group_add_device(). Can you please hack some debug messages iniommu_group_add_device()so that we can know where the EBUSY returns?Best regards,
Baolu
The error code is returned by __iommu_attach_device().
Quoted 83 lines
Quoted 83 lines
[ 258.938053] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 258.938196] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 258.938229] mlx5_core 0000:81:00.2: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting [ 258.938315] mlx5_core 0000:81:00.2: init_one:1308:(pid 265): mlx5_load_one failed with error code -12 [ 258.938540] mlx5_core: probe of 0000:81:00.2 failed with error -12 [ 258.938597] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 258.938657] pci 0000:81:00.3: enabling Extended Tags [ 258.939431] pci 0000:81:00.3: Failed to add to iommu group 52: -16 [ 258.939928] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 258.940039] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 258.940071] mlx5_core 0000:81:00.3: mlx5_function_setup:923:(pid 265): Failed initializing command interface, aborting [ 258.940158] mlx5_core 0000:81:00.3: init_one:1308:(pid 265): mlx5_load_one failed with error code -12 [ 258.940400] mlx5_core: probe of 0000:81:00.3 failed with error -12On previous patch
0e31a7266508("iommu/vt-d: Remove startup parameter fromdevice_def_domain_type()") in the series same sequence of actions doesn't trigger any iommu errors:[ 164.252254] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 164.288724] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 164.394839] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 164.394938] pci 0000:81:00.2: enabling Extended Tags [ 164.396087] pci 0000:81:00.2: Adding to iommu group 52 [ 164.396154] pci 0000:81:00.2: Using iommu direct mapping [ 164.396679] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 164.396803] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 164.619320] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 164.625754] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 164.625922] mlx5_core 0000:81:00.2: Assigned random MAC address 5e:1e:9b:ca:c8:e5 [ 164.739694] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 164.774637] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 164.774709] pci 0000:81:00.3: enabling Extended Tags [ 164.775816] pci 0000:81:00.3: Adding to iommu group 53 [ 164.775886] pci 0000:81:00.3: Using iommu direct mapping [ 164.776610] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 164.776734] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 164.999360] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 165.007118] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 165.007327] mlx5_core 0000:81:00.3: Assigned random MAC address 82:4a:7a:5f:81:55 [ 165.123927] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 172.063665] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 172.103306] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 173.033033] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.091605] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.129258] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 173.129863] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 173.203879] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 173.204002] ens1f0_0: renamed from eth1 [ 173.289454] ens1f0_1: renamed from eth0 [ 186.720692] pci 0000:81:00.2: Removing from iommu group 52 [ 186.720994] pci 0000:81:00.3: Removing from iommu group 53 [ 187.771549] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 188.141758] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 188.394072] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 191.116400] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 191.128965] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 191.235151] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 191.235250] pci 0000:81:00.2: enabling Extended Tags [ 191.236463] pci 0000:81:00.2: Adding to iommu group 52 [ 191.236531] pci 0000:81:00.2: Using iommu direct mapping [ 191.237037] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 191.237161] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 191.457369] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 191.463355] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 191.463509] mlx5_core 0000:81:00.2: Assigned random MAC address e6:f2:0c:b4:e3:2e [ 191.572884] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 191.608592] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 191.608664] pci 0000:81:00.3: enabling Extended Tags [ 191.609434] pci 0000:81:00.3: Adding to iommu group 53 [ 191.609466] pci 0000:81:00.3: Using iommu direct mapping [ 191.609760] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 191.609862] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 191.826324] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 191.832558] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 191.832730] mlx5_core 0000:81:00.3: Assigned random MAC address a2:dc:76:30:18:6c [ 191.949625] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0Thanks,
Vlad
Hi,
Quoted 54 lines · Vlad Buslov
Quoted 53 lines · Lu Baolu
Hi Vilad,
Quoted 50 lines · Vlad Buslov
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patchb7297783c2bb("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any valueQuoted 1 line
0.
[ 223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 223.663864] pci 0000:81:00.2: enabling Extended Tags [ 223.665143] pci 0000:81:00.2: Adding to iommu group 52 [ 223.665215] pci 0000:81:00.2: Using iommu direct mapping [ 223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 224.041711] pci 0000:81:00.3: enabling Extended Tags [ 224.043660] pci 0000:81:00.3: Adding to iommu group 53 [ 224.043738] pci 0000:81:00.3: Using iommu direct mapping [ 224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846] ens1f0_0: renamed from eth0 [ 237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 237.526880] ens1f0_1: renamed from eth1 [ 248.953873] pci 0000:81:00.2: Removing from iommu group 52 [ 248.954114] pci 0000:81:00.3: Removing from iommu group 53 [ 249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 258.936258] pci 0000:81:00.2: enabling Extended Tags [ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16
Quoted 8 lines
Quoted 6 lines
It seems that an EBUSY error returned from
iommu_group_add_device(). Can you please hack some debug messages iniommu_group_add_device()so that we can know where the EBUSY returns?Best regards,
BaoluThe error code is returned by
__iommu_attach_device().
Thanks!
It looks like the system has already a domain for specific pci bdf device. Does this VF share the bdf with other devices? Or has been previously created, and system failed to get chance to remove it?
Best regards,
Baolu
Robin Murphy
reviewerQuoted 63 lines · Lu Baolu
Hi,
Quoted 60 lines · Vlad Buslov
Quoted 59 lines · Lu Baolu
Hi Vilad,
Quoted 42 lines · Vlad Buslov
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patchb7297783c2bb("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any valueQuoted 1 line
0.
[ 223.525282]
mlx5_core0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027]mlx5_core0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class0x020000
[ 223.663864] pci 0000:81:00.2: enabling Extended Tags
[ 223.665143] pci 0000:81:00.2: Adding to iommu group 52
[ 223.665215] pci 0000:81:00.2: Using iommu direct mapping
[ 223.665771]mlx5_core0000:81:00.2: enabling device (0000 -> 0002)
[ 223.665890]mlx5_core0000:81:00.2: firmware version: 16.26.148
[ 223.889908]mlx5_core0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438]mlx5_core0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636]mlx5_core0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905]mlx5_core0000:81:00.2 ens1f0v0: renamed from eth0
[ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class0x020000
[ 224.041711] pci 0000:81:00.3: enabling Extended Tags
[ 224.043660] pci 0000:81:00.3: Adding to iommu group 53
[ 224.043738] pci 0000:81:00.3: Using iommu direct mapping
[ 224.044196]mlx5_core0000:81:00.3: enabling device (0000 -> 0002)
[ 224.044298]mlx5_core0000:81:00.3: firmware version: 16.26.148
[ 224.268099]mlx5_core0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983]mlx5_core0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195]mlx5_core0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359]mlx5_core0000:81:00.3 ens1f0v1: renamed from eth0
[ 236.325027]mlx5_core0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766]mlx5_core0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052]mlx5_core0000:81:00.0 ens1f0: renamed from eth0
[ 237.390768]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846]ens1f0_0: renamed from eth0
[ 237.460399]mlx5_core0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[ 237.526880]ens1f0_1: renamed from eth1
[ 248.953873] pci 0000:81:00.2: Removing from iommu group 52
[ 248.954114] pci 0000:81:00.3: Removing from iommu group 53Quoted 8 lines
[ 249.960570]
mlx5_core0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 250.559431]mlx5_core0000:81:00.0 ens1f0: renamed from eth0
[ 258.819162]mlx5_core0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625]mlx5_core0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class0x020000
[ 258.936258] pci 0000:81:00.2: enabling Extended Tags
[ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16It seems that an EBUSY error returned from
iommu_group_add_device(). Can you please hack some debug messages iniommu_group_add_device()so that we can know where the EBUSY returns?Best regards,
Baolu
Quoted 8 lines
Quoted 2 lines
The error code is returned by
__iommu_attach_device().Thanks!
It looks like the system has already a domain for specific pci bdf device. Does this VF share the bdf with other devices? Or has been previously created, and system failed to get chance to remove it?
At a glance, it looks like it might be down to intel_iommu_remove_device() not calling dmar_remove_one_dev_info() like the old notifier did. If the group is getting torn down and recreated, but the driver still has a stale pointer to the old default domain cached, which dmar_insert_one_dev_info() finds and returns, that would seem to explain the observed behaviour.
Robin.
Hi,
Quoted 67 lines · Robin Murphy
Quoted 66 lines · Lu Baolu
Hi,
Quoted 62 lines · Vlad Buslov
Quoted 53 lines · Lu Baolu
Hi Vilad,
Quoted 42 lines · Vlad Buslov
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patchb7297783c2bb("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any valueQuoted 1 line
0.
[ 223.525282]
mlx5_core0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027]mlx5_core0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class0x020000
[ 223.663864] pci 0000:81:00.2: enabling Extended Tags
[ 223.665143] pci 0000:81:00.2: Adding to iommu group 52
[ 223.665215] pci 0000:81:00.2: Using iommu direct mapping
[ 223.665771]mlx5_core0000:81:00.2: enabling device (0000 -> 0002)
[ 223.665890]mlx5_core0000:81:00.2: firmware version: 16.26.148
[ 223.889908]mlx5_core0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438]mlx5_core0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636]mlx5_core0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905]mlx5_core0000:81:00.2 ens1f0v0: renamed from eth0
[ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class0x020000
[ 224.041711] pci 0000:81:00.3: enabling Extended Tags
[ 224.043660] pci 0000:81:00.3: Adding to iommu group 53
[ 224.043738] pci 0000:81:00.3: Using iommu direct mapping
[ 224.044196]mlx5_core0000:81:00.3: enabling device (0000 -> 0002)
[ 224.044298]mlx5_core0000:81:00.3: firmware version: 16.26.148
[ 224.268099]mlx5_core0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983]mlx5_core0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195]mlx5_core0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359]mlx5_core0000:81:00.3 ens1f0v1: renamed from eth0
[ 236.325027]mlx5_core0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766]mlx5_core0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052]mlx5_core0000:81:00.0 ens1f0: renamed from eth0
[ 237.390768]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846]ens1f0_0: renamed from eth0
[ 237.460399]mlx5_core0000:81:00.0: E-Switch: SRIOV enabled: active vports(3)
[ 237.526880]ens1f0_1: renamed from eth1
[ 248.953873] pci 0000:81:00.2: Removing from iommu group 52
[ 248.954114] pci 0000:81:00.3: Removing from iommu group 53Quoted 8 lines
[ 249.960570]
mlx5_core0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135]mlx5_core0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 250.559431]mlx5_core0000:81:00.0 ens1f0: renamed from eth0
[ 258.819162]mlx5_core0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625]mlx5_core0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class0x020000
[ 258.936258] pci 0000:81:00.2: enabling Extended Tags
[ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16Quoted 6 lines
It seems that an EBUSY error returned from
iommu_group_add_device(). Can you please hack some debug messages iniommu_group_add_device()so that we can know where the EBUSY returns?Best regards,
BaoluThe error code is returned by
__iommu_attach_device().
Quoted 7 lines
Quoted 5 lines
Thanks!
It looks like the system has already a domain for specific pci bdf device. Does this VF share the bdf with other devices? Or has been previously created, and system failed to get chance to remove it?
At a glance, it looks like it might be down to
intel_iommu_remove_device()not callingdmar_remove_one_dev_info()like the old notifier did. If the group is getting torn down and recreated, but the driver still has a stale pointer to the old default domain cached, whichdmar_insert_one_dev_info()finds and returns, that would seem to explain the observed behaviour.
Yes agreed.
Vlad,
Can you please try below change?
drivers/iommu/intel-iommu.c +2 -0
@@ -5575,6 +5575,8 @@ static void intel_iommu_remove_device(struct device *dev)if (!iommu)return ; +dmar_remove_one_dev_info (dev); +iommu_group_remove_device (dev);iommu_device_unlink (&iommu->iommu, dev);
Best regards,
Baolu
Quoted 120 lines · Lu Baolu
Hi,
Quoted 91 lines · Robin Murphy
Quoted 81 lines · Lu Baolu
Hi,
Quoted 78 lines · Vlad Buslov
Quoted 77 lines · Lu Baolu
Hi Vilad,
Quoted 60 lines · Vlad Buslov
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patchb7297783c2bb("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any valueQuoted 1 line
0.
[ 223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 223.663864] pci 0000:81:00.2: enabling Extended Tags [ 223.665143] pci 0000:81:00.2: Adding to iommu group 52 [ 223.665215] pci 0000:81:00.2: Using iommu direct mapping [ 223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 224.041711] pci 0000:81:00.3: enabling Extended Tags [ 224.043660] pci 0000:81:00.3: Adding to iommu group 53 [ 224.043738] pci 0000:81:00.3: Using iommu direct mapping [ 224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846] ens1f0_0: renamed from eth0 [ 237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 237.526880] ens1f0_1: renamed from eth1 [ 248.953873] pci 0000:81:00.2: Removing from iommu group 52 [ 248.954114] pci 0000:81:00.3: Removing from iommu group 53 [ 249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8)StrdSz(2048) RxCqeCmprss(0)
Quoted 8 lines
[ 250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 258.936258] pci 0000:81:00.2: enabling Extended Tags [ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16It seems that an EBUSY error returned from
iommu_group_add_device(). Can you please hack some debug messages iniommu_group_add_device()so that we can know where the EBUSY returns?Best regards,
BaoluQuoted 8 lines
Quoted 2 lines
The error code is returned by
__iommu_attach_device().Thanks!
It looks like the system has already a domain for specific pci bdf device. Does this VF share the bdf with other devices? Or has been previously created, and system failed to get chance to remove it?
Quoted 5 lines
At a glance, it looks like it might be down to
intel_iommu_remove_device()not callingdmar_remove_one_dev_info()like the old notifier did. If the group is getting torn down and recreated, but the driver still has a stale pointer to the old default domain cached, whichdmar_insert_one_dev_info()finds and returns, that would seem to explain the observed behaviour.Yes agreed.
Vlad,
Can you please try below change?
drivers/iommu/intel-iommu.c +2
-0@@ -5575,6 +5575,8 @@ static void intel_iommu_remove_device(struct device *dev)if (!iommu)return ; +dmar_remove_one_dev_info (dev); +iommu_group_remove_device (dev);iommu_device_unlink (&iommu->iommu, dev);
Quoted 2 lines
Best regards,
Baolu
Hi Baolu,
This patch fixes the issue for me.
Thanks,
Vlad
Hi,
Quoted 120 lines · Vlad Buslov
Quoted 118 lines · Lu Baolu
Hi,
Quoted 91 lines · Robin Murphy
Quoted 81 lines · Lu Baolu
Hi,
Quoted 78 lines · Vlad Buslov
Quoted 77 lines · Lu Baolu
Hi Vilad,
Quoted 60 lines · Vlad Buslov
Hi Lu Baolu,
Our mlx5 driver fails to recreate VFs when cmdline includes "intel_iommu=on iommu=pt" after recent merge of patch set "iommu/vt-d:
Delegate DMA domain to generic iommu". I've bisected the failure to patchb7297783c2bb("iommu/vt-d: Remove duplicated code for device hotplug"). Here is the dmesg log for following case: enable switchdev mode, set number of VFs to 0, then set it back to any valueQuoted 1 line
0.
[ 223.525282] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 223.562027] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 223.663766] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 223.663864] pci 0000:81:00.2: enabling Extended Tags [ 223.665143] pci 0000:81:00.2: Adding to iommu group 52 [ 223.665215] pci 0000:81:00.2: Using iommu direct mapping [ 223.665771] mlx5_core 0000:81:00.2: enabling device (0000 -> 0002) [ 223.665890] mlx5_core 0000:81:00.2: firmware version: 16.26.148 [ 223.889908] mlx5_core 0000:81:00.2: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 223.896438] mlx5_core 0000:81:00.2: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 223.896636] mlx5_core 0000:81:00.2: Assigned random MAC address 56:1f:95:e0:51:d6 [ 224.012905] mlx5_core 0000:81:00.2 ens1f0v0: renamed from eth0 [ 224.041651] pci 0000:81:00.3: [15b3:101a] type 00 class 0x020000 [ 224.041711] pci 0000:81:00.3: enabling Extended Tags [ 224.043660] pci 0000:81:00.3: Adding to iommu group 53 [ 224.043738] pci 0000:81:00.3: Using iommu direct mapping [ 224.044196] mlx5_core 0000:81:00.3: enabling device (0000 -> 0002) [ 224.044298] mlx5_core 0000:81:00.3: firmware version: 16.26.148 [ 224.268099] mlx5_core 0000:81:00.3: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps [ 224.274983] mlx5_core 0000:81:00.3: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 224.275195] mlx5_core 0000:81:00.3: Assigned random MAC address a6:1e:56:0a:d9:f2 [ 224.388359] mlx5_core 0000:81:00.3 ens1f0v1: renamed from eth0 [ 236.325027] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(1) [ 236.362766] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (2) [ 237.290066] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.350215] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.373052] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 237.390768] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0) [ 237.447846] ens1f0_0: renamed from eth0 [ 237.460399] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 237.526880] ens1f0_1: renamed from eth1 [ 248.953873] pci 0000:81:00.2: Removing from iommu group 52 [ 248.954114] pci 0000:81:00.3: Removing from iommu group 53 [ 249.960570] mlx5_core 0000:81:00.0: E-Switch: disable SRIOV: active vports(3) mode(2) [ 250.319135] mlx5_core 0000:81:00.0: MLX5E: StrdRq(1) RqSz(8)StrdSz(2048) RxCqeCmprss(0)
Quoted 8 lines
[ 250.559431] mlx5_core 0000:81:00.0 ens1f0: renamed from eth0 [ 258.819162] mlx5_core 0000:81:00.0: E-Switch: E-Switch enable SRIOV: nvfs(2) mode (1) [ 258.831625] mlx5_core 0000:81:00.0: E-Switch: SRIOV enabled: active vports(3) [ 258.936160] pci 0000:81:00.2: [15b3:101a] type 00 class 0x020000 [ 258.936258] pci 0000:81:00.2: enabling Extended Tags [ 258.937438] pci 0000:81:00.2: Failed to add to iommu group 52: -16It seems that an EBUSY error returned from
iommu_group_add_device(). Can you please hack some debug messages iniommu_group_add_device()so that we can know where the EBUSY returns?Best regards,
BaoluQuoted 8 lines
Quoted 2 lines
The error code is returned by
__iommu_attach_device().Thanks!
It looks like the system has already a domain for specific pci bdf device. Does this VF share the bdf with other devices? Or has been previously created, and system failed to get chance to remove it?
Quoted 5 lines
At a glance, it looks like it might be down to
intel_iommu_remove_device()not callingdmar_remove_one_dev_info()like the old notifier did. If the group is getting torn down and recreated, but the driver still has a stale pointer to the old default domain cached, whichdmar_insert_one_dev_info()finds and returns, that would seem to explain the observed behaviour.Yes agreed.
Vlad,
Can you please try below change?
drivers/iommu/intel-iommu.c +2
-0@@ -5575,6 +5575,8 @@ static void intel_iommu_remove_device(struct device *dev)if (!iommu)return ; +dmar_remove_one_dev_info (dev); +iommu_group_remove_device (dev);
Quoted 9 lines
Quoted 4 lines
drivers/iommu/intel-iommu.c +0
-0iommu_device_unlink (&iommu->iommu, dev);Best regards,
BaoluHi Baolu,
This patch fixes the issue for me.
Great! Thanks for testing. I will submit a fix soon.
Best regards,
Baolu