Skip to contentSkip to conversation

eal: forcing IOVA as PA in ppc

PATCH In review Takeshi Yoshimura 4 participants 22 messages Jul 31, 2019 – Aug 2, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 eal: forcing IOVA as PA in ppc

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

---
 lib/librte_eal/linux/eal/eal.c | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/linux/eal/eal.c +6 -0
index 946222ccd..db2dec922 100644
@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
 				RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
 			}
 		}
+#endif
+#ifdef RTE_ARCH_PPC_64
+		if (iova_mode == RTE_IOVA_VA) {
+			iova_mode = RTE_IOVA_PA;
+			RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
+		}
 #endif
 		rte_eal_get_configuration()->iova_mode = iova_mode;
 	} else {
Signature · 1 line
2.17.1
Acked-by on v1 1/1
#1 lore raw
Quoted 22 lines · Takeshi Yoshimura

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

---
  lib/librte_eal/linux/eal/eal.c | 6 ++++++
  1 file changed, 6 insertions(+)
lib/librte_eal/linux/eal/eal.c +0 -0
index 946222ccd..db2dec922 100644

lib/librte_eal/linux/eal/eal.c rte_eal_init(int argc, char **argv) · line 1130

Quoted 10 lines
lib/librte_eal/linux/eal/eal.c +6 -0
@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
  				RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
  			}
  		}
+#endif
+#ifdef RTE_ARCH_PPC_64
+		if (iova_mode == RTE_IOVA_VA) {
+			iova_mode = RTE_IOVA_PA;
+			RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
+		}
  #endif

I would have preferred a space between #endif and a new #ifdef, but otherwise

Signature · 2 lines
Thanks,
Anatoly
#2 lore raw
In reply to Takeshi Yoshimura
Quoted 7 lines · Takeshi Yoshimura

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.

What is the "ppc driver" you are referring to?
Do you mean the vfio implementation for ppc?

Quoted 5 lines

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?

Signature · 1 line
David Marchand
#3 lore raw

-----David Marchand <[email protected]> wrote: -----

Quoted 22 lines

To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Quoted 5 lines · Takeshi Yoshimura

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA

as

Quoted 2 lines

PA in ppc as before because current ppc driver does not support VA mode.

What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?

Quoted 1 line
Quoted 13 lines
Quoted 4 lines

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?

Signature · 1 line
David Marchand

Current code does not implement VA mode for ppc. So, yes. VA mode is not working under current dpdk on ppc.

#4 lore raw
In reply to David Marchand

-----David Marchand <[email protected]> wrote: -----

Quoted 22 lines

To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Quoted 5 lines · Takeshi Yoshimura

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA

as

Quoted 2 lines

PA in ppc as before because current ppc driver does not support VA mode.

What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?

Quoted 1 line
Quoted 13 lines
Quoted 4 lines

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?

Signature · 1 line
David Marchand

Ah, I overlooked your first question. The "ppc driver" is in drivers/bus/pci/linux/pci.c. In pci.c, pci_one_device_iommu_support_va() always returns false in ppc. so, the driver does not allow VA mode in ppc.

This return value is passed to rte_pci_probe_one_driver() in drivers/bus/pci/pci_common.c. And the error log appeared like:

EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
#5 lore raw
Quoted 11 lines · Takeshi T Yoshimura

Ah, I overlooked your first question. The "ppc driver" is in drivers/bus/pci/linux/pci.c. In pci.c, pci_one_device_iommu_support_va() always returns false in ppc. so, the driver does not allow VA mode in ppc.

This return value is passed to rte_pci_probe_one_driver() in drivers/bus/pci/pci_common.c. And the error log appeared like:

EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing

In which situation does this error happen?
When probing the device at eal init time? or later when hotplugging? Can you give the full eal logs with --log-level lib.*:debug?

Signature · 1 line
David Marchand
#6 lore raw
Quoted 1 line · David Marchand

--log-level lib.*:debug

Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 4096
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:8 hugepage_sz:1073741824
EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x100030000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: Creating 4 segment lists: n_segs:32 socket_id:8 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2200000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2240000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2a40000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2a80000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3280000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x32c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3ac0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x3b00000000 (size = 0x800000000)
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 4 is ready (tid=7fffab066c90;cpuset=[4])
EAL: lcore 9 is ready (tid=7fffa917dc90;cpuset=[9])
<snip> ...

EAL: lcore 74 is ready (tid=7ffea9fbdc90;cpuset=[74])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.1 cannot be used
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
testpmd: No probed ethernet devices

Interactive-mode selected
Auto-start selected
Set rxonly packet forwarding mode
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=723456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 2048MB
Done
Start automatic packet forwarding
rxonly packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support enabled, MP allocation mode: native

rxonly packet forwarding packets/burst=32
nb forwarding cores=64 - nb forwarding ports=0
testpmd>

Dave

#7 lore raw
Quoted 7 lines · David Christensen
Quoted 1 line · David Marchand

--log-level lib.*:debug

Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).

Quoted 16 lines

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'

Ok, I think I understand now.

Can you try this hunk ?

drivers/bus/pci/pci_common.c +0 -3
index 9794552..6d3b553 100644
@@ -620,9 +620,6 @@ rte_pci_get_iommu_class(void)
        FOREACH_DEVICE_ON_PCIBUS(dev) {
                if (pci_ignore_device(dev))
                        continue;
-               if (dev->kdrv == RTE_KDRV_UNKNOWN ||
-                   dev->kdrv == RTE_KDRV_NONE)
-                       continue;
                FOREACH_DRIVER_ON_PCIBUS(drv) {
                        enum rte_iova_mode dev_iova_mode;
 
Signature · 1 line
David Marchand
#8 lore raw
Quoted 20 lines
Quoted 19 lines

Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'

Quoted 4 lines

Ok, I think I understand now.

Can you try this hunk ?

IOVA mode is now reported as PA.

MLX5 PMD fails to load correctly, but that also occurs without the patch and when the EAL command-line option "--iova-mode=pa" is used. This worked previously in RC1 and may be a separate issue.

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib /home/davec/src/dpdk/build/app/testpmd --log-level="lib.*:debug" -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: PCI driver net_mlx5 for device 0000:01:00.0 wants IOVA as 'PA'
EAL: PCI driver net_mlx5 for device 0000:01:00.1 wants IOVA as 'PA'
EAL: Bus pci wants IOVA as 'PA'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 4096
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:8 hugepage_sz:1073741824
EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x100030000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: Creating 4 segment lists: n_segs:32 socket_id:8 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2200000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2240000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2a40000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2a80000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3280000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x32c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3ac0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x3b00000000 (size = 0x800000000)
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 4 is ready (tid=7fffabb46c90;cpuset=[4])
EAL: lcore 9 is ready (tid=7fffa9c5dc90;cpuset=[9])
<snip> ...

EAL: lcore 73 is ready (tid=7ffea77edc90;cpuset=[73])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
EAL: Mem event callback 'MLX5_MEM_EVENT_CB:(nil)' registered
net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1019 net_mlx5
net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
testpmd: /home/davec/src/dpdk/drivers/net/mlx5/mlx5_ethdev.c:520: mlx5_set_txlimit_params: Assertion `config->txq_inline_min >= 0' failed.

Aborted

Dave

#9 lore raw
Quoted 21 lines · David Christensen
Quoted 21 lines
Quoted 20 lines

Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).

sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'

Quoted 6 lines
Quoted 4 lines

Ok, I think I understand now.

Can you try this hunk ?

IOVA mode is now reported as PA.

Ok thanks.
Still trying to understand Takeshi Yoshimura issue which looks different.

Quoted 3 lines

MLX5 PMD fails to load correctly, but that also occurs without the patch and when the EAL command-line option "--iova-mode=pa" is used. This worked previously in RC1 and may be a separate issue.

Saw your patch on mlx5, yes separate issue.

Signature · 1 line
David Marchand
#10 lore raw
In reply to Takeshi T Yoshimura
Quoted 43 lines · Takeshi T Yoshimura

-----David Marchand <[email protected]> wrote: -----

Quoted 27 lines

To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Quoted 5 lines · Takeshi Yoshimura

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA

as

Quoted 2 lines

PA in ppc as before because current ppc driver does not support VA mode.

What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?

Quoted 5 lines

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

Quoted 8 lines

Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?

--
David Marchand

Ah, I overlooked your first question. The "ppc driver" is in drivers/bus/pci/linux/pci.c. In pci.c, pci_one_device_iommu_support_va() always returns false in ppc. so, the driver does not allow VA mode in ppc.

Quoted 6 lines

This return value is passed to rte_pci_probe_one_driver() in drivers/bus/pci/pci_common.c. And the error log appeared like:

EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing

Can you confirm on which kernel driver this device was bound ? I'd like to be sure this is the same issue than David.

Signature · 1 line
David Marchand
#11 lore raw

-----David Marchand <[email protected]> wrote: -----

Quoted 63 lines

To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 04:45AM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Quoted 8 lines · Takeshi T Yoshimura

-----David Marchand <[email protected]> wrote: -----

Quoted 5 lines

To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA

in

Quoted 3 lines
Quoted 3 lines

ppc

On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura

<[email protected]>

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines · wrote:

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force

IOVA

Quoted 2 lines
Quoted 2 lines

as

Quoted 1 line

PA in ppc as before because current ppc driver does not support

VA

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

mode.

What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?

Quoted 2 lines

Theoretically, ppc can support VA mode, but I suspect that ppc

with

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change

the

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

code to check device capability in ppc.

Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?

Quoted 7 lines
Quoted 4 lines

--
David Marchand

Ah, I overlooked your first question. The "ppc driver" is in drivers/bus/pci/linux/pci.c. In pci.c,

pci_one_device_iommu_support_va()

Quoted 1 line

always returns false in ppc. so, the driver does not allow VA mode

in ppc.

Quoted 4 lines

This return value is passed to rte_pci_probe_one_driver() in drivers/bus/pci/pci_common.c. And the error log appeared like:

Quoted 13 lines
Quoted 3 lines
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not

initializing

Can you confirm on which kernel driver this device was bound ? I'd like to be sure this is the same issue than David.

Signature · 1 line
David Marchand

Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922. The operating system is Ubuntu 18.04 and Linux 4.17.

Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]

EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=77e7345a44a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used
#12 lore raw

Hello,

Quoted 5 lines · Takeshi T Yoshimura

Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922. The operating system is Ubuntu 18.04 and Linux 4.17.

Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]

We are missing the initial logs here (eal log level is set twice, but this should not be a problem).
Is it truncated by spdk ? Is there a way to get them ?

Thanks.

Signature · 1 line
David Marchand
#13 lore raw

-----David Marchand <[email protected]> wrote: -----

Quoted 21 lines

To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 02:01PM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Hello,

Quoted 1 line · Takeshi T Yoshimura

Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM

Power9 AC922.

Quoted 4 lines

The operating system is Ubuntu 18.04 and Linux 4.17.

Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization... [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0

--log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]

Quoted 10 lines

We are missing the initial logs here (eal log level is set twice, but this should not be a problem).
Is it truncated by spdk ? Is there a way to get them ?

Thanks.

Signature · 1 line
David Marchand

Okay. I should have set the --log-level=lib.*:debug. Here is the log.

Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.*:debug --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid159692 ]

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 4 on socket 0
EAL: Detected lcore 6 as core 4 on socket 0
EAL: Detected lcore 7 as core 4 on socket 0
EAL: Detected lcore 8 as core 8 on socket 0
EAL: Detected lcore 9 as core 8 on socket 0
EAL: Detected lcore 10 as core 8 on socket 0
EAL: Detected lcore 11 as core 8 on socket 0
EAL: Detected lcore 12 as core 12 on socket 0
EAL: Detected lcore 13 as core 12 on socket 0
EAL: Detected lcore 14 as core 12 on socket 0
EAL: Detected lcore 15 as core 12 on socket 0
EAL: Detected lcore 16 as core 24 on socket 0
EAL: Detected lcore 17 as core 24 on socket 0
EAL: Detected lcore 18 as core 24 on socket 0
EAL: Detected lcore 19 as core 24 on socket 0
EAL: Detected lcore 20 as core 28 on socket 0
EAL: Detected lcore 21 as core 28 on socket 0
EAL: Detected lcore 22 as core 28 on socket 0
EAL: Detected lcore 23 as core 28 on socket 0
EAL: Detected lcore 24 as core 32 on socket 0
EAL: Detected lcore 25 as core 32 on socket 0
EAL: Detected lcore 26 as core 32 on socket 0
EAL: Detected lcore 27 as core 32 on socket 0
EAL: Detected lcore 28 as core 36 on socket 0
EAL: Detected lcore 29 as core 36 on socket 0
EAL: Detected lcore 30 as core 36 on socket 0
EAL: Detected lcore 31 as core 36 on socket 0
EAL: Detected lcore 32 as core 48 on socket 0
EAL: Detected lcore 33 as core 48 on socket 0
EAL: Detected lcore 34 as core 48 on socket 0
EAL: Detected lcore 35 as core 48 on socket 0
EAL: Detected lcore 36 as core 52 on socket 0
EAL: Detected lcore 37 as core 52 on socket 0
EAL: Detected lcore 38 as core 52 on socket 0
EAL: Detected lcore 39 as core 52 on socket 0
EAL: Detected lcore 40 as core 56 on socket 0
EAL: Detected lcore 41 as core 56 on socket 0
EAL: Detected lcore 42 as core 56 on socket 0
EAL: Detected lcore 43 as core 56 on socket 0
EAL: Detected lcore 44 as core 60 on socket 0
EAL: Detected lcore 45 as core 60 on socket 0
EAL: Detected lcore 46 as core 60 on socket 0
EAL: Detected lcore 47 as core 60 on socket 0
EAL: Detected lcore 48 as core 64 on socket 0
EAL: Detected lcore 49 as core 64 on socket 0
EAL: Detected lcore 50 as core 64 on socket 0
EAL: Detected lcore 51 as core 64 on socket 0
EAL: Detected lcore 52 as core 68 on socket 0
EAL: Detected lcore 53 as core 68 on socket 0
EAL: Detected lcore 54 as core 68 on socket 0
EAL: Detected lcore 55 as core 68 on socket 0
EAL: Detected lcore 56 as core 72 on socket 0
EAL: Detected lcore 57 as core 72 on socket 0
EAL: Detected lcore 58 as core 72 on socket 0
EAL: Detected lcore 59 as core 72 on socket 0
EAL: Detected lcore 60 as core 76 on socket 0
EAL: Detected lcore 61 as core 76 on socket 0
EAL: Detected lcore 62 as core 76 on socket 0
EAL: Detected lcore 63 as core 76 on socket 0
EAL: Detected lcore 64 as core 80 on socket 0
EAL: Detected lcore 65 as core 80 on socket 0
EAL: Detected lcore 66 as core 80 on socket 0
EAL: Detected lcore 67 as core 80 on socket 0
EAL: Detected lcore 68 as core 84 on socket 0
EAL: Detected lcore 69 as core 84 on socket 0
EAL: Detected lcore 70 as core 84 on socket 0
EAL: Detected lcore 71 as core 84 on socket 0
EAL: Detected lcore 72 as core 88 on socket 0
EAL: Detected lcore 73 as core 88 on socket 0
EAL: Detected lcore 74 as core 88 on socket 0
EAL: Detected lcore 75 as core 88 on socket 0
EAL: Detected lcore 76 as core 92 on socket 0
EAL: Detected lcore 77 as core 92 on socket 0
EAL: Detected lcore 78 as core 92 on socket 0
EAL: Detected lcore 79 as core 92 on socket 0
EAL: Detected lcore 80 as core 2048 on socket 8
EAL: Detected lcore 81 as core 2048 on socket 8
EAL: Detected lcore 82 as core 2048 on socket 8
EAL: Detected lcore 83 as core 2048 on socket 8
EAL: Detected lcore 84 as core 2052 on socket 8
EAL: Detected lcore 85 as core 2052 on socket 8
EAL: Detected lcore 86 as core 2052 on socket 8
EAL: Detected lcore 87 as core 2052 on socket 8
EAL: Detected lcore 88 as core 2056 on socket 8
EAL: Detected lcore 89 as core 2056 on socket 8
EAL: Detected lcore 90 as core 2056 on socket 8
EAL: Detected lcore 91 as core 2056 on socket 8
EAL: Detected lcore 92 as core 2060 on socket 8
EAL: Detected lcore 93 as core 2060 on socket 8
EAL: Detected lcore 94 as core 2060 on socket 8
EAL: Detected lcore 95 as core 2060 on socket 8
EAL: Detected lcore 96 as core 2064 on socket 8
EAL: Detected lcore 97 as core 2064 on socket 8
EAL: Detected lcore 98 as core 2064 on socket 8
EAL: Detected lcore 99 as core 2064 on socket 8
EAL: Detected lcore 100 as core 2068 on socket 8
EAL: Detected lcore 101 as core 2068 on socket 8
EAL: Detected lcore 102 as core 2068 on socket 8
EAL: Detected lcore 103 as core 2068 on socket 8
EAL: Detected lcore 104 as core 2072 on socket 8
EAL: Detected lcore 105 as core 2072 on socket 8
EAL: Detected lcore 106 as core 2072 on socket 8
EAL: Detected lcore 107 as core 2072 on socket 8
EAL: Detected lcore 108 as core 2076 on socket 8
EAL: Detected lcore 109 as core 2076 on socket 8
EAL: Detected lcore 110 as core 2076 on socket 8
EAL: Detected lcore 111 as core 2076 on socket 8
EAL: Detected lcore 112 as core 2080 on socket 8
EAL: Detected lcore 113 as core 2080 on socket 8
EAL: Detected lcore 114 as core 2080 on socket 8
EAL: Detected lcore 115 as core 2080 on socket 8
EAL: Detected lcore 116 as core 2084 on socket 8
EAL: Detected lcore 117 as core 2084 on socket 8
EAL: Detected lcore 118 as core 2084 on socket 8
EAL: Detected lcore 119 as core 2084 on socket 8
EAL: Detected lcore 120 as core 2088 on socket 8
EAL: Detected lcore 121 as core 2088 on socket 8
EAL: Detected lcore 122 as core 2088 on socket 8
EAL: Detected lcore 123 as core 2088 on socket 8
EAL: Detected lcore 124 as core 2092 on socket 8
EAL: Detected lcore 125 as core 2092 on socket 8
EAL: Detected lcore 126 as core 2092 on socket 8
EAL: Detected lcore 127 as core 2092 on socket 8
EAL: Detected lcore 128 as core 2096 on socket 8
EAL: Detected lcore 129 as core 2096 on socket 8
EAL: Detected lcore 130 as core 2096 on socket 8
EAL: Detected lcore 131 as core 2096 on socket 8
EAL: Detected lcore 132 as core 2100 on socket 8
EAL: Detected lcore 133 as core 2100 on socket 8
EAL: Detected lcore 134 as core 2100 on socket 8
EAL: Detected lcore 135 as core 2100 on socket 8
EAL: Detected lcore 136 as core 2104 on socket 8
EAL: Detected lcore 137 as core 2104 on socket 8
EAL: Detected lcore 138 as core 2104 on socket 8
EAL: Detected lcore 139 as core 2104 on socket 8
EAL: Detected lcore 140 as core 2108 on socket 8
EAL: Detected lcore 141 as core 2108 on socket 8
EAL: Detected lcore 142 as core 2108 on socket 8
EAL: Detected lcore 143 as core 2108 on socket 8
EAL: Detected lcore 144 as core 2112 on socket 8
EAL: Detected lcore 145 as core 2112 on socket 8
EAL: Detected lcore 146 as core 2112 on socket 8
EAL: Detected lcore 147 as core 2112 on socket 8
EAL: Detected lcore 148 as core 2116 on socket 8
EAL: Detected lcore 149 as core 2116 on socket 8
EAL: Detected lcore 150 as core 2116 on socket 8
EAL: Detected lcore 151 as core 2116 on socket 8
EAL: Detected lcore 152 as core 2136 on socket 8
EAL: Detected lcore 153 as core 2136 on socket 8
EAL: Detected lcore 154 as core 2136 on socket 8
EAL: Detected lcore 155 as core 2136 on socket 8
EAL: Detected lcore 156 as core 2140 on socket 8
EAL: Detected lcore 157 as core 2140 on socket 8
EAL: Detected lcore 158 as core 2140 on socket 8
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No shared files mode enabled, IPC will be disabled
EAL: No shared files mode enabled, IPC is disabled
EAL: VFIO PCI modules not loaded
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=7430194644a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used

No NVMe controllers found.

#14 lore raw
Quoted 4 lines · Takeshi T Yoshimura

Okay. I should have set the --log-level=lib.*:debug. Here is the log.

Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.*:debug --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid159692 ]

[snip]

Quoted 3 lines

EAL: No shared files mode enabled, IPC will be disabled
EAL: No shared files mode enabled, IPC is disabled
EAL: VFIO PCI modules not loaded

EAL thinks that vfio kmod is not loaded at this point.

Quoted 1 line

EAL: Bus pci wants IOVA as 'DC'

We see no log here for the devices that are used later.

- Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?
- Is the spdk_nvme driver loaded at this point?
- I suppose forcing --iova-mode=pa resolves the issue, but can you confirm this?

Quoted 4 lines

EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: Selected IOVA mode 'VA'

[snip]

Quoted 13 lines
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used

No NVMe controllers found.

Signature · 1 line
David Marchand
#15 lore raw
Quoted 1 line

- Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?

Yes, vfio-pci is bound to 0030:01:00.0 in the log.

Quoted 1 line

- Is the spdk_nvme driver loaded at this point?

No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.

Quoted 1 line

- I suppose forcing --iova-mode=pa resolves the issue, but can you confirm this?

Yes, you are right. I needed to modify the spdk core to do this, anyway.. Here is the log when I specify --iova-mode=pa for an example app of spdk (identify)

Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.*:debug --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid140641 ]

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 4 on socket 0
EAL: Detected lcore 6 as core 4 on socket 0
EAL: Detected lcore 7 as core 4 on socket 0
EAL: Detected lcore 8 as core 8 on socket 0
EAL: Detected lcore 9 as core 8 on socket 0
EAL: Detected lcore 10 as core 8 on socket 0
EAL: Detected lcore 11 as core 8 on socket 0
EAL: Detected lcore 12 as core 12 on socket 0
EAL: Detected lcore 13 as core 12 on socket 0
EAL: Detected lcore 14 as core 12 on socket 0
EAL: Detected lcore 15 as core 12 on socket 0
EAL: Detected lcore 16 as core 24 on socket 0
EAL: Detected lcore 17 as core 24 on socket 0
EAL: Detected lcore 18 as core 24 on socket 0
EAL: Detected lcore 19 as core 24 on socket 0
EAL: Detected lcore 20 as core 28 on socket 0
EAL: Detected lcore 21 as core 28 on socket 0
EAL: Detected lcore 22 as core 28 on socket 0
EAL: Detected lcore 23 as core 28 on socket 0
EAL: Detected lcore 24 as core 32 on socket 0
EAL: Detected lcore 25 as core 32 on socket 0
EAL: Detected lcore 26 as core 32 on socket 0
EAL: Detected lcore 27 as core 32 on socket 0
EAL: Detected lcore 28 as core 36 on socket 0
EAL: Detected lcore 29 as core 36 on socket 0
EAL: Detected lcore 30 as core 36 on socket 0
EAL: Detected lcore 31 as core 36 on socket 0
EAL: Detected lcore 32 as core 48 on socket 0
EAL: Detected lcore 33 as core 48 on socket 0
EAL: Detected lcore 34 as core 48 on socket 0
EAL: Detected lcore 35 as core 48 on socket 0
EAL: Detected lcore 36 as core 52 on socket 0
EAL: Detected lcore 37 as core 52 on socket 0
EAL: Detected lcore 38 as core 52 on socket 0
EAL: Detected lcore 39 as core 52 on socket 0
EAL: Detected lcore 40 as core 56 on socket 0
EAL: Detected lcore 41 as core 56 on socket 0
EAL: Detected lcore 42 as core 56 on socket 0
EAL: Detected lcore 43 as core 56 on socket 0
EAL: Detected lcore 44 as core 60 on socket 0
EAL: Detected lcore 45 as core 60 on socket 0
EAL: Detected lcore 46 as core 60 on socket 0
EAL: Detected lcore 47 as core 60 on socket 0
EAL: Detected lcore 48 as core 64 on socket 0
EAL: Detected lcore 49 as core 64 on socket 0
EAL: Detected lcore 50 as core 64 on socket 0
EAL: Detected lcore 51 as core 64 on socket 0
EAL: Detected lcore 52 as core 68 on socket 0
EAL: Detected lcore 53 as core 68 on socket 0
EAL: Detected lcore 54 as core 68 on socket 0
EAL: Detected lcore 55 as core 68 on socket 0
EAL: Detected lcore 56 as core 72 on socket 0
EAL: Detected lcore 57 as core 72 on socket 0
EAL: Detected lcore 58 as core 72 on socket 0
EAL: Detected lcore 59 as core 72 on socket 0
EAL: Detected lcore 60 as core 76 on socket 0
EAL: Detected lcore 61 as core 76 on socket 0
EAL: Detected lcore 62 as core 76 on socket 0
EAL: Detected lcore 63 as core 76 on socket 0
EAL: Detected lcore 64 as core 80 on socket 0
EAL: Detected lcore 65 as core 80 on socket 0
EAL: Detected lcore 66 as core 80 on socket 0
EAL: Detected lcore 67 as core 80 on socket 0
EAL: Detected lcore 68 as core 84 on socket 0
EAL: Detected lcore 69 as core 84 on socket 0
EAL: Detected lcore 70 as core 84 on socket 0
EAL: Detected lcore 71 as core 84 on socket 0
EAL: Detected lcore 72 as core 88 on socket 0
EAL: Detected lcore 73 as core 88 on socket 0
EAL: Detected lcore 74 as core 88 on socket 0
EAL: Detected lcore 75 as core 88 on socket 0
EAL: Detected lcore 76 as core 92 on socket 0
EAL: Detected lcore 77 as core 92 on socket 0
EAL: Detected lcore 78 as core 92 on socket 0
EAL: Detected lcore 79 as core 92 on socket 0
EAL: Detected lcore 80 as core 2048 on socket 8
EAL: Detected lcore 81 as core 2048 on socket 8
EAL: Detected lcore 82 as core 2048 on socket 8
EAL: Detected lcore 83 as core 2048 on socket 8
EAL: Detected lcore 84 as core 2052 on socket 8
EAL: Detected lcore 85 as core 2052 on socket 8
EAL: Detected lcore 86 as core 2052 on socket 8
EAL: Detected lcore 87 as core 2052 on socket 8
EAL: Detected lcore 88 as core 2056 on socket 8
EAL: Detected lcore 89 as core 2056 on socket 8
EAL: Detected lcore 90 as core 2056 on socket 8
EAL: Detected lcore 91 as core 2056 on socket 8
EAL: Detected lcore 92 as core 2060 on socket 8
EAL: Detected lcore 93 as core 2060 on socket 8
EAL: Detected lcore 94 as core 2060 on socket 8
EAL: Detected lcore 95 as core 2060 on socket 8
EAL: Detected lcore 96 as core 2064 on socket 8
EAL: Detected lcore 97 as core 2064 on socket 8
EAL: Detected lcore 98 as core 2064 on socket 8
EAL: Detected lcore 99 as core 2064 on socket 8
EAL: Detected lcore 100 as core 2068 on socket 8
EAL: Detected lcore 101 as core 2068 on socket 8
EAL: Detected lcore 102 as core 2068 on socket 8
EAL: Detected lcore 103 as core 2068 on socket 8
EAL: Detected lcore 104 as core 2072 on socket 8
EAL: Detected lcore 105 as core 2072 on socket 8
EAL: Detected lcore 106 as core 2072 on socket 8
EAL: Detected lcore 107 as core 2072 on socket 8
EAL: Detected lcore 108 as core 2076 on socket 8
EAL: Detected lcore 109 as core 2076 on socket 8
EAL: Detected lcore 110 as core 2076 on socket 8
EAL: Detected lcore 111 as core 2076 on socket 8
EAL: Detected lcore 112 as core 2080 on socket 8
EAL: Detected lcore 113 as core 2080 on socket 8
EAL: Detected lcore 114 as core 2080 on socket 8
EAL: Detected lcore 115 as core 2080 on socket 8
EAL: Detected lcore 116 as core 2084 on socket 8
EAL: Detected lcore 117 as core 2084 on socket 8
EAL: Detected lcore 118 as core 2084 on socket 8
EAL: Detected lcore 119 as core 2084 on socket 8
EAL: Detected lcore 120 as core 2088 on socket 8
EAL: Detected lcore 121 as core 2088 on socket 8
EAL: Detected lcore 122 as core 2088 on socket 8
EAL: Detected lcore 123 as core 2088 on socket 8
EAL: Detected lcore 124 as core 2092 on socket 8
EAL: Detected lcore 125 as core 2092 on socket 8
EAL: Detected lcore 126 as core 2092 on socket 8
EAL: Detected lcore 127 as core 2092 on socket 8
EAL: Detected lcore 128 as core 2096 on socket 8
EAL: Detected lcore 129 as core 2096 on socket 8
EAL: Detected lcore 130 as core 2096 on socket 8
EAL: Detected lcore 131 as core 2096 on socket 8
EAL: Detected lcore 132 as core 2100 on socket 8
EAL: Detected lcore 133 as core 2100 on socket 8
EAL: Detected lcore 134 as core 2100 on socket 8
EAL: Detected lcore 135 as core 2100 on socket 8
EAL: Detected lcore 136 as core 2104 on socket 8
EAL: Detected lcore 137 as core 2104 on socket 8
EAL: Detected lcore 138 as core 2104 on socket 8
EAL: Detected lcore 139 as core 2104 on socket 8
EAL: Detected lcore 140 as core 2108 on socket 8
EAL: Detected lcore 141 as core 2108 on socket 8
EAL: Detected lcore 142 as core 2108 on socket 8
EAL: Detected lcore 143 as core 2108 on socket 8
EAL: Detected lcore 144 as core 2112 on socket 8
EAL: Detected lcore 145 as core 2112 on socket 8
EAL: Detected lcore 146 as core 2112 on socket 8
EAL: Detected lcore 147 as core 2112 on socket 8
EAL: Detected lcore 148 as core 2116 on socket 8
EAL: Detected lcore 149 as core 2116 on socket 8
EAL: Detected lcore 150 as core 2116 on socket 8
EAL: Detected lcore 151 as core 2116 on socket 8
EAL: Detected lcore 152 as core 2136 on socket 8
EAL: Detected lcore 153 as core 2136 on socket 8
EAL: Detected lcore 154 as core 2136 on socket 8
EAL: Detected lcore 155 as core 2136 on socket 8
EAL: Detected lcore 156 as core 2140 on socket 8
EAL: Detected lcore 157 as core 2140 on socket 8
EAL: Detected lcore 158 as core 2140 on socket 8
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No shared files mode enabled, IPC will be disabled
EAL: No shared files mode enabled, IPC is disabled
EAL: VFIO PCI modules not loaded
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=7d49a88844a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   set IOMMU type 1 (Type 1) failed, error 19 (No such device)
EAL:   using IOMMU type 7 (sPAPR)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
EAL: VFIO reports MSI-X BAR as mappable
EAL:   PCI memory mapped at 0x202001000000
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 4MB
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 4MB

=====================================================
NVMe Controller at 0030:01:00.0 [144d:a822]
=====================================================
Controller Capabilities/Features
================================

Vendor ID:                             144d
Subsystem Vendor ID:                   1014
Serial Number:                         S3RXNA0K300102
Model Number:                          PCIe3 6.4TB NVMe Flash Adapter II x8
Firmware Version:                      MN12MN12
Recommended Arb Burst:                 8
IEEE OUI Identifier:                   38 25 00
Multi-path I/O
  May have multiple subsystem ports:   Yes
  May be connected to multiple hosts:  Yes
  Associated with SR-IOV VF:           No
Max Data Transfer Size:                1048576
Error Recovery Timeout:                Unlimited
NVMe Specification Version (VS):       1.2.1
NVMe Specification Version (Identify): 1.2.1
Maximum Queue Entries:                 1024
Contiguous Queues Required:            Yes
Arbitration Mechanisms Supported
  Weighted Round Robin:                Not Supported
  Vendor Specific:                     Not Supported
Reset Timeout:                         60000 ms
Doorbell Stride:                       4 bytes
NVM Subsystem Reset:                   Supported
Command Sets Supported
  NVM Command Set:                     Supported
Boot Partition:                        Not Supported
Memory Page Size Minimum:              4096 bytes
Memory Page Size Maximum:              4096 bytes
Optional Asynchronous Events Supported
  Namespace Attribute Notices:         Supported
  Firmware Activation Notices:         Supported
128-bit Host Identifier:               Not Supported

Controller Memory Buffer Support
================================
Supported: No

Admin Command Set Attributes
============================

Security Send/Receive:                 Not Supported
Format NVM:                            Supported
Firmware Activate/Download:            Supported
Namespace Management:                  Supported
Device Self-Test:                      Not Supported
Directives:                            Not Supported
NVMe-MI:                               Not Supported
Virtualization Management:             Not Supported
Doorbell Buffer Config:                Not Supported
Abort Command Limit:                   8
Async Event Request Limit:             16
Number of Firmware Slots:              3
Firmware Slot 1 Read-Only:             No
Firmware Update Granularity:           No Information Provided
Per-Namespace SMART Log:               No
Command Effects Log Page:              Supported
Get Log Page Extended Data:            Not Supported
Telemetry Log Pages:                   Not Supported
Error Log Page Entries Supported:      256
Keep Alive:                            Not Supported

NVM Command Set Attributes
==========================
Submission Queue Entry Size
Max: 64
Min: 64
Completion Queue Entry Size

  Max:                       16
  Min:                       16
Number of Namespaces:        32
Compare Command:             Not Supported
Write Uncorrectable Command: Supported
Dataset Management Command:  Supported
Write Zeroes Command:        Supported
Set Features Save Field:     Supported
Reservations:                Supported
Timestamp:                   Not Supported
Volatile Write Cache:        Not Present
Atomic Write Unit (Normal):  65536
Atomic Write Unit (PFail):   1

Atomic Compare & Write Unit: 1
Scatter-Gather List
SGL Command Set: Supported
SGL Keyed: Not Supported
SGL Bit Bucket Descriptor: Supported

  SGL Metadata Pointer:      Supported
  Oversized SGL:             Supported
  SGL Metadata Address:      Not Supported
  SGL Offset:                Not Supported
  Transport SGL Data Block:  Not Supported

Replay Protected Memory Block: Not Supported

Firmware Slot Information
=========================

Active slot:                 1
Next controller reset slot:  1
Slot 1 Firmware Revision:    MN12MN12
Slot 2 Firmware Revision:    MN12MN12
Slot 3 Firmware Revision:    MN12MN12

Commands Supported and Effects
==============================
Admin Commands
--------------
Delete I/O Submission Queue (00h): Supported
Create I/O Submission Queue (01h): Supported
Get Log Page (02h): Supported
Delete I/O Completion Queue (04h): Supported
Create I/O Completion Queue (05h): Supported
Identify (06h): Supported
Abort (08h): Supported
Set Features (09h): Supported
Get Features (0Ah): Supported
Asynchronous Event Request (0Ch): Supported
Namespace Management (0Dh): Supported NS-Cap-Change NS-Inventory-Change All-NS-Exclusive
Firmware Commit (10h): Supported
Firmware Image Download (11h): Supported
Namespace Attachment (15h): Supported All-NS-Exclusive
Format NVM (80h): Supported LBA-Change NS-Cap-Change Per-NS-Exclusive
I/O Commands
------------
Flush (00h): Supported
Write (01h): Supported LBA-Change
Read (02h): Supported
Write Uncorrectable (04h): Supported LBA-Change
Write Zeroes (08h): Supported LBA-Change
Dataset Management (09h): Supported LBA-Change
Reservation Register (0Dh): Supported
Reservation Report (0Eh): Supported
Reservation Acquire (11h): Supported
Reservation Release (15h): Supported

Error Log
=========

Entry: 0
Error Count:            0x2
Submission Queue Id:    0x0
Command Id:             0x1a
Phase Bit:              0
Status Code:            0x1c
Status Code Type:       0x1
Do Not Retry:           0
Error Location:         0x0
LBA:                    0x0
Namespace:              0x1
Vendor Log Page:        0x0
-----------
Entry: 1
Error Count:            0x1
Submission Queue Id:    0x0
Command Id:             0x5
Phase Bit:              0
Status Code:            0x1c
Status Code Type:       0x1
Do Not Retry:           0
Error Location:         0x0
LBA:                    0x0
Namespace:              0x1
Vendor Log Page:        0x0

Power Management
================

Number of Power States:      1
Current Power State:         Power State #0

Power State #0: Max Power: 25.00 W
Non-Operational Permissive Mode: Not Supported

Health Information
==================
Critical Warnings:

  Available Spare Space:     OK
  Temperature:               OK
  Device Reliability:        OK
  Read Only:                 No
  Volatile Memory Backup:    OK
Current Temperature:         319 Kelvin (46 Celsius)
Temperature Threshold:       353 Kelvin (80 Celsius)
Available Spare:             100%
Available Spare Threshold:   10%
Life Percentage Used:        0%
Data Units Read:             44913828
Data Units Written:          67060524
Host Read Commands:          1363758490
Host Write Commands:         2804327733
Controller Busy Time:        715 minutes
Power Cycles:                104
Power On Hours:              6530 hours
Unsafe Shutdowns:            41
Unrecoverable Media Errors:  0
Lifetime Error Log Entries:  2
Warning Temperature Time:    0 minutes
Critical Temperature Time:   0 minutes
Temperature Sensor 1:        319 Kelvin (46 Celsius)
Temperature Sensor 2:        316 Kelvin (43 Celsius)
Temperature Sensor 3:        316 Kelvin (43 Celsius)

Number of Queues
================

Number of I/O Submission Queues:      128
Number of I/O Completion Queues:      128

Namespace ID:1

Deallocate:                            Supported
Deallocated/Unwritten Error:           Not Supported
Deallocated Read Value:                Unknown
Deallocate in Write Zeroes:            Not Supported
Deallocated Guard Field:               0xFFFF
Flush:                                 Not Supported
Reservation:                           Supported
Namespace Sharing Capabilities:        Private
Size (in LBAs):                        1562805846 (1490M)
Capacity (in LBAs):                    1562805846 (1490M)
Utilization (in LBAs):                 1562805846 (1490M)
NGUID:                                 335258304B3001020025384100000001
Thin Provisioning:                     Not Supported
Per-NS Atomic Units:                   No
NGUID/EUI64 Never Reused:              No
Number of LBA Formats:                 4
Current LBA Format:                    LBA Format #00
LBA Format #00: Data Size:  4096  Metadata Size:     0
LBA Format #01: Data Size:  4096  Metadata Size:     8
LBA Format #02: Data Size:   512  Metadata Size:     0
LBA Format #03: Data Size:   512  Metadata Size:     8
EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was shrunk by 4MB
EAL: Calling mem event callback 'spdk:(nil)'
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was shrunk by 4MB
EAL: request: eal_dev_mp_request
EAL: No shared files mode enabled, IPC is disabled
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   remove driver: 144d:a822 spdk_nvme
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered
EAL: Releasing pci mapped resource for 0030:01:00.0
EAL: Calling pci_unmap_resource for 0030:01:00.0 at 0x202001000000
EAL:   PCI memory unmapped at 0x202001000000
#16 lore raw
Quoted 9 lines · Takeshi T Yoshimura
Quoted 2 lines

- Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?

Yes, vfio-pci is bound to 0030:01:00.0 in the log.

Quoted 1 line

- Is the spdk_nvme driver loaded at this point?

No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.

Do you mean that spdk registers its drivers after rte_eal_init() has returned?

Nothing should prevent from registering the drivers them before calling rte_eal_init(), did you try this?
If the drivers are in the form of shared libraries (.so files), did you try to load them using the -d eal option?

Signature · 1 line
David Marchand
#17 lore raw

-----David Marchand <[email protected]> wrote: -----

Quoted 20 lines

To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 06:47PM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Quoted 2 lines · Takeshi T Yoshimura
Quoted 2 lines

- Are the (below) devices bound to vfio-pci or any other kmod at

this point of the init?

Quoted 7 lines

Yes, vfio-pci is bound to 0030:01:00.0 in the log.

Quoted 1 line

- Is the spdk_nvme driver loaded at this point?

No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.

Quoted 13 lines

Do you mean that spdk registers its drivers after rte_eal_init() has returned?

Nothing should prevent from registering the drivers them before calling rte_eal_init(), did you try this?
If the drivers are in the form of shared libraries (.so files), did you try to load them using the -d eal option?

Signature · 1 line
David Marchand

SPDK apps have to call rte_eal_init() before registering drivers,
since the driver allocates DMA memory.
Do I have to change existing apps and spdk core to workaround this issue to use new versions of dpdk?

#18 lore raw
Quoted 32 lines · Takeshi T Yoshimura

-----David Marchand <[email protected]> wrote: -----

Quoted 29 lines

To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 06:47PM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Quoted 2 lines · Takeshi T Yoshimura
Quoted 2 lines

- Are the (below) devices bound to vfio-pci or any other kmod at

this point of the init?

Quoted 7 lines

Yes, vfio-pci is bound to 0030:01:00.0 in the log.

Quoted 1 line

- Is the spdk_nvme driver loaded at this point?

No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.

Do you mean that spdk registers its drivers after rte_eal_init() has returned?

Nothing should prevent from registering the drivers them before calling rte_eal_init(), did you try this?
If the drivers are in the form of shared libraries (.so files), did you try to load them using the -d eal option?

Quoted 7 lines
Quoted 4 lines

--
David Marchand

SPDK apps have to call rte_eal_init() before registering drivers, since the driver allocates DMA memory.

Registering a driver before calling rte_eal_init is something that works most of the time.
Not sure I get your point here.

Quoted 1 line

Do I have to change existing apps and spdk core to workaround this issue to use new versions of dpdk?

I understand this is a regression and I am looking at it. If we are too short for 19.08, then we can go with your workaround, but it needs a better commitlog.

Signature · 1 line
David Marchand
#19 lore raw
In reply to David Marchand

-----David Marchand <[email protected]> wrote: -----

Quoted 63 lines

To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 04:45AM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc

Quoted 8 lines · Takeshi T Yoshimura

-----David Marchand <[email protected]> wrote: -----

Quoted 5 lines

To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA

in

Quoted 3 lines
Quoted 3 lines

ppc

On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura

<[email protected]>

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines · wrote:

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force

IOVA

Quoted 2 lines
Quoted 2 lines

as

Quoted 1 line

PA in ppc as before because current ppc driver does not support

VA

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

mode.

What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?

Quoted 2 lines

Theoretically, ppc can support VA mode, but I suspect that ppc

with

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change

the

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

code to check device capability in ppc.

Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?

Quoted 7 lines
Quoted 4 lines

--
David Marchand

Ah, I overlooked your first question. The "ppc driver" is in drivers/bus/pci/linux/pci.c. In pci.c,

pci_one_device_iommu_support_va()

Quoted 1 line

always returns false in ppc. so, the driver does not allow VA mode

in ppc.

Quoted 4 lines

This return value is passed to rte_pci_probe_one_driver() in drivers/bus/pci/pci_common.c. And the error log appeared like:

Quoted 13 lines
Quoted 3 lines
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not

initializing

Can you confirm on which kernel driver this device was bound ? I'd like to be sure this is the same issue than David.

Signature · 1 line
David Marchand

Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922. The operating system is Ubuntu 18.04 and Linux 4.17.

Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]

EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 7 (sPAPR) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x200000000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 1048576
EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
EAL: Detected memory type: socket_id:8 hugepage_sz:2097152
EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200000030000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200000200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200400200000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200400400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200800400000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200800600000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x200c00600000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x200c00800000 (size = 0x400000000)
EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201000800000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201000a00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201400a00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201400c00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201800c00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201800e00000 (size = 0x400000000)
EAL: Ask a virtual area of 0x70000 bytes
EAL: Virtual area found at 0x201c00e00000 (size = 0x70000)
EAL: Memseg list allocated: 0x800kB at socket 8
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x201c01000000 (size = 0x400000000)
EAL: Hugepages will be freed exactly as allocated.
EAL: No shared files mode enabled, IPC is disabled
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 0 is ready (tid=77e7345a44a0;cpuset=[0])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: No shared files mode enabled, IPC is disabled
EAL: Heap on socket 0 was expanded by 2MB
EAL: No shared files mode enabled, IPC is disabled
EAL: Mem event callback 'spdk:(nil)' registered
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0000:01:00.0 cannot be used
EAL: PCI device 0003:01:00.0 on NUMA socket 0
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0003:01:00.0 cannot be used
EAL: PCI device 0030:01:00.0 on NUMA socket 8
EAL:   probe driver: 144d:a822 spdk_nvme
EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
EAL: Requested device 0030:01:00.0 cannot be used
#20 lore raw
In reply to Takeshi Yoshimura

Hello Anatoly,

Quoted 36 lines · Takeshi Yoshimura

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

---
 lib/librte_eal/linux/eal/eal.c | 6 ++++++
 1 file changed, 6 insertions(+)
lib/librte_eal/linux/eal/eal.c +6 -1
index 946222ccd..db2dec922 100644
@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
                                RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
                        }
                }
+#endif
+#ifdef RTE_ARCH_PPC_64
+               if (iova_mode == RTE_IOVA_VA) {
+                       iova_mode = RTE_IOVA_PA;
+                       RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
+               }
 #endif
                rte_eal_get_configuration()->iova_mode = iova_mode;
        } else {
--
Quoted 2 lines

2.17.1

Afaiu, this problem is linked to a specific iommu implementation. Does it make sense to introduce a check "iommu_support_va" in eal, a bit like what we have in the pci bus ?

Signature · 1 line
David Marchand
#21 lore raw
Quoted 37 lines · David Marchand

Hello Anatoly,

Quoted 34 lines · Takeshi Yoshimura

Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.

Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.

---
  lib/librte_eal/linux/eal/eal.c | 6 ++++++
  1 file changed, 6 insertions(+)
lib/librte_eal/linux/eal/eal.c +6 -0
index 946222ccd..db2dec922 100644
@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv)
                                 RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
                         }
                 }
+#endif
+#ifdef RTE_ARCH_PPC_64
+               if (iova_mode == RTE_IOVA_VA) {
+                       iova_mode = RTE_IOVA_PA;
+                       RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because PPC uses PA mode.\n");
+               }
  #endif
                 rte_eal_get_configuration()->iova_mode = iova_mode;
Quoted 9 lines
Quoted 4 lines
lib/librte_eal/linux/eal/eal.c +0 -1
         } else {
--

2.17.1

Afaiu, this problem is linked to a specific iommu implementation. Does it make sense to introduce a check "iommu_support_va" in eal, a bit like what we have in the pci bus ?

It probably does. Some platforms simply don't support VA mode (FreeBSD, for one), so that would be useful instead of hardcoding IOVA as PA/VA as defaults.

Signature · 2 lines
Thanks,
Anatoly