eal: forcing IOVA as PA in ppc
PATCH 1/1 eal: forcing IOVA as PA in ppc
Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (or RTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.
Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
--- lib/librte_eal/linux/eal/eal.c | 6 ++++++ 1 file changed, 6 insertions(+)
lib/librte_eal/linux/eal/eal.c +6 -0
@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv) RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n"); } } +#endif +#ifdef RTE_ARCH_PPC_64 +if (iova_mode ==RTE_IOVA_VA ) { + iova_mode =RTE_IOVA_PA ; +RTE_LOG (WARNING ,EAL ,"Forcing IOVA as 'PA' because PPC uses PA mode.\n" ); + }#endif rte_eal_get_configuration ()->iova_mode = iova_mode; }else {
Signature · 1 line
2.17.1
Quoted 22 lines · Takeshi Yoshimura
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
Signed-off-by Takeshi Yoshimura <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 6 ++++++ 1 file changed, 6 insertions(+)lib/librte_eal/linux/eal/eal.c +0
-0
lib/librte_eal/linux/eal/eal.c rte_eal_init(int argc, char **argv) · line 1130
Quoted 10 lines
lib/librte_eal/linux/eal/eal.c +6
-0@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv) RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n"); } } +#endif +#ifdef RTE_ARCH_PPC_64 +if (iova_mode ==RTE_IOVA_VA ) { + iova_mode =RTE_IOVA_PA ; +RTE_LOG (WARNING ,EAL ,"Forcing IOVA as 'PA' because PPC uses PA mode.\n" ); + }#endif
I would have preferred a space between #endif and a new #ifdef, but otherwise
Signature · 2 lines
Thanks, Anatoly
Quoted 7 lines · Takeshi Yoshimura
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.
What is the "ppc driver" you are referring to?
Do you mean the vfio implementation for ppc?
Quoted 5 lines
Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?
Signature · 1 line
David Marchand
-----David Marchand <[email protected]> wrote: -----
Quoted 22 lines
To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcQuoted 5 lines · Takeshi Yoshimura
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVAas
Quoted 2 lines
PA in ppc as before because current ppc driver does not support VA mode.
What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?
Quoted 1 line
Quoted 13 lines
Quoted 4 lines
Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?Signature · 1 line
David Marchand
Current code does not implement VA mode for ppc. So, yes. VA mode is not working under current dpdk on ppc.
-----David Marchand <[email protected]> wrote: -----
Quoted 22 lines
To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcQuoted 5 lines · Takeshi Yoshimura
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVAas
Quoted 2 lines
PA in ppc as before because current ppc driver does not support VA mode.
What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?
Quoted 1 line
Quoted 13 lines
Quoted 4 lines
Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?Signature · 1 line
David Marchand
Ah, I overlooked your first question. The "ppc driver" is in drivers/bus/pci/linux/pci.c. In pci.c, pci_one_device_iommu_support_va() always returns false in ppc. so, the driver does not allow VA mode in ppc.
This return value is passed to rte_pci_probe_one_driver() in drivers/bus/pci/pci_common.c. And the error log appeared like:
EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
Quoted 11 lines · Takeshi T Yoshimura
Ah, I overlooked your first question. The "ppc driver" is in
drivers/bus/pci/linux/pci.c. Inpci.c,pci_one_device_iommu_support_va()always returns false in ppc. so, the driver does not allow VA mode in ppc.This return value is passed to
rte_pci_probe_one_driver()indrivers/bus/pci/pci_common.c. And the error log appeared like:EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
In which situation does this error happen?
When probing the device at eal init time? or later when hotplugging? Can you give the full eal logs with --log-level lib.*:debug?
Signature · 1 line
David Marchand
Quoted 1 line · David Marchand
--log-level lib.*:debug
Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).
sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib /home/davec/src/dpdk/build/app/testpmd --log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 4096
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:8 hugepage_sz:1073741824
EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x100030000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: Creating 4 segment lists: n_segs:32 socket_id:8 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2200000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2240000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2a40000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2a80000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3280000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x32c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3ac0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x3b00000000 (size = 0x800000000)
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 4 is ready (tid=7fffab066c90;cpuset=[4])
EAL: lcore 9 is ready (tid=7fffa917dc90;cpuset=[9])
<snip> ...
EAL: lcore 74 is ready (tid=7ffea9fbdc90;cpuset=[74]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: Heap on socket 0 was expanded by 1024MB EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 15b3:1019 net_mlx5 EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:01:00.0 cannot be used EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 15b3:1019 net_mlx5 EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:01:00.1 cannot be used EAL: Module /sys/module/vfio not found! error 2 (No such file or directory) testpmd: No probed ethernet devices
Interactive-mode selected
Auto-start selected
Set rxonly packet forwarding mode
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=723456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 2048MB
Done
Start automatic packet forwarding
rxonly packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support enabled, MP allocation mode: native
rxonly packet forwarding packets/burst=32
nb forwarding cores=64 - nb forwarding ports=0
testpmd>
Dave
Quoted 7 lines · David Christensen
Quoted 1 line · David Marchand
--log-level lib.*:debug
Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).
Quoted 16 lines
sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
/home/davec/src/dpdk/build/app/testpmd--log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket/var/run/dpdk/rte/mp_socket
EAL: Module/sys/module/vfio_pcinot found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'
Ok, I think I understand now.
Can you try this hunk ?
drivers/bus/pci/pci_common.c +0 -3
@@ -620,9 +620,6 @@ rte_pci_get_iommu_class(void)FOREACH_DEVICE_ON_PCIBUS (dev) {if (pci_ignore_device (dev))continue ; -if (dev->kdrv ==RTE_KDRV_UNKNOWN || - dev->kdrv ==RTE_KDRV_NONE ) -continue ;FOREACH_DRIVER_ON_PCIBUS (drv) {enum rte_iova_mode dev_iova_mode;
Signature · 1 line
David Marchand
Quoted 20 lines
Quoted 19 lines
Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).
sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
/home/davec/src/dpdk/build/app/testpmd--log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket/var/run/dpdk/rte/mp_socket
EAL: Module/sys/module/vfio_pcinot found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'
Quoted 4 lines
Ok, I think I understand now.
Can you try this hunk ?
IOVA mode is now reported as PA.
MLX5 PMD fails to load correctly, but that also occurs without the patch and when the EAL command-line option "--iova-mode=pa" is used. This worked previously in RC1 and may be a separate issue.
sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib /home/davec/src/dpdk/build/app/testpmd --log-level="lib.*:debug" -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: PCI driver net_mlx5 for device 0000:01:00.0 wants IOVA as 'PA'
EAL: PCI driver net_mlx5 for device 0000:01:00.1 wants IOVA as 'PA'
EAL: Bus pci wants IOVA as 'PA'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Ask a virtual area of 0x30000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x30000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 4096
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:8 hugepage_sz:1073741824
EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x100030000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: Creating 4 segment lists: n_segs:32 socket_id:8 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2200000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2240000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x2a40000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x2a80000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3280000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x32c0000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x10000 bytes
EAL: Virtual area found at 0x3ac0000000 (size = 0x10000)
EAL: Memseg list allocated: 0x100000kB at socket 8
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x3b00000000 (size = 0x800000000)
EAL: TSC frequency is ~510000 KHz
EAL: Master lcore 4 is ready (tid=7fffabb46c90;cpuset=[4])
EAL: lcore 9 is ready (tid=7fffa9c5dc90;cpuset=[9])
<snip> ...
EAL: lcore 73 is ready (tid=7ffea77edc90;cpuset=[73]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: Heap on socket 0 was expanded by 1024MB EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 15b3:1019 net_mlx5 EAL: Mem event callback 'MLX5_MEM_EVENT_CB:(nil)' registered net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration EAL: PCI device 0000:01:00.1 on NUMA socket 0 EAL: probe driver: 15b3:1019 net_mlx5 net_mlx5: mlx5.c:1683: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration EAL: Module /sys/module/vfio not found! error 2 (No such file or directory) testpmd: /home/davec/src/dpdk/drivers/net/mlx5/mlx5_ethdev.c:520: mlx5_set_txlimit_params: Assertion `config->txq_inline_min >= 0' failed.
Aborted
Dave
Quoted 21 lines · David Christensen
Quoted 21 lines
Quoted 20 lines
Here's the output from my Power 9 system running 19.08-RC2 (I'm not using VFIO since I have an MLX5 adapter).
sudo -E LD_LIBRARY_PATH=/home/davec/src/dpdk/build/lib
/home/davec/src/dpdk/build/app/testpmd--log-level lib.\*:debug -w 0000:01:00.0 -w 0000:01:00.1 -l 4,8-79 -n 4 -- --rxq=32 --rxd=1024 --txq=32 --txd=4096 --nb-cores=64 -i -a --numa --forward-mode=rxonly
EAL: Detected lcore 0 as core 0 on socket 0
<snip> ...
EAL: Detected lcore 159 as core 2140 on socket 8
EAL: Support maximum 1536 logical core(s) by configuration.
EAL: Detected 160 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket/var/run/dpdk/rte/mp_socket
EAL: Module/sys/module/vfio_pcinot found! error 2 (No such file or directory)
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Bus pci wants IOVA as 'DC'
Quoted 6 lines
Quoted 4 lines
Ok, I think I understand now.
Can you try this hunk ?
IOVA mode is now reported as PA.
Ok thanks.
Still trying to understand Takeshi Yoshimura issue which looks different.
Quoted 3 lines
MLX5 PMD fails to load correctly, but that also occurs without the patch and when the EAL command-line option "--iova-mode=pa" is used. This worked previously in RC1 and may be a separate issue.
Saw your patch on mlx5, yes separate issue.
Signature · 1 line
David Marchand
Quoted 43 lines · Takeshi T Yoshimura
-----David Marchand <[email protected]> wrote: -----
Quoted 27 lines
To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcQuoted 5 lines · Takeshi Yoshimura
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVAas
Quoted 2 lines
PA in ppc as before because current ppc driver does not support VA mode.
What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?
Quoted 5 lines
Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
Quoted 8 lines
Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?--
David MarchandAh, I overlooked your first question. The "ppc driver" is in
drivers/bus/pci/linux/pci.c. Inpci.c,pci_one_device_iommu_support_va()always returns false in ppc. so, the driver does not allow VA mode in ppc.
Quoted 6 lines
This return value is passed to
rte_pci_probe_one_driver()indrivers/bus/pci/pci_common.c. And the error log appeared like:EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
Can you confirm on which kernel driver this device was bound ? I'd like to be sure this is the same issue than David.
Signature · 1 line
David Marchand
-----David Marchand <[email protected]> wrote: -----
Quoted 63 lines
To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 04:45AM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcQuoted 8 lines · Takeshi T Yoshimura
-----David Marchand <[email protected]> wrote: -----
Quoted 5 lines
To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PAin
Quoted 3 lines
Quoted 3 lines
ppc
On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines · wrote:
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to forceIOVA
Quoted 2 lines
Quoted 2 lines
as
Quoted 1 line
PA in ppc as before because current ppc driver does not support
VA
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
mode.
What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?
Quoted 2 lines
Theoretically, ppc can support VA mode, but I suspect that ppc
with
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change
the
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
code to check device capability in ppc.
Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?Quoted 7 lines
Quoted 4 lines
--
David MarchandAh, I overlooked your first question. The "ppc driver" is in
drivers/bus/pci/linux/pci.c. Inpci.c,
pci_one_device_iommu_support_va()Quoted 1 line
always returns false in ppc. so, the driver does not allow VA mode
in ppc.
Quoted 4 lines
This return value is passed to
rte_pci_probe_one_driver()indrivers/bus/pci/pci_common.c. And the error log appeared like:
Quoted 13 lines
Quoted 3 lines
EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', notinitializing
Can you confirm on which kernel driver this device was bound ? I'd like to be sure this is the same issue than David.
Signature · 1 line
David Marchand
Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922. The operating system is Ubuntu 18.04 and Linux 4.17.
Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]
EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is not supported EAL: IOMMU type 7 (sPAPR) is supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x30000 bytes EAL: Virtual area found at 0x200000000000 (size = 0x30000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket_id:0 hugepage_sz:2097152 EAL: Detected memory type: socket_id:8 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200000030000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200000200000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200400200000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200400400000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200800400000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200800600000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200c00600000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200c00800000 (size = 0x400000000) EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201000800000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201000a00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201400a00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201400c00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201800c00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201800e00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201c00e00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201c01000000 (size = 0x400000000) EAL: Hugepages will be freed exactly as allocated. EAL: No shared files mode enabled, IPC is disabled EAL: TSC frequency is ~510000 KHz EAL: Master lcore 0 is ready (tid=77e7345a44a0;cpuset=[0]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was expanded by 2MB EAL: No shared files mode enabled, IPC is disabled EAL: Mem event callback 'spdk:(nil)' registered EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:01:00.0 cannot be used EAL: PCI device 0003:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0003:01:00.0 cannot be used EAL: PCI device 0030:01:00.0 on NUMA socket 8 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0030:01:00.0 cannot be used
Hello,
Quoted 5 lines · Takeshi T Yoshimura
Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922. The operating system is Ubuntu 18.04 and Linux 4.17.
Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c0x1-n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]
We are missing the initial logs here (eal log level is set twice, but this should not be a problem).
Is it truncated by spdk ? Is there a way to get them ?
Thanks.
Signature · 1 line
David Marchand
-----David Marchand <[email protected]> wrote: -----
Quoted 21 lines
To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 02:01PM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcHello,
Quoted 1 line · Takeshi T Yoshimura
Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM
Power9 AC922.
Quoted 4 lines
The operating system is Ubuntu 18.04 and Linux 4.17.
Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization... [ DPDK EAL parameters: identify --no-shconf -c
0x1-n 1 -m 0--log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]
Quoted 10 lines
We are missing the initial logs here (eal log level is set twice, but this should not be a problem).
Is it truncated by spdk ? Is there a way to get them ?Thanks.
Signature · 1 line
David Marchand
Okay. I should have set the --log-level=lib.*:debug. Here is the log.
Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.*:debug --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid159692 ]
EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Detected lcore 2 as core 0 on socket 0 EAL: Detected lcore 3 as core 0 on socket 0 EAL: Detected lcore 4 as core 4 on socket 0 EAL: Detected lcore 5 as core 4 on socket 0 EAL: Detected lcore 6 as core 4 on socket 0 EAL: Detected lcore 7 as core 4 on socket 0 EAL: Detected lcore 8 as core 8 on socket 0 EAL: Detected lcore 9 as core 8 on socket 0 EAL: Detected lcore 10 as core 8 on socket 0 EAL: Detected lcore 11 as core 8 on socket 0 EAL: Detected lcore 12 as core 12 on socket 0 EAL: Detected lcore 13 as core 12 on socket 0 EAL: Detected lcore 14 as core 12 on socket 0 EAL: Detected lcore 15 as core 12 on socket 0 EAL: Detected lcore 16 as core 24 on socket 0 EAL: Detected lcore 17 as core 24 on socket 0 EAL: Detected lcore 18 as core 24 on socket 0 EAL: Detected lcore 19 as core 24 on socket 0 EAL: Detected lcore 20 as core 28 on socket 0 EAL: Detected lcore 21 as core 28 on socket 0 EAL: Detected lcore 22 as core 28 on socket 0 EAL: Detected lcore 23 as core 28 on socket 0 EAL: Detected lcore 24 as core 32 on socket 0 EAL: Detected lcore 25 as core 32 on socket 0 EAL: Detected lcore 26 as core 32 on socket 0 EAL: Detected lcore 27 as core 32 on socket 0 EAL: Detected lcore 28 as core 36 on socket 0 EAL: Detected lcore 29 as core 36 on socket 0 EAL: Detected lcore 30 as core 36 on socket 0 EAL: Detected lcore 31 as core 36 on socket 0 EAL: Detected lcore 32 as core 48 on socket 0 EAL: Detected lcore 33 as core 48 on socket 0 EAL: Detected lcore 34 as core 48 on socket 0 EAL: Detected lcore 35 as core 48 on socket 0 EAL: Detected lcore 36 as core 52 on socket 0 EAL: Detected lcore 37 as core 52 on socket 0 EAL: Detected lcore 38 as core 52 on socket 0 EAL: Detected lcore 39 as core 52 on socket 0 EAL: Detected lcore 40 as core 56 on socket 0 EAL: Detected lcore 41 as core 56 on socket 0 EAL: Detected lcore 42 as core 56 on socket 0 EAL: Detected lcore 43 as core 56 on socket 0 EAL: Detected lcore 44 as core 60 on socket 0 EAL: Detected lcore 45 as core 60 on socket 0 EAL: Detected lcore 46 as core 60 on socket 0 EAL: Detected lcore 47 as core 60 on socket 0 EAL: Detected lcore 48 as core 64 on socket 0 EAL: Detected lcore 49 as core 64 on socket 0 EAL: Detected lcore 50 as core 64 on socket 0 EAL: Detected lcore 51 as core 64 on socket 0 EAL: Detected lcore 52 as core 68 on socket 0 EAL: Detected lcore 53 as core 68 on socket 0 EAL: Detected lcore 54 as core 68 on socket 0 EAL: Detected lcore 55 as core 68 on socket 0 EAL: Detected lcore 56 as core 72 on socket 0 EAL: Detected lcore 57 as core 72 on socket 0 EAL: Detected lcore 58 as core 72 on socket 0 EAL: Detected lcore 59 as core 72 on socket 0 EAL: Detected lcore 60 as core 76 on socket 0 EAL: Detected lcore 61 as core 76 on socket 0 EAL: Detected lcore 62 as core 76 on socket 0 EAL: Detected lcore 63 as core 76 on socket 0 EAL: Detected lcore 64 as core 80 on socket 0 EAL: Detected lcore 65 as core 80 on socket 0 EAL: Detected lcore 66 as core 80 on socket 0 EAL: Detected lcore 67 as core 80 on socket 0 EAL: Detected lcore 68 as core 84 on socket 0 EAL: Detected lcore 69 as core 84 on socket 0 EAL: Detected lcore 70 as core 84 on socket 0 EAL: Detected lcore 71 as core 84 on socket 0 EAL: Detected lcore 72 as core 88 on socket 0 EAL: Detected lcore 73 as core 88 on socket 0 EAL: Detected lcore 74 as core 88 on socket 0 EAL: Detected lcore 75 as core 88 on socket 0 EAL: Detected lcore 76 as core 92 on socket 0 EAL: Detected lcore 77 as core 92 on socket 0 EAL: Detected lcore 78 as core 92 on socket 0 EAL: Detected lcore 79 as core 92 on socket 0 EAL: Detected lcore 80 as core 2048 on socket 8 EAL: Detected lcore 81 as core 2048 on socket 8 EAL: Detected lcore 82 as core 2048 on socket 8 EAL: Detected lcore 83 as core 2048 on socket 8 EAL: Detected lcore 84 as core 2052 on socket 8 EAL: Detected lcore 85 as core 2052 on socket 8 EAL: Detected lcore 86 as core 2052 on socket 8 EAL: Detected lcore 87 as core 2052 on socket 8 EAL: Detected lcore 88 as core 2056 on socket 8 EAL: Detected lcore 89 as core 2056 on socket 8 EAL: Detected lcore 90 as core 2056 on socket 8 EAL: Detected lcore 91 as core 2056 on socket 8 EAL: Detected lcore 92 as core 2060 on socket 8 EAL: Detected lcore 93 as core 2060 on socket 8 EAL: Detected lcore 94 as core 2060 on socket 8 EAL: Detected lcore 95 as core 2060 on socket 8 EAL: Detected lcore 96 as core 2064 on socket 8 EAL: Detected lcore 97 as core 2064 on socket 8 EAL: Detected lcore 98 as core 2064 on socket 8 EAL: Detected lcore 99 as core 2064 on socket 8 EAL: Detected lcore 100 as core 2068 on socket 8 EAL: Detected lcore 101 as core 2068 on socket 8 EAL: Detected lcore 102 as core 2068 on socket 8 EAL: Detected lcore 103 as core 2068 on socket 8 EAL: Detected lcore 104 as core 2072 on socket 8 EAL: Detected lcore 105 as core 2072 on socket 8 EAL: Detected lcore 106 as core 2072 on socket 8 EAL: Detected lcore 107 as core 2072 on socket 8 EAL: Detected lcore 108 as core 2076 on socket 8 EAL: Detected lcore 109 as core 2076 on socket 8 EAL: Detected lcore 110 as core 2076 on socket 8 EAL: Detected lcore 111 as core 2076 on socket 8 EAL: Detected lcore 112 as core 2080 on socket 8 EAL: Detected lcore 113 as core 2080 on socket 8 EAL: Detected lcore 114 as core 2080 on socket 8 EAL: Detected lcore 115 as core 2080 on socket 8 EAL: Detected lcore 116 as core 2084 on socket 8 EAL: Detected lcore 117 as core 2084 on socket 8 EAL: Detected lcore 118 as core 2084 on socket 8 EAL: Detected lcore 119 as core 2084 on socket 8 EAL: Detected lcore 120 as core 2088 on socket 8 EAL: Detected lcore 121 as core 2088 on socket 8 EAL: Detected lcore 122 as core 2088 on socket 8 EAL: Detected lcore 123 as core 2088 on socket 8 EAL: Detected lcore 124 as core 2092 on socket 8 EAL: Detected lcore 125 as core 2092 on socket 8 EAL: Detected lcore 126 as core 2092 on socket 8 EAL: Detected lcore 127 as core 2092 on socket 8 EAL: Detected lcore 128 as core 2096 on socket 8 EAL: Detected lcore 129 as core 2096 on socket 8 EAL: Detected lcore 130 as core 2096 on socket 8 EAL: Detected lcore 131 as core 2096 on socket 8 EAL: Detected lcore 132 as core 2100 on socket 8 EAL: Detected lcore 133 as core 2100 on socket 8 EAL: Detected lcore 134 as core 2100 on socket 8 EAL: Detected lcore 135 as core 2100 on socket 8 EAL: Detected lcore 136 as core 2104 on socket 8 EAL: Detected lcore 137 as core 2104 on socket 8 EAL: Detected lcore 138 as core 2104 on socket 8 EAL: Detected lcore 139 as core 2104 on socket 8 EAL: Detected lcore 140 as core 2108 on socket 8 EAL: Detected lcore 141 as core 2108 on socket 8 EAL: Detected lcore 142 as core 2108 on socket 8 EAL: Detected lcore 143 as core 2108 on socket 8 EAL: Detected lcore 144 as core 2112 on socket 8 EAL: Detected lcore 145 as core 2112 on socket 8 EAL: Detected lcore 146 as core 2112 on socket 8 EAL: Detected lcore 147 as core 2112 on socket 8 EAL: Detected lcore 148 as core 2116 on socket 8 EAL: Detected lcore 149 as core 2116 on socket 8 EAL: Detected lcore 150 as core 2116 on socket 8 EAL: Detected lcore 151 as core 2116 on socket 8 EAL: Detected lcore 152 as core 2136 on socket 8 EAL: Detected lcore 153 as core 2136 on socket 8 EAL: Detected lcore 154 as core 2136 on socket 8 EAL: Detected lcore 155 as core 2136 on socket 8 EAL: Detected lcore 156 as core 2140 on socket 8 EAL: Detected lcore 157 as core 2140 on socket 8 EAL: Detected lcore 158 as core 2140 on socket 8 EAL: Detected lcore 159 as core 2140 on socket 8 EAL: Support maximum 1536 logical core(s) by configuration. EAL: Detected 160 lcore(s) EAL: Detected 2 NUMA nodes EAL: No shared files mode enabled, IPC will be disabled EAL: No shared files mode enabled, IPC is disabled EAL: VFIO PCI modules not loaded EAL: Bus pci wants IOVA as 'DC' EAL: Buses did not request a specific IOVA mode. EAL: IOMMU is available, selecting IOVA as VA mode. EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory) EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is not supported EAL: IOMMU type 7 (sPAPR) is supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x30000 bytes EAL: Virtual area found at 0x200000000000 (size = 0x30000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket_id:0 hugepage_sz:2097152 EAL: Detected memory type: socket_id:8 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200000030000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200000200000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200400200000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200400400000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200800400000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200800600000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200c00600000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200c00800000 (size = 0x400000000) EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201000800000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201000a00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201400a00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201400c00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201800c00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201800e00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201c00e00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201c01000000 (size = 0x400000000) EAL: Hugepages will be freed exactly as allocated. EAL: No shared files mode enabled, IPC is disabled EAL: TSC frequency is ~510000 KHz EAL: Master lcore 0 is ready (tid=7430194644a0;cpuset=[0]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was expanded by 2MB EAL: No shared files mode enabled, IPC is disabled EAL: Mem event callback 'spdk:(nil)' registered EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:01:00.0 cannot be used EAL: PCI device 0003:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0003:01:00.0 cannot be used EAL: PCI device 0030:01:00.0 on NUMA socket 8 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0030:01:00.0 cannot be used
No NVMe controllers found.
Quoted 4 lines · Takeshi T Yoshimura
Okay. I should have set the --log-level=lib.*:debug. Here is the log.
Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c0x1-n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.*:debug --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid159692 ]
[snip]
Quoted 3 lines
EAL: No shared files mode enabled, IPC will be disabled
EAL: No shared files mode enabled, IPC is disabled
EAL: VFIO PCI modules not loaded
EAL thinks that vfio kmod is not loaded at this point.
Quoted 1 line
EAL: Bus pci wants IOVA as 'DC'
We see no log here for the devices that are used later.
- Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?
- Is the spdk_nvme driver loaded at this point?
- I suppose forcing --iova-mode=pa resolves the issue, but can you confirm this?
Quoted 4 lines
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Module/sys/module/rte_kninot found! error 2 (No such file or directory)
EAL: Selected IOVA mode 'VA'
[snip]
Quoted 13 lines
EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:01:00.0 cannot be used EAL: PCI device 0003:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0003:01:00.0 cannot be used EAL: PCI device 0030:01:00.0 on NUMA socket 8 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0030:01:00.0 cannot be usedNo NVMe controllers found.
Signature · 1 line
David Marchand
Quoted 1 line
- Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?
Yes, vfio-pci is bound to 0030:01:00.0 in the log.
Quoted 1 line
- Is the
spdk_nvmedriver loaded at this point?
No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.
Quoted 1 line
- I suppose forcing --iova-mode=pa resolves the issue, but can you confirm this?
Yes, you are right. I needed to modify the spdk core to do this, anyway.. Here is the log when I specify --iova-mode=pa for an example app of spdk (identify)
Starting SPDK v19.10-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.*:debug --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid140641 ]
EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Detected lcore 2 as core 0 on socket 0 EAL: Detected lcore 3 as core 0 on socket 0 EAL: Detected lcore 4 as core 4 on socket 0 EAL: Detected lcore 5 as core 4 on socket 0 EAL: Detected lcore 6 as core 4 on socket 0 EAL: Detected lcore 7 as core 4 on socket 0 EAL: Detected lcore 8 as core 8 on socket 0 EAL: Detected lcore 9 as core 8 on socket 0 EAL: Detected lcore 10 as core 8 on socket 0 EAL: Detected lcore 11 as core 8 on socket 0 EAL: Detected lcore 12 as core 12 on socket 0 EAL: Detected lcore 13 as core 12 on socket 0 EAL: Detected lcore 14 as core 12 on socket 0 EAL: Detected lcore 15 as core 12 on socket 0 EAL: Detected lcore 16 as core 24 on socket 0 EAL: Detected lcore 17 as core 24 on socket 0 EAL: Detected lcore 18 as core 24 on socket 0 EAL: Detected lcore 19 as core 24 on socket 0 EAL: Detected lcore 20 as core 28 on socket 0 EAL: Detected lcore 21 as core 28 on socket 0 EAL: Detected lcore 22 as core 28 on socket 0 EAL: Detected lcore 23 as core 28 on socket 0 EAL: Detected lcore 24 as core 32 on socket 0 EAL: Detected lcore 25 as core 32 on socket 0 EAL: Detected lcore 26 as core 32 on socket 0 EAL: Detected lcore 27 as core 32 on socket 0 EAL: Detected lcore 28 as core 36 on socket 0 EAL: Detected lcore 29 as core 36 on socket 0 EAL: Detected lcore 30 as core 36 on socket 0 EAL: Detected lcore 31 as core 36 on socket 0 EAL: Detected lcore 32 as core 48 on socket 0 EAL: Detected lcore 33 as core 48 on socket 0 EAL: Detected lcore 34 as core 48 on socket 0 EAL: Detected lcore 35 as core 48 on socket 0 EAL: Detected lcore 36 as core 52 on socket 0 EAL: Detected lcore 37 as core 52 on socket 0 EAL: Detected lcore 38 as core 52 on socket 0 EAL: Detected lcore 39 as core 52 on socket 0 EAL: Detected lcore 40 as core 56 on socket 0 EAL: Detected lcore 41 as core 56 on socket 0 EAL: Detected lcore 42 as core 56 on socket 0 EAL: Detected lcore 43 as core 56 on socket 0 EAL: Detected lcore 44 as core 60 on socket 0 EAL: Detected lcore 45 as core 60 on socket 0 EAL: Detected lcore 46 as core 60 on socket 0 EAL: Detected lcore 47 as core 60 on socket 0 EAL: Detected lcore 48 as core 64 on socket 0 EAL: Detected lcore 49 as core 64 on socket 0 EAL: Detected lcore 50 as core 64 on socket 0 EAL: Detected lcore 51 as core 64 on socket 0 EAL: Detected lcore 52 as core 68 on socket 0 EAL: Detected lcore 53 as core 68 on socket 0 EAL: Detected lcore 54 as core 68 on socket 0 EAL: Detected lcore 55 as core 68 on socket 0 EAL: Detected lcore 56 as core 72 on socket 0 EAL: Detected lcore 57 as core 72 on socket 0 EAL: Detected lcore 58 as core 72 on socket 0 EAL: Detected lcore 59 as core 72 on socket 0 EAL: Detected lcore 60 as core 76 on socket 0 EAL: Detected lcore 61 as core 76 on socket 0 EAL: Detected lcore 62 as core 76 on socket 0 EAL: Detected lcore 63 as core 76 on socket 0 EAL: Detected lcore 64 as core 80 on socket 0 EAL: Detected lcore 65 as core 80 on socket 0 EAL: Detected lcore 66 as core 80 on socket 0 EAL: Detected lcore 67 as core 80 on socket 0 EAL: Detected lcore 68 as core 84 on socket 0 EAL: Detected lcore 69 as core 84 on socket 0 EAL: Detected lcore 70 as core 84 on socket 0 EAL: Detected lcore 71 as core 84 on socket 0 EAL: Detected lcore 72 as core 88 on socket 0 EAL: Detected lcore 73 as core 88 on socket 0 EAL: Detected lcore 74 as core 88 on socket 0 EAL: Detected lcore 75 as core 88 on socket 0 EAL: Detected lcore 76 as core 92 on socket 0 EAL: Detected lcore 77 as core 92 on socket 0 EAL: Detected lcore 78 as core 92 on socket 0 EAL: Detected lcore 79 as core 92 on socket 0 EAL: Detected lcore 80 as core 2048 on socket 8 EAL: Detected lcore 81 as core 2048 on socket 8 EAL: Detected lcore 82 as core 2048 on socket 8 EAL: Detected lcore 83 as core 2048 on socket 8 EAL: Detected lcore 84 as core 2052 on socket 8 EAL: Detected lcore 85 as core 2052 on socket 8 EAL: Detected lcore 86 as core 2052 on socket 8 EAL: Detected lcore 87 as core 2052 on socket 8 EAL: Detected lcore 88 as core 2056 on socket 8 EAL: Detected lcore 89 as core 2056 on socket 8 EAL: Detected lcore 90 as core 2056 on socket 8 EAL: Detected lcore 91 as core 2056 on socket 8 EAL: Detected lcore 92 as core 2060 on socket 8 EAL: Detected lcore 93 as core 2060 on socket 8 EAL: Detected lcore 94 as core 2060 on socket 8 EAL: Detected lcore 95 as core 2060 on socket 8 EAL: Detected lcore 96 as core 2064 on socket 8 EAL: Detected lcore 97 as core 2064 on socket 8 EAL: Detected lcore 98 as core 2064 on socket 8 EAL: Detected lcore 99 as core 2064 on socket 8 EAL: Detected lcore 100 as core 2068 on socket 8 EAL: Detected lcore 101 as core 2068 on socket 8 EAL: Detected lcore 102 as core 2068 on socket 8 EAL: Detected lcore 103 as core 2068 on socket 8 EAL: Detected lcore 104 as core 2072 on socket 8 EAL: Detected lcore 105 as core 2072 on socket 8 EAL: Detected lcore 106 as core 2072 on socket 8 EAL: Detected lcore 107 as core 2072 on socket 8 EAL: Detected lcore 108 as core 2076 on socket 8 EAL: Detected lcore 109 as core 2076 on socket 8 EAL: Detected lcore 110 as core 2076 on socket 8 EAL: Detected lcore 111 as core 2076 on socket 8 EAL: Detected lcore 112 as core 2080 on socket 8 EAL: Detected lcore 113 as core 2080 on socket 8 EAL: Detected lcore 114 as core 2080 on socket 8 EAL: Detected lcore 115 as core 2080 on socket 8 EAL: Detected lcore 116 as core 2084 on socket 8 EAL: Detected lcore 117 as core 2084 on socket 8 EAL: Detected lcore 118 as core 2084 on socket 8 EAL: Detected lcore 119 as core 2084 on socket 8 EAL: Detected lcore 120 as core 2088 on socket 8 EAL: Detected lcore 121 as core 2088 on socket 8 EAL: Detected lcore 122 as core 2088 on socket 8 EAL: Detected lcore 123 as core 2088 on socket 8 EAL: Detected lcore 124 as core 2092 on socket 8 EAL: Detected lcore 125 as core 2092 on socket 8 EAL: Detected lcore 126 as core 2092 on socket 8 EAL: Detected lcore 127 as core 2092 on socket 8 EAL: Detected lcore 128 as core 2096 on socket 8 EAL: Detected lcore 129 as core 2096 on socket 8 EAL: Detected lcore 130 as core 2096 on socket 8 EAL: Detected lcore 131 as core 2096 on socket 8 EAL: Detected lcore 132 as core 2100 on socket 8 EAL: Detected lcore 133 as core 2100 on socket 8 EAL: Detected lcore 134 as core 2100 on socket 8 EAL: Detected lcore 135 as core 2100 on socket 8 EAL: Detected lcore 136 as core 2104 on socket 8 EAL: Detected lcore 137 as core 2104 on socket 8 EAL: Detected lcore 138 as core 2104 on socket 8 EAL: Detected lcore 139 as core 2104 on socket 8 EAL: Detected lcore 140 as core 2108 on socket 8 EAL: Detected lcore 141 as core 2108 on socket 8 EAL: Detected lcore 142 as core 2108 on socket 8 EAL: Detected lcore 143 as core 2108 on socket 8 EAL: Detected lcore 144 as core 2112 on socket 8 EAL: Detected lcore 145 as core 2112 on socket 8 EAL: Detected lcore 146 as core 2112 on socket 8 EAL: Detected lcore 147 as core 2112 on socket 8 EAL: Detected lcore 148 as core 2116 on socket 8 EAL: Detected lcore 149 as core 2116 on socket 8 EAL: Detected lcore 150 as core 2116 on socket 8 EAL: Detected lcore 151 as core 2116 on socket 8 EAL: Detected lcore 152 as core 2136 on socket 8 EAL: Detected lcore 153 as core 2136 on socket 8 EAL: Detected lcore 154 as core 2136 on socket 8 EAL: Detected lcore 155 as core 2136 on socket 8 EAL: Detected lcore 156 as core 2140 on socket 8 EAL: Detected lcore 157 as core 2140 on socket 8 EAL: Detected lcore 158 as core 2140 on socket 8 EAL: Detected lcore 159 as core 2140 on socket 8 EAL: Support maximum 1536 logical core(s) by configuration. EAL: Detected 160 lcore(s) EAL: Detected 2 NUMA nodes EAL: No shared files mode enabled, IPC will be disabled EAL: No shared files mode enabled, IPC is disabled EAL: VFIO PCI modules not loaded EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is not supported EAL: IOMMU type 7 (sPAPR) is supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x30000 bytes EAL: Virtual area found at 0x200000000000 (size = 0x30000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket_id:0 hugepage_sz:2097152 EAL: Detected memory type: socket_id:8 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200000030000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200000200000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200400200000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200400400000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200800400000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200800600000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200c00600000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200c00800000 (size = 0x400000000) EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201000800000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201000a00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201400a00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201400c00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201800c00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201800e00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201c00e00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201c01000000 (size = 0x400000000) EAL: Hugepages will be freed exactly as allocated. EAL: No shared files mode enabled, IPC is disabled EAL: TSC frequency is ~510000 KHz EAL: Master lcore 0 is ready (tid=7d49a88844a0;cpuset=[0]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was expanded by 2MB EAL: No shared files mode enabled, IPC is disabled EAL: Mem event callback 'spdk:(nil)' registered EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0003:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0030:01:00.0 on NUMA socket 8 EAL: probe driver: 144d:a822 spdk_nvme EAL: set IOMMU type 1 (Type 1) failed, error 19 (No such device) EAL: using IOMMU type 7 (sPAPR) EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered EAL: Installed memory event callback for VFIO EAL: VFIO reports MSI-X BAR as mappable EAL: PCI memory mapped at 0x202001000000 EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: Calling mem event callback 'spdk:(nil)' EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)' EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was expanded by 4MB EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: Calling mem event callback 'spdk:(nil)' EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)' EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was expanded by 4MB
=====================================================
NVMe Controller at 0030:01:00.0 [144d:a822]
=====================================================
Controller Capabilities/Features
================================
Vendor ID: 144d Subsystem Vendor ID: 1014 Serial Number: S3RXNA0K300102 Model Number: PCIe3 6.4TB NVMe Flash Adapter II x8 Firmware Version: MN12MN12 Recommended Arb Burst: 8 IEEE OUI Identifier: 38 25 00 Multi-path I/O May have multiple subsystem ports: Yes May be connected to multiple hosts: Yes Associated with SR-IOV VF: No Max Data Transfer Size: 1048576 Error Recovery Timeout: Unlimited NVMe Specification Version (VS): 1.2.1 NVMe Specification Version (Identify): 1.2.1 Maximum Queue Entries: 1024 Contiguous Queues Required: Yes Arbitration Mechanisms Supported Weighted Round Robin: Not Supported Vendor Specific: Not Supported Reset Timeout: 60000 ms Doorbell Stride: 4 bytes NVM Subsystem Reset: Supported Command Sets Supported NVM Command Set: Supported Boot Partition: Not Supported Memory Page Size Minimum: 4096 bytes Memory Page Size Maximum: 4096 bytes Optional Asynchronous Events Supported Namespace Attribute Notices: Supported Firmware Activation Notices: Supported 128-bit Host Identifier: Not Supported
Controller Memory Buffer Support
================================
Supported: No
Admin Command Set Attributes
============================
Security Send/Receive: Not Supported Format NVM: Supported Firmware Activate/Download: Supported Namespace Management: Supported Device Self-Test: Not Supported Directives: Not Supported NVMe-MI: Not Supported Virtualization Management: Not Supported Doorbell Buffer Config: Not Supported Abort Command Limit: 8 Async Event Request Limit: 16 Number of Firmware Slots: 3 Firmware Slot 1 Read-Only: No Firmware Update Granularity: No Information Provided Per-Namespace SMART Log: No Command Effects Log Page: Supported Get Log Page Extended Data: Not Supported Telemetry Log Pages: Not Supported Error Log Page Entries Supported: 256 Keep Alive: Not Supported
NVM Command Set Attributes
==========================
Submission Queue Entry Size
Max: 64
Min: 64
Completion Queue Entry Size
Max: 16 Min: 16 Number of Namespaces: 32 Compare Command: Not Supported Write Uncorrectable Command: Supported Dataset Management Command: Supported Write Zeroes Command: Supported Set Features Save Field: Supported Reservations: Supported Timestamp: Not Supported Volatile Write Cache: Not Present Atomic Write Unit (Normal): 65536 Atomic Write Unit (PFail): 1
Atomic Compare & Write Unit: 1
Scatter-Gather List
SGL Command Set: Supported
SGL Keyed: Not Supported
SGL Bit Bucket Descriptor: Supported
SGL Metadata Pointer: Supported Oversized SGL: Supported SGL Metadata Address: Not Supported SGL Offset: Not Supported Transport SGL Data Block: Not Supported
Replay Protected Memory Block: Not Supported
Firmware Slot Information
=========================
Active slot: 1 Next controller reset slot: 1 Slot 1 Firmware Revision: MN12MN12 Slot 2 Firmware Revision: MN12MN12 Slot 3 Firmware Revision: MN12MN12
Commands Supported and Effects
==============================
Admin Commands
--------------
Delete I/O Submission Queue (00h): Supported
Create I/O Submission Queue (01h): Supported
Get Log Page (02h): Supported
Delete I/O Completion Queue (04h): Supported
Create I/O Completion Queue (05h): Supported
Identify (06h): Supported
Abort (08h): Supported
Set Features (09h): Supported
Get Features (0Ah): Supported
Asynchronous Event Request (0Ch): Supported
Namespace Management (0Dh): Supported NS-Cap-Change NS-Inventory-Change All-NS-Exclusive
Firmware Commit (10h): Supported
Firmware Image Download (11h): Supported
Namespace Attachment (15h): Supported All-NS-Exclusive
Format NVM (80h): Supported LBA-Change NS-Cap-Change Per-NS-Exclusive
I/O Commands
------------
Flush (00h): Supported
Write (01h): Supported LBA-Change
Read (02h): Supported
Write Uncorrectable (04h): Supported LBA-Change
Write Zeroes (08h): Supported LBA-Change
Dataset Management (09h): Supported LBA-Change
Reservation Register (0Dh): Supported
Reservation Report (0Eh): Supported
Reservation Acquire (11h): Supported
Reservation Release (15h): Supported
Error Log
=========
Entry: 0 Error Count: 0x2 Submission Queue Id: 0x0 Command Id: 0x1a Phase Bit: 0 Status Code: 0x1c Status Code Type: 0x1 Do Not Retry: 0 Error Location: 0x0 LBA: 0x0 Namespace: 0x1 Vendor Log Page: 0x0 ----------- Entry: 1 Error Count: 0x1 Submission Queue Id: 0x0 Command Id: 0x5 Phase Bit: 0 Status Code: 0x1c Status Code Type: 0x1 Do Not Retry: 0 Error Location: 0x0 LBA: 0x0 Namespace: 0x1 Vendor Log Page: 0x0
Power Management
================
Number of Power States: 1 Current Power State: Power State #0
Power State #0: Max Power: 25.00 W
Non-Operational Permissive Mode: Not Supported
Health Information
==================
Critical Warnings:
Available Spare Space: OK Temperature: OK Device Reliability: OK Read Only: No Volatile Memory Backup: OK Current Temperature: 319 Kelvin (46 Celsius) Temperature Threshold: 353 Kelvin (80 Celsius) Available Spare: 100% Available Spare Threshold: 10% Life Percentage Used: 0% Data Units Read: 44913828 Data Units Written: 67060524 Host Read Commands: 1363758490 Host Write Commands: 2804327733 Controller Busy Time: 715 minutes Power Cycles: 104 Power On Hours: 6530 hours Unsafe Shutdowns: 41 Unrecoverable Media Errors: 0 Lifetime Error Log Entries: 2 Warning Temperature Time: 0 minutes Critical Temperature Time: 0 minutes Temperature Sensor 1: 319 Kelvin (46 Celsius) Temperature Sensor 2: 316 Kelvin (43 Celsius) Temperature Sensor 3: 316 Kelvin (43 Celsius)
Number of Queues
================
Number of I/O Submission Queues: 128 Number of I/O Completion Queues: 128
Namespace ID:1
Deallocate: Supported Deallocated/Unwritten Error: Not Supported Deallocated Read Value: Unknown Deallocate in Write Zeroes: Not Supported Deallocated Guard Field: 0xFFFF Flush: Not Supported Reservation: Supported Namespace Sharing Capabilities: Private Size (in LBAs): 1562805846 (1490M) Capacity (in LBAs): 1562805846 (1490M) Utilization (in LBAs): 1562805846 (1490M) NGUID: 335258304B3001020025384100000001 Thin Provisioning: Not Supported Per-NS Atomic Units: No NGUID/EUI64 Never Reused: No Number of LBA Formats: 4 Current LBA Format: LBA Format #00 LBA Format #00: Data Size: 4096 Metadata Size: 0 LBA Format #01: Data Size: 4096 Metadata Size: 8 LBA Format #02: Data Size: 512 Metadata Size: 0 LBA Format #03: Data Size: 512 Metadata Size: 8
EAL: Calling mem event callback 'spdk:(nil)' EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)' EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was shrunk by 4MB EAL: Calling mem event callback 'spdk:(nil)' EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)' EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was shrunk by 4MB EAL: request: eal_dev_mp_request EAL: No shared files mode enabled, IPC is disabled EAL: PCI device 0030:01:00.0 on NUMA socket 8 EAL: remove driver: 144d:a822 spdk_nvme EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered EAL: Releasing pci mapped resource for 0030:01:00.0 EAL: Calling pci_unmap_resource for 0030:01:00.0 at 0x202001000000 EAL: PCI memory unmapped at 0x202001000000
Quoted 9 lines · Takeshi T Yoshimura
Quoted 2 lines
- Are the (below) devices bound to vfio-pci or any other kmod at this point of the init?
Yes, vfio-pci is bound to 0030:01:00.0 in the log.
Quoted 1 line
- Is the
spdk_nvmedriver loaded at this point?No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.
Do you mean that spdk registers its drivers after rte_eal_init() has returned?
Nothing should prevent from registering the drivers them before calling rte_eal_init(), did you try this?
If the drivers are in the form of shared libraries (.so files), did you try to load them using the -d eal option?
Signature · 1 line
David Marchand
-----David Marchand <[email protected]> wrote: -----
Quoted 20 lines
To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 06:47PM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcQuoted 2 lines · Takeshi T Yoshimura
Quoted 2 lines
- Are the (below) devices bound to vfio-pci or any other kmod at
this point of the init?
Quoted 7 lines
Yes, vfio-pci is bound to 0030:01:00.0 in the log.
Quoted 1 line
- Is the
spdk_nvmedriver loaded at this point?No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.
Quoted 13 lines
Do you mean that spdk registers its drivers after
rte_eal_init()has returned?Nothing should prevent from registering the drivers them before calling
rte_eal_init(), did you try this?
If the drivers are in the form of shared libraries (.so files), did you try to load them using the -d eal option?Signature · 1 line
David Marchand
SPDK apps have to call rte_eal_init() before registering drivers,
since the driver allocates DMA memory.
Do I have to change existing apps and spdk core to workaround this issue to use new versions of dpdk?
Quoted 32 lines · Takeshi T Yoshimura
-----David Marchand <[email protected]> wrote: -----
Quoted 29 lines
To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 06:47PM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcQuoted 2 lines · Takeshi T Yoshimura
Quoted 2 lines
- Are the (below) devices bound to vfio-pci or any other kmod at
this point of the init?
Quoted 7 lines
Yes, vfio-pci is bound to 0030:01:00.0 in the log.
Quoted 1 line
- Is the
spdk_nvmedriver loaded at this point?No, spdk just attempts to initialize dpdk at this point. After the initialization finishes, spdk probes the nvme driver.
Do you mean that spdk registers its drivers after
rte_eal_init()has returned?Nothing should prevent from registering the drivers them before calling
rte_eal_init(), did you try this?
If the drivers are in the form of shared libraries (.so files), did you try to load them using the -d eal option?
Quoted 7 lines
Quoted 4 lines
--
David MarchandSPDK apps have to call
rte_eal_init()before registering drivers, since the driver allocates DMA memory.
Registering a driver before calling rte_eal_init is something that works most of the time.
Not sure I get your point here.
Quoted 1 line
Do I have to change existing apps and spdk core to workaround this issue to use new versions of dpdk?
I understand this is a regression and I am looking at it. If we are too short for 19.08, then we can go with your workaround, but it needs a better commitlog.
Signature · 1 line
David Marchand
-----David Marchand <[email protected]> wrote: -----
Quoted 63 lines
To: Takeshi T Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 08/01/2019 04:45AM
Cc: dev <[email protected]>, David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppcQuoted 8 lines · Takeshi T Yoshimura
-----David Marchand <[email protected]> wrote: -----
Quoted 5 lines
To: Takeshi Yoshimura <[email protected]>
From: David Marchand <[email protected]>
Date: 07/31/2019 06:29PM
Cc: dev <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PAin
Quoted 3 lines
Quoted 3 lines
ppc
On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines · wrote:
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to forceIOVA
Quoted 2 lines
Quoted 2 lines
as
Quoted 1 line
PA in ppc as before because current ppc driver does not support
VA
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
mode.
What is the "ppc driver" you are referring to? Do you mean the vfio implementation for ppc?
Quoted 2 lines
Theoretically, ppc can support VA mode, but I suspect that ppc
with
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change
the
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
code to check device capability in ppc.
Sorry, I am confused.
Is VA mode not working at all? or is it a performance issue?Quoted 7 lines
Quoted 4 lines
--
David MarchandAh, I overlooked your first question. The "ppc driver" is in
drivers/bus/pci/linux/pci.c. Inpci.c,
pci_one_device_iommu_support_va()Quoted 1 line
always returns false in ppc. so, the driver does not allow VA mode
in ppc.
Quoted 4 lines
This return value is passed to
rte_pci_probe_one_driver()indrivers/bus/pci/pci_common.c. And the error log appeared like:
Quoted 13 lines
Quoted 3 lines
EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', notinitializing
Can you confirm on which kernel driver this device was bound ? I'd like to be sure this is the same issue than David.
Signature · 1 line
David Marchand
Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 AC922. The operating system is Ubuntu 18.04 and Linux 4.17.
Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --log-level=lib.eal:8 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid139059 ]
EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is not supported EAL: IOMMU type 7 (sPAPR) is supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x30000 bytes EAL: Virtual area found at 0x200000000000 (size = 0x30000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket_id:0 hugepage_sz:2097152 EAL: Detected memory type: socket_id:8 hugepage_sz:2097152 EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200000030000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200000200000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200400200000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200400400000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200800400000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200800600000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200c00600000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200c00800000 (size = 0x400000000) EAL: Creating 4 segment lists: n_segs:8192 socket_id:8 hugepage_sz:2097152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201000800000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201000a00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201400a00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201400c00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201800c00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201800e00000 (size = 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201c00e00000 (size = 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201c01000000 (size = 0x400000000) EAL: Hugepages will be freed exactly as allocated. EAL: No shared files mode enabled, IPC is disabled EAL: TSC frequency is ~510000 KHz EAL: Master lcore 0 is ready (tid=77e7345a44a0;cpuset=[0]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp_malloc_sync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was expanded by 2MB EAL: No shared files mode enabled, IPC is disabled EAL: Mem event callback 'spdk:(nil)' registered EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:01:00.0 cannot be used EAL: PCI device 0003:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0003:01:00.0 cannot be used EAL: PCI device 0030:01:00.0 on NUMA socket 8 EAL: probe driver: 144d:a822 spdk_nvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0030:01:00.0 cannot be used
Hello Anatoly,
Quoted 36 lines · Takeshi Yoshimura
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
Signed-off-by Takeshi Yoshimura <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 6 ++++++ 1 file changed, 6 insertions(+)lib/librte_eal/linux/eal/eal.c +6
-1@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv) RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n"); } } +#endif +#ifdef RTE_ARCH_PPC_64 +if (iova_mode ==RTE_IOVA_VA ) { + iova_mode =RTE_IOVA_PA ; +RTE_LOG (WARNING ,EAL ,"Forcing IOVA as 'PA' because PPC uses PA mode.\n" ); + }#endif rte_eal_get_configuration ()->iova_mode = iova_mode; }else { --
Quoted 2 lines
2.17.1
Afaiu, this problem is linked to a specific iommu implementation. Does it make sense to introduce a check "iommu_support_va" in eal, a bit like what we have in the pci bus ?
Signature · 1 line
David Marchand
Quoted 37 lines · David Marchand
Hello Anatoly,
Quoted 34 lines · Takeshi Yoshimura
Commit
b76fafb174d2("eal: fix IOVA mode selection as VA for PCI drivers") breaks ppc apps with no IOVA configs (orRTE_IOVA_DC) because of the inconsistency of user's request and the result of device capability for IOVA mode. I updated the code to force IOVA as PA in ppc as before because current ppc driver does not support VA mode.Theoretically, ppc can support VA mode, but I suspect that ppc with VA mode may have performance issues to create a big DMA window (VA often uses higher addresses than PA). So, I didn't change the code to check device capability in ppc.
Signed-off-by Takeshi Yoshimura <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 6 ++++++ 1 file changed, 6 insertions(+)lib/librte_eal/linux/eal/eal.c +6
-0@@ -1121,6 +1121,12 @@ rte_eal_init(int argc, char **argv) RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n"); } } +#endif +#ifdef RTE_ARCH_PPC_64 +if (iova_mode ==RTE_IOVA_VA ) { + iova_mode =RTE_IOVA_PA ; +RTE_LOG (WARNING ,EAL ,"Forcing IOVA as 'PA' because PPC uses PA mode.\n" ); + }#endif rte_eal_get_configuration ()->iova_mode = iova_mode;
Quoted 9 lines
Quoted 4 lines
lib/librte_eal/linux/eal/eal.c +0
-1}else { --2.17.1
Afaiu, this problem is linked to a specific iommu implementation. Does it make sense to introduce a check "
iommu_support_va" in eal, a bit like what we have in the pci bus ?
It probably does. Some platforms simply don't support VA mode (FreeBSD, for one), so that would be useful instead of hardcoding IOVA as PA/VA as defaults.
Signature · 2 lines
Thanks, Anatoly