Sporadic errors while initializing NICs in example applications, dpdk-1.5.0r1
Hi, I'm experiencing weird problems with running dpdk examples on my server running ubuntu-12.04. Application either manages to use ethernet ports or doesn't. For example, this is results of two identical sequental runs of l2fwd:
******************************************************************************
dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ s -E ./build/l2fwd -c 0x3 -n 2 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Detected lcore 4 as core 0 on socket 0 EAL: Detected lcore 5 as core 1 on socket 0 EAL: Detected lcore 6 as core 2 on socket 0 EAL: Detected lcore 7 as core 3 on socket 0 EAL: Skip lcore 8 (not detected) <LINES DELETED> EAL: Setting up memory... EAL: Ask a virtual area of 0x4194304 bytes EAL: Virtual area found at 0x7f6b82400000 (size = 0x400000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f6b82000000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f6b81c00000 (size = 0x200000) EAL: Ask a virtual area of 0x1056964608 bytes EAL: Virtual area found at 0x7f6b42a00000 (size = 0x3f000000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f6b42600000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f6b42200000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f6b41e00000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f6b41a00000 (size = 0x200000) EAL: Requesting 512 pages of size 2MB from socket 0 EAL: TSC frequency is ~1600000 KHz EAL: Master core 0 is ready (tid=836da800) EAL: Core 1 is ready (tid=40ff8700) EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83687000 EAL: PCI memory mapped at 0x7f6b83683000 EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83663000 EAL: PCI memory mapped at 0x7f6b8365f000 EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:03:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:04:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:04:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:05:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:05:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:06:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:06:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:07:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:07:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:08:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:08:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:09:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:09:00.0 not managed by UIO driver, skipping EAL: Error - exiting with code: 1 Cause: No Ethernet ports - bye
************************************************************************************
dev@econat-tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ s -E ./build/l2fwd -c 0x3 -n 2 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Detected lcore 4 as core 0 on socket 0 EAL: Detected lcore 5 as core 1 on socket 0 EAL: Detected lcore 6 as core 2 on socket 0 EAL: Detected lcore 7 as core 3 on socket 0 EAL: Skip lcore 8 (not detected) <LINES DELETED> EAL: Setting up memory... EAL: Ask a virtual area of 0x4194304 bytes EAL: Virtual area found at 0x7f538a600000 (size = 0x400000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f538a200000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f5389e00000 (size = 0x200000) EAL: Ask a virtual area of 0x1056964608 bytes EAL: Virtual area found at 0x7f534ac00000 (size = 0x3f000000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f534a800000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f534a400000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f534a000000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f5349c00000 (size = 0x200000) EAL: Requesting 512 pages of size 2MB from socket 0 EAL: TSC frequency is ~3301000 KHz EAL: Master core 0 is ready (tid=8b98d800) EAL: Core 1 is ready (tid=491f8700) EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f538b93a000 EAL: PCI memory mapped at 0x7f538b936000 EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f538b916000 EAL: PCI memory mapped at 0x7f538b912000 EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:03:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:04:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:04:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:05:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:05:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:06:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:06:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:07:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:07:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:08:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:08:00.0 not managed by UIO driver, skipping EAL: PCI device 0000:09:00.0 on NUMA socket -1 EAL: probe driver: 8086:10d3 rte_em_pmd EAL: 0000:09:00.0 not managed by UIO driver, skipping Skipping disabled port 0 Skipping disabled port 1 EAL: Error - exiting with code: 1 Cause: All available ports are disabled. Please set portmask.
******************************************************************
You see, final messages differ. Almost all the time I get "Cause: No Ethernet ports - bye" but sometimes (with probability of 1/10 or so) NIC is initialized successfully.
Some info about the system:
dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 58 model name : Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz stepping : 9 microcode : 0x10 cpu MHz : 1600.000 cache size : 8192 KB physical id : 0 siblings : 8 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms bogomips : 6584.75 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ lspci 00:00.0 Host bridge: Intel Corporation Ivy Bridge DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Ivy Bridge PCI Express Root Port (rev 09) 00:01.1 PCI bridge: Intel Corporation Ivy Bridge PCI Express Root Port (rev 09) 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) 00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 2 (rev b5) 00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b5) 00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b5) 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5) 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5) 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5) 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) 00:1f.0 ISA bridge: Intel Corporation C206 Chipset Family LPC Controller (rev 05) 00:1f.2 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 4 port SATA IDE Controller (rev 05) 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05) 00:1f.5 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 2 port SATA IDE Controller (rev 05) 02:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 02:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 08:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
dev@tiny-one:~/dpdk-1.5.0r1/examples/l2fwd$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 12.04.2 LTS Release: 12.04 Codename: precise
Any ideas how to investigate this?
Regards,
Dmitry
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello,
22/11/2013 13:29, Dmitry Vyal :
Quoted 8 lines
EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83687000 EAL: PCI memory mapped at 0x7f6b83683000 EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83663000 EAL: PCI memory mapped at 0x7f6b8365f000
[...]
Quoted 4 lines
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - byeAny ideas how to investigate this?
Could you try this patch in order to see the root cause of your issue ?
lib/librte_pmd_ixgbe/ixgbe_logs.h +18 -15
@@ -34,41 +34,44 @@#ifndef _IXGBE_LOGS_H_#define _IXGBE_LOGS_H_ +#define PMD_LOG(level, fmt, args...) \ + RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ##args) +#ifdef RTE_LIBRTE_IXGBE_DEBUG_INIT -#define PMD_INIT_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_INIT_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#define PMD_INIT_FUNC_TRACE () PMD_INIT_LOG(DEBUG, " >>")#else -#define PMD_INIT_LOG(level, fmt, args...) do { } while(0) +#define PMD_INIT_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#define PMD_INIT_FUNC_TRACE () do { } while(0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_RX -#define PMD_RX_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_RX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_RX_LOG(level, fmt, args...) do { } while(0) +#define PMD_RX_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX -#define PMD_TX_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_TX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_TX_LOG(level, fmt, args...) do { } while(0) +#define PMD_TX_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX_FREE -#define PMD_TX_FREE_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_TX_FREE_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_TX_FREE_LOG(level, fmt, args...) do { } while(0) +#define PMD_TX_FREE_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_DRIVER -#define PMD_DRV_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_DRV_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_DRV_LOG(level, fmt, args...) do { } while(0) +#define PMD_DRV_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #endif /* _IXGBE_LOGS_H_ */
Signature · 1 line
Thomas
A message that is not in the archive (sent off-list or not yet received): 201311221348.02307.thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org
Looks like I finally found the reason. After applying this patch I can no longer reproduce the error.
lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c +0 -0
@@ -2347,7 +2347,7 @@ s32 ixgbe_reset_pipeline_82599(struct ixgbe_hw *hw)/* Write AUTOC register with toggled LMS[2] bit and Restart_AN */ IXGBE_WRITE_REG(hw, IXGBE_AUTOC, autoc_reg ^
IXGBE_AUTOC_LMS_1G_AN);
/* Wait for AN to leave state 0 */
- for (i = 0; i < 10; i++) {
+ for (i = 0; i < 100; i++) {
msec_delay(4);
anlp1_reg = IXGBE_READ_REG(hw, IXGBE_ANLP1);
if (anlp1_reg & IXGBE_ANLP1_AN_STATE_MASK)Quoted 81 lines · Thomas Monjalon
Hello,
22/11/2013 13:29, Dmitry Vyal :
Quoted 8 lines
EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83687000 EAL: PCI memory mapped at 0x7f6b83683000 EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83663000 EAL: PCI memory mapped at 0x7f6b8365f000[...]
Quoted 4 lines
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - byeAny ideas how to investigate this?
Could you try this patch in order to see the root cause of your issue ?
lib/librte_pmd_ixgbe/ixgbe_logs.h +18
-15@@ -34,41 +34,44 @@#ifndef _IXGBE_LOGS_H_#define _IXGBE_LOGS_H_ +#define PMD_LOG(level, fmt, args...) \ + RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ##args) +#ifdef RTE_LIBRTE_IXGBE_DEBUG_INIT -#define PMD_INIT_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_INIT_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#define PMD_INIT_FUNC_TRACE () PMD_INIT_LOG(DEBUG, " >>")#else -#define PMD_INIT_LOG(level, fmt, args...) do { } while(0) +#define PMD_INIT_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#define PMD_INIT_FUNC_TRACE () do { } while(0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_RX -#define PMD_RX_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_RX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_RX_LOG(level, fmt, args...) do { } while(0) +#define PMD_RX_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX -#define PMD_TX_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_TX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_TX_LOG(level, fmt, args...) do { } while(0) +#define PMD_TX_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX_FREE -#define PMD_TX_FREE_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_TX_FREE_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_TX_FREE_LOG(level, fmt, args...) do { } while(0) +#define PMD_TX_FREE_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_DRIVER -#define PMD_DRV_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_DRV_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_DRV_LOG(level, fmt, args...) do { } while(0) +#define PMD_DRV_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #endif /* _IXGBE_LOGS_H_ */
A message that is not in the archive (sent off-list or not yet received): [email protected]
I had exactly same problem and fixed it with same patch since release 1.4. But somehow it stopped to appear even without the patch. I have no idea what I have done since both the kernel and the driver have been updated during these days.
Quoted 105 lines · Dmitry Vyal
Looks like I finally found the reason. After applying this patch I can no longer reproduce the error.
lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c +1
-1@@ -2347,7 +2347,7 @@ s32 ixgbe_reset_pipeline_82599(struct ixgbe_hw *hw)/* Write AUTOC register with toggled LMS[2] bit and Restart_AN */ IXGBE_WRITE_REG (hw,IXGBE_AUTOC , autoc_reg ^IXGBE_AUTOC_LMS_1G_AN );/* Wait for AN to leave state 0 */ -for (i =0 ; i <10 ; i++) { +for (i =0 ; i <100 ; i++) {msec_delay (4 ); anlp1_reg =IXGBE_READ_REG (hw,IXGBE_ANLP1 );if (anlp1_reg &IXGBE_ANLP1_AN_STATE_MASK )Quoted 85 lines · Thomas Monjalon
Hello,
22/11/2013 13:29, Dmitry Vyal :
Quoted 9 lines
EAL: PCI device 0000:02:00.0 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83687000 EAL: PCI memory mapped at 0x7f6b83683000 EAL: PCI device 0000:02:00.1 on NUMA socket -1 EAL: probe driver: 8086:10fb rte_ixgbe_pmd EAL: PCI memory mapped at 0x7f6b83663000 EAL: PCI memory mapped at 0x7f6b8365f000[...]
Quoted 5 lines
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - byeAny ideas how to investigate this?
Could you try this patch in order to see the root cause of your issue ?
lib/librte_pmd_ixgbe/ixgbe_logs.h +17
-15@@ -34,41 +34,44 @@#ifndef _IXGBE_LOGS_H_#define _IXGBE_LOGS_H_ +#define PMD_LOG(level, fmt, args...) \ + RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ##args) +#ifdef RTE_LIBRTE_IXGBE_DEBUG_INIT -#define PMD_INIT_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_INIT_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#define PMD_INIT_FUNC_TRACE () PMD_INIT_LOG(DEBUG, " >>")#else -#define PMD_INIT_LOG(level, fmt, args...) do { } while(0) +#define PMD_INIT_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#define PMD_INIT_FUNC_TRACE () do { } while(0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_RX -#define PMD_RX_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_RX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_RX_LOG(level, fmt, args...) do { } while(0) +#define PMD_RX_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX -#define PMD_TX_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_TX_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_TX_LOG(level, fmt, args...) do { } while(0) +#define PMD_TX_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_TX_FREE -#define PMD_TX_FREE_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_TX_FREE_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_TX_FREE_LOG(level, fmt, args...) do { } while(0) +#define PMD_TX_FREE_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #ifdef RTE_LIBRTE_IXGBE_DEBUG_DRIVER -#define PMD_DRV_LOG(level, fmt, args...) \ - RTE_LOG(level, PMD, "%s(): " fmt "\n", __func__, ## args) +#define PMD_DRV_LOG(level, fmt, args...) PMD_LOG(level, fmt, ##args)#else -#define PMD_DRV_LOG(level, fmt, args...) do { } while(0) +#define PMD_DRV_LOG(level, fmt, args...) \ + (void)(RTE_LOG_##level <= RTE_LOG_ERR ? PMD_LOG(level, fmt, ##args) : 0)#endif #endif /* _IXGBE_LOGS_H_ */
27/11/2013 15:06, Dmitry Vyal :
lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c
Quoted 11 lines
Looks like I finally found the reason. After applying this patch I can no longer reproduce the error.
lib/librte_pmd_ixgbe/ixgbe/ixgbe_82599.c +1
-1/* Wait for AN to leave state 0 */ -for (i =0 ; i <10 ; i++) { +for (i =0 ; i <100 ; i++) {msec_delay (4 ); anlp1_reg =IXGBE_READ_REG (hw,IXGBE_ANLP1 );if (anlp1_reg &IXGBE_ANLP1_AN_STATE_MASK )
It's probably due to a frequency scaling.
The timer based is initialized when DPDK initialize and the CPU can change its frequency, breaking next timers.
The fix is to control the CPU frequency.
Please try this, without your patch:
for g in /sys/devices/system/cpu/*/cpufreq/scaling_governor; do echo performance >$g; done
The right fix for applications (examples and testpmd included)
could be to call rte_power_init(). Patches are welcomed.
Thank you for your analysis
Signature · 1 line
Thomas
A message that is not in the archive (sent off-list or not yet received): 201311271542.05288.thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org
Quoted 11 lines
It's probably due to a frequency scaling.
The timer based is initialized when DPDK initialize and the CPU can change its frequency, breaking next timers.The fix is to control the CPU frequency.
Please try this, without your patch:
for g in/sys/devices/system/cpu/*/cpufreq/scaling_governor; do
echo performance >$g; done The right fix for applications (examples and testpmd included) could be to callrte_power_init(). Patches are welcomed.
[BR] Frequency changes should not affect timers for modern Intel CPUs. Please see the " Intel(r) 64 and IA-32 Architectures Software Developer's Manual" Volume 3 (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf) , Section 17.13 for more details on this.
A message that is not in the archive (sent off-list or not yet received): 59AF69C657FD0841A61C55336867B5B01A9781FC-kPTMFJFq+rELt2AQoY/[email protected]
Hmm, that's strange. I don't know how to interpret my observations then. I have access to two platforms, one is based on Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz and another on Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz. Both running ubuntu-12.04 server. I see repeating errors on NIC initialisation phase. The error frequency greatly reduces if I patch loop limit as I described earlier or if I call rte_power_init and rte_power_freq_max as Thomas suggested.
But the only way to get rid of them completely is to set performance governor.
Quoted 12 lines · Bruce
Quoted 10 lines
It's probably due to a frequency scaling.
The timer based is initialized when DPDK initialize and the CPU can change its frequency, breaking next timers.The fix is to control the CPU frequency.
Please try this, without your patch:
for g in/sys/devices/system/cpu/*/cpufreq/scaling_governor; do
echo performance >$g; done The right fix for applications (examples and testpmd included) could be to callrte_power_init(). Patches are welcomed.[BR] Frequency changes should not affect timers for modern Intel CPUs. Please see the " Intel(r) 64 and IA-32 Architectures Software Developer's Manual" Volume 3 (http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf) , Section 17.13 for more details on this.
A message that is not in the archive (sent off-list or not yet received): [email protected]
29/11/2013 14:53, Dmitry Vyal :
Quoted 30 lines · Bruce
Quoted 14 lines
Quoted 5 lines
It's probably due to a frequency scaling.
The timer based is initialized when DPDK initialize and the CPU can change
its frequency, breaking next timers.Quoted 8 lines
The fix is to control the CPU frequency.
Please try this, without your patch:
for g in/sys/devices/system/cpu/*/cpufreq/scaling_governor; doecho performance >$g; done The right fix for applications (examples and testpmd included) could be to call
rte_power_init(). Patches are welcomed.Quoted 6 lines
[BR] Frequency changes should not affect timers for modern Intel CPUs. Please see the " Intel(r) 64 and IA-32 Architectures Software Developer's Manual" Volume 3
(http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-i
a-32-architectures-software-developer-system-programming-manual-325384.pdf ) , Section 17.13 for more details on this.Hmm, that's strange. I don't know how to interpret my observations then. I have access to two platforms, one is based on Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz and another on Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz. Both running ubuntu-12.04 server. I see repeating errors on NIC initialisation phase. The error frequency greatly reduces if I patch loop limit as I described earlier or if I call
rte_power_initandrte_power_freq_maxas Thomas suggested.
Quoted 2 lines
But the only way to get rid of them completely is to set performance governor.
Please check that your hardware do not support invariant TSC. It would explain why you need to fix frequency.
I attach a simple code to test CPU feature "Invariant TSC".
Signature · 1 line
Thomas
29/11/2013 13:25, Thomas Monjalon :
Quoted 19 lines
29/11/2013 14:53, Dmitry Vyal :
Quoted 18 lines · Bruce
Quoted 8 lines
[BR] Frequency changes should not affect timers for modern Intel CPUs. Please see the " Intel(r) 64 and IA-32 Architectures Software Developer's
Manual" Volume 3
(http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-> > > i a-32-architectures-software-developer-system-programming-manual-325384.p df
) , Section 17.13 for more details on this.Hmm, that's strange. I don't know how to interpret my observations then. I have access to two platforms, one is based on Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz and another on Intel(R) Xeon(R) CPU E3-1270 v3 @ 3.50GHz. Both running ubuntu-12.04 server. I see repeating errors on NIC initialisation phase. The error frequency greatly reduces if I patch loop limit as I described earlier or if I call
rte_power_initandrte_power_freq_maxas Thomas suggested.
Quoted 7 lines
Quoted 2 lines
But the only way to get rid of them completely is to set performance governor.
Please check that your hardware do not support invariant TSC. It would explain why you need to fix frequency.
I attach a simple code to test CPU feature "Invariant TSC".
It seems that the file is stripped on the mailing-list. Code inlined:
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <stdint.h>
int main()
{
uint32_t a = 0x80000000;
uint32_t b, d;
__asm__("cpuid;"
:"=a"(b)
:"0"(a));
if (b >= 0x80000007) {
a = 0x80000007;
__asm__("cpuid;"
:"=a"(b), "=d"(d)
:"0"(a));
if (d & (1<<8)) {
printf("Invariant TSC is supported\n");
} else{
printf("Invariant TSC is NOT supported\n");
}
} else {
printf("No support for Advanced Power Management Information in
CPUID\n");
}
return 0;
}Quoted 6 lines · Thomas Monjalon
29/11/2013 13:25, Thomas Monjalon :
Please check that your hardware do not support invariant TSC. It would explain why you need to fix frequency.
I attach a simple code to test CPU feature "Invariant TSC".
I compiled and ran the code on all the platforms I had troubles on. Invariant TSC is supported everywhere.
Quoted 37 lines
It seems that the file is stripped on the mailing-list. Code inlined:
#include <
stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <stdint.h>int
main(){ uint32_t a = 0x80000000; uint32_t b, d; __asm__("cpuid;" :"=a"(b) :"0"(a)); if (b >= 0x80000007) { a = 0x80000007; __asm__("cpuid;" :"=a"(b), "=d"(d) :"0"(a)); if (d & (1<<8)) { printf("Invariant TSC is supported\n"); } else{ printf("Invariant TSC is NOT supported\n"); } } else { printf("No support for Advanced Power Management Information in CPUID\n"); } return 0; }
Thomas29/11/2013 13:25, Monjalon :
Quoted 8 lines
29/11/2013 14:53, Dmitry Vyal :
Quoted 7 lines · Bruce
Quoted 1 line
[BR] Frequency changes should not affect timers for modern Intel CPUs.
The error frequency greatly reduces if I patch
loop limit as I described earlier or if I callrte_power_initandrte_power_freq_maxas Thomas suggested.
Quoted 7 lines
Quoted 2 lines
But the only way to get rid of them completely is to set performance governor.
Please check that your hardware do not support invariant TSC. It would explain why you need to fix frequency.
I attach a simple code to test CPU feature "Invariant TSC".
Note that this feature is called constant_tsc in /proc/cpuinfo:
grep --color -m1 constant_tsc /proc/cpuinfo
It is checked by check_tsc_flags() at DPDK initialization and should print a warning if missing:
http://dpdk.org/browse/dpdk/commit/?id=e7c8996e13b9abb706ea0de53271346f4f86ca03
Signature · 1 line
Thomas