Debugging EAL PCI / Driver Init

DISCUSSION Matthew Hall 3 participants 9 messages Aug 1, 2014 – Aug 3, 2014 9 messages not in the archive

Share & tools

Matthew Hall

author

2014-08-01 17:51 UTC #0 lore raw

Hello,

I am running into a problem where Eth driver init works fine in a sample app and finds my NICs, and the NICs appear in rte_eal_pci_dump(stdout) but they don't show up in rte_eth_dev_count() even after rte_eal_pci_probe() is called the same as the sample apps, so my app won't boot.

I have a lot of experience using the older versions of the DPDK where you had to call the PMD init functions manually but no experience with the later versions where the DPDK is supposed to init the PMDs itself automatically.

What do I have to do to dump the most possible debug output on why the driver list for my PCI devices always seems empty? Any places I should look to see the issue? Maybe I didn't link it together with the right DPDK libs? I used the combined DPDK static lib libintel_dpdk.a to make things simpler as I had seen recommended in various places.

Thanks,
Matthew.

A message that is not in the archive (sent off-list or not yet received): [email protected]

Matthew Hall

author

2014-08-02 15:29 UTC #2 lore raw

I did a bit more experimentation and found the following. If I unmark the rte_igb_pmd_init function as static, and call it directly from my code, the driver will load, and the port count increments to 2:

EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   PCI memory mapped at 0x7f09d45f2000
EAL:   PCI memory mapped at 0x7f09d4730000
PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x1521

EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   PCI memory mapped at 0x7f09d0f00000
EAL:   PCI memory mapped at 0x7f09d472c000
PMD: eth_igb_dev_init(): port_id 1 vendorID=0x8086 deviceID=0x1521

So it seems like when you enable these options:

# Combine to one single library
CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_LIBNAME="intel_dpdk"

You don't really get a working DPDK inside of "-lintel_dpdk". Someone suggested linking with "-Xlinker -lintel_dpdk" but that didn't seem to help.

Is there a secret to getting a single integrated static library, where all of the PMD's end up in the PCI driver list so rte_eal_pci_probe can find them? Or some secret to linking against the combined library which works properly?

Matthew.

Quoted 19 lines · Matthew Hall

Hello,
I am running into a problem where Eth driver init works fine in a sample app and finds my NICs, and the NICs appear in rte_eal_pci_dump(stdout) but they don't show up in rte_eth_dev_count() even after rte_eal_pci_probe() is called the same as the sample apps, so my app won't boot.
I have a lot of experience using the older versions of the DPDK where you had to call the PMD init functions manually but no experience with the later versions where the DPDK is supposed to init the PMDs itself automatically.
What do I have to do to dump the most possible debug output on why the driver list for my PCI devices always seems empty? Any places I should look to see the issue? Maybe I didn't link it together with the right DPDK libs? I used the combined DPDK static lib libintel_dpdk.a to make things simpler as I had seen recommended in various places.
Thanks,
Matthew.

A message that is not in the archive (sent off-list or not yet received): [email protected]

Matthew Hall

author

2014-08-02 15:43 UTC #4 lore raw

Also, when using the separate libraries the problem still happens:

-lethdev -lrte_cfgfile -lrte_cmdline -lrte_distributor -lrte_hash -lrte_ip_frag -lrte_lpm -lrte_malloc -lrte_mbuf -lrte_mempool -lrte_pmd_e1000 -lrte_pmd_pcap -lrte_pmd_virtio_uio -lrte_pmd_vmxnet3_uio -lrte_port -lrte_ring -lrte_table -lrte_timer -lrte_eal -lbsd -ldl -lpcap -lpthread

So it seems there is a special order or link technique which must be used, or rte_eal's PCI code won't be able to load the PMD's during PCI probing. Is it documented anywhere how to get that to work?

Thanks,
Matthew.

Quoted 51 lines · Matthew Hall

I did a bit more experimentation and found the following. If I unmark the rte_igb_pmd_init function as static, and call it directly from my code, the driver will load, and the port count increments to 2:
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   PCI memory mapped at 0x7f09d45f2000
EAL:   PCI memory mapped at 0x7f09d4730000
PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x1521
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   probe driver: 8086:1521 rte_igb_pmd
EAL:   PCI memory mapped at 0x7f09d0f00000
EAL:   PCI memory mapped at 0x7f09d472c000
PMD: eth_igb_dev_init(): port_id 1 vendorID=0x8086 deviceID=0x1521
So it seems like when you enable these options:
# Combine to one single library
CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_LIBNAME="intel_dpdk"
You don't really get a working DPDK inside of "-lintel_dpdk". Someone suggested linking with "-Xlinker -lintel_dpdk" but that didn't seem to help.
Is there a secret to getting a single integrated static library, where all of the PMD's end up in the PCI driver list so rte_eal_pci_probe can find them? Or some secret to linking against the combined library which works properly?
Matthew.
Quoted 19 lines · Matthew Hall
Hello,
I am running into a problem where Eth driver init works fine in a sample app and finds my NICs, and the NICs appear in rte_eal_pci_dump(stdout) but they don't show up in rte_eth_dev_count() even after rte_eal_pci_probe() is called the same as the sample apps, so my app won't boot.
I have a lot of experience using the older versions of the DPDK where you had to call the PMD init functions manually but no experience with the later versions where the DPDK is supposed to init the PMDs itself automatically.
What do I have to do to dump the most possible debug output on why the driver list for my PCI devices always seems empty? Any places I should look to see the issue? Maybe I didn't link it together with the right DPDK libs? I used the combined DPDK static lib libintel_dpdk.a to make things simpler as I had seen recommended in various places.
Thanks,
Matthew.

A message that is not in the archive (sent off-list or not yet received): CAFLKUzKOPxC4jQBG6C07QXJ=MrcydxG9FdHD6h7omj6BY=99LQ@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): CAFLKUzKOPxC4jQBG6C07QXJ=MrcydxG9FdHD6h7omj6BY=99LQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Matthew Hall

author

2014-08-02 16:46 UTC #7 lore raw

Quoted 4 lines · Masaru Oki

cc links library funtion from archive only if call from other object. but new dpdk pmd library has constractor section and not call directly. ld always links library funtion with constractor section. use -Xlinker, or use ld instead of cc.

Hello Oki-san,

The trick to fix it was this, I finally found it in the example Makefiles with V=1 flag.

-Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group -Wl,--no-whole-archive

Thank you for the advice you provided, I couldn't have fixed it without your suggestions... it got me to look more closely at the linking. Importantly, "-Wl,--whole-archive" includes the entire archive whether or not it's called from other objects, so we don't lose the constructors, just like you said.

Matthew.

A message that is not in the archive (sent off-list or not yet received): [email protected]

Alex Markuze

2014-08-03 10:38 UTC #9 lore raw

Hi Matt, Dev
I'm Trying to compile ann app linking to dpdk and dpdk based libs. And I'm seeing the same issue you've reported.
The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports. Same code compiled as a dpdk app works fine.

In your solution to this issue you are referring to -lintel_dpdk? I couldn't find any reference to it.

Thanks
Alex.

Quoted 19 lines · Matthew Hall

Quoted 4 lines · Masaru Oki
cc links library funtion from archive only if call from other object. but new dpdk pmd library has constractor section and not call directly. ld always links library funtion with constractor section. use -Xlinker, or use ld instead of cc.
Hello Oki-san,
The trick to fix it was this, I finally found it in the example Makefiles with V=1 flag.
-Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group -Wl,--no-whole-archive
Thank you for the advice you provided, I couldn't have fixed it without your suggestions... it got me to look more closely at the linking. Importantly, "-Wl,--whole-archive" includes the entire archive whether or not it's called from other objects, so we don't lose the constructors, just like you said.
Matthew.

A message that is not in the archive (sent off-list or not yet received): CAKfHP0VrDBJD9UeFwJurUNBfXTcBpZxa_VfeKBNJF56jY52LDA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Alex Markuze

2014-08-03 11:41 UTC #11 lore raw

Resolved just as Matt has described.
To remove any ambiguity (for future reference).

This line in the gcc command resolves the issue in my case (a different nic may need a different lib):
--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,

The problem is that the probe command polls over all pci devices and tries to find a matching driver, these drivers register With these macros which will only be called when the --whole-archive option is provided and its actually the reason for this flags existence. Without this flag the driver list is empty.

PMD_REGISTER_DRIVER(rte_ixgbe_driver);
PMD_REGISTER_DRIVER(rte_ixgbevf_driver);

Quoted 33 lines · Alex Markuze

Hi Matt, Dev
I'm Trying to compile ann app linking to dpdk and dpdk based libs. And I'm seeing the same issue you've reported.
The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports. Same code compiled as a dpdk app works fine.
In your solution to this issue you are referring to -lintel_dpdk? I couldn't find any reference to it.
Thanks
Alex.
Quoted 19 lines · Matthew Hall
Quoted 4 lines · Masaru Oki
cc links library funtion from archive only if call from other object. but new dpdk pmd library has constractor section and not call directly. ld always links library funtion with constractor section. use -Xlinker, or use ld instead of cc.
Hello Oki-san,
The trick to fix it was this, I finally found it in the example Makefiles with V=1 flag.
-Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group -Wl,--no-whole-archive
Thank you for the advice you provided, I couldn't have fixed it without your suggestions... it got me to look more closely at the linking. Importantly, "-Wl,--whole-archive" includes the entire archive whether or not it's called from other objects, so we don't lose the constructors, just like you said.
Matthew.

A message that is not in the archive (sent off-list or not yet received): CAKfHP0UQRtDLU4pDwSoJF0AJfhPFkuo6R6vLrHcAd20VRHWfWA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Matthew Hall

author

2014-08-03 17:23 UTC #13 lore raw

There is an option in the DPDK build config which compiles every DPDK lib into a single static lib. I don't have the name of the option in front of me but it had COMBINE in its name. When this option is used you can get every function in the whole DPDK with a single library. After that I had a lot fewer linking issues.

Matthew.

Signature · 1 line

Sent from my mobile device.

Quoted 60 lines · Alex Markuze

Resolved just as Matt has described.
To remove any ambiguity (for future reference).
This line in the gcc command resolves the issue in my case (a different nic may need a different lib):
--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,
The problem is that the probe command polls over all pci devices and tries to find a matching driver, these drivers register With these macros which will only be called when the --whole-archive option is provided and its actually the reason for this flags existence. Without this flag the driver list is empty.
PMD_REGISTER_DRIVER(rte_ixgbe_driver);
PMD_REGISTER_DRIVER(rte_ixgbevf_driver);
Quoted 14 lines · Alex Markuze
Hi Matt, Dev
I'm Trying to compile ann app linking to dpdk and dpdk based libs. And I'm seeing the same issue you've reported.
The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports. Same code compiled as a dpdk app works fine.
In your solution to this issue you are referring to -lintel_dpdk? I couldn't find any reference to it.
Thanks
Alex.
On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Masaru Oki
Quoted 1 line
cc links library funtion from archive only if call from other
object.
Quoted 1 line
Quoted 1 line
Quoted 1 line
but new dpdk pmd library has constractor section and not call
directly.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
ld always links library funtion with constractor section. use -Xlinker, or use ld instead of cc.
Hello Oki-san,
The trick to fix it was this, I finally found it in the example
Makefiles with
Quoted 3 lines
Quoted 3 lines
V=1 flag.
-Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group
-Wl,--no-whole-archive
Quoted 2 lines
Quoted 2 lines
Thank you for the advice you provided, I couldn't have fixed it
without your
Quoted 1 line
Quoted 1 line
suggestions... it got me to look more closely at the linking.
Importantly,
Quoted 1 line
Quoted 1 line
"-Wl,--whole-archive" includes the entire archive whether or not
it's called
Quoted 1 line
Quoted 1 line
from other objects, so we don't lose the constructors, just like you
said.
Quoted 2 lines
Quoted 2 lines
Matthew.

A message that is not in the archive (sent off-list or not yet received): bf0653a3-6e73-4e51-a626-cfb4544d3fe3-2ueSQiBKiTY7tOexoI0I+QC/[email protected]

Jayakumar, Muthurajan

2014-08-03 18:13 UTC #15 lore raw

Are you referring to CONFIG_RTE_BUILD_COMBINE_LIBS ?

(ps: referenced here http://dpdk.org/ml/archives/dev/2013-October/000639.html)

Thanks,
M Jay

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Matthew Hall
Sent: Sunday, August 03, 2014 10:23 AM
To: Alex Markuze
Cc: [email protected]
Subject: Re: [dpdk-dev] Debugging EAL PCI / Driver Init

Matthew.
--
Sent from my mobile device.

Quoted 60 lines · Alex Markuze

Resolved just as Matt has described.
To remove any ambiguity (for future reference).
This line in the gcc command resolves the issue in my case (a different nic may need a different lib):
--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,
The problem is that the probe command polls over all pci devices and tries to find a matching driver, these drivers register With these macros which will only be called when the --whole-archive option is provided and its actually the reason for this flags existence. Without this flag the driver list is empty.
PMD_REGISTER_DRIVER(rte_ixgbe_driver);
PMD_REGISTER_DRIVER(rte_ixgbevf_driver);
Quoted 14 lines · Alex Markuze
Hi Matt, Dev
I'm Trying to compile ann app linking to dpdk and dpdk based libs. And I'm seeing the same issue you've reported.
The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports. Same code compiled as a dpdk app works fine.
In your solution to this issue you are referring to -lintel_dpdk? I couldn't find any reference to it.
Thanks
Alex.
On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Masaru Oki
Quoted 1 line
cc links library funtion from archive only if call from other
object.
Quoted 1 line
Quoted 1 line
Quoted 1 line
but new dpdk pmd library has constractor section and not call
directly.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
ld always links library funtion with constractor section. use -Xlinker, or use ld instead of cc.
Hello Oki-san,
The trick to fix it was this, I finally found it in the example
Makefiles with
Quoted 3 lines
Quoted 3 lines
V=1 flag.
-Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group
-Wl,--no-whole-archive
Quoted 2 lines
Quoted 2 lines
Thank you for the advice you provided, I couldn't have fixed it
without your
Quoted 1 line
Quoted 1 line
suggestions... it got me to look more closely at the linking.
Importantly,
Quoted 1 line
Quoted 1 line
"-Wl,--whole-archive" includes the entire archive whether or not
it's called
Quoted 1 line
Quoted 1 line
from other objects, so we don't lose the constructors, just like you
said.
Quoted 2 lines
Quoted 2 lines
Matthew.

A message that is not in the archive (sent off-list or not yet received): 5D695A7F6F10504DBD9B9187395A21797D0F92FA-8oqHQFITsIFcIJlls4ac1rfspsVTdybXVpNB7YpNyf8@public.gmane.org

Matthew Hall

author

2014-08-03 21:39 UTC #17 lore raw

Yes, that's the one! It made a lot of my linking problems go away when I used that along with the whole-archive flags.

Matthew

Signature · 1 line

Sent from my mobile device.

Quoted 92 lines · Muthurajan

Are you referring to CONFIG_RTE_BUILD_COMBINE_LIBS ?
(ps: referenced here
http://dpdk.org/ml/archives/dev/2013-October/000639.html)
Thanks,
M Jay
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Matthew Hall
Sent: Sunday, August 03, 2014 10:23 AM
To: Alex Markuze
Cc: [email protected]
Subject: Re: [dpdk-dev] Debugging EAL PCI / Driver Init
There is an option in the DPDK build config which compiles every DPDK lib into a single static lib. I don't have the name of the option in front of me but it had COMBINE in its name. When this option is used you can get every function in the whole DPDK with a single library. After that I had a lot fewer linking issues.
Matthew.
--
Sent from my mobile device.
Quoted 4 lines · Alex Markuze
Resolved just as Matt has described.
To remove any ambiguity (for future reference).
This line in the gcc command resolves the issue in my case (a
different
Quoted 46 lines
nic may need a different lib):
--whole-archive -Wl,--start-group -lrte_pmd_ixgbe -Wl,--end-group -Wl,
The problem is that the probe command polls over all pci devices and tries to find a matching driver, these drivers register With these macros which will only be called when the --whole-archive option is provided and its actually the reason for this flags existence. Without
this flag the driver list is empty.
PMD_REGISTER_DRIVER(rte_ixgbe_driver);
PMD_REGISTER_DRIVER(rte_ixgbevf_driver);
Quoted 15 lines · Alex Markuze
Hi Matt, Dev
I'm Trying to compile ann app linking to dpdk and dpdk based libs. And I'm seeing the same issue you've reported.
The probe function doesn't seem to find any ixgbevf(SRIOV VM) ports. Same code compiled as a dpdk app works fine.
In your solution to this issue you are referring to -lintel_dpdk? I
couldn't find any reference to it.
Thanks
Alex.
On Sat, Aug 2, 2014 at 7:46 PM, Matthew Hall <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines · Masaru Oki
Quoted 1 line
cc links library funtion from archive only if call from other
object.
Quoted 1 line
Quoted 1 line
Quoted 1 line
but new dpdk pmd library has constractor section and not call
directly.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
ld always links library funtion with constractor section. use -Xlinker, or use ld instead of cc.
Hello Oki-san,
The trick to fix it was this, I finally found it in the example
Makefiles with
Quoted 3 lines
Quoted 3 lines
V=1 flag.
-Wl,--whole-archive -Wl,--start-group -lintel_dpdk -Wl,--end-group
Quoted 9 lines
-Wl,--no-whole-archive
Quoted 2 lines
Quoted 2 lines
Thank you for the advice you provided, I couldn't have fixed it
without your
Quoted 1 line
Quoted 1 line
suggestions... it got me to look more closely at the linking.
Importantly,
Quoted 1 line
Quoted 1 line
"-Wl,--whole-archive" includes the entire archive whether or not
it's called
Quoted 1 line
Quoted 1 line
from other objects, so we don't lose the constructors, just like
you
Quoted 3 lines
said.
Quoted 2 lines
Quoted 2 lines
Matthew.