Any ideas how to stop DPDK from banning me from the box.

Greetings!

I've been playing with dpdk on my desktop for some time and decided to finally test in on a server. It has a plenty of nics:

dev@box:~$ lspci |grep Ether
01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0c:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0c:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
12:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
82:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
82:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
8a:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
8a:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

I connect to the server using ssh through one of gigabit ports. The problem is that no matter which cards I blacklist with -b option then starting my dpdk application it takes away all the interfaces from the kernel and later returns blacklisted ones. This disrupts my ssh session and I can no longer connect to the server.

Any ideas how to overcome this?

Best regards,
Dmitry Vyal.

Dmitry,

Please post the exact blacklist option. I use the '-b' option all the time with my test box (Dell T620).

Thanks,

Patrick

Sent from my iPad

Il 18/07/2013 17:42, Dmitry Vyal ha scritto:

Hi

I do the very same thing and I have no such problem.

Maybe you are using wrong IDs, verify that.

Here a working example:

pino:~/dpdk-1.2.3r4 ~> lspci | grep -i ethernet
02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) 03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 04:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

pino:~/dpdk-1.2.3r4 ~> sudo build/app/testpmd -c 0xf -n 3 -r 1 -b 0000:05:00.0 -b 0000:05:00.1
EAL: coremask set to f
...

Initializing port 0... done:  Link Down
Initializing port 1... done:  Link Down
Initializing port 2... done:  Link Up - speed 10000 Mbps - full-duplex
Initializing port 3... done:  Link Up - speed 10000 Mbps - full-duplex

No commandline core given, start packet forwarding
io packet forwarding - CRC stripping disabled - packets/burst=16 nb forwarding cores=1 - nb forwarding ports=4
RX queues=1 - RX desc=128 - RX free threshold=0
RX threshold registers: pthresh=8 hthresh=8 wthresh=4 TX queues=1 - TX desc=512 - TX free threshold=0
TX threshold registers: pthresh=36 hthresh=0 wthresh=0 TX RS bit threshold=0
Press enter to exit

[ssh is still up]

Regards,
Marco

Hi Marco,

thanks for posting the example I carefully composed mine using your as a basis and, guess, all works I don't have the exact cmdline I issued yesterday. Looks like bash didn't have a chance to save history But I guess I put all the -b options after the -- delimiter so they weren't parsed by rte_eal_init. You saved me a day!

Il 19/07/2013 09:11, Dmitry Vyal ha scritto:

Hi Dmitry,

I'm glad I could help!
By the way, you can also avoid using the PMD igb driver entirely if you don't need it (I don't as I just care about 10Gbit interfaces, but since you have lots of 82576 NICs I'm not that sure); just change your compile configuration, either the default one or a custom one (depending on what you are using):

pino:~/dpdk-1.2.3r4 -> grep IGB config/defconfig_x86_64-marco-linuxapp-gcc # Compile burst-oriented IGB PMD driver CONFIG_RTE_LIBRTE_IGB_PMD=n
CONFIG_RTE_LIBRTE_IGB_DEBUG_INIT=n CONFIG_RTE_LIBRTE_IGB_DEBUG_RX=n CONFIG_RTE_LIBRTE_IGB_DEBUG_TX=n CONFIG_RTE_LIBRTE_IGB_DEBUG_TX_FREE=n CONFIG_RTE_LIBRTE_IGB_DEBUG_DRIVER=n

As you can see here I keep using the kernel space igb driver:

pino:~/dpdk-1.2.3r4 -> lspci -kD | grep -A 2 -i ether
0000:02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
Subsystem: Mellanox Technologies Device 0016
Kernel driver in use: mlx4_core
0000:03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Kernel driver in use: igb_uio
0000:03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Kernel driver in use: igb_uio
0000:04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Kernel driver in use: igb_uio
0000:04:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Kernel driver in use: igb_uio
0000:05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
Subsystem: Super Micro Computer Inc Device 10c9
Kernel driver in use: igb
0000:05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
Subsystem: Super Micro Computer Inc Device 10c9
Kernel driver in use: igb

Regards,
Marco

Semi-related: why does DPDK take a blacklist instead of a list of devices that it should use (perhaps with a special value "all" in case someone really wants that behavior)? Historical reasons?

(yes, I've also bricked my ssh connections several times)

Antti, you are right. Should you need to make it better with a device name, your patch can be included.

About such ssh connection, we did not feel the pain at 6WIND since we have a stack that manages such issues so it does not break, but your request is valid.

Looking for any patch that would help,
Vincent