DCA
Hi,
I would like to ask if there is any reason why DPDK doesn't have support for DCA feature?
thanks,
vlad
A message that is not in the archive (sent off-list or not yet received): 5534CFFF.7000404-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org
Quoted 6 lines · Vlad Zolotarov
Hi,
I would like to ask if there is any reason why DPDK doesn't have support for DCA feature?thanks,
vlad
With modern platforms with DDIO the data written by the NIC automatically goes into the cache of the CPU without us needing to use DCA.
/Bruce
Quoted 9 lines · Bruce Richardson
Quoted 6 lines · Vlad Zolotarov
Hi,
I would like to ask if there is any reason why DPDK doesn't have support for DCA feature?thanks,
vladWith modern platforms with DDIO the data written by the NIC automatically goes into the cache of the CPU without us needing to use DCA.
Thanks for a reply, Bruce.
One question though. According to DDIO documentation it only affects the CPUs "local" relatively to the NIC. DCA, on the other hand may be configured to work with any CPU. Modern platforms usually have a few NUMA nodes and requirement of binding network handling threads only to CPUs "local" to the NIC is very limiting.
Could u, pls., comment on this?
thanks in advance,
vlad
Quoted 2 lines
/Bruce
A message that is not in the archive (sent off-list or not yet received): 55360F9C.7070601-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org
Quoted 19 lines · Vlad Zolotarov
Quoted 9 lines · Bruce Richardson
Quoted 6 lines · Vlad Zolotarov
Hi,
I would like to ask if there is any reason why DPDK doesn't have support for DCA feature?thanks,
vladWith modern platforms with DDIO the data written by the NIC automatically goes into the cache of the CPU without us needing to use DCA.
Thanks for a reply, Bruce.
One question though. According to DDIO documentation it only affects the CPUs "local" relatively to the NIC. DCA, on the other hand may be configured to work with any CPU. Modern platforms usually have a few NUMA nodes and requirement of binding network handling threads only to CPUs "local" to the NIC is very limiting.
Quoted 5 lines
Could u, pls., comment on this?
thanks in advance,
vlad
My main comment is that yes, you are correct. DDIO only works with the local socket, while DCA can be made to work with remote sockets. If you need to do polling on a device from a remote socket you may need to look at DCA.
Can you perhaps comment on the use-case where you find this binding limiting? Modern platforms have multiple NUMA nodes, but they also generally have PCI slots connected to those multiple NUMA nodes also, so that you can have your NIC ports similarly NUMA partitionned?
/Bruce
Quoted 24 lines · Bruce Richardson
Quoted 18 lines · Vlad Zolotarov
Quoted 9 lines · Bruce Richardson
Quoted 6 lines · Vlad Zolotarov
Hi,
I would like to ask if there is any reason why DPDK doesn't have support for DCA feature?thanks,
vladWith modern platforms with DDIO the data written by the NIC automatically goes into the cache of the CPU without us needing to use DCA.
Thanks for a reply, Bruce.
One question though. According to DDIO documentation it only affects the CPUs "local" relatively to the NIC. DCA, on the other hand may be configured to work with any CPU. Modern platforms usually have a few NUMA nodes and requirement of binding network handling threads only to CPUs "local" to the NIC is very limiting.Quoted 5 lines
Could u, pls., comment on this?
thanks in advance,
vlad
Quoted 8 lines
My main comment is that yes, you are correct. DDIO only works with the local socket, while DCA can be made to work with remote sockets. If you need to do polling on a device from a remote socket you may need to look at DCA.
Can you perhaps comment on the use-case where you find this binding limiting? Modern platforms have multiple NUMA nodes, but they also generally have PCI slots connected to those multiple NUMA nodes also, so that you can have your NIC ports similarly NUMA partitionned?
The immediate example where this could be problematic is an AWS Guest with Enhanced Netowrking case: in c3.8xlarge instance u get a 2 NUMA nodes, 32 CPU cores and u can bind as many 82599 Intel VFs as u need, each providing 4 Rx and 4 Tx queues. AFAIR nothing is promised about the locality of PFs VFs belong to. To utilize all CPUs we'll need 4 or 8 VFs depending on the queues layout we decide (a separate CPU for each queue or a separate CPU for each Rx + Tx queue pair). In this case u may get absolutely different NUMA layouts:
- all VFs reside on the same PF: half of the queues will be remote
to one of the NUMA node or all of them are remote to all CPUs.
- VFs come from two PFs which may reside in the same NUMA nodes as
CPUs or not...
- VFs come from more than two different PFs...
So, in the above example DCA would cover all our needs while DDIO won't be able to cover them in most of the cases.
Quoted 2 lines
/Bruce
Quoted 4 lines · Bruce Richardson
Can you perhaps comment on the use-case where you find this binding limiting? Modern platforms have multiple NUMA nodes, but they also generally have PCI slots connected to those multiple NUMA nodes also, so that you can have your NIC ports similarly NUMA partitionned?
Hi Bruce,
I was wondering if you have tried to do this on COTS (commerical off-the-shelf) hardware before. What I found each time I tried it was that PCIe slots are not very evenly distributed across the NUMA nodes unlike what you'd expect.
Sometimes the PCIe lanes on CPU 0 get partly used up by Super IO or other integrated peripherals. Other times the motherboards give you 2 x8 when you needed 1 x16 or they give you a bundh of x4 when you needed x8, etc.
It's actually pretty difficult to find the mapping, for one, and even when you do, even harder to get the right slots for your cards and so on. In the ixgbe kernel driver you'll sometimes get some cryptic debug prints when it's been munged and performance will suffer. But in the ixgbe PMD driver you're on your own mostly.
Matthew.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Matthew Hall
Quoted 4 lines · Bruce Richardson
Can you perhaps comment on the use-case where you find this binding limiting? Modern platforms have multiple NUMA nodes, but they also generally have PCI slots connected to those multiple NUMA nodes also, so that you can have your NIC ports similarly NUMA partitionned?
Quoted 7 lines
Hi Bruce,
I was wondering if you have tried to do this on COTS (commerical off-the-shelf) hardware before. What I found each time I tried it was that PCIe slots are not very evenly distributed across the NUMA nodes unlike what you'd expect.
I doubt I've tried it on regular commercial boards as much as you guys have, though it does happen!
Quoted 3 lines
Sometimes the PCIe lanes on CPU 0 get partly used up by Super IO or other integrated peripherals. Other times the motherboards give you 2 x8 when you needed 1 x16 or they give you a bundh of x4 when you needed x8, etc.
Point taken!
Quoted 6 lines
It's actually pretty difficult to find the mapping, for one, and even when you do, even harder to get the right slots for your cards and so on. In the ixgbe kernel driver you'll sometimes get some cryptic debug prints when it's been munged and performance will suffer. But in the ixgbe PMD driver you're on your own mostly.
It was to try and make the NUMA mapping of PCI clearer that we added in the printing of the NUMA node on PCI scan:
EAL: PCI device 0000:86:00.0 on NUMA socket 1 EAL: probe driver: 8086:154a rte_ixgbe_pmd EAL: PCI memory mapped at 0x7fb452f04000 EAL: PCI memory mapped at 0x7fb453004000
Is there something more than this you feel we could do in the PMD to help with slot identification?
/Bruce
Quoted 2 lines
Matthew.