Skip to contentSkip to conversation

how to get a DMA channel near a process?

DISCUSSION Brice Goglin 3 participants 4 messages Aug 25, 2009 – Aug 26, 2009

lore
Share & tools
lore
#0 lore raw

Hello,

I am playing with DMA engine on a Nehalem box with two X58 chipsets (Supermicro X8DAH). My understanding is that there are 8 DMA channels on each chipset, so 8 channels near each processor. Unfortunately, my BIOS and 2.6.31-rc still wrongly reports the physical location of my devices (it claims all PCI devices are near the first processor) but I worked around the problem manually.

The offloaded copy performance changes a lot depending on whether the process memory is allocated near the DMA device. So first I would like to know if DMA channels are allocated near the requesting processor/process. Then I guess it's possible to read the cpu mask near a given chan by following chan->dev.device up to the pci device, right? But is there any way to request a DMA channel near a specific socket or NUMA node or cpu mask?

Thanks,
Brice

#1 lore raw
Quoted 5 lines · Brice Goglin

Hello,

I am playing with DMA engine on a Nehalem box with two X58 chipsets (Supermicro X8DAH). My understanding is that there are 8 DMA channels on each chipset, so 8 channels near each processor. Unfortunately, my BIOS

Don't think that DMA channel is the term you are looking for, maybe PCI Express lanes? DMA channels are an ancient ISA thing.

Quoted 3 lines

and 2.6.31-rc still wrongly reports the physical location of my devices (it claims all PCI devices are near the first processor) but I worked around the problem manually.

Presumably incorrect information in the BIOS tables (SLIT or MADT, maybe?)

Quoted 5 lines

The offloaded copy performance changes a lot depending on whether the process memory is allocated near the DMA device. So first I would like to know if DMA channels are allocated near the requesting processor/process. Then I guess it's possible to read the cpu mask near

Not sure quite what you are trying to say here. Normally when memory is allocated for DMA use for a specific device I believe the kernel tries to allocate it on the same node that the device is attached to.

Quoted 8 lines

a given chan by following chan->dev.device up to the pci device, right? But is there any way to request a DMA channel near a specific socket or NUMA node or cpu mask?

Thanks,
Brice

#2 lore raw
Quoted 9 lines · Robert Hancock
Quoted 5 lines · Brice Goglin

Hello,

I am playing with DMA engine on a Nehalem box with two X58 chipsets (Supermicro X8DAH). My understanding is that there are 8 DMA channels on each chipset, so 8 channels near each processor. Unfortunately, my BIOS

Don't think that DMA channel is the term you are looking for, maybe PCI Express lanes? DMA channels are an ancient ISA thing.

Talking about DMA Engine devices/channels such as Intel I/OAT. See include/linux/dmaengine.h

Brice

#3 lore raw
In reply to Brice Goglin
Quoted 9 lines · Brice Goglin

Hello,

I am playing with DMA engine on a Nehalem box with two X58 chipsets (Supermicro X8DAH). My understanding is that there are 8 DMA channels on each chipset, so 8 channels near each processor. Unfortunately, my BIOS and 2.6.31-rc still wrongly reports the physical location of my devices (it claims all PCI devices are near the first processor) but I worked around the problem manually.

Quoted 7 lines

The offloaded copy performance changes a lot depending on whether the process memory is allocated near the DMA device. So first I would like to know if DMA channels are allocated near the requesting processor/process. Then I guess it's possible to read the cpu mask near a given chan by following chan->dev.device up to the pci device, right? But is there any way to request a DMA channel near a specific socket or NUMA node or cpu mask?

If we can determine the topology of which chipset is closer to which cpu package then we can modify the dmaengine channel_table to return "local" channels via dma_find_channel(). The current allocation policy is blind per-cpu round robin such that cpu-0 gets channel-0, cpu-1 gets channel-1... As you pointed out the challenge will be finding a reliable source for the topology information.

--
Dan