Skip to contentSkip to conversation

scatter-gather ops within host memory on a PC

OT Mark Frazer 2 participants 3 messages May 25, 2004

lore
Share & tools
lore
#0 lore raw

I'd like to aggregate a bunch of ethernet packet payloads into a single large buffer after examining the packet headers to determine the order of the packets. The data volume makes offloading this to a DMA controller desirable.

Do PC chipsets have DMA controllers on them to do these scatter-gather operations? I know a lot of embedded SoC processors do, but am not to familiar with the current PC chipsets.

I've glanced at the NFS code but haven't studied it a lot. rxrpc/transport.c seems to copy some data around, so I assume that linux uses the host processor to gather the individual ethernet packets into pages that go into the page cache.

Are there any good books anyone would recommend on PC architecture?

thanks
-mark

Signature · 1 line
The laws of science be a harsh mistress. - Bender
#1 lore raw
Quoted 14 lines · Mark Frazer

I'd like to aggregate a bunch of ethernet packet payloads into a single large buffer after examining the packet headers to determine the order of the packets. The data volume makes offloading this to a DMA controller desirable.

Do PC chipsets have DMA controllers on them to do these scatter-gather operations? I know a lot of embedded SoC processors do, but am not to familiar with the current PC chipsets.

I've glanced at the NFS code but haven't studied it a lot. rxrpc/transport.c seems to copy some data around, so I assume that linux uses the host processor to gather the individual ethernet packets into pages that go into the page cache.

Quoted 5 lines

Are there any good books anyone would recommend on PC architecture?

thanks
-mark
--

Scatter-lists with DMA operations are quite common. Many Ethernet controllers that perform bus-master DMA use a chip that is either a (PLX) PCI 9080 or pretty-much a clone.

The problem is that DMA does data-transfer to physical locations. If you only have one physical location (a page in DMA-able memory) its virtual to physical translation needs to be done only once. But, if you expect to write to/from user-mode buffers, the translation needs to be done every time the scatter-list is built. This is not some simple operation since user's pages are assembled from anywhere into what looks like contiguous space to the user, but physically can be anywhere. So somebody has got to scan a bunch of page-tables looking for the user's pages every time you build the scatter-list, i.e., every DMA transfer. This will surely take much more time than the current scheme.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.

#2 lore raw

Richard B. Johnson <[email protected]> [04/05/25 13:52]:

Quoted 3 lines

Scatter-lists with DMA operations are quite common. Many Ethernet controllers that perform bus-master DMA use a chip that is either a (PLX) PCI 9080 or pretty-much a clone.

Yup, you're talking about PCI adapter to host scatter gather. I was staying within the host memory. I was wanting to use the standard ethernet drivers to do the transfer from the cards to the host. Then peek at the headers of the rx'd skbufs to set up another scatter-gather operation to unmix the possibly out-of-order and certainly discontiguous in host memory packets.

Quoted 11 lines

The problem is that DMA does data-transfer to physical locations. If you only have one physical location (a page in DMA-able memory) its virtual to physical translation needs to be done only once. But, if you expect to write to/from user-mode buffers, the translation needs to be done every time the scatter-list is built. This is not some simple operation since user's pages are assembled from anywhere into what looks like contiguous space to the user, but physically can be anywhere. So somebody has got to scan a bunch of page-tables looking for the user's pages every time you build the scatter-list, i.e., every DMA transfer. This will surely take much more time than the current scheme.

You're getting at DIO: get_user_pages, pci_map_sg, sg_dma_address, sg_dma_len and friends. I can't quite do that for this application as I want to reorder the data in the driver and present the application with the in-order reassembled image. The destination buffer for the un-mixing SG operation will be the result of get_user_pages and friends though.

cheers
-mark

Signature · 1 line
They're great! They're like sex except I'm having them. - Fry