Can I/OAT DMA engineer access PCI MMIO space
I try to use ioatdam to copy data from system memory to pci MMIO space:
=======================
void mmio_copy(void *dst, void *src, size_t len) struct async_submit_ctl submit;
struct memcpy_context ctx;
atomic_set(&ctx.cnt, 1); init_completion(&ctx.cmp); init_async_submit(&submit, ASYNC_TX_ACK, NULL, done_memcpy, &ctx, NULL);
while (len) {size_t clen = (len > PAGE_SIZE) ? PAGE_SIZE : len; struct page *p_dst, *p_src;
p_dst = virt_to_page(dst);p_src = virt_to_page(src);
atomic_inc(&ctx.cnt);
ntb_async_memcpy(p_dst, p_src, 0, 0, clen, &submit);
dst += clen;
src += clen;
len -= clen;
}
if (atomic_dec_and_test(&ctx.cnt))
return;
async_tx_issue_pending_all();wait_for_completion(&ctx.cmp);
=================
If dst points to a memory space, the operation would pass. But if dst points to a pci MMIO space, it failed with kernel oops. It seems the code:
BUG_ON(is_ioat_bug(chanerr));
in drivers/dma/ioat/dma_v3.c, line 365 cause the oops. Is there anyway to access pci MMIO space using ioat? The datasheet says that ioat supports MMIO access.
Quoted 9 lines · 康剑斌
I try to use ioatdam to copy data from system memory to pci MMIO space:
If dst points to a memory space, the operation would pass. But if dst points to a pci MMIO space, it failed with kernel oops. It seems the code:
BUG_ON(is_ioat_bug(chanerr));
indrivers/dma/ioat/dma_v3.c, line 365 cause the oops. Is there anyway to access pci MMIO space using ioat? The datasheet says that ioat supports MMIO access.
Did you map the IO memory in kernel using ioremap and friends first?
Signature · 1 line
~Vinod
Quoted 11 lines
Quoted 9 lines
I try to use ioatdam to copy data from system memory to pci MMIO space:
If dst points to a memory space, the operation would pass. But if dst points to a pci MMIO space, it failed with kernel oops. It seems the code:
BUG_ON(is_ioat_bug(chanerr));
indrivers/dma/ioat/dma_v3.c, line 365 cause the oops.
Is there anyway to access pci MMIO space using ioat?
The datasheet says that ioat supports MMIO access.Did you map the IO memory in kernel using ioremap and friends first?
yes, I had used 'ioremap_nocache' to map the IO memory and I can use
memcpy to copy data to this region. The async_tx should have been correctly configured as I can use aync_memcpy to copy data between different system memory address.
Quoted 8 lines · 康剑斌
Quoted 8 lines
Quoted 8 lines
I try to use ioatdam to copy data from system memory to pci MMIO space:
If dst points to a memory space, the operation would pass. But if dst points to a pci MMIO space, it failed with kernel oops. It seems the code:
BUG_ON(is_ioat_bug(chanerr));
indrivers/dma/ioat/dma_v3.c, line 365 cause the oops.
Is there anyway to access pci MMIO space using ioat?
The datasheet says that ioat supports MMIO access.
Quoted 6 lines
Quoted 2 lines
Did you map the IO memory in kernel using ioremap and friends first?
yes, I had used '
ioremap_nocache' to map the IO memory and I can use memcpy to copy data to this region. Theasync_txshould have been correctly configured as
I can useaync_memcpyto copy data between different system memory address.
Then you should be using memcpy_toio() and friends
~Vinod
Signature · 1 line
~Vinod
Quoted 7 lines
Quoted 5 lines
yes, I had used '
ioremap_nocache' to map the IO memory and I can use memcpy to copy data to this region. Theasync_txshould have been correctly configured as
I can useaync_memcpyto copy data between different system memory address.Then you should be using
memcpy_toio()and friends
Do you mean that if I have mapped the mmio, I can' use I/OAT dma transfer to this region any more? I can use memcpy to copy data, but it consumes lots of cpu as PCI access is too slow. If I can use i/oat dma and asyc_tx api to do the job, the performance should be imporved. Thanks
Quoted 7 lines · 康剑斌
Quoted 7 lines
Quoted 5 lines
yes, I had used '
ioremap_nocache' to map the IO memory and I can use memcpy to copy data to this region. Theasync_txshould have been correctly configured as
I can useaync_memcpyto copy data between different system memory address.Then you should be using
memcpy_toio()and friends
Quoted 7 lines
Do you mean that if I have mapped the mmio, I can' use I/OAT dma transfer to this region any more?
I can use memcpy to copy data, but it consumes lots of cpu as PCI access is too slow.
If I can use i/oat dma andasyc_txapi to do the job, the performance should be imporved.
Thanks
The async_tx api only supports memory-to-memory transfers. To write to mmio space with ioatdma you would need a custom method, like the dma-slave support in other drivers, to program the descriptors with the physical mmio bus address.
--
Dan
Quoted 9 lines · 康剑斌
Quoted 9 lines
Quoted 9 lines
I try to use ioatdam to copy data from system memory to pci MMIO space:
If dst points to a memory space, the operation would pass. But if dst points to a pci MMIO space, it failed with kernel oops. It seems the code:
BUG_ON(is_ioat_bug(chanerr));
indrivers/dma/ioat/dma_v3.c, line 365 cause the oops.
Is there anyway to access pci MMIO space using ioat?
The datasheet says that ioat supports MMIO access.
Quoted 7 lines
Quoted 2 lines
Did you map the IO memory in kernel using ioremap and friends first?
yes, I had used '
ioremap_nocache' to map the IO memory and I can use memcpy to copy data to this region. Theasync_txshould have been correctly configured as
I can useaync_memcpyto copy data between different system memory address.
ioremap maps mmio space for the cpu not bus mastering devices. You need to program the physical address into the descriptor.
--
Dan
Quoted 10 lines · Dan Williams
Quoted 8 lines
Do you mean that if I have mapped the mmio, I can' use I/OAT dma transfer to this region any more?
I can use memcpy to copy data, but it consumes lots of cpu as PCI access is too slow.
If I can use i/oat dma andasyc_txapi to do the job, the performance should be imporved.
Thanks
Quoted 4 lines
The
async_txapi only supports memory-to-memory transfers. To write to mmio space with ioatdma you would need a custom method, like the dma-slave support in other drivers, to program the descriptors with the physical mmio bus address.Signature · 1 line
Dan
Thanks.
I directly read pci bar address and program it into descriptors, ioatdma works.
Some problem is, when PCI transfer failed (Using a NTB connect to another system, and the system power down), ioatdma will cause kernel oops.
BUG_ON(is_ioat_bug(chanerr));
in drivers/dma/ioat/dma_v3.c, line 365
It seems that HW reports a 'IOAT_CHANERR_DEST_ADDR_ERR', and drivers can't recover from this situation. What does dma-slave mean? Just like DMA_SLAVE flag existing in other DMA drivers?
[ adding Dave ]
Quoted 7 lines · 康剑斌
Thanks.
I directly read pci bar address and program it into descriptors, ioatdma works.
Some problem is, when PCI transfer failed (Using a NTB connect to another system, and the system power down),
ioatdma will cause kernel oops.
Quoted 5 lines
BUG_ON(is_ioat_bug(chanerr));
indrivers/dma/ioat/dma_v3.c, line 365It seems that HW reports a '
IOAT_CHANERR_DEST_ADDR_ERR', and drivers can't recover from this situation.
Ah ok, this is expected with the current upstream ioatdma driver. The driver assumes that all transfers are mem-to-mem (ASYNC_TX_DMA or NET_DMA) and that a destination address error is a fatal error (similar to a kernel page fault).
With NTB, where failures are expected, the driver would need to be modified to expect the error, recover from it, and report it to the application.
Quoted 2 lines
What does dma-slave mean? Just like
DMA_SLAVEflag existing in other DMA drivers?
Yes, DMA_SLAVE is the generic framework to associate a dma offload device with an mmio peripheral.
--
Dan