Skip to contentSkip to conversation

New API for non cache coherent ppc cpu's

DISCUSSION Armin Kuster 4 participants 11 messages Nov 20, 2001 – Nov 23, 2001

lore
Share & tools
lore
#0 lore raw

To all NOT_COHERENT_CACHE users

About a month ago a new API made its way into the ppc, consistent_sync_page. For CONFIG_NOT_COHERENT_CACHE processors, requires
proper flushing of the page being used. Please review and provide feed back

-- armin

sample from patch

void consistent_sync_page(struct page *page, unsigned long offset, size_t size, int direction)

{
        unsigned long start = (unsigned long)page->virtual;
        unsigned long end   = start + size;

        switch (direction) {
        case PCI_DMA_NONE:
                BUG();
        case PCI_DMA_FROMDEVICE:        /* invalidate only */
                invalidate_dcache_range(start, end);
                break;
        case PCI_DMA_TODEVICE:          /* writeback only */
                clean_dcache_range(start, end);
                break;
        case PCI_DMA_BIDIRECTIONAL:     /* writeback and invalidate */
                flush_dcache_range(start, end);
                break;
        }

}

#1 lore raw

Hi,

Quoted 2 lines · Armin Kuster

About a month ago a new API made its way into the ppc, consistent_sync_page.

Another one?
We have now 3 APIs for this:
- cache_(push|clear): that's the old m68k API (and also used by APUS)
- dma_cache_(inv|wback|wback_inv): used by mips(64), parisc, sh
- consistent_sync_page: ppc

Quoted 4 lines

For CONFIG_NOT_COHERENT_CACHE processors,
requires
proper flushing of the page being used. Please review and provide feed back

Two comments:
- I'm not sure about the page argument, although I'd like to see it, the problem is the drivers usually don't have a page pointer.
- I think it was never defined, what should be done if offset/size isn't cache line aligned. That's especially a problem in the invalidate only case. I'd prefer to make this illegal, as it's mostly not a problem for drivers.

bye, Roman

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#2 lore raw
Quoted 14 lines · Roman Zippel

Hi,

Quoted 2 lines · Armin Kuster

About a month ago a new API made its way into the ppc, consistent_sync_page.

Another one?
We have now 3 APIs for this:
- cache_(push|clear): that's the old m68k API (and also used by APUS)
- dma_cache_(inv|wback|wback_inv): used by mips(64), parisc, sh
- consistent_sync_page: ppc

Quoted 8 lines
Quoted 4 lines

For CONFIG_NOT_COHERENT_CACHE processors,
requires
proper flushing of the page being used. Please review and provide feed back

Two comments:
- I'm not sure about the page argument, although I'd like to see it, the problem is the drivers usually don't have a page pointer.

example as it is used today.

mapping = pci_map_page(pdev,

			       virt_to_page(cmd->request_buffer),
			       ((unsigned long)cmd->request_buffer &
				~PAGE_MASK),
			       cmd->request_bufflen, dma_dir);

/*
* pci_{map,unmap}_single_page maps a kernel page to a dma_addr_t.
identical
* to pci_map_single, but takes a struct page instead of a virtual
address
*/
static inline dma_addr_t pci_map_page(struct pci_dev *hwdev, struct page *page,
unsigned long offset, size_t size, int direction)

{
	if (direction == PCI_DMA_NONE)
		BUG();
	consistent_sync_page(page, offset, size, direction);
	return (page - mem_map) * PAGE_SIZE + PCI_DRAM_OFFSET + offset;
}
Quoted 5 lines

- I think it was never defined, what should be done if offset/size isn't cache line aligned. That's especially a problem in the invalidate only case. I'd prefer to make this illegal, as it's mostly not a problem for drivers.

wouldn't it also be a problem for consistent_sync? we don't check for offset/size are cache line aligned.

Quoted 1 line

bye, Roman

-- armin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#3 lore raw

Hi,

Quoted 2 lines · Armin Kuster

mapping = pci_map_page(pdev,

                               virt_to_page(cmd->request_buffer),

This is the problem, virt_to_page doesn't work kmap'ed highmem pages.

Quoted 7 lines
Quoted 4 lines

- I think it was never defined, what should be done if offset/size isn't cache line aligned. That's especially a problem in the invalidate only case. I'd prefer to make this illegal, as it's mostly not a problem for drivers.

wouldn't it also be a problem for consistent_sync? we don't check for offset/size are cache line aligned.

Yes, same problem.

bye, Roman

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#4 lore raw
In reply to Armin Kuster
Quoted 7 lines · Armin Kuster

To all NOT_COHERENT_CACHE users

About a month ago a new API made its way into the ppc, consistent_sync_page. For CONFIG_NOT_COHERENT_CACHE processors, requires
proper flushing of the page being used. Please review and provide feed back

If we have to have a consistent_sync_page, it should be purely a local function in our implementation of the official DMA mapping API - see Documentation/DMA-mapping.txt. Drivers should be using functions such as pci_alloc_consistent, pci_map_single, pci_dma_sync_single, pci_unmap_single, etc. The implementation of those routines should do the correct cache flushing - if it doesn't then we need to fix it.

If you're talking about non-PCI devices, use the pci DMA API but just pass NULL for the dev (we need to make sure that will work ok on the non-cache-coherent cpus).

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#5 lore raw
Quoted 3 lines · Paul Mackerras

If we have to have a consistent_sync_page, it should be purely a local function in our implementation of the official DMA mapping API - see Documentation/DMA-mapping.txt.

That's exactly what it is. We are not creating a new API, and it shouldn't have been described as such.

Quoted 4 lines

.... Drivers should be using functions such
as pci_alloc_consistent, pci_map_single, pci_dma_sync_single, pci_unmap_single, etc. The implementation of those routines should do the correct cache flushing - if it doesn't then we need to fix it.

They do. Recently a pci_sync_page was added for some reason (because Linux doesn't have a real VM implementation I guess :-). We just needed to add the underlying function to support it. I think it popped up in one of the SCSI drivers.

Quoted 3 lines

If you're talking about non-PCI devices, use the pci DMA API but just pass NULL for the dev (we need to make sure that will work ok on the non-cache-coherent cpus).

Ummm....I don't normally do this because using these functions requires the CONFIG_PCI option, which isn't supported or has bad effects when requested on processors that don't have PCI. For non-PCI devices, I call the consistent_* functions, which other architectures support and do as well. So, we kind of have a common interface here for non-PCI devices as well.

I stole all of this cache coherency stuff from ARM and Sparc (or MIPS, I don't remember) when I did it the first time.

Thanks.

	-- Dan

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#9 lore raw
In reply to Paul Mackerras

Hi,

Quoted 6 lines · Paul Mackerras

If we have to have a consistent_sync_page, it should be purely a local function in our implementation of the official DMA mapping API - see Documentation/DMA-mapping.txt. Drivers should be using functions such as pci_alloc_consistent, pci_map_single, pci_dma_sync_single, pci_unmap_single, etc. The implementation of those routines should do the correct cache flushing - if it doesn't then we need to fix it.

This document only describes DMA _mappings_, it doesn't say anything about cache coherency.

Quoted 3 lines

If you're talking about non-PCI devices, use the pci DMA API but just pass NULL for the dev (we need to make sure that will work ok on the non-cache-coherent cpus).

That's the other problem, "non-PCI" sounds like ISA there, what about other buses?

bye, Roman

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#6 lore raw
In reply to Dan Malek

Hi,

Quoted 7 lines · Dan Malek

For non-PCI devices,
I call the consistent_* functions, which other architectures support and do as well. So, we kind of have a common interface here for non-PCI devices as well.

I stole all of this cache coherency stuff from ARM and Sparc (or MIPS, I don't remember) when I did it the first time.

The only other architecture which defines it is ARM?! The only API which exists at least as dummys everywhere is dma_cache_(inv|wback).

bye, Roman

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#10 lore raw
In reply to Roman Zippel
Quoted 2 lines · Roman Zippel

This document only describes DMA _mappings_, it doesn't say anything about cache coherency.

It talks about the conditions under which the cpu and the DMA device will see a consistent view of memory. The section on the streaming DMA mappings could be more explicit, but the idea is that between the pci_map_{single,sg} and the corresponding pci_unmap_*, the device owns the region and the cpu shouldn't touch it without first calling pci_dma_sync_*. (Hmmm, there is possibly a hole here - the API should maybe require that you call a sync function both before and after touching the memory.) At other times the cpu owns the region and should see a consistent view of that memory. That implies some cache flushing in pci_map_* and/or pci_unmap_* on non-cache-coherent systems.

Quoted 2 lines

That's the other problem, "non-PCI" sounds like ISA there, what about other buses?

That is implementation-dependent. My view is that we should make this API work for every kind of bus we have, if possible.

Note also that in 2.5 the struct pci_dev is going to transmogrify into a "struct device" that will be used for I/O devices on any kind of bus.

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#7 lore raw
In reply to Dan Malek
Quoted 2 lines · Dan Malek

That's exactly what it is. We are not creating a new API, and it shouldn't have been described as such.

OK... Armin's message wasn't exactly clear... :)

Quoted 1 line

They do. Recently a pci_sync_page was added for some reason (because

To go with pci_map_page and pci_unmap_page. These give you the ability to do DMA to highmem pages without using bounce buffers.

Quoted 10 lines
Quoted 3 lines

If you're talking about non-PCI devices, use the pci DMA API but just pass NULL for the dev (we need to make sure that will work ok on the non-cache-coherent cpus).

Ummm....I don't normally do this because using these functions requires the CONFIG_PCI option, which isn't supported or has bad effects when requested on processors that don't have PCI. For non-PCI devices, I call the consistent_* functions, which other architectures support and do as well. So, we kind of have a common interface here for non-PCI devices as well.

My preference would be to make the one set of functions work everywhere. OTOH these non-PCI devices you're talking about are presumably very specific to PPC embedded chips so maybe it doesn't matter so much.

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#8 lore raw
Quoted 2 lines · Paul Mackerras

To go with pci_map_page and pci_unmap_page. These give you the ability to do DMA to highmem pages without using bounce buffers.

IIRC, there was a virt_to_bus in the code at one time, making it unusable for highmem pages, which is why I couldn't understand the appearance of the function.

Quoted 2 lines

My preference would be to make the one set of functions work everywhere.

Mine, too. I just wish this was a common concept across a variety of our I/O interfaces.

Quoted 3 lines

.... OTOH these non-PCI devices you're talking about are presumably very specific to PPC embedded chips so maybe it doesn't matter so much.

That is always a challenge with the embedded parts. It is often very convenient to make the (promper, IMHO) assumption this software isn't going to be used outside of these parts.

Thanks.

	-- Dan

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/