Reintroduce O_SYNC flag to make DRAM non-cached.

From: Nick Spence <[email protected]>

This patch re-introduces the O_SYNC flag to make DRAM non-cached, which is the default behavior for non PowerPC architectures in /drivers/char/mem.c

---

index 5ccb579..f994b0a 100644
@@ -101,7 +101,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
 	if (ppc_md.phys_mem_access_prot)
 		return ppc_md.phys_mem_access_prot(file, pfn, size, vma_prot);
 
-	if (!page_is_ram(pfn))
+	if ((file->f_flags & O_SYNC) || (!page_is_ram(pfn)))
 		vma_prot = __pgprot(pgprot_val(vma_prot)
 				    | _PAGE_GUARDED | _PAGE_NO_CACHE);
 	return vma_prot;

1.5.5

This says nothing about why you want to do this---what bad things arise from the present code, or what good things would happen if the change was made. It needs to.

The change isn't acceptable as it is, because IBM machines with a hypervisor don't allow system memory to be mapped uncacheable. It would at least have to have some platform-specific check in there.

On other powerpc platforms, this would introduce possible cache paradoxes, where a page is mapped cacheable at one address (via the linear mapping) and uncacheable at another address (via /dev/mem). If you really really want to do this, you'll need to add code to bust up the large pages (or TLB entries) used for the linear mapping so that the corresponding pages of the linear mapping can be unmapped, or mapped cacheable. Plus you'll need to add the appropriate cache flushing code to make sure there aren't any cache lines for the page in cache. Further, you'll need to arrange somehow to make sure that the kernel stack has a resident TLB entry during the unrecoverable parts of the exception entry and exit paths.

Paul.

We found the problem when porting code from Linux 2.4 to 2.6, where a user space application maps a 1 MByte region of DRAM with the O_SYNC=20 flag to communicate with an internal core that shares access to the DRAM but does not have any cache snooping logic.

In 2.4 the mem driver honors the O_SYNC flag and makes the requested memory
memory non-cached so that writes from user space are immediately available
to the second core.

In 2.6 the powerpc mem driver no longer honors the O_SYNC flag so the data
is only written out to DRAM (and becoming visible to the second core) as the
cache reloads with other data.

This is getting to be a much more elaborate project.

In our case the memory area is reserved so that the main processor will not
touch it except when mapped by the mem driver.

Alternative solutions are to modify the user space code to flush out the caches by reading a large area, or to implement a board specific=20 ppc_md.phys_mem_access_prot() function to do this.

Either of these will meet my needs, but this will leave the O_SYNC flag ignored on the PowerPC architecture in 2.6, but honored by Linux 2.4 and other architectures. Probably not a problem for most people.

Thanks,
Nick

Then you should completely carve it out of the LMB's which will ensure it's not seen as RAM by /dev/mem and not mapped by the linear mapping (well, the later depends ... if it's carved out of the top of RAM it should work fine, if it's a hole, I'm not sure we handle holes in the linear mapping on 32 bits).

Unfortunately the memory window is 4 MBytes into the DDR and not at the end. It would be hard to put it at the end because the contents are location dependent and we can change the memory size.

The region is reserved with an lmb_reserve().

/* Set the location and size of the shared DDR memory region */ void* shared_mem_base = (void *)0x400000;
long shared_mem_size = 0x100000;

....
lmb_reserve((unsigned long)shared_mem_base, shared_mem_size);

The page protection seemed to be allocated on a per pte basis, where each PTE is a small fixed size. I will need to check the TLB setup further.

Nick

The problem is that it will then be part of the linear mapping, which means you'll end up with a double cacheable & non-cacheable mapping for that memory, this is not nice ... it might work as long as we stick to having G bit set for the whole linear mapping but it's going to come back and bite.

I still think you should look closely whether it could be carved out of the end of memory... even if that involves changing application code.

Now, if you aren't afraid of cache paradox caused by the linear mapping, then carve it out of the LMB wherever it is rather than lmb_reserve() it. That would probably work.

Ben.

If you only need to write from the core running Linux, and never read the memory back, and the secondary core never writes the memory, your userland program can simply do dcbst on it after it wrote it.

Otherwise, if your CPU has BAT registers, you could use those to force that memory region as non-cacheable (be careful, BATs are not allowed to overlap each other).

If these things won't work for you, you need to do some simple device driver, and do non-coherent DMA from it (you can view the secondary core as an I/O device like any other).

Segher