Question about cache flushing and fork

Hi,

Can someone please explain why it is necessary to flush the cache during fork()? (i.e. call to flush_cache_mm() in dup_mmap)

It seems that after fork, the parent and child have access to the same vm, so it should be sufficient to flush the tlb, and create two pte's for the processes. I can see that during COW processing there can be kernel/user cache aliasing issues on virtually indexed caches, but that seems to be taken care of by copy_cow_page().

I've read through cachetlb.txt, but it just says:

This interface is used to handle whole address space page table operations such as what happens during fork, exit, and exec.

I can see why this is needed for exit(), but why fork()? and i don't see this used for exec() ?

Also is there an updated version of the "Linux Cache Flush Architecture" document? (http://en.tldp.org/LDP/khg/HyperNews/get/memory/flush.html)
This is a very nicely written doc, but it seems a bit out of date for 2.6 (e.g. flush_page_to_ram is gone)

thanks
randolph

Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/

Writable pages that will be shared between the child and parent are marked read-only and COW, some cpu caches store protection information in the cache lines in order to avoid TLB lookups etc. so the caches must be flushed since the page protection information is changing.

Writable pages that will be shared between the child and parent are marked read-only and COW, some cpu caches store protection information in the cache lines in order to avoid TLB lookups etc. so the caches must be flushed since the page protection information is changing.

On PARISC, the cache line contains the following elements:
1) data (obviously)
2) physical page
3) dirty/clean/public/private/etc state

A cache access hits or misses depending on whether or not the physical page from the TLB matches the physical page stored in the cache line.

If flushing is required during fork on PARISC, then there are cache consistency issues elsewhere, something is horribly broken in the design (and it should be falling all over anyway).

lamont

On PARISC, the cache line contains the following elements:
1) data (obviously)
2) physical page
3) dirty/clean/public/private/etc state

A cache access hits or misses depending on whether or not the physical page from the TLB matches the physical page stored in the cache line.

If flushing is required during fork on PARISC, then there are cache consistency issues elsewhere, something is horribly broken in the design (and it should be falling all over anyway).

lamont

I don't know if it is strictly necessary - we might well be able to just do the right thing in the page fault COW handler. But doing the cache flush at fork time just means that we should never have a page that is marked read-only but that may have dirty data in the virtual caches. That could easily get confusing. In fact, I wouldn't be totally surprised if some architecture refused to do write-backs through a read-only mapping.

But quite frankly, I suspect that since only a few CPU's have virtually indexed caches, the cache-flush code hasn't gotten that much testing, and there may be somewhat of an overkill approach there. As you found out, it's not _that_ well documented.

Linus

I don't know if it is strictly necessary - we might well be able to just do the right thing in the page fault COW handler. But doing the cache flush at fork time just means that we should never have a page that is marked read-only but that may have dirty data in the virtual caches. That could easily get confusing. In fact, I wouldn't be totally surprised if some architecture refused to do write-backs through a read-only mapping.

But quite frankly, I suspect that since only a few CPU's have virtually indexed caches, the cache-flush code hasn't gotten that much testing, and there may be somewhat of an overkill approach there. As you found out, it's not _that_ well documented.

Linus

I have a different cache question. GCC uses a stack trampoline to call nested functions. This trampoline is on the stack and has to be flushed before being used. A few weeks ago a trampoline testcase was added. This testcase failed under hppa64-hpux. It turned out that the code assumed that the PA 2.0 cache line was 32 bytes (it was apparently derived from the original PA 1.1 code).

According to my understanding, all PA 2.0 machines use a 64 byte cache line. On the otherhand, the PA 2.0 documentation states that a cache line on a PA 2.0 machine can be 16, 32 or 64 bytes if I remember correctly.

After I changed the cache line to 64 bytes, the GCC testcase passed under hpux 11.11 on the A500 that I have. However, I noticed recently that it still fails on a C200 running 11.00. It dies with an illegal insn (it's a mfia) about the 5th time through one of the trampolines that the code creates. The mfia insn is the entry to the trampoline and it's 16 bytes from the start of the trampoline data. If I copy the C200 binary to the A500, it executes without error. So, there appears to be a subtle hardware or OS dependence involved.

I believe that code follows the stated rules for self-modifying code. The code looks fine when examined with gdb.

Anybody have have any ideas what might be going on?

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Correct. I believe that PDC_CACHE returns the actual size (among other things.)

If the C200 is genuinely a PA2.0 machine, then that's, umm, interesting, since the arch document doesn't say that it's hversion dependent... The instruction is undefined(*) on PA1.1 - you must bl .+8,<register>, which will get you the address of .+8 (regardless of where you branch.)

lamont

(*) undefined is defined as resulting in any CPU state, with the sole requirement that there is some defined set of instructions, operating at the same privilege level that results in the same CPU state.

Is it possible to do PDC calls from user space under hpux to determine cache parameters?

I revised GCC's trampoline code to assume a line length of 32 instead of 64 on hppa64. It can now flush arbitrarily long regions of memory. The test now passes. This seems to imply that the cache length of a C200 under hpux11.00 is 32 bytes.

Thanks,
Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Re:

Not that I know of. I have a driver that lets me do it. PDC calls generally run with virtual address translation turned off.

Wait ... I forgot ... yes, sort of. /dev/diag/diag2 allows you to do (some) PDC calls. Unfortunately, it's not documented :(

Stan Sieler                                           [email protected]
www.allegro.com/sieler/wanted/index.html                  www.sieler.com

The architecture requires that pdc calls be done at ring 0, with PSW.D=0, so no. Having said that, many of the interesting return values are available from the kernel, either through sysconf or (sigh) /dev/kmem.

Cache line size is fixed on every CPU that I've seen, so OS shouldn't matter... One can assume a 16-byte cache line and it will work everywhere, of course, although at 1/2 or 1/4 speed in many cases. AFAIK, all PA1.1 machines use at least a 32 byte cache-line.

lamont

Hi,

HP-UX's /dev/diag/diag2 definitely seems to return cache information.

However, it's going to be up to HP to reveal the calling sequence and semantics / cautions.

Stan Sieler                                           [email protected]
www.allegro.com/sieler/wanted/index.html                  www.sieler.com

Could you elaborate on the context for this? It was my understanding that to flush an area for execution, you simply loop over the area striding by the cache line width, issuing fdc; fic. (at least, that's how we do it in the kernel).

If we think the cache line is *smaller* than it actually is, all this does is issue more flushes than necessary to the cache lines, I don't understand how it results in an observable failure.

James

In the particular case at hand, I noticed that calls to nested functions were broken on hppa64-hpux11.11. Nested calls in GCC are done with a stack trampoline. The 64-bit trampoline was in fact 72 bytes long. GCC was using a line length of 32. Because the trampoline has 16 byte alignment, it was possible to hit an alignment situation were the stack code didn't get flushed.

I changed the length to 64 on hppa64 to fix this problem. I based this on the following code in include/parisc/cache.h:

/*
* PA 2.0 processors have 64-byte cachelines; PA 1.1 processors have
* 32-byte cachelines. The default configuration is not for SMP anyway,
* so if you're building for SMP, you should select the appropriate
* processor type. There is a potential livelock danger when running
* a machine with this value set too small, but it's more probable you'll
* just ruin performance.
*/
#ifdef CONFIG_PA20
#define L1_CACHE_BYTES 64
#else
#define L1_CACHE_BYTES 32
#endif

After I made the GCC change, I noticed the trampoline-1.c test failed on a C200 but not on the A500 that I did the original fix on. It appears the C200 has line length of 32 bytes. As the C200 contains a PA 2.0 processor, the above comment would appear incorrect. Probably, trying to run a kernel built with CONFIG_PA20 defined on a C200 would fail because the line length specified above is *larger* than the actual line length. On the otherhand, PDC_CACHE allows setting some cache parameters.

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

OK. Note also that the sp is supposed to always be 64-byte aligned...

I expect you are correct here - again, no hard knowledge of that processor on my part.

Assuming a 32-byte cache line on a 64-byte-cache-line machine should not cause any issues (other than performance issues, that is..)

lamont

That's not correct for 64 bits. See page 15 of the "64-Bit Runtime Architecture for PA-RISIC 2.0, Version 3.3". The alignment specified is 16 bytes. This is also the case for the HP compilers that I have. Thus, you can't assume that a stack frame starts at the beginning of a cacheline in the 64-bit runtime.

Regarding the frame marker, there is a discrepancy between the size of the frame marker mentioned in this document and that which HP compilers generate (16 versus 48 bytes).

Dave

J. David Anglin                                  [email protected]
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)