Skip to contentSkip to conversation

cache coherence problem

DISCUSSION Juergen Kienhoefer 7 participants 12 messages Nov 18, 2003 – Nov 19, 2003

lore
Share & tools
lore
#0 lore raw

Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen

#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <signal.h>
#include <sched.h>

int main( int argc, char** argv ) {
    long* loc;
    int (*init)(void);
    long returninstr0 = 0x38600009; // li r3,4
    long returninstr1 = 0x4e800020; // blr
    int returned;
    int fd;
    char zero;
    char tempfile[20] = { "test-XXXXXX" };

    printf( "start of test\n" );


    fd = mkstemp( tempfile );
    printf( "memfile name test-XXXXXX, fd %d\n", fd );
    if (fchmod(fd, 0777) < 0){
	perror("fchmod");
	exit(1);
    }
    if( lseek( fd, 32768*1024, SEEK_SET ) != 32768*1024 ){ //32M
	perror("lseek");
	exit(1);
    }
    zero = 0;
    if( write(fd, &zero, 1 ) != 1){
	perror("write");
	exit(1);
    }
    if( fcntl(fd, F_SETFD, FD_CLOEXEC ) != 0)
	perror("Setting FD_CLOEXEC failed");


    loc = mmap( (void *) 0x62800000, 12288,
	        PROT_READ | PROT_WRITE | PROT_EXEC,
	        MAP_SHARED | MAP_FIXED,
	        fd, 0x3a0000);

    printf( "memory allocated at addr %x\n", loc );
    printf( "data %x\n", loc[0] );

    // random behaviour, sometimes works, sometimes illegal instruction
    //loc = (long*)0x62800000; // set the address
    loc = (long*)0x62802000; // set the address
    // always illegal instruction
    //loc = (long*)0x62801b44;
    //loc = (long*)0x62801000;
    loc[0] = returninstr0;
    loc[1] = returninstr1;

    printf( "new code: addr %x, %x\n", loc, loc[0] );

    init = (int (*)(void))loc;
    returned = init();

    printf( "function returned %d\n", returned );

}

#1 lore raw
Quoted 7 lines · Juergen Kienhoefer

Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?

What processor is this on?

-Matt

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#2 lore raw
In reply to Juergen Kienhoefer
Quoted 8 lines · Juergen Kienhoefer

Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!

I don't think this is the kernel's responsibility. I think this is just life with PPCs. If you dump random instructions into random memory, you have to make sure everything is flushed and the icache is invalidated before executing the code. You have to do the same thing when you do self-modifying code.

Why would you want to do this, anyway?

--
----------------------------------------------
Michael Zucca - [email protected]
----------------------------------------------
"I'm too old to use Emacs." -- Rod MacDonald
----------------------------------------------

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#4 lore raw
In reply to Juergen Kienhoefer
Quoted 9 lines · Juergen Kienhoefer

Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen

It's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages, but it will certainly not enforce flush of your writes to memory.

You need to first flush the data cache to memory using dcbf or dcbst, then sync for this to complete, then invalidate the instruction cache, sync and isync.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#5 lore raw

Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci
sync
isync
for every 32 bytes of the memory block I put code in. Right?
Cheers
Juergen

Quoted 19 lines · Benjamin Herrenschmidt
Quoted 11 lines · Juergen Kienhoefer

Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen

It's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages, but it will certainly not enforce flush of your writes to memory.

Quoted 8 lines

You need to first flush the data cache to memory using dcbf or dcbst, then sync for this to complete, then invalidate the instruction cache, sync and isync.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#6 lore raw
Quoted 6 lines · Juergen Kienhoefer

Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci

^^^^^^^
No: icbi

Quoted 8 lines

sync
isync
for every 32 bytes of the memory block I put code in. Right?
Cheers
Juergen

Quoted 31 lines
Quoted 27 lines · Benjamin Herrenschmidt
Quoted 11 lines · Juergen Kienhoefer

Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen

It's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages, but it will certainly not enforce flush of your writes to memory.

You need to first flush the data cache to memory using dcbf or dcbst, then sync for this to complete, then invalidate the instruction cache, sync and isync.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#7 lore raw
Quoted 13 lines · Benjamin Herrenschmidt
Quoted 6 lines · Juergen Kienhoefer

Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci

^^^^^^^
No: icbi

Quoted 3 lines

sync
isync
for every 32 bytes of the memory block I put code in.

Actually, to be precise, you need:

1) A loop of dcbst's over every cache line crossed by your code
2) one sync
3) A loop of icbi's over every cache line crossed by your code
4) one sync, one isync

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#9 lore raw
In reply to Juergen Kienhoefer
Quoted 2 lines · In message <[email protected]> you wrote:

Basically, what I need to do is:

...

Quoted 1 line

for every 32 bytes of the memory block I put code in.

...

s/32/CACHE_LINE_SIZE/g

Best regards,

Wolfgang Denk

--
Software Engineering: Embedded and Realtime Systems, Embedded Linux
Phone: (+49)-8142-4596-87 Fax: (+49)-8142-4596-88 Email: [email protected] Windows95 = graphical user interface for a single-threaded interrupt handler (DOS)

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#10 lore raw
In reply to Benjamin Herrenschmidt
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 9 lines · Juergen Kienhoefer

Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen

Quoted 2 lines

It's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages,

If you mean zeroed pages for blanked, I believe that it is wrong. The reason being that 0 is an invalid instruction so that the code would trap in any case.

Maybe I'm wrong, but I seem to remember this as an optimization that Paulus implemented some time ago.

Gabriel

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#8 lore raw

Re: cache coherence problem: FIXED

In reply to Benjamin Herrenschmidt

Ben, et al.
Thanks, the test program is fixed and working.
I appreciate your help very much!!!
See attachment
Juergen

Quoted 25 lines · Benjamin Herrenschmidt
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 8 lines · Juergen Kienhoefer

Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci

Quoted 10 lines

^^^^^^^
No: icbi

Quoted 5 lines

sync
isync
for every 32 bytes of the memory block I put code in.

Quoted 11 lines

Actually, to be precise, you need:

1) A loop of dcbst's over every cache line crossed by your code
2) one sync
3) A loop of icbi's over every cache line crossed by your code
4) one sync, one isync

Ben.

#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <signal.h>
#include <sched.h>

static inline void clear_icache(void *p)

{
    asm volatile ("dcbst 0,%0;
                   sync;
                   icbi 0,%0;
                   sync;
                   isync" : : "r" (p));
}
int main( int argc, char** argv ) {
    long* loc;
    int (*init)(void);
    long returninstr0 = 0x38600009; // li r3,4
    long returninstr1 = 0x4e800020; // blr
    int returned;
    int fd;
    char zero;
    char tempfile[20] = { "test-XXXXXX" };

    printf( "start of test\n" );


    fd = mkstemp( tempfile );
    printf( "memfile name test-XXXXXX, fd %d\n", fd );
    if (fchmod(fd, 0777) < 0){
	perror("fchmod");
	exit(1);
    }
    if( lseek( fd, 32768*1024, SEEK_SET ) != 32768*1024 ){ //32M
	perror("lseek");
	exit(1);
    }
    zero = 0;
    if( write(fd, &zero, 1 ) != 1){
	perror("write");
	exit(1);
    }
    if( fcntl(fd, F_SETFD, FD_CLOEXEC ) != 0)
	perror("Setting FD_CLOEXEC failed");


    loc = mmap( (void *) 0x62800000, 12288,
	        PROT_READ | PROT_WRITE | PROT_EXEC,
	        MAP_SHARED | MAP_FIXED,
	        fd, 0x3a0000);

    printf( "memory allocated at addr %x\n", loc );
    printf( "data %x\n", loc[0] );

    // random behaviour, sometimes works, sometimes illegal instruction
    //loc = (long*)0x62800000; // set the address
    //loc = (long*)0x62802000; // set the address
    // always illegal instruction
    loc = (long*)0x62801b44;
    //loc = (long*)0x62801000;
    loc[0] = returninstr0;
    loc[1] = returninstr1;

    printf( "new code: addr %x, %x\n", loc, loc[0] );

    clear_icache( loc );

    init = (int (*)(void))loc;
    returned = init();

    printf( "function returned %d\n", returned );

}

#11 lore raw
In reply to Gabriel Paubert
Quoted 9 lines

newly mapped in blank pages,

If you mean zeroed pages for blanked, I believe that it is wrong. The reason being that 0 is an invalid instruction so that the code would trap in any case.

Maybe I'm wrong, but I seem to remember this as an optimization that Paulus implemented some time ago.

And that we had to undo because glibc relied on it, not invalidating cache lines in some conditions assuming newly mapped zeroed pages are delivered icache-clean by the kernel.

One of the arguments of the glibc folks for not fixing that was that it would be a security hole to let stale icache content leak, so the kernel has to invalidate them anyway.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#3 lore raw
In reply to Michael R. Zucca
Quoted 6 lines · Michael R. Zucca

have to make sure everything is flushed and the icache is invalidated before executing the code. You have to do the same thing when you do self-modifying code.

Why would you want to do this, anyway?

I don't know what he's doing, but sometimes one has static if tests inside of tight inner loops, but you don't know (at compile time) what the compare op will be until runtime. One way to solve this is build an optimized loop for each possible compare op.

This optimization strategy fails if you have more than a handful of if-tests, cause then you have a combinatorial explosion (i.e. dozens of nearly identical .o files eating up megabytes with nearly identical
code). So one way around this to have a few basic loops, and patch
them up at run time.

--linas

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/