cache coherence problem
Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <signal.h>
#include <sched.h>
int main( int argc, char** argv ) {
long* loc;
int (*init)(void);
long returninstr0 = 0x38600009; // li r3,4
long returninstr1 = 0x4e800020; // blr
int returned;
int fd;
char zero;
char tempfile[20] = { "test-XXXXXX" };
printf( "start of test\n" );
fd = mkstemp( tempfile );
printf( "memfile name test-XXXXXX, fd %d\n", fd );
if (fchmod(fd, 0777) < 0){
perror("fchmod");
exit(1);
}
if( lseek( fd, 32768*1024, SEEK_SET ) != 32768*1024 ){ //32M
perror("lseek");
exit(1);
}
zero = 0;
if( write(fd, &zero, 1 ) != 1){
perror("write");
exit(1);
}
if( fcntl(fd, F_SETFD, FD_CLOEXEC ) != 0)
perror("Setting FD_CLOEXEC failed");
loc = mmap( (void *) 0x62800000, 12288,
PROT_READ | PROT_WRITE | PROT_EXEC,
MAP_SHARED | MAP_FIXED,
fd, 0x3a0000);
printf( "memory allocated at addr %x\n", loc );
printf( "data %x\n", loc[0] );
// random behaviour, sometimes works, sometimes illegal instruction
//loc = (long*)0x62800000; // set the address
loc = (long*)0x62802000; // set the address
// always illegal instruction
//loc = (long*)0x62801b44;
//loc = (long*)0x62801000;
loc[0] = returninstr0;
loc[1] = returninstr1;
printf( "new code: addr %x, %x\n", loc, loc[0] );
init = (int (*)(void))loc;
returned = init();
printf( "function returned %d\n", returned );}
Quoted 7 lines · Juergen Kienhoefer
Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
What processor is this on?
-Matt
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 8 lines · Juergen Kienhoefer
Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
I don't think this is the kernel's responsibility. I think this is just life with PPCs. If you dump random instructions into random memory, you have to make sure everything is flushed and the icache is invalidated before executing the code. You have to do the same thing when you do self-modifying code.
Why would you want to do this, anyway?
--
----------------------------------------------
Michael Zucca - [email protected]
----------------------------------------------
"I'm too old to use Emacs." -- Rod MacDonald
----------------------------------------------
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 6 lines · Michael R. Zucca
have to make sure everything is flushed and the icache is invalidated before executing the code. You have to do the same thing when you do self-modifying code.
Why would you want to do this, anyway?
I don't know what he's doing, but sometimes one has static if tests inside of tight inner loops, but you don't know (at compile time) what the compare op will be until runtime. One way to solve this is build an optimized loop for each possible compare op.
This optimization strategy fails if you have more than a handful of if-tests, cause then you have a combinatorial explosion (i.e. dozens of nearly identical .o files eating up megabytes with nearly identical
code). So one way around this to have a few basic loops, and patch
them up at run time.
--linas
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 9 lines · Juergen Kienhoefer
Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen
It's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages, but it will certainly not enforce flush of your writes to memory.
You need to first flush the data cache to memory using dcbf or dcbst, then sync for this to complete, then invalidate the instruction cache, sync and isync.
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci
sync
isync
for every 32 bytes of the memory block I put code in. Right?
Cheers
Juergen
Quoted 19 lines · Benjamin Herrenschmidt
Quoted 11 lines · Juergen Kienhoefer
Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
JuergenIt's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages, but it will certainly not enforce flush of your writes to memory.
Quoted 8 lines
You need to first flush the data cache to memory using dcbf or dcbst, then sync for this to complete, then invalidate the instruction cache, sync and isync.
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 6 lines · Juergen Kienhoefer
Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci
^^^^^^^
No: icbi
Quoted 8 lines
sync
isync
for every 32 bytes of the memory block I put code in. Right?
Cheers
Juergen
Quoted 31 lines
Quoted 27 lines · Benjamin Herrenschmidt
Quoted 11 lines · Juergen Kienhoefer
Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
JuergenIt's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages, but it will certainly not enforce flush of your writes to memory.
You need to first flush the data cache to memory using dcbf or dcbst, then sync for this to complete, then invalidate the instruction cache, sync and isync.
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 13 lines · Benjamin Herrenschmidt
Quoted 6 lines · Juergen Kienhoefer
Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccci^^^^^^^
No: icbiQuoted 3 lines
sync
isync
for every 32 bytes of the memory block I put code in.
Actually, to be precise, you need:
1) A loop of dcbst's over every cache line crossed by your code
2) one sync
3) A loop of icbi's over every cache line crossed by your code
4) one sync, one isync
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Re: cache coherence problem: FIXED
Ben, et al.
Thanks, the test program is fixed and working.
I appreciate your help very much!!!
See attachment
Juergen
Quoted 25 lines · Benjamin Herrenschmidt
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 8 lines · Juergen Kienhoefer
Guys,
Thank you very much for the ideas.
Basically, what I need to do is:
dcbst
sync
iccciQuoted 10 lines
^^^^^^^
No: icbiQuoted 5 lines
sync
isync
for every 32 bytes of the memory block I put code in.
Quoted 11 lines
Actually, to be precise, you need:
1) A loop of dcbst's over every cache line crossed by your code
2) one sync
3) A loop of icbi's over every cache line crossed by your code
4) one sync, one isyncBen.
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <sys/ptrace.h>
#include <signal.h>
#include <sched.h>
static inline void clear_icache(void *p)
{
asm volatile ("dcbst 0,%0;
sync;
icbi 0,%0;
sync;
isync" : : "r" (p));
}int main( int argc, char** argv ) {
long* loc;
int (*init)(void);
long returninstr0 = 0x38600009; // li r3,4
long returninstr1 = 0x4e800020; // blr
int returned;
int fd;
char zero;
char tempfile[20] = { "test-XXXXXX" };
printf( "start of test\n" );
fd = mkstemp( tempfile );
printf( "memfile name test-XXXXXX, fd %d\n", fd );
if (fchmod(fd, 0777) < 0){
perror("fchmod");
exit(1);
}
if( lseek( fd, 32768*1024, SEEK_SET ) != 32768*1024 ){ //32M
perror("lseek");
exit(1);
}
zero = 0;
if( write(fd, &zero, 1 ) != 1){
perror("write");
exit(1);
}
if( fcntl(fd, F_SETFD, FD_CLOEXEC ) != 0)
perror("Setting FD_CLOEXEC failed");
loc = mmap( (void *) 0x62800000, 12288,
PROT_READ | PROT_WRITE | PROT_EXEC,
MAP_SHARED | MAP_FIXED,
fd, 0x3a0000);
printf( "memory allocated at addr %x\n", loc );
printf( "data %x\n", loc[0] );
// random behaviour, sometimes works, sometimes illegal instruction
//loc = (long*)0x62800000; // set the address
//loc = (long*)0x62802000; // set the address
// always illegal instruction
loc = (long*)0x62801b44;
//loc = (long*)0x62801000;
loc[0] = returninstr0;
loc[1] = returninstr1;
printf( "new code: addr %x, %x\n", loc, loc[0] );
clear_icache( loc );
init = (int (*)(void))loc;
returned = init();
printf( "function returned %d\n", returned );}
Quoted 2 lines · In message <[email protected]> you wrote:
Basically, what I need to do is:
...
Quoted 1 line
for every 32 bytes of the memory block I put code in.
...
s/32/CACHE_LINE_SIZE/g
Best regards,
Wolfgang Denk
--
Software Engineering: Embedded and Realtime Systems, Embedded Linux
Phone: (+49)-8142-4596-87 Fax: (+49)-8142-4596-88 Email: [email protected] Windows95 = graphical user interface for a single-threaded interrupt handler (DOS)
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 9 lines · Juergen Kienhoefer
Guys,
Look at the folloging test program. It mmaps memory, puts some instructions in it and executes it.
Sometimes it works, sometimes it crashes with illegal instruction. This smells like cache problems.
Should the kernel clean the instruction cache for these addresses in mmap system call?
Thanks for any thoughts!
Juergen
Quoted 2 lines
It's your duty to ensure cache coherency. Actually, the kernel will eventually clean the icache for newly mapped in blank pages,
If you mean zeroed pages for blanked, I believe that it is wrong. The reason being that 0 is an invalid instruction so that the code would trap in any case.
Maybe I'm wrong, but I seem to remember this as an optimization that Paulus implemented some time ago.
Gabriel
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
Quoted 9 lines
newly mapped in blank pages,
If you mean zeroed pages for blanked, I believe that it is wrong. The reason being that 0 is an invalid instruction so that the code would trap in any case.
Maybe I'm wrong, but I seem to remember this as an optimization that Paulus implemented some time ago.
And that we had to undo because glibc relied on it, not invalidating cache lines in some conditions assuming newly mapped zeroed pages are delivered icache-clean by the kernel.
One of the arguments of the glibc folks for not fixing that was that it would be a security hole to let stale icache content leak, so the kernel has to invalidate them anyway.
Ben.
** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/