IA64 Kernel Question

[I'm cc'ing the reply to linux-ia64 in the hope that it goes through,
as I think this might be of interest to others.]

Rob,

I haven't tried running your code, but from looking at it, it appears that it fails to establish coherency with the i-cache. With gcc, you can use a routine along the lines of:

static void
flush_cache (void *addr, unsigned long len)
{
void *end = (char *) addr + len;

while (addr < end)

    {
      asm volatile ("fc %0" :: "r"(addr));
      addr = (char *) addr + 32;
    }

asm volatile (";;sync.i;;srlz.i;;");
}

For example, a call of the form:

	flush_cache(pBuffer1, 0x1000);

should do.

The reason this is needed is that on ia64, CPU local stores are not coherent with respect to i-cache fetches (everything else *is* cache-coherent).

The memory allocated by malloc() does indeed have execute permission turned on. Linux does this for historical reasons. One performance
caveat: when executing malloc()'d memory, you'll get one additional page fault for each page that is actually executed so it is advantageous to use as few pages as possible for dynamicaly generated code.

If your code is multi-threaded, there are additional consideration to ensure all CPUs see the right version of the code at the right time. See the IA-64 architecture manual for details.

Hope this helps,

	--david

Rob> David,
Rob> I am sorry for sending this directly to you, but I am unable to send Rob> email to the IA64 kernel list for some reason. I thought you may know Rob> the answer off hand, or could forward it to the list for me.

Rob> I have noticed a problem when trying to execute code in a user mode app Rob> from an allocated buffer. The code below does a malloc to get a buffer, Rob> and then copies code from another function to the buffer. Being careful Rob> to treat function pointers properly as structures, I believe that the Rob> buffer function is called properly.
Rob> Unfortunately it seg faults upon execution, although it does at least Rob> display the correct fault address. Is using the same GP value from the Rob> other function the correct thing to do in a case like this? Is the Rob> memory region user mode malloc uses being set to allow execution? Rob> Perhaps there is something else that needs to be done in my code to Rob> allow this to work. I would appreciate any insights anyone might have.

Rob> Rob

Rob> #include <stdio.h>
Rob> #include <stdlib.h>
Rob> #include <string.h>
Rob> #include <malloc.h>

  Rob> typedef struct _fp
  Rob> {
  Rob> long addr;
  Rob> long gp;

Rob> } IA64_FUNCTION;

  Rob> void TestApp(void)
  Rob> {
   
  Rob> __asm__ __volatile__ ("nop.i 0");
  Rob> __asm__ __volatile__ ("nop.i 0");
  Rob> __asm__ __volatile__ ("nop.i 0");
  Rob> __asm__ __volatile__ ("nop.i 0");

Rob> return;
Rob> }

  Rob> int main(int argc, char *argv[])
  Rob> {
  Rob> void  
  Rob> (*pSubroutine)(void);   
  Rob> unsigned char
  Rob> *pBuffer1;
  Rob> long
  Rob> alignment;

Rob> IA64_FUNCTION *fp;
Rob> IA64_FUNCTION newfp;

  Rob> printf("Test ***\n");
   
  Rob> // Allocate and align buffer on 16 byte boundary
  Rob> pBuffer1 = (unsigned char *)malloc(0x1000);
  Rob> alignment = ((unsigned long)pBuffer1 % 16);
  Rob> pBuffer1 = pBuffer1 + 16 - alignment;
   
  Rob> fp = (IA64_FUNCTION *)TestApp;
  Rob> printf("pSub Addr = 0x%lX GP = 0x%lX\n", fp->addr, fp->gp);
   
  Rob> memcpy(pBuffer1, (unsigned char *)fp->addr, 256);
   
  Rob> newfp.gp = fp->gp;
  Rob> newfp.addr = (long)pBuffer1;
  Rob> printf("pSub Addr = 0x%lX GP = 0x%lX\n", newfp.addr, newfp.gp);
  Rob> pSubroutine = (void (*)(void))&newfp;

  Rob> (*pSubroutine)();

  Rob> return(0);
  Rob> }

To followup on this issue: Robert tried the flush_memory() and the program still didn't work.

David> The memory allocated by malloc() does indeed have execute David> permission turned on. Linux does this for historical David> reasons.

This paragraph is correct in that it's describing the (traditionally) intended behavior of Linux. However, it turns out that there is a platform-independent bug in the kernel which has the effect that the first few bytes returned by brk() may not be executable. The reason this happens is that the ELF executable maps the data (and bss) section with READ+WRITE rights only and the kernel doesn't turn on EXECUTE right for the last such page, which usually contains the initial break value.

So, the upshot of this is that for the time being, on ia64 you'll have to do mprotect() to be certain that the memory returned by brk() is executable.

I'm not sure what the long-term solution will be. My inclination is to break with tradition and make anonymous memory not exectuable by default, but it's not clear whether this will be acceptable for all platforms (x86 won't be affected because, there, the READ right implies the EXECUTE right). I'll follow up with a separate mail to the LKML get a discussion going (I'll cc this list).

	--david