shared memory problem on ARM v5TE using threads

Hello,

I have the following shared mem problem on a ARM v5TE processor using Linux version 2.6.28,

-bash-3.2# cat /proc/cpuinfo
Processor : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS : 799.53
Features : swp half thumb fastmult edsp
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant : 0x2
CPU part : 0x131
CPU revision : 1

The testscript [1] starts 2 processes. One write process with one thread, which writes in a shared memory.

The second process starts 4 threads, which all read from this shared memory. This don;t work on this processor [4]. The same demoprogramm works fine on ppc, i386 or on a ARMv6 based board [3][6] ...

If I start 4 read processes, which themselves starts only one readthread [5], the demoprogramm works fine!

Also, if I start one read process, which only attaches the shared memory once with shmat(), and then starts 4 read threads, and all this 4 read threads using the same shared memory addr, returned from shmat(), this works as expected.

Any ideas, hints ... ?

TIA

bye,
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany


[1] Testscript

#!/bin/sh
echo "Compile"
gcc -o shmtest2 shmtest2.c -lpthread

echo "Run shmtest2"
./shmtest2 write 1 &
./shmtest2 read 4 &


[2] demoprogramm, shmtest2.c

#include <pthread.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

extern void      exit();

struct Entry
{
 char          ident_name[1000];
 unsigned int  tipc_nr;
 unsigned int  pid;
 unsigned int  in_msg;
 unsigned int  out_msg;
 unsigned int  rxQueueLength;
};

void* attachSharedMem(int shmid)
{
  void* addr = shmat(shmid, NULL, 0);
  if ((addr != 0) && (0xffffffff != (unsigned int)addr))
  {
    printf("attach shared mem:%x\n",addr);
  }
  else
  {
    printf("shmat failed");
    addr = 0;
  }
  return addr;
}

int createSharedMem()
{
  key_t   key     = 1000;          /* key to be passed to shmget() */
  int     shmflg;                  /* shmflg to be passed to shmget() */
  int     shmid;                   /* return value from shmget() */
  int     size;                    /* size to be passed to shmget() */

  size   = 60000;
  shmflg = IPC_CREAT | 0666;
  if ((shmid = shmget (key, size, shmflg)) == -1)
  {
    printf("shmget failed");
    shmid = 0;
  }

  printf("Shared memory Id:%d\n",shmid);

  return shmid;
}


void* setupSharedMem()
{
  int shmid = createSharedMem();
  void* addrShm = attachSharedMem(shmid);
  return addrShm;
}

void *readThread(void *t)
{
  struct Entry* entry   = 0;

  int shmid = (int)t;
  void* addrShm = attachSharedMem(shmid);

  if (addrShm != 0)
  {
    printf("Start Read Thread addr:%x\n",addrShm);
    entry = (struct Entry*)addrShm;
    entry->in_msg  = 0;
    entry->out_msg = 0;

    int i=0;
    while(i < 60)
    {
      entry->in_msg += 1000;
      sleep(1);
      printf("Read from entry in_msg=%d, out_msg=%d, addr=%x\n",entry->in_msg,entry->out_msg, addrShm);
      i++;
    }
  }

  pthread_exit(NULL);
}

void *writeThread(void *t)
{
  struct Entry* entry   = 0;
  unsigned int threadId = (unsigned int)t;
  void* addrShm         = setupSharedMem();

  if (addrShm != 0)
  {
    printf("Start Write Thread %d, addr:%x\n",threadId,addrShm);
    entry = (struct Entry*)addrShm;
    strcpy(entry->ident_name,"this is a test entry");
    entry->in_msg  = 0;
    entry->out_msg = 0;
    entry->rxQueueLength = 20000;
    entry->pid     = threadId;
    entry->tipc_nr = 1000;

    int i=0;
    while(i < 60)
    {
      entry->out_msg += 1000;
      sleep(1);
      i++;
    }
  }

  pthread_exit(NULL);
}

main(int argc, char* argv[])
{

    //check the arguments
    if (argc != 3)
    {
       printf("Arguments are [read|write] [number of threads]\n");
       exit(1);
    }

    unsigned int mode         = 0;
    unsigned int nbrOfThreads = 0;

    if (strcmp(argv[1],"write") == 0)
    {
      printf("Write to in_msg\n");
      mode = 1;
    }

    if (strcmp(argv[1],"read") == 0)
    {
      printf("Read from in_msg\n");
      mode = 2;
    }

    nbrOfThreads = atoi(argv[2]);

    pthread_t threads[nbrOfThreads];
    pthread_attr_t attr;

    /* Initialize and set thread detached attribute */
    pthread_attr_init(&attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

    unsigned int t;
    int rc;
    for(t=0; t<nbrOfThreads; t++)
    {
      printf("Creating thread %ld\n", t);
      if (mode == 1)
      {
        rc = pthread_create(&threads[t], &attr, writeThread, (void *)t);
      }
      else
      {
        int shmid = createSharedMem(t);
        rc = pthread_create(&threads[t], &attr, readThread, (void *)shmid);
      }

      if (rc)
      {
        printf("ERROR; return code from pthread_create() is %d\n", rc);
        exit(-1);
      }
    }

    void* status;
    pthread_attr_destroy(&attr);
    for(t=0; t<nbrOfThreads; t++)
    {
      pthread_join(threads[t], &status);
    }

    printf("All %s threads finished, exit\n",mode == 1 ? "write":"read");

    exit(0);
}

[3] ARMv6 processor cpuinfo
-bash-3.2# cat /proc/version
Linux version 2.6.32-rc6 ([email protected]) (gcc version 4.2.2) #4 Thu Dec 3 12:31:13 CET 2009
-bash-3.2# cat /proc/cpuinfo
Processor       : ARMv6-compatible processor rev 4 (v6l)
BogoMIPS        : 398.13
Features        : swp half thumb fastmult vfp edsp java
CPU implementer : 0x41
CPU architecture: 6TEJ
CPU variant     : 0x0
CPU part        : 0xb36
CPU revision    : 4

[4] Log from Demoprogramm, not working

-bash-3.2# ./shtest2.sh
Run shmtest2
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Write to in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Write Thread 0, addr:40961000
348: write new mesg: 0
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Creating thread 1
Shared memory Id:0
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
attach shared mem:41170000
Start Read Thread addr:41170000
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
-bash-3.2# Read from entry in_msg=1000, out_msg=0, addr=40961000
Read from entry in_msg=0, out_msg=0, addr=40961000
Read from entry in_msg=2000, out_msg=1000, addr=40961000
Read from entry in_msg=1000, out_msg=0, addr=41170000
Read from entry in_msg=3000, out_msg=1000, addr=40961000
Read from entry in_msg=4000, out_msg=1000, addr=40961000
Read from entry in_msg=2000, out_msg=0, addr=40961000
Read from entry in_msg=5000, out_msg=2000, addr=40961000
Read from entry in_msg=3000, out_msg=0, addr=41170000
Read from entry in_msg=6000, out_msg=2000, addr=40961000
Read from entry in_msg=7000, out_msg=2000, addr=40961000
Read from entry in_msg=4000, out_msg=0, addr=40961000
Read from entry in_msg=8000, out_msg=3000, addr=40961000
Read from entry in_msg=5000, out_msg=0, addr=41170000
Read from entry in_msg=9000, out_msg=3000, addr=40961000
Read from entry in_msg=10000, out_msg=3000, addr=40961000
[...]

[5] Testscript which starts 4 read processes each with
    one read thread

#!/bin/sh
echo "Compile"
gcc -o shmtest2 shmtest2.c -lpthread

echo "Run shmtest2"
./shmtest2 write 1 &
./shmtest2 read 1 &
./shmtest2 read 1 &
./shmtest2 read 1 &
./shmtest2 read 1 &

[6] working log on ARMv6

-bash-3.2# ./shtest2.sh
Run shmtest2
Write to in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40960000
Start Write Thread 0, addr:40960000
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40960000
Start Read Thread addr:40960000
Creating thread 1
Shared memory Id:0
Creating thread 2
Shared memory Id:0
Creating thread 3
Shared memory Id:0
attach shared mem:4216f000
Start Read Thread addr:4216f000
attach shared mem:4217e000
Start Read Thread addr:4217e000
attach shared mem:4218d000
Start Read Thread addr:4218d000
-bash-3.2# Read from entry in_msg=1000, out_msg=1000, addr=40960000
Read from entry in_msg=2000, out_msg=1000, addr=4216f000
Read from entry in_msg=3000, out_msg=1000, addr=4217e000
Read from entry in_msg=4000, out_msg=1000, addr=4218d000
Read from entry in_msg=5000, out_msg=2000, addr=40960000
Read from entry in_msg=6000, out_msg=2000, addr=4216f000
Read from entry in_msg=7000, out_msg=2000, addr=4217e000
Read from entry in_msg=8000, out_msg=2000, addr=4218d000
Read from entry in_msg=9000, out_msg=3000, addr=40960000
Read from entry in_msg=10000, out_msg=3000, addr=4216f000
Read from entry in_msg=11000, out_msg=3000, addr=4217e000
Read from entry in_msg=12000, out_msg=3000, addr=4218d000
Read from entry in_msg=13000, out_msg=4000, addr=40960000
Read from entry in_msg=14000, out_msg=4000, addr=4216f000
Read from entry in_msg=15000, out_msg=4000, addr=4217e000
Read from entry in_msg=16000, out_msg=4000, addr=4218d000
[...]

Could you elaborate what you are expecting to see?
You are "read-modify-write"'ing a shared location without using atomic instructions, so it seems reasonable that the updates of the entry->in_msg are not behaving like you want them to behave.

Joerg

I think this is messed up - this is not from your first script but your second script which starts four independent read processes.

I determined this because:
(1) the read thread addresses are mostly the same
(2) there are four "Read form in_msg" strings, which you only print once at the start of the program.

Hello Russell King,

Ups, sorry for the confusion, here 2 logs with the right values:

-bash-3.2# cat shtest2.sh
#!/bin/sh
echo "Run shmtest2"
./shmtest2 write 1 &
./shmtest2 read 4 &

-bash-3.2# cat shmtest2.c
#include <pthread.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

extern void exit();

struct Entry
{

 char          ident_name[1000];
 unsigned int  tipc_nr;
 unsigned int  pid;
 unsigned int  in_msg;
 unsigned int  out_msg;
 unsigned int  rxQueueLength;

};

void* attachSharedMem(int shmid)
{
void* addr = shmat(shmid, NULL, 0);
if ((addr != 0) && (0xffffffff != (unsigned int)addr))

  {
    printf("attach shared mem:%x\n",addr);
  }
  else
  {
    printf("shmat failed");
    addr = 0;

}
return addr;
}

int createSharedMem()
{

  key_t   key     = 1000;          /* key to be passed to shmget() */
  int     shmflg;                  /* shmflg to be passed to shmget() */
  int     shmid;                   /* return value from shmget() */
  int     size;                    /* size to be passed to shmget() */

  size   = 60000;
  shmflg = IPC_CREAT | 0666;
  if ((shmid = shmget (key, size, shmflg)) == -1)
  {
    printf("shmget failed");
    shmid = 0;
  }

  printf("Shared memory Id:%d\n",shmid);

return shmid;
}

void* setupSharedMem()
{
int shmid = createSharedMem();
void* addrShm = attachSharedMem(shmid);
return addrShm;
}

void *readThread(void *t)
{
struct Entry* entry = 0;

  int shmid = (int)t;
  void* addrShm = attachSharedMem(shmid);

  if (addrShm != 0)
  {
    printf("Start Read Thread addr:%x\n",addrShm);
    entry = (struct Entry*)addrShm;
    entry->in_msg  = 0;
    entry->out_msg = 0;

    int i=0;
    while(i < 60)
    {
      entry->in_msg += 1000;
      sleep(1);
      printf("%d Read from entry in_msg=%d, out_msg=%d, addr=%x\n", getpid(), entry->in_msg,entry->out_msg, addrShm);
      i++;
    }

}

pthread_exit(NULL);
}

void *writeThread(void *t)
{
struct Entry* entry = 0;
unsigned int threadId = (unsigned int)t;
void* addrShm = setupSharedMem();

  if (addrShm != 0)
  {
    printf("Start Write Thread %d, addr:%x\n",threadId,addrShm);
    entry = (struct Entry*)addrShm;
    strcpy(entry->ident_name,"this is a test entry");
    entry->in_msg  = 0;
    entry->out_msg = 0;
    entry->rxQueueLength = 20000;
    entry->pid     = threadId;
    entry->tipc_nr = 1000;

    int i=0;
    while(i < 60)
    {
        printf("%d: write new mesg: %d\n", getpid(), entry->out_msg);
      entry->out_msg += 1000;
      sleep(1);
      //printf("Write in entry with threadId=%d\n", threadId);
      i++;
    }

}

pthread_exit(NULL);
}

main(int argc, char* argv[])
{

    //check the arguments
    if (argc != 3)
    {
       printf("Arguments are [read|write] [number of threads]\n");
       exit(1);
    }

    unsigned int mode         = 0;
    unsigned int nbrOfThreads = 0;

    if (strcmp(argv[1],"write") == 0)
    {
      printf("Write to in_msg\n");
      mode = 1;
    }

    if (strcmp(argv[1],"read") == 0)
    {
      printf("Read from in_msg\n");
      mode = 2;
    }

    nbrOfThreads = atoi(argv[2]);

    pthread_t threads[nbrOfThreads];
    pthread_attr_t attr;

    /* Initialize and set thread detached attribute */
    pthread_attr_init(&attr);
    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

    unsigned int t;
    int rc;
    for(t=0; t<nbrOfThreads; t++)
    {
      printf("Creating thread %ld\n", t);
      if (mode == 1)
      {
        rc = pthread_create(&threads[t], &attr, writeThread, (void *)t);
      }
      else
      {
        int shmid = createSharedMem(t);
        rc = pthread_create(&threads[t], &attr, readThread, (void *)shmid);
      }

      if (rc)
      {
        printf("ERROR; return code from pthread_create() is %d\n", rc);
        exit(-1);
      }
    }

    void* status;
    pthread_attr_destroy(&attr);
    for(t=0; t<nbrOfThreads; t++)
    {
      pthread_join(threads[t], &status);
    }

    printf("All %s threads finished, exit\n",mode == 1 ? "write":"read");

    exit(0);
}

-bash-3.2# ./shtest2.sh
Run shmtest2
Write to in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Write Thread 0, addr:40961000
411: write new mesg: 0
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Creating thread 1
Shared memory Id:0
attach shared mem:41170000
Start Read Thread addr:41170000
Creating thread 2
Shared memory Id:0
attach shared mem:4197f000
Start Read Thread addr:4197f000
Creating thread 3
Shared memory Id:0
attach shared mem:4218e000
Start Read Thread addr:4218e000
-bash-3.2# 411: write new mesg: 0
413 Read from entry in_msg=1000, out_msg=0, addr=40961000 413 Read from entry in_msg=2000, out_msg=0, addr=41170000 413 Read from entry in_msg=3000, out_msg=0, addr=4197f000 413 Read from entry in_msg=4000, out_msg=0, addr=4218e000 411: write new mesg: 1000
413 Read from entry in_msg=1000, out_msg=1000, addr=40961000 413 Read from entry in_msg=2000, out_msg=1000, addr=41170000 413 Read from entry in_msg=3000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=4000, out_msg=1000, addr=4218e000 411: write new mesg: 2000
413 Read from entry in_msg=5000, out_msg=1000, addr=40961000 413 Read from entry in_msg=6000, out_msg=1000, addr=41170000 413 Read from entry in_msg=7000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=8000, out_msg=1000, addr=4218e000 411: write new mesg: 3000
413 Read from entry in_msg=9000, out_msg=1000, addr=40961000 413 Read from entry in_msg=10000, out_msg=1000, addr=41170000 413 Read from entry in_msg=11000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=12000, out_msg=1000, addr=4218e000 411: write new mesg: 4000
413 Read from entry in_msg=13000, out_msg=1000, addr=40961000 413 Read from entry in_msg=14000, out_msg=1000, addr=41170000 413 Read from entry in_msg=15000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=16000, out_msg=1000, addr=4218e000 411: write new mesg: 5000
413 Read from entry in_msg=17000, out_msg=1000, addr=40961000 413 Read from entry in_msg=18000, out_msg=1000, addr=41170000 413 Read from entry in_msg=19000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=20000, out_msg=1000, addr=4218e000 411: write new mesg: 6000
413 Read from entry in_msg=21000, out_msg=1000, addr=40961000 413 Read from entry in_msg=22000, out_msg=1000, addr=41170000 413 Read from entry in_msg=23000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=24000, out_msg=1000, addr=4218e000 411: write new mesg: 7000
413 Read from entry in_msg=25000, out_msg=1000, addr=40961000 413 Read from entry in_msg=26000, out_msg=1000, addr=41170000 413 Read from entry in_msg=27000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=28000, out_msg=1000, addr=4218e000 411: write new mesg: 8000
413 Read from entry in_msg=29000, out_msg=1000, addr=40961000 413 Read from entry in_msg=30000, out_msg=1000, addr=41170000 413 Read from entry in_msg=31000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=32000, out_msg=1000, addr=4218e000 411: write new mesg: 9000
413 Read from entry in_msg=33000, out_msg=1000, addr=40961000 413 Read from entry in_msg=34000, out_msg=1000, addr=41170000 413 Read from entry in_msg=35000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=36000, out_msg=1000, addr=4218e000 411: write new mesg: 10000
413 Read from entry in_msg=37000, out_msg=1000, addr=40961000 413 Read from entry in_msg=38000, out_msg=1000, addr=41170000 413 Read from entry in_msg=39000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=40000, out_msg=1000, addr=4218e000 411: write new mesg: 11000
413 Read from entry in_msg=41000, out_msg=1000, addr=40961000 413 Read from entry in_msg=42000, out_msg=1000, addr=41170000 413 Read from entry in_msg=43000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=44000, out_msg=1000, addr=4218e000 411: write new mesg: 12000
413 Read from entry in_msg=45000, out_msg=1000, addr=40961000 413 Read from entry in_msg=46000, out_msg=1000, addr=41170000 413 Read from entry in_msg=47000, out_msg=1000, addr=4197f000 413 Read from entry in_msg=48000, out_msg=1000, addr=4218e000 411: write new mesg: 13000

-bash-3.2# cat shtest2.sh
#!/bin/sh
echo "Run shmtest2"
./shmtest2 write 1 &
./shmtest2 read 1 &
./shmtest2 read 1 &
./shmtest2 read 1 &
./shmtest2 read 1 &

-bash-3.2# ./shtest2.sh
Run shmtest2
Write to in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Write Thread 0, addr:40961000
423: write new mesg: 0
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
-bash-3.2# 423: write new mesg: 0
425 Read from entry in_msg=1000, out_msg=1000, addr=40961000 427 Read from entry in_msg=2000, out_msg=1000, addr=40961000 429 Read from entry in_msg=3000, out_msg=1000, addr=40961000 431 Read from entry in_msg=4000, out_msg=1000, addr=40961000 423: write new mesg: 1000
425 Read from entry in_msg=5000, out_msg=2000, addr=40961000 427 Read from entry in_msg=6000, out_msg=2000, addr=40961000 429 Read from entry in_msg=7000, out_msg=2000, addr=40961000 431 Read from entry in_msg=8000, out_msg=2000, addr=40961000 423: write new mesg: 2000
425 Read from entry in_msg=9000, out_msg=3000, addr=40961000 427 Read from entry in_msg=10000, out_msg=3000, addr=40961000 429 Read from entry in_msg=11000, out_msg=3000, addr=40961000 431 Read from entry in_msg=12000, out_msg=3000, addr=40961000 423: write new mesg: 3000
425 Read from entry in_msg=13000, out_msg=4000, addr=40961000 427 Read from entry in_msg=14000, out_msg=4000, addr=40961000 429 Read from entry in_msg=15000, out_msg=4000, addr=40961000 431 Read from entry in_msg=16000, out_msg=4000, addr=40961000 423: write new mesg: 4000
425 Read from entry in_msg=17000, out_msg=5000, addr=40961000 427 Read from entry in_msg=18000, out_msg=5000, addr=40961000 429 Read from entry in_msg=19000, out_msg=5000, addr=40961000 431 Read from entry in_msg=20000, out_msg=5000, addr=40961000 423: write new mesg: 5000
425 Read from entry in_msg=21000, out_msg=6000, addr=40961000 427 Read from entry in_msg=22000, out_msg=6000, addr=40961000 429 Read from entry in_msg=23000, out_msg=6000, addr=40961000 431 Read from entry in_msg=24000, out_msg=6000, addr=40961000 423: write new mesg: 6000
425 Read from entry in_msg=25000, out_msg=7000, addr=40961000 427 Read from entry in_msg=26000, out_msg=7000, addr=40961000 429 Read from entry in_msg=27000, out_msg=7000, addr=40961000 431 Read from entry in_msg=28000, out_msg=7000, addr=40961000 423: write new mesg: 7000
425 Read from entry in_msg=29000, out_msg=8000, addr=40961000 427 Read from entry in_msg=30000, out_msg=8000, addr=40961000 429 Read from entry in_msg=31000, out_msg=8000, addr=40961000 431 Read from entry in_msg=32000, out_msg=8000, addr=40961000 423: write new mesg: 8000
425 Read from entry in_msg=33000, out_msg=9000, addr=40961000 427 Read from entry in_msg=34000, out_msg=9000, addr=40961000 429 Read from entry in_msg=35000, out_msg=9000, addr=40961000 431 Read from entry in_msg=36000, out_msg=9000, addr=40961000

Thanks!

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

I don't remember whether 2.6.28 has highmem. Can you check whether you're using highmem please?

Hello Russell King,

No, there is no HIGHMEM in the defconfig, if you mean this config option.

You can find the complete defconfig here:

http://git.denx.de/?p=linux-2.6-denx.git;a=blob;f=arch/arm/configs/suen3_defconfig;h=905f9bc6a36c03ab41b97a292fca5b3a0011a2a0;hb=3dc3e4dc957f216ec5cdcd2a770f5c4b0cefca4a

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Please apply this patch and re-run your program. Expect a fair bit of output - the last 50 or so lines from the kernel messages should cover everything that's required. Also useful would be the output from the program giving the addresses of the shm regions.

(This patch is a little messy because it contains a bit of restructuring for 2.6.32 bug fixing...)

index 7296022..c457920 100644
@@ -27,6 +27,8 @@
 
 static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE;
 
+#define is_debug() (strcmp(current->comm, "shmtest2") == 0)
+
 /*
  * We take the easy way out of this problem - we make the
  * PTE uncacheable.  However, we leave the write buffer on.

@@ -36,33 +38,18 @@ static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE;
  * Therefore those configurations which might call adjust_pte (those
  * without CONFIG_CPU_CACHE_VIPT) cannot support split page_table_lock.
  */
-static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
+static int do_adjust_pte(struct vm_area_struct *vma, unsigned long address,
+	pte_t *ptep)
 {
-	pgd_t *pgd;
-	pmd_t *pmd;
-	pte_t *pte, entry;
+	pte_t entry = *ptep;
 	int ret;
 
-	pgd = pgd_offset(vma->vm_mm, address);
-	if (pgd_none(*pgd))
-		goto no_pgd;
-	if (pgd_bad(*pgd))
-		goto bad_pgd;
-
-	pmd = pmd_offset(pgd, address);
-	if (pmd_none(*pmd))
-		goto no_pmd;
-	if (pmd_bad(*pmd))
-		goto bad_pmd;
-
-	pte = pte_offset_map(pmd, address);
-	entry = *pte;
-
 	/*
 	 * If this page is present, it's actually being shared.
 	 */
 	ret = pte_present(entry);
-
+if (is_debug()) printk("%s:%d: vma %p addr %lx pte %08lx\n",
+  current->comm, current->pid, vma, address, pte_val(entry));
 	/*
 	 * If this page isn't present, or is already setup to
 	 * fault (ie, is old), we can safely ignore any issues.

@@ -74,23 +61,36 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
 				  (pfn << PAGE_SHIFT) + PAGE_SIZE);
 		pte_val(entry) &= ~L_PTE_MT_MASK;
 		pte_val(entry) |= shared_pte_mask;
-		set_pte_at(vma->vm_mm, address, pte, entry);
+		set_pte_at(vma->vm_mm, address, ptep, entry);
 		flush_tlb_page(vma, address);
+if (is_debug()) printk("%s:%d: modified %08lx\n", current->comm, current->pid, pte_val(entry));
 	}
-	pte_unmap(pte);
 	return ret;
+}
+
+static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
+{
+	pgd_t *pgd;
+	pmd_t *pmd;
+	pte_t *pte;
+	int ret;
 
-bad_pgd:
-	pgd_ERROR(*pgd);
-	pgd_clear(pgd);
-no_pgd:
-	return 0;
-
-bad_pmd:
-	pmd_ERROR(*pmd);
-	pmd_clear(pmd);
-no_pmd:
-	return 0;
+	pgd = pgd_offset(vma->vm_mm, address);
+	if (pgd_none_or_clear_bad(pgd))
+		return 0;
+
+	pmd = pmd_offset(pgd, address);
+	if (pmd_none_or_clear_bad(pmd))
+		return 0;
+
+	/*
+	 * This may be called while another page table is mapped.
+	 */
+	pte = pte_offset_map_nested(pmd, address);
+	ret = do_adjust_pte(vma, address, pte);
+	pte_unmap_nested(pte);
+
+	return ret;
 }
 
 static void

@@ -105,6 +105,9 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
 
 	pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);
 
+if (is_debug()) printk("%s:%d: %p %p %lx %lx %lx\n",
+ current->comm, current->pid, mapping, vma, pgoff, addr, pfn);
+
 	/*
 	 * If we have any shared mappings that are in the same mm
 	 * space, then we need to handle them specially to maintain

@@ -125,6 +128,7 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
 		aliases += adjust_pte(mpnt, mpnt->vm_start + offset);
 	}
 	flush_dcache_mmap_unlock(mapping);
+if (is_debug()) printk("%s:%d: aliases %d\n", current->comm, current->pid, aliases);
 	if (aliases)
 		adjust_pte(vma, addr);
 	else

BTW, so that other people know: the use of highmem on VIVT caches _will_ result in buggy shared mmap[*] behaviour similar to what you're seeing; the ARM fixup in update_mmu_cache() is rendered inoperative for the existing mappings.

Luckily, however, when the conditions for applying the fixup are triggered, you _will_ get a kernel oops.

Since the fix for this involves touching every other architecture (because we need the page table pointer passed into update_mmu_cache), I will not be forwarding this fix for 2.6.32.X - the patch is just far too big to consider.

That all said, Heiko, since you're not using highmem, this will not be affecting you.

* - this occurs when the same physical pages are mapped multiple times into the same MM space. Our work-around for this is to mark the pages non-cachable bufferable, or if the write buffer fails the coherency test, non-cachable non-bufferable.

Could you apply this patch and test again:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=08e445bd6a

Nicolas

That's almost certainly the problem, thanks Nicolas.

Hello Nicolas Pitre,

Thanks for your reply. I applied this patch, but it didn;t help. Same output as without it:

-bash-3.2# cat shtest2.sh
#!/bin/sh
echo "Run shmtest2"
./shmtest2 write 1 &
./shmtest2 read 4 &

-bash-3.2# ./shtest2.sh
Run shmtest2
Write to in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Write Thread 0, addr:40961000
358: write new mesg: 0
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Creating thread 1
Shared memory Id:0
attach shared mem:41170000
Start Read Thread addr:41170000
Creating thread 2
Shared memory Id:0
attach shared mem:4197f000
Start Read Thread addr:4197f000
Creating thread 3
Shared memory Id:0
attach shared mem:4218e000
Start Read Thread addr:4218e000
-bash-3.2# 358: write new mesg: 0
360 Read from entry in_msg=1000, out_msg=0, addr=40961000 360 Read from entry in_msg=2000, out_msg=0, addr=41170000 360 Read from entry in_msg=3000, out_msg=0, addr=4197f000 360 Read from entry in_msg=4000, out_msg=0, addr=4218e000 358: write new mesg: 1000
360 Read from entry in_msg=5000, out_msg=0, addr=40961000 360 Read from entry in_msg=6000, out_msg=0, addr=41170000 360 Read from entry in_msg=7000, out_msg=0, addr=4197f000 360 Read from entry in_msg=8000, out_msg=0, addr=4218e000 358: write new mesg: 2000
360 Read from entry in_msg=9000, out_msg=0, addr=40961000 360 Read from entry in_msg=10000, out_msg=0, addr=41170000 360 Read from entry in_msg=11000, out_msg=0, addr=4197f000 360 Read from entry in_msg=12000, out_msg=0, addr=4218e000 358: write new mesg: 3000
360 Read from entry in_msg=13000, out_msg=0, addr=40961000 360 Read from entry in_msg=14000, out_msg=0, addr=41170000 360 Read from entry in_msg=15000, out_msg=0, addr=4197f000 360 Read from entry in_msg=16000, out_msg=0, addr=4218e000 358: write new mesg: 4000
360 Read from entry in_msg=17000, out_msg=0, addr=40961000 360 Read from entry in_msg=18000, out_msg=0, addr=41170000 360 Read from entry in_msg=19000, out_msg=0, addr=4197f000 360 Read from entry in_msg=20000, out_msg=0, addr=4218e000 358: write new mesg: 5000

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Hello Russell King,

Thanks for your patch. I tried it and it gives with my testprogramm the following output:

-bash-3.2# ./shtest2.sh
Run shmtest2
shmtest2:348: c6c0bea8 c723bf98 0 40000000 7404 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 0 40000000 7404 shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 1 40001000 7403 shmtest2:349: aliases 0
-bash-3.2# shmtest2:348: c6c0bea8 c723bf98 1 40001000 7403 shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c721cee8 1b 40023000 7410 shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 16 40016000 740e shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c7217700 1b 40023000 7410 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 8 40008000 7422 shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 13 40013000 7425 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 16 40016000 740e shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 14 40014000 7424 shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 15 40015000 7409 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 8 40008000 7422 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 2 40002000 741b shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 3 40003000 741a shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 13 40013000 7425 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 9 40009000 7421 shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 a 4000a000 7420 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 14 40014000 7424 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 19 40019000 7412 shmtest2:348: aliases 0
shmtest2:348: c6ccff08 c7217338 0 8000 64d0 shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 7 40007000 741c shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 15 40015000 7409 shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 2 40002000 741b shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 1a 4001a000 7411 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 3 40003000 741a shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 17 40017000 7414 shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 e 4000e000 7423 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 9 40009000 7421 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 b 4000b000 741f shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 c 4000c000 741e shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 a 4000a000 7420 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 d 4000d000 741d shmtest2:348: aliases 0
shmtest2:348: c6c0bea8 c723bf98 5 40005000 7418 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 19 40019000 7412 shmtest2:349: aliases 0
shmtest2:349: c6ccff08 c723bee8 0 8000 64d0 shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 7 40007000 741c shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 6 40006000 7417 shmtest2:348: aliases 0
shmtest2:348: c6c94d08 c722c498 12 4003f000 69bd shmtest2:348: aliases 0
shmtest2:348: c6c94d08 c722ce38 0 40025000 6991 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 1a 4001a000 7411 shmtest2:349: aliases 0
shmtest2:348: c6c94d08 c722ce38 3 40028000 69cd shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 17 40017000 7414 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 10 40010000 7428 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226230 111 4015c000 7435 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 e 4000e000 7423 shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 b 4000b000 741f shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 0 40043000 742f shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 c 4000c000 741e shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 d 4000d000 741d shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 10 40053000 7446 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 5 40005000 7418 shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 6 40006000 7417 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 4 40004000 7419 shmtest2:348: aliases 0
shmtest2:349: c6c94d08 c722c128 12 4003f000 69bd shmtest2:349: aliases 0
shmtest2:349: c6c94d08 c722c0d0 0 40025000 6991 shmtest2:349: aliases 0
shmtest2:348: c6c0bea8 c723bf98 f 4000f000 7429 shmtest2:348: aliases 0
shmtest2:349: c6c94d08 c722c0d0 3 40028000 69cd shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 10 40010000 7428 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c230 111 4015c000 7435 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 11 40054000 744c shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 0 40043000 742f shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 10 40053000 7446 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226230 110 4015b000 7436 shmtest2:348: aliases 0
shmtest2:349: c6c0bea8 c721c968 4 40004000 7419 shmtest2:349: aliases 0
shmtest2:349: c6c0bea8 c721c968 f 4000f000 7429 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 12 40055000 744b shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 11 40054000 744c shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c230 110 4015b000 7436 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 13 40056000 744a shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 12 40055000 744b shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 13 40056000 744a shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 14 40057000 7449 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 14 40057000 7449 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 5 40048000 7450 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 1 40044000 742e shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 5 40048000 7450 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 8 4004b000 7459 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 f 40052000 7440 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 1 40044000 742e shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 2 40045000 7453 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 3 40046000 7452 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 8 4004b000 7459 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 b 4004e000 7444 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 4 40047000 7451 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 f 40052000 7440 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 a 4004d000 7445 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 c 4004f000 7443 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 2 40045000 7453 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 6 40049000 7454 shmtest2:348: aliases 0
shmtest2:348: c6c94d08 c722ce38 1 40026000 69b0 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 3 40046000 7452 shmtest2:349: aliases 0
shmtest2:348: c6c94d08 c722ce38 2 40027000 69ce shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 9 4004c000 743f shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 b 4004e000 7444 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 7 4004a000 745a shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 d 40050000 7442 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 4 40047000 7451 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 e 40051000 7441 shmtest2:348: aliases 0
shmtest2:348: c6c94d08 c722ce38 4 40029000 69cc shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 a 4004d000 7445 shmtest2:349: aliases 0
shmtest2:348: c6c94d08 c722ce38 10 40035000 69bf shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 6d 400b0000 746a shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 c 4004f000 7443 shmtest2:349: aliases 0
shmtest2:348: c6c94d08 c722ce38 e 40033000 69ba shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 6 40049000 7454 shmtest2:349: aliases 0
shmtest2:349: c6c94d08 c722c0d0 1 40026000 69b0 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 6e 400b1000 7470 shmtest2:348: aliases 0
shmtest2:349: c6c94d08 c722c0d0 2 40027000 69ce shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 9 4004c000 743f shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 b5 400f8000 7458 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 7 4004a000 745a shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 d 40050000 7442 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 8d 400d0000 749e shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 e 40051000 7441 shmtest2:349: aliases 0
shmtest2:349: c6c94d08 c722c0d0 4 40029000 69cc shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 b6 400f9000 7457 shmtest2:348: aliases 0
shmtest2:349: c6c94d08 c722c0d0 10 40035000 69bf shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 6d 400b0000 746a shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 f1 40134000 7508 shmtest2:348: aliases 0
shmtest2:349: c6c94d08 c722c0d0 e 40033000 69ba shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 6e 400b1000 7470 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 c9 4010c000 7511 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 b5 400f8000 7458 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 8d 400d0000 749e shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 bc 400ff000 745d shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 b6 400f9000 7457 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 f1 40134000 7508 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 6c 400af000 746b shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 c9 4010c000 7511 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 bc 400ff000 745d shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 2c 4006f000 747e shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 6c 400af000 746b shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 2c 4006f000 747e shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 29 4006c000 747a shmtest2:349: aliases 0
shmtest2:349: c6ccff08 c723bee8 1 9000 64cf shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 29 4006c000 747a shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 57 4009a000 74a5 shmtest2:349: aliases 0
shmtest2:348: c6ccff08 c7217338 1 9000 64cf shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 58 4009b000 74a4 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 61 400a4000 74af shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 57 4009a000 74a5 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 58 4009b000 74a4 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 62 400a5000 74b5 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 61 400a4000 74af shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 55 40098000 74cb shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 62 400a5000 74b5 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 ad 400f0000 7492 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 55 40098000 74cb shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 af 400f2000 7490 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 ad 400f0000 7492 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 b9 400fc000 7460 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 af 400f2000 7490 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 63 400a6000 74b4 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 b9 400fc000 7460 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 f2 40135000 7507 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 ae 400f1000 7491 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 63 400a6000 74b4 shmtest2:348: aliases 0
Read from in_msgshmtest2:349: c6c0b848 c722c180 2b 4006e000 7478

shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 2d 40070000 7484 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 f2 40135000 7507 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 2e 40071000 7483 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 f5 40138000 7504 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 ae 400f1000 7491 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 101 40144000 74f0 shmtest2:349: aliases 0
shmtest2:349: c6c94d08 c722c0d0 6 4002b000 69ca shmtest2:349: aliases 0
Write to in_msgshmtest2:348: c6c0b848 c7226e38 2b 4006e000 7478

shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 43 40086000 751e shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 2d 40070000 7484 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 2e 40071000 7483 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 3b 4007e000 74b8 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 f5 40138000 7504 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 101 40144000 74f0 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 3c 4007f000 74b7 shmtest2:349: aliases 0
shmtest2:348: c6c94d08 c722ce38 6 4002b000 69ca shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 43 40086000 751e shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 54 40097000 74c5 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 3b 4007e000 74b8 shmtest2:348: aliases 0
shmtest2:349: c6c94d08 c722c0d0 c 40031000 69ae shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 102 40145000 74ef shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 3c 4007f000 74b7 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 39 4007c000 74ba shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 3d 40080000 74bd shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 54 40097000 74c5 shmtest2:348: aliases 0
Creating thread shmtest2:349: c6c0b848 c722c180 3f 40082000 74c2 0
shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 3e 40081000 74c3 shmtest2:349: aliases 0
shmtest2:348: c6c94d08 c722ce38 c 40031000 69ae shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 be 40101000 7462 shmtest2:349: aliases 0
Shared memory Idshmtest2:349: c6c94d08 c722c0d0 5 4002a000 69cb :0
shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 102 40145000 74ef shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 6a 400ad000 746d shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 39 4007c000 74ba shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 3d 40080000 74bd shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 69 400ac000 746e shmtest2:349: aliases 0
Creating thread shmtest2:348: c6c0b848 c7226e38 3f 40082000 74c2 0
shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 65 400a8000 74b2 shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 66 400a9000 7463 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 3e 40081000 74c3 shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 f0 40133000 7509 shmtest2:349: aliases 0
shmtest2:349: c6c94d08 c722c0d0 7 4002c000 69c9 shmtest2:349: aliases 0
shmtest2:348: c6c94d08 c722ce38 5 4002a000 69cb shmtest2:348: aliases 0
shmtest2:349: c6c94d08 c722c0d0 9 4002e000 69d5 shmtest2:349: aliases 0
shmtest2:349: c6c94d08 c722c0d0 8 4002d000 69cf shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 6a 400ad000 746d shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 68 400ab000 746f shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 67 400aa000 7469 shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 69 400ac000 746e shmtest2:348: aliases 0
shmtest2:349: c6c0b848 c722c180 ba 400fd000 745f shmtest2:349: aliases 0
shmtest2:349: c6c0b848 c722c180 bd 40100000 745c shmtest2:349: aliases 0
shmtest2:348: c6c0b848 c7226e38 65 400a8000 74b2 shmtest2:348: aliases 0
attach shared meshmtest2:350: c6c0b848 c722c180 c8 4010b000 750b m:40961000
shmtest2:350: aliases 0
shmtest2:348: c6c0b848 c7226e38 66 400a9000 7463 shmtest2:348: aliases 0
shmtest2:348: c6c0b848 c7226e38 f0 40133000 7509 shmtest2:348: aliases 0
Start Read Threashmtest2:350: c7220140 c7217968 0 40961000 64a1 shmtest2:350: aliases 0

Creating thread shmtest2:348: c6c94d08 c722ce38 7 4002c000 69c9 1
Shared memoryshmtest2:348: aliases 0
Id:0
shmtest2:350: c6c0b848 c722c180 8b 400ce000 74a0 shmtest2:350: aliases 0
shmtest2:348: c6c94d08 c722ce38 9 4002e000 69d5 shmtest2:348: aliases 0
shmtest2:348: c6c94d08 c722ce38 8 4002d000 69cf shmtest2:348: aliases 0
attach shared meshmtest2:351: c7220140 c72179c0 0 41170000 64a1 m:41170000
Starshmtest2:351: vma c7217968 addr 40961000 pte 064a13cf shmtest2:351: modified 064a13c7
t Read Thread adshmtest2:351: aliases 1
dr:41170000
shmtest2:351: vma c72179c0 addr 41170000 pte 064a13cf shmtest2:351: modified 064a13c7
shmtest2:348: c6c0b848 c7226e38 68 400ab000 746f shmtest2:348: aliases 0
Creating thread shmtest2:352: c7220140 c7217548 0 4197f000 64a1 2
Shared memoryshmtest2:352: vma c7217968 addr 40961000 pte 064a13c7 shmtest2:352: vma c72179c0 addr 41170000 pte 064a13c7
Id:0
attach shshmtest2:352: aliases 2
ared mem:4197f00shmtest2:352: vma c7217548 addr 4197f000 pte 064a13cf 0
Start Read Thshmtest2:352: modified 064a13c7
read addr:4197f0shmtest2:348: c6c0b848 c7226e38 67 400aa000 7469 00
shmtest2:348: aliases 0
Creating thread shmtest2:348: c6c0b848 c7226e38 ba 400fd000 745f 3
Shared memoryshmtest2:348: aliases 0
Id:0
shmtest2:348: c6c0b848 c7226e38 bd 40100000 745c shmtest2:348: aliases 0
shmtest2:353: c6c0b848 c7226e38 be 40101000 7462 shmtest2:353: aliases 0
attach shared meshmtest2:354: c7220140 c7217650 0 4218e000 64a1 m:4218e000
Starshmtest2:354: vma c7217968 addr 40961000 pte 064a13c7 shmtest2:354: vma c72179c0 addr 41170000 pte 064a13c7 shmtest2:354: vma c7217548 addr 4197f000 pte 064a13c7
t Read Thread adshmtest2:354: aliases 3
dr:4218e000
shmtest2:354: vma c7217650 addr 4218e000 pte 064a13cf shmtest2:354: modified 064a13c7
Shared memory Idshmtest2:353: c6c0b848 c7226e38 c8 4010b000 750b :0
shmtest2:353: aliases 0
attach shared meshmtest2:353: c7220140 c7217288 0 40961000 64a1 m:40961000
Starshmtest2:353: aliases 0
t Write Thread 0shmtest2:353: c6c0b848 c7226e38 8c 400cf000 749f shmtest2:353: aliases 0

348: write new mshmtest2:353: c6c0b848 c7226e38 8b 400ce000 74a0
esg: 0
shmtest2:353: aliases 0
shmtest2:350: c6c0b848 c722c180 8c 400cf000 749f shmtest2:350: aliases 0
349 Read from entry in_msg=0, out_msg=1000, addr=40961000 349 Read from entry in_msg=1000, out_msg=1000, addr=41170000 349 Read from entry in_msg=2000, out_msg=1000, addr=4197f000 349 Read from entry in_msg=3000, out_msg=1000, addr=4218e000 348: write new mesg: 1000
349 Read from entry in_msg=4000, out_msg=1000, addr=40961000 349 Read from entry in_msg=5000, out_msg=1000, addr=41170000 349 Read from entry in_msg=6000, out_msg=1000, addr=4197f000 349 Read from entry in_msg=7000, out_msg=1000, addr=4218e000 348: write new mesg: 2000
349 Read from entry in_msg=8000, out_msg=1000, addr=40961000 349 Read from entry in_msg=9000, out_msg=1000, addr=41170000 349 Read from entry in_msg=10000, out_msg=1000, addr=4197f000 349 Read from entry in_msg=11000, out_msg=1000, addr=4218e000 348: write new mesg: 3000
349 Read from entry in_msg=12000, out_msg=1000, addr=40961000 349 Read from entry in_msg=13000, out_msg=1000, addr=41170000 349 Read from entry in_msg=14000, out_msg=1000, addr=4197f000 349 Read from entry in_msg=15000, out_msg=1000, addr=4218e000 348: write new mesg: 4000

If you want a log without the printfs from the userspaceprg., I can make it.

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

OK. Then could you test with Linux version 2.6.32, just to be sure the issue still exists there?

Nicolas

Well, the kernel messages of relevance from the dump are:

shmtest2:351: c7220140 c72179c0 0 41170000 64a1 shmtest2:351: vma c7217968 addr 40961000 pte 064a13cf shmtest2:351: modified 064a13c7
shmtest2:351: aliases 1
shmtest2:351: vma c72179c0 addr 41170000 pte 064a13cf shmtest2:351: modified 064a13c7
shmtest2:352: c7220140 c7217548 0 4197f000 64a1 shmtest2:352: vma c7217968 addr 40961000 pte 064a13c7 shmtest2:352: vma c72179c0 addr 41170000 pte 064a13c7 shmtest2:352: aliases 2
shmtest2:352: vma c7217548 addr 4197f000 pte 064a13cf shmtest2:352: modified 064a13c7
shmtest2:354: c7220140 c7217650 0 4218e000 64a1 shmtest2:354: vma c7217968 addr 40961000 pte 064a13c7 shmtest2:354: vma c72179c0 addr 41170000 pte 064a13c7 shmtest2:354: vma c7217548 addr 4197f000 pte 064a13c7 shmtest2:354: aliases 3
shmtest2:354: vma c7217650 addr 4218e000 pte 064a13cf shmtest2:354: modified 064a13c7

which shows that the PTEs are having their 'cacheable' bit correctly cleared, thus making changes to the page in RAM immediately visible to the user program.

Could it be that the CPU you're using doesn't support the C=0 B=1 PTE encoding properly, and still caches data in such a region?

Hello Nicolas Pitre,

I try to start a 2.6.32 kernel, but maybe I need some time for it.

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Hello Russell King,

Ok, sounds good :-)

Hmm.. I try to find this out, thanks!

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Hi

There isn't any known issues related to working with C=0 B=1 on Kirkwood. Can you pls run this test on some other ARMv5 platform and approve it is working.

Thanks

-----Original Message-----
From: [email protected] [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Heiko Schocher
Sent: Friday, December 04, 2009 9:35 PM
To: Russell King - ARM Linux
Cc: linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

Hello Russell King,

Ok, sounds good :-)

Hmm.. I try to find this out, thanks!

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

I ran it on an ARM926EJ-S, which is ARMv5 and worked fine.

If there's no problem with C=0 B=1 mappings on Kirkwood, I've no idea what's going on, and I don't have any suggestion on what to try next.

The log shows that the kernel is doing the right thing: when we detect two mappings for the same page in the same MM space, we clean and invalidate any existing cacheable mappings visible in the MM space (both L1 and L2), and switch all visible mappings to C=0 B=1 mappings. This makes the area non-cacheable.

That means updates to the area should be visible via any mapping in the same MM space as soon as they're written to any other mapping.

The above is the result of the second mapping of the same physical page occuring in the current MM space: we switch the mappings at 0x40961000 and 0x41170000 virtual to C=0 B=1 mode.

We detect a third mapping, and the previous two mappings are already C=0 B=1 mode, so there's no need to touch them. The new mapping at 0x4197f000 is modified to C=0 B=1 mode.

And here is the forth mapping, and the previous two mappings are already C=0 B=1 mode, so there's no need to touch them. The new mapping at 0x4218e000 is modified to C=0 B=1 mode.

Just run the test on kernel 2.6.31 and reproduce the failure, I will look into it and update...

Regards

-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Sunday, December 06, 2009 4:16 PM
To: Ronen Shitrit
Cc: hs at denx.de; linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

I ran it on an ARM926EJ-S, which is ARMv5 and worked fine.

If there's no problem with C=0 B=1 mappings on Kirkwood, I've no idea what's going on, and I don't have any suggestion on what to try next.

The log shows that the kernel is doing the right thing: when we detect two mappings for the same page in the same MM space, we clean and invalidate any existing cacheable mappings visible in the MM space (both L1 and L2), and switch all visible mappings to C=0 B=1 mappings. This makes the area non-cacheable.

That means updates to the area should be visible via any mapping in the same MM space as soon as they're written to any other mapping.

The above is the result of the second mapping of the same physical page occuring in the current MM space: we switch the mappings at 0x40961000 and 0x41170000 virtual to C=0 B=1 mode.

We detect a third mapping, and the previous two mappings are already C=0 B=1 mode, so there's no need to touch them. The new mapping at 0x4197f000 is modified to C=0 B=1 mode.

And here is the forth mapping, and the previous two mappings are already C=0 B=1 mode, so there's no need to touch them. The new mapping at 0x4218e000 is modified to C=0 B=1 mode.

Hello Russell,

Gilles Chanteperdrix <[email protected]> suggested

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

does it have L2 cache?

what about the PTE of the MM space of the write process? if it remains C=1 B=1, then it's data will be at the L2, and as the L2 is not flushed on context switch, then that explains this behavior.

No.

That's probably the issue, and it means that _all_ shared writable mappings on your processor will be broken.

Oh dear, that really is bad news.

There are two solutions to this which I can currently think of:
1. flush the L2 cache on every context switch
2. make all shared writable mappings non-cacheable

Neither of those two options appeals. Since it's only one set of CPUs which are affected, we really don't want to apply any fix for this to the generic ARM kernel code - especially when all other L2 caches are sensibly implemented as PIPT rather than VIVT.

Can we please forget that Feroceon CPUs exist? ;)

It's definitely this issue.
If I disable L2 everything works.
Also using Russell debug prints shows that adjust_pte is never called for the write process page.

I think option 2 below is preferred, we don't want to flush the entire L2 for each context switch, it will be a performance killer.

Regards

-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Monday, December 07, 2009 1:42 PM
To: saeed bishara
Cc: Ronen Shitrit; hs at denx.de; linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

No.

That's probably the issue, and it means that _all_ shared writable mappings on your processor will be broken.

Oh dear, that really is bad news.

There are two solutions to this which I can currently think of:
1. flush the L2 cache on every context switch
2. make all shared writable mappings non-cacheable

Neither of those two options appeals. Since it's only one set of CPUs which are affected, we really don't want to apply any fix for this to the generic ARM kernel code - especially when all other L2 caches are sensibly implemented as PIPT rather than VIVT.

Can we please forget that Feroceon CPUs exist? ;)

Hello Russell,

[...]

Hmm.. I tried also the testprg with CACHE_FEROCEON_L2 deaktivated, same result ...

Indeed.

To clarify, the testprg runs fine, if I start 4 processes each with only one read thread. In this case all works as expected. The mess begins only, if one read process starts more than one read thread ...

;-)

bye
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Hello Ronen,

How did you exactly disabled the L2 Cache? And on which system did you this test?

thanks
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Disable CONFIG_CACHE_FEROCEON_L2 in the config.
I'm using KW-DB kernel 2.6.31.4.

Regards

-----Original Message-----
From: Heiko Schocher [mailto:hs at denx.de]
Sent: Monday, December 07, 2009 2:27 PM
To: Ronen Shitrit
Cc: Russell King - ARM Linux; saeed bishara; linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

Hello Ronen,

How did you exactly disabled the L2 Cache? And on which system did you this test?

thanks
Heiko

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

-----Original Message-----
From: Heiko Schocher [mailto:hs at denx.de]
Sent: Monday, December 07, 2009 2:24 PM
To: Russell King - ARM Linux
Cc: saeed bishara; Ronen Shitrit; linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

Hello Russell,

[...]

Hmm.. I tried also the testprg with CACHE_FEROCEON_L2 deaktivated, same result ...

Indeed.

To clarify, the testprg runs fine, if I start 4 processes each with only one read thread. In this case all works as expected. The mess begins only, if one read process starts more than one read thread ...

[Ronen Shitrit] That also match the theory:
When using different processes, the shared area will stay C=1 B=1, On each context switch L1 will be flushed,
Since L2 is PIPT next process will get the correct data...

Hang on - if L2 is PIPT, then there shouldn't be a problem provided it's searched with C=0 B=1 mappings. Is that the case?

Beware that if you're using the original Marvell provided U-Boot, you need to disable L2 in the kernel _and_ in the U-Boot environment. If L2 is disabled in the kernel but not in U-Boot then stranger things may happen. Best thing to do is to keep L2 in U-Boot always disabled and let the kernel enable it as needed.

Nicolas

I don't have the time to properly wrap my brain around the current issue at the moment. However there are 3 facts to account for:

1) Only 2 ARMv5 CPU variants with L2 cache exist: Feroceon and XSC3. However this issue should affect both equally.

2) L2 cache is PIPT in both cases.

3) From commit 08e445bd6a which fixed such a similar issue on Feroceon and XSC3:

Ideally, we would make L1 uncacheable and L2 cacheable as L2 is PIPT. But Feroceon does not support that combination, and the TEX=5 C=0 B=0 encoding for XSc3 doesn't appear to work in practice.

Hope this helps.

Nicolas

Officially, ARMv5 does not support the 'extended small page' format for the 2nd level descriptors (ARMv6 and Xscale CPUs added this support.)

That means ARMv5 officially only has support for two bits to control the caching attributes - the C and B bits. This means we can't specify the policy for the L2 cache... unless Feroceon also supports the 'extended small page' format. This might be a third solution to the problem, and probably the best - provided Feroceon will allow us to specify 'inner non-cacheable outer write-back' (TEX=111 CB=00).

If Feroceon did support the 'extended small page' format, why isn't it already using this? (It's setup to use the architecturally defined 'small page' format for ARMv3 to ARMv5.)

Last time I checked the Feroceon doc, there was no way to have inner non-cacheable outer write-back behavior. And as I mentioned in my previous email, while debugging the issue on an XSC3, the TEX=111 CB=00 combination didn't appear to behave as expected (no one bothered to verify my findings at the time either). So I concluded that there is no such thing as inner non-cacheable outer write-back on ARMv5. This was consigned in commit 08e445bd6a.

Nicolas

No TEX bits in KW.

-----Original Message-----
From: Nicolas Pitre [mailto:nico at fluxnic.net]
Sent: Monday, December 07, 2009 5:57 PM
To: Russell King - ARM Linux
Cc: Ronen Shitrit; hs at denx.de; saeed bishara; linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

Last time I checked the Feroceon doc, there was no way to have inner non-cacheable outer write-back behavior. And as I mentioned in my previous email, while debugging the issue on an XSC3, the TEX=111 CB=00 combination didn't appear to behave as expected (no one bothered to verify my findings at the time either). So I concluded that there is no such thing as inner non-cacheable outer write-back on ARMv5. This was consigned in commit 08e445bd6a.

Nicolas

Sigh, why do people create this kind of hardware brokenness.

It seems the original commit (08e445bd6a) only partly addresses the problem; it's broken in so many other ways, as is highlighted by this test case. Was it originally created for Xscale3 or Feroceon? Was the problem actually found to exist on Xscale3 and Feroceon?

Any read or write via another cacheable mapping will result in the L2 being loaded with data. One instance is as shown in the original posters test program - where a shared writable mapping exists in another process.

Another case would be having a shared writable mapping, and using read()/ write() on the mapped file. This is normally taken care of with flush_dcache_page(), but this does not do any L2 cache maintainence on Feroceon.

Another case is any kind of mmap() of the same file - in other words, it doesn't have to be another shared mmap to bring data into the L2 cache.

Now, at first throught, if we disable the cache for all shared writable mappings in addition to what we're already doing, does this solve the problem? Well, it means that the writes will bypass the caches and hit the RAM directly. The reads from the other shared mappings will read direct from the RAM.

A private mapping using the same page will use the same page, and it will not be marked uncacheable. Accesses to it will draw data into the L2 cache.

PIO kernel mode accesses will also use the cached copy, and that _is_ a problem - it means when we update the backing file on disk, we'll write out the L2 cached data rather than what really should be written out - the updated data from the writable shared mappings.

So it seems that at least these affected CPUs need flush_dcache_page() to also do L2 cache maintainence. I don't think that's enough to cover all cases though - it probably also needs to do L2 cache maintainence in all the other flush_cache_* functions as well.

This is something that should be benchmarked on the affected CPUs and compared with the unmodified code with L2 cache disabled.

As a side note, I'm currently concerned that the sequence:

	mmap(MAP_SHARED);
	write to shared mapping;
	msync(MS_SYNC);

may not result in the written data hitting the disk (due to missing a cache flush) but as yet I'm unable to prove it. Since I now get lost reading the Linux VFS/MM code, I can't prove this by code inspection.

Checking for this isn't going to be easy - (a) munmapping the region will cause the data to hit RAM, (b) any context switch will cause the data to hit RAM, (c) merely reading back the file via read() will trigger flush_dcache_page()... Need some way to externally monitor what gets written to the storage device...

Probably because either no one cared or no one had the hardware to be able to check.

Well, there is no such thing as L2 cache on ARMv5 architecture. L2 cache is present on Xscale3 and Feroceon purely as a CPU vendor addition.

On Xscale3, it was extended to have the TEX bits in the page table. However, Feroceon, being an independently designed CPU, appears to have L2 cache but without the TEX bits - which gives us less options to solve this issue.

However, I don't think tricks like making all shared writable mappings uncacheable is going to fix it either as I mentioned in my previous message.

Consider the effect of that kind of fix: take for instance a SHM mapping between the X server and a client for transferring bitmap data (eg, mplayer). Do we really want it marked non-cacheable? I think you can say goodbye to video playback on these platforms.

It fixed a test case that was discovered on XSC3 and turned up to be valid on Feroceon as well. I probably have the source for it somewhere. The case was multiple mmap() of the same memory area within the same process. I think (but that needs confirmation) that this fixed a real life db4 issue as well.

I thought those were already handled by making L1 uncacheable (and L2 cleaned) as soon as a second user of a shared mapping was encountered.

But that case is fine, no? L2 being PIPT you get the same cached data for both mappings, and a write will COW the page.

Hmmm...

/me starts to feel the head ache

Nicolas

I have a test case for this as well.

That doesn't work - the kernel mapping will still be cacheable, and it is the kernel mapping that read() and write() will use. Their coherency issue is resolved by flush_dcache_page() performed _before_ the access (note: there is no coherency after a write access, so WBWA caches are probably broken wrt this.)

So:

	Process/Thread
	1/1		2/1		2/2		3/1	kernel
	map(MAP_SHARED)	map(MAP_SHARED)	map(MAP_SHARED)
	write mapping
								flush L1
			read mapping
					read mapping

Now, lets say we flush the L1 and L2 caches, and we mark all these mappings uncachable. Now, another process does a read from the file backing this mapping:

flush L1
read
flush_dcache_page() read data (loads L2 cache) flush L1
write mapping
(does not hit L2)
flush L1
read mapping
(does not hit L2,
sees data from 1/1)
read mapping
(does not hit L2,
sees data from 1/1)
flush L1
read
flush_dcache_page() read data (from L2 cache
and doesn't see updates from 1/1)

What I'm slightly more worried about as well is whether PIO writeouts will write the data from process 1/1 onto disk.

The point is its a way to get data into the L2 cache, which will be visible via other cachable mappings and mask the shared-mapped updates. I wasn't considering a COW to a private mapping.

You're not the only one...

I'm going to try and prove the msync() problem I mentioned at the end of my mail - it's probably going to be easier to prove and solve (and impacts more ARM CPUs than this problem.)

As for this problem, I'm not certain what the solution is.

In the mean time, as a work-around, I suggest that any CPU with L1 VIVT cache (which thereby requires the make_coherent() code) has its L2 cache disabled. That should at least allow the system to behave as correctly as it does for other ARM VIVT CPUs.

Another idea is to change the shared mapping handling, in case of vivt with pipt L2, so it won't remap the shared area as non-cacheable:
- make_coherent won't use adjust_pte and leave only the regular flush.
- Flush L1 for all context switches, also for the case that the new process is using same mm (thread context switch).

I tried it and it solved the original issue posted on this thread.

Any thoughts? Any pitfalls this solution is missing?

Regards

-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Monday, December 07, 2009 7:56 PM
To: Nicolas Pitre
Cc: Ronen Shitrit; hs at denx.de; saeed bishara; linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

I have a test case for this as well.

That doesn't work - the kernel mapping will still be cacheable, and it is the kernel mapping that read() and write() will use. Their coherency issue is resolved by flush_dcache_page() performed _before_ the access (note: there is no coherency after a write access, so WBWA caches are probably broken wrt this.)

So:

	Process/Thread
	1/1		2/1		2/2		3/1	kernel
	map(MAP_SHARED)	map(MAP_SHARED)	map(MAP_SHARED)
	write mapping
								flush L1
			read mapping
					read mapping

Now, lets say we flush the L1 and L2 caches, and we mark all these mappings uncachable. Now, another process does a read from the file backing this mapping:

flush L1
read
flush_dcache_page() read data (loads L2 cache) flush L1
write mapping
(does not hit L2)
flush L1
read mapping
(does not hit L2,
sees data from 1/1)
read mapping
(does not hit L2,
sees data from 1/1)
flush L1
read
flush_dcache_page() read data (from L2 cache
and doesn't see updates from 1/1)

What I'm slightly more worried about as well is whether PIO writeouts will write the data from process 1/1 onto disk.

The point is its a way to get data into the L2 cache, which will be visible via other cachable mappings and mask the shared-mapped updates. I wasn't considering a COW to a private mapping.

You're not the only one...

I'm going to try and prove the msync() problem I mentioned at the end of my mail - it's probably going to be easier to prove and solve (and impacts more ARM CPUs than this problem.)

As for this problem, I'm not certain what the solution is.

In the mean time, as a work-around, I suggest that any CPU with L1 VIVT cache (which thereby requires the make_coherent() code) has its L2 cache disabled. That should at least allow the system to behave as correctly as it does for other ARM VIVT CPUs.

That doesn't work.

Well, with a VIVT L1, we flush the L1 on all MM switches anyway. Flushing it on any thread switch is not going to help that much.

The problem with shared mmaps is that if you have multiple within the same thread, it is required that they are _all_ coherent with respect to each other, whether or not a context switch has occurred.

I believe there are databases (eg, db4) which are non-threaded, makes use of multiple shared mappings and requires that it works, otherwise the database becomes corrupted.

Unfortunately, this means your suggestion is not a solution at all.

I'm afraid to say that the only solution I can see to this problem is to disable the L2 cache outright on these CPUs - the choice seems to be between correct system behaviour and lower performance, or performance but buggy system behaviour in certain cases leading to data corruption.

I know which I'd prefer.

BTW, if no one can come up with a solution for this, we need to consider making the kernel by default disable L2 cache on Feroceon, and display a warning message if L2 is to be enabled.

We really can not have a known data corrupting issue like this exist silently in the system - it's not fair for users nor developers to waste time trying to work out why their data is being corrupted.

There is another possible solution we are looking into:
1) Force all NC writes to go through L2:
If there is a hit in the L2, the write will update the L2 together with the DRAM. If there is a miss, the write will continue to the DRAM only.
2) Force all NC reads to wait till L2 write buffer is drained, to avoid any racing with above...
3) Set L2 in WT mode.

I still can't determine if this solution is valid from HW point of view, but I will really appreciate any comment on this. Any thoughts? Any pitfalls this solution is missing?

Thanks

-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Sunday, December 13, 2009 2:06 PM
To: Ronen Shitrit
Cc: hs at denx.de; saeed bishara; linux-arm-kernel at lists.infradead.org; Nicolas Pitre
Subject: Re: shared memory problem on ARM v5TE using threads

BTW, if no one can come up with a solution for this, we need to consider making the kernel by default disable L2 cache on Feroceon, and display a warning message if L2 is to be enabled.

We really can not have a known data corrupting issue like this exist silently in the system - it's not fair for users nor developers to waste time trying to work out why their data is being corrupted.

Hi, I've been using the patch below for a couple of days now on an a Feroceon based system. It tries to be smart in deciding when to mark a mapping uncacheable and I haven't noticed many of them (but I am *not* running it on a X11 system where maybe the situation could be different). It solves the situation of the test program above. Please be patient if there is something really wrong with it because I don't have a deep understanding of the workings of Linux VM and neither the Feroceon manuals (unfortunately I found that these are not available *after* buying the hardware). Thank you for looking at this, I think this problem should be fixed soon so we can trust our systems.

index d0d17b6..36dc4a5 100644
@@ -34,7 +34,7 @@ static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE;
  * Therefore those configurations which might call adjust_pte (those
  * without CONFIG_CPU_CACHE_VIPT) cannot support split page_table_lock.
  */
-static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
+static int adjust_pte(struct vm_area_struct *vma, unsigned long

address, int update, int only_shared)
{
pgd_t *pgd;
pmd_t *pmd;
@@ -65,7 +65,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
* If this page isn't present, or is already setup to
* fault (ie, is old), we can safely ignore any issues. */
- if (ret && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask) {
+ if (ret && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask && update) {
unsigned long pfn = pte_pfn(entry); flush_cache_page(vma, address, pfn); outer_flush_range((pfn << PAGE_SHIFT),
@@ -74,7 +74,12 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
pte_val(entry) |= shared_pte_mask; set_pte_at(vma->vm_mm, address, pte, entry); flush_tlb_page(vma, address);
+ printk(KERN_INFO "Uncached vma %08x (phy %08x) from pid %d\n",
+ (unsigned int) vma, (unsigned int) (pfn << PAGE_SHIFT),
+ current->pid);
}
+ if (only_shared && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask)
+ ret = 0;
pte_unmap(pte);
return ret;

@@ -100,6 +105,9 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
unsigned long offset;
pgoff_t pgoff;
int aliases = 0;
+#if defined(CONFIG_OUTER_CACHE) && defined(CONFIG_CPU_CACHE_VIVT)
+ int run;
+#endif

pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);

@@ -109,6 +117,32 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
* cache coherency.
*/
flush_dcache_mmap_lock(mapping);
+#if defined(CONFIG_OUTER_CACHE) && defined(CONFIG_CPU_CACHE_VIVT)
+ /*
+ * In the first run we just check if we have to make some
+ * address space uncacheable cause of L1 VIVT. In the second
+ * we check if there is an uncached map in other process. If
+ * one of the previous condition is true we proceed to make
+ * *all* (both in current process VMA and that of others) of
+ * them so. This should solve both cases of multiple shared
+ * memories attached in the same process but not impact the
+ * common case of just one mapping per process.
+ */
+ for(run = 0; run < 3; run++) {
+ vma_prio_tree_foreach(mpnt, &iter, &mapping->i_mmap, pgoff, pgoff) {
+ if ((mpnt->vm_mm != mm || mpnt == vma) && run == 0)
+ continue;
+ if (!(mpnt->vm_flags & VM_MAYSHARE))
+ continue;
+ offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT;
+ aliases += adjust_pte(mpnt, mpnt->vm_start + offset,
+ run == 2, /* update only on the last run */
+ run == 1); /* on the second run catch shared in other procs */
+ }
+ if (aliases == 0 && run == 1)
+ break;
+ }
+#else
vma_prio_tree_foreach(mpnt, &iter, &mapping->i_mmap, pgoff, pgoff) {
/*
* If this VMA is not in our MM, we can ignore it.
@@ -120,11 +154,12 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
if (!(mpnt->vm_flags & VM_MAYSHARE))
continue;
offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT;
- aliases += adjust_pte(mpnt, mpnt->vm_start + offset);
+ aliases += adjust_pte(mpnt, mpnt->vm_start + offset, 1, 0);
}
+#endif
flush_dcache_mmap_unlock(mapping);
if (aliases)
- adjust_pte(vma, addr);
+ adjust_pte(vma, addr, 1, 0);
else
flush_cache_page(vma, addr, pfn);
}

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

Quoting Russell from this thread:
"
Now, at first throught, if we disable the cache for all shared writable mappings in addition to what we're already doing, does this solve the problem? Well, it means that the writes will bypass the caches and hit the RAM directly. The reads from the other shared mappings will read direct from the RAM.

A private mapping using the same page will use the same page, and it will not be marked uncacheable. Accesses to it will draw data into the L2 cache.

PIO kernel mode accesses will also use the cached copy, and that _is_ a problem - it means when we update the backing file on disk, we'll write out the L2 cached data rather than what really should be written out - the updated data from the writable shared mappings. "

I think your patch doesn't cover the PIO mode...

-----Original Message-----
From: [email protected] [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of christian pellegrin
Sent: Monday, December 14, 2009 3:13 PM
To: linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

Hi, I've been using the patch below for a couple of days now on an a Feroceon based system. It tries to be smart in deciding when to mark a mapping uncacheable and I haven't noticed many of them (but I am *not* running it on a X11 system where maybe the situation could be different). It solves the situation of the test program above. Please be patient if there is something really wrong with it because I don't have a deep understanding of the workings of Linux VM and neither the Feroceon manuals (unfortunately I found that these are not available *after* buying the hardware). Thank you for looking at this, I think this problem should be fixed soon so we can trust our systems.

index d0d17b6..36dc4a5 100644
@@ -34,7 +34,7 @@ static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE;
  * Therefore those configurations which might call adjust_pte (those
  * without CONFIG_CPU_CACHE_VIPT) cannot support split page_table_lock.
  */
-static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
+static int adjust_pte(struct vm_area_struct *vma, unsigned long

address, int update, int only_shared)
{
pgd_t *pgd;
pmd_t *pmd;
@@ -65,7 +65,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
* If this page isn't present, or is already setup to
* fault (ie, is old), we can safely ignore any issues. */
- if (ret && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask) {
+ if (ret && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask && update) {
unsigned long pfn = pte_pfn(entry); flush_cache_page(vma, address, pfn); outer_flush_range((pfn << PAGE_SHIFT),
@@ -74,7 +74,12 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
pte_val(entry) |= shared_pte_mask; set_pte_at(vma->vm_mm, address, pte, entry); flush_tlb_page(vma, address);
+ printk(KERN_INFO "Uncached vma %08x (phy %08x) from pid %d\n",
+ (unsigned int) vma, (unsigned int) (pfn << PAGE_SHIFT),
+ current->pid);
}
+ if (only_shared && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask)
+ ret = 0;
pte_unmap(pte);
return ret;

@@ -100,6 +105,9 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
unsigned long offset;
pgoff_t pgoff;
int aliases = 0;
+#if defined(CONFIG_OUTER_CACHE) && defined(CONFIG_CPU_CACHE_VIVT)
+ int run;
+#endif

pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);

@@ -109,6 +117,32 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
* cache coherency.
*/
flush_dcache_mmap_lock(mapping);
+#if defined(CONFIG_OUTER_CACHE) && defined(CONFIG_CPU_CACHE_VIVT)
+ /*
+ * In the first run we just check if we have to make some
+ * address space uncacheable cause of L1 VIVT. In the second
+ * we check if there is an uncached map in other process. If
+ * one of the previous condition is true we proceed to make
+ * *all* (both in current process VMA and that of others) of
+ * them so. This should solve both cases of multiple shared
+ * memories attached in the same process but not impact the
+ * common case of just one mapping per process.
+ */
+ for(run = 0; run < 3; run++) {
+ vma_prio_tree_foreach(mpnt, &iter, &mapping->i_mmap, pgoff, pgoff) {
+ if ((mpnt->vm_mm != mm || mpnt == vma) && run == 0)
+ continue;
+ if (!(mpnt->vm_flags & VM_MAYSHARE))
+ continue;
+ offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT;
+ aliases += adjust_pte(mpnt, mpnt->vm_start + offset,
+ run == 2, /* update only on the last run */
+ run == 1); /* on the second run catch shared in other procs */
+ }
+ if (aliases == 0 && run == 1)
+ break;
+ }
+#else
vma_prio_tree_foreach(mpnt, &iter, &mapping->i_mmap, pgoff, pgoff) {
/*
* If this VMA is not in our MM, we can ignore it.
@@ -120,11 +154,12 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
if (!(mpnt->vm_flags & VM_MAYSHARE))
continue;
offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT;
- aliases += adjust_pte(mpnt, mpnt->vm_start + offset);
+ aliases += adjust_pte(mpnt, mpnt->vm_start + offset, 1, 0);
}
+#endif
flush_dcache_mmap_unlock(mapping);
if (aliases)
- adjust_pte(vma, addr);
+ adjust_pte(vma, addr, 1, 0);
else
flush_cache_page(vma, addr, pfn);
}

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Has someone a suggestion on how to write a test-case for this? I tried various forms of mixing read/write syscalls (or a mmap wit MAP_PRIVATE) and access via two mmap with MAP_SHARED aliasing the same address (I checked that adjust_pte was called by them) but didn't get any problem. I have to admit that I really don't understand how PIO in kernel mode can refer to mappings that are aliased (and so market uncacheable) made by user-space programs. Thank you for your patience.

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

Look at:

http://www.marvell.com/products/embedded_processors/kirkwood/index.jsp

Nicolas

I don't think there is any peripheral that runs in PIO mode on Kirkwood... except the NAND flash.

PIO transfer will store data in the kernel direct mapped memory which is always cached. Normally the cache coherency with user space is handled by flush_dcache_page(), but that deals only with L1 and data can move to L2 where it will be invisible to the uncached user space shared mappings.

I don't know off hand how PIO on a page that is already mapped and shared in user space may be produced though.

Nicolas

the sata can be configured to use pio instead of dma, this can be done by adding libata.force=pio to the command line.

Unfortunately the chapaters about the core and the L2 cache are missing. I gugled and found confirmation from openrd forum that they are available only with a NDA.

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

Thanks for your answers, unfortunately it turned out to be quite easy to write a program [0] that shows exactly what Russell described. :-(

On the Ferocecon [1] sometimes *both* the read via private mapping and via simple lseek/read give the wrong result. On another arm without L2 cache [2] it happens sometimes that private mapping gives the wrong result (I'm not sure it's ok, but it seems reasonable because the mapping is private) but the read via lseek/read is always ok.

[0]:
/*
dd if=/dev/zero of=prova count=1 bs=1k
before running this
*/

#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char *argv[])
{
int fd, fd1;
char *s1, *s2, *p;
volatile char x;

  fd = open("prova", O_RDWR);
  assert(fd);

  fd1 = open("prova", O_RDWR);
  assert(fd1);

p = mmap(NULL, 1024, PROT_WRITE|PROT_READ, MAP_PRIVATE,

	   fd1, 0);

assert((long) p != -1);

  x = *p;

s1 = mmap(NULL, 1024, PROT_WRITE|PROT_READ, MAP_SHARED,

	     fd, 0);
  assert((long) s1 != -1);
  s2 = mmap(NULL, 1024, PROT_WRITE|PROT_READ, MAP_SHARED,
	     fd, 0);

assert((long) s2 != -1);

  fprintf(stderr,"s1 %p s2 %p p %p\n", s1, s2, p);
  fprintf(stderr, "1: s1 %d s2 %d p %d\n", s1[0], s2[0], p[0]);

  s1[0] = 'a';
  s2[0] = 'b';

  msync(s1, 1024, MS_SYNC);
  msync(s2, 1024, MS_SYNC);

  assert(lseek(fd1, SEEK_SET, 0) == 0);
  assert(read(fd1, &x, 1) == 1);
  fprintf(stderr, "2: s1 %d s2 %d p %d x %d\n", s1[0], s2[0], p[0], x);

return 0;
}

[1]:

debian-armel:~/cachep# uname -a
Linux debian-armel 2.6.32-rc7-openrd #36 PREEMPT Tue Dec 15 11:07:00
CET 2009 armv5tel GNU/Linux
debian-armel:~/cachep# cat /proc/cpuinfo
Processor	: Feroceon 88FR131 rev 1 (v5l)
BogoMIPS	: 1196.03
Features	: swp half thumb fastmult edsp
CPU implementer	: 0x56
CPU architecture: 5TE
CPU variant	: 0x2
CPU part	: 0x131
CPU revision	: 1

Hardware : Marvell OpenRD Base Board
Revision : 0000
Serial : 0000000000000000

[2]:
root at familiar:/var/tmp# uname -a
Linux familiar 2.6.23-dixe #58 PREEMPT Fri Dec 4 11:33:26 CET 2009 armv4tl unknown
root at familiar:/var/tmp# cat /proc/cpuinfo
Processor : ARM920T rev 0 (v4l)
BogoMIPS : 88.84
Features : swp half thumb
CPU implementer : 0x41
CPU architecture: 4T
CPU variant : 0x1
CPU part : 0x920
CPU revision : 0
Cache type : write-back
Cache clean : cp15 c7 ops
Cache lockdown : format A
Cache format : Harvard
I size : 16384
I assoc : 64
I line length : 32
I sets : 8
D size : 16384
D assoc : 64
D line length : 32
D sets : 8

Hardware : Atmel AT91RM9200-DK
Revision : 0000
Serial : 0000000000000000

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

The patch here does what was described by Russell and seems to solve the read/write and the MAP_PRIVATE in test cases like the one I posted. As noted it may not be enough. If you don't think it's complete crap I can try to check for other cases too by installing a more complete system (X11, db4) and using ftrace to watch for which flush_cache_* we actually use. Any suggestion is appreciated, it would be a pity to have to turn off L2 cache.

index d0d17b6..53ad00f 100644
@@ -34,7 +34,7 @@ static unsigned long shared_pte_mask = L_PTE_MT_BUFFERABLE;
  * Therefore those configurations which might call adjust_pte (those
  * without CONFIG_CPU_CACHE_VIPT) cannot support split page_table_lock.
  */
-static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
+static int adjust_pte(struct vm_area_struct *vma, unsigned long

address, int update, int only_shared)
{
pgd_t *pgd;
pmd_t *pmd;
@@ -65,7 +65,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
* If this page isn't present, or is already setup to
* fault (ie, is old), we can safely ignore any issues. */
- if (ret && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask) {
+ if (ret && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask && update) {
unsigned long pfn = pte_pfn(entry); flush_cache_page(vma, address, pfn); outer_flush_range((pfn << PAGE_SHIFT),
@@ -74,7 +74,13 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address)
pte_val(entry) |= shared_pte_mask; set_pte_at(vma->vm_mm, address, pte, entry); flush_tlb_page(vma, address);
+ printk(KERN_INFO "Uncached vma %08x (addr %08lx flahs %08lx phy
%08x) from pid %d\n",

+		       (unsigned int) vma, vma->vm_start, vma->vm_flags,
+		       (unsigned int) (pfn << PAGE_SHIFT),
+		       current->pid);

}
+ if (only_shared && (pte_val(entry) & L_PTE_MT_MASK) != shared_pte_mask)
+ ret = 0;
pte_unmap(pte);
return ret;

@@ -100,6 +106,9 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
unsigned long offset;
pgoff_t pgoff;
int aliases = 0;
+#if defined(CONFIG_OUTER_CACHE) && defined(CONFIG_CPU_CACHE_VIVT)
+ int run;
+#endif

pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);

@@ -109,6 +118,32 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
* cache coherency.
*/
flush_dcache_mmap_lock(mapping);
+#if defined(CONFIG_OUTER_CACHE) && defined(CONFIG_CPU_CACHE_VIVT)
+ /*
+ * In the first run we just check if we have to make some
+ * address space uncacheable cause of L1 VIVT. In the second
+ * we check if there is an uncached map in other process. If
+ * one of the previous condition is true we proceed to make
+ * *all* (both in current process VMA and that of others) of
+ * them so. This should solve both cases of multiple shared
+ * memories attached in the same process but not impact the
+ * common case of just one mapping per process.
+ */
+ for(run = 0; run < 3; run++) {
+ vma_prio_tree_foreach(mpnt, &iter, &mapping->i_mmap, pgoff, pgoff) {
+ if ((mpnt->vm_mm != mm || mpnt == vma) && run == 0)
+ continue;
+ if (!(mpnt->vm_flags & VM_MAYSHARE) && run != 2) /* update all mappings */
+ continue;
+ offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT;
+ aliases += adjust_pte(mpnt, mpnt->vm_start + offset,
+ run == 2, /* update only on the last run */
+ run == 1); /* on the second run catch shared in other procs */
+ }
+ if (aliases == 0 && run == 1)
+ break;
+ }
+#else
vma_prio_tree_foreach(mpnt, &iter, &mapping->i_mmap, pgoff, pgoff) {
/*
* If this VMA is not in our MM, we can ignore it.
@@ -120,11 +155,12 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, unsigne
if (!(mpnt->vm_flags & VM_MAYSHARE))
continue;
offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT;
- aliases += adjust_pte(mpnt, mpnt->vm_start + offset);
+ aliases += adjust_pte(mpnt, mpnt->vm_start + offset, 1, 0);
}
+#endif
flush_dcache_mmap_unlock(mapping);
if (aliases)
- adjust_pte(vma, addr);
+ adjust_pte(vma, addr, 1, 0);
else
flush_cache_page(vma, addr, pfn);
}

index 7f294f3..b7c6986 100644
@@ -209,6 +209,13 @@ void flush_dcache_page(struct page *page)
 			__flush_dcache_aliases(mapping, page);
 		else if (mapping)
 			__flush_icache_all();
+#ifdef CONFIG_OUTER_CACHE
+		{
+		  unsigned long pfn = page_to_pfn(page);
+		  outer_flush_range((pfn << PAGE_SHIFT),
+				    (pfn << PAGE_SHIFT) + PAGE_SIZE);
+		}
+#endif
 	}
 }
 EXPORT_SYMBOL(flush_dcache_page);

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

Try this:

- create a 4K file
- create three programs:
1. repeatedly uses read() and write() to increment the first word of this file
2. uses one shared mmap of this file to increment the 256th word (offset

     1024)

3. uses two shared mmaps of this file, reading the 128th word from one shared mmap, reads the 128th word from the other, increments the first value and writes it to the 128th word in the second map, printing the two values read. It also reads the first and 256th word and prints the result

This should be an adequate test of most of the important scenarios: the third program should show an increasing number for both locations.

If it doesn't, then the value is being cached somewhere. I suspect what you'll find is that on Feroceon, the only values you will see increment is the 128th value. The remainder will be static.

That means multiple shared mmaps on Feroceon are incoherent with:
1. file updates via read/write.
2. other shared mappings in other processes.

And because of (1) I expect that a read() followed by another read() of the same file offset as being updated by (3) will not show updates - so it should be both lost write()s and stale data on read()s (which will mean PIO write-outs - eg swap out, file update - will be affected.)

As a note, you want s1, s2 and p to be volatile, otherwise the compiler can cache the result.

These msyncs should not be required for data via one shared mmap to be visible via another mmap.

This might be enough (but needs thorough testing). However, adding the requirement for L2 cache flushing in flush_dcache_page() because you have an L2 cache is unfair on those L2's which don't suffer from this problem.

I think we need to have a little more information about the behaviour of the L2 cache so that we can decide how much flushing is required, and where.

-----Original Message-----
From: [email protected] [mailto:linux-arm-kernel-bounces at lists.infradead.org] On Behalf Of Russell King - ARM Linux
Sent: Tuesday, December 15, 2009 7:19 PM
To: christian pellegrin
Cc: linux-arm-kernel at lists.infradead.org
Subject: Re: shared memory problem on ARM v5TE using threads

This might be enough (but needs thorough testing). However, adding the requirement for L2 cache flushing in flush_dcache_page() because you have an L2 cache is unfair on those L2's which don't suffer from this problem.

I think we need to have a little more information about the behaviour of the L2 cache so that we can decide how much flushing is required, and where.
[Ronen Shitrit] What information do you need?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

I played a bit with the program implementing this idea. With an unpatched 2.6.32-rc7 from orion git tree all the numbers are actually incrementing. Of course it's impossible to catch an intermittent problem with this test because maybe I'm just seeing one update out of n. I had a similar feeling with my test program about read/write
consistency: sometimes the uncached mapping got the right result and sometimes not (I did the test on an unload system via console, so I really doubt I was implicitly flushing the cache). My guess is that sometimes the data written through L2 hit the SDRAM. I double checked, I have L2 working in WB and *not* WT. It's much easier to show the consistency problem if we print the locations in process 2, now the 128th location is *never* incrementing until I don't trash the cache running a big program. Somehow read/write is still incrementing. With the patch I proposed I don't see any problems.

I'm trying some more elaborate tests where just one case of inconsistency will stop the counting.

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

Here is the program that implements Russell's ideas (at least I think so) but is easier to use. By giving the parameter 1 or -1 you can test different kind of consistency issues (missing flush in r/w or inconsistent mapping's cacheness). It is also quite fun to watch at with the buggy kernel on an idle system: it looks like that every couple of seconds the 256kb L2 cache get flushed anyway (so even on the kernel without the patch every now and then you get some progress). I had it running for tens of minutes on a patched kernel without stops.

*** rt.sh:

#!/bin/sh

RT=./rta

dd if=/dev/zero of=rt count=1 bs=4k

$RT 0 $1 &
$RT 1 $1 &
$RT 2 $1 &

*** rt.c:
/*
arm-none-linux-gnueabi-gcc -Wall -o rta rt.c
*/
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char *argv[])
{
int fd, dir;
volatile unsigned int *s1, *s2;
volatile int x;

  if (argc != 3) {
    printf("Usage: %s [0/1/2] [dir]\n", argv[0]);
    exit(1);
  }
  dir = atoi(argv[2]);

  fd = open("rt",O_RDWR);
    assert(fd != -1);

switch(atoi(argv[1])) {
case 0:

    x = 3000000;
    assert(lseek(fd, sizeof(unsigned int) * 128, SEEK_SET) ==
(sizeof(unsigned int) * 128));
    assert(write(fd, &x, sizeof(x)) == sizeof(x));
    while(1) {
      do {
	assert(lseek(fd, sizeof(unsigned int) * 128, SEEK_SET) ==
(sizeof(unsigned int) * 128));
	assert(read(fd, &x, sizeof(x)) == sizeof(x));
      } while(x % 3 != 0);
      printf("%d rw\n",x);
      x += dir;
      assert(lseek(fd, sizeof(unsigned int) * 128, SEEK_SET) ==
(sizeof(unsigned int) * 128));
      assert(write(fd, &x, sizeof(x)) == sizeof(x));
      if (dir > 0)
	assert(lseek(fd, 0, SEEK_SET) == 0);
      else
	assert(lseek(fd, sizeof(unsigned int) * 256, SEEK_SET) ==
sizeof(unsigned int) * 256);
      assert(write(fd, &x, sizeof(x)) == sizeof(x));
    }
    break;

case 1:

    s1 = mmap(NULL, 4096, PROT_WRITE|PROT_READ, MAP_SHARED,
	      fd, 0);
    assert((long) s1 != -1);
    while(1) {
      while (s1[0] % 3 != 1);
      printf("%d 1map\n",s1[0]);
      s1[0] += dir;
      if (dir > 0)
	s1[256] = s1[0];
      else
	s1[128] = s1[0];
    }
    break;

default:

    s1 = mmap(NULL, 4096, PROT_WRITE|PROT_READ, MAP_SHARED,
	      fd, 0);
    assert((long) s1 != -1);
    s2 = mmap(NULL, 4096, PROT_WRITE|PROT_READ, MAP_SHARED,
	      fd, 0);
    assert((long) s2 != -1);
    while (1) {
      while (s1[256] % 3 != 2);
      printf("%d 2map\n",s1[256]);
      s2[256] += dir;
      if (dir > 0)
	s2[128] =  s1[256];
      else
	s2[0] = s1[256];
    }

}

return 0;
}

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

-----Original Message-----
From: christian pellegrin [mailto:chripell at gmail.com]
Sent: Wednesday, December 16, 2009 7:39 PM
To: Russell King - ARM Linux
Cc: saeed bishara; Ronen Shitrit; linux-arm-kernel at lists.infradead.org; Nicolas Pitre
Subject: Re: shared memory problem on ARM v5TE using threads

Here is the program that implements Russell's ideas (at least I think
so) but is easier to use. By giving the parameter 1 or -1 you can test
different kind of consistency issues (missing flush in r/w or
inconsistent mapping's cacheness). It is also quite fun to watch at
with the buggy kernel on an idle system: it looks like that every
couple of seconds the 256kb L2 cache get flushed anyway
[Ronen Shitrit] The L2 dirty line eviction is always a result of L1 line fill request, or a specific CP15 flush command.

(so even on
the kernel without the patch every now and then you get some progress). I had it running for tens of minutes on a patched kernel without stops.
[Ronen Shitrit] Nice :)

Hello Christian,

FYI:
I tried your patch:
http://lists.infradead.org/pipermail/linux-arm-kernel/2009-December/006192.html

with the testprogramm, I posted here:
http://lists.infradead.org/pipermail/linux-arm-kernel/2009-December/005489.html

and on my board [1] it works fine [2] with your patch.

Another question comes in my mind:

As we have problems with shared memory, if using more than one mapping, do we have this problems also when using other "standard" Linux IPC, like for example semaphores, queues, ... ?

bye,
Heiko

[1]
-bash-3.2# cat /proc/cpuinfo
Processor : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS : 799.53
Features : swp half thumb fastmult edsp
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant : 0x2
CPU part : 0x131
CPU revision : 1

Hardware : Keymile SUEN3 Board
Revision : 0000
Serial : 0000000000000000
-bash-3.2#

[2] log from testprogramm with patch from christian:

-bash-3.2# cat shtest2.sh
#!/bin/sh
echo "Run shmtest2"
./shmtest2 write 1 &
./shmtest2 read 4 &
-bash-3.2# ./shtest2.sh
Run shmtest2
Write to in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Write Thread 0, addr:40961000
359: write new mesg: 0
Read from in_msg
Creating thread 0
Shared memory Id:0
attach shared mem:40961000
Start Read Thread addr:40961000
Creating thread 1
Shared memory Id:0
attach shared mem:41170000
Start Read Thread addr:41170000
Creating thread 2
Shared memory Id:0
attach shared mem:4197f000
Start Read Thread addr:4197f000
Creating thread 3
Shared memory Id:0
attach shared mem:4218e000
Start Read Thread addr:4218e000
-bash-3.2# 359: write new mesg: 0
361 Read from entry in_msg=1000, out_msg=1000, addr=40961000 361 Read from entry in_msg=2000, out_msg=1000, addr=41170000 361 Read from entry in_msg=3000, out_msg=1000, addr=4197f000 361 Read from entry in_msg=4000, out_msg=1000, addr=4218e000 359: write new mesg: 1000
361 Read from entry in_msg=5000, out_msg=2000, addr=40961000 361 Read from entry in_msg=6000, out_msg=2000, addr=41170000 361 Read from entry in_msg=7000, out_msg=2000, addr=4197f000 361 Read from entry in_msg=8000, out_msg=2000, addr=4218e000 359: write new mesg: 2000
361 Read from entry in_msg=9000, out_msg=3000, addr=40961000 361 Read from entry in_msg=10000, out_msg=3000, addr=41170000 361 Read from entry in_msg=11000, out_msg=3000, addr=4197f000 361 Read from entry in_msg=12000, out_msg=3000, addr=4218e000 359: write new mesg: 3000
[...]
349: write new mesg: 57000
350 Read from entry in_msg=230000, out_msg=58000, addr=4197f000 350 Read from entry in_msg=231000, out_msg=58000, addr=41170000 350 Read from entry in_msg=232000, out_msg=58000, addr=4218e000 350 Read from entry in_msg=233000, out_msg=58000, addr=40961000 349: write new mesg: 58000
350 Read from entry in_msg=234000, out_msg=59000, addr=4197f000 350 Read from entry in_msg=235000, out_msg=59000, addr=41170000 350 Read from entry in_msg=236000, out_msg=59000, addr=4218e000 350 Read from entry in_msg=237000, out_msg=59000, addr=40961000 350 Read from entry in_msg=237000, out_msg=59000, addr=4197f000 350 Read from entry in_msg=237000, out_msg=59000, addr=4218e000 350 Read from entry in_msg=237000, out_msg=59000, addr=41170000 All write threads finished, exit
All read threads finished, exit

-bash-3.2#

out_msg increments as suggested!

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Hi

AFAIK no. The problem you discovered is caused by the access of the same physical memory area via a cached and an uncached mapping without proper L2 cache flushing. The only way that I know about how to create an uncached one in user space is by mapping the same physical memory to different virtual address (so triggering the L1 VIVT cache aliasing avoidance mechanism).

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

But that's pretty unusual situation, right?

So what about...

a) flush L2 on context switch

b) disable L2 when thread has maps one physical address twice

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

No. This is too big a performance killer.

Nicolas

It may be uncommon, but it's something that must work - not only is it a quality of implementation issue, but its also a data corruption issue. Get it wrong and you silently corrupt files on your filesystems.

As I've said earlier in the thread, we know that db4 does this. Not fixing this means that we really don't care about any program which uses db4, or data contained within a db4 database.

If you go to that extent, the system will probably perform better with the L2 cache permanently disabled.

Due to the way the L2 cache works, you have to disable L2 and flush it when switching to the thread, and re-enable L2 when you switch away. Merely flushing it when switching away won't work.

In order to disable or enable the L2 cache, the L1 cache must be flushed and disabled - and that would have to be done with all IRQs (including FIQs) disabled to ensure that its done atomically.

I don't think this is a sane option.

Could you please repost your patch, adding CONFIG_CPU_CACHE_VIVT to the conditional code flushing the L2 cache in flush_dcache_page()?

I think this would be a good thing to have merged. If I'm not mistaken, this patch appears to fix all identified coherency cases so far.

Nicolas

No, that's not sufficient as I pointed out earlier. We really do not want to be doing the L2 cache stuff on things which aren't Feroceon until it's shown that others are affected.

The other annoying thing is that it's touching the same area of code which is currently broken with highpte - and I have changes in the pipeline for fixing that.

Aren't we saying the same thing?

As I said earlier already, this is concerning only L2 equipped VIVT processor models. There are very few of those in existence, namely some Feroceons and XSC3s.

On both Feroceon and XSC3, there used to be some issues with DB4 when creating two shared writable mappings in the same process. This was fixed in commit 08e445bd. Both processors used to fail and both were happy with the change. And according to the tests I've made and the conclusion I've consigned in that commit, none of those processors could support L1 uncacheable L2 cacheable.

This didn't cover all cases but we apparently have a solution for those now. So by surrounding the code with #if defined(CONFIG_CPU_CACHE_VIVT) && defined(CONFIG_OUTER_CACHE) we actually affect only the same processor models i.e. Feroceon (proven to need it) and XSC3 (unproven).

So, even if XSC3 is included while it might potentially not need it, I'd cover it anyway given past track records with similar issues to be on the safe side, at least until someone with the hardware is able to confirm the fix is not needed there (and I doubt it will ever happen). We're talking about some seriously rare occurrences anyway, so this is not like if this would impose a significant useless performance killing overhead on an XSC3.

That's a valid concern. In which case I'd suggest that Christian waits until your changes are visible somewhere for him to rebase his patch on top and retest it before reposting. The current patch (with my suggested modification) might be worth considering for linux-stable nevertheless.

Nicolas

Not quite. You're saying "all VIVT + outer cache will have this fix". I'm saying "those which have the problem should have the fix".

I believe treating them as being the same is not the right approach. At the moment, they're the same group of processors, but that doesn't mean it stays that way, and should it change, finding such hidden conditions isn't going to be easy.

So I'd like to see something other than CONFIG_CPU_CACHE_VIVT+ CONFIG_OUTER_CACHE in the code - instead, make a new config symbol which is enabled when both are currently selected, and put a useful help text with it so people understand what its about.

It'll be a while before my patches are ready - they touch every other architecture in the kernel because we need access to the kmapped page table to modify the PTE...

http://ftp.arm.linux.org.uk/git/gitweb.cgi?p=linux-2.6-arm.git;a=shortlog;h=refs/heads/umc

GIT:
http://ftp.arm.linux.org.uk/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git umc

(or something like that.)

I'm going to send a proper patch (at least for checkpatch.pl standards) in response to this mail in case someone cares to add it to theirs trees. I'm cloning the mentioned git repository and will try to do some testing on umc branch.The modifications seem important so one more tester won't hurt I guess.

Christian Pellegrin, see http://www.evolware.org/chri/
"Real Programmers don't play tennis, or any other sport which requires
you to change clothes. Mountain climbing is OK, and Real Programmers
wear their climbing boots to work in case a mountain should suddenly
spring up in the middle of the computer room."

This patch fixes coherency problems on ARM systems that have a VIVT L1 cache and a PIPT L2 one. The modifications to fault-armv.c assure that we don't mix cached and uncached mappings of the same physical memory area after we made a mapping uncached to prevent problems with L1 aliasing. The modifications to flush.c assure coherency with read/write because kernel PIO memory access is always cached.

---
 arch/arm/mm/Kconfig      |    8 ++++++
 arch/arm/mm/fault-armv.c |   55 ++++++++++++++++++++++++++++++++++++++++++---
 arch/arm/mm/flush.c      |    7 ++++++
 3 files changed, 66 insertions(+), 4 deletions(-)

index 9264d81..2eacf29 100644
@@ -776,3 +776,11 @@ config ARM_L1_CACHE_SHIFT
 	int
 	default 6 if ARCH_OMAP3
 	default 5
+
+config ARM_ARMV5_L2_CACHE_COHERENCY_FIX
+       bool "Enable L2 coherency problem fix on ARMv5"
+       depends on OUTER_CACHE && CPU_CACHE_VIVT
+       default y
+       help
+         Fixes coherency problems on ARM systems that have
+	 a non-disableable VIVT L1 cache and a PIPT L2 one.

index 7f294f3..779a7f9 100644
@@ -209,6 +209,13 @@ void flush_dcache_page(struct page *page)
 			__flush_dcache_aliases(mapping, page);
 		else if (mapping)
 			__flush_icache_all();
+#ifdef CONFIG_ARM_ARMV5_L2_CACHE_COHERENCY_FIX
+		{
+		  unsigned long pfn = page_to_pfn(page);
+		  outer_flush_range((pfn << PAGE_SHIFT),
+				    (pfn << PAGE_SHIFT) + PAGE_SIZE);
+		}
+#endif
 	}
 }
 EXPORT_SYMBOL(flush_dcache_page);

1.5.6.5

Eek?

Alternative is "disable L2". I guess "use L2, but flush it occasionally" is still better than "disable L2".

Of course, it should be measured. It looks like hardware already is braindead, so maybe logic does not work there.

(And it may depend if you want max throughput or min latency).
Pavel

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

I'm not trying to argue it does not need to be fixed; I'm trying to say that performance penalty in unusual case may be ok.

Are you sure? Context switches are not that common (3 times a second in number-crunching cases?) and L2 cache should help performance quite a lot.

Well, it sounds slow and it will not be easy to implement. It really depends how much L2 cache helps on those systems. (I'd guess PC with L2 disabled would run at half a speed. If performance penalty for L2 disabled is similar on those systems, extra complexity may be worth it.)

So I guess someone with affected system needs to do some benchmarking.
Pavel

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Do you own such hardware?

If no, why do you care?

Do you really have something constructive to add to this thread besides "hardware is braindead" or other worthless speculations about some issue for which you apparently don't know whit about?

For your information, a patch has been proposed already which does not imply disabling L2 nor flushing it on every context switches. Would you care to familiarize yourself with it before commenting further please?

Nicolas

No.

Curiosity? You spreading statements that seem strange? Me wanting to understand VIVT caches?

Really? The patch seemed to solve cached vs. uncached accesses, but did not seem to solve two (cached) mappings of same file.
Pavel

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html