Skip to contentSkip to conversation

mmap() of /dev/kmem issue

DISCUSSION Mario Smarduch 6 participants 20 messages May 4, 2004 – May 5, 2004

lore
Share & tools
lore
#0 lore raw

I have problem of viewing memory contents via mmap() of /dev/kmem. After mmap()'ing /dev/kmem for reading it appears that data accessed in the kernel lives in different cache lines from data accessed in the user space. The lseek()/read() from kernel reflects latest updates, but same reads from user space mmap fetches old data, and vice versa when updating from user. When I flush the caches (cat of a big file to another file), then user mapped value reflects the latest changes but then again remains frozen. I'm having trouble understanding how can this happen since the caches are physically indexed. Any ideas what the problem could be?

- Mario.

#1 lore raw
Quoted 4 lines · Mario Smarduch

I have problem of viewing memory contents via mmap() of /dev/kmem.
...
Any ideas what the problem could be?

Can you post a test case? What kernel are you running?

#2 lore raw
In reply to Mario Smarduch
Quoted 1 line · Mario Smarduch

I have problem of viewing memory contents via mmap() of /dev/kmem.

1) which regions/addresses are you accessing?
2) what values do you see/should see?
3) test case?

grant

Quoted 18 lines

After mmap()'ing /dev/kmem for reading
it appears that data accessed in the kernel lives in different cache lines from data accessed in the user space. The lseek()/read() from kernel reflects latest updates, but same reads from user space mmap fetches old data, and vice versa when updating from user. When I flush the caches (cat of a big file to another file), then user mapped value reflects the latest changes but then again remains frozen. I'm having trouble understanding how can this happen since the caches are physically indexed. Any ideas what the problem could be?

- Mario.

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Mario Smarduch
Quoted 7 lines · Bjorn Helgaas
Quoted 4 lines · Mario Smarduch

I have problem of viewing memory contents via mmap() of /dev/kmem.
...
Any ideas what the problem could be?

Can you post a test case? What kernel are you running?

To run the test first verify the PAGE_MASK matches. Secondly compile tstmmapkmem.c, and run runtval.sh it expects an upto date /boot/System.map file. It seems like the region doesn't appear to be mapped cachable.

runtval.sh
-------

#!/bin/ksh

jiffies=0xgrep "D jiffies" /boot/System.map | awk '{ print $1 }' ./tstmmapkmem $jiffies

tstmmapkmem.c - cc -o tstmmapkmem tstmmapkmem.c
----------
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <libgen.h>
#include <sys/time.h>
#include <stdlib.h>
#include <sys/mman.h>

#define IDENBASE 0xe000000000000000
#define DEVICE "/dev/kmem"
#define PAGE_MASK (0x4000 - 1) /* page is 16k */

#define wmb() __asm__ __volatile__ ("": : :"memory")

extern int errno;
main(int argc, char *argv[])
{
int fd;
ulong mapaddr;
volatile long *jifptr, jiffies;

   jiffies = strtoul(argv[1], 0, 16);
   printf("jiffies = 0x%lx+AFw-n", jiffies);
   printf("pid = %d+AFw-n", getpid());

   if((fd = open(DEVICE, O_RDWR)) < 0)
        printf("open failed - errno = %d+AFw-n", errno), exit(errno);

jiffies -= IDENBASE;
mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
PROT_WRITE|PROT_R
EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK);

   if(mapaddr = -1UL) {
      printf("mmap failed errno = %d+AFw-n", errno);
      exit(-1);

}
printf("mapaddr = 0x%lx+AFw-n", mapaddr);
jiffies = mapaddr +- (jiffies & PAGE_MASK);
jifptr = (long *) jiffies;
printf("jifptr = 0x%lx+AFw-n", jifptr);
pragain:
printf("jiffies: %ld+AFw-n", *jifptr);

printf("Type return to read again (and flush cashe in meantime):+AFw-n");

getchar();
goto pragain;
}

- Mario.

#4 lore raw
In reply to Mario Smarduch
Quoted 2 lines · Mario Smarduch

jiffies=0xgrep "D jiffies" /boot/System.map | awk '{ print $1 }' ./tstmmapkmem $jiffies

Maybe the process is being prempted?
Ie maybe the problem is the contents are "stale" by the time they get displayed on the console/terminal. jiffies gets updated 1000/second so on occasion I would expect this to happen.
Maybe pick something that doesn't get updated so frequently?

grant

Quoted 46 lines

#define DEVICE "/dev/kmem"
#define PAGE_MASK (0x4000 - 1) /* page is 16k */

#define wmb() __asm__ __volatile__ ("": : :"memory")

extern int errno;
main(int argc, char *argv[])

{
   int fd;
   ulong mapaddr;
   volatile long *jifptr, jiffies;

   jiffies = strtoul(argv[1], 0, 16);
   printf("jiffies = 0x%lx\n", jiffies);
   printf("pid = %d\n", getpid());

   if((fd = open(DEVICE, O_RDWR)) < 0)
        printf("open failed - errno = %d\n", errno), exit(errno);

   jiffies -= IDENBASE;
   mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
PROT_WRITE|PROT_R
EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & ~PAGE_MASK);
   if(mapaddr = -1UL) {
      printf("mmap failed errno = %d\n", errno);
      exit(-1);
   }
   printf("mapaddr = 0x%lx\n", mapaddr);
   jiffies = mapaddr + (jiffies & PAGE_MASK);
   jifptr = (long *) jiffies;
   printf("jifptr = 0x%lx\n", jifptr);
pragain:
   printf("jiffies: %ld\n",  *jifptr);

   printf("Type return to read again (and flush cashe in meantime):\n");

   getchar();
   goto pragain;
}

- Mario.

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
In reply to Mario Smarduch
Quoted 10 lines · Grant Grundler
Quoted 2 lines · Mario Smarduch

jiffies=0xgrep "D jiffies" /boot/System.map | awk '{ print $1 }' ./tstmmapkmem $jiffies

Maybe the process is being prempted?
Ie maybe the problem is the contents are "stale" by the time they get displayed on the console/terminal. jiffies gets updated 1000/second so on occasion I would expect this to happen.
Maybe pick something that doesn't get updated so frequently?

I can't see how that could have a bearing if both kernel/user are hitting the cache you should see the same values everytime.

- Mario

Quoted 50 lines

grant

Quoted 46 lines

#define DEVICE "/dev/kmem"
#define PAGE_MASK (0x4000 - 1) /* page is 16k */

#define wmb() __asm__ __volatile__ ("": : :"memory")

extern int errno;
main(int argc, char *argv[])

{
   int fd;
   ulong mapaddr;
   volatile long *jifptr, jiffies;

   jiffies = strtoul(argv[1], 0, 16);
   printf("jiffies = 0x%lx+AFw-n", jiffies);
   printf("pid = %d+AFw-n", getpid());

   if((fd = open(DEVICE, O_RDWR)) < 0)
        printf("open failed - errno = %d+AFw-n", errno), exit(errno);

   jiffies -= IDENBASE;
   mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
PROT_WRITE|PROT_R
EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK);
   if(mapaddr = -1UL) {
      printf("mmap failed errno = %d+AFw-n", errno);
      exit(-1);
   }
   printf("mapaddr = 0x%lx+AFw-n", mapaddr);
   jiffies = mapaddr  (jiffies & PAGE_MASK);
   jifptr = (long *) jiffies;
   printf("jifptr = 0x%lx+AFw-n", jifptr);
pragain:
   printf("jiffies: %ld+AFw-n",  *jifptr);

   printf("Type return to read again (and flush cashe in meantime):+AFw-n");

   getchar();
   goto pragain;
}

- Mario.

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
In reply to Mario Smarduch

It appears that from mmap_mem() noncached_address() succeeds on PageReserved for ranges in: KERNEL_START- KERNEL_END, I tried a temporary workaround by making an exception to addresses in that range and things work (that is for this program only) but what sort of problems would such a change introduce?

if(PageReserved(page) && !((va >= KERNEL_START) && (va < KERNEL_END)) )

            return (1);

I would think allot of people would like to mmap() /dev/kmem to take a quick look at kernel values, atleast that's what I've seen on other Unix variants.

- mario.

Quoted 59 lines · Grant Grundler
Quoted 2 lines · Mario Smarduch

jiffies=0xgrep "D jiffies" /boot/System.map | awk '{ print $1 }' ./tstmmapkmem $jiffies

Maybe the process is being prempted?
Ie maybe the problem is the contents are "stale" by the time they get displayed on the console/terminal. jiffies gets updated 1000/second so on occasion I would expect this to happen.
Maybe pick something that doesn't get updated so frequently?

grant

Quoted 46 lines

#define DEVICE "/dev/kmem"
#define PAGE_MASK (0x4000 - 1) /* page is 16k */

#define wmb() __asm__ __volatile__ ("": : :"memory")

extern int errno;
main(int argc, char *argv[])

{
   int fd;
   ulong mapaddr;
   volatile long *jifptr, jiffies;

   jiffies = strtoul(argv[1], 0, 16);
   printf("jiffies = 0x%lx+AFw-n", jiffies);
   printf("pid = %d+AFw-n", getpid());

   if((fd = open(DEVICE, O_RDWR)) < 0)
        printf("open failed - errno = %d+AFw-n", errno), exit(errno);

   jiffies -= IDENBASE;
   mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
PROT_WRITE|PROT_R
EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK);
   if(mapaddr = -1UL) {
      printf("mmap failed errno = %d+AFw-n", errno);
      exit(-1);
   }
   printf("mapaddr = 0x%lx+AFw-n", mapaddr);
   jiffies = mapaddr  (jiffies & PAGE_MASK);
   jifptr = (long *) jiffies;
   printf("jifptr = 0x%lx+AFw-n", jifptr);
pragain:
   printf("jiffies: %ld+AFw-n",  *jifptr);

   printf("Type return to read again (and flush cashe in meantime):+AFw-n");

   getchar();
   goto pragain;
}

- Mario.

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
In reply to Mario Smarduch
Quoted 3 lines · Mario Smarduch

I would think allot of people would like to mmap() /dev/kmem to take a quick look at kernel values, atleast that's what I've seen on other Unix variants.

yes - and that's a religious issue. I know people would like like to see /dev/kmem completely removed.

Many wise kernel engineers have advised me to add /proc or similar "architected" interfaces. That way a program will merely get an error instead of crashing the box when the kernel changes it's data structures or how said structures get used.

grant

#8 lore raw
In reply to Mario Smarduch
Quoted 8 lines · Grant Grundler
Quoted 3 lines · Mario Smarduch

I would think allot of people would like to mmap() /dev/kmem to take a quick look at kernel values, atleast that's what I've seen on other Unix variants.

yes - and that's a religious issue. I know people would like like to see /dev/kmem completely removed.

Quoted 7 lines

Many wise kernel engineers have advised me to add /proc or similar "architected" interfaces. That way a program will merely get an error instead of crashing the box when the kernel changes it's data structures or how said structures get used.

grant

That's understandable. But there are cases where kernel values just need to be read timely. As things are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed, but I'm not sure how mapping it as uncachable helps? To me this behaviour seems buggy.

- Mario.

#9 lore raw
In reply to Mario Smarduch
Quoted 8 lines

That's understandable. But there are cases where kernel values just need to be read timely. As things are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed, but I'm not sure how mapping it as uncachable helps? To me this behaviour seems buggy.

Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).

-Tony

#10 lore raw
In reply to Mario Smarduch
Quoted 2 lines · Mario Smarduch

But there are cases where
kernel values just need to be read timely.

Watching kernel data from userspace is an impossible task. Unless one has a "real time" scheduler and *halts* all kernel activity, the kernel values visible to user space will lag what the kernel does. Something like KGDB can do that. It's not something I expect to work correctly.
Ergo define an interface (eg gettimeofday() or /proc/sys/kernel/*) to get "sane" values and move on.

Quoted 5 lines

As things
are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed,

Maybe. I don't understand the VM well enough.

Quoted 3 lines

but I'm
not sure how mapping it as uncachable helps? To me this behaviour seems buggy.

I'm still inclined to think the approach is wrong. Maybe you have a better test case that doesn't involve user interaction and shows why the data has to come from /dev/kmem?

grant

#11 lore raw
In reply to Mario Smarduch
Quoted 3 lines · Tony

Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).

... which is why we don't do that.

See drivers/char/mem.c::mmap_mem(). We call uncached_access() which does:

#elif defined(CONFIG_IA64)
/*
* On ia64, we ignore O_SYNC because we cannot tolerate memory attribute
aliases.

         */
        return !(efi_mem_attributes(addr) & EFI_MEMORY_WB);
Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#12 lore raw
In reply to Mario Smarduch
Quoted 9 lines · Tony
Quoted 8 lines

That's understandable. But there are cases where kernel values just need to be read timely. As things are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed, but I'm not sure how mapping it as uncachable helps? To me this behaviour seems buggy.

Quoted 5 lines

Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).

-Tony

The background on the issue is that I'm trying to mmap() some data in KERNEL_START-KERNEL_END range
to read some data, without having to go through a system call. This results in uncacheble mapping in
user space, which can results in what you stated above. Its something I did routinly on other Unix variants. And I think that restriction is not there on Linux IA32 boxes. The reason its mapped uncached its because that range is marked reserved. I'm guessing that reserved memory map ranges are uncachable (just guessing here), but in this case the range in question is accessed cacheble by the kernel.

- Mario.

#13 lore raw
In reply to Mario Smarduch
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines · Tony

Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).

... which is why we don't do that.

Quoted 8 lines

See drivers/char/mem.c::mmap_mem(). We call uncached_access() which does:

#elif defined(CONFIG_IA64)
/*
* On ia64, we ignore O_SYNC because we cannot tolerate memory attribute
aliases.

         */
        return !(efi_mem_attributes(addr) & EFI_MEMORY_WB);

That's great but what release is that in?

- mario.

Quoted 13 lines

--
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain -
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw
In reply to Mario Smarduch
Quoted 1 line · Mario Smarduch

That's great but what release is that in?

This is in 2.6.6-rc3. I don't know when it was added.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#16 lore raw
In reply to Mario Smarduch
Quoted 9 lines · Grant Grundler
Quoted 2 lines · Mario Smarduch

But there are cases where
kernel values just need to be read timely.

Watching kernel data from userspace is an impossible task. Unless one has a "real time" scheduler and *halts* all kernel activity, the kernel values visible to user space will lag what the kernel does. Something like KGDB can do that. It's not something I expect to work correctly.

I've worked with protocol applications (Frame Relay/LAPD) that used /dev/kmem to synchronize across kernel allocated buffers to bypass standard system call overhead. The tasks ran at highest priority and were locked. I'm not advacating that kind of use but its been done.

Quoted 3 lines

Ergo define an interface (eg gettimeofday() or /proc/sys/kernel/*) to get "sane" values and move on.

That's kind of what this is intended for.

- Mario.

Quoted 19 lines
Quoted 7 lines

As things
are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed,

Maybe. I don't understand the VM well enough.

Quoted 3 lines

but I'm
not sure how mapping it as uncachable helps? To me this behaviour seems buggy.

I'm still inclined to think the approach is wrong. Maybe you have a better test case that doesn't involve user interaction and shows why the data has to come from /dev/kmem?

grant

#17 lore raw
In reply to Mario Smarduch
Quoted 5 lines · Matthew Wilcox
Quoted 1 line · Mario Smarduch

That's great but what release is that in?

This is in 2.6.6-rc3. I don't know when it was added.

Quoted 7 lines

--
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain

Is the patch applicable to 2.4 variants?

#18 lore raw
In reply to Mario Smarduch
Quoted 4 lines · Mario Smarduch
Quoted 1 line

That's great but what release is that in?

This is in 2.6.6-rc3. I don't know when it was added.

N.B. your test program will need some changes to work under 2.6 as the kernel is linked into virtual addresses in region 5 instead of region 7.

So "IDENBASE" and "jiffies - IDENBASE" would need to be changed.

In fact there might be another whole layer of difficulty writing your test program, as figuring the physical address of the kernel object that you want to look at is a lot harder as the bootloader may have picked any physical address to load the kernel (though in practice at the moment it only does this on SGI sn2 machines).

-Tony

#19 lore raw
In reply to Mario Smarduch
Quoted 11 lines · Tony
Quoted 4 lines · Mario Smarduch
Quoted 1 line

That's great but what release is that in?

This is in 2.6.6-rc3. I don't know when it was added.

N.B. your test program will need some changes to work under 2.6 as the kernel is linked into virtual addresses in region 5 instead of region 7.

So "IDENBASE" and "jiffies - IDENBASE" would need to be changed.

Quoted 7 lines

In fact there might be another whole layer of difficulty writing your test program, as figuring the physical address of the kernel object that you want to look at is a lot harder as the bootloader may have picked any physical address to load the kernel (though in practice at the moment it only does this on SGI sn2 machines).

-Tony

Tony thanks for the pointers. I actually need to
have this work on 2.4 though.