mmap() of /dev/kmem issue
I have problem of viewing memory contents via mmap() of /dev/kmem. After mmap()'ing /dev/kmem for reading it appears that data accessed in the kernel lives in different cache lines from data accessed in the user space. The lseek()/read() from kernel reflects latest updates, but same reads from user space mmap fetches old data, and vice versa when updating from user. When I flush the caches (cat of a big file to another file), then user mapped value reflects the latest changes but then again remains frozen. I'm having trouble understanding how can this happen since the caches are physically indexed. Any ideas what the problem could be?
- Mario.
Quoted 4 lines · Mario Smarduch
I have problem of viewing memory contents via
mmap()of/dev/kmem.
...
Any ideas what the problem could be?
Can you post a test case? What kernel are you running?
Quoted 1 line · Mario Smarduch
I have problem of viewing memory contents via
mmap()of/dev/kmem.
1) which regions/addresses are you accessing?
2) what values do you see/should see?
3) test case?
grant
Quoted 18 lines
After mmap()'ing
/dev/kmemfor reading
it appears that data accessed in the kernel lives in different cache lines from data accessed in the user space. The lseek()/read() from kernel reflects latest updates, but same reads from user space mmap fetches old data, and vice versa when updating from user. When I flush the caches (cat of a big file to another file), then user mapped value reflects the latest changes but then again remains frozen. I'm having trouble understanding how can this happen since the caches are physically indexed. Any ideas what the problem could be?- Mario.
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Bjorn Helgaas
Quoted 4 lines · Mario Smarduch
I have problem of viewing memory contents via
mmap()of/dev/kmem.
...
Any ideas what the problem could be?Can you post a test case? What kernel are you running?
To run the test first verify the PAGE_MASK matches. Secondly compile tstmmapkmem.c, and run runtval.sh it expects an upto date /boot/System.map file. It seems like the region doesn't appear to be mapped cachable.
runtval.sh
-------
#!/bin/ksh
jiffies=0xgrep "D jiffies" /boot/System.map | awk '{ print $1 }' ./tstmmapkmem $jiffies
tstmmapkmem.c - cc -o tstmmapkmem tstmmapkmem.c
----------
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <libgen.h>
#include <sys/time.h>
#include <stdlib.h>
#include <sys/mman.h>
#define IDENBASE 0xe000000000000000
#define DEVICE "/dev/kmem"
#define PAGE_MASK (0x4000 - 1) /* page is 16k */
#define wmb() __asm__ __volatile__ ("": : :"memory")
extern int errno;main(int argc, char *argv[])
{
int fd;
ulong mapaddr;
volatile long *jifptr, jiffies;
jiffies = strtoul(argv[1], 0, 16);
printf("jiffies = 0x%lx+AFw-n", jiffies);
printf("pid = %d+AFw-n", getpid());
if((fd = open(DEVICE, O_RDWR)) < 0)
printf("open failed - errno = %d+AFw-n", errno), exit(errno); jiffies -= IDENBASE;
mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
PROT_WRITE|PROT_R
EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK);
if(mapaddr = -1UL) {
printf("mmap failed errno = %d+AFw-n", errno);
exit(-1); }
printf("mapaddr = 0x%lx+AFw-n", mapaddr);
jiffies = mapaddr +- (jiffies & PAGE_MASK);
jifptr = (long *) jiffies;
printf("jifptr = 0x%lx+AFw-n", jifptr);
pragain:
printf("jiffies: %ld+AFw-n", *jifptr);
printf("Type return to read again (and flush cashe in meantime):+AFw-n");
getchar();
goto pragain;
}
- Mario.
Quoted 2 lines · Mario Smarduch
jiffies=0x
grep "D jiffies" /boot/System.map | awk '{ print $1 }'./tstmmapkmem $jiffies
Maybe the process is being prempted?
Ie maybe the problem is the contents are "stale" by the time they get displayed on the console/terminal. jiffies gets updated 1000/second so on occasion I would expect this to happen.
Maybe pick something that doesn't get updated so frequently?
grant
Quoted 46 lines
#define DEVICE "
/dev/kmem"
#definePAGE_MASK(0x4000- 1) /* page is 16k */#define
wmb()__asm____volatile__("": : :"memory")extern int errno;
main(int argc, char *argv[]){ int fd; ulong mapaddr; volatile long *jifptr, jiffies; jiffies = strtoul(argv[1], 0, 16); printf("jiffies = 0x%lx\n", jiffies); printf("pid = %d\n", getpid()); if((fd = open(DEVICE, O_RDWR)) < 0) printf("open failed - errno = %d\n", errno), exit(errno); jiffies -= IDENBASE; mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000, PROT_WRITE|PROT_R EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & ~PAGE_MASK); if(mapaddr = -1UL) { printf("mmap failed errno = %d\n", errno); exit(-1); } printf("mapaddr = 0x%lx\n", mapaddr); jiffies = mapaddr + (jiffies & PAGE_MASK); jifptr = (long *) jiffies; printf("jifptr = 0x%lx\n", jifptr); pragain: printf("jiffies: %ld\n", *jifptr); printf("Type return to read again (and flush cashe in meantime):\n"); getchar(); goto pragain; }- Mario.
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Grant Grundler
Quoted 2 lines · Mario Smarduch
jiffies=0x
grep "D jiffies" /boot/System.map | awk '{ print $1 }'./tstmmapkmem $jiffiesMaybe the process is being prempted?
Ie maybe the problem is the contents are "stale" by the time they get displayed on the console/terminal. jiffies gets updated 1000/second so on occasion I would expect this to happen.
Maybe pick something that doesn't get updated so frequently?
I can't see how that could have a bearing if both kernel/user are hitting the cache you should see the same values everytime.
- Mario
Quoted 50 lines
grant
Quoted 46 lines
#define DEVICE "
/dev/kmem"
#definePAGE_MASK(0x4000- 1) /* page is 16k */#define
wmb()__asm____volatile__("": : :"memory")extern int errno;
main(int argc, char *argv[]){ int fd; ulong mapaddr; volatile long *jifptr, jiffies; jiffies = strtoul(argv[1], 0, 16); printf("jiffies = 0x%lx+AFw-n", jiffies); printf("pid = %d+AFw-n", getpid()); if((fd = open(DEVICE, O_RDWR)) < 0) printf("open failed - errno = %d+AFw-n", errno), exit(errno); jiffies -= IDENBASE; mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000, PROT_WRITE|PROT_R EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK); if(mapaddr = -1UL) { printf("mmap failed errno = %d+AFw-n", errno); exit(-1); } printf("mapaddr = 0x%lx+AFw-n", mapaddr); jiffies = mapaddr (jiffies & PAGE_MASK); jifptr = (long *) jiffies; printf("jifptr = 0x%lx+AFw-n", jifptr); pragain: printf("jiffies: %ld+AFw-n", *jifptr); printf("Type return to read again (and flush cashe in meantime):+AFw-n"); getchar(); goto pragain; }- Mario.
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
It appears that from mmap_mem() noncached_address() succeeds on PageReserved for ranges in: KERNEL_START- KERNEL_END, I tried a temporary workaround by making an exception to addresses in that range and things work (that is for this program only) but what sort of problems would such a change introduce?
if(PageReserved(page) && !((va >= KERNEL_START) && (va < KERNEL_END)) )
return (1);
I would think allot of people would like to mmap() /dev/kmem to take a quick look at kernel values, atleast that's what I've seen on other Unix variants.
- mario.
Quoted 59 lines · Grant Grundler
Quoted 2 lines · Mario Smarduch
jiffies=0x
grep "D jiffies" /boot/System.map | awk '{ print $1 }'./tstmmapkmem $jiffiesMaybe the process is being prempted?
Ie maybe the problem is the contents are "stale" by the time they get displayed on the console/terminal. jiffies gets updated 1000/second so on occasion I would expect this to happen.
Maybe pick something that doesn't get updated so frequently?grant
Quoted 46 lines
#define DEVICE "
/dev/kmem"
#definePAGE_MASK(0x4000- 1) /* page is 16k */#define
wmb()__asm____volatile__("": : :"memory")extern int errno;
main(int argc, char *argv[]){ int fd; ulong mapaddr; volatile long *jifptr, jiffies; jiffies = strtoul(argv[1], 0, 16); printf("jiffies = 0x%lx+AFw-n", jiffies); printf("pid = %d+AFw-n", getpid()); if((fd = open(DEVICE, O_RDWR)) < 0) printf("open failed - errno = %d+AFw-n", errno), exit(errno); jiffies -= IDENBASE; mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000, PROT_WRITE|PROT_R EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK); if(mapaddr = -1UL) { printf("mmap failed errno = %d+AFw-n", errno); exit(-1); } printf("mapaddr = 0x%lx+AFw-n", mapaddr); jiffies = mapaddr (jiffies & PAGE_MASK); jifptr = (long *) jiffies; printf("jifptr = 0x%lx+AFw-n", jifptr); pragain: printf("jiffies: %ld+AFw-n", *jifptr); printf("Type return to read again (and flush cashe in meantime):+AFw-n"); getchar(); goto pragain; }- Mario.
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Mario Smarduch
I would think allot of people would like to
mmap()/dev/kmemto take a quick look at kernel values, atleast that's what I've seen on other Unix variants.
yes - and that's a religious issue. I know people would like like to see /dev/kmem completely removed.
Many wise kernel engineers have advised me to add /proc or similar "architected" interfaces. That way a program will merely get an error instead of crashing the box when the kernel changes it's data structures or how said structures get used.
grant
Quoted 8 lines · Grant Grundler
Quoted 3 lines · Mario Smarduch
I would think allot of people would like to
mmap()/dev/kmemto take a quick look at kernel values, atleast that's what I've seen on other Unix variants.yes - and that's a religious issue. I know people would like like to see
/dev/kmemcompletely removed.
Quoted 7 lines
Many wise kernel engineers have advised me to add /proc or similar "architected" interfaces. That way a program will merely get an error instead of crashing the box when the kernel changes it's data structures or how said structures get used.
grant
That's understandable. But there are cases where kernel values just need to be read timely. As things are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed, but I'm not sure how mapping it as uncachable helps? To me this behaviour seems buggy.
- Mario.
Quoted 8 lines
That's understandable. But there are cases where kernel values just need to be read timely. As things are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed, but I'm not sure how mapping it as uncachable helps? To me this behaviour seems buggy.
Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).
-Tony
Quoted 2 lines · Mario Smarduch
But there are cases where
kernel values just need to be read timely.
Watching kernel data from userspace is an impossible task. Unless one has a "real time" scheduler and *halts* all kernel activity, the kernel values visible to user space will lag what the kernel does. Something like KGDB can do that. It's not something I expect to work correctly.
Ergo define an interface (eg gettimeofday() or /proc/sys/kernel/*) to get "sane" values and move on.
Quoted 5 lines
As things
are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed,
Maybe. I don't understand the VM well enough.
Quoted 3 lines
but I'm
not sure how mapping it as uncachable helps? To me this behaviour seems buggy.
I'm still inclined to think the approach is wrong. Maybe you have a better test case that doesn't involve user interaction and shows why the data has to come from /dev/kmem?
grant
Quoted 3 lines · Tony
Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).
... which is why we don't do that.
See drivers/char/mem.c::mmap_mem(). We call uncached_access() which does:
#elif defined(CONFIG_IA64)
/*
* On ia64, we ignore O_SYNC because we cannot tolerate memory attribute
aliases.
*/
return !(efi_mem_attributes(addr) & EFI_MEMORY_WB);Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Quoted 9 lines · Tony
Quoted 8 lines
That's understandable. But there are cases where kernel values just need to be read timely. As things are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed, but I'm not sure how mapping it as uncachable helps? To me this behaviour seems buggy.
Quoted 5 lines
Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).
-Tony
The background on the issue is that I'm trying to mmap() some data in KERNEL_START-KERNEL_END range
to read some data, without having to go through a system call. This results in uncacheble mapping in
user space, which can results in what you stated above. Its something I did routinly on other Unix variants. And I think that restriction is not there on Linux IA32 boxes. The reason its mapped uncached its because that range is marked reserved. I'm guessing that reserved memory map ranges are uncachable (just guessing here), but in this case the range in question is accessed cacheble by the kernel.
- Mario.
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines · Tony
Mapping kernel memory as uncacheable into a user process seems like a recipe for disaster (well at least for an MCA when the processor catches you using mixed attributes).
... which is why we don't do that.
Quoted 8 lines
See drivers/char/mem.c::mmap_mem(). We call
uncached_access()which does:#elif
defined(CONFIG_IA64)
/*
* On ia64, we ignoreO_SYNCbecause we cannot tolerate memory attribute
aliases.*/ return !(efi_mem_attributes(addr) & EFI_MEMORY_WB);
That's great but what release is that in?
- mario.
Quoted 13 lines
--
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain -
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Mario Smarduch
That's great but what release is that in?
This is in 2.6.6-rc3. I don't know when it was added.
Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Quoted 1 line · Matthew Wilcox
This is in 2.6.6-rc3. I don't know when it was added.
Almost a year ago:
http://linux.bkbits.net:8080/linux-2.5/cset%401.1046.264.19?nav=index.html|src/.|src/drivers|src/drivers/char|related/drivers/char/mem.c
(there are some relevant changes since then too obviously)
--cw
Quoted 9 lines · Grant Grundler
Quoted 2 lines · Mario Smarduch
But there are cases where
kernel values just need to be read timely.Watching kernel data from userspace is an impossible task. Unless one has a "real time" scheduler and *halts* all kernel activity, the kernel values visible to user space will lag what the kernel does. Something like KGDB can do that. It's not something I expect to work correctly.
I've worked with protocol applications (Frame Relay/LAPD) that used /dev/kmem to synchronize across kernel allocated buffers to bypass standard system call overhead. The tasks ran at highest priority and were locked. I'm not advacating that kind of use but its been done.
Quoted 3 lines
Ergo define an interface (eg
gettimeofday()or/proc/sys/kernel/*) to get "sane" values and move on.
That's kind of what this is intended for.
- Mario.
Quoted 19 lines
Quoted 7 lines
As things
are right now you can still corrupt the kernel but are unable to do anything useful because of attribute aliasing to that range. Perhaps a rejection of mapping such regions for writes should be imposed,Maybe. I don't understand the VM well enough.
Quoted 3 lines
but I'm
not sure how mapping it as uncachable helps? To me this behaviour seems buggy.I'm still inclined to think the approach is wrong. Maybe you have a better test case that doesn't involve user interaction and shows why the data has to come from
/dev/kmem?grant
Quoted 5 lines · Matthew Wilcox
Quoted 1 line · Mario Smarduch
That's great but what release is that in?
This is in 2.6.6-rc3. I don't know when it was added.
Quoted 7 lines
--
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Is the patch applicable to 2.4 variants?
Quoted 4 lines · Mario Smarduch
Quoted 1 line
That's great but what release is that in?
This is in 2.6.6-rc3. I don't know when it was added.
N.B. your test program will need some changes to work under 2.6 as the kernel is linked into virtual addresses in region 5 instead of region 7.
So "IDENBASE" and "jiffies - IDENBASE" would need to be changed.
In fact there might be another whole layer of difficulty writing your test program, as figuring the physical address of the kernel object that you want to look at is a lot harder as the bootloader may have picked any physical address to load the kernel (though in practice at the moment it only does this on SGI sn2 machines).
-Tony
Quoted 11 lines · Tony
Quoted 4 lines · Mario Smarduch
Quoted 1 line
That's great but what release is that in?
This is in 2.6.6-rc3. I don't know when it was added.
N.B. your test program will need some changes to work under 2.6 as the kernel is linked into virtual addresses in region 5 instead of region 7.
So "IDENBASE" and "jiffies - IDENBASE" would need to be changed.
Quoted 7 lines
In fact there might be another whole layer of difficulty writing your test program, as figuring the physical address of the kernel object that you want to look at is a lot harder as the bootloader may have picked any physical address to load the kernel (though in practice at the moment it only does this on SGI sn2 machines).
-Tony
Tony thanks for the pointers. I actually need to
have this work on 2.4 though.