mmap question
I'm trying to implement a completely synchronous mmap in my filesystem, but am running into some difficulty and was wondering if someone could give me some insight/clue.
I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap. The problem is that mmap writes don't seem to register right away. If I have a test app:
start = mmap(0, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
if((int)start==-1) {
perror("mmap:");
exit(-1);
}
memcpy(buffer, start, 10);
buffer[10] = 0;
printf("reply: (%s)\n", buffer);
sprintf(start, "hello world\n");
memcpy(buffer, start, 10);
buffer[10] = 0;
printf("reply2: (%s)\n", buffer);
munmap(start, 8192);
close(fd);I get the right results (because the read is also an mmap through the buffer, but it looks like I don't see the file's set_page_dirty method called until after I see the second read. If I do use normal reads (instead of mmap reads) I get the wrong results because my normal file read method doesn't go through the page cache.
What am I doing wrong? Is what I'm trying to do impossible, and if so, how can I get as close as possible?
Thanks in advance for any help.
-eric
Quoted 5 lines
I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap.
...
What am I doing wrong? Is what I'm trying to do impossible, and if so, how can I get as close as possible?
It looks to me like you're running into the fundamental limitation that the CPU doesn't notify Linux every time you store into a memory location. It does, though, set the dirty flag in the page table, and Linux eventually inspects that flag and finds out that you have stored in the past. At that time, it can call set_page_dirty.
Without knowing what properties of not having a cache you were hoping for, I couldn't say what alternative would be closest to this.
Hypothetically, if you had a backing storage device that could do memory mapped I/O, you could have mmapped direct I/O.
--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems
Quoted 5 lines · Bryan Henderson
It looks to me like you're running into the fundamental limitation that the CPU doesn't notify Linux every time you store into a memory location. It does, though, set the dirty flag in the page table, and Linux eventually inspects that flag and finds out that you have stored in the past. At that time, it can call
set_page_dirty.
well, we *could* know ... never map this page writable. have a per-vma flag that says "emulate writes", and call the filesystem to update backing storage before returning to the application.
the price of doing this would be exorbitant.
Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Quoted 3 lines
well, we *could* know ... never map this page writable. have a per-vma flag that says "emulate writes", and call the filesystem to update backing storage before returning to the application.
Ah yes, you mean, I take it, that the page fault handler would look at the user's program and emulate the faulting store instruction and return to the instruction after it. Very clever.
And as long as we're going down that path, we should also consider changing exec() so instead of branching into the program, it just interprets it!
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Bryan Henderson
Quoted 5 lines
I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap.
...
What am I doing wrong? Is what I'm trying to do impossible, and if so, how can I get as close as possible?
Quoted 6 lines
It looks to me like you're running into the fundamental limitation that the CPU doesn't notify Linux every time you store into a memory location. It does, though, set the dirty flag in the page table, and Linux eventually inspects that flag and finds out that you have stored in the past. At that time, it can call
set_page_dirty.
Is there an existing interface to force it to check if the page is dirty -- for example, one way to maintain coherency would be for my normal read/write path to check if the page is dirty and write it back and invalidate the page before executing the normal-path read/write. This may be an unlikely scenerio, but I think it is one of the corner cases fsx tickles.
-eric
Quoted 2 lines
Is there an existing interface to force it to check if the page is dirty
The msync() system call and libc function does that. And then it does the same thing as fsync().
--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems
I forgot you were talking about code inside the kernel. In that case, filemap_sync().
Just wanted to thank everybody for their help. I ended up backing off the lofty goals and worked towards just passing fsx for now. The main thing needed was an invalidate_inode_pages2(inode->i_mapping) when I wrote the file through the uncached method (in order to invalidate any read buffers). It turns out fsx was msync'ing the mmap writes so that path was already clear.
I've got plenty of other fires to fight before I go after the more extreme case, and perhaps direct I/O is the appropriate way to go about it.
Thanks again.
-eric
Quoted 3 lines · Eric Van Hensbergen
I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap.
I am only a kernel newbie but I belive that you need a pagefault every time the memory is accessed. Don't know how to do that or if you would need to be platform specific, though.
HTH
Martin Jambor