Skip to contentSkip to conversation

mmap question

DISCUSSION Eric Van Hensbergen 4 participants 9 messages Mar 21, 2005 – Mar 22, 2005 1 message not in the archive VFS

lore
Share & tools
lore
#0 lore raw

I'm trying to implement a completely synchronous mmap in my filesystem, but am running into some difficulty and was wondering if someone could give me some insight/clue.

I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap. The problem is that mmap writes don't seem to register right away. If I have a test app:

        start = mmap(0, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
        if((int)start==-1) {
                perror("mmap:");
                exit(-1);
        }
        memcpy(buffer, start, 10);
        buffer[10] = 0;
        printf("reply: (%s)\n", buffer);
        sprintf(start, "hello world\n");
        memcpy(buffer, start, 10);
        buffer[10] = 0;
        printf("reply2: (%s)\n", buffer);
        munmap(start, 8192);
        close(fd);

I get the right results (because the read is also an mmap through the buffer, but it looks like I don't see the file's set_page_dirty method called until after I see the second read. If I do use normal reads (instead of mmap reads) I get the wrong results because my normal file read method doesn't go through the page cache.

What am I doing wrong? Is what I'm trying to do impossible, and if so, how can I get as close as possible?

Thanks in advance for any help.

-eric

#1 lore raw
Quoted 5 lines

I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap.
...
What am I doing wrong? Is what I'm trying to do impossible, and if so, how can I get as close as possible?

It looks to me like you're running into the fundamental limitation that the CPU doesn't notify Linux every time you store into a memory location. It does, though, set the dirty flag in the page table, and Linux eventually inspects that flag and finds out that you have stored in the past. At that time, it can call set_page_dirty.

Without knowing what properties of not having a cache you were hoping for, I couldn't say what alternative would be closest to this.

Hypothetically, if you had a backing storage device that could do memory mapped I/O, you could have mmapped direct I/O.

--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems

#2 lore raw
Quoted 5 lines · Bryan Henderson

It looks to me like you're running into the fundamental limitation that the CPU doesn't notify Linux every time you store into a memory location. It does, though, set the dirty flag in the page table, and Linux eventually inspects that flag and finds out that you have stored in the past. At that time, it can call set_page_dirty.

well, we *could* know ... never map this page writable. have a per-vma flag that says "emulate writes", and call the filesystem to update backing storage before returning to the application.

the price of doing this would be exorbitant.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#5 lore raw
In reply to a message not in the archive
Quoted 6 lines · Bryan Henderson
Quoted 5 lines

I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap.
...
What am I doing wrong? Is what I'm trying to do impossible, and if so, how can I get as close as possible?

Quoted 6 lines

It looks to me like you're running into the fundamental limitation that the CPU doesn't notify Linux every time you store into a memory location. It does, though, set the dirty flag in the page table, and Linux eventually inspects that flag and finds out that you have stored in the past. At that time, it can call set_page_dirty.

Is there an existing interface to force it to check if the page is dirty -- for example, one way to maintain coherency would be for my normal read/write path to check if the page is dirty and write it back and invalidate the page before executing the normal-path read/write. This may be an unlikely scenerio, but I think it is one of the corner cases fsx tickles.

-eric

#3 lore raw
In reply to Matthew Wilcox
Quoted 3 lines

well, we *could* know ... never map this page writable. have a per-vma flag that says "emulate writes", and call the filesystem to update backing storage before returning to the application.

Ah yes, you mean, I take it, that the page fault handler would look at the user's program and emulate the faulting store instruction and return to the instruction after it. Very clever.

And as long as we're going down that path, we should also consider changing exec() so instead of branching into the program, it just interprets it!

#6 lore raw
In reply to Eric Van Hensbergen
Quoted 2 lines

Is there an existing interface to force it to check if the page is dirty

The msync() system call and libc function does that. And then it does the same thing as fsync().

--
Bryan Henderson IBM Almaden Research Center
San Jose CA Filesystems

#9 lore raw
In reply to Eric Van Hensbergen
Quoted 3 lines · Eric Van Hensbergen

I want all my file system's operations to be complete uncached and synchronous, but I also want to support mmap.

I am only a kernel newbie but I belive that you need a pagefault every time the memory is accessed. Don't know how to do that or if you would need to be platform specific, though.

HTH

Martin Jambor

#8 lore raw
In reply to Bryan Henderson

Just wanted to thank everybody for their help. I ended up backing off the lofty goals and worked towards just passing fsx for now. The main thing needed was an invalidate_inode_pages2(inode->i_mapping) when I wrote the file through the uncached method (in order to invalidate any read buffers). It turns out fsx was msync'ing the mmap writes so that path was already clear.

I've got plenty of other fires to fight before I go after the more extreme case, and perhaps direct I/O is the appropriate way to go about it.

Thanks again.

-eric