How to find out which pages were copied-on-write?
Hi,
in an application that MAP_PRIVATEly mmap()s a file it would be quite helpful for me to find out which pages have been copied-on-write.
I found that mincore() does a similar thing by reporting which pages are currently residing in physical memory, but what I want to know is which pages differ from the original file image on disk.
Can you recommend a way to do that? (does not need to be portable beyond Linux)
Alternatively, it would be sufficient if I could turn a private mapping into a shared one (and possibly do an msync() afterwards if I need to make sure the changes have been written out). Would such a feature need a lot of effort to implement?
Yet another feature that I could use if it were available:
A "copy-on-read"-mapping. There, a page would become a private copy of a process once _another_ process wrote data to the corresponding file location. But I suspect that feature could be very hard to implement...
Regards,
Lutz Vieweg
OK, now that I am considering this problem, I am trying to figure out what problem we are trying to solve.
By reading your email, I gather that you have a single threaded application which is doing an mmap on a file as a MAP_PRIVATE mapping. The memory area is then handed to a library which may modify some pages. You want to decide after the return if you had success and thereby control the writing of the updated data back to the file. Because of the size of the file, doing a second mapping and comparing/copying pages is unreasonable and you would like to only modify the pages that have actually changed.
If that is not what you are trying to do, please give me a similar description of _WHAT_ you are trying to do and not the _HOW_ you think the kernel can make this easier.
Quoted 21 lines · Lutz Vieweg
Hi,
in an application that
MAP_PRIVATElymmap()s a file it would be quite helpful for me to find out which pages have been copied-on-write.I found that
mincore()does a similar thing by reporting which pages are currently residing in physical memory, but what I want to know is which pages differ from the original file image on disk.Can you recommend a way to do that? (does not need to be portable beyond Linux)
Alternatively, it would be sufficient if I could turn a private mapping into a shared one (and possibly do an
msync()afterwards if I need to make sure the changes have been written out). Would such a feature need a lot of effort to implement?
Quoted 5 lines
Yet another feature that I could use if it were available:
A "copy-on-read"-mapping. There, a page would become a private copy of a process once _another_ process wrote data to the corresponding file location. But I suspect that feature could be very hard to implement...
This is a different way of thinking of copy-on-write. I believe you are thinking of the time when there are two processes sharing the page. When one process takes the write fault, the page is copied and the by that process and the other process becomes the exclusive owner of the page.
Thanks,
Robin Holt
Quoted 11 lines · Robin Holt
OK, now that I am considering this problem, I am trying to figure out what problem we are trying to solve.
By reading your email, I gather that you have a single threaded application which is doing an mmap on a file as a
MAP_PRIVATEmapping. The memory area is then handed to a library which may modify some pages. You want to decide after the return if you had success and thereby control the writing of the updated data back to the file. Because of the size of the file, doing a second mapping and comparing/copying pages is unreasonable and you would like to only modify the pages that have actually changed.
That's about it, the most important issue is that I want to avoid having an inconsistent file on the disk for long periods, because a) the application could crash and b) another process might want to map the same file (read-only). And since the application is reaching points where the data is consistent, while it is not in between, it would be nice to have a private mapping while it is inconsistent and commit the changes only at the points where the application knows the data is consistent.
Turning a private into a shared mapping would be a perfect solution since that would mean another process could map the file at any time and find consistent data. The second best solution would be the one where the application just manually writes out the changed pages at the time of consistence, this would at least reduce the times when the data on disk is inconsistent to a minimum.
Quoted 10 lines
Quoted 5 lines
Yet another feature that I could use if it were available:
A "copy-on-read"-mapping. There, a page would become a private copy of a process once _another_ process wrote data to the corresponding file location. But I suspect that feature could be very hard to implement...This is a different way of thinking of copy-on-write. I believe you are thinking of the time when there are two processes sharing the page. When one process takes the write fault, the page is copied and the by that process and the other process becomes the exclusive owner of the page.
A little different: Think of N processes (N may be 8 or so) that mmap() a file using a new mode "MAP_SNAPSHOT" (which could be read-only if a mix with private copy-on-write pages was too hard to realize), and 1 process mmap()ing the same file using MAP_SHARED. Once the N processes mmap()ed the file using MAP_SNAPSHOT, their "view" of the file content would never change, that is, if the one process that mmap()ed the file with MAP_SHARED writes to a page, that page _is_ written to disk the usual way, but the other N processes get a copy of the page before it has been changed, so they will always see the same data.
Once the processes that mmap()ed using MAP_SNAPSHOT unmap the file, the copies of the pages that were changed on disk are simply discarded.
That would - similar to the features mentioned above - allow one process to efficiently work on portions of a huge file over a longer period of time, and only at times when the file in total contains consistent data, other processes could be instructed to mmap() them again to obtain a newer version.
Regards,
Lutz Vieweg
HPAs library LPSM sounds like what you're looking for.
http://freshmeat.net/projects/lpsm/
Or you can do what you want the hard way using mprotect and a SEGV handler.
~mc
Quoted 73 lines · Lutz Vieweg
Quoted 11 lines · Robin Holt
OK, now that I am considering this problem, I am trying to figure out what problem we are trying to solve.
By reading your email, I gather that you have a single threaded application which is doing an mmap on a file as a
MAP_PRIVATEmapping. The memory area is then handed to a library which may modify some pages. You want to decide after the return if you had success and thereby control the writing of the updated data back to the file. Because of the size of the file, doing a second mapping and comparing/copying pages is unreasonable and you would like to only modify the pages that have actually changed.That's about it, the most important issue is that I want to avoid having an inconsistent file on the disk for long periods, because a) the application could crash and b) another process might want to map the same file (read-only). And since the application is reaching points where the data is consistent, while it is not in between, it would be nice to have a private mapping while it is inconsistent and commit the changes only at the points where the application knows the data is consistent.
Turning a private into a shared mapping would be a perfect solution since that would mean another process could map the file at any time and find consistent data. The second best solution would be the one where the application just manually writes out the changed pages at the time of consistence, this would at least reduce the times when the data on disk is inconsistent to a minimum.
Quoted 11 lines
Quoted 5 lines
Yet another feature that I could use if it were available:
A "copy-on-read"-mapping. There, a page would become a private copy of a process once _another_ process wrote data to the corresponding file location. But I suspect that feature could be very hard to implement...This is a different way of thinking of copy-on-write. I believe you are thinking of the time when there are two processes sharing the page. When one process takes the write fault, the page is copied and the by that process and the other process becomes the exclusive owner of the page.
A little different: Think of N processes (N may be 8 or so) that
mmap()a file using a new mode "MAP_SNAPSHOT" (which could be read-only if a mix with private copy-on-write pages was too hard to realize), and 1 process mmap()ing the same file usingMAP_SHARED. Once the N processes mmap()ed the file usingMAP_SNAPSHOT, their "view" of the file content would never change, that is, if the one process that mmap()ed the file withMAP_SHAREDwrites to a page, that page _is_ written to disk the usual way, but the other N processes get a copy of the page before it has been changed, so they will always see the same data.
Once the processes that mmap()ed usingMAP_SNAPSHOTunmap the file, the copies of the pages that were changed on disk are simply discarded.That would - similar to the features mentioned above - allow one process to efficiently work on portions of a huge file over a longer period of time, and only at times when the file in total contains consistent data, other processes could be instructed to
mmap()them again to obtain a newer version.Regards,
Lutz Vieweg
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
Michael Clark, . . . . . . . . . . . . [email protected] Metaparadigm Pte. Ltd . . . . . . . . http://www.metaparadigm.com "Explore Operations Research: The Science of Better at www.scienceofbetter.org "
Quoted 5 lines · Michael Clark
HPAs library LPSM sounds like what you're looking for.
http://freshmeat.net/projects/lpsm/
Or you can do what you want the hard way using mprotect and a SEGV handler.
Certainly a valid idea to consider - doing all those things in userspace... so thanks for the hint!
But wouldn't that introduce a significant overhead and undermine all of the nice advantages the kernel might have in scheduling I/O operations?
However, I shall really consider and profile the mprotect/sighandler approach...
Regards,
Lutz Vieweg
PS: I'm using my own allocator already, so using the C-library implementation
wouldn't gain me much...
Quoted 6 lines · Lutz Vieweg
Quoted 4 lines · Michael Clark
HPAs library LPSM sounds like what you're looking for.
Quoted 8 lines
Quoted 1 line
Or you can do what you want the hard way using mprotect and a SEGV handler.
Certainly a valid idea to consider - doing all those things in userspace... so thanks for the hint!
But wouldn't that introduce a significant overhead and undermine all of the nice advantages the kernel might have in scheduling I/O operations?
Not really. Plain read/write IO is generally faster than mmap IO anyway. You don't use mmap for speed but rather for convenience.
Quoted 8 lines
However, I shall really consider and profile the mprotect/sighandler approach...
Regards,
Lutz Vieweg
PS: I'm using my own allocator already, so using the C-library implementation
wouldn't gain me much...
This wasn't why I suggested it. It's has the commit semantics on memory mapped files that you were asking about (the allocator is optional I believe).
~mc
Quoted 4 lines · Michael Clark
Quoted 2 lines
But wouldn't that introduce a significant overhead and undermine all of the nice advantages the kernel might have in scheduling I/O operations?
Not really. Plain read/write IO is generally faster than mmap IO anyway.
Well, that was my result, too, when I measured mmap() vs. read()/write() with the 2.4.x kernels, however, I was quite impressed recently when I measured write operations with MAP_SHARED regions under 2.6.7 (CPU x86_64), they were not at all slower than ordinary write()s. (congratulations to the involved kernel hackers on that! :-)
Quoted 1 line
You don't use mmap for speed but rather for convenience.
But isn't an advantage with mmap() that there's no need for the kernel to copy what is to be written to a dedicated buffer? The kernel could initiate DMA writes directly from the working memory...
Regards,
Lutz Vieweg
Quoted 6 lines · Lutz Vieweg
Quoted 1 line
You don't use mmap for speed but rather for convenience.
But isn't an advantage with
mmap()that there's no need for the kernel to copy what is to be written to a dedicated buffer? The kernel could initiate DMA writes directly from the working memory...
Yes, but page faults are expensive too. Each time a page is written out it needs to be marked read only again and will cause a page fault for the next write access from userspace. For certain workloads this can easily add up to more than copy_(to|from)_user in read/write.
read/write also gives you more explicit control on IO batching and scheduling (when to read or write). Less need for the kernel to employ tricks to effectively coaslesce IOs on dirtied pages or sense streaming access patterns.
~mc
Quoted 4 lines · Michael Clark
Quoted 2 lines · Lutz Vieweg
Quoted 1 line
You don't use mmap for speed but rather for convenience.
Quoted 8 lines
Quoted 3 lines
But isn't an advantage with
mmap()that there's no need for the kernel to copy what is to be written to a dedicated buffer? The kernel could initiate DMA writes directly from the working memory...Yes, but page faults are expensive too. Each time a page is written out it needs to be marked read only again and will cause a page fault for the next write access from userspace. For certain workloads this can easily add up to more than copy_(to|from)_user in read/write.
But I would need exactly the same number of pagefaults if I implemented the "mark-dirty-on-write" logic in userspace using SIGSEGV and signal handlers, as it is done by the LPSM software...
Quoted 4 lines
read/write also gives you more explicit control on IO batching and scheduling (when to read or write). Less need for the kernel to employ tricks to effectively coaslesce IOs on dirtied pages or sense streaming access patterns.
But if the kernel would turn a private copy of a c-o-w page into a "dirty"-page that is marked for writing out to disk, another process could mmap() the very same page even before it has been written to disk, while if I write out dirty pages using write() in userspace, other processes probably won't notice before all the data has reached the disk, which could take quite some time.
And unlike the user space application, the kernel knows which writes go to which physical disk so it can e.g. make better use of striping.
Regards,
Lutz Vieweg
Quoted 18 lines · Lutz Vieweg
Quoted 10 lines · Michael Clark
Quoted 6 lines · Lutz Vieweg
Quoted 1 line
You don't use mmap for speed but rather for convenience.
But isn't an advantage with
mmap()that there's no need for the kernel to copy what is to be written to a dedicated buffer? The kernel could initiate DMA writes directly from the working memory...Quoted 4 lines
Yes, but page faults are expensive too. Each time a page is written out it needs to be marked read only again and will cause a page fault for the next write access from userspace. For certain workloads this can easily add up to more than copy_(to|from)_user in read/write.
Quoted 3 lines
But I would need exactly the same number of pagefaults if I implemented the "mark-dirty-on-write" logic in userspace using SIGSEGV and signal handlers, as it is done by the LPSM software...
Yes, that's sort of my point although you get the commit semantics you want, albeit a little more usersapce signal overhead and an mprotect call (you've already taken the exception so the extra signal overhead shouldn't be too much), but perhaps less overall page faults than MAP_SHARED (which is the mmap variant that writes dirtied pages back to backing store) as you control when to mark the page clean again ie. do the writeout at your commit point and not before.
Really my point was you don't use mmap for speed but rather for convenience.
~mc