Question on mmap / expecting more calls to vfs_read
Hello!
Intro / context
===============
I'm trying to get a precise list of file names, offsets and byte counts of all read and write operations. I need that information to feed it into a simulator I am working on. I am running qemu-kvm with a slightly patched Linux kernel (additions of calls to printk only) sucking the kernel log from the serial console (kernel option earlyprintk=serial,keep) to a file. The setup itself works okay for me.
Question / problem
==================
When analyzing the kernel logs I produced I noticed stumbled over the case of a few libraries where only the beginning of the file (maybe its ELF header) was read repeated times, but nothing after:
file /lib/ld-2.11.2.so max offset+count 456 total bytes read ever 502840
Especially as libc is among them, I guess some reads pass by me somehow. Any ideas what is shoveling the file bytes around?
I found calls to do_mmap_pgoff for these file, though. That's when I thought I may need help understanding how mmap works.
Is data read into the mapped region instantly all at once? Or is it feed into it on demand when the kernel gets read requests to a mapped area? In either case, where does the data come from if neither vfs_read nor do_readv_writev are ever called on the rest of the file?
Thanks in advance for any help.
Sebastian
Hi...
Quoted 8 lines · Sebastian Pipping
When analyzing the kernel logs I produced I noticed stumbled over the case of a few libraries where only the beginning of the file (maybe its ELF header) was read repeated times, but nothing after:
?file ? ? ? ? ? ? ? ? ?
/lib/ld-2.11.2.so
?max offset+count ? ? ? 456
?total bytes read ever ?502840
Quoted 5 lines
Especially as libc is among them, I guess some reads pass by me somehow. Any ideas what is shoveling the file bytes around?
I found calls to
do_mmap_pgofffor these file, though. That's when I thought I may need help understanding how mmap works.
AFAIK, mmap will eventually calls VFS read in the case of major fault. However, in other cases the content could already be in page cache, that's why you don't see any filesystem or block device related operation.
As we know, ld.so and libc.so are one of the most linked library.... therefore it is very likely they are already in the page cache. "But I saw initial read, why is that?". IMHO that's because filesystem layer need to make sure, it is the same file which is already cached in the page cache.
To be precise, inode number, in which major and minor block device it resides...and possibly other determinant factor. Just after it is decided it's the same, page cache could be 100% utilized..again..assuming all the content of ld.so and libc.so stays in cache and never get flushed back.
Take the above hint with lots of salt, ok? :D
Signature · 7 lines
regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Hi Sebastian,
you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.
Quoted 1 line
From the file system perspective, mmap does not use
vfs_reador
vfs_write calls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method of address_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.
Hope this makes it clearer.
Rajat
Quoted 50 lines · Mulyadi Santosa
Hi...
Quoted 8 lines · Sebastian Pipping
When analyzing the kernel logs I produced I noticed stumbled over the case of a few libraries where only the beginning of the file (maybe its ELF header) was read repeated times, but nothing after:
?file ? ? ? ? ? ? ? ? ?
/lib/ld-2.11.2.so
?max offset+count ? ? ? 456
?total bytes read ever ?502840Quoted 5 lines
Especially as libc is among them, I guess some reads pass by me somehow. Any ideas what is shoveling the file bytes around?
I found calls to
do_mmap_pgofffor these file, though. That's when I thought I may need help understanding how mmap works.AFAIK, mmap will eventually calls VFS read in the case of major fault. However, in other cases the content could already be in page cache, that's why you don't see any filesystem or block device related operation.
As we know, ld.so and libc.so are one of the most linked library.... therefore it is very likely they are already in the page cache. "But I saw initial read, why is that?". IMHO that's because filesystem layer need to make sure, it is the same file which is already cached in the page cache.
To be precise, inode number, in which major and minor block device it resides...and possibly other determinant factor. Just after it is decided it's the same, page cache could be 100% utilized..again..assuming all the content of ld.so and libc.so stays in cache and never get flushed back.
Take the above hint with lots of salt, ok? :D
--
regards,Mulyadi Santosa
Freelance Linux trainer and consultantblog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Quoted 21 lines · Rajat Sharma
Hi Sebastian,
you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.
From the file system perspective, mmap does not use
vfs_readorvfs_writecalls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method ofaddress_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.
Quoted 1 line
Hope this makes it clearer.
Excellent, thank you! I find calls to readpages on file /lib/libc-2.11.2.so now. That may be the missing reads.
Any ideas how get offset and length (like with vfs_read) for a certain page passed to readpage(file, page) ?
Best,
Sebastian
Quoted 22 lines · Sebastian Pipping
Quoted 21 lines · Rajat Sharma
Hi Sebastian,
you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.
From the file system perspective, mmap does not use
vfs_readorvfs_writecalls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method ofaddress_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.
Quoted 7 lines
Quoted 1 line
Hope this makes it clearer.
Excellent, thank you! ?I find calls to readpages on file
/lib/libc-2.11.2.sonow. ?That may be the missing reads.Any ideas how get offset and length (like with
vfs_read) for a certain page passed toreadpage(file, page)?
Add your stats hooks to do_mmap or mmap_region() ???
Signature · 5 lines
Thanks - Manish ================================== [$\*.^ -- I miss being one of them ==================================
Quoted 1 line
Add your stats hooks to
do_mmapormmap_region()???
mmap is one time call to initialize virtual address space of current process with corresponding region on file's disk image. Once this setting is done, this the actual calls to read data from file is accomplish through memory area's (vma) page fault handlers which in turn call readpage address space operation of an inode:
inode->i_mapping->a_ops->readpage(file, page).
so, suitable position is to add hooks on readpage a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child after nfs_readdir is finished.
As far as offset and length of I/O are concerned, page->index gives you its index in the page cache which in turn is equivalent to file offset (page->index << PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will be inode->i_size & ~PAGE_MASK, otherwise it can be PAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.
Having said above, it will still be better if you can state what you want to achieve in little layman language.
Rajat
Quoted 32 lines · Manish Katiyar
Quoted 22 lines · Sebastian Pipping
Quoted 21 lines · Rajat Sharma
Hi Sebastian,
you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.
From the file system perspective, mmap does not use
vfs_readorvfs_writecalls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method ofaddress_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.Quoted 7 lines
Quoted 1 line
Hope this makes it clearer.
Excellent, thank you! I find calls to readpages on file
/lib/libc-2.11.2.sonow. That may be the missing reads.Any ideas how get offset and length (like with
vfs_read) for a certain page passed toreadpage(file, page)?
Quoted 9 lines
Add your stats hooks to
do_mmapormmap_region()???--
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110107/3f190eca/attachment.html
I meant 'file holes' not 'wholes' in last mail... my bad in shooting mails :(
Rajat
Quoted 27 lines · Rajat Sharma
Quoted 1 line
Add your stats hooks to
do_mmapormmap_region()???mmap is one time call to initialize virtual address space of current process with corresponding region on file's disk image. Once this setting is done, this the actual calls to read data from file is accomplish through memory area's (vma) page fault handlers which in turn call readpage address space operation of an inode:
inode->i_mapping->a_ops->readpage(file, page).so, suitable position is to add hooks on readpage
a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child afternfs_readdiris finished.As far as offset and length of I/O are concerned,
page->indexgives you its index in the page cache which in turn is equivalent to file offset (page->index<<PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will beinode->i_size& ~PAGE_MASK, otherwise it can bePAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.
Quoted 51 lines
Having said above, it will still be better if you can state what you want to achieve in little layman language.
Rajat
Quoted 41 lines · Manish Katiyar
Quoted 22 lines · Sebastian Pipping
Quoted 21 lines · Rajat Sharma
Hi Sebastian,
you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.
From the file system perspective, mmap does not use
vfs_readorvfs_writecalls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method ofaddress_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.Quoted 7 lines
Quoted 1 line
Hope this makes it clearer.
Excellent, thank you! I find calls to readpages on file
/lib/libc-2.11.2.sonow. That may be the missing reads.Any ideas how get offset and length (like with
vfs_read) for a certain page passed toreadpage(file, page)?Add your stats hooks to
do_mmapormmap_region()???--
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110107/2216691d/attachment-0001.html
Quoted 8 lines · Rajat Sharma
so, suitable position is to add hooks on readpage
a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child afternfs_readdiris
finished.
I see.
Quoted 7 lines
As far as offset and length of I/O are concerned,
page->indexgives you its index in the page cache which in turn is equivalent to file offset (page->index<<PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will beinode->i_size& ~PAGE_MASK, otherwise it can bePAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.
That sounds great. I have added these lines of code to read_pages() of mm/readahead.c:
================================================
{
struct page *page = list_to_page(pages);
for (page_idx = 0; page_idx < nr_pages;
page_idx++,
page = list_to_page(page->lru.next)) {
pgoff_t offset = 0;
pgoff_t len = PAGE_SIZE;
if (page) {
offset = page->index << PAGE_SHIFT;
if (page->mapping && page->mapping->host) {
struct inode *inode = page->mapping->host;
len = inode->i_size & ~PAGE_MASK;
}
}
printk(KERN_DEBUG "VFS: BODY at 1 read_pages(...)["
"page_idx=%u, offset=%lu, len=%lu, page_size=%lu]",
page_idx, offset, len, PAGE_SIZE); }
}
================================================
Suprisingly I never get len != PAGE_SIZE, i.e. partial pages. Also, in a list of pages given, the offsets are all the same, all 0 in this case:
================================================
[ 2811.993564] VFS: BODY at 1 read_pages(...)[page_idx=0, offset=0, len=4096, page_size=4096] [ 2811.996456] VFS: BODY at 1 read_pages(...)[page_idx=1, offset=0, len=4096, page_size=4096] [ 2811.999305] VFS: BODY at 1 read_pages(...)[page_idx=2, offset=0, len=4096, page_size=4096] [ 2812.002152] VFS: BODY at 1 read_pages(...)[page_idx=3, offset=0,
len=4096, page_size=4096]
================================================
Is page = list_to_page(page->lru.next) the correct way of walking pages? I am wondering how the main loop of read_pages()
for (page_idx = 0; page_idx < nr_pages; page_idx++) {
struct page *page = list_to_page(pages);
[..]
}really iterates, if neither the address of pages changes in the loop nor page_idx is passed to anything. I would think the loop is actually accessing the same page again and again, but somehow my own
page = list_to_page(page->lru.next)
above doesn't seem to do better. Any insights?
Thanks!
Quoted 2 lines
Having said above, it will still be better if you can state what you want to achieve in little layman language.
I want to trace down all reads and writes to the file system including filename, offset and byte count into a log that I can reply inside a simulator.
Best,
Sebastian
Hi Sebastian,
read_pages is a small piece of code, please paste the complete function with your changes over here, I am sure there must be some minor bug otherwise page->lru.next looks fine to me.
Quoted 10 lines
I am wondering how the main loop of
read_pages()for (page_idx = 0; page_idx < nr_pages; page_idx++) { struct page *page = list_to_page(pages); [..] }really iterates, if neither the address of pages changes in the loop nor
page_idxis passed to anything. I would think the loop is actually accessing the same page again and again, but somehow my own
No its not iterating same page, you missed the call to list_del(&page->lru) in the main loop. Remember it removes the first element from the head of the list (as this page is now responsibility of page cache) and next element automatically becomes head of the list ie. list_head *pages.
Quoted 4 lines
I want to trace down all reads and writes to the file system including filename, offset and byte count into a log that I can reply inside a simulator.
for filename you have to track open calls to file system but remember for resolving the complete path name, you have to get into depths of namespace and play with vfsmount data structure.
Rajat
Quoted 98 lines · Sebastian Pipping
Quoted 8 lines · Rajat Sharma
so, suitable position is to add hooks on readpage
a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child afternfs_readdirisfinished.
I see.
Quoted 7 lines
As far as offset and length of I/O are concerned,
page->indexgives you its index in the page cache which in turn is equivalent to file offset (page->index<<PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will beinode->i_size& ~PAGE_MASK, otherwise it can bePAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.That sounds great. ?I have added these lines of code to
read_pages()ofmm/readahead.c:================================================
{
?struct page *page =list_to_page(pages);
?for (page_idx= 0;page_idx<nr_pages;
? ? ?page_idx++,
? ? ?page =list_to_page(page->lru.next)) {
? ?pgoff_t offset = 0;
? ?pgoff_t len =PAGE_SIZE;? ?if (page) {
? ? ?offset =page->index<<PAGE_SHIFT;? ? ?if (
page->mapping&&page->mapping->host) {
? ? ? ?struct inode *inode =page->mapping->host;
? ? ? ?len =inode->i_size& ~PAGE_MASK;
? ? ?}
? ?}? ?printk(KERN_DEBUG "VFS: BODY at 1 read_pages(...)[" ? ? ? ?"page_idx=%u, offset=%lu, len=%lu, page_size=%lu]", ? ? ? ?page_idx, offset, len,
PAGE_SIZE);
?}
}
================================================Suprisingly I never get len !=
PAGE_SIZE, i.e. partial pages. Also, in a list of pages given, the offsets are all the same, all 0 in this case:================================================
[ 2811.993564] VFS: BODY at 1 read_pages(...)[page_idx=0, offset=0, len=4096, page_size=4096] [ 2811.996456] VFS: BODY at 1 read_pages(...)[page_idx=1, offset=0, len=4096, page_size=4096] [ 2811.999305] VFS: BODY at 1 read_pages(...)[page_idx=2, offset=0, len=4096, page_size=4096] [ 2812.002152] VFS: BODY at 1 read_pages(...)[page_idx=3, offset=0,len=4096, page_size=4096]
================================================Is ?page =
list_to_page(page->lru.next)?the correct way of walking pages? ?I am wondering how the main loop ofread_pages()? ? ? ?for (
page_idx= 0;page_idx<nr_pages; page_idx++) { ? ? ? ? ? ? ? ?struct page *page =list_to_page(pages); ? ? ? ? ? ? ? ?[..]
? ? ? ?}really iterates, if neither the address of pages changes in the loop nor
page_idxis passed to anything. ?I would think the loop is actually accessing the same page again and again, but somehow my own?page =
list_to_page(page->lru.next)above doesn't seem to do better. ?Any insights?
Thanks!
Quoted 2 lines
Having said above, it will still be better if you can state what you want to achieve in little layman language.
I want to trace down all reads and writes to the file system including filename, offset and byte count into a log that I can reply inside a simulator.
Best,
Sebastian