Skip to contentSkip to conversation

Question on mmap / expecting more calls to vfs_read

DISCUSSION Sebastian Pipping 4 participants 9 messages Jan 5, 2011 – Jan 12, 2011

lore
Share & tools
lore
#0 lore raw

Hello!

Intro / context
===============
I'm trying to get a precise list of file names, offsets and byte counts of all read and write operations. I need that information to feed it into a simulator I am working on. I am running qemu-kvm with a slightly patched Linux kernel (additions of calls to printk only) sucking the kernel log from the serial console (kernel option earlyprintk=serial,keep) to a file. The setup itself works okay for me.

Question / problem
==================
When analyzing the kernel logs I produced I noticed stumbled over the case of a few libraries where only the beginning of the file (maybe its ELF header) was read repeated times, but nothing after:

  file                   /lib/ld-2.11.2.so
  max offset+count       456
  total bytes read ever  502840

Especially as libc is among them, I guess some reads pass by me somehow. Any ideas what is shoveling the file bytes around?

I found calls to do_mmap_pgoff for these file, though. That's when I thought I may need help understanding how mmap works.

Is data read into the mapped region instantly all at once? Or is it feed into it on demand when the kernel gets read requests to a mapped area? In either case, where does the data come from if neither vfs_read nor do_readv_writev are ever called on the rest of the file?

Thanks in advance for any help.

Sebastian

#1 lore raw

Hi...

Quoted 8 lines · Sebastian Pipping

When analyzing the kernel logs I produced I noticed stumbled over the case of a few libraries where only the beginning of the file (maybe its ELF header) was read repeated times, but nothing after:

?file ? ? ? ? ? ? ? ? ? /lib/ld-2.11.2.so
?max offset+count ? ? ? 456
?total bytes read ever ?502840

Quoted 5 lines

Especially as libc is among them, I guess some reads pass by me somehow. Any ideas what is shoveling the file bytes around?

I found calls to do_mmap_pgoff for these file, though. That's when I thought I may need help understanding how mmap works.

AFAIK, mmap will eventually calls VFS read in the case of major fault. However, in other cases the content could already be in page cache, that's why you don't see any filesystem or block device related operation.

As we know, ld.so and libc.so are one of the most linked library.... therefore it is very likely they are already in the page cache. "But I saw initial read, why is that?". IMHO that's because filesystem layer need to make sure, it is the same file which is already cached in the page cache.

To be precise, inode number, in which major and minor block device it resides...and possibly other determinant factor. Just after it is decided it's the same, page cache could be 100% utilized..again..assuming all the content of ld.so and libc.so stays in cache and never get flushed back.

Take the above hint with lots of salt, ok? :D

Signature · 7 lines
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com
#2 lore raw

Hi Sebastian,

you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.

Quoted 1 line

From the file system perspective, mmap does not use vfs_read or

vfs_write calls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method of address_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.

Hope this makes it clearer.

Rajat

Quoted 50 lines · Mulyadi Santosa

Hi...

Quoted 8 lines · Sebastian Pipping

When analyzing the kernel logs I produced I noticed stumbled over the case of a few libraries where only the beginning of the file (maybe its ELF header) was read repeated times, but nothing after:

?file ? ? ? ? ? ? ? ? ? /lib/ld-2.11.2.so
?max offset+count ? ? ? 456
?total bytes read ever ?502840

Quoted 5 lines

Especially as libc is among them, I guess some reads pass by me somehow. Any ideas what is shoveling the file bytes around?

I found calls to do_mmap_pgoff for these file, though. That's when I thought I may need help understanding how mmap works.

AFAIK, mmap will eventually calls VFS read in the case of major fault. However, in other cases the content could already be in page cache, that's why you don't see any filesystem or block device related operation.

As we know, ld.so and libc.so are one of the most linked library.... therefore it is very likely they are already in the page cache. "But I saw initial read, why is that?". IMHO that's because filesystem layer need to make sure, it is the same file which is already cached in the page cache.

To be precise, inode number, in which major and minor block device it resides...and possibly other determinant factor. Just after it is decided it's the same, page cache could be 100% utilized..again..assuming all the content of ld.so and libc.so stays in cache and never get flushed back.

Take the above hint with lots of salt, ok? :D

--
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

#3 lore raw
Quoted 21 lines · Rajat Sharma

Hi Sebastian,

you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.

From the file system perspective, mmap does not use vfs_read or vfs_write calls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method of address_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.

Quoted 1 line

Hope this makes it clearer.

Excellent, thank you! I find calls to readpages on file /lib/libc-2.11.2.so now. That may be the missing reads.

Any ideas how get offset and length (like with vfs_read) for a certain page passed to readpage(file, page) ?

Best,

Sebastian

#4 lore raw
Quoted 22 lines · Sebastian Pipping
Quoted 21 lines · Rajat Sharma

Hi Sebastian,

you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.

From the file system perspective, mmap does not use vfs_read or vfs_write calls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method of address_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.

Quoted 7 lines
Quoted 1 line

Hope this makes it clearer.

Excellent, thank you! ?I find calls to readpages on file /lib/libc-2.11.2.so now. ?That may be the missing reads.

Any ideas how get offset and length (like with vfs_read) for a certain page passed to readpage(file, page) ?

Add your stats hooks to do_mmap or mmap_region() ???

Signature · 5 lines
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================
#5 lore raw
Quoted 1 line

Add your stats hooks to do_mmap or mmap_region() ???

mmap is one time call to initialize virtual address space of current process with corresponding region on file's disk image. Once this setting is done, this the actual calls to read data from file is accomplish through memory area's (vma) page fault handlers which in turn call readpage address space operation of an inode:

inode->i_mapping->a_ops->readpage(file, page).

so, suitable position is to add hooks on readpage a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child after nfs_readdir is finished.

As far as offset and length of I/O are concerned, page->index gives you its index in the page cache which in turn is equivalent to file offset (page->index << PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will be inode->i_size & ~PAGE_MASK, otherwise it can be PAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.

Having said above, it will still be better if you can state what you want to achieve in little layman language.

Rajat

Quoted 32 lines · Manish Katiyar
Quoted 22 lines · Sebastian Pipping
Quoted 21 lines · Rajat Sharma

Hi Sebastian,

you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.

From the file system perspective, mmap does not use vfs_read or vfs_write calls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method of address_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.

Quoted 7 lines
Quoted 1 line

Hope this makes it clearer.

Excellent, thank you! I find calls to readpages on file /lib/libc-2.11.2.so now. That may be the missing reads.

Any ideas how get offset and length (like with vfs_read) for a certain page passed to readpage(file, page) ?

Quoted 9 lines

Add your stats hooks to do_mmap or mmap_region() ???

--
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110107/3f190eca/attachment.html

#6 lore raw

I meant 'file holes' not 'wholes' in last mail... my bad in shooting mails :(

Rajat

Quoted 27 lines · Rajat Sharma
Quoted 1 line

Add your stats hooks to do_mmap or mmap_region() ???

mmap is one time call to initialize virtual address space of current process with corresponding region on file's disk image. Once this setting is done, this the actual calls to read data from file is accomplish through memory area's (vma) page fault handlers which in turn call readpage address space operation of an inode:

inode->i_mapping->a_ops->readpage(file, page).

so, suitable position is to add hooks on readpage a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child after nfs_readdir is finished.

As far as offset and length of I/O are concerned, page->index gives you its index in the page cache which in turn is equivalent to file offset (page->index << PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will be inode->i_size & ~PAGE_MASK, otherwise it can be PAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.

Quoted 51 lines

Having said above, it will still be better if you can state what you want to achieve in little layman language.

Rajat

Quoted 41 lines · Manish Katiyar
Quoted 22 lines · Sebastian Pipping
Quoted 21 lines · Rajat Sharma

Hi Sebastian,

you guess for ELF header seems to be valid to me. When executables or binaries are loaded memory, it is done through mmap call to the file, and to understand what file is and what binary handler in kernel can handle its section, kernel needs to know its header first, which is within the first page of the header with fixed location for a magic number (identifier for binary handler e.g. ELF handler which further loads its other sections by reading section table). Note that there are multiple binary format handles within the kernel e.g. ELF, A.OUT which are tried sequentially to identify the file format.

From the file system perspective, mmap does not use vfs_read or vfs_write calls at all, thats why you don't see them. It directly works on address space operations of an inode (file) to populate data in page-cache. For a mmapped region, if you don't see a page in memory, kenel page faults and tries to read-in the page using readpage method of address_space_operations. Similarly when you modify a page, writepage method is called, but since executables are accessed read-only, you won't see writepage method getting called either.

Quoted 7 lines
Quoted 1 line

Hope this makes it clearer.

Excellent, thank you! I find calls to readpages on file /lib/libc-2.11.2.so now. That may be the missing reads.

Any ideas how get offset and length (like with vfs_read) for a certain page passed to readpage(file, page) ?

Add your stats hooks to do_mmap or mmap_region() ???

--
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110107/2216691d/attachment-0001.html

#7 lore raw
In reply to Rajat Sharma
Quoted 8 lines · Rajat Sharma

so, suitable position is to add hooks on readpage a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child after nfs_readdir is

finished.

I see.

Quoted 7 lines

As far as offset and length of I/O are concerned, page->index gives you its index in the page cache which in turn is equivalent to file offset (page->index << PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will be inode->i_size & ~PAGE_MASK, otherwise it can be PAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.

That sounds great. I have added these lines of code to read_pages() of mm/readahead.c:

================================================
{
struct page *page = list_to_page(pages);

  for (page_idx = 0; page_idx < nr_pages;
      page_idx++,
      page = list_to_page(page->lru.next)) {
    pgoff_t offset = 0;
    pgoff_t len = PAGE_SIZE;

    if (page) {
      offset = page->index << PAGE_SHIFT;

      if (page->mapping && page->mapping->host) {
        struct inode *inode = page->mapping->host;
        len = inode->i_size & ~PAGE_MASK;
      }
    }

    printk(KERN_DEBUG "VFS: BODY at 1 read_pages(...)["
        "page_idx=%u, offset=%lu, len=%lu, page_size=%lu]",
        page_idx, offset, len, PAGE_SIZE);

}
}
================================================

Suprisingly I never get len != PAGE_SIZE, i.e. partial pages. Also, in a list of pages given, the offsets are all the same, all 0 in this case:

================================================

[ 2811.993564] VFS: BODY at 1 read_pages(...)[page_idx=0, offset=0,
len=4096, page_size=4096]
[ 2811.996456] VFS: BODY at 1 read_pages(...)[page_idx=1, offset=0,
len=4096, page_size=4096]
[ 2811.999305] VFS: BODY at 1 read_pages(...)[page_idx=2, offset=0,
len=4096, page_size=4096]
[ 2812.002152] VFS: BODY at 1 read_pages(...)[page_idx=3, offset=0,

len=4096, page_size=4096]
================================================

Is page = list_to_page(page->lru.next) the correct way of walking pages? I am wondering how the main loop of read_pages()

	for (page_idx = 0; page_idx < nr_pages; page_idx++) {
		struct page *page = list_to_page(pages);
		[..]
	}

really iterates, if neither the address of pages changes in the loop nor page_idx is passed to anything. I would think the loop is actually accessing the same page again and again, but somehow my own

page = list_to_page(page->lru.next)

above doesn't seem to do better. Any insights?

Thanks!

Quoted 2 lines

Having said above, it will still be better if you can state what you want to achieve in little layman language.

I want to trace down all reads and writes to the file system including filename, offset and byte count into a log that I can reply inside a simulator.

Best,

Sebastian

#8 lore raw

Hi Sebastian,

read_pages is a small piece of code, please paste the complete function with your changes over here, I am sure there must be some minor bug otherwise page->lru.next looks fine to me.

Quoted 10 lines

I am wondering how the main loop of read_pages()

       for (page_idx = 0; page_idx < nr_pages; page_idx++) {
               struct page *page = list_to_page(pages);
               [..]
       }

really iterates, if neither the address of pages changes in the loop nor page_idx is passed to anything. I would think the loop is actually accessing the same page again and again, but somehow my own

No its not iterating same page, you missed the call to list_del(&page->lru) in the main loop. Remember it removes the first element from the head of the list (as this page is now responsibility of page cache) and next element automatically becomes head of the list ie. list_head *pages.

Quoted 4 lines

I want to trace down all reads and writes to the file system including filename, offset and byte count into a log that I can reply inside a simulator.

for filename you have to track open calls to file system but remember for resolving the complete path name, you have to get into depths of namespace and play with vfsmount data structure.

Rajat

Quoted 98 lines · Sebastian Pipping
Quoted 8 lines · Rajat Sharma

so, suitable position is to add hooks on readpage a_op. And of-course for doing that, you may have to capture various path thorugh which inode can come in memory, e.g. lookup and create directory inode operation (for regular files). For your worst nightmare, NFS implements its readdir with an additional feature with v3 protocol called READDIR PLUS, which not only gives you name of children of a directory, but also initializes their inodes in memory, so you may have to hook readdir as well and trap aops of all regular file child after nfs_readdir is

finished.

I see.

Quoted 7 lines

As far as offset and length of I/O are concerned, page->index gives you its index in the page cache which in turn is equivalent to file offset (page->index << PAGE_SHIFT). readpage is invoked to bring in complete page in memory. It may so happen that page is a partial page (e.g. last page of file), in that case your I/O lenght will be inode->i_size & ~PAGE_MASK, otherwise it can be PAGE_SIZE. don't worry about file wholes, that is taken care by filesystem's original readpage method.

That sounds great. ?I have added these lines of code to read_pages() of mm/readahead.c:

================================================
{
?struct page *page = list_to_page(pages);
?for (page_idx = 0; page_idx < nr_pages;
? ? ?page_idx++,
? ? ?page = list_to_page(page->lru.next)) {
? ?pgoff_t offset = 0;
? ?pgoff_t len = PAGE_SIZE;

? ?if (page) {
? ? ?offset = page->index << PAGE_SHIFT;

? ? ?if (page->mapping && page->mapping->host) {
? ? ? ?struct inode *inode = page->mapping->host;
? ? ? ?len = inode->i_size & ~PAGE_MASK;
? ? ?}
? ?}

? ?printk(KERN_DEBUG "VFS: BODY at 1 read_pages(...)[" ? ? ? ?"page_idx=%u, offset=%lu, len=%lu, page_size=%lu]", ? ? ? ?page_idx, offset, len, PAGE_SIZE);
?}
}
================================================

Suprisingly I never get len != PAGE_SIZE, i.e. partial pages. Also, in a list of pages given, the offsets are all the same, all 0 in this case:

================================================

[ 2811.993564] VFS: BODY at 1 read_pages(...)[page_idx=0, offset=0,
len=4096, page_size=4096]
[ 2811.996456] VFS: BODY at 1 read_pages(...)[page_idx=1, offset=0,
len=4096, page_size=4096]
[ 2811.999305] VFS: BODY at 1 read_pages(...)[page_idx=2, offset=0,
len=4096, page_size=4096]
[ 2812.002152] VFS: BODY at 1 read_pages(...)[page_idx=3, offset=0,

len=4096, page_size=4096]
================================================

Is ?page = list_to_page(page->lru.next) ?the correct way of walking pages? ?I am wondering how the main loop of read_pages()

? ? ? ?for (page_idx = 0; page_idx < nr_pages; page_idx++) { ? ? ? ? ? ? ? ?struct page *page = list_to_page(pages); ? ? ? ? ? ? ? ?[..]
? ? ? ?}

really iterates, if neither the address of pages changes in the loop nor page_idx is passed to anything. ?I would think the loop is actually accessing the same page again and again, but somehow my own

?page = list_to_page(page->lru.next)

above doesn't seem to do better. ?Any insights?

Thanks!

Quoted 2 lines

Having said above, it will still be better if you can state what you want to achieve in little layman language.

I want to trace down all reads and writes to the file system including filename, offset and byte count into a log that I can reply inside a simulator.

Best,

Sebastian