Skip to contentSkip to conversation

mmap vs mtime in 2.6.26

DISCUSSION Ferenc Wagner 3 participants 11 messages May 8, 2009 – Oct 15, 2009

lore
Share & tools
lore
#0 lore raw

Hi,

I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny). It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch
http://lkml.org/lkml/2008/1/22/370 is present in my kernel. Still, the test program (version 3 from the bug report) gives failures:

$ ./mmap_test junk
Modifying junk...
Flushing data using sync()...
Failure: time not changed.
Not modifying junk...
Flushing data using msync()...
Success: time not changed.
Not modifying junk...
Flushing data using fsync()...
Success: time not changed.
Modifying junk...
Flushing data using msync()...
Failure: time not changed.
Modifying junk...
Flushing data using fsync()...
Failure: time not changed.

Is this as expected? I reckon the bug wasn't solved fully, but it seems to be worse that that.

I don't know if it's related or not, but another anomaly appeared after the upgrade: minutes-long remount times. The machine always was under a fair load (10), but mount /usr -oremount,rw never took noticeable time. The IO load pushes two other filesystems. Now strace -tt shows:

[...]
01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory) 01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0 01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
[...]

After this, remount ro and rw again are quick for some time.

I can not easily test newer kernels in this setup, so thought to ask around before delving into it further. Is this a known problem perhaps? Does it ring any bells? If already fixed, can the fix be backported to 2.6.26 with reasonable effort? If not, how can I best help investigate this?

Signature · 2 lines
Thanks,
Feri.
#1 lore raw

mmap vs mtime in 2.6.26 and up

Hi,

I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck). It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels. Still, the test program (version 3 from the bug report) gives failures:

$ ./mmap_test junk
Modifying junk...
Flushing data using sync()...
Failure: time not changed.
Not modifying junk...
Flushing data using msync()...
Success: time not changed.
Not modifying junk...
Flushing data using fsync()...
Success: time not changed.
Modifying junk...
Flushing data using msync()...
Failure: time not changed.
Modifying junk...
Flushing data using fsync()...
Failure: time not changed.

Is this as expected? I reckon the bug wasn't solved fully, but it seems to be worse that that.

I don't know if it's related or not, but another anomaly appeared after the upgrade: minutes-long remount times. The machine always was under a fair load (10), but mount /usr -oremount,rw never took noticeable time. The IO load pushes two other filesystems. Now strace -tt shows:

[...]
01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory) 01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0 01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
[...]

After this, remount ro and rw again are quick for some time. (I couldn't test this part with 2.6.30-rc5, but again, this may be totally unrelated.)

This bug is quite serious in our environment, as it thwarts the backup software. The file system is xfs; rw,nosuid,nodev,nobarrier,noquota.

Signature · 2 lines
Regards,
Feri.
#2 lore raw

Re: mmap vs mtime in 2.6.26 and up

(Adding Christoph Hellwig, xfs list, Andrew Morton to cc:)

Quoted 27 lines · Ferenc Wagner

Hi,

I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck).  It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels.  Still, the test program (version 3 from the bug report) gives failures:

$ ./mmap_test junk
Modifying junk...
Flushing data using sync()...
Failure: time not changed.
Not modifying junk...
Flushing data using msync()...
Success: time not changed.
Not modifying junk...
Flushing data using fsync()...
Success: time not changed.
Modifying junk...
Flushing data using msync()...
Failure: time not changed.
Modifying junk...
Flushing data using fsync()...
Failure: time not changed.
Quoted 2 lines

Is this as expected?  I reckon the bug wasn't solved fully, but it seems to be worse that that.

I don't think that's expected. Further, the test program (v3) from that bugzilla http://bugzilla.kernel.org/show_bug.cgi?id=2645 works for me under ext3, so perhaps this is an xfs-specific issue.

Quoted 17 lines

I don't know if it's related or not, but another anomaly appeared after the upgrade: minutes-long remount times.  The machine always was under a fair load (10), but mount /usr -oremount,rw never took noticeable time.  The IO load pushes two other filesystems.  Now strace -tt shows:

[...]
01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory) 01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0 01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
[...]

After this, remount ro and rw again are quick for some time. (I couldn't test this part with 2.6.30-rc5, but again, this may be totally unrelated.)

Quoted 5 lines

This bug is quite serious in our environment, as it thwarts the backup software.  The file system is xfs; rw,nosuid,nodev,nobarrier,noquota. --
Regards,
Feri.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw

Re: mmap vs mtime in 2.6.26 and up

Quoted 7 lines · Ray Lee
Quoted 7 lines

I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck). ??It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels. ??Still, the test program (version 3 from the bug report) gives failures:

The problem is pretty simple. do_wp_page and __do_fault use file_update_time to update ctime and mtime. But this function is only a helper for simply filesystems that have a binary inode dirty/non dirty state and keep the m/ctime purely in the Linux inode. It must not be called from generic code as more complex filesystems need a notification through ->setattr to update the timestamps. This will also affect other filesystems like ubifs. I'm not entirely sure why it ever worked before, we must have picked up those c/mtime updates by accident somehow.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

Re: mmap vs mtime in 2.6.26 and up

Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Ray Lee
Quoted 7 lines

I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck). ??It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels. ??Still, the test program (version 3 from the bug report) gives failures:

Quoted 9 lines

The problem is pretty simple. do_wp_page and __do_fault use file_update_time to update ctime and mtime. But this function is only a helper for simply filesystems that have a binary inode dirty/non dirty state and keep the m/ctime purely in the Linux inode. It must not be called from generic code as more complex filesystems need a notification through ->setattr to update the timestamps. This will also affect other filesystems like ubifs. I'm not entirely sure why it ever worked before, we must have picked up those c/mtime updates by accident somehow.

Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?

Signature · 6 lines
Feri.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

Re: mmap vs mtime in 2.6.26 and up

Quoted 3 lines · Ferenc Wagner

Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?

Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw

Re: mmap vs mtime in 2.6.26 and up

Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Ferenc Wagner

Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?

Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.

Doing this correctly in the framework of the current codee is unfortunately not so easy, as calling ->setattr requires taking i_mutex which we can't in the pagefaul path.

To fix this properly we need to actually update the timestamps during msync and co as done by the patches from Miklos:

	http://lkml.org/lkml/2007/2/28/166

and Peter:

	http://lkml.org/lkml/2006/5/31/176

----- End forwarded message -----

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw

Re: mmap vs mtime in 2.6.26 and up

Quoted 13 lines · Christoph Hellwig
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Ferenc Wagner

Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?

Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.

Doing this correctly in the framework of the current codee is unfortunately not so easy, as calling ->setattr requires taking i_mutex which we can't in the pagefaul path.

Quoted 5 lines

To fix this properly we need to actually update the timestamps during msync and co as done by the patches from Miklos:

	http://lkml.org/lkml/2007/2/28/166
and Peter:
	http://lkml.org/lkml/2006/5/31/176

Hi Christoph,

http://bugzilla.kernel.org/show_bug.cgi?id=2645#c53 shows that Anton doesn't quite agree with you on this. I really can't tell, would you (or anybody from the accused XFS community) please comment? Or did you perhaps fix it meanwhile? I can't easily test never kernels, but I will if there's some chance.

Signature · 7 lines
Thanks,
Feri.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw

Re: mmap vs mtime in 2.6.26 and up

In reply to Christoph Hellwig
Quoted 23 lines · Ferenc Wagner
Quoted 15 lines · Christoph Hellwig
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Ferenc Wagner

Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?

Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.

Doing this correctly in the framework of the current codee is unfortunately not so easy, as calling ->setattr requires taking i_mutex which we can't in the pagefaul path.

Quoted 5 lines

To fix this properly we need to actually update the timestamps during msync and co as done by the patches from Miklos:

	http://lkml.org/lkml/2007/2/28/166
and Peter:
	http://lkml.org/lkml/2006/5/31/176
Quoted 5 lines

http://bugzilla.kernel.org/show_bug.cgi?id=2645#c53 shows that Anton doesn't quite agree with you on this. I really can't tell, would you (or anybody from the accused XFS community) please comment? Or did you perhaps fix it meanwhile? I can't easily test newer kernels, but I will if there's some chance.

I added some fresh test results to
http://bugzilla.kernel.org/show_bug.cgi?id=2645. In short, under 2.6.30-rc8 the test program reports full failure on XFS and RAMFS, full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:

$ ./mmap /dev/shm/testfile 
Modifying /dev/shm/testfile...
Flushing data using sync()...
Failure: time not changed.
Not modifying /dev/shm/testfile...
Flushing data using msync()...
Success: time not changed.
Not modifying /dev/shm/testfile...
Flushing data using fsync()...
Success: time not changed.
Modifying /dev/shm/testfile...
Flushing data using msync()...
Failure: time not changed.
Modifying /dev/shm/testfile...
Flushing data using fsync()...
Failure: time not changed.

This doesn't look like az XFS-specific issue, but XFS is definitely affected. Anybody's got an idea how to tackle this?

Signature · 7 lines
Regards,
Feri.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw

Re: mmap vs mtime in 2.6.26 and up

Quoted 4 lines · Ferenc Wagner

I added some fresh test results to
http://bugzilla.kernel.org/show_bug.cgi?id=2645. In short, under 2.6.30-rc8 the test program reports full failure on XFS and RAMFS, full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:

There's a patch queued up for inclusion in 2.6.32 to work around this issue in XFS. The lack of a proper callout from the VFS still makes this a much less than ideal solution.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw

Re: mmap vs mtime in 2.6.26 and up

Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Ferenc Wagner

I added some fresh test results to
http://bugzilla.kernel.org/show_bug.cgi?id=2645. In short, under 2.6.30-rc8 the test program reports full failure on XFS and RAMFS, full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:

There's a patch queued up for inclusion in 2.6.32 to work around this issue in XFS. The lack of a proper callout from the VFS still makes this a much less than ideal solution.

Great, anyway! Somehow I managed to gloss over this mail until now, but happened to test 2.6.32-rc4 and found the issue fixed. At least for XFS, as my latest addition to the bug report details it. Thank you very much for taking care of this issue!

Signature · 7 lines
Regards,
Feri.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs