mmap vs mtime in 2.6.26
Hi,
I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny). It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch
http://lkml.org/lkml/2008/1/22/370 is present in my kernel. Still, the test program (version 3 from the bug report) gives failures:
$ ./mmap_test junk Modifying junk... Flushing data using sync()... Failure: time not changed. Not modifying junk... Flushing data using msync()... Success: time not changed. Not modifying junk... Flushing data using fsync()... Success: time not changed. Modifying junk... Flushing data using msync()... Failure: time not changed. Modifying junk... Flushing data using fsync()... Failure: time not changed.
Is this as expected? I reckon the bug wasn't solved fully, but it seems to be worse that that.
I don't know if it's related or not, but another anomaly appeared after the upgrade: minutes-long remount times. The machine always was under a fair load (10), but mount /usr -oremount,rw never took noticeable time. The IO load pushes two other filesystems. Now strace -tt shows:
[...]
01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory) 01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0 01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
[...]
After this, remount ro and rw again are quick for some time.
I can not easily test newer kernels in this setup, so thought to ask around before delving into it further. Is this a known problem perhaps? Does it ring any bells? If already fixed, can the fix be backported to 2.6.26 with reasonable effort? If not, how can I best help investigate this?
Signature · 2 lines
Thanks, Feri.
mmap vs mtime in 2.6.26 and up
Hi,
I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck). It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels. Still, the test program (version 3 from the bug report) gives failures:
$ ./mmap_test junk Modifying junk... Flushing data using sync()... Failure: time not changed. Not modifying junk... Flushing data using msync()... Success: time not changed. Not modifying junk... Flushing data using fsync()... Success: time not changed. Modifying junk... Flushing data using msync()... Failure: time not changed. Modifying junk... Flushing data using fsync()... Failure: time not changed.
Is this as expected? I reckon the bug wasn't solved fully, but it seems to be worse that that.
I don't know if it's related or not, but another anomaly appeared after the upgrade: minutes-long remount times. The machine always was under a fair load (10), but mount /usr -oremount,rw never took noticeable time. The IO load pushes two other filesystems. Now strace -tt shows:
[...]
01:24:08.463813 stat64("/sbin/mount.xfs", 0xbfa40eb0) = -1 ENOENT (No such file or directory) 01:24:08.463904 mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL) = 0 01:26:58.705387 readlink("/dev", 0xbfa3ef3b, 4096) = -1 EINVAL (Invalid argument)
[...]
After this, remount ro and rw again are quick for some time. (I couldn't test this part with 2.6.30-rc5, but again, this may be totally unrelated.)
This bug is quite serious in our environment, as it thwarts the backup software. The file system is xfs; rw,nosuid,nodev,nobarrier,noquota.
Signature · 2 lines
Regards, Feri.
Re: mmap vs mtime in 2.6.26 and up
(Adding Christoph Hellwig, xfs list, Andrew Morton to cc:)
Quoted 27 lines · Ferenc Wagner
Hi,
I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck). It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels. Still, the test program (version 3 from the bug report) gives failures:
$ ./mmap_test junk Modifying junk... Flushing data using sync()... Failure: time not changed. Not modifying junk... Flushing data using msync()... Success: time not changed. Not modifying junk... Flushing data using fsync()... Success: time not changed. Modifying junk... Flushing data using msync()... Failure: time not changed. Modifying junk... Flushing data using fsync()... Failure: time not changed.
Quoted 2 lines
Is this as expected? I reckon the bug wasn't solved fully, but it seems to be worse that that.
I don't think that's expected. Further, the test program (v3) from that bugzilla http://bugzilla.kernel.org/show_bug.cgi?id=2645 works for me under ext3, so perhaps this is an xfs-specific issue.
Quoted 17 lines
I don't know if it's related or not, but another anomaly appeared after the upgrade: minutes-long remount times. The machine always was under a fair load (10), but mount /usr -oremount,rw never took noticeable time. The IO load pushes two other filesystems. Now strace -tt shows:
[...]
01:24:08.463813stat64("/sbin/mount.xfs", 0xbfa40eb0)= -1 ENOENT (No such file or directory) 01:24:08.463904mount("/dev/mapper/noc7-usr", "/usr", 0x894a2c8, MS_MGC_VAL|MS_NODEV|MS_REMOUNT, NULL)= 0 01:26:58.705387readlink("/dev", 0xbfa3ef3b, 4096)= -1 EINVAL (Invalid argument)
[...]After this, remount ro and rw again are quick for some time. (I couldn't test this part with 2.6.30-rc5, but again, this may be totally unrelated.)
Quoted 5 lines
This bug is quite serious in our environment, as it thwarts the backup software. The file system is xfs; rw,nosuid,nodev,nobarrier,noquota. --
Regards,
Feri.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
Quoted 7 lines · Ray Lee
Quoted 7 lines
I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck). ??It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels. ??Still, the test program (version 3 from the bug report) gives failures:
The problem is pretty simple. do_wp_page and __do_fault use file_update_time to update ctime and mtime. But this function is only a helper for simply filesystems that have a binary inode dirty/non dirty state and keep the m/ctime purely in the Linux inode. It must not be called from generic code as more complex filesystems need a notification through ->setattr to update the timestamps. This will also affect other filesystems like ubifs. I'm not entirely sure why it ever worked before, we must have picked up those c/mtime updates by accident somehow.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Ray Lee
Quoted 7 lines
I've noticed that the last modification times of our RRD files got stuck after upgrading from 2.6.24 to 2.6.26 (Debian Etch -> Lenny; I also tested with 2.6.30-rc5, they are still stuck). ??It has some literature, most notably kernel bug #2645, but that's closed long ago and the resulting patch http://lkml.org/lkml/2008/1/22/370 is present in my kernels. ??Still, the test program (version 3 from the bug report) gives failures:
Quoted 9 lines
The problem is pretty simple.
do_wp_pageand__do_faultusefile_update_timeto update ctime and mtime. But this function is only a helper for simply filesystems that have a binary inode dirty/non dirty state and keep the m/ctime purely in the Linux inode. It must not be called from generic code as more complex filesystems need a notification through ->setattr to update the timestamps. This will also affect other filesystems like ubifs. I'm not entirely sure why it ever worked before, we must have picked up those c/mtime updates by accident somehow.
Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?
Signature · 6 lines
Feri. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
Quoted 3 lines · Ferenc Wagner
Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?
Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Ferenc Wagner
Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?
Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.
Doing this correctly in the framework of the current codee is unfortunately not so easy, as calling ->setattr requires taking i_mutex which we can't in the pagefaul path.
To fix this properly we need to actually update the timestamps during msync and co as done by the patches from Miklos:
http://lkml.org/lkml/2007/2/28/166
and Peter:
http://lkml.org/lkml/2006/5/31/176
----- End forwarded message -----
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
Quoted 13 lines · Christoph Hellwig
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Ferenc Wagner
Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?
Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.
Doing this correctly in the framework of the current codee is unfortunately not so easy, as calling ->setattr requires taking
i_mutexwhich we can't in the pagefaul path.
Quoted 5 lines
To fix this properly we need to actually update the timestamps during msync and co as done by the patches from Miklos:
http://lkml.org/lkml/2007/2/28/166 and Peter: http://lkml.org/lkml/2006/5/31/176
Hi Christoph,
http://bugzilla.kernel.org/show_bug.cgi?id=2645#c53 shows that Anton doesn't quite agree with you on this. I really can't tell, would you (or anybody from the accused XFS community) please comment? Or did you perhaps fix it meanwhile? I can't easily test never kernels, but I will if there's some chance.
Signature · 7 lines
Thanks, Feri. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
In reply to Christoph HellwigQuoted 23 lines · Ferenc Wagner
Quoted 15 lines · Christoph Hellwig
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Ferenc Wagner
Thanks for the analysis. Unfortunately I don't nearly know enough to work on this issue, but would like to track it as it affects our backup system. So, shouldn't #2645 be reopened again?
Yes, definitively as the current "fix" is incorrected. I'll try to cook up a correct version once I get some time.
Doing this correctly in the framework of the current codee is unfortunately not so easy, as calling ->setattr requires taking
i_mutexwhich we can't in the pagefaul path.Quoted 5 lines
To fix this properly we need to actually update the timestamps during msync and co as done by the patches from Miklos:
http://lkml.org/lkml/2007/2/28/166 and Peter: http://lkml.org/lkml/2006/5/31/176
Quoted 5 lines
http://bugzilla.kernel.org/show_bug.cgi?id=2645#c53 shows that Anton doesn't quite agree with you on this. I really can't tell, would you (or anybody from the accused XFS community) please comment? Or did you perhaps fix it meanwhile? I can't easily test newer kernels, but I will if there's some chance.
I added some fresh test results to
http://bugzilla.kernel.org/show_bug.cgi?id=2645. In short, under 2.6.30-rc8 the test program reports full failure on XFS and RAMFS, full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:
$ ./mmap /dev/shm/testfile Modifying /dev/shm/testfile... Flushing data using sync()... Failure: time not changed. Not modifying /dev/shm/testfile... Flushing data using msync()... Success: time not changed. Not modifying /dev/shm/testfile... Flushing data using fsync()... Success: time not changed. Modifying /dev/shm/testfile... Flushing data using msync()... Failure: time not changed. Modifying /dev/shm/testfile... Flushing data using fsync()... Failure: time not changed.
This doesn't look like az XFS-specific issue, but XFS is definitely affected. Anybody's got an idea how to tackle this?
Signature · 7 lines
Regards, Feri. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
Quoted 4 lines · Ferenc Wagner
I added some fresh test results to
http://bugzilla.kernel.org/show_bug.cgi?id=2645. In short, under 2.6.30-rc8 the test program reports full failure on XFS and RAMFS, full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:
There's a patch queued up for inclusion in 2.6.32 to work around this issue in XFS. The lack of a proper callout from the VFS still makes this a much less than ideal solution.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: mmap vs mtime in 2.6.26 and up
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Ferenc Wagner
I added some fresh test results to
http://bugzilla.kernel.org/show_bug.cgi?id=2645. In short, under 2.6.30-rc8 the test program reports full failure on XFS and RAMFS, full success on EXT3, VFAT and ReiserFS, and mixed results on TMPFS:There's a patch queued up for inclusion in 2.6.32 to work around this issue in XFS. The lack of a proper callout from the VFS still makes this a much less than ideal solution.
Great, anyway! Somehow I managed to gloss over this mail until now, but happened to test 2.6.32-rc4 and found the issue fixed. At least for XFS, as my latest addition to the bug report details it. Thank you very much for taking care of this issue!
Signature · 7 lines
Regards, Feri. _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs