Page cache corruption when creating a snapshot
We have observed an apparent kernel memory corruption bug when creating an LVM snapshot. This has been reproduced on two different machines, so it does not appear to be a memory hardware issue.
The reproduction recipe looks like:
rm -rf /tmp/test mkdir /tmp/test # Put around 60MB of files into /tmp/test find /tmp/test -type f | xargs md5sum > /tmp/sum.pre lvcreate --size 2G --snapshot /dev/dink/gutsy-i386-sbuild --name testsnapshot find /tmp/test -type f | xargs md5sum > /tmp/sum.post lvremove -f /dev/dink/testsnapshot diff -u /tmp/sum.pre /tmp/sum.post
Line 5 naturally needs to be adjusted for the LVM configuration of the test machine. On my machine, /dev/dink/gutsy-i386-sbuild is an unmounted 2GB logical volume containing a build chroot; it lives in a different volume group from the one /tmp's filesystem is located in.
Not all of the time, but some of the time when I do this, one of the files in /tmp/test will have a different md5sum. It's always a one-byte difference at offset 156 within a 1K block (but a different block each time), and the incorrect value of that byte is always one less than the correct value. For example:
@@ -471431,7 +471431,7 @@
0731860: 4d46 6ae3 0252 6864 e634 15eb 7ac1 f0ee MFj..Rhd.4..z...
0731870: 9f2b 8d82 33e3 138b 31a2 8da5 4594 5648 .+..3...1...E.VH
0731880: 74fd 00e0 bc48 fe09 d557 f501 70a8 7dfd t....H...W..p.}.
-0731890: ea8f 5010 b963 e2ec 7b84 8ef7 e851 fdfa ..P..c..{....Q..
+0731890: ea8f 5010 b963 e2ec 7b84 8ef7 e751 fdfa ..P..c..{....Q..
07318a0: 6031 670b cd54 fe01 20d6 f3fb c662 dfc3 `1g..T.. ....b..
07318b0: 7605 acd2 1be6 3fee 54ff e15b bc60 77fa v.....?.T..[.`w.
07318c0: 368e 99f9 60a0 a1a2 fbdf ef0d 4bca a201 6...`.......K...If the machine is rebooted (after moving /tmp/test to another location so it doesn't get blown away by init scripts), the apparently modified file reverts to the correct contents. Thus, the issue appears to be page cache corruption, not actual filesystem corruption.
Version information:
root@linux-build-10:~# uname -a Linux linux-build-10 2.6.22-14-server #1 SMP Thu Jan 31 23:57:25 UTC 2008 x86_64 GNU/Linux root@linux-build-10:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 7.10 Release: 7.10 Codename: gutsy root@linux-build-10:~# dpkg -s lvm2 | grep Version Version: 2.02.26-1ubuntu4 root@linux-build-10:~# pvscan PV /dev/sdb VG dink lvm2 [136.73 GB / 110.73 GB free] PV /dev/sda5 VG LINUX-BUILD-10.mit.edu lvm2 [68.12 GB / 0 free] Total: 2 [204.85 GB] / in use: 2 [204.85 GB] / in no VG: 0 [0 ] root@linux-build-10:~# vgscan Reading all physical volumes. This may take a while... Found volume group "dink" using metadata type lvm2 Found volume group "LINUX-BUILD-10.mit.edu" using metadata type lvm2
(I sent a slightly different variant of this yesterday without subscribing to the list, which I think was black-holed. Apologies if this shows up twice. Also, I filed a similar bug report with Ubuntu which can be seen at:
https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/196784
)
Quoted 7 lines · On Fri, Feb 29, 2008 at 12:32:41PM -0500, [email protected] wrote:
The reproduction recipe looks like:
rm -rf/tmp/test
mkdir/tmp/test
# Put around 60MB of files into/tmp/test
find/tmp/test-type f | xargs md5sum >/tmp/sum.pre
lvcreate --size 2G --snapshot/dev/dink/gutsy-i386-sbuild--name testsnapshot find/tmp/test-type f | xargs md5sum >/tmp/sum.post
Can you do that twice?
find /tmp/test -type f | xargs md5sum > /tmp/sum.post2
and check the two post files are the same?
And add some syncs/blockdev --flushbufs at different places in the script to see if you can make it go away.
Alasdair
Signature · 1 line
[email protected]
Quoted 9 lines · Alasdair G Kergon
Quoted 7 lines · On Fri, Feb 29, 2008 at 12:32:41PM -0500, [email protected] wrote:
The reproduction recipe looks like:
rm -rf/tmp/test
mkdir/tmp/test
# Put around 60MB of files into/tmp/test
find/tmp/test-type f | xargs md5sum >/tmp/sum.pre
lvcreate --size 2G --snapshot/dev/dink/gutsy-i386-sbuild--name testsnapshot find/tmp/test-type f | xargs md5sum >/tmp/sum.post
Quoted 3 lines
Can you do that twice?
find/tmp/test-type f | xargs md5sum >/tmp/sum.post2
and check the two post files are the same?
In three reproductions of the page cache corruption, sum.post2 was always the same as sum.post.
In my experiences with this problem in general, the page cache corruption is not particularly transient; once it happens, the file continues to appear modified (with the same incorrect contents) for the indefinite future, until the machine is rebooted.
Quoted 2 lines
And add some syncs/blockdev --flushbufs at different places in the script to see if you can make it go away.
Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot. (I could churn the page cache into oblivion with a bazillion reads of different files, but I'd have no way of knowing when I had succeeded in reusing the corrupted cache page.)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 8 lines · Greg Hudson
Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot. (I could churn the page cache into oblivion with a bazillion reads of different files, but I'd have no way of knowing when I had succeeded in reusing the corrupted cache page.)
You should be able to achieve that via the /proc/sys/vm/drop_caches sysctl:
Documentation/filesystems/proc.txt:
drop_caches
- -----------
Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.
To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches
I guess a 1 would be a good first try & if that doesn't clear it, a 3 should force the entire inode to be re-read from disk.
Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFHyFia6YSQoMYUY94RArInAJ9dRVLCGYXcllL4RDz4pei4qGnUVQCg4LL1 d22/7jgF5zxsGnFPeCfpkwA=
=v99d
-----END PGP SIGNATURE-----
Quoted 9 lines · Bryn M. Reeves
Quoted 5 lines · Greg Hudson
Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot.
To free pagecache:
echo 1 >/proc/sys/vm/drop_caches
Thank you! Verified that freeing the page cache causes the apparently-corrupted file's md5sum to revert to the correct value:
[Abbreviated diff output:]
--- /tmp/sum.pre 2008-02-29 14:30:29.000000000 -0500
+++ /tmp/sum.post 2008-02-29 14:30:32.000000000 -0500
-3e497fef20ce899c5621dcfbcbfec9a3 /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp +e9725c38b630515dc304a26bde28fa51 /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp
root@linux-build-10:~# md5sum /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp e9725c38b630515dc304a26bde28fa51 /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp root@linux-build-10:~# echo 1 > /proc/sys/vm/drop_caches root@linux-build-10:~# md5sum /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp 3e497fef20ce899c5621dcfbcbfec9a3 /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp
Quoted 5 lines · Greg Hudson
Quoted 1 line · Bryn M. Reeves
Greg Hudson wrote:
Thank you! Verified that freeing the page cache causes the apparently-corrupted file's md5sum to revert to the correct value:
I notice this is ubuntu: Can you eliminate the distribution, by reproducing on a different distribution, or using a current upstream development kernel like 2.6.25-rc3?
What is /tmp ?
Alasdair
Signature · 1 line
[email protected]
Quoted 3 lines · Alasdair G Kergon
I notice this is ubuntu: Can you eliminate the distribution, by reproducing on a different distribution, or using a current upstream development kernel like 2.6.25-rc3?
Apologies for the delay. I was unable to reproduce this bug in many trials on 2.6.25-rc3, so it may have been either introduced by Ubuntu in gutsy or (more likely) fixed in the mainline kernel source since 2.6.22.
Anders Kaseorg has also seen this problem on Gutsy but was unable to reproduce it on Hardy, which lends credence to the second theory.
Quoted 1 line
What is /tmp ?
A regular old directory on the root fileystem, which is ext3fs on a different LVM volume group on a different disk from the volume group containing the LV's I'm snapshotting.
Quoted 22 lines · Greg Hudson [[email protected]] wrote:
Quoted 9 lines · Alasdair G Kergon
Quoted 7 lines · On Fri, Feb 29, 2008 at 12:32:41PM -0500, [email protected] wrote:
The reproduction recipe looks like:
rm -rf/tmp/test
mkdir/tmp/test
# Put around 60MB of files into/tmp/test
find/tmp/test-type f | xargs md5sum >/tmp/sum.pre
lvcreate --size 2G --snapshot/dev/dink/gutsy-i386-sbuild--name testsnapshot find/tmp/test-type f | xargs md5sum >/tmp/sum.postQuoted 3 lines
Can you do that twice?
find/tmp/test-type f | xargs md5sum >/tmp/sum.post2
and check the two post files are the same?In three reproductions of the page cache corruption, sum.post2 was always the same as sum.post.
In my experiences with this problem in general, the page cache corruption is not particularly transient; once it happens, the file continues to appear modified (with the same incorrect contents) for the indefinite future, until the machine is rebooted.
Quoted 8 lines
Quoted 2 lines
And add some syncs/blockdev --flushbufs at different places in the script to see if you can make it go away.
Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot.
Not an expert on O_DIRECT, but it is supposed to read from the disk without creating page cache. I don't really know what it does if page cache exists! The "dd" command has O_DIRECT support and see if you notice any change with the corrupted file when you do "dd" with and without O_DIRECT.
--Malahal.
Quoted 4 lines · On Fri, 29 Feb 2008 [email protected] wrote:
rm -rf
/tmp/test
mkdir/tmp/test
# Put around 60MB of files into/tmp/testfind/tmp/test-type f | xargs md5sum >/tmp/sum.pre
Ran the last four lines several times on my Centos-5 system with /tmp and the snapshot VG on different volume groups. (/tmp on rootvg)
# du /tmp/test 84688 /tmp/test # lvs backvg2/DFL LV VG Attr LSize Origin Snap% Move Log Copy% DFL backvg2 -wi-a- 10.00G
Quoted 4 lines
lvcreate --size 2G --snapshot
/dev/backvg2/DFL--nameDFL_snapfind/tmp/test-type f | xargs md5sum >/tmp/sum.postlvremove -f/dev/backvg2/DFL_snap
diff -u/tmp/sum.pre/tmp/sum.
Doesn't seem to fail for us. (Sighs with relief.)
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.