Skip to contentSkip to conversation

Page cache corruption when creating a snapshot

linux-lvm [email protected] 5 participants 9 messages Feb 29, 2008 – Mar 4, 2008

lore
Share & tools
lore
#0 lore raw

We have observed an apparent kernel memory corruption bug when creating an LVM snapshot. This has been reproduced on two different machines, so it does not appear to be a memory hardware issue.

The reproduction recipe looks like:

  rm -rf /tmp/test
  mkdir /tmp/test
  # Put around 60MB of files into /tmp/test
  find /tmp/test -type f | xargs md5sum > /tmp/sum.pre
  lvcreate --size 2G --snapshot /dev/dink/gutsy-i386-sbuild --name testsnapshot
  find /tmp/test -type f | xargs md5sum > /tmp/sum.post
  lvremove -f /dev/dink/testsnapshot
  diff -u /tmp/sum.pre /tmp/sum.post

Line 5 naturally needs to be adjusted for the LVM configuration of the test machine. On my machine, /dev/dink/gutsy-i386-sbuild is an unmounted 2GB logical volume containing a build chroot; it lives in a different volume group from the one /tmp's filesystem is located in.

Not all of the time, but some of the time when I do this, one of the files in /tmp/test will have a different md5sum. It's always a one-byte difference at offset 156 within a 1K block (but a different block each time), and the incorrect value of that byte is always one less than the correct value. For example:

@@ -471431,7 +471431,7 @@

 0731860: 4d46 6ae3 0252 6864 e634 15eb 7ac1 f0ee  MFj..Rhd.4..z...
 0731870: 9f2b 8d82 33e3 138b 31a2 8da5 4594 5648  .+..3...1...E.VH
 0731880: 74fd 00e0 bc48 fe09 d557 f501 70a8 7dfd  t....H...W..p.}.
-0731890: ea8f 5010 b963 e2ec 7b84 8ef7 e851 fdfa  ..P..c..{....Q..
+0731890: ea8f 5010 b963 e2ec 7b84 8ef7 e751 fdfa  ..P..c..{....Q..
 07318a0: 6031 670b cd54 fe01 20d6 f3fb c662 dfc3  `1g..T.. ....b..
 07318b0: 7605 acd2 1be6 3fee 54ff e15b bc60 77fa  v.....?.T..[.`w.
 07318c0: 368e 99f9 60a0 a1a2 fbdf ef0d 4bca a201  6...`.......K...

If the machine is rebooted (after moving /tmp/test to another location so it doesn't get blown away by init scripts), the apparently modified file reverts to the correct contents. Thus, the issue appears to be page cache corruption, not actual filesystem corruption.

Version information:

root@linux-build-10:~# uname -a
Linux linux-build-10 2.6.22-14-server #1 SMP Thu Jan 31 23:57:25 UTC 2008 x86_64 GNU/Linux
root@linux-build-10:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 7.10
Release:        7.10
Codename:       gutsy
root@linux-build-10:~# dpkg -s lvm2 | grep Version
Version: 2.02.26-1ubuntu4
root@linux-build-10:~# pvscan
  PV /dev/sdb    VG dink                     lvm2 [136.73 GB / 110.73 GB free]
  PV /dev/sda5   VG LINUX-BUILD-10.mit.edu   lvm2 [68.12 GB / 0    free]
  Total: 2 [204.85 GB] / in use: 2 [204.85 GB] / in no VG: 0 [0   ]
root@linux-build-10:~# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "dink" using metadata type lvm2
  Found volume group "LINUX-BUILD-10.mit.edu" using metadata type lvm2

(I sent a slightly different variant of this yesterday without subscribing to the list, which I think was black-holed. Apologies if this shows up twice. Also, I filed a similar bug report with Ubuntu which can be seen at:
https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/196784
)

#1 lore raw
Quoted 7 lines · On Fri, Feb 29, 2008 at 12:32:41PM -0500, [email protected] wrote:

The reproduction recipe looks like:
rm -rf /tmp/test
mkdir /tmp/test
# Put around 60MB of files into /tmp/test
find /tmp/test -type f | xargs md5sum > /tmp/sum.pre
lvcreate --size 2G --snapshot /dev/dink/gutsy-i386-sbuild --name testsnapshot find /tmp/test -type f | xargs md5sum > /tmp/sum.post

Can you do that twice?
find /tmp/test -type f | xargs md5sum > /tmp/sum.post2
and check the two post files are the same?

And add some syncs/blockdev --flushbufs at different places in the script to see if you can make it go away.

Alasdair

Signature · 1 line
[email protected]
#2 lore raw
Quoted 9 lines · Alasdair G Kergon
Quoted 7 lines · On Fri, Feb 29, 2008 at 12:32:41PM -0500, [email protected] wrote:

The reproduction recipe looks like:
rm -rf /tmp/test
mkdir /tmp/test
# Put around 60MB of files into /tmp/test
find /tmp/test -type f | xargs md5sum > /tmp/sum.pre
lvcreate --size 2G --snapshot /dev/dink/gutsy-i386-sbuild --name testsnapshot find /tmp/test -type f | xargs md5sum > /tmp/sum.post

Quoted 3 lines

Can you do that twice?
find /tmp/test -type f | xargs md5sum > /tmp/sum.post2
and check the two post files are the same?

In three reproductions of the page cache corruption, sum.post2 was always the same as sum.post.

In my experiences with this problem in general, the page cache corruption is not particularly transient; once it happens, the file continues to appear modified (with the same incorrect contents) for the indefinite future, until the machine is rebooted.

Quoted 2 lines

And add some syncs/blockdev --flushbufs at different places in the script to see if you can make it go away.

Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot. (I could churn the page cache into oblivion with a bazillion reads of different files, but I'd have no way of knowing when I had succeeded in reusing the corrupted cache page.)

#3 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 8 lines · Greg Hudson

Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot. (I could churn the page cache into oblivion with a bazillion reads of different files, but I'd have no way of knowing when I had succeeded in reusing the corrupted cache page.)

You should be able to achieve that via the /proc/sys/vm/drop_caches sysctl:

Documentation/filesystems/proc.txt:

drop_caches
- -----------

Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

I guess a 1 would be a good first try & if that doesn't clear it, a 3 should force the entire inode to be re-read from disk.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFHyFia6YSQoMYUY94RArInAJ9dRVLCGYXcllL4RDz4pei4qGnUVQCg4LL1 d22/7jgF5zxsGnFPeCfpkwA=
=v99d
-----END PGP SIGNATURE-----

#4 lore raw
Quoted 9 lines · Bryn M. Reeves
Quoted 5 lines · Greg Hudson

Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot.

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches

Thank you! Verified that freeing the page cache causes the apparently-corrupted file's md5sum to revert to the correct value:

[Abbreviated diff output:]
--- /tmp/sum.pre 2008-02-29 14:30:29.000000000 -0500
+++ /tmp/sum.post 2008-02-29 14:30:32.000000000 -0500
-3e497fef20ce899c5621dcfbcbfec9a3 /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp +e9725c38b630515dc304a26bde28fa51 /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp

root@linux-build-10:~# md5sum /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp
e9725c38b630515dc304a26bde28fa51  /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp
root@linux-build-10:~# echo 1 > /proc/sys/vm/drop_caches
root@linux-build-10:~# md5sum /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp
3e497fef20ce899c5621dcfbcbfec9a3  /tmp/test/krb5-1.6.dfsg/src/windows/cns/kerbnet.hlp
#5 lore raw
Quoted 5 lines · Greg Hudson
Quoted 1 line · Bryn M. Reeves

Greg Hudson wrote:

Thank you! Verified that freeing the page cache causes the apparently-corrupted file's md5sum to revert to the correct value:

I notice this is ubuntu: Can you eliminate the distribution, by reproducing on a different distribution, or using a current upstream development kernel like 2.6.25-rc3?

What is /tmp ?

Alasdair

Signature · 1 line
[email protected]
#6 lore raw
Quoted 3 lines · Alasdair G Kergon

I notice this is ubuntu: Can you eliminate the distribution, by reproducing on a different distribution, or using a current upstream development kernel like 2.6.25-rc3?

Apologies for the delay. I was unable to reproduce this bug in many trials on 2.6.25-rc3, so it may have been either introduced by Ubuntu in gutsy or (more likely) fixed in the mainline kernel source since 2.6.22.

Anders Kaseorg has also seen this problem on Gutsy but was unable to reproduce it on Hardy, which lends credence to the second theory.

Quoted 1 line

What is /tmp ?

A regular old directory on the root fileystem, which is ext3fs on a different LVM volume group on a different disk from the volume group containing the LV's I'm snapshotting.

#7 lore raw
In reply to Greg Hudson
Quoted 22 lines · Greg Hudson [[email protected]] wrote:
Quoted 9 lines · Alasdair G Kergon
Quoted 7 lines · On Fri, Feb 29, 2008 at 12:32:41PM -0500, [email protected] wrote:

The reproduction recipe looks like:
rm -rf /tmp/test
mkdir /tmp/test
# Put around 60MB of files into /tmp/test
find /tmp/test -type f | xargs md5sum > /tmp/sum.pre
lvcreate --size 2G --snapshot /dev/dink/gutsy-i386-sbuild --name testsnapshot find /tmp/test -type f | xargs md5sum > /tmp/sum.post

Quoted 3 lines

Can you do that twice?
find /tmp/test -type f | xargs md5sum > /tmp/sum.post2
and check the two post files are the same?

In three reproductions of the page cache corruption, sum.post2 was always the same as sum.post.

In my experiences with this problem in general, the page cache corruption is not particularly transient; once it happens, the file continues to appear modified (with the same incorrect contents) for the indefinite future, until the machine is rebooted.

Quoted 8 lines
Quoted 2 lines

And add some syncs/blockdev --flushbufs at different places in the script to see if you can make it go away.

Nope, that never made it go away. I'm not sure in what situations flushing write buffers would have any effect. If I had a way to throw away the read-only page cache and force a file reload from disk, I would expect that to eliminate the visible effect of the corruption; at the moment the only reliable way I know how to do that is to reboot.

Not an expert on O_DIRECT, but it is supposed to read from the disk without creating page cache. I don't really know what it does if page cache exists! The "dd" command has O_DIRECT support and see if you notice any change with the corrupted file when you do "dd" with and without O_DIRECT.

--Malahal.

#8 lore raw
In reply to [email protected]
Quoted 4 lines · On Fri, 29 Feb 2008 [email protected] wrote:

rm -rf /tmp/test
mkdir /tmp/test
# Put around 60MB of files into /tmp/test find /tmp/test -type f | xargs md5sum > /tmp/sum.pre

Ran the last four lines several times on my Centos-5 system with /tmp and the snapshot VG on different volume groups. (/tmp on rootvg)

# du /tmp/test
84688   /tmp/test
# lvs backvg2/DFL
  LV   VG      Attr   LSize  Origin Snap%  Move Log Copy% 
  DFL  backvg2 -wi-a- 10.00G 
Quoted 4 lines

lvcreate --size 2G --snapshot /dev/backvg2/DFL --name DFL_snap find /tmp/test -type f | xargs md5sum > /tmp/sum.post lvremove -f /dev/backvg2/DFL_snap
diff -u /tmp/sum.pre /tmp/sum.

Doesn't seem to fail for us. (Sighs with relief.)

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.