Skip to contentSkip to conversation

Block device cache issue

DISCUSSION Apollon Oikonomopoulos 3 participants 3 messages Apr 2, 2009 – Apr 7, 2009

lore
Share & tools
lore
#0 lore raw

Greetings to the list,

At my company, we have come across something that we think is a design limitation in the way the Linux kernel handles block device caches. I will first describe the incident we encountered, before speculating on the actual cause.

As part of our infrastructure, we are running some Linux servers used as Xen Dom0s, using SAN LUNs as the VMs' disk images, so these LUNs contain normal MBR partition tables. At some point we came across a VM, that - due to a misconfiguration of GRUB - failed on a reboot. We used multipath-tools' kpartx to create a device-mapper device pointing to the first partition of the LUN, mounted the filesystem, changed boot/grub/menu.lst, unmounted it and proceeded to boot the VM once more. To our surprise, Xen's pygrub showed the boot menu exactly as it was before the changes we made. We double-checked that the changes we made were indeed there and tried to find out what was actually going on.

As it turned out, the LUN device's read buffers had not been updated; losetup'ing the LUN device with the proper offset to the first partition and mounting it gave us exactly the image of the filesystem as it was _before_ our changes. We started digging into the kernel's buffer internals and came along the conclusion [1] that every block device has its own pagecache, attached to a hash of (major,minor), that is independent from the caches of its containing or contained devices.

Now, in practice one rarely - if ever - accesses the same data from these two different paths (disk + partition), except in scenarios like this. However currently there seems to be an implicit assumption that these two paths should not be used in the same "uptime" cycle at all, at least not without dropping the caches. For the record, I managed to reproduce the whole issue by reading a single block through sda, dd'ing random data to it through sda1 and re-reading it through sda: its contents were intact (even hours later) and were up-to-date only when using O_DIRECT and finally when I dropped all caches (using /proc/sys/vm/drop_caches).

And now we come to the question part: Can someone please verify that the above statements are correct, or am I missing something? If they are, should it perhaps be the case that the partition's buffers somehow be linked with those of the containing device, or even be part of them? I don't even know if this is possible without significant overhead in the page cache (of which my understanding is very shallow), but keep in mind that this behaviour almost led to filesystem corruption (luckily we only changed a single file and hit a single inode).

Thank you for your time. Cheers,
Apollon

PS: I am not subscribed to the list, so I would appreciate if you could
Cc any answers to my address.

[1] If I interpret the contents of fs/buffer.c and include/linux/buffer_head.h correctly. Unfortunately, I'm not a kernel hacker, so I apologise if I'm mistaken at this point.

Signature · 4 lines
-----------------------------------------------------------
 Apollon Oikonomopoulos - GRNET Network Operations Centre
 Greek Research & Technology Network - http://www.grnet.gr
----------------------------------------------------------- 
#1 lore raw
Quoted 37 lines · On Thu, 2 Apr 2009 17:52:05 +0300 Apollon Oikonomopoulos…

Greetings to the list,

At my company, we have come across something that we think is a design limitation in the way the Linux kernel handles block device caches. I will first describe the incident we encountered, before speculating on the actual cause.

As part of our infrastructure, we are running some Linux servers used as Xen Dom0s, using SAN LUNs as the VMs' disk images, so these LUNs contain normal MBR partition tables. At some point we came across a VM, that - due to a misconfiguration of GRUB - failed on a reboot. We used multipath-tools' kpartx to create a device-mapper device pointing to the first partition of the LUN, mounted the filesystem, changed boot/grub/menu.lst, unmounted it and proceeded to boot the VM once more. To our surprise, Xen's pygrub showed the boot menu exactly as it was before the changes we made. We double-checked that the changes we made were indeed there and tried to find out what was actually going on.

As it turned out, the LUN device's read buffers had not been updated; losetup'ing the LUN device with the proper offset to the first partition and mounting it gave us exactly the image of the filesystem as it was _before_ our changes. We started digging into the kernel's buffer internals and came along the conclusion [1] that every block device has its own pagecache, attached to a hash of (major,minor), that is independent from the caches of its containing or contained devices.

Now, in practice one rarely - if ever - accesses the same data from these two different paths (disk + partition), except in scenarios like this. However currently there seems to be an implicit assumption that these two paths should not be used in the same "uptime" cycle at all, at least not without dropping the caches. For the record, I managed to reproduce the whole issue by reading a single block through sda, dd'ing random data to it through sda1 and re-reading it through sda: its contents were intact (even hours later) and were up-to-date only when using O_DIRECT and finally when I dropped all caches (using /proc/sys/vm/drop_caches).

Quoted 2 lines

And now we come to the question part: Can someone please verify that the above statements are correct, or am I missing something?

The above statements are correct ;)

Similarly, the pagecache for /etc/password is separate from the pagecache for the device upon which /etc is mounted.

Quoted 7 lines

If they are,
should it perhaps be the case that the partition's buffers somehow be linked with those of the containing device, or even be part of them? I don't even know if this is possible without significant overhead in the page cache (of which my understanding is very shallow), but keep in mind that this behaviour almost led to filesystem corruption (luckily we only changed a single file and hit a single inode).

It would incur overhead. We could perhaps fix it by having a single cache for /dev/sda and then just making /dev/sda1 access that cache with an offset. But it rarely if ever comes up - I guess the few applications which do this sort of thing are taking suitable steps to avoid it - fsync, ioctl(BKLFLSBUF), posix_fadvise(FADV_DONTNEED), O_DIRECT, etc.

#2 lore raw
Quoted 7 lines · Andrew Morton
Quoted 3 lines

should it perhaps be the case that the partition's buffers somehow be linked with those of the containing device, or even be part of them? I don't even know if this is possible without significant overhead in the page cache (of which my understanding is very shallow), but keep in mind that this behaviour almost led to filesystem corruption (luckily we only changed a single file and hit a single inode).

It would incur overhead. We could perhaps fix it by having a single cache for /dev/sda and then just making /dev/sda1 access that cache with an offset.

The offset can be non PAGE_SIZE aligned (and usually isn't, 63 sectors difference with normal partitioning).

Signature · 1 line
error compiling committee.c: too many arguments to function