DM: Possible disk corruption?

linux-lvm Jan-Benedict Glaw 2 participants 4 messages Jan 7, 2003

Share & tools

Jan-Benedict Glaw

author

2003-01-07 03:28 UTC #0 lore raw

Hi!

For testing purposes, I've placed a HDD with a number of PVs (assembling to one VG with one LV) into a test box and installed 2.4.19 with the dewnloadable DM patch (official release) and installed LVM2. Initially, this ran quite well, and I added another HDD (this is the third HDD. One is for the system, the other two large ones to store data).

After playing a bit with DM+LVM2, I shutted the box down to bring it to another location, but it didn't any longer boot:-( I reinstalled the base system (the LVs came up again with no problems:-), but rebooting the box again, it didn't boot up again:-(

This time, I used gpart to try to recover, and it found useful data to assemble a new partition table. Before writing it, I backed up the first 1.44MB of /dev/hda (guess where I stored it on:-> to do post-mortem-inspection.

The first 512 byted of mentioned boot HDD where overwritten by some Samba log! This is followed by some KB of 0x00. However, hda was never ever itself (or any partition on it) used with LVM or anything else what could IMHO caused this. The only difference compared to dozends of machines I installed is DM + LVM2 userland tools (used with LVM1 metadata). Is it possible that the DM version downloadable at sistina isn't really stable (for basic use)? Should I try some CVS checkout?

MfG, JBG
PS: If you would like to have the floppy's contents for your own inspection, I can place it somewhere...

Signature · 4 lines

   Jan-Benedict Glaw       [email protected]    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur
    fuer einen Freien Staat voll Freier Bürger" | im Internet!
   Shell Script APT-Proxy: http://lug-owl.de/~jbglaw/software/ap2/

Alasdair G Kergon

2003-01-07 08:52 UTC #1 lore raw

Quoted 2 lines · Jan-Benedict Glaw

The first 512 byted of mentioned boot HDD where overwritten by some Samba log! This is followed by some KB of 0x00.

If you read/write directly to /dev/hda do you see any corruption? Do you see any errors for the device in the system logs?

Quoted 4 lines

After playing a bit with DM+LVM2, I shutted the box down to bring it to another location, but it didn't any longer boot:-( I reinstalled the base system (the LVs came up again with no problems:-), but rebooting the box again, it didn't boot up again:-(

I reckon it's unlikely DM/LVM2 can be causing this - not least because the device you're seeing problems on is apparently outside the DM/LVM2 system.

Nevertheless, you can block LVM2 userspace from reading/writing to/from /dev/hda and its partitions by adding a filter to lvm.conf (see example.conf) e.g. "r|^/dev/hda|"
and then running vgscan to apply the new filter. [Personally I recommend using a positive filter though - (a)ccept the devices you want to use with LVM2 and (r)eject everything else.]

You can see what block devices the kernel device-mapper is really using by running 'dmsetup' after you have activated your LVs:
For each entry in /dev/mapper, run:
dmsetup deps name e.g. dmsetup deps vg0-lvol0
to get a list of major/minor numbers used.
dmsetup table <name>
shows you the mappings.

Did you keep a complete log of the commands you ran so you can check retrospectively for typos? (e.g. "/dev/hda" where "/dev/hda2" was intended) [Always a good idea to set shell history to a big number, or (esp. on a shared box) to use 'sudo' to record all commands run as root.]

If the problem is reproducible, I'd check for all occurrences of "/dev/hda" on your system (e.g. with 'rgrep') and make sure none of them have have a missing partition ID. I'd check the block device major/minors are correct & there isn't another /dev/something the same as /dev/hda & that the physical devices hda, hdb, hdc etc. haven't got reordered by mistake & the geometries & jumpers are correct.

Then I'd make sure syslog was set up to log *all* messages (many distributions annoyingly throw many debug messages away by default) and run through the system startup & shutdown sequence manually, to discover at what point the sector gets corrupted, and dealing with any error messages that appear either on the screen or in the system logs.

Quoted 2 lines

PS: If you would like to have the floppy's contents for your own inspection,

I don't think it would help in isolation: to follow that line of investigation you need to start by working out where those 512 bytes of samba log were meant to be on your system - e.g. are they a copy of existing data, or is that sector missing from where it should be? etc.

Alasdair

Signature · 1 line

[email protected]

Jan-Benedict Glaw

author

2003-01-07 09:50 UTC #2 lore raw

On Tue, 2003-01-07 14:52:49 +0000, Alasdair G Kergon <[email protected]> wrote in message <[email protected]>:

Quoted 5 lines · Jan-Benedict Glaw

Quoted 2 lines
The first 512 byted of mentioned boot HDD where overwritten by some Samba log! This is followed by some KB of 0x00.
If you read/write directly to /dev/hda do you see any corruption?

I've not seen any. The system is quite busy at at some time swapping. Swapspace is on hda so I would have seen killed processes. But despite not booting at some time, the box is rock solid.

Quoted 1 line

Do you see any errors for the device in the system logs?

No.

Quoted 6 lines

Nevertheless, you can block LVM2 userspace from reading/writing to/from /dev/hda and its partitions by adding a filter to lvm.conf (see example.conf) e.g. "r|^/dev/hda|"
and then running vgscan to apply the new filter. [Personally I recommend using a positive filter though - (a)ccept the devices you want to use with LVM2 and (r)eject everything else.]

As the box isn't really productive, I can continue to play with it. I'll do some DM/LVM activity and occassionally sfdisk -l /dev/hda.

Quoted 5 lines

Did you keep a complete log of the commands you ran so you can check retrospectively for typos? (e.g. "/dev/hda" where "/dev/hda2" was intended) [Always a good idea to set shell history to a big number, or (esp. on a shared box) to use 'sudo' to record all commands run as root.]

Sorry, nothing there:

mirror:~# cat .bash_history |egrep '(vg|lv|pv)'
pvdisplay -v |less -S
cat /etc/resolv.conf

This has been the second scribbled boot sector on this box. Using LVM on daily basis (and on more than only that one box) I don't think that I did the same mistake twice.

Well, I have to do some LVM administration in near future, I'll pay attention to my /dev/hda:-)

Basically, I liked to know if there are known bugs which could have lead to such a corruption.

Oh, the samba los snipplet should have gone to /var/log/samba/<some_file_name>, which is on /dev/hda3. /dev/hda2 is swap and /dev/hda1 is /boot. So the block ended up in a wrong partition, at a wrong offset but on the right device (hey, at least one out of three is right:-)

I'll have an eye on it.

MfG, JBG