Skip to contentSkip to conversation

Bad disk?

linux-lvm Mauricio Tavares 4 participants 9 messages Nov 10, 2010 – Nov 11, 2010

lore
Share & tools
lore
#0 lore raw

Yesterday I added a hard drive (to put extra stuff on it) to my ubuntu 10.10 box and created a LVM in it. Then copied some files to it and restarted the machine to see if it would mount into the right mountpoint. It didn't. So I decided to see if it was there (vg in question is export):

raub@strangepork:~$ sudo vgscan
  Reading all physical volumes.  This may take a while...
  /dev/dm-0: read failed after 0 of 4096 at 429496664064: Input/output 
error
  /dev/dm-0: read failed after 0 of 4096 at 429496721408: Input/output 
error
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  Found volume group "export" using metadata type lvm2
  Found volume group "root" using metadata type lvm2
raub@strangepork:~$

Those dm-0 messages do not make me happy. dmesg and vgchange make me think the problem is on the new drive:

[  268.024593] scsi 0:0:0:0: Direct-Access     ATA      ST3500320NS 
 SN04 PQ: 0 ANSI: 5
[  268.024900] sd 0:0:0:0: [sdc] 976773168 512-byte logical blocks: (500 GB/465 GiB)
[  268.024918] sd 0:0:0:0: Attached scsi generic sg2 type 0
[  268.024996] sd 0:0:0:0: [sdc] Write Protect is off
[  268.025003] sd 0:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[  268.025046] sd 0:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  268.025377]  sdc: sdc1
[  268.049853] sd 0:0:0:0: [sdc] Attached SCSI disk
[  335.467482] quiet_error: 3 callbacks suppressed
[  335.467492] Buffer I/O error on device dm-0, logical block 104857584
[  335.467540] Buffer I/O error on device dm-0, logical block 104857584
[  335.467589] Buffer I/O error on device dm-0, logical block 104857598
[  335.467615] Buffer I/O error on device dm-0, logical block 104857598
[  335.467647] Buffer I/O error on device dm-0, logical block 0
[  335.467671] Buffer I/O error on device dm-0, logical block 0
[  335.467703] Buffer I/O error on device dm-0, logical block 1
[  335.467734] Buffer I/O error on device dm-0, logical block 104857599
[  335.467762] Buffer I/O error on device dm-0, logical block 104857599
[  335.467788] Buffer I/O error on device dm-0, logical block 104857599
raub@strangepork:~$ sudo vgchange -a y
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  1 logical volume(s) in volume group "export" now active
  4 logical volume(s) in volume group "root" now active
raub@strangepork:~$

Glancing through http://readlist.com/lists/centos.org/centos/1/5704.html I wondered if I had the wrong file system descriptor:

raub@strangepork:~$ sudo sfdisk -l /dev/sdc

Disk /dev/sdc: 60801 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1          0+  60800   60801- 488384001   8e  Linux LVM
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty

raub@strangepork:~$

I guess that rules that out. Does that mean the disk is bad? If so, is there a way to retrieve the data I put in it?

#1 lore raw
Quoted 5 lines · Mauricio Tavares

Yesterday I added a hard drive (to put extra stuff on it) to my ubuntu 10.10 box and created a LVM in it. Then copied some files to it and restarted the machine to see if it would mount into the right mountpoint. It didn't. So I decided to see if it was there (vg in question is export):

None of the errors you posted mention a physical disk at all, so there isn't anything wrong with them. To find out what dm-0 is you need to poke around with dmsetup.

#2 lore raw
Quoted 10 lines · Phillip Susi
Quoted 5 lines · Mauricio Tavares

Yesterday I added a hard drive (to put extra stuff on it) to my ubuntu 10.10 box and created a LVM in it. Then copied some files to it and restarted the machine to see if it would mount into the right mountpoint. It didn't. So I decided to see if it was there (vg in question is export):

None of the errors you posted mention a physical disk at all, so there isn't anything wrong with them. To find out what dm-0 is you need to poke around with dmsetup.

Now that might be interesting since I honestly do not know how to use
it (and my google-fu seems weak today). But this is a bit more info I found:

raub@strangepork:~$ sudo pvs -a
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 429496664064: Input/output 
error
  /dev/dm-0: read failed after 0 of 4096 at 429496721408: Input/output 
error
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  PV         VG     Fmt  Attr PSize   PFree
  /dev/dm-0              --        0       0
  /dev/dm-1  root   lvm2 a-   277.05g 192.55g
  /dev/dm-2              --        0       0
  /dev/dm-3              --        0       0
  /dev/dm-4              --        0       0
  /dev/dm-5              --        0       0
  /dev/ram0              --        0       0
  /dev/ram1              --        0       0
  /dev/ram10             --        0       0
  /dev/ram11             --        0       0
  /dev/ram12             --        0       0
  /dev/ram13             --        0       0
  /dev/ram14             --        0       0
  /dev/ram15             --        0       0
  /dev/ram2              --        0       0
  /dev/ram3              --        0       0
  /dev/ram4              --        0       0
  /dev/ram5              --        0       0
  /dev/ram6              --        0       0
  /dev/ram7              --        0       0
  /dev/ram8              --        0       0
  /dev/ram9              --        0       0
  /dev/root              --        0       0
  /dev/sda1              --        0       0
  /dev/sda3              --        0       0
  /dev/sdc1  export lvm2 a-   465.76g  65.76g
raub@strangepork:~$

FYI, the root vg in inside /dev/sda3, which is an encrypted partition. How can I find which drive /dev/dm-0 is?

#3 lore raw
Quoted 1 line · Mauricio Tavares

How can I find which drive /dev/dm-0 is?

Try:
cat /sys/block/dm-0/dm/name

On my system, I created dm2device so I can tell which snapshot is full or whatever device has a problem:

# cat /usr/local/bin/dm2device
#!/bin/sh

cat /sys/block/$1/dm/name

--
Ray Morris
[email protected]

Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/

Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/

Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php

Quoted 61 lines · Mauricio Tavares
Quoted 10 lines · Phillip Susi
Quoted 5 lines · Mauricio Tavares

Yesterday I added a hard drive (to put extra stuff on it) to my ubuntu 10.10 box and created a LVM in it. Then copied some files to it and restarted the machine to see if it would mount into the right mountpoint. It didn't. So I decided to see if it was there (vg in question is export):

None of the errors you posted mention a physical disk at all, so there isn't anything wrong with them. To find out what dm-0 is you need to poke around with dmsetup.

Now that might be interesting since I honestly do not know how
to use it (and my google-fu seems weak today). But this is a bit more info I found:

raub@strangepork:~$ sudo pvs -a
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 429496664064:  
Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 429496721408:  
Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  PV         VG     Fmt  Attr PSize   PFree
  /dev/dm-0              --        0       0
  /dev/dm-1  root   lvm2 a-   277.05g 192.55g
  /dev/dm-2              --        0       0
  /dev/dm-3              --        0       0
  /dev/dm-4              --        0       0
  /dev/dm-5              --        0       0
  /dev/ram0              --        0       0
  /dev/ram1              --        0       0
  /dev/ram10             --        0       0
  /dev/ram11             --        0       0
  /dev/ram12             --        0       0
  /dev/ram13             --        0       0
  /dev/ram14             --        0       0
  /dev/ram15             --        0       0
  /dev/ram2              --        0       0
  /dev/ram3              --        0       0
  /dev/ram4              --        0       0
  /dev/ram5              --        0       0
  /dev/ram6              --        0       0
  /dev/ram7              --        0       0
  /dev/ram8              --        0       0
  /dev/ram9              --        0       0
  /dev/root              --        0       0
  /dev/sda1              --        0       0
  /dev/sda3              --        0       0
  /dev/sdc1  export lvm2 a-   465.76g  65.76g
raub@strangepork:~$

FYI, the root vg in inside /dev/sda3, which is an encrypted partition. How can I find which drive /dev/dm-0 is?

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
Quoted 7 lines · Ray Morris
Quoted 1 line · Mauricio Tavares

How can I find which drive /dev/dm-0 is?

Try:
cat /sys/block/dm-0/dm/name

Quoted 8 lines

On my system, I created dm2device so I can tell which snapshot is full or whatever device has a problem:

# cat /usr/local/bin/dm2device
#!/bin/sh

cat /sys/block/$1/dm/name

Thanks!

cat /sys/block/dm-0/dm/name returned export-vms, which is /dev/sdc1.

Quoted 84 lines

--
Ray Morris
[email protected]

Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/

Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/

Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php

Quoted 54 lines · Mauricio Tavares
Quoted 10 lines · Phillip Susi
Quoted 5 lines · Mauricio Tavares

Yesterday I added a hard drive (to put extra stuff on it) to my ubuntu 10.10 box and created a LVM in it. Then copied some files to it and restarted the machine to see if it would mount into the right mountpoint. It didn't. So I decided to see if it was there (vg in question is export):

None of the errors you posted mention a physical disk at all, so there isn't anything wrong with them. To find out what dm-0 is you need to poke around with dmsetup.

Now that might be interesting since I honestly do not know how to use it (and my google-fu seems weak today). But this is a bit more info I found:

raub@strangepork:~$ sudo pvs -a
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 429496664064: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 429496721408: Input/output
error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
PV VG Fmt Attr PSize PFree
/dev/dm-0 -- 0 0
/dev/dm-1 root lvm2 a- 277.05g 192.55g
/dev/dm-2 -- 0 0
/dev/dm-3 -- 0 0
/dev/dm-4 -- 0 0
/dev/dm-5 -- 0 0
/dev/ram0 -- 0 0
/dev/ram1 -- 0 0
/dev/ram10 -- 0 0
/dev/ram11 -- 0 0
/dev/ram12 -- 0 0
/dev/ram13 -- 0 0
/dev/ram14 -- 0 0
/dev/ram15 -- 0 0
/dev/ram2 -- 0 0
/dev/ram3 -- 0 0
/dev/ram4 -- 0 0
/dev/ram5 -- 0 0
/dev/ram6 -- 0 0
/dev/ram7 -- 0 0
/dev/ram8 -- 0 0
/dev/ram9 -- 0 0
/dev/root -- 0 0
/dev/sda1 -- 0 0
/dev/sda3 -- 0 0
/dev/sdc1 export lvm2 a- 465.76g 65.76g
raub@strangepork:~$
Quoted 9 lines

FYI, the root vg in inside /dev/sda3, which is an encrypted partition. How can I find which drive /dev/dm-0 is?

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw
In reply to Mauricio Tavares
Quoted 2 lines · Mauricio Tavares

Those dm-0 messages do not make me happy. dmesg and vgchange make me think the problem is on the new drive:

Those dm-0 messages are probably a logical error. For instance, a snapshot that is full would give those errors. You need to tell us what dm-0 is mapped to. Look in /dev/mapper for starters.

Quoted 11 lines
[  268.024593] scsi 0:0:0:0: Direct-Access     ATA      ST3500320NS  SN04 PQ:
0 ANSI: 5
[  268.024900] sd 0:0:0:0: [sdc] 976773168 512-byte logical blocks: (500
GB/465 GiB)
[  268.024918] sd 0:0:0:0: Attached scsi generic sg2 type 0
[  268.024996] sd 0:0:0:0: [sdc] Write Protect is off
[  268.025003] sd 0:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[  268.025046] sd 0:0:0:0: [sdc] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[  268.025377]  sdc: sdc1
[  268.049853] sd 0:0:0:0: [sdc] Attached SCSI disk

This is normal for your new disk.

Quoted 11 lines
[  335.467482] quiet_error: 3 callbacks suppressed
[  335.467492] Buffer I/O error on device dm-0, logical block 104857584
[  335.467540] Buffer I/O error on device dm-0, logical block 104857584
[  335.467589] Buffer I/O error on device dm-0, logical block 104857598
[  335.467615] Buffer I/O error on device dm-0, logical block 104857598
[  335.467647] Buffer I/O error on device dm-0, logical block 0
[  335.467671] Buffer I/O error on device dm-0, logical block 0
[  335.467703] Buffer I/O error on device dm-0, logical block 1
[  335.467734] Buffer I/O error on device dm-0, logical block 104857599
[  335.467762] Buffer I/O error on device dm-0, logical block 104857599
[  335.467788] Buffer I/O error on device dm-0, logical block 104857599

Again, this is on dm-0, not sdc.

Quoted 10 lines
raub@strangepork:~$ sudo sfdisk -l /dev/sdc

Disk /dev/sdc: 60801 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1          0+  60800   60801- 488384001   8e  Linux LVM
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty

This is normal, not sure what is has to do with filesystem desciptors.

Tell us exactly what you mean by "put a LVM on it". Did you run pvcreate? vgcreate? lvcreate? You might find the output of "pvs" enlightening. That will tell us what PVs you have created.
And list /dev/mapper so we know what dm-0 is, and include the output of "lvs".

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#6 lore raw
Quoted 9 lines · Stuart D. Gathman
Quoted 2 lines · Mauricio Tavares

Those dm-0 messages do not make me happy. dmesg and vgchange make me think the problem is on the new drive:

Those dm-0 messages are probably a logical error. For instance, a snapshot that is full would give those errors. You need to tell us what dm-0 is mapped to. Look in /dev/mapper for starters.

Sorry for that: I did not think it would be there. But, as you said, it is:

lrwxrwxrwx 1 root root 7 2010-11-10 09:30 export-vms -> ../dm-0

the export vg is in /dev/sdc1.

Quoted 39 lines
Quoted 11 lines
[  268.024593] scsi 0:0:0:0: Direct-Access     ATA      ST3500320NS  SN04 PQ:
0 ANSI: 5
[  268.024900] sd 0:0:0:0: [sdc] 976773168 512-byte logical blocks: (500
GB/465 GiB)
[  268.024918] sd 0:0:0:0: Attached scsi generic sg2 type 0
[  268.024996] sd 0:0:0:0: [sdc] Write Protect is off
[  268.025003] sd 0:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[  268.025046] sd 0:0:0:0: [sdc] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[  268.025377]  sdc: sdc1
[  268.049853] sd 0:0:0:0: [sdc] Attached SCSI disk

This is normal for your new disk.

Quoted 11 lines
[  335.467482] quiet_error: 3 callbacks suppressed
[  335.467492] Buffer I/O error on device dm-0, logical block 104857584
[  335.467540] Buffer I/O error on device dm-0, logical block 104857584
[  335.467589] Buffer I/O error on device dm-0, logical block 104857598
[  335.467615] Buffer I/O error on device dm-0, logical block 104857598
[  335.467647] Buffer I/O error on device dm-0, logical block 0
[  335.467671] Buffer I/O error on device dm-0, logical block 0
[  335.467703] Buffer I/O error on device dm-0, logical block 1
[  335.467734] Buffer I/O error on device dm-0, logical block 104857599
[  335.467762] Buffer I/O error on device dm-0, logical block 104857599
[  335.467788] Buffer I/O error on device dm-0, logical block 104857599

Again, this is on dm-0, not sdc.

Quoted 10 lines
raub@strangepork:~$ sudo sfdisk -l /dev/sdc

Disk /dev/sdc: 60801 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System

/dev/sdc1          0+  60800   60801- 488384001   8e  Linux LVM
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty
Quoted 7 lines

This is normal, not sure what is has to do with filesystem desciptors.

Tell us exactly what you mean by "put a LVM on it". Did you run pvcreate? vgcreate? lvcreate? You might find the output of "pvs" enlightening. That will tell us what PVs you have created.
And list /dev/mapper so we know what dm-0 is, and include the output of "lvs".

Let me put this way, I thought I did. I mean, after creating the
partition, setting it to LVM (8e), then running

pvcreate /dev/sdc1
vgcreate export /dev/sdc1
lvcreate -L 400G --name vms export

I used mkfs.ext4 to create partition (on /dev/mapper/export-vms) and off I went. Do you think I missed a step?

#7 lore raw
Quoted 9 lines · Mauricio Tavares
Quoted 6 lines

Tell us exactly what you mean by "put a LVM on it". Did you run pvcreate? vgcreate? lvcreate? You might find the output of "pvs" enlightening. That will tell us what PVs you have created. And list /dev/mapper so we know what dm-0 is, and include the output of "lvs".

Let me put this way, I thought I did. I mean, after creating the
partition, setting it to LVM (8e), then running

Quoted 6 lines

pvcreate /dev/sdc1
vgcreate export /dev/sdc1
lvcreate -L 400G --name vms export

I used mkfs.ext4 to create partition (on /dev/mapper/export-vms) and off I went. Do you think I missed a step?

Great. Now include output of "lvs"

BTW, if you really suspect a disk error, test for it directly. E.g., you can run

# dd if=/dev/sdc1 of=/dev/null bs=256k

to read through the partition or

# smartctl -t long /dev/sdc

To initiate a long self test of the disk (need smartmontools installed).

A brand new disk that flunks self test is indeed defective.

However, for real physical I/O errors, there would be errors logged in /var/log/messages referencing sdc (as opposed to dm-0), so I still think it is a logical error.

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
#8 lore raw
Quoted 11 lines · Stuart D. Gathman
Quoted 9 lines · Mauricio Tavares
Quoted 6 lines

Tell us exactly what you mean by "put a LVM on it". Did you run pvcreate? vgcreate? lvcreate? You might find the output of "pvs" enlightening. That will tell us what PVs you have created. And list /dev/mapper so we know what dm-0 is, and include the output of "lvs".

Let me put this way, I thought I did. I mean, after creating the
partition, setting it to LVM (8e), then running

Quoted 8 lines
Quoted 6 lines

pvcreate /dev/sdc1
vgcreate export /dev/sdc1
lvcreate -L 400G --name vms export

I used mkfs.ext4 to create partition (on /dev/mapper/export-vms) and off I went. Do you think I missed a step?

Great. Now include output of "lvs"

raub@strangepork:~$ sudo lvs export
  /dev/dm-0: read failed after 0 of 4096 at 0: Input/output error
  LV   VG     Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  vms  export -wi-a- 400.00g
raub@strangepork:~$
Quoted 17 lines

BTW, if you really suspect a disk error, test for it directly. E.g., you can run

# dd if=/dev/sdc1 of=/dev/null bs=256k

to read through the partition or

# smartctl -t long /dev/sdc

To initiate a long self test of the disk (need smartmontools installed).

A brand new disk that flunks self test is indeed defective.

However, for real physical I/O errors, there would be errors logged in /var/log/messages referencing sdc (as opposed to dm-0), so I still think it is a logical error.