Linux software RAID assistance

So I tried running mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 missing /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 --chunk=64

as it seemed that /dev/sdd1 failed to be added to the array. This did nothing.

dmesg contains:

md: invalid superblock checksum on sdd1
md: sdd1 does not have a valid v1.1 superblock, not importing!
md: md_import_device returned -22

The output of mdadm -E is as follows:

proxmox:~# mdadm -E /dev/sd[d-m]1
/dev/sdd1:
         Magic : a92b4efc
       Version : 1.1
   Feature Map : 0x0
    Array UUID : c4f62f32:4244a1db:b7746203:f10b5227
          Name : proølox:0
 Creation Time : Sat Aug 21 19:16:38 2010
    Raid Level : raid5
  Raid Devices : 10

Avail Dev Size : 1953099512 (931.31 GiB 999.99 GB)

    Array Size : 17577894528 (8381.79 GiB 8999.88 GB)

Used Dev Size : 1953099392 (931.31 GiB 999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
Device UUID : b0ffd0a1:866676af:3ae4c03c:d219676e

   Update Time : Mon Feb  7 21:08:29 2011
      Checksum : 13aa9685 - expected 93aa9672
        Events : 60802

        Layout : left-symmetric
    Chunk Size : 64K

Device Role : Active device 0
Array State : AAAAAAAAAA ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 7d55c29e:273c35da:f6438197:0365f95f
Name : ubuntu:0
Creation Time : Thu Feb 10 11:59:48 2011
Raid Level : raid5
Raid Devices : 10

Avail Dev Size : 1953099512 (931.31 GiB 999.99 GB)

    Array Size : 17577894528 (8381.79 GiB 8999.88 GB)

Used Dev Size : 1953099392 (931.31 GiB 999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
Device UUID : 4121fa31:7543218c:f42a937d:fddf04e8

Update Time : Thu Feb 10 12:17:59 2011
Checksum : 5f965c84 - correct
Events : 4

Layout : left-symmetric
Chunk Size : 64K

Device Role : Active device 1
Array State : .AAAAAAAAA ('A' == active, '.' == missing)

Hi all

I use a 3ware 9500-12 port sata card (JBOD) which will not work without a 128mb sodimm. The sodimm socket is flakey and the result is that the machine occasionally crashes. Yesterday I finally gave in and put together another
machine so that I can rsync between them. When I turned the machine on today to set up rync, the RAID array was not gone, but corrupted.
Typical...

I built the array in Aug 2010 using the following command:

mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{b,c,d,e,f,g,h,i,j,k}1 --chunk=64

Using LVM, I did the following:
pvscan
pvcreate -M2 /dev/md0
vgcreate lvm-raid /dev/md0
vgdisplay lvm-raid
vgscan
lvscan
lvcreate -v -l 100%VG -n RAID lvm-raid
lvdisplay /dev/lvm-raid/lvm0

I then formatted using:
mkfs -t ext4 -v -m .1 -b 4096 -E stride=16,stripe-width=144 /dev/lvm-raid/RAID

This worked perfectly since I created the array. Now mdadm is coming up with

proxmox:/dev/md# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/ubuntu:0

mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: cannot open device /dev/sdm1: Device or resource busy
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: cannot open device /dev/sdl1: Device or resource busy
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: cannot open device /dev/sdk1: Device or resource busy
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: cannot open device /dev/sdh1: Device or resource busy
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: no RAID superblock on /dev/sdd
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sdd1 is identified as a member of /dev/md/pro�lox:0, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 2 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 3 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 4 of /dev/md/pro�lox:0

mdadm: no uptodate device for slot 5 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 6 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 7 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 8 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 9 of /dev/md/pro�lox:0
mdadm: failed to add /dev/sdd1 to /dev/md/pro�lox:0: Invalid argument
mdadm: /dev/md/pro�lox:0 assembled from 0 drives - not enough to start the array.

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/sdd
mdadm: No arrays found in config file or automatically

pvscan and vgscan show nothing.

So I tried running mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 missing /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 --chunk=64

as it seemed that /dev/sdd1 failed to be added to the array. This did nothing.

dmesg contains:

md: invalid superblock checksum on sdd1

Quoted 9 lines · On Thu, 10 Feb 2011 16:16:44 +0000 Simon McNair<[email protected]> wrote:

Hi all

I use a 3ware 9500-12 port sata card (JBOD) which will not work without a 128mb sodimm. The sodimm socket is flakey and the result is that the machine occasionally crashes. Yesterday I finally gave in and put together another
machine so that I can rsync between them. When I turned the machine on today to set up rync, the RAID array was not gone, but corrupted.
Typical...

Presumably the old machine was called 'ubuntu' and the new machine 'proølox'

Quoted 19 lines

I built the array in Aug 2010 using the following command:

mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{b,c,d,e,f,g,h,i,j,k}1 --chunk=64

Using LVM, I did the following:
pvscan
pvcreate -M2 /dev/md0
vgcreate lvm-raid /dev/md0
vgdisplay lvm-raid
vgscan
lvscan
lvcreate -v -l 100%VG -n RAID lvm-raid
lvdisplay /dev/lvm-raid/lvm0

I then formatted using:
mkfs -t ext4 -v -m .1 -b 4096 -E stride=16,stripe-width=144 /dev/lvm-raid/RAID

Quoted 6 lines

This worked perfectly since I created the array. Now mdadm is coming up with

proxmox:/dev/md# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/ubuntu:0

And it seems that ubuntu:0 have been successfully assembled.
It is missing one device for some reason (sdd1) but RAID can cope with that.

Quoted 30 lines

mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: cannot open device /dev/sdm1: Device or resource busy
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: cannot open device /dev/sdl1: Device or resource busy
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: cannot open device /dev/sdk1: Device or resource busy
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: cannot open device /dev/sdh1: Device or resource busy
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: no RAID superblock on /dev/sdd
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sdd1 is identified as a member of /dev/md/pro�lox:0, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 2 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 3 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 4 of /dev/md/pro�lox:0

Quoted 8 lines

mdadm: no uptodate device for slot 5 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 6 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 7 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 8 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 9 of /dev/md/pro�lox:0
mdadm: failed to add /dev/sdd1 to /dev/md/pro�lox:0: Invalid argument
mdadm: /dev/md/pro�lox:0 assembled from 0 drives - not enough to start the array.

This looks like it is *after* to trying the --create command you give below.. It is best to report things in the order they happen, else you can confuse people (or get caught out!).

Quoted 4 lines

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/sdd
mdadm: No arrays found in config file or automatically

Quoted 8 lines

pvscan and vgscan show nothing.

So I tried running mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 missing /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 --chunk=64

as it seemed that /dev/sdd1 failed to be added to the array. This did nothing.

It did not to nothing. It wrote a superblock to /dev/sdd1 and complained that it couldn't write to all the others --- didn't it?

Quoted 3 lines

dmesg contains:

md: invalid superblock checksum on sdd1

I guess that is why sdd1 was missing from 'ubuntu:0'. Though as I cannot tell if this happened before or after any of the various things reported above, it is hard to be sure.

The real mystery is why 'pvscan' reports nothing.

What about
pvscan --verbose

or

blkid -p /dev/md/ubuntu:0

or even

   dd of=/dev/md/ubuntu:0 count=8 | od -c

??

NeilBrown

Quoted 10 lines · On Thu, 10 Feb 2011 16:16:44 +0000 Simon McNair <[email protected]> wrote:

Hi all

I use a 3ware 9500-12 port sata card (JBOD) which will not work without a 128mb sodimm. The sodimm socket is flakey and the result is that the machine occasionally crashes. Yesterday I finally gave in and put together another
machine so that I can rsync between them. When I turned the machine on today to set up rync, the RAID array was not gone, but corrupted.
Typical...

Presumably the old machine was called 'ubuntu' and the new machine 'proølox'

Quoted 23 lines

I built the array in Aug 2010 using the following command:

mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{b,c,d,e,f,g,h,i,j,k}1 --chunk=64

Using LVM, I did the following:
pvscan
pvcreate -M2 /dev/md0
vgcreate lvm-raid /dev/md0
vgdisplay lvm-raid
vgscan
lvscan
lvcreate -v -l 100%VG -n RAID lvm-raid
lvdisplay /dev/lvm-raid/lvm0

I then formatted using:
mkfs -t ext4 -v -m .1 -b 4096 -E stride=16,stripe-width=144 /dev/lvm-raid/RAID

This worked perfectly since I created the array. Now mdadm is coming up with

Quoted 3 lines

proxmox:/dev/md# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/ubuntu:0

And it seems that ubuntu:0 have been successfully assembled.
It is missing one device for some reason (sdd1) but RAID can cope with that.

Quoted 5 lines

mdadm: no uptodate device for slot 8 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 9 of /dev/md/pro�lox:0
mdadm: failed to add /dev/sdd1 to /dev/md/pro�lox:0: Invalid argument
mdadm: /dev/md/pro�lox:0 assembled from 0 drives - not enough to start the array.

This looks like it is *after* to trying the --create command you give below.. It is best to report things in the order they happen, else you can confuse people (or get caught out!).

Quoted 10 lines

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/sdd
mdadm: No arrays found in config file or automatically

pvscan and vgscan show nothing.

So I tried running mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 missing /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 --chunk=64

Quoted 2 lines

as it seemed that /dev/sdd1 failed to be added to the array. This did nothing.

It did not to nothing. It wrote a superblock to /dev/sdd1 and complained that it couldn't write to all the others --- didn't it?

Quoted 3 lines

dmesg contains:

md: invalid superblock checksum on sdd1

I guess that is why sdd1 was missing from 'ubuntu:0'. Though as I cannot tell if this happened before or after any of the various things reported above, it is hard to be sure.

The real mystery is why 'pvscan' reports nothing.

What about
pvscan --verbose

or

blkid -p /dev/md/ubuntu:0

or even

dd of=/dev/md/ubuntu:0 count=8 | od -c

Hi Neil,

Since Simon has responded, let me summarize the assistance I provided per his off-list request:

Quoted 33 lines · NeilBrown

Quoted 9 lines · On Thu, 10 Feb 2011 16:16:44 +0000 Simon McNair<[email protected]> wrote:

Hi all

I use a 3ware 9500-12 port sata card (JBOD) which will not work without a 128mb sodimm. The sodimm socket is flakey and the result is that the machine occasionally crashes. Yesterday I finally gave in and put together another
machine so that I can rsync between them. When I turned the machine on today to set up rync, the RAID array was not gone, but corrupted.
Typical...

Presumably the old machine was called 'ubuntu' and the new machine 'proølox'

Quoted 19 lines

I built the array in Aug 2010 using the following command:

mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{b,c,d,e,f,g,h,i,j,k}1 --chunk=64

Using LVM, I did the following:
pvscan
pvcreate -M2 /dev/md0
vgcreate lvm-raid /dev/md0
vgdisplay lvm-raid
vgscan
lvscan
lvcreate -v -l 100%VG -n RAID lvm-raid
lvdisplay /dev/lvm-raid/lvm0

I then formatted using:
mkfs -t ext4 -v -m .1 -b 4096 -E stride=16,stripe-width=144 /dev/lvm-raid/RAID

Quoted 8 lines

Quoted 6 lines

This worked perfectly since I created the array. Now mdadm is coming up with

proxmox:/dev/md# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/ubuntu:0

And it seems that ubuntu:0 have been successfully assembled.
It is missing one device for some reason (sdd1) but RAID can cope with that.

3ware card is compromised, with a loose buffer memory dimm. Some of its ECC errors were caught and reported in dmesg. Its likely, based on the loose memory socket, that many multiple-bit errors got through.

[trim /]

Quoted 8 lines

Quoted 5 lines

mdadm: no uptodate device for slot 8 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 9 of /dev/md/pro�lox:0
mdadm: failed to add /dev/sdd1 to /dev/md/pro�lox:0: Invalid argument
mdadm: /dev/md/pro�lox:0 assembled from 0 drives - not enough to start the array.

This looks like it is *after* to trying the --create command you give below.. It is best to report things in the order they happen, else you can confuse people (or get caught out!).

Yes, this was after.

Quoted 6 lines

Quoted 6 lines

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/sdd
mdadm: No arrays found in config file or automatically

pvscan and vgscan show nothing.

Quoted 8 lines

Quoted 6 lines

So I tried running mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 missing /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 --chunk=64

as it seemed that /dev/sdd1 failed to be added to the array. This did nothing.

It did not to nothing. It wrote a superblock to /dev/sdd1 and complained that it couldn't write to all the others --- didn't it?

There were multiple attempts to create. One wrote to just sdd1, another succeeded with all but sdd1.

Quoted 9 lines

Quoted 3 lines

dmesg contains:

md: invalid superblock checksum on sdd1

I guess that is why sdd1 was missing from 'ubuntu:0'. Though as I cannot tell if this happened before or after any of the various things reported above, it is hard to be sure.

The real mystery is why 'pvscan' reports nothing.

The original array was created with mdadm v2.6.7, and had a data offset of 264 sectors. After Simon's various attempts to --create, he ended up with data offset of 2048, using mdadm v3.1.4. The mdadm -E reports he posted to the list showed the 264 offset. We didn't realize the offset had been updated until somewhat later in our troubleshooting efforts.

In any case, pvscan couldn't see the LVM signature because it wasn't there (at offset 2048).

Quoted 9 lines

What about
pvscan --verbose

or

blkid -p /dev/md/ubuntu:0

or even

Quoted 1 line

   dd of=/dev/md/ubuntu:0 count=8 | od -c

Fortunately, Simon did have a copy of his LVM configuration. With the help of dd, strings, and grep, we did locate his LVM sig at the correct location on sdd1 (for data offset 264). After a number of attempts to bypass LVM and access his single LV with dmsetup (based on his backed up configuration, on the assembled new array less sdd1), I realized that the data offset was wrong on the recreated array, and went looking for the cause. I found your git commit that changed that logic last spring, and recommended that Simon revert to the default package for his ubuntu install, which is v2.6.7.

Simon has now attempted to recreate the array with v2.6.7, but the controller is throwing too many errors to succeed, and I suggested it was too flakey to trust any further. Based on the existence of the LVM sig on sdd1, I believe Simon's data is (mostly) intact, and only needs a successful create operation with a properly functioning controller. (He might also need to perform an lvm vgcfgrestore, but he has the necessary backup file.)

A new controller is on order.

Phil

Phil,
Thanks for filling in the gaps, I had forgotton quite how much help and assistance you had provided up to now. You're a real godsend.

To fill in some of the other pieces of info:
The original machine was called proxmox (KVM virtualisation machine) and I used an Ubuntu live cd to see if it was the software which was preventing progress or not. It does seem like there is data corruption in the machine name for some reason.

The original company I ordered the controller through sent me an email at 4pm saying that they could not fulfill the order, 4pm was too late for them to pick & pack so there was another 24hr turn around delay. The supermicro card should arrive here tomorrow and hopefully I'll be able to get a dd of each of the drives prior to Phil coming online.

For some reason blkid doesn't exist on my machine even though I have e2fs-utils installed. V weird.

Quoted 2 lines

For some reason blkid doesn't exist on my machine even though I have e2fs-utils installed. V weird.

FWIW, on my VPS running Ubuntu Server 10.10, the blkid executable is part of util-linux, and installed in /sbin. You might need to re-install it.

Quoted 2 lines · On Tue, 15 Feb 2011 14:37:55 -0500 Phil Turmel<[email protected]> wrote:

Quoted 2 lines

For some reason blkid doesn't exist on my machine even though I have e2fs-utils installed. V weird.

Weird is that you think it is somehow a "e2fs-utils" program.

dpkg -S blkid
...
util-linux: /sbin/blkid
...

Quoted 2 lines

FWIW, on my VPS running Ubuntu Server 10.10, the blkid executable is part of util-linux, and installed in /sbin. You might need to re-install it.

Quoted 2 lines · On Tue, 15 Feb 2011 14:37:55 -0500 Phil Turmel <[email protected]> wrote:

Quoted 2 lines

For some reason blkid doesn't exist on my machine even though I have e2fs-utils installed.  V weird.

Weird is that you think it is somehow a "e2fs-utils" program.

dpkg -S blkid
...
util-linux: /sbin/blkid
...

Quoted 1 line

Quoted 2 lines

FWIW, on my VPS running Ubuntu Server 10.10, the blkid executable is part of util-linux, and installed in /sbin.  You might need to re-install it.

--
With respect,
Roman

proxmox:~# dpkg -S blkid
libblkid1: /lib/libblkid.so.1.0
libblkid1: /lib/libblkid.so.1
libblkid1: /usr/share/doc/libblkid1/changelog.Debian.gz
libblkid1: /usr/share/doc/libblkid1/copyright
libblkid1: /usr/share/doc/libblkid1
e2fsprogs-dbg: /usr/lib/debug/sbin/blkid

proxmox:~# apt-get install e2fsprogs-dbg
Reading package lists... Done
Building dependency tree
Reading state information... Done
e2fsprogs-dbg is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I still can't blkid installed, even though I have e2fsprogs-dbg installed...

Quoted 16 lines · On Thu, 17 Feb 2011 15:10:32 +0000 Simon Mcnair<[email protected]> wrote:

proxmox:~# dpkg -S blkid
libblkid1: /lib/libblkid.so.1.0
libblkid1: /lib/libblkid.so.1
libblkid1: /usr/share/doc/libblkid1/changelog.Debian.gz
libblkid1: /usr/share/doc/libblkid1/copyright
libblkid1: /usr/share/doc/libblkid1
e2fsprogs-dbg: /usr/lib/debug/sbin/blkid

proxmox:~# apt-get install e2fsprogs-dbg
Reading package lists... Done
Building dependency tree
Reading state information... Done
e2fsprogs-dbg is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Quoted 1 line

I still can't blkid installed, even though I have e2fsprogs-dbg installed...

/usr/lib/debug/sbin/ is not in system 'path', so you won't be able to run the program just by entering 'blkid' into the shell prompt.
Either use the complete file name and path, e.g.

   # /usr/lib/debug/sbin/blkid --help

or symlink /usr/lib/debug/sbin/blkid to something like /usr/local/sbin/blkid.

Hi Roman,
Sorry to be so persistent on something that is not Linux raid specific, but I still can't get blkid to run, the message is:

proxmox:~# ls -la /usr/lib/debug/sbin/blkid
-rwx------ 1 root root 19887 2008-10-13 04:54 /usr/lib/debug/sbin/blkid

proxmox:~# /usr/lib/debug/sbin/blkid --help
-bash: /usr/lib/debug/sbin/blkid: cannot execute binary file

If I can't run it directly I'm pretty sure symlinking it will not make any difference. It's weird, normally when I apt-get things it modifies the path etc and does everything required to make it work, I would have thought if I installed a debug version of something it should add the debug path as well ?

Quoted 7 lines · Simon McNair

Hi Roman,
Sorry to be so persistent on something that is not Linux raid specific, but I still can't get blkid to run, the message is:

proxmox:~# ls -la /usr/lib/debug/sbin/blkid
-rwx------ 1 root root 19887 2008-10-13 04:54 /usr/lib/debug/sbin/blkid

Quoted 9 lines

proxmox:~# /usr/lib/debug/sbin/blkid --help
-bash: /usr/lib/debug/sbin/blkid: cannot execute binary file

If I can't run it directly I'm pretty sure symlinking it will not make any difference.  It's weird, normally when I apt-get things it modifies the path etc and does everything required to make it work, I would have thought if I installed a debug version of something it should add the debug path as well ?

I suspect that those are just the debug symbols - they're split off from the binary so they can be automatically pulled in when needed, but don't bloat the application itself.  From the package naming, the actual blkid binary should be in the e2fsprogs package - do you have that installed?

Cheers,
   Robin
--
    ___

   ( ' }     |       Robin Hill        <[email protected]> |
  / / )      | Little Jim says ....                            |
 // !!       |      "He fallen in de water !!"                 |

Yeah, It's installed :-)

Quoted 1 line · Simon Mcnair

Yeah, It's installed :-)

apt-get install --reinstall e2fsprogs

yay. Thanks very much. That worked. I don't quite understand why something should need a reinstall to work, but thanks v much.

Hi Neil,

Since Simon has responded, let me summarize the assistance I provided per his off-list request:

Quoted 33 lines · NeilBrown

Quoted 9 lines · On Thu, 10 Feb 2011 16:16:44 +0000 Simon McNair<[email protected]> wrote:

Hi all

I use a 3ware 9500-12 port sata card (JBOD) which will not work without a 128mb sodimm. The sodimm socket is flakey and the result is that the machine occasionally crashes. Yesterday I finally gave in and put together another
machine so that I can rsync between them. When I turned the machine on today to set up rync, the RAID array was not gone, but corrupted.
Typical...

Presumably the old machine was called 'ubuntu' and the new machine 'proølox'

Quoted 19 lines

I built the array in Aug 2010 using the following command:

mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{b,c,d,e,f,g,h,i,j,k}1 --chunk=64

Using LVM, I did the following:
pvscan
pvcreate -M2 /dev/md0
vgcreate lvm-raid /dev/md0
vgdisplay lvm-raid
vgscan
lvscan
lvcreate -v -l 100%VG -n RAID lvm-raid
lvdisplay /dev/lvm-raid/lvm0

I then formatted using:
mkfs -t ext4 -v -m .1 -b 4096 -E stride=16,stripe-width=144 /dev/lvm-raid/RAID

Quoted 8 lines

Quoted 6 lines

This worked perfectly since I created the array. Now mdadm is coming up with

proxmox:/dev/md# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/ubuntu:0

And it seems that ubuntu:0 have been successfully assembled.
It is missing one device for some reason (sdd1) but RAID can cope with that.

3ware card is compromised, with a loose buffer memory dimm. Some of its ECC errors were caught and reported in dmesg. Its likely, based on the loose memory socket, that many multiple-bit errors got through.

[trim /]

Quoted 8 lines

Quoted 5 lines

mdadm: no uptodate device for slot 8 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 9 of /dev/md/pro�lox:0
mdadm: failed to add /dev/sdd1 to /dev/md/pro�lox:0: Invalid argument
mdadm: /dev/md/pro�lox:0 assembled from 0 drives - not enough to start the array.

This looks like it is *after* to trying the --create command you give below.. It is best to report things in the order they happen, else you can confuse people (or get caught out!).

Yes, this was after.

Quoted 6 lines

Quoted 6 lines

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/sdd
mdadm: No arrays found in config file or automatically

pvscan and vgscan show nothing.

Quoted 8 lines

Quoted 6 lines

So I tried running mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 missing /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 --chunk=64

as it seemed that /dev/sdd1 failed to be added to the array. This did nothing.

It did not to nothing. It wrote a superblock to /dev/sdd1 and complained that it couldn't write to all the others --- didn't it?

There were multiple attempts to create. One wrote to just sdd1, another succeeded with all but sdd1.

Quoted 9 lines

Quoted 3 lines

dmesg contains:

md: invalid superblock checksum on sdd1

I guess that is why sdd1 was missing from 'ubuntu:0'. Though as I cannot tell if this happened before or after any of the various things reported above, it is hard to be sure.

The real mystery is why 'pvscan' reports nothing.

The original array was created with mdadm v2.6.7, and had a data offset of 264 sectors. After Simon's various attempts to --create, he ended up with data offset of 2048, using mdadm v3.1.4. The mdadm -E reports he posted to the list showed the 264 offset. We didn't realize the offset had been updated until somewhat later in our troubleshooting efforts.

In any case, pvscan couldn't see the LVM signature because it wasn't there (at offset 2048).

Quoted 9 lines

What about
pvscan --verbose

or

blkid -p /dev/md/ubuntu:0

or even

Quoted 1 line

   dd of=/dev/md/ubuntu:0 count=8 | od -c

Fortunately, Simon did have a copy of his LVM configuration. With the help of dd, strings, and grep, we did locate his LVM sig at the correct location on sdd1 (for data offset 264). After a number of attempts to bypass LVM and access his single LV with dmsetup (based on his backed up configuration, on the assembled new array less sdd1), I realized that the data offset was wrong on the recreated array, and went looking for the cause. I found your git commit that changed that logic last spring, and recommended that Simon revert to the default package for his ubuntu install, which is v2.6.7.

Simon has now attempted to recreate the array with v2.6.7, but the controller is throwing too many errors to succeed, and I suggested it was too flakey to trust any further. Based on the existence of the LVM sig on sdd1, I believe Simon's data is (mostly) intact, and only needs a successful create operation with a properly functioning controller. (He might also need to perform an lvm vgcfgrestore, but he has the necessary backup file.)

A new controller is on order.

Phil

latest update. A bit long I'm afraid...

hi, card installed and devices plugged in. Now have

sda   sda2  sdb1  sdc1  sde   sdf1  sdg1  sdi   sdj1  sdk1  sdl1  sdm1
sda1  sdb   sdc   sdd   sdf   sdg   sdh   sdj   sdk   sdl   sdm

proxmox:/home/simon# ./lsdrv.sh
Controller device @ pci0000:00/0000:00:01.0/0000:01:00.0 [mvsas]
  SCSI storage controller: Marvell Technology Group Ltd. MV64460/64461/64462 Sys                                         tem Controller, Revision B (rev 01)
    host4: /dev/sdf ATA Hitachi HDS72101 {SN: GTA000PAGABXRA}
    host4: /dev/sdg ATA Hitachi HDS72101 {SN: GTA000PAGAA5DA}
    host4: /dev/sdh ATA Hitachi HDS72101 {SN: GTA000PAG9NL9A}
    host4: /dev/sdi ATA Hitachi HDS72101 {SN: GTA000PAGA8V4A}
    host4: /dev/sdj ATA Hitachi HDS72101 {SN: GTD000PAGMT9GD}
    host4: /dev/sdk ATA Hitachi HDS72101 {SN: GTG000PAG18BJC}
    host4: /dev/sdl ATA Hitachi HDS72101 {SN: GTG000PAG1DPLC}
    host4: /dev/sdm ATA Hitachi HDS72101 {SN: GTA000PAG7WMEA}
Controller device @ pci0000:00/0000:00:1a.7/usb1/1-4/1-4.1/1-4.1.1/1-4.1.1:1.0 [                                         usb-storage]
  Bus 001 Device 007: ID 0424:2228 Standard Microsystems Corp. 9-in-2 Card Reade                                         r {SN: 08050920003A}
    host9: /dev/sdd Generic Flash HS-CF
    host9: /dev/sde Generic Flash HS-COMBO
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.0 [ahci]
  SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller                                          (rev 03)
    host7: [Empty]
    host8: [Empty]
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.1 [pata_jmicron]
  IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (r                                         ev 03)
    host5: [Empty]
    host6: [Empty]

Controller device @ pci0000:00/0000:00:1f.2 [ata_piix]
  IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Contro                                         ller #1
    host0: /dev/sda ATA STM3500418AS {SN: 9VM3QJ5C}
    host1: /dev/sr0 Optiarc DVD RW AD-5240S
Controller device @ pci0000:00/0000:00:1f.5 [ata_piix]
  IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Contro                                         ller #2
    host2: /dev/sdb ATA Hitachi HDS72101 {SN: GTD000PAGMT8DD}
    host3: /dev/sdc ATA Hitachi HDS72101 {SN: GTG000PAG04V0C}

Phil,
sg3-utils did the job :-)
Sorry for doubting you.

    host6: /dev/sdd AMCC 9500S-12 DISK {SN: PAGA8V4A3B8378002254}
    host6: /dev/sde AMCC 9500S-12 DISK {SN: PAG9NL9A3B8387004C88}
    host6: /dev/sdf AMCC 9500S-12 DISK {SN: PAGAA5DA3B8396001AE0}
    host6: /dev/sdg AMCC 9500S-12 DISK {SN: PAGABXRA3B83A0005EFB}
    host6: /dev/sdh AMCC 9500S-12 DISK {SN: PAG7WMEA3B83AA003E4F}
    host6: /dev/sdi AMCC 9500S-12 DISK {SN: PAG1DPLC3B83B40026E6}
    host6: /dev/sdj AMCC 9500S-12 DISK {SN: PAG18BJC3B83C3004760}
    host6: /dev/sdk AMCC 9500S-12 DISK {SN: PAGMT9GD3B83CD004B18}
    host6: /dev/sdl AMCC 9500S-12 DISK {SN: PAGMT8DD3B83D70021FF}
    host6: /dev/sdm AMCC 9500S-12 DISK {SN: PAG04V0C3B83E100ACA0}

Simon

proxmox:/home/simon# parted -l
Model: ATA STM3500418AS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End    Size   Type     File system  Flags
 1      32.8kB  537MB  537MB  primary  ext3         boot
 2      537MB   500GB  500GB  primary               lvm

Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? c

Error: /dev/sdh: unrecognised disk label

Error: /dev/sdi: unrecognised disk label

proxmox:/home/simon# for x in sd{d..m} ; do parted -s /dev/$x unit s
print ; done
Model: AMCC 9500S-12 DISK (scsi)
Disk /dev/sdd: 1953103872s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Good morning, Simon,

Quoted 5 lines · Simon McNair

latest update. A bit long I'm afraid...

hi, card installed and devices plugged in. Now have

sda   sda2  sdb1  sdc1  sde   sdf1  sdg1  sdi   sdj1  sdk1  sdl1  sdm1
sda1  sdb   sdc   sdd   sdf   sdg   sdh   sdj   sdk   sdl   sdm

Hmmm.

Quoted 23 lines

proxmox:/home/simon# ./lsdrv.sh
Controller device @ pci0000:00/0000:00:01.0/0000:01:00.0 [mvsas]
   SCSI storage controller: Marvell Technology Group Ltd. MV64460/64461/64462 Sys                                         tem Controller, Revision B (rev 01)
     host4: /dev/sdf ATA Hitachi HDS72101 {SN: GTA000PAGABXRA}
     host4: /dev/sdg ATA Hitachi HDS72101 {SN: GTA000PAGAA5DA}
     host4: /dev/sdh ATA Hitachi HDS72101 {SN: GTA000PAG9NL9A}
     host4: /dev/sdi ATA Hitachi HDS72101 {SN: GTA000PAGA8V4A}
     host4: /dev/sdj ATA Hitachi HDS72101 {SN: GTD000PAGMT9GD}
     host4: /dev/sdk ATA Hitachi HDS72101 {SN: GTG000PAG18BJC}
     host4: /dev/sdl ATA Hitachi HDS72101 {SN: GTG000PAG1DPLC}
     host4: /dev/sdm ATA Hitachi HDS72101 {SN: GTA000PAG7WMEA}
Controller device @ pci0000:00/0000:00:1a.7/usb1/1-4/1-4.1/1-4.1.1/1-4.1.1:1.0 [                                         usb-storage]
   Bus 001 Device 007: ID 0424:2228 Standard Microsystems Corp. 9-in-2 Card Reade                                         r {SN: 08050920003A}
     host9: /dev/sdd Generic Flash HS-CF
     host9: /dev/sde Generic Flash HS-COMBO
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.0 [ahci]
   SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller                                          (rev 03)
     host7: [Empty]
     host8: [Empty]
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.1 [pata_jmicron]
   IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (r                                         ev 03)
     host5: [Empty]
     host6: [Empty]

Quoted 8 lines

Controller device @ pci0000:00/0000:00:1f.2 [ata_piix]
   IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Contro                                         ller #1
     host0: /dev/sda ATA STM3500418AS {SN: 9VM3QJ5C}
     host1: /dev/sr0 Optiarc DVD RW AD-5240S
Controller device @ pci0000:00/0000:00:1f.5 [ata_piix]
   IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Contro                                         ller #2
     host2: /dev/sdb ATA Hitachi HDS72101 {SN: GTD000PAGMT8DD}
     host3: /dev/sdc ATA Hitachi HDS72101 {SN: GTG000PAG04V0C}

Good thing we recorded the serial numbers. From an earlier run of "lsdrv":

Quoted 15 lines

Phil,
sg3-utils did the job :-)
Sorry for doubting you.

     host6: /dev/sdd AMCC 9500S-12 DISK {SN: PAGA8V4A3B8378002254}
     host6: /dev/sde AMCC 9500S-12 DISK {SN: PAG9NL9A3B8387004C88}
     host6: /dev/sdf AMCC 9500S-12 DISK {SN: PAGAA5DA3B8396001AE0}
     host6: /dev/sdg AMCC 9500S-12 DISK {SN: PAGABXRA3B83A0005EFB}
     host6: /dev/sdh AMCC 9500S-12 DISK {SN: PAG7WMEA3B83AA003E4F}
     host6: /dev/sdi AMCC 9500S-12 DISK {SN: PAG1DPLC3B83B40026E6}
     host6: /dev/sdj AMCC 9500S-12 DISK {SN: PAG18BJC3B83C3004760}
     host6: /dev/sdk AMCC 9500S-12 DISK {SN: PAGMT9GD3B83CD004B18}
     host6: /dev/sdl AMCC 9500S-12 DISK {SN: PAGMT8DD3B83D70021FF}
     host6: /dev/sdm AMCC 9500S-12 DISK {SN: PAG04V0C3B83E100ACA0}

Quoted 1 line

Simon

I don't know why the serial numbers are formatted differently, but we can still tell them apart (the eight characters starting with "PAG").

So, our device order in your new setup is: [ihgfmlkjbc], where /dev/sdi corresponds to the original report's /dev/sdd, which matches the sig grep in your other note.

Another note: The controller for sd[abc] is still showing ata_piix as its controller. That means you cannot hot-plug those ports. If you change your BIOS to AHCI mode instead of "Compatibility" or "Emulation", the full-featured ahci driver will run those ports. Not urgent, but I highly recommend it.

Quoted 11 lines

proxmox:/home/simon# parted -l
Model: ATA STM3500418AS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags

  1      32.8kB  537MB  537MB  primary  ext3         boot
  2      537MB   500GB  500GB  primary               lvm

Quoted 3 lines

Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? c

The 3ware controller must have reserved some space at the end of each drive for its own use. Didn't know it'd do that. You will have to fix that.

[trim /]

Quoted 3 lines

Error: /dev/sdh: unrecognised disk label

Error: /dev/sdi: unrecognised disk label

It seems the flaky controller took out these partition tables. Hope that's all it got. They'll have to be re-created with parted.

Please run parted on each of the ten drives, and make sure they end up like so:

Quoted 7 lines

proxmox:/home/simon# for x in sd{d..m} ; do parted -s /dev/$x unit s
print ; done
Model: AMCC 9500S-12 DISK (scsi)
Disk /dev/sdd: 1953103872s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Quoted 2 lines

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Make sure you request "unit s" before your other commands so we can make sure it matches.

When you think you are done, check them all:

for x in /dev/sd{i,h,g,f,m,l,k,j,b,c} ; do parted -s $x unit s print ; done

After that, create:

mdadm --create --verbose --assume-clean /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{i,h,g,f,m,l,k,j,b,c}1 --chunk=64

And finally:

pvscan --verbose

Phil

Hi Phil,
A couple of questions please.

Quoted 7 lines

Quoted 1 line

Simon

I don't know why the serial numbers are formatted differently, but we can still tell them apart (the eight characters starting with "PAG").

So, our device order in your new setup is: [ihgfmlkjbc], where /dev/sdi corresponds to the original report's /dev/sdd, which matches the sig grep in your other note.

Another note: The controller for sd[abc] is still showing ata_piix as its controller. That means you cannot hot-plug those ports. If you change your BIOS to AHCI mode instead of "Compatibility" or "Emulation", the full-featured ahci driver will run those ports. Not urgent, but I highly recommend it.

Will do that now, before I forget

Quoted 7 lines

Quoted 3 lines

Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? c

The 3ware controller must have reserved some space at the end of each drive for its own use. Didn't know it'd do that. You will have to fix that.

[trim /]

Do you have any suggestions on how I can fix that ? I don't have a clue

when I was trying to figure out the command for this using 'man parted' I came across this:
" rescue start end
Rescue a lost partition that was located somewhere between start and end. If a partition is found, parted will ask if you want to create an entry for it in the partition table."
Is it worth trying ?

I originally created the partitions like so:
parted -s /dev/sdb rm 1
parted -s /dev/sdb mklabel gpt
parted -s --align optimal /dev/sdb mkpart primary ext4 .512 100% parted -s /dev/sdb set 1 raid on
parted -s /dev/sdb align-check optimal 1

so to recreate the above I would do:
parted -s /dev/sdb mkpart primary ext4 2048s 1953101823s parted -s /dev/sdc mkpart primary ext4 2048s 1953101823s parted -s /dev/sdf mkpart primary ext4 2048s 1953101823s parted -s /dev/sdg mkpart primary ext4 2048s 1953101823s parted -s /dev/sdh mkpart primary ext4 2048s 1953101823s parted -s /dev/sdi mkpart primary ext4 2048s 1953101823s parted -s /dev/sdj mkpart primary ext4 2048s 1953101823s parted -s /dev/sdk mkpart primary ext4 2048s 1953101823s parted -s /dev/sdl mkpart primary ext4 2048s 1953101823s parted -s /dev/sdm mkpart primary ext4 2048s 1953101823s

I'm guessing the backups that I want to do can wait until any potential fsck ?

sorry if the questions are dumb but I'm not sure what I'm doing and I'd rather ask more questions than fewer and understand the implications of what I'm doing.

Quoted 2 lines · Simon McNair

Hi Phil,
A couple of questions please.

[trim /]

Quoted 8 lines

Quoted 7 lines

Quoted 1 line

Simon

I don't know why the serial numbers are formatted differently, but we can still tell them apart (the eight characters starting with "PAG").

So, our device order in your new setup is: [ihgfmlkjbc], where /dev/sdi corresponds to the original report's /dev/sdd, which matches the sig grep in your other note.

Another note: The controller for sd[abc] is still showing ata_piix as its controller. That means you cannot hot-plug those ports. If you change your BIOS to AHCI mode instead of "Compatibility" or "Emulation", the full-featured ahci driver will run those ports. Not urgent, but I highly recommend it.

Will do that now, before I forget

Hot-pluggability with suitable trays is very handy! :)

[trim /]

Quoted 8 lines

Quoted 7 lines

Quoted 3 lines

Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? c

The 3ware controller must have reserved some space at the end of each drive for its own use. Didn't know it'd do that. You will have to fix that.

[trim /]

Do you have any suggestions on how I can fix that ? I don't have a clue

Just do 'parted /dev/sd?' and on the ones it offers to fix, say yes. Then request 'unit s' and 'print' to verify that it is correct.

[trim /]

Quoted 5 lines

when I was trying to figure out the command for this using 'man parted' I came across this:
" rescue start end
Rescue a lost partition that was located somewhere between start and end. If a partition is found, parted will ask if you want to create an entry for it in the partition table."
Is it worth trying ?

Nah. That's for when you don't know exactly where the partition is. We know.

Quoted 7 lines

I originally created the partitions like so:
parted -s /dev/sdb rm 1
parted -s /dev/sdb mklabel gpt
parted -s --align optimal /dev/sdb mkpart primary ext4 .512 100% parted -s /dev/sdb set 1 raid on
parted -s /dev/sdb align-check optimal 1

Quoted 11 lines

so to recreate the above I would do:
parted -s /dev/sdb mkpart primary ext4 2048s 1953101823s parted -s /dev/sdc mkpart primary ext4 2048s 1953101823s parted -s /dev/sdf mkpart primary ext4 2048s 1953101823s parted -s /dev/sdg mkpart primary ext4 2048s 1953101823s parted -s /dev/sdh mkpart primary ext4 2048s 1953101823s parted -s /dev/sdi mkpart primary ext4 2048s 1953101823s parted -s /dev/sdj mkpart primary ext4 2048s 1953101823s parted -s /dev/sdk mkpart primary ext4 2048s 1953101823s parted -s /dev/sdl mkpart primary ext4 2048s 1953101823s parted -s /dev/sdm mkpart primary ext4 2048s 1953101823s

Only recreate the partition tables where you have to, i.e., the 'Fix' option above didn't work. And don't specify a filesystem.

Probably just /dev/sdh and /dev/sdi. Like so, though:

parted -s /dev/sdh mklabel gpt mkpart primary 2048s 1953101823s set 1 raid on parted -s /dev/sdi mklabel gpt mkpart primary 2048s 1953101823s set 1 raid on

Quoted 1 line

I'm guessing the backups that I want to do can wait until any potential fsck ?

Do an 'fsck -N' first, and if it passes, or has few errors, mount the filesystem readonly and grab your backup. Then let fsck have at it for real. If anything gets fixed, compare your backup from the read-only fs to the fixed fs.

Given your flaky old controller, I expect there'll be *some* problems.

Quoted 1 line

sorry if the questions are dumb but I'm not sure what I'm doing and I'd rather ask more questions than fewer and understand the implications of what I'm doing.

Oh, no. You are right to be paranoid. If anything looks funny, stop.

Phil

parted -l gives

Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? fix
Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 421296 blocks) or continue with the current setting?
Fix/Ignore?

I'm guessing ignore ?

Quoted 9 lines · Simon McNair

parted -l gives

Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? fix
Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 421296 blocks) or continue with the current setting?
Fix/Ignore?

Quoted 1 line

I'm guessing ignore ?

Either should work. If you 'Fix' for all of the drives, you could later 'grow' your array to include the extra space.

Quoted 10 lines · Phil Turmel

Quoted 9 lines · Simon McNair

parted -l gives

Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? fix
Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 421296 blocks) or continue with the current setting?
Fix/Ignore?

Quoted 2 lines

Quoted 1 line

I'm guessing ignore ?

Either should work. If you 'Fix' for all of the drives, you could later 'grow' your array to include the extra space.

It's vital that the start sector be 2048, though. Please do recheck after all are done.

Phil

have done this.... Results are:

proxmox:/home/simon# ./lsdrv.sh
Controller device @ pci0000:00/0000:00:01.0/0000:01:00.0 [mvsas]
  SCSI storage controller: Marvell Technology Group Ltd. MV64460/64461/64462 System Controller, Revision B (rev 01)
    host0: /dev/sdf ATA Hitachi HDS72101 {SN: GTA000PAGABXRA}
    host0: /dev/sdg ATA Hitachi HDS72101 {SN: GTA000PAGAA5DA}
    host0: /dev/sdh ATA Hitachi HDS72101 {SN: GTA000PAG9NL9A}
    host0: /dev/sdi ATA Hitachi HDS72101 {SN: GTA000PAGA8V4A}
    host0: /dev/sdj ATA Hitachi HDS72101 {SN: GTD000PAGMT9GD}
    host0: /dev/sdk ATA Hitachi HDS72101 {SN: GTG000PAG18BJC}
    host0: /dev/sdl ATA Hitachi HDS72101 {SN: GTG000PAG1DPLC}
    host0: /dev/sdm ATA Hitachi HDS72101 {SN: GTA000PAG7WMEA}
Controller device @ pci0000:00/0000:00:1a.7/usb1/1-4/1-4.1/1-4.1.1/1-4.1.1:1.0 [usb-storage]
  Bus 001 Device 007: ID 0424:2228 Standard Microsystems Corp. 9-in-2 Card Reader {SN: 08050920003A}
    host11: /dev/sdb Generic Flash HS-CF
    host11: /dev/sdc Generic Flash HS-COMBO
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.0 [ahci]
  SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 03)
    host9: [Empty]
    host10: [Empty]
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.1 [pata_jmicron]
  IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 03)
    host1: [Empty]
    host2: [Empty]
Controller device @ pci0000:00/0000:00:1f.2 [ahci]
  SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
    host3: /dev/sda ATA STM3500418AS {SN: 9VM3QJ5C}
    host4: /dev/sr0 Optiarc DVD RW AD-5240S
    host5: [Empty]
    host6: [Empty]
    host7: /dev/sdd ATA Hitachi HDS72101 {SN: GTD000PAGMT8DD}
    host8: /dev/sde ATA Hitachi HDS72101 {SN: GTG000PAG04V0C}

enabling achi has switched the drive letters around for a couple of drives again

parted list is:

proxmox:/home/simon# for x in /dev/sd{i,h,g,f,m,l,k,j,d,e} ; do parted -s $x unit s print ; done
Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdi: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdh: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdg: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdf: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdm: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdl: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdk: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdj: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdd: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sde: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name     Flags
 1      2048s  1953101823s  1953099776s               primary  raid

is it now:

mdadm --create --verbose --assume-clean /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{i,h,g,f,m,l,k,j,d,e}1 --chunk=64

Quoted 111 lines · Simon McNair

have done this.... Results are:

proxmox:/home/simon# ./lsdrv.sh
Controller device @ pci0000:00/0000:00:01.0/0000:01:00.0 [mvsas]
   SCSI storage controller: Marvell Technology Group Ltd. MV64460/64461/64462 System Controller, Revision B (rev 01)
     host0: /dev/sdf ATA Hitachi HDS72101 {SN: GTA000PAGABXRA}
     host0: /dev/sdg ATA Hitachi HDS72101 {SN: GTA000PAGAA5DA}
     host0: /dev/sdh ATA Hitachi HDS72101 {SN: GTA000PAG9NL9A}
     host0: /dev/sdi ATA Hitachi HDS72101 {SN: GTA000PAGA8V4A}
     host0: /dev/sdj ATA Hitachi HDS72101 {SN: GTD000PAGMT9GD}
     host0: /dev/sdk ATA Hitachi HDS72101 {SN: GTG000PAG18BJC}
     host0: /dev/sdl ATA Hitachi HDS72101 {SN: GTG000PAG1DPLC}
     host0: /dev/sdm ATA Hitachi HDS72101 {SN: GTA000PAG7WMEA}
Controller device @ pci0000:00/0000:00:1a.7/usb1/1-4/1-4.1/1-4.1.1/1-4.1.1:1.0 [usb-storage]
   Bus 001 Device 007: ID 0424:2228 Standard Microsystems Corp. 9-in-2 Card Reader {SN: 08050920003A}
     host11: /dev/sdb Generic Flash HS-CF
     host11: /dev/sdc Generic Flash HS-COMBO
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.0 [ahci]
   SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 03)
     host9: [Empty]
     host10: [Empty]
Controller device @ pci0000:00/0000:00:1c.4/0000:04:00.1 [pata_jmicron]
   IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 03)
     host1: [Empty]
     host2: [Empty]
Controller device @ pci0000:00/0000:00:1f.2 [ahci]
   SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
     host3: /dev/sda ATA STM3500418AS {SN: 9VM3QJ5C}
     host4: /dev/sr0 Optiarc DVD RW AD-5240S
     host5: [Empty]
     host6: [Empty]
     host7: /dev/sdd ATA Hitachi HDS72101 {SN: GTD000PAGMT8DD}
     host8: /dev/sde ATA Hitachi HDS72101 {SN: GTG000PAG04V0C}

enabling achi has switched the drive letters around for a couple of drives again

parted list is:

proxmox:/home/simon# for x in /dev/sd{i,h,g,f,m,l,k,j,d,e} ; do parted -s $x unit s print ; done
Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdi: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdh: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdg: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdf: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdm: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdl: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdk: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdj: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sdd: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

Quoted 7 lines

Model: ATA Hitachi HDS72101 (scsi)
Disk /dev/sde: 1953525168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 1953101823s 1953099776s primary raid

All looks good.

Quoted 3 lines

is it now:

mdadm --create --verbose --assume-clean /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{i,h,g,f,m,l,k,j,d,e}1 --chunk=64

Yes.

Phil

pvscan is:

proxmox:/home/simon# pvscan --verbose
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
    Walking through all physical volumes
  PV /dev/sda2   VG pve        lvm2 [465.26 GB / 4.00 GB free]
  PV /dev/md0    VG lvm-raid   lvm2 [8.19 TB / 0    free]
  Total: 2 [655.05 GB] / in use: 2 [655.05 GB] / in no VG: 0 [0   ]

lvm-raid is there ....

proxmox:/home/simon# fsck.ext4 -n /dev/md0

e2fsck 1.41.3 (12-Oct-2008)
fsck.ext4: Superblock invalid, trying backup blocks... Superblock has an invalid ext3 journal (inode 8). Clear? no

fsck.ext4: Illegal inode number while checking ext3 journal for /dev/md0

I thought I needed to run fsck against the lvm group ? I used to do... fsck.ext4 /dev/lvm-raid/RAID so do I need to mount it or something ?

Quoted 10 lines · Simon McNair

pvscan is:

proxmox:/home/simon# pvscan --verbose
     Wiping cache of LVM-capable devices
     Wiping internal VG cache
     Walking through all physical volumes
   PV /dev/sda2   VG pve        lvm2 [465.26 GB / 4.00 GB free]
   PV /dev/md0    VG lvm-raid   lvm2 [8.19 TB / 0    free]
   Total: 2 [655.05 GB] / in use: 2 [655.05 GB] / in no VG: 0 [0   ]

lvm-raid is there ....

Very good.

Quoted 1 line

proxmox:/home/simon# fsck.ext4 -n /dev/md0

Uh, no.

Quoted 8 lines

e2fsck 1.41.3 (12-Oct-2008)
fsck.ext4: Superblock invalid, trying backup blocks... Superblock has an invalid ext3 journal (inode 8). Clear? no

fsck.ext4: Illegal inode number while checking ext3 journal for /dev/md0

I thought I needed to run fsck against the lvm group ? I used to do... fsck.ext4 /dev/lvm-raid/RAID so do I need to mount it or something ?

vgscan --verbose

lvscan --verbose

Then either:

fsck -N /dev/mapper/lvm-raid-RAID

or:

fsck -N /dev/lvm-raid/RAID

depends on what udev is doing.

Phil

proxmox:/home/simon# vgscan --verbose
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
    Finding all volume groups
    Finding volume group "pve"
  Found volume group "pve" using metadata type lvm2
    Finding volume group "lvm-raid"
  Found volume group "lvm-raid" using metadata type lvm2
proxmox:/home/simon#
proxmox:/home/simon# lvscan --verbose
    Finding all logical volumes
  ACTIVE            '/dev/pve/swap' [11.00 GB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GB] inherit
  ACTIVE            '/dev/pve/data' [354.26 GB] inherit
  inactive          '/dev/lvm-raid/RAID' [8.19 TB] inherit

proxmox:/home/simon# vgchange -ay
  3 logical volume(s) in volume group "pve" now active
  1 logical volume(s) in volume group "lvm-raid" now active

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/lvm-raid-RAID

e2fsck 1.41.3 (12-Oct-2008)
fsck.ext4: No such file or directory while trying to open /dev/mapper/lvm-raid-RAID

The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:

    e2fsck -b 8193 <device>

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/
control         lvm--raid-RAID  pve-data        pve-root        pve-swap

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/lvm--raid-RAID
e2fsck 1.41.3 (12-Oct-2008)
/dev/mapper/lvm--raid-RAID has unsupported feature(s): FEATURE_I31
e2fsck: Get a newer version of e2fsck!

my version of e2fsck always worked before ?

Quoted 17 lines · Simon McNair

proxmox:/home/simon# vgscan --verbose
     Wiping cache of LVM-capable devices
     Wiping internal VG cache
   Reading all physical volumes.  This may take a while...
     Finding all volume groups
     Finding volume group "pve"
   Found volume group "pve" using metadata type lvm2
     Finding volume group "lvm-raid"
   Found volume group "lvm-raid" using metadata type lvm2
proxmox:/home/simon#
proxmox:/home/simon# lvscan --verbose
     Finding all logical volumes
   ACTIVE            '/dev/pve/swap' [11.00 GB] inherit
   ACTIVE            '/dev/pve/root' [96.00 GB] inherit
   ACTIVE            '/dev/pve/data' [354.26 GB] inherit
   inactive          '/dev/lvm-raid/RAID' [8.19 TB] inherit

Quoted 3 lines

proxmox:/home/simon# vgchange -ay
   3 logical volume(s) in volume group "pve" now active
   1 logical volume(s) in volume group "lvm-raid" now active

Heh. Figures.

Quoted 1 line

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/lvm-raid-RAID

Actually, I wanted you to try with a capital N. Lower case 'n' is similar, but not quite the same.

Quoted 11 lines

e2fsck 1.41.3 (12-Oct-2008)
fsck.ext4: No such file or directory while trying to open /dev/mapper/lvm-raid-RAID

The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:

     e2fsck -b 8193<device>

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/
control         lvm--raid-RAID  pve-data        pve-root        pve-swap

Strange. I guess it does that to distinguish dashes in the VG name from dashes between VG and LV names.

Quoted 6 lines

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/lvm--raid-RAID
e2fsck 1.41.3 (12-Oct-2008)
/dev/mapper/lvm--raid-RAID has unsupported feature(s): FEATURE_I31
e2fsck: Get a newer version of e2fsck!

my version of e2fsck always worked before ?

v1.41.14 was release 7 weeks ago. But, I suspect there's corruption in the superblock. Do you still have your disk images tucked away somewhere safe?

If so, try:

1) The '-b' option to e2fsck. We need to experiment with '-n -b offset' to find the alternate superblock. Trying 'offset' = to 8193, 16384, and 32768, per the man-page.

2) A newer e2fsprogs.

Finally,
3) mount -r /dev/lvm-raid/RAID /mnt/whatever

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Phil,
Jeez I'm having a bad week (my windows 7x64 machine has just started randomly crashing, my Thecus n5200 is playing up and the weather has been dire so I've not been able to put my new shed up...oh and I have ongoing 'other' issues as you're well aware ;-)) The thecus n5200 has 5x2TB hdd's. I wiped out the existing raid5 array to create a jbod span of 10tb in order to hold my 9tb of backups. The Thecus has had a hissy fit and I've had to set the process off again, so you can bet it'll be a day or two before it get's the drives formatted (it's not a very powerful nas), then I'll do the backups, then I'll try as you suggested.

Thanks for the ongoing assistance.

Quoted 2 lines · Simon McNair

Phil,
Jeez I'm having a bad week (my windows 7x64 machine has just started randomly crashing, my Thecus n5200 is playing up and the weather has been dire so I've not been able to put my new shed up...oh and I have ongoing 'other' issues as you're well aware ;-)) The thecus n5200 has 5x2TB hdd's. I wiped out the existing raid5 array to create a jbod span of 10tb in order to hold my 9tb of backups. The Thecus has had a hissy fit and I've had to set the process off again, so you can bet it'll be a day or two before it get's the drives formatted (it's not a very powerful nas), then I'll do the backups, then I'll try as you suggested.

That's fine.

Quoted 2 lines

Thanks for the ongoing assistance.

No problem.

Phil

I just went to bed and one last question popped in to my mind. Since there is a fair timezone gap I thought I'd be presumptuous and ask it in the hope I can turn it around a bit quicker in the morning.

My suspicion is that once I have 5 formatted 2tb drives I may be lucky to get 10x 1tb dd images on to it. Can I feed the dd process in to tar, bzip2, zip or something else which will give me enough space to fit the images on ?

Will I get more usable space from 5x2tb partitions or from 1xspanned volume ? (the thecus pretty much only allows you to create raid volumes so a jbod needs to be 5x1tb arrays or 1 spanned volume or stripe).

Quoted 9 lines · Simon Mcnair

I just went to bed and one last question popped in to my mind. Since there is a fair timezone gap I thought I'd be presumptuous and ask it in the hope I can turn it around a bit quicker in the morning.

My suspicion is that once I have 5 formatted 2tb drives I may be lucky to get 10x 1tb dd images on to it. Can I feed the dd process in to tar, bzip2, zip or something else which will give me enough space to fit the images on ?

Quoted 4 lines

Will I get more usable space from 5x2tb partitions or from 1xspanned volume ? (the thecus pretty much only allows you to create raid volumes so a jbod needs to be 5x1tb arrays or 1 spanned volume or stripe).

I'd use one spanned volume, and gzip.  I'd simultaneously generate an md5sum while streaming to your thecus.  A script like so:

#! /bin/bash
#
function usage() {
       printf "Usage:\n\t%s devname\n\n" "basename \"$0\""
       printf "'devname' must be a relative path in /dev/ to the desired block device.\n" exit 1
}

# Verify the supplied name is a device
test -b "/dev/$1" || usage

# Convert path separators and spaces into dashes
outfile="echo \"$1\" |sed -r -e 's:[ /]+:-:g'"

# Create a side-stream for computing the MD5 of the data read fifo=mktemp -u
mkfifo $fifo || exit
md5sum -b <$fifo >/mnt/thecus/$outfile.md5 &

# Read the device and compress it
dd if="/dev/$1" bs=1M | tee 2>$fifo | gzip >/mnt/thecus/$outfile.gz

# Wait for the background task to close
wait

Phil,
After a couple of attempts I realised that all I needed to do was ./backup.sh sdi rather than specifying ./backup.sh /dev/sdi schoolboy error huh, rtfm.

I tried modifying the code, so I could kick them all off at once, but I wanted to check that this would work from a multithreaded/sane perspective (yes I know it's a bit of IO, but iotop seems to infer I'm only getting 10M/s throughput from the disk anyway).

is this code kosha ?

#! /bin/bash
#
function usage() {
printf "Usage:\n\t%s devname\n\n" "basename \"$0\"" printf "'devname' must be a relative path in /dev/ to the
desired block device.\n"
exit 1
}

# Verify the supplied name is a device
test -b "/dev/$1" || usage

# Convert path separators and spaces into dashes
outfile="echo \"$1\" |sed -r -e 's:[ /]+:-:g'"

# Create a side-stream for computing the MD5 of the data read fifo=mktemp -u
mkfifo $fifo || exit
md5sum -b <$fifo >$2/$outfile.md5 &

# Read the device and compress it
dd if="/dev/$1" bs=1M | tee 2>$fifo | gzip >$2/$outfile.gz

# Wait for the background task to close
wait

I was a little concerned as I didn't see the hdd drive LED light up for my attempt even though the file was growing nicely on my CIFS share. I'm dumping it on the 5x2TB disks (JBOD) in my windows 7 box as my Thecus is not a happy chappy and I need time to mount the DOM module in another machine to find out why.

Quoted 9 lines · Simon Mcnair

Phil,
After a couple of attempts I realised that all I needed to do was ./backup.sh sdi rather than specifying ./backup.sh /dev/sdi schoolboy error huh, rtfm.

I tried modifying the code, so I could kick them all off at once, but I wanted to check that this would work from a multithreaded/sane perspective (yes I know it's a bit of IO, but iotop seems to infer I'm only getting 10M/s throughput from the disk anyway).

That sounds suspiciously like a 100 meg ethernet bottleneck, so parallel operation won't help. If so, this'll take a very long time.

Quoted 1 line

is this code kosha ?

Looks pretty good. I've attached a slight update with your changes and a bugfix.

Quoted 23 lines

#! /bin/bash
#
function usage() {
printf "Usage:\n\t%s devname\n\n" "basename \"$0\"" printf "'devname' must be a relative path in /dev/ to the
desired block device.\n"
exit 1
}

# Verify the supplied name is a device
test -b "/dev/$1" || usage

# Convert path separators and spaces into dashes
outfile="echo \"$1\" |sed -r -e 's:[ /]+:-:g'"

# Create a side-stream for computing the MD5 of the data read fifo=mktemp -u
mkfifo $fifo || exit
md5sum -b <$fifo >$2/$outfile.md5 &

# Read the device and compress it
dd if="/dev/$1" bs=1M | tee 2>$fifo | gzip >$2/$outfile.gz

Quoted 8 lines

# Wait for the background task to close
wait

I was a little concerned as I didn't see the hdd drive LED light up for my attempt even though the file was growing nicely on my CIFS share. I'm dumping it on the 5x2TB disks (JBOD) in my windows 7 box as my Thecus is not a happy chappy and I need time to mount the DOM module in another machine to find out why.

If you're willing to dismantle the thecus, directly connecting its drives to your crippled system will make things go much faster. You've got four empty motherboard SATA ports. You just have to watch out for the power load.

Phil
<block2gz.sh>

Phil, I think I need to connect them to the crippled machine directly, I was just trying to leave it alone, I know that doesn't make much sense but I'm more familiar with windows than Linux. The cards that I have are all connected at gigabit and bonded. That machine has two ports, so does the other. I'm running two asus p6ts (1 deluxe and 1 se) with i7 920's

I cant understand why it's so slow though I have seen some reports that the supermicro card is quite slow.
Simon
Ps thanks for the bugfixes and updates
Simon

Quoted 53 lines · Phil Turmel

Quoted 9 lines · Simon Mcnair

Phil,
After a couple of attempts I realised that all I needed to do was ./backup.sh sdi rather than specifying ./backup.sh /dev/sdi  schoolboy error huh, rtfm.

I tried modifying the code, so I could kick them all off at once, but I wanted to check that this would work from a multithreaded/sane perspective (yes I know it's a bit of IO, but iotop seems to infer I'm only getting 10M/s throughput from the disk anyway).

That sounds suspiciously like a 100 meg ethernet bottleneck, so parallel operation won't help.  If so, this'll take a very long time.

Quoted 1 line

is this code kosha ?

Looks pretty good.  I've attached a slight update with your changes and a bugfix.

Quoted 23 lines

#! /bin/bash
#
function usage() {
      printf "Usage:\n\t%s devname\n\n" "basename \"$0\"" printf "'devname' must be a relative path in /dev/ to the desired block device.\n"
      exit 1
}

# Verify the supplied name is a device
test -b "/dev/$1" || usage

# Convert path separators and spaces into dashes
outfile="echo \"$1\" |sed -r -e 's:[ /]+:-:g'"

# Create a side-stream for computing the MD5 of the data read fifo=mktemp -u
mkfifo $fifo || exit
md5sum -b <$fifo >$2/$outfile.md5 &

# Read the device and compress it
dd if="/dev/$1" bs=1M | tee 2>$fifo | gzip >$2/$outfile.gz

Quoted 8 lines

# Wait for the background task to close
wait

I was a little concerned as I didn't see the hdd drive LED light up for my attempt even though the file was growing nicely on my CIFS share.  I'm dumping it on the 5x2TB disks (JBOD) in my windows 7 box as my Thecus is not a happy chappy and I need time to mount the DOM module in another machine to find out why.

If you're willing to dismantle the thecus, directly connecting its drives to your crippled system will make things go much faster.  You've got four empty motherboard SATA ports.  You just have to watch out for the power load.

Phil
<block2gz.sh>

19390 root       17.49 M/s       0 B/s  0.00 % 21.36 % dd if /dev/sde bs 1M
19333 root       16.52 M/s       0 B/s  0.00 % 17.07 % dd if /dev/sdd bs 1M
19503 root       15.79 M/s       0 B/s  0.00 % 13.91 % dd if /dev/sdf bs 1M
18896 root           0 B/s   16.10 M/s  0.00 %  0.00 % ntfs-3g

/dev/sdb1 /media/ntfs3g/sdb
18909 root 0 B/s 13.49 M/s 0.00 % 0.00 % ntfs-3g
/dev/sdc1 /media/ntfs3g/sdc
18920 root 0 B/s 15.73 M/s 0.00 % 0.00 % ntfs-3g
/dev/sdn1 /media/ntfs3g/sdn

this will certainly be quicker :-)

Quoted 10 lines · Simon Mcnair

19390 root       17.49 M/s       0 B/s  0.00 % 21.36 % dd if /dev/sde bs 1M 19333 root       16.52 M/s       0 B/s  0.00 % 17.07 % dd if /dev/sdd bs 1M 19503 root       15.79 M/s       0 B/s  0.00 % 13.91 % dd if /dev/sdf bs 1M 18896 root           0 B/s   16.10 M/s  0.00 %  0.00 % ntfs-3g /dev/sdb1 /media/ntfs3g/sdb
18909 root           0 B/s   13.49 M/s  0.00 %  0.00 % ntfs-3g /dev/sdc1 /media/ntfs3g/sdc
18920 root           0 B/s   15.73 M/s  0.00 %  0.00 % ntfs-3g /dev/sdn1 /media/ntfs3g/sdn

Quoted 1 line

this will certainly be quicker :-)

But still a long time...

Let me know.

Phil

Phil,
Sorry for the spamming, but I'm just keeping you informed :-).

proxmox:/home/simon# ./block2gz.sh sdd /media/ntfs3g/sdb
953869+1 records in
953869+1 records out
1000204886016 bytes (1.0 TB) copied, 95384 s, 10.5 MB/s

Have just started sdj,sdk,sdl,sdm. I was thinking of renaming the .gz's with the serial number as this would seem to me as more useful, is this a good idea ?

These numbers are not at all reflective of the drive or controller speed as I took the lazy route and was writing 2 sets of images at the same time to the same drive and in addition gzip was also running (although not really stressing the system from what I could tell).

I suspect also that the drives may be pretty fragmented as the space was not allocated at the start of the write, so that may have had some impact too.

as attached (hopefully).
regards
Simon

Quoted 3 lines · Simon Mcnair

as attached (hopefully).
regards
Simon

Nothing obvious to me.

Are you running irqbalance?

cat /proc/interrupts ?

Phil,
I don't know how to find out if I'm running irqbalance, it's whatever was in the proxmox iso that I installed the OS from. I ran a ps -aux | grep irq in case it shows anything of interest:

proxmox:/home/simon# ps -aux | grep irq
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
root         3  0.0  0.0      0     0 ?        S    Feb17   0:22 [ksoftirqd/0]
root         7  0.0  0.0      0     0 ?        S    Feb17   0:05 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    Feb17   0:05 [ksoftirqd/2]
root        13  0.0  0.0      0     0 ?        S    Feb17   0:05 [ksoftirqd/3]
root        16  0.0  0.0      0     0 ?        S    Feb17   0:11 [ksoftirqd/4]
root        19  0.0  0.0      0     0 ?        S    Feb17   0:13 [ksoftirqd/5]
root        22  0.0  0.0      0     0 ?        S    Feb17   0:06 [ksoftirqd/6]
root        25  0.0  0.0      0     0 ?        S    Feb17   0:06 [ksoftirqd/7]
root      4024  0.0  0.0      0     0 ?        S    Feb17   0:00
[kvm-irqfd-clean]

proxmox:/home/simon# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4
CPU5       CPU6       CPU7
   0:   66008972          0          0          0          0
0          0          0  IR-IO-APIC-edge      timer
   1:     265829          0          0          0          0
0          0          0  IR-IO-APIC-edge      i8042
   8:          1          0          0          0          0
0          0          0  IR-IO-APIC-edge      rtc0
   9:          0          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   acpi
  16:    4432639          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb3, ahci
  17:     124325          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   pata_jmicron, eth1
  18:     954710          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb1,
uhci_hcd:usb8
  19:       5994          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb5,
uhci_hcd:usb7, firewire_ohci
  21:          0          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb4
  23:         62          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb2,
uhci_hcd:usb6
  24:     273558          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   nvidia
  28:   30234710          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   mvsas
  64:          0          0          0          0          0
0          0          0  DMAR_MSI-edge      dmar0
  65:          0          0          0          0          0
0          0          0  DMAR_MSI-edge      dmar1
  73:   26838727          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0
  74:   27057631          0          0          0          0
0          0          0  IR-PCI-MSI-edge      ahci
  75:        247          0          0          0          0
0          0          0  IR-PCI-MSI-edge      hda_intel
 NMI:          0          0          0          0          0
0          0          0   Non-maskable interrupts
 LOC:   23087679   23088282   21638323   20580136   22974769
20856094   20144310   20084801   Local timer interrupts
 SPU:          0          0          0          0          0
0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0
0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0          0
0          0          0   Performance pending work
 RES:   15236622    9388860    8532928    7181879   13444866
6005599    4856933    3713707   Rescheduling interrupts
 CAL:       3384       5773       5839       5859       4590
5748       5781       5691   Function call interrupts
 TLB:     201989     195383     193545     196155     251338
249053     272032     319233   TLB shootdowns
 TRM:          0          0          0          0          0
0          0          0   Thermal event interrupts

 THR:          0          0          0          0          0
0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0
0          0          0   Machine check exceptions
 MCP:        830        830        830        830        830
830        830        830   Machine check polls
 ERR:          7
 MIS:          0

Quoted 5 lines · Simon Mcnair

Phil,
I don't know how to find out if I'm running irqbalance, it's whatever was in the proxmox iso that I installed the OS from.  I ran a ps -aux | grep irq in case it shows anything of interest:

Quoted 12 lines

proxmox:/home/simon# ps -aux | grep irq
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
root         3  0.0  0.0      0     0 ?        S    Feb17   0:22 [ksoftirqd/0]
root         7  0.0  0.0      0     0 ?        S    Feb17   0:05 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    Feb17   0:05 [ksoftirqd/2]
root        13  0.0  0.0      0     0 ?        S    Feb17   0:05 [ksoftirqd/3]
root        16  0.0  0.0      0     0 ?        S    Feb17   0:11 [ksoftirqd/4]
root        19  0.0  0.0      0     0 ?        S    Feb17   0:13 [ksoftirqd/5]
root        22  0.0  0.0      0     0 ?        S    Feb17   0:06 [ksoftirqd/6]
root        25  0.0  0.0      0     0 ?        S    Feb17   0:06 [ksoftirqd/7]
root      4024  0.0  0.0      0     0 ?        S    Feb17   0:00
[kvm-irqfd-clean]

Not there.  On ubuntu 10.10, the package is called "irqbalance", and the executable daemon is "irqbalance".

Quoted 58 lines

proxmox:/home/simon# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4
CPU5       CPU6       CPU7
   0:   66008972          0          0          0          0
0          0          0  IR-IO-APIC-edge      timer
   1:     265829          0          0          0          0
0          0          0  IR-IO-APIC-edge      i8042
   8:          1          0          0          0          0
0          0          0  IR-IO-APIC-edge      rtc0
   9:          0          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   acpi
  16:    4432639          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb3, ahci
  17:     124325          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   pata_jmicron, eth1
  18:     954710          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb1,
uhci_hcd:usb8
  19:       5994          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb5,
uhci_hcd:usb7, firewire_ohci
  21:          0          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb4
  23:         62          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb2,
uhci_hcd:usb6
  24:     273558          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   nvidia
  28:   30234710          0          0          0          0
0          0          0  IR-IO-APIC-fasteoi   mvsas
  64:          0          0          0          0          0
0          0          0  DMAR_MSI-edge      dmar0
  65:          0          0          0          0          0
0          0          0  DMAR_MSI-edge      dmar1
  73:   26838727          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0
  74:   27057631          0          0          0          0
0          0          0  IR-PCI-MSI-edge      ahci
  75:        247          0          0          0          0
0          0          0  IR-PCI-MSI-edge      hda_intel
 NMI:          0          0          0          0          0
0          0          0   Non-maskable interrupts
 LOC:   23087679   23088282   21638323   20580136   22974769
20856094   20144310   20084801   Local timer interrupts
 SPU:          0          0          0          0          0
0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0
0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0          0
0          0          0   Performance pending work
 RES:   15236622    9388860    8532928    7181879   13444866
6005599    4856933    3713707   Rescheduling interrupts
 CAL:       3384       5773       5839       5859       4590
5748       5781       5691   Function call interrupts
 TLB:     201989     195383     193545     196155     251338
249053     272032     319233   TLB shootdowns
 TRM:          0          0          0          0          0
0          0          0   Thermal event interrupts

Quoted 8 lines

 THR:          0          0          0          0          0
0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0
0          0          0   Machine check exceptions
 MCP:        830        830        830        830        830
830        830        830   Machine check polls
 ERR:          7
 MIS:          0

CPU0 is handling every single I/O interrupt.  I really think you need irqbalance.

Phil

Phil,
You are a god amongst men :-)

I seem to have the vast majority of my data. I can't see anything in lost+found and the fsck log doesn't list any filenames etc of missing data. Is there any way for me to find anything that I've lost ?

Quoted 2 lines · Simon Mcnair

Phil,
You are a god amongst men :-)

Heh.  I read that out loud to my wife....

Quoted 3 lines

I seem to have the vast majority of my data.  I can't see anything in lost+found and the fsck log doesn't list any filenames etc of missing data.  Is there any way for me to find anything that I've lost ?

Not that I know of, short of a list of names from before the crash.

Glad to hear that the family treasures are safe.

Phil

Phil,
I thought it was fixed, but every time I do a reboot it disappears again.  I did the:
pvcreate --restorefile 'simons-lvm-backup' --uuid 9fAJEz-HcaP-RQ51-fV8b-nxrN-Uqwb-PPnOLJ /dev/md0 and the vgcfgrestore

the only piece of your advice that I didn't follow was "zero the superblock on the removed drive" as I thought re-adding it to the array would do that anyway.  Did this cause me problems ?

proxmox:~# mdadm  --assemble --scan -v
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: /dev/dm-2 has wrong uuid.
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: /dev/dm-1 has wrong uuid.
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: /dev/dm-0 has wrong uuid.
mdadm: /dev/sdo1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdo
mdadm: /dev/sdo has wrong uuid.
mdadm: /dev/sdn1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdn
mdadm: /dev/sdn has wrong uuid.
mdadm: /dev/sdm1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdm
mdadm: /dev/sdm has wrong uuid.
mdadm: /dev/sdl1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdl
mdadm: /dev/sdl has wrong uuid.
mdadm: /dev/sdk1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk
mdadm: /dev/sdk has wrong uuid.
mdadm: /dev/sdj1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdj
mdadm: /dev/sdj has wrong uuid.
mdadm: /dev/sdi1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi
mdadm: /dev/sdi has wrong uuid.
mdadm: /dev/sdh1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh
mdadm: /dev/sdh has wrong uuid.
mdadm: /dev/sdg1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg
mdadm: /dev/sdg has wrong uuid.
mdadm: /dev/sdf1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: /dev/sda2 has wrong uuid.
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.

I'm getting frustrated now.  It seems like it's fixzed, then next reboot, BAM, it's gone again.  I'm so out of my depth here.

Please can you help me fix this once and for all ?

cheers
Simon

Quoted 16 lines · Phil Turmel

Quoted 2 lines · Simon Mcnair

Phil,
You are a god amongst men :-)

Heh.  I read that out loud to my wife....

Quoted 3 lines

I seem to have the vast majority of my data.  I can't see anything in lost+found and the fsck log doesn't list any filenames etc of missing data.  Is there any way for me to find anything that I've lost ?

Not that I know of, short of a list of names from before the crash.

Glad to hear that the family treasures are safe.

Phil

was this anything to do with the dist-upgrade that I performed ?

mdadm has upgraded again to

proxmox:~# mdadm -V
mdadm - v3.1.4 - 31st August 2010

DEVICE /dev/sd[a-z][1-9]

ARRAY /dev/md23 UUID=c3cbe096:fc43d939:8aa66230:708c5670 ARRAY /dev/md22 UUID=1438a239:aa03c3f9:68e051d7:b59a6219 ARRAY /dev/md21 UUID=4b47d0f6:16f0e352:c67a2185:feb0b573 ARRAY /dev/md20 UUID=7676c77e:6ae70f65:30a170a2:b1a1b242

Quoted 5 lines · Simon Mcnair

was this anything to do with the dist-upgrade that I performed ?

mdadm has upgraded again to

proxmox:~# mdadm -V
mdadm - v3.1.4 - 31st August 2010

No, but you probably can't do any more "--create --assume-clean" operations with that array.

As for the assembly errors, its probably an out-of-date mdadm.conf.

My server's looks like this:

Quoted 2 lines

DEVICE /dev/sd[a-z][1-9]

Quoted 4 lines

ARRAY /dev/md23 UUID=c3cbe096:fc43d939:8aa66230:708c5670 ARRAY /dev/md22 UUID=1438a239:aa03c3f9:68e051d7:b59a6219 ARRAY /dev/md21 UUID=4b47d0f6:16f0e352:c67a2185:feb0b573 ARRAY /dev/md20 UUID=7676c77e:6ae70f65:30a170a2:b1a1b242

Yours probably needs to be updated with the new md0 uuid. Its not your boot drive, so your initramfs shouldn't matter.

Phil

Phil,
phew I didn't know that:
"mdadm: /dev/sdo has wrong uuid.
mdadm: /dev/sdn1 has wrong uuid."

was just that the array UUID didn't match mdadm.conf, it would be nice if it said:

"mdadm: /dev/sdo uuid does not match mdadm.conf.
mdadm: /dev/sdn1 uuid does not match mdadm.conf" my mdadm.conf now reads

DEVICE partitions
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
# ARRAY /dev/md/0 level=raid5 metadata=1.1 num-devices=10 UUID=0a72e40f:aec6f80f:a7004457:1a84a7a8 name=pro�lox:0 ARRAY /dev/md/0 metadata=1.1 UUID=12c2af00:10681e10:fb17e449:1404739c name=proxmox:0

I'm afraid to do another reboot incase something else goes wrong ;-). Just for my info, when I only have a single array there isn't much chance of it being assigned anything other than md0, so could I not just leave mdadm.conf empty ?

Quoted 17 lines · Simon McNair

proxmox:/home/simon# vgscan --verbose
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
    Finding all volume groups
    Finding volume group "pve"
  Found volume group "pve" using metadata type lvm2
    Finding volume group "lvm-raid"
  Found volume group "lvm-raid" using metadata type lvm2
proxmox:/home/simon#
proxmox:/home/simon# lvscan --verbose
    Finding all logical volumes
  ACTIVE            '/dev/pve/swap' [11.00 GB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GB] inherit
  ACTIVE            '/dev/pve/data' [354.26 GB] inherit
  inactive          '/dev/lvm-raid/RAID' [8.19 TB] inherit

Quoted 3 lines

proxmox:/home/simon# vgchange -ay
  3 logical volume(s) in volume group "pve" now active
  1 logical volume(s) in volume group "lvm-raid" now active

Heh.  Figures.

Quoted 1 line

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/lvm-raid-RAID

Actually, I wanted you to try with a capital N.  Lower case 'n' is similar, but not quite the same.

Quoted 11 lines

e2fsck 1.41.3 (12-Oct-2008)
fsck.ext4: No such file or directory while trying to open /dev/mapper/lvm-raid-RAID

The superblock could not be read or does not describe a correct ext2 filesystem.  If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/
control         lvm--raid-RAID  pve-data        pve-root        pve-swap

Strange.  I guess it does that to distinguish dashes in the VG name from dashes between VG and LV names.

Quoted 6 lines

proxmox:/home/simon# fsck.ext4 -n /dev/mapper/lvm--raid-RAID
e2fsck 1.41.3 (12-Oct-2008)
/dev/mapper/lvm--raid-RAID has unsupported feature(s): FEATURE_I31
e2fsck: Get a newer version of e2fsck!

my version of e2fsck always worked before ?

v1.41.14 was release 7 weeks ago.  But, I suspect there's corruption in the superblock.  Do you still have your disk images tucked away somewhere safe?

If so, try:

1) The '-b' option to e2fsck.  We need to experiment with '-n -b offset' to find the alternate superblock.  Trying 'offset' = to 8193, 16384, and 32768, per the man-page.

2) A newer e2fsprogs.

Finally,
3) mount -r /dev/lvm-raid/RAID /mnt/whatever

Phil

time passes. You are eaten by a Grue. Sheesh this is taking a long time.

simon@proxmox:~$ top

top - 09:26:19 up 20:36, 11 users, load average: 2.46, 2.49, 2.39
Tasks: 334 total, 9 running, 325 sleeping, 0 stopped, 0 zombie
Cpu(s): 52.8%us, 9.4%sy, 0.0%ni, 35.2%id, 2.5%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 12299244k total, 12227584k used, 71660k free, 11042868k buffers
Swap: 11534332k total, 0k used, 11534332k free, 208204k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18896 root      20   0 12288 1072  460 S   63  0.0 431:16.55 ntfs-3g
18909 root      20   0 12288 1064  460 R   62  0.0 419:17.11 ntfs-3g
18920 root      20   0 12288 1100  492 R   62  0.0 428:17.33 ntfs-3g
 9210 root      20   0  4068  520  328 S   54  0.0 661:59.92 gzip
 9138 root      20   0  4068  524  328 R   53  0.0 647:29.07 gzip
 9247 root      20   0  4068  524  328 S   53  0.0 651:28.36 gzip
25678 root      20   0  4068  524  328 R   52  0.0 439:49.20 gzip
24957 root      20   0  4068  524  328 R   51  0.0 437:44.01 gzip
25792 root      20   0  4068  524  328 S   48  0.0 433:28.14 gzip

This is hardly touching the CPU on the box (in my opinion), any advice on using renice ? I've never used it before, but now seems like a good time ?

Quoted 21 lines

time passes.  You are eaten by a Grue.  Sheesh this is taking a long time.

simon@proxmox:~$ top

top - 09:26:19 up 20:36, 11 users,  load average: 2.46, 2.49, 2.39
Tasks: 334 total,   9 running, 325 sleeping,   0 stopped,   0 zombie Cpu(s): 52.8%us,  9.4%sy,  0.0%ni, 35.2%id,  2.5%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  12299244k total, 12227584k used,    71660k free, 11042868k buffers
Swap: 11534332k total,        0k used, 11534332k free,   208204k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 18896 root      20   0 12288 1072  460 S   63  0.0 431:16.55 ntfs-3g 18909 root      20   0 12288 1064  460 R   62  0.0 419:17.11 ntfs-3g 18920 root      20   0 12288 1100  492 R   62  0.0 428:17.33 ntfs-3g 9210 root      20   0  4068  520  328 S   54  0.0 661:59.92 gzip 9138 root      20   0  4068  524  328 R   53  0.0 647:29.07 gzip 9247 root      20   0  4068  524  328 S   53  0.0 651:28.36 gzip 25678 root      20   0  4068  524  328 R   52  0.0 439:49.20 gzip 24957 root      20   0  4068  524  328 R   51  0.0 437:44.01 gzip 25792 root      20   0  4068  524  328 S   48  0.0 433:28.14 gzip

Quoted 3 lines

This is hardly touching the CPU on the box (in my opinion), any advice on using renice ?  I've never used it before, but now seems like a good time ?

Probably wouldn't help.

You have a total usage of 65%.  I suspect that each of the processors running gzip are nearly pegged, and everything else is loafing along.  Hit '1' in top to show the CPU usage per-cpu.  Single-threaded gzip is holding you back.

If you had enough space for saving the partitions uncompressed, it would be much faster.  The PCIe x4 interface on the SuperMicro has a theoretical performance of 1GByte/s, which would be ~ 125MB/s per drive.  From what I've read, that card actually delivers ~ 75MB/s per drive when they're all busy.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

pci express x4 = 2.5*4gbps = 10gbps not?!

Quoted 21 lines · Simon Mcnair

time passes.  You are eaten by a Grue.  Sheesh this is taking a long time.

simon@proxmox:~$ top

top - 09:26:19 up 20:36, 11 users,  load average: 2.46, 2.49, 2.39
Tasks: 334 total,   9 running, 325 sleeping,   0 stopped,   0 zombie Cpu(s): 52.8%us,  9.4%sy,  0.0%ni, 35.2%id,  2.5%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  12299244k total, 12227584k used,    71660k free, 11042868k buffers
Swap: 11534332k total,        0k used, 11534332k free,   208204k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 18896 root      20   0 12288 1072  460 S   63  0.0 431:16.55 ntfs-3g 18909 root      20   0 12288 1064  460 R   62  0.0 419:17.11 ntfs-3g 18920 root      20   0 12288 1100  492 R   62  0.0 428:17.33 ntfs-3g 9210 root      20   0  4068  520  328 S   54  0.0 661:59.92 gzip 9138 root      20   0  4068  524  328 R   53  0.0 647:29.07 gzip 9247 root      20   0  4068  524  328 S   53  0.0 651:28.36 gzip 25678 root      20   0  4068  524  328 R   52  0.0 439:49.20 gzip 24957 root      20   0  4068  524  328 R   51  0.0 437:44.01 gzip 25792 root      20   0  4068  524  328 S   48  0.0 433:28.14 gzip

Quoted 3 lines

This is hardly touching the CPU on the box (in my opinion), any advice on using renice ?  I've never used it before, but now seems like a good time ?

Probably wouldn't help.

You have a total usage of 65%.  I suspect that each of the processors running gzip are nearly pegged, and everything else is loafing along.  Hit '1' in top to show the CPU usage per-cpu.  Single-threaded gzip is holding you back.

If you had enough space for saving the partitions uncompressed, it would be much faster.  The PCIe x4 interface on the SuperMicro has a theoretical performance of 1GByte/s, which would be ~ 125MB/s per drive.  From what I've read, that card actually delivers ~ 75MB/s per drive when they're all busy.

Phil

top - 13:28:45 up 1 day, 39 min, 12 users, load average: 2.65, 2.49, 2.37
Tasks: 340 total, 7 running, 333 sleeping, 0 stopped, 0 zombie
Cpu0 : 47.9%us, 15.0%sy, 0.0%ni, 13.7%id, 21.5%wa, 0.0%hi, 2.0%si, 0.0%st Cpu1 : 54.8%us, 8.7%sy, 0.0%ni, 33.3%id, 3.2%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu2 : 46.5%us, 13.7%sy, 0.0%ni, 37.9%id, 1.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 49.5%us, 9.5%sy, 0.0%ni, 37.6%id, 3.4%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 48.7%us, 8.2%sy, 0.0%ni, 17.3%id, 25.5%wa, 0.0%hi, 0.3%si, 0.0%st Cpu5 : 50.3%us, 7.2%sy, 0.0%ni, 41.8%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 53.5%us, 6.6%sy, 0.0%ni, 38.4%id, 1.6%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 52.0%us, 6.2%sy, 0.0%ni, 41.2%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 12299244k total, 12227064k used, 72180k free, 10754600k buffers
Swap: 11534332k total, 0k used, 11534332k free, 175916k cached

Quoted 4 lines · Simon Mcnair

top - 13:28:45 up 1 day, 39 min, 12 users, load average: 2.65, 2.49, 2.37
Tasks: 340 total, 7 running, 333 sleeping, 0 stopped, 0 zombie
Cpu0 : 47.9%us, 15.0%sy, 0.0%ni, 13.7%id, 21.5%wa, 0.0%hi, 2.0%si, 0.0%st Cpu1 : 54.8%us, 8.7%sy, 0.0%ni, 33.3%id, 3.2%wa, 0.0%hi, 0.0%si, 0.0%st

Quoted 8 lines

Cpu2 : 46.5%us, 13.7%sy, 0.0%ni, 37.9%id, 1.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 49.5%us, 9.5%sy, 0.0%ni, 37.6%id, 3.4%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 48.7%us, 8.2%sy, 0.0%ni, 17.3%id, 25.5%wa, 0.0%hi, 0.3%si, 0.0%st Cpu5 : 50.3%us, 7.2%sy, 0.0%ni, 41.8%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 53.5%us, 6.6%sy, 0.0%ni, 38.4%id, 1.6%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 52.0%us, 6.2%sy, 0.0%ni, 41.2%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 12299244k total, 12227064k used, 72180k free, 10754600k buffers
Swap: 11534332k total, 0k used, 11534332k free, 175916k cached

Hmmm. Not what I expected at all.

I'd be very interested in some performance tests of the Supermicro ports vs. motherboard ports on that system.

Phil

Phil,
Tell me what you want me to do and I'm game for it. When I bought it the reviews were not all that promising (http://www.google.co.uk/products/catalog?hl=&q=AOC-SASLP-MV8&cid=15596376504116122379&os=reviews) but I figure getting up and running is more important than sata speed at the moment.

Quoted 2 lines

Phil,
Tell me what you want me to do and I'm game for it.  When I bought it the reviews were not all that promising (http://www.google.co.uk/products/catalog?hl=&q=AOC-SASLP-MV8&cid=15596376504116122379&os=reviews) but I figure getting up and running is more important than sata speed at the moment.

I perused the cards reviewed here:

http://blog.zorinaq.com/?e=10

I was aiming for the ~$100 mark, for personal use.  I'd have chosen the one of LSI 1068E units, but the cheaper ones are reversed boards (parts on opposite side of normal).  The Marvell 6480 was apparently the next best deal.  I may have to rethink that.

In any case, some benchmarking options:

1) Spot check with "hdparm -t /dev/sd?".  Not considered very accurate.  It's read-only, though, so can run in parallel with other device activity (w/ corresponding reduced accuracy).

2) Run a packaged set of benchmarks:
       http://www.coker.com.au/bonnie++/
       http://www.iozone.org/
       http://dbench.samba.org/

3) Script a series of basic tests of various tasks:
       https://raid.wiki.kernel.org/index.php/Home_grown_testing_methods

Most of these tests require a writable filesystem on the target device to work in.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Phil,

Is this fsck (fsck.ext4 -n -b 32768 /dev/mapper/lvm--raid-RAID

Quoted 1 line

fsck.txt) as bad as it looks ? :-(

Quoted 4 lines · Simon Mcnair

Hi Phil,

Is this fsck (fsck.ext4 -n -b 32768 /dev/mapper/lvm--raid-RAID

Quoted 1 line

fsck.txt) as bad as it looks ? :-(

It's bad.  Either the original sdd has a lot more corruption than I expected, or the 3ware spread corruption over all the drives.

If the former, failing it out of the array might help.  If the latter, your data is likely toast.  Some identifiable data is being found, based on the used vs. free block/inode/directory counts in that report.  That's good.

I suggest you do "mdadm /dev/md0 --fail /dev/sdi1" and repeat the "fsck -n" as above.

(It'll be noticably slower, as it'll be using parity to reconstruct 1 out every 9 chunks.)

If it the fsck results improve, or stay the same, proceed to "fsck -y", and we'll see.

Wouldn't hurt to run "iostat -xm 5" in another terminal during the fsck to see what kind of performance that array is getting.

Phil

Quoted 2 lines · Simon McNair

Hi Phil,
 A couple of questions please.

[trim /]

Quoted 8 lines

Quoted 7 lines

Quoted 1 line

Simon

I don't know why the serial numbers are formatted differently, but we can still tell them apart (the eight characters starting with "PAG").

So, our device order in your new setup is: [ihgfmlkjbc], where /dev/sdi corresponds to the original report's /dev/sdd, which matches the sig grep in your other note.

Another note:  The controller for sd[abc] is still showing ata_piix as its controller.  That means you cannot hot-plug those ports.  If you change your BIOS to AHCI mode instead of "Compatibility" or "Emulation", the full-featured ahci driver will run those ports.  Not urgent, but I highly recommend it.

Will do that now, before I forget

Hot-pluggability with suitable trays is very handy! :)

[trim /]

Quoted 8 lines

Quoted 7 lines

Quoted 3 lines

Error: The backup GPT table is not at the end of the disk, as it should be.  This might mean that another operating system believes the disk is smaller.  Fix, by moving the backup to the end (and removing the old backup)? Fix/Cancel? c

The 3ware controller must have reserved some space at the end of each drive for its own use.  Didn't know it'd do that.  You will have to fix that.

[trim /]

Do you have any suggestions on how I can fix that ?  I don't have a clue

Just do 'parted /dev/sd?' and on the ones it offers to fix, say yes.  Then request 'unit s' and 'print' to verify that it is correct.

[trim /]

Quoted 5 lines

when I was trying to figure out the command for this using 'man parted' I came across this:
" rescue start end
                     Rescue  a  lost  partition  that was located somewhere between start and end.  If a partition is found, parted will ask if you want to create an entry for it in the partition table." Is it worth trying ?

Nah.  That's for when you don't know exactly where the partition is.  We know.

Quoted 7 lines

I originally created the partitions like so:
parted -s /dev/sdb rm 1
parted -s /dev/sdb mklabel gpt
parted -s --align optimal /dev/sdb mkpart primary ext4 .512 100% parted -s /dev/sdb set 1 raid on
parted -s /dev/sdb align-check optimal 1

Quoted 11 lines

so to recreate the above I would do:
parted -s /dev/sdb mkpart primary ext4 2048s 1953101823s parted -s /dev/sdc mkpart primary ext4 2048s 1953101823s parted -s /dev/sdf mkpart primary ext4 2048s 1953101823s parted -s /dev/sdg mkpart primary ext4 2048s 1953101823s parted -s /dev/sdh mkpart primary ext4 2048s 1953101823s parted -s /dev/sdi mkpart primary ext4 2048s 1953101823s parted -s /dev/sdj mkpart primary ext4 2048s 1953101823s parted -s /dev/sdk mkpart primary ext4 2048s 1953101823s parted -s /dev/sdl mkpart primary ext4 2048s 1953101823s parted -s /dev/sdm mkpart primary ext4 2048s 1953101823s

Only recreate the partition tables where you have to, i.e., the 'Fix' option above didn't work.  And don't specify a filesystem.

Probably just /dev/sdh and /dev/sdi.  Like so, though:

parted -s /dev/sdh mklabel gpt mkpart primary 2048s 1953101823s set 1 raid on parted -s /dev/sdi mklabel gpt mkpart primary 2048s 1953101823s set 1 raid on

Quoted 1 line

I'm guessing the backups that I want to do can wait until any potential fsck ?

Do an 'fsck -N' first, and if it passes, or has few errors, mount the filesystem readonly and grab your backup.  Then let fsck have at it for real.  If anything gets fixed, compare your backup from the read-only fs to the fixed fs.

Given your flaky old controller, I expect there'll be *some* problems.

Quoted 1 line

sorry if the questions are dumb but I'm not sure what I'm doing and I'd rather ask more questions than fewer and understand the implications of what I'm doing.

Oh, no.  You are right to be paranoid.  If anything looks funny, stop.

Phil

Hi Neil,

Since Simon has responded, let me summarize the assistance I provided per his off-list request:

Quoted 33 lines · NeilBrown

Quoted 9 lines · On Thu, 10 Feb 2011 16:16:44 +0000 Simon McNair<[email protected]> wrote:

Hi all

I use a 3ware 9500-12 port sata card (JBOD) which will not work without a 128mb sodimm. The sodimm socket is flakey and the result is that the machine occasionally crashes. Yesterday I finally gave in and put together another
machine so that I can rsync between them. When I turned the machine on today to set up rync, the RAID array was not gone, but corrupted.
Typical...

Presumably the old machine was called 'ubuntu' and the new machine 'proølox'

Quoted 19 lines

I built the array in Aug 2010 using the following command:

mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 /dev/sd{b,c,d,e,f,g,h,i,j,k}1 --chunk=64

Using LVM, I did the following:
pvscan
pvcreate -M2 /dev/md0
vgcreate lvm-raid /dev/md0
vgdisplay lvm-raid
vgscan
lvscan
lvcreate -v -l 100%VG -n RAID lvm-raid
lvdisplay /dev/lvm-raid/lvm0

I then formatted using:
mkfs -t ext4 -v -m .1 -b 4096 -E stride=16,stripe-width=144 /dev/lvm-raid/RAID

Quoted 8 lines

Quoted 6 lines

This worked perfectly since I created the array. Now mdadm is coming up with

proxmox:/dev/md# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/ubuntu:0

And it seems that ubuntu:0 have been successfully assembled.
It is missing one device for some reason (sdd1) but RAID can cope with that.

3ware card is compromised, with a loose buffer memory dimm. Some of its ECC errors were caught and reported in dmesg. Its likely, based on the loose memory socket, that many multiple-bit errors got through.

[trim /]

Quoted 8 lines

Quoted 5 lines

mdadm: no uptodate device for slot 8 of /dev/md/pro�lox:0
mdadm: no uptodate device for slot 9 of /dev/md/pro�lox:0
mdadm: failed to add /dev/sdd1 to /dev/md/pro�lox:0: Invalid argument
mdadm: /dev/md/pro�lox:0 assembled from 0 drives - not enough to start the array.

This looks like it is *after* to trying the --create command you give below.. It is best to report things in the order they happen, else you can confuse people (or get caught out!).

Yes, this was after.

Quoted 6 lines

Quoted 6 lines

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/sdd
mdadm: No arrays found in config file or automatically

pvscan and vgscan show nothing.

Quoted 8 lines

Quoted 6 lines

So I tried running mdadm --create --verbose /dev/md0 --metadata=1.1 --level=5 --raid-devices=10 missing /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 --chunk=64

as it seemed that /dev/sdd1 failed to be added to the array. This did nothing.

It did not to nothing. It wrote a superblock to /dev/sdd1 and complained that it couldn't write to all the others --- didn't it?

There were multiple attempts to create. One wrote to just sdd1, another succeeded with all but sdd1.

Quoted 9 lines

Quoted 3 lines

dmesg contains:

md: invalid superblock checksum on sdd1

I guess that is why sdd1 was missing from 'ubuntu:0'. Though as I cannot tell if this happened before or after any of the various things reported above, it is hard to be sure.

The real mystery is why 'pvscan' reports nothing.

The original array was created with mdadm v2.6.7, and had a data offset of 264 sectors. After Simon's various attempts to --create, he ended up with data offset of 2048, using mdadm v3.1.4. The mdadm -E reports he posted to the list showed the 264 offset. We didn't realize the offset had been updated until somewhat later in our troubleshooting efforts.

In any case, pvscan couldn't see the LVM signature because it wasn't there (at offset 2048).

Quoted 9 lines

What about
pvscan --verbose

or

blkid -p /dev/md/ubuntu:0

or even

Quoted 1 line

   dd of=/dev/md/ubuntu:0 count=8 | od -c

Fortunately, Simon did have a copy of his LVM configuration. With the help of dd, strings, and grep, we did locate his LVM sig at the correct location on sdd1 (for data offset 264). After a number of attempts to bypass LVM and access his single LV with dmsetup (based on his backed up configuration, on the assembled new array less sdd1), I realized that the data offset was wrong on the recreated array, and went looking for the cause. I found your git commit that changed that logic last spring, and recommended that Simon revert to the default package for his ubuntu install, which is v2.6.7.

Simon has now attempted to recreate the array with v2.6.7, but the controller is throwing too many errors to succeed, and I suggested it was too flakey to trust any further. Based on the existence of the LVM sig on sdd1, I believe Simon's data is (mostly) intact, and only needs a successful create operation with a properly functioning controller. (He might also need to perform an lvm vgcfgrestore, but he has the necessary backup file.)

A new controller is on order.

Phil