raid related kernel hang in 2.6.36
Hi,
We just quick tested the raid stability under the vanilla 2.6.36 kernel and got an unstable result.
first, we create a raid5 array using 16 sata disks:
mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q]
then use dd to stress the io:
dd if=/dev/zero of=/dev/md10 bs=1M
After a while, usually several minutes, the system hangs. It looks like some kind of kernel deadlock. ping to this machine could get timely response, however, any other process just hangs.
It is easily reproducible and everytime the test result is a system hang.
Johnson
Hi,
I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds), mdadm -D md10, zcat /proc/config.gz, mdadm version)
Regards,
// Mathias
2010/11/24 Du Jun <[email protected]>:
Quoted 16 lines
Hi,
We just quick tested the raid stability under the vanilla 2.6.36 kernel and got an unstable result.first, we create a raid5 array using 16 sata disks:
mdadm -C
/dev/md10-l 5 -n 16 /dev/sd[b-q]then use dd to stress the io:
dd if=/dev/zero of=/dev/md10 bs=1M
After a while, usually several minutes, the system hangs. It looks like some kind of kernel deadlock. ping to this machine could get timely response, however, any other process just hangs.
Quoted 9 lines
It is easily reproducible and everytime the test result is a system hang.
Johnson
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Their's no dmesg information at all before the hang. The system just hangs suddenly without any kernel printk or oops.
mdadm version is debian lenny version 2.6.7.2-3
I will get mdadm -E & mdadm -D informations tomorrow but I doubt they are useful.
Johnson
Quoted 36 lines · Mathias Burén
Hi,
I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds), mdadm -D md10, zcat
/proc/config.gz, mdadm version)Regards,
// Mathias2010/11/24 Du Jun<[email protected]>:
Quoted 16 lines
Hi,
We just quick tested the raid stability under the vanilla 2.6.36 kernel and got an unstable result.first, we create a raid5 array using 16 sata disks:
mdadm -C
/dev/md10-l 5 -n 16 /dev/sd[b-q]then use dd to stress the io:
dd if=/dev/zero of=/dev/md10 bs=1M
After a while, usually several minutes, the system hangs. It looks like some kind of kernel deadlock. ping to this machine could get timely response, however, any other process just hangs.
Quoted 10 lines
It is easily reproducible and everytime the test result is a system hang.
Johnson
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · On Wed, 24 Nov 2010 22:34:21 +0800 Du Jun <[email protected]> wrote:
Their's no dmesg information at all before the hang. The system just hangs suddenly without any kernel printk or oops.
mdadm version is debian lenny version 2.6.7.2-3
I will get mdadm -E & mdadm -D informations tomorrow but I doubt they are useful.
Probably not, but they wouldn't hurt either.
What would really help is if you could capture the result of
sysrq-T
which I accept might be a challenge depending on how hard the machine really has frozen.
Also, make sure the LOCKUP_DETECTOR config options are set, and wait a few minutes after the hand to see if something pops up.
I'll try to see if I can reproduce, but I don't have 16 disks so it may not work.
NeilBrown
Quoted 47 lines
Johnson
Quoted 36 lines · Mathias Burén
Hi,
I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds), mdadm -D md10, zcat
/proc/config.gz, mdadm version)Regards,
// Mathias2010/11/24 Du Jun<[email protected]>:
Quoted 16 lines
Hi,
We just quick tested the raid stability under the vanilla 2.6.36 kernel and got an unstable result.first, we create a raid5 array using 16 sata disks:
mdadm -C
/dev/md10-l 5 -n 16 /dev/sd[b-q]then use dd to stress the io:
dd if=/dev/zero of=/dev/md10 bs=1M
After a while, usually several minutes, the system hangs. It looks like some kind of kernel deadlock. ping to this machine could get timely response, however, any other process just hangs.
Quoted 10 lines
It is easily reproducible and everytime the test result is a system hang.
Johnson
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 7 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Neil Brown ([email protected]) wrote on 25 November 2010 07:01:
Quoted 12 lines · On Wed, 24 Nov 2010 22:34:21 +0800 Du Jun <[email protected]> wrote:
Quoted 8 lines
Their's no dmesg information at all before the hang. The system just hangs suddenly without any kernel printk or oops.
mdadm version is debian lenny version 2.6.7.2-3
I will get mdadm -E & mdadm -D informations tomorrow but I doubt they are useful.
Probably not, but they wouldn't hurt either.
Quoted 8 lines
What would really help is if you could capture the result of
sysrq-Twhich I accept might be a challenge depending on how hard the machine really has frozen.
Also, make sure the
LOCKUP_DETECTORconfig options are set, and wait a few minutes after the hand to see if something pops up.
You could also boot with the NMI watchdog. Using netconsole to capture the full traces would be useful as well.
Hi, Neil,
attached is the mdadm -E and mdadm -D information for md10(raid5 with 16 disks) and md11 (raid6 with 16 disks).
I recompiled the 2.6.36 kernel with kernel sysrq and softlock dectet option, then setup two mds to reproduce the softlock bug.
First create two md using mdadm:
mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q] mdadm -C /dev/md10 -l 6 -n 16 /dev/sd[r-z] /dev/sda[a-g]
then use dd :
dd if=/dev/zero of=/dev/md10 bs=1M & dd if=/dev/zero of=/dev/md11 bs=1M&
After a while, the system is hung with lots of softlock kernel trace on screen. following are some netconsole output:
Nov 25 11:01:35 10.10.20.20 [ 1236.027001] BUG: soft lockup - CPU#0 stuck for 67s! [top:5690] Nov 25 11:01:40 10.10.20.20 [ 1236.027001] Stack: Nov 25 11:01:45 10.10.20.20 [ 1236.027001] Call Trace: Nov 25 11:01:50 10.10.20.20 [ 1236.027001] Code: Nov 25 11:01:55 10.10.20.20 03 Nov 25 11:02:00 10.10.20.20 81 Nov 25 11:02:05 10.10.20.20 48 Nov 25 11:02:10 10.10.20.20 c7 Nov 25 11:02:15 10.10.20.20 c1 Nov 25 11:02:20 10.10.20.20 91 Nov 25 11:02:25 10.10.20.20 19 Nov 25 11:02:30 10.10.20.20 03 Nov 25 11:02:35 10.10.20.20 81 Nov 25 11:02:40 10.10.20.20 e9 Nov 25 11:02:45 10.10.20.20 dd Nov 25 11:02:50 10.10.20.20 fe Nov 25 11:02:55 10.10.20.20 ff Nov 25 11:03:00 10.10.20.20 ff Nov 25 11:03:05 10.10.20.20 90 Nov 25 11:03:10 10.10.20.20 90 Nov 25 11:03:15 10.10.20.20 90 Nov 25 11:03:20 10.10.20.20 90 Nov 25 11:03:25 10.10.20.20 90 Nov 25 11:03:30 10.10.20.20 90 Nov 25 11:03:35 10.10.20.20 90 Nov 25 11:03:40 10.10.20.20 90 Nov 25 11:03:45 10.10.20.20 90 Nov 25 11:03:50 10.10.20.20 90 Nov 25 11:03:55 10.10.20.20 90 Nov 25 11:04:00 10.10.20.20 90 Nov 25 11:04:05 10.10.20.20 90 Nov 25 11:04:10 10.10.20.20 b8 Nov 25 11:04:15 10.10.20.20 00 Nov 25 11:04:20 10.10.20.20 01 Nov 25 11:04:25 10.10.20.20 00 Nov 25 11:04:30 10.10.20.20 00 Nov 25 11:04:35 10.10.20.20 f0 Nov 25 11:04:40 10.10.20.20 66 Nov 25 11:04:45 10.10.20.20 0f Nov 25 11:04:50 10.10.20.20 c1 Nov 25 11:04:55 10.10.20.20 07 Nov 25 11:05:00 10.10.20.20 38 Nov 25 11:05:05 10.10.20.20 e0 Nov 25 11:05:10 10.10.20.20 74 Nov 25 11:05:15 10.10.20.20 06 Nov 25 11:05:20 10.10.20.20 f3 Nov 25 11:05:25 10.10.20.20 90
Message from [email protected] at Nov 25 11:05:30 ...
a>
Nov 25 11:05:35 10.10.20.20 07 Nov 25 11:05:40 10.10.20.20 eb Nov 25 11:05:45 10.10.20.20 f6 Nov 25 11:05:50 10.10.20.20 c3 Nov 25 11:05:55 10.10.20.20 66 Nov 25 11:06:00 10.10.20.20 66 Nov 25 11:06:05 10.10.20.20 2e Nov 25 11:06:10 10.10.20.20 0f Nov 25 11:06:15 10.10.20.20 1f Nov 25 11:06:20 10.10.20.20 84 Nov 25 11:06:25 10.10.20.20 00 Nov 25 11:06:30 10.10.20.20 00 Nov 25 11:06:35 10.10.20.20 00 Nov 25 11:06:40 10.10.20.20 00 Nov 25 11:06:45 10.10.20.20 00 Nov 25 11:06:50 10.10.20.20 0f Nov 25 11:06:55 10.10.20.20 b7 Nov 25 11:07:00 10.10.20.20 07 Nov 25 11:07:05 10.10.20.20 38 Nov 25 11:07:10 10.10.20.20 e0 Nov 25 11:07:15 10.10.20.20 Nov 25 11:07:21 10.10.20.20 [ 1236.099000] BUG: soft lockup - CPU#1 stuck for 67s! [pvscan:6172] Nov 25 11:07:26 10.10.20.20 [ 1236.099001] Stack: Nov 25 11:07:31 10.10.20.20 [ 1236.099001] Call Trace: Nov 25 11:07:36 10.10.20.20 [ 1236.099001] Code: Nov 25 11:07:41 10.10.20.20 39 Nov 25 11:07:46 10.10.20.20 ea Nov 25 11:07:51 10.10.20.20 0f Nov 25 11:07:56 10.10.20.20 1f Nov 25 11:08:01 10.10.20.20 00 Nov 25 11:08:06 10.10.20.20 77 Nov 25 11:08:11 10.10.20.20 a8 Nov 25 11:08:16 10.10.20.20 89 Nov 25 11:08:21 10.10.20.20 de Nov 25 11:08:26 10.10.20.20 89 Nov 25 11:08:31 10.10.20.20 e9 Nov 25 11:08:36 10.10.20.20 49 Nov 25 11:08:41 10.10.20.20 8d Nov 25 11:08:46 10.10.20.20 34 Nov 25 11:08:51 10.10.20.20 f7 Nov 25 11:08:56 10.10.20.20 29 Nov 25 11:09:01 10.10.20.20 d9 Nov 25 11:09:06 10.10.20.20 45 Nov 25 11:09:11 10.10.20.20 89 Nov 25 11:09:16 10.10.20.20 e1 Nov 25 11:09:21 10.10.20.20 4c Nov 25 11:09:26 10.10.20.20 8d Nov 25 11:09:31 10.10.20.20 44 Nov 25 11:09:36 10.10.20.20 24 Nov 25 11:09:41 10.10.20.20 08
the sysrq-T just print "showstate" on screen. no more useful information.
thanks.
Johnson
Quoted 18 lines · 在 2010-11-25四的 07:01 +1100,Neil Brown写道:
Quoted 7 lines · On Wed, 24 Nov 2010 22:34:21 +0800 Du Jun <[email protected]> wrote:
Their's no dmesg information at all before the hang. The system just hangs suddenly without any kernel printk or oops.
mdadm version is debian lenny version 2.6.7.2-3
I will get mdadm -E & mdadm -D informations tomorrow but I doubt they are useful.
Probably not, but they wouldn't hurt either.
What would really help is if you could capture the result of
sysrq-Twhich I accept might be a challenge depending on how hard the machine really has frozen.
Quoted 62 lines
Also, make sure the
LOCKUP_DETECTORconfig options are set, and wait a few minutes after the hand to see if something pops up.I'll try to see if I can reproduce, but I don't have 16 disks so it may not work.
NeilBrown
Quoted 51 lines
Johnson
Quoted 36 lines · Mathias Burén
Hi,
I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds), mdadm -D md10, zcat
/proc/config.gz, mdadm version)Regards,
// Mathias2010/11/24 Du Jun<[email protected]>:
Quoted 16 lines
Hi,
We just quick tested the raid stability under the vanilla 2.6.36 kernel and got an unstable result.first, we create a raid5 array using 16 sata disks:
mdadm -C
/dev/md10-l 5 -n 16 /dev/sd[b-q]then use dd to stress the io:
dd if=/dev/zero of=/dev/md10 bs=1M
After a while, usually several minutes, the system hangs. It looks like some kind of kernel deadlock. ping to this machine could get timely response, however, any other process just hangs.
Quoted 10 lines
It is easily reproducible and everytime the test result is a system hang.
Johnson
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 7 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
/dev/md10:
Version : 00.90
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Persistence : Superblock is persistent
Update Time : Thu Nov 25 10:42:10 2010
State : clean, degraded, recovering
Active Devices : 15
Working Devices : 16
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 2% complete
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Events : 0.4
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
3 8 64 3 active sync /dev/sde
4 8 80 4 active sync /dev/sdf
5 8 96 5 active sync /dev/sdg
6 8 112 6 active sync /dev/sdh
7 8 128 7 active sync /dev/sdi
8 8 144 8 active sync /dev/sdj
9 8 160 9 active sync /dev/sdk
10 8 176 10 active sync /dev/sdl
11 8 192 11 active sync /dev/sdm
12 8 208 12 active sync /dev/sdn
13 8 224 13 active sync /dev/sdo
14 8 240 14 active sync /dev/sdp
16 65 0 15 spare rebuilding /dev/sdq
/dev/md11:
Version : 00.90
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Persistence : Superblock is persistent
Update Time : Thu Nov 25 10:43:51 2010
State : clean, degraded, recovering
Active Devices : 15
Working Devices : 16
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 2% complete
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Events : 0.2
Number Major Minor RaidDevice State
0 65 16 0 active sync /dev/sdr
1 65 32 1 active sync /dev/sds
2 65 48 2 active sync /dev/sdt
3 65 64 3 active sync /dev/sdu
4 65 80 4 active sync /dev/sdv
5 65 96 5 active sync /dev/sdw
6 65 112 6 active sync /dev/sdx
7 65 128 7 active sync /dev/sdy
8 65 144 8 active sync /dev/sdz
9 65 160 9 active sync /dev/sdaa
10 65 176 10 active sync /dev/sdab
11 65 192 11 active sync /dev/sdac
12 65 208 12 active sync /dev/sdad
13 65 224 13 active sync /dev/sdae
14 65 240 14 active sync /dev/sdaf
16 66 0 15 spare rebuilding /dev/sdag
/dev/sdaa:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d61 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 9 65 160 9 active sync /dev/sdaa
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdab:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d73 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 10 65 176 10 active sync /dev/sdab
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdac:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d85 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 11 65 192 11 active sync /dev/sdac
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdad:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d97 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 12 65 208 12 active sync /dev/sdad
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdae:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62da9 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 13 65 224 13 active sync /dev/sdae
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdaf:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62dbb - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 14 65 240 14 active sync /dev/sdaf
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdag:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cca - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 16 66 0 16 spare /dev/sdag
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdb:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4090 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 16 0 active sync /dev/sdb
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdc:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40a2 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 32 1 active sync /dev/sdc
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdd:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40b4 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 48 2 active sync /dev/sdd
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sde:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40c6 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 64 3 active sync /dev/sde
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdf:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40d8 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 80 4 active sync /dev/sdf
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdg:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40ea - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 96 5 active sync /dev/sdg
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdh:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40fc - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 112 6 active sync /dev/sdh
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdi:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a410e - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 8 128 7 active sync /dev/sdi
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdj:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4120 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 144 8 active sync /dev/sdj
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdk:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4132 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 9 8 160 9 active sync /dev/sdk
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdl:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4144 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 10 8 176 10 active sync /dev/sdl
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdm:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4156 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 11 8 192 11 active sync /dev/sdm
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdn:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4168 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 12 8 208 12 active sync /dev/sdn
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdo:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a417a - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 13 8 224 13 active sync /dev/sdo
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdp:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a418c - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 14 8 240 14 active sync /dev/sdp
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdq:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40d3 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 16 65 0 16 spare /dev/sdq
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdr:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cbf - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 65 16 0 active sync /dev/sdr
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sds:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cd1 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 65 32 1 active sync /dev/sds
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdt:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62ce3 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 65 48 2 active sync /dev/sdt
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdu:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cf5 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 65 64 3 active sync /dev/sdu
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdv:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d07 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 65 80 4 active sync /dev/sdv
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdw:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d19 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 65 96 5 active sync /dev/sdw
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdx:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d2b - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 65 112 6 active sync /dev/sdx
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdy:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d3d - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 65 128 7 active sync /dev/sdy
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdz:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB) Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d4f - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 65 144 8 active sync /dev/sdz
0 0 65 16 0 active sync /dev/sdr 1 1 65 32 1 active sync /dev/sds 2 2 65 48 2 active sync /dev/sdt 3 3 65 64 3 active sync /dev/sdu 4 4 65 80 4 active sync /dev/sdv 5 5 65 96 5 active sync /dev/sdw 6 6 65 112 6 active sync /dev/sdx 7 7 65 128 7 active sync /dev/sdy 8 8 65 144 8 active sync /dev/sdz 9 9 65 160 9 active sync /dev/sdaa 10 10 65 176 10 active sync /dev/sdab 11 11 65 192 11 active sync /dev/sdac 12 12 65 208 12 active sync /dev/sdad 13 13 65 224 13 active sync /dev/sdae 14 14 65 240 14 active sync /dev/sdaf 15 15 0 0 15 faulty removed 16 16 66 0 16 spare /dev/sdag
Hi Neil,
We are currently using 2.6.36 on a production machine running mdadm, and this morning we experienced a very similar issue, but we didn't have the LOCKUP_DETECTOR config options set, so there was absolutely no output. We also experienced this on a non-production test platform a few days ago (same hardware and software as our production machine).
Has this bug been diagnosed/resolved? If so, past which kernel version? If not, can you suggest some recourse for us, like perhaps moving back to 2.6.35 for our production server, and moving up to the latest stable on our test platform, and enabling the LOCKUP_DETECTOR options?
Is this bug totally reproducible Du Jun? If so, should I just try what you outlined in your previous message?
Many thanks,
Iordan Iordanov
Hi Neil,
As requested, here is more background.
Quoted 1 line · Iordan Iordanov
We are currently using 2.6.36 on a production machine running mdadm
Our distribution is Debian Lenny. We have 10 3-way RAID1 mirrors (30 1TB disks) striped together with LVM in one large volume group, out of which we create logical drives. The actual drives are iscsi targets we import from separate machines. The system is serving about 3500 active (not logged-in at the same time) accounts to our department. Also, we have a "development" platform, with no users using it, also with lvm, mdadm, and iscsi, but with 5 disks instead of 30. The development platform was sitting idle the whole time.
Quoted 2 lines
this morning we experienced a very similar issue, but we didn't have the
LOCKUP_DETECTORconfig options set, so there was absolutely no output.
What I mean by a similar issue is that our development platform hung without any output in the logs and was completely unresponsive on Feb 7, and our production system hung and was completely unresponsive, save for pings, on Feb 10. I don't know if the two hangs were caused by the same issue.
I thought that the potential lock-up reported by Du Jun was acknowledged as reality. I was wrong. We will recompile a longterm kernel (this time) with the LOCKUP_DETECTOR flags set, and will wait to see if the issue recurs. Do you have any suggestions to us?
Many thanks,
Iordan