Raid0 rebuild slows to a crawl

DISCUSSION David Liontooth 4 participants 5 messages Jan 13, 2012 – Jan 14, 2012 MD RAID

Share & tools

David Liontooth

author

2012-01-13 08:01 UTC #0 lore raw

I'm trying to use a WD3200AAKX-001CA0 drive in a raid1 configuration (Debian squeeze mdadm 3.1.4-1+8efb9d1), but adding it is unusably slow:

# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2] sda3[1]
      73015360 blocks [2/1] [_U]
      [>....................]  recovery =  0.0% (28928/73015360) 
finish=2521.6min speed=482K/sec

dmesg shows no errors:

md: recovery of RAID array md0
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.

The drive itself doesn't seem to have a problem, though it's not particularly fast:

# hdparm -tT /dev/sdb

/dev/sdb:
Timing cached reads: 11220 MB in 2.00 seconds = 5614.03 MB/sec
Timing buffered disk reads: 174 MB in 6.44 seconds = 27.01 MB/sec

# smartctl -a /dev/sdb

5 Reallocated_Sector_Ct 0x0033 133 133 140 Pre-fail Always FAILING_NOW 1265

# hdparm -i /dev/sdb

/dev/sdb:

Model=WDC WD3200AAKX-001CA0, FwRev=15.01H15, SerialNo=WD-WCAYUEJ66523 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50 BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=625142448 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7

* signifies the current active mode

Is this a known problem? Only see it with this WD drive.

Cheers,
Dave

Kay Diederichs

2012-01-13 08:44 UTC #1 lore raw

Quoted 51 lines · David Liontooth

I'm trying to use a WD3200AAKX-001CA0 drive in a raid1 configuration (Debian squeeze mdadm 3.1.4-1+8efb9d1), but adding it is unusably slow:
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2] sda3[1]
73015360 blocks [2/1] [_U]
[>....................] recovery = 0.0% (28928/73015360)
finish=2521.6min speed=482K/sec
dmesg shows no errors:
md: recovery of RAID array md0
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
The drive itself doesn't seem to have a problem, though it's not particularly fast:
# hdparm -tT /dev/sdb
/dev/sdb:
Timing cached reads: 11220 MB in 2.00 seconds = 5614.03 MB/sec Timing buffered disk reads: 174 MB in 6.44 seconds = 27.01 MB/sec
# smartctl -a /dev/sdb
5 Reallocated_Sector_Ct 0x0033 133 133 140 Pre-fail Always FAILING_NOW 1265
# hdparm -i /dev/sdb
/dev/sdb:
Model=WDC WD3200AAKX-001CA0, FwRev=15.01H15, SerialNo=WD-WCAYUEJ66523 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50 BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=625142448 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7
* signifies the current active mode
Is this a known problem? Only see it with this WD drive.

Quoted 9 lines

Cheers,
Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Dave,

I'd say your drive is dying if its Reallocated_Sector_Ct is 1265. Dying drives often are really slow; they spend the time on trying to overcome error states.

HTH,
Kay

David Liontooth

author

2012-01-13 09:36 UTC #2 lore raw

Much appreciated! I'll notify next-of-kin.

Cheers,
Dave

Quoted 63 lines · Kay Diederichs

Quoted 47 lines · David Liontooth
I'm trying to use a WD3200AAKX-001CA0 drive in a raid1 configuration (Debian squeeze mdadm 3.1.4-1+8efb9d1), but adding it is unusably slow:
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2] sda3[1]
73015360 blocks [2/1] [_U]
[>....................] recovery = 0.0% (28928/73015360)
finish=2521.6min speed=482K/sec
dmesg shows no errors:
md: recovery of RAID array md0
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
The drive itself doesn't seem to have a problem, though it's not particularly fast:
# hdparm -tT /dev/sdb
/dev/sdb:
Timing cached reads: 11220 MB in 2.00 seconds = 5614.03 MB/sec Timing buffered disk reads: 174 MB in 6.44 seconds = 27.01 MB/sec
# smartctl -a /dev/sdb
5 Reallocated_Sector_Ct 0x0033 133 133 140 Pre-fail Always FAILING_NOW 1265
# hdparm -i /dev/sdb
/dev/sdb:
Model=WDC WD3200AAKX-001CA0, FwRev=15.01H15, SerialNo=WD-WCAYUEJ66523 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50 BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=625142448 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7
Quoted 7 lines
* signifies the current active mode
Is this a known problem? Only see it with this WD drive.
Cheers,
Dave
Hi Dave,
I'd say your drive is dying if its Reallocated_Sector_Ct is 1265. Dying drives often are really slow; they spend the time on trying to overcome error states.
HTH,
Kay

Robin Hill

2012-01-13 08:46 UTC #3 lore raw

Re: Raid1 rebuild slows to a crawl

In reply to David Liontooth

Quoted 23 lines · David Liontooth

I'm trying to use a WD3200AAKX-001CA0 drive in a raid1 configuration (Debian squeeze mdadm 3.1.4-1+8efb9d1), but adding it is unusably slow:
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2] sda3[1]
       73015360 blocks [2/1] [_U]
       [>....................]  recovery =  0.0% (28928/73015360) 
finish=2521.6min speed=482K/sec
dmesg shows no errors:
md: recovery of RAID array md0
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
The drive itself doesn't seem to have a problem, though it's not particularly fast:
# hdparm -tT /dev/sdb

Quoted 9 lines

/dev/sdb:
Timing cached reads: 11220 MB in 2.00 seconds = 5614.03 MB/sec
Timing buffered disk reads: 174 MB in 6.44 seconds = 27.01 MB/sec
# smartctl -a /dev/sdb
5 Reallocated_Sector_Ct 0x0033 133 133 140 Pre-fail Always
FAILING_NOW 1265

It has 1265 reallocated sectors, reports FAILING_NOW, and you say it doesn't seem to have a problem? At a guess, it's repeatedly getting a write failure, hitting the error retry timeout (which is quite long for consumer drives),failing the sector, then reallocating it and moving on to do the same with the next sector.

Get rid of this drive and get a replacement - there's no way you can rely on it at all.

Cheers,
Robin

Signature · 4 lines

     ___        
    ( ' }     |       Robin Hill        <[email protected]> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

Stefan /St0fF/ Hübner

2012-01-14 20:13 UTC #4 lore raw

In reply to David Liontooth

This drive is failing now - just as smartctl indicates. I suggest you stop that rebuild (as it was dropped for a reason), exchange the drive, then restart/rebuild with a new drive.

Stefan

Am 13.01.2012 09:01, schrieb David Liontooth:

Quoted 60 lines

I'm trying to use a WD3200AAKX-001CA0 drive in a raid1 configuration (Debian squeeze mdadm 3.1.4-1+8efb9d1), but adding it is unusably slow:
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb3[2] sda3[1]
      73015360 blocks [2/1] [_U]
      [>....................]  recovery =  0.0% (28928/73015360)
finish=2521.6min speed=482K/sec
dmesg shows no errors:
md: recovery of RAID array md0
md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
The drive itself doesn't seem to have a problem, though it's not particularly fast:
# hdparm -tT /dev/sdb
/dev/sdb:
Timing cached reads: 11220 MB in 2.00 seconds = 5614.03 MB/sec
Timing buffered disk reads: 174 MB in 6.44 seconds = 27.01 MB/sec
# smartctl -a /dev/sdb
5 Reallocated_Sector_Ct 0x0033 133 133 140 Pre-fail Always
FAILING_NOW 1265
# hdparm -i /dev/sdb
/dev/sdb:
Model=WDC WD3200AAKX-001CA0, FwRev=15.01H15, SerialNo=WD-WCAYUEJ66523 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50 BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=625142448 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7
* signifies the current active mode
Is this a known problem? Only see it with this WD drive.
Cheers,
Dave
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html