Skip to contentSkip to conversation

RAID 5: weird size results after Grow

DISCUSSION Marko Berg 4 participants 9 messages Oct 13, 2007 – Oct 14, 2007 MD RAID

lore
Share & tools
lore

Marko Berg

author
#0 lore raw

Hi folks,

I added a fourth drive to a RAID 5 array. After some complications related to adding a new HD controller at the same time, and thus changing some device names, I re-created the array and got it working (in the sense "nothing degraded"). But size results are weird. Each component partition is 320 G, does anyone have an explanation for the "Used Dev Size" field value below? The 960 G total size is as it should be, but in practice Linux reports the array only having 625,019,608 blocks. How can this be, even though the array should be clean with 4 active devices?

$  mdadm -D /dev/md0
/dev/md0:
       Version : 01.02.03
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
    Array Size : 937705344 (894.27 GiB 960.21 GB)
 Used Dev Size : 625136896 (298.09 GiB 320.07 GB)
  Raid Devices : 4
 Total Devices : 4
Preferred Minor : 0
   Persistence : Superblock is persistent

Update Time : Sat Oct 13 05:11:38 2007
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Name : 0
UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
Events : 2

Number Major Minor RaidDevice State
0 253 2 0 active sync
/dev/VolGroup01/LogVol02

      1       8       33        1      active sync   /dev/sdc1
      2       8       49        2      active sync   /dev/sdd1
      3       8       17        3      active sync   /dev/sdb1

Results for mdadm -E <partition> on all devices appear like this one, with positions changed:

$ mdadm -E /dev/sdc1
/dev/sdc1:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
          Name : 0
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
  Raid Devices : 4

Used Dev Size : 625137010 (298.09 GiB 320.07 GB)

    Array Size : 1875410688 (894.27 GiB 960.21 GB)
     Used Size : 625136896 (298.09 GiB 320.07 GB)

Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9b2037fb:231a8ebf:1aaa5577:140795cc

Update Time : Sat Oct 13 10:56:02 2007
Checksum : c729f5a1 - correct
Events : 2

Layout : left-symmetric
Chunk Size : 64K

Array Slot : 1 (0, 1, 2, 3)
Array State : uUuu

Particularly, "Used Dev Size" and "Used Size" report an amount twice the size of the partition (and device). Array size is here twice the actual size, even though shown correctly within parentheses.

Finally, mdstat shows the block count as it should be.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[3] sdd1[2] sdc1[1] dm-2[0]
     937705344 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] 
[UUUU]

unused devices: <none>

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

Signature · 1 line
Marko
#1 lore raw
Quoted 3 lines · Marko Berg

Hi folks,

I added a fourth drive to a RAID 5 array. After some complications related to adding a new HD controller at the same time, and thus changing some device names, I re-created the array and got it working (in the sense "nothing degraded"). But size results are weird. Each component partition is 320 G, does anyone have an explanation for the "Used Dev Size" field value below? The 960 G total size is as it should be, but in practice Linux reports the array only having 625,019,608 blocks.

I don't see that number below, what command reported this?

Quoted 61 lines

How can this be, even though the array should be clean with 4 active devices?

$  mdadm -D /dev/md0
/dev/md0:
       Version : 01.02.03
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
    Array Size : 937705344 (894.27 GiB 960.21 GB)
 Used Dev Size : 625136896 (298.09 GiB 320.07 GB)
  Raid Devices : 4
 Total Devices : 4
Preferred Minor : 0
   Persistence : Superblock is persistent

Update Time : Sat Oct 13 05:11:38 2007
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Name : 0
UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
Events : 2

Number Major Minor RaidDevice State
0 253 2 0 active sync
/dev/VolGroup01/LogVol02

      1       8       33        1      active sync   /dev/sdc1
      2       8       49        2      active sync   /dev/sdd1
      3       8       17        3      active sync   /dev/sdb1

Results for mdadm -E <partition> on all devices appear like this one, with positions changed:

$ mdadm -E /dev/sdc1
/dev/sdc1:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
          Name : 0
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
  Raid Devices : 4

Used Dev Size : 625137010 (298.09 GiB 320.07 GB)

    Array Size : 1875410688 (894.27 GiB 960.21 GB)
     Used Size : 625136896 (298.09 GiB 320.07 GB)

Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9b2037fb:231a8ebf:1aaa5577:140795cc

Update Time : Sat Oct 13 10:56:02 2007
Checksum : c729f5a1 - correct
Events : 2

Quoted 8 lines

Layout : left-symmetric
Chunk Size : 64K

   Array Slot : 1 (0, 1, 2, 3)

Array State : uUuu

Particularly, "Used Dev Size" and "Used Size" report an amount twice the size of the partition (and device). Array size is here twice the actual size, even though shown correctly within parentheses.

Sectors are 512 bytes.

Quoted 11 lines

Finally, mdstat shows the block count as it should be.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[3] sdd1[2] sdc1[1] dm-2[0]
     937705344 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] 
[UUUU]
    unused devices: <none>
Quoted 2 lines

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

I'm not sure what you're trying to "fix" here, everything you posted looks sane.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
author
#2 lore raw
Quoted 4 lines · Bill Davidsen
Quoted 1 line · Marko Berg

I added a fourth drive to a RAID 5 array. After some complications related to adding a new HD controller at the same time, and thus changing some device names, I re-created the array and got it working (in the sense "nothing degraded"). But size results are weird. Each component partition is 320 G, does anyone have an explanation for the "Used Dev Size" field value below? The 960 G total size is as it should be, but in practice Linux reports the array only having 625,019,608 blocks.

I don't see that number below, what command reported this?

For instance df:

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0             625019608 358223356 235539408  61% /usr/pub
Quoted 64 lines
Quoted 64 lines

How can this be, even though the array should be clean with 4 active devices?

$  mdadm -D /dev/md0
/dev/md0:
       Version : 01.02.03
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
    Array Size : 937705344 (894.27 GiB 960.21 GB)
 Used Dev Size : 625136896 (298.09 GiB 320.07 GB)
  Raid Devices : 4
 Total Devices : 4
Preferred Minor : 0
   Persistence : Superblock is persistent

Update Time : Sat Oct 13 05:11:38 2007
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Name : 0
UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
Events : 2

Number Major Minor RaidDevice State
0 253 2 0 active sync
/dev/VolGroup01/LogVol02

      1       8       33        1      active sync   /dev/sdc1
      2       8       49        2      active sync   /dev/sdd1
      3       8       17        3      active sync   /dev/sdb1

Results for mdadm -E <partition> on all devices appear like this one, with positions changed:

$ mdadm -E /dev/sdc1
/dev/sdc1:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
          Name : 0
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
  Raid Devices : 4

Used Dev Size : 625137010 (298.09 GiB 320.07 GB)

    Array Size : 1875410688 (894.27 GiB 960.21 GB)
     Used Size : 625136896 (298.09 GiB 320.07 GB)

Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9b2037fb:231a8ebf:1aaa5577:140795cc

Update Time : Sat Oct 13 10:56:02 2007
Checksum : c729f5a1 - correct
Events : 2

Layout : left-symmetric
Chunk Size : 64K

Quoted 7 lines
Quoted 5 lines
   Array Slot : 1 (0, 1, 2, 3)

Array State : uUuu

Particularly, "Used Dev Size" and "Used Size" report an amount twice the size of the partition (and device). Array size is here twice the actual size, even though shown correctly within parentheses.

Sectors are 512 bytes.

So "Used Dev Size" above uses sector size, while "Array Size" uses 1k blocks? I'm pretty sure, though, that previously "Used Dev Size" was in 1k blocks too. That's also what most of the examples in the net seem to have.

Quoted 10 lines
Quoted 10 lines

Finally, mdstat shows the block count as it should be.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[3] sdd1[2] sdc1[1] dm-2[0]
     937705344 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] 
[UUUU]
    unused devices: <none>
Quoted 3 lines
Quoted 2 lines

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

I'm not sure what you're trying to "fix" here, everything you posted looks sane.

I'm trying to find the missing 300 GB that, as df reports, are not available. I ought to have a 900 GB array, consisting of four 300 GB devices, while only 600 GB are available. Adding the fourth device didn't increase the capacity of the array (visible, at least). E.g. fdisk reports the array size to be 900 G, but df still claims 600 capacity. Any clues why?

Signature · 1 line
Marko
#3 lore raw
Quoted 7 lines · Marko Berg
Quoted 4 lines · Bill Davidsen
Quoted 2 lines · Marko Berg

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

I'm not sure what you're trying to "fix" here, everything you posted looks sane.

I'm trying to find the missing 300 GB that, as df reports, are not available. I ought to have a 900 GB array, consisting of four 300 GB devices, while only 600 GB are available. Adding the fourth device didn't increase the capacity of the array (visible, at least). E.g. fdisk reports the array size to be 900 G, but df still claims 600 capacity. Any clues why?

df reports the size of the filesystem, which is still about 600GB--the filesystem doesn't resize automatically when the size of the underlying device changes.

You'll need to use resize2fs, resize_reiserfs, or whatever other tool is appropriate for your type of filesystem.

-Corey

author
#4 lore raw
Quoted 7 lines · Corey Hickey
Quoted 6 lines · Marko Berg
Quoted 4 lines · Bill Davidsen
Quoted 2 lines · Marko Berg

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

I'm not sure what you're trying to "fix" here, everything you posted looks sane.

Quoted 7 lines
Quoted 1 line

I'm trying to find the missing 300 GB that, as df reports, are not available. I ought to have a 900 GB array, consisting of four 300 GB devices, while only 600 GB are available. Adding the fourth device didn't increase the capacity of the array (visible, at least). E.g. fdisk reports the array size to be 900 G, but df still claims 600 capacity. Any clues why?

df reports the size of the filesystem, which is still about 600GB--the filesystem doesn't resize automatically when the size of the underlying device changes.

You'll need to use resize2fs, resize_reiserfs, or whatever other tool is appropriate for your type of filesystem.

-Corey

Right, so this isn't one of my sharpest days... Thanks a bunch, Corey!

Signature · 1 line
Marko
#5 lore raw
Quoted 10 lines · Marko Berg
Quoted 9 lines · Corey Hickey
Quoted 7 lines · Marko Berg
Quoted 4 lines · Bill Davidsen
Quoted 2 lines · Marko Berg

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

I'm not sure what you're trying to "fix" here, everything you posted looks sane.

I'm trying to find the missing 300 GB that, as df reports, are not available. I ought to have a 900 GB array, consisting of four 300 GB devices, while only 600 GB are available. Adding the fourth device didn't increase the capacity of the array (visible, at least). E.g. fdisk reports the array size to be 900 G, but df still claims 600 capacity. Any clues why?

Quoted 15 lines
Quoted 5 lines

df reports the size of the filesystem, which is still about 600GB--the filesystem doesn't resize automatically when the size of the underlying device changes.

You'll need to use resize2fs, resize_reiserfs, or whatever other tool is appropriate for your type of filesystem.

-Corey

Right, so this isn't one of my sharpest days... Thanks a bunch, Corey!

Signature · 5 lines
Marko
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Ah, already answered.

Justin.

#6 lore raw
Quoted 9 lines · Justin Piszcz
Quoted 6 lines · Marko Berg
Quoted 5 lines · Corey Hickey
Quoted 4 lines · Marko Berg
Quoted 3 lines · Bill Davidsen
Quoted 2 lines · Marko Berg

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Quoted 1 line

I'm not sure what you're trying to "fix" here, everything you posted looks sane.

I'm trying to find the missing 300 GB that, as df reports, are not available. I ought to have a 900 GB array, consisting of four 300 GB devices, while only 600 GB are available. Adding the fourth device didn't increase the capacity of the array (visible, at least). E.g. fdisk reports the array size to be 900 G, but df still claims 600 capacity. Any clues why?

df reports the size of the filesystem, which is still about 600GB--the filesystem doesn't resize automatically when the size of the underlying device changes.

You'll need to use resize2fs, resize_reiserfs, or whatever other tool is appropriate for your type of filesystem.

-Corey

Right, so this isn't one of my sharpest days... Thanks a bunch, Corey!

No problem.

Quoted 1 line

Ah, already answered.

vger.kernel.org greylisted my smtp server, so it took my message a while to get to the list.

-Corey

#7 lore raw
In reply to Marko Berg
Quoted 96 lines · Marko Berg
Quoted 4 lines · Bill Davidsen
Quoted 1 line · Marko Berg

I added a fourth drive to a RAID 5 array. After some complications related to adding a new HD controller at the same time, and thus changing some device names, I re-created the array and got it working (in the sense "nothing degraded"). But size results are weird. Each component partition is 320 G, does anyone have an explanation for the "Used Dev Size" field value below? The 960 G total size is as it should be, but in practice Linux reports the array only having 625,019,608 blocks.

I don't see that number below, what command reported this?

For instance df:

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0             625019608 358223356 235539408  61% /usr/pub
Quoted 64 lines
Quoted 64 lines

How can this be, even though the array should be clean with 4 active devices?

$  mdadm -D /dev/md0
/dev/md0:
       Version : 01.02.03
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
    Array Size : 937705344 (894.27 GiB 960.21 GB)
 Used Dev Size : 625136896 (298.09 GiB 320.07 GB)
  Raid Devices : 4
 Total Devices : 4
Preferred Minor : 0
   Persistence : Superblock is persistent

Update Time : Sat Oct 13 05:11:38 2007
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Name : 0
UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
Events : 2

Number Major Minor RaidDevice State
0 253 2 0 active sync
/dev/VolGroup01/LogVol02

      1       8       33        1      active sync   /dev/sdc1
      2       8       49        2      active sync   /dev/sdd1
      3       8       17        3      active sync   /dev/sdb1

Results for mdadm -E <partition> on all devices appear like this one, with positions changed:

$ mdadm -E /dev/sdc1
/dev/sdc1:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 9bf903f8:7fc9eec1:2ff25011:37e9607b
          Name : 0
 Creation Time : Sat Oct 13 01:25:26 2007
    Raid Level : raid5
  Raid Devices : 4

Used Dev Size : 625137010 (298.09 GiB 320.07 GB)

    Array Size : 1875410688 (894.27 GiB 960.21 GB)
     Used Size : 625136896 (298.09 GiB 320.07 GB)

Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9b2037fb:231a8ebf:1aaa5577:140795cc

Update Time : Sat Oct 13 10:56:02 2007
Checksum : c729f5a1 - correct
Events : 2

Layout : left-symmetric
Chunk Size : 64K

Quoted 7 lines
Quoted 5 lines
   Array Slot : 1 (0, 1, 2, 3)

Array State : uUuu

Particularly, "Used Dev Size" and "Used Size" report an amount twice the size of the partition (and device). Array size is here twice the actual size, even though shown correctly within parentheses.

Sectors are 512 bytes.

So "Used Dev Size" above uses sector size, while "Array Size" uses 1k blocks? I'm pretty sure, though, that previously "Used Dev Size" was in 1k blocks too. That's also what most of the examples in the net seem to have.

Quoted 10 lines
Quoted 10 lines

Finally, mdstat shows the block count as it should be.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[3] sdd1[2] sdc1[1] dm-2[0]
     937705344 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] 
[UUUU]
    unused devices: <none>
Quoted 13 lines
Quoted 3 lines
Quoted 2 lines

Any suggestions on how to fix this, or what to investigate next, would be appreciated!

I'm not sure what you're trying to "fix" here, everything you posted looks sane.

I'm trying to find the missing 300 GB that, as df reports, are not available. I ought to have a 900 GB array, consisting of four 300 GB devices, while only 600 GB are available. Adding the fourth device didn't increase the capacity of the array (visible, at least). E.g. fdisk reports the array size to be 900 G, but df still claims 600 capacity. Any clues why?

Signature · 5 lines
Marko
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

You have to expand the filesystem.

#8 lore raw
In reply to Marko Berg
Quoted 4 lines · Marko Berg
Quoted 3 lines · Bill Davidsen
Quoted 1 line · Marko Berg

I added a fourth drive to a RAID 5 array. After some complications related to adding a new HD controller at the same time, and thus changing some device names, I re-created the array and got it working (in the sense "nothing degraded"). But size results are weird. Each component partition is 320 G, does anyone have an explanation for the "Used Dev Size" field value below? The 960 G total size is as it should be, but in practice Linux reports the array only having 625,019,608 blocks.

Quoted 9 lines
Quoted 1 line

I don't see that number below, what command reported this?

For instance df:

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0             625019608 358223356 235539408  61% /usr/pub
Quoted 1 line
Quoted 1 line

How can this be, even though the array should be clean with 4 active devices?

df reports the size of the filesystem, mdadm reports the size of the array.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979