Skip to contentSkip to conversation

Device utilization with RAID-1

DISCUSSION Harald Nikolisin 6 participants 7 messages Aug 18, 2011 – Aug 20, 2011 MD RAID

lore
Share & tools
lore
#0 lore raw

hi,

I didn't want to complain in general about SW RAID-1 performance. I simply think something is wrong with my setup and I have currently no idea how to improve.

The basic questions (where I did not find an answer, neither in FAQ's nor in forum discussions) are.
a) Is it normal that the hard drives show an permanent utilization (around 20%) without any noticeable actions on the computer?
b) Should (as long as no resync happens) the state of mdadm active or clean?

cheers,
harald

well, I have only 2 hard drives and no space for more..

Am 16.08.2011 03:29, schrieb Roberto Spadim:

Quoted 51 lines

try raid10 far layout

2011/8/15 Harald Nikolisin <[email protected]
<mailto:[email protected]>>

    Since a long time I'm unhappy with the performance of my RAID-1 system.
    Investigation with atop and iostat unveils that the disk utilization is
    always on a certain level although nothing happens on the system. In the
    case of reading or writing files the utilization boosts always to 100%
    for a long time. Very ugly examples are "Firefox starting" or "zypper
    updates".
    That is snapshot of the output of iostat:


    Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
    avgrq-sz avgqu-sz   await  svctm  %util
    sda               0,00     0,00    0,00    7,33     0,00    43,33
    5,91     0,33   43,18  33,32  24,43
    sdb               0,00     0,00    0,00    7,33     0,00    43,33
    5,91     0,35   45,59  39,73  29,13
    md0               0,00     0,00    0,00    0,67     0,00     5,33
    8,00     0,00    0,00   0,00   0,00
    md1               0,00     0,00    0,00    0,33     0,00     5,33
    16,00     0,00    0,00   0,00   0,00
    md2               0,00     0,00    0,00    0,33     0,00     1,00
    3,00     0,00    0,00   0,00   0,00
    md3               0,00     0,00    0,00    0,00     0,00     0,00
    0,00     0,00    0,00   0,00   0,00
    md4               0,00     0,00    0,00    0,00     0,00     0,00
    0,00     0,00    0,00   0,00   0,00
    md5               0,00     0,00    0,00    0,33     0,00     0,67
    2,00     0,00    0,00   0,00   0,00

    I checked with mdadm if a resync happens or so, but this is not the
    case. The state says "active" on all RAID devices - btw. what is the
    difference to "clean" ?

    thanks for any hints,
     harald
    --
    To unsubscribe from this list: send the line "unsubscribe linux-raid" in
    the body of a message to [email protected]
    <mailto:[email protected]>
    More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Roberto Spadim
Spadim Technology / SPAEmpresarial
#1 lore raw
Quoted 10 lines · On Thu, 18 Aug 2011 02:26:17 +0200 Harald Nikolisin <[email protected]> wrote:

hi,

I didn't want to complain in general about SW RAID-1 performance. I simply think something is wrong with my setup and I have currently no idea how to improve.

The basic questions (where I did not find an answer, neither in FAQ's nor in forum discussions) are.
a) Is it normal that the hard drives show an permanent utilization (around 20%) without any noticeable actions on the computer?

No. If the array is resyncing or recovering then you would expect utilization for as many hours as it takes - but that would show in /proc/mdstat.

Quoted 1 line

b) Should (as long as no resync happens) the state of mdadm active or clean?

If anything has been written to the device in the last 200msec (including e.g. access time updates) then expect it to be 'active'.
If nothing has been written for 200msecc or more, then expect it to be clean.

If you crash while it is active, a resync is needed. If you crash while it is clean, no resync is needed. If you don't crash at all .... that is best :-)

NeilBrown

Quoted 64 lines

cheers,
harald

well, I have only 2 hard drives and no space for more..

Am 16.08.2011 03:29, schrieb Roberto Spadim:

Quoted 37 lines

try raid10 far layout

2011/8/15 Harald Nikolisin <[email protected]
<mailto:[email protected]>>

    Since a long time I'm unhappy with the performance of my RAID-1 system.
    Investigation with atop and iostat unveils that the disk utilization is
    always on a certain level although nothing happens on the system. In the
    case of reading or writing files the utilization boosts always to 100%
    for a long time. Very ugly examples are "Firefox starting" or "zypper
    updates".
    That is snapshot of the output of iostat:


    Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
    avgrq-sz avgqu-sz   await  svctm  %util
    sda               0,00     0,00    0,00    7,33     0,00    43,33
    5,91     0,33   43,18  33,32  24,43
    sdb               0,00     0,00    0,00    7,33     0,00    43,33
    5,91     0,35   45,59  39,73  29,13
    md0               0,00     0,00    0,00    0,67     0,00     5,33
    8,00     0,00    0,00   0,00   0,00
    md1               0,00     0,00    0,00    0,33     0,00     5,33
    16,00     0,00    0,00   0,00   0,00
    md2               0,00     0,00    0,00    0,33     0,00     1,00
    3,00     0,00    0,00   0,00   0,00
    md3               0,00     0,00    0,00    0,00     0,00     0,00
    0,00     0,00    0,00   0,00   0,00
    md4               0,00     0,00    0,00    0,00     0,00     0,00
    0,00     0,00    0,00   0,00   0,00
    md5               0,00     0,00    0,00    0,33     0,00     0,67
    2,00     0,00    0,00   0,00   0,00

    I checked with mdadm if a resync happens or so, but this is not the
    case. The state says "active" on all RAID devices - btw. what is the
    difference to "clean" ?
Quoted 14 lines
    thanks for any hints,
     harald
    --
    To unsubscribe from this list: send the line "unsubscribe linux-raid" in
    the body of a message to [email protected]
    <mailto:[email protected]>
    More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Roberto Spadim
Spadim Technology / SPAEmpresarial

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

Am 18.08.2011 03:42, schrieb NeilBrown:

Quoted 24 lines
Quoted 10 lines · On Thu, 18 Aug 2011 02:26:17 +0200 Harald Nikolisin <[email protected]> wrote:

hi,

I didn't want to complain in general about SW RAID-1 performance. I simply think something is wrong with my setup and I have currently no idea how to improve.

The basic questions (where I did not find an answer, neither in FAQ's nor in forum discussions) are.
a) Is it normal that the hard drives show an permanent utilization (around 20%) without any noticeable actions on the computer?

No. If the array is resyncing or recovering then you would expect utilization for as many hours as it takes - but that would show in /proc/mdstat.

Quoted 1 line

b) Should (as long as no resync happens) the state of mdadm active or clean?

If anything has been written to the device in the last 200msec (including e.g. access time updates) then expect it to be 'active'.
If nothing has been written for 200msecc or more, then expect it to be clean.

Quoted 6 lines

If you crash while it is active, a resync is needed. If you crash while it is clean, no resync is needed. If you don't crash at all .... that is best :-)

NeilBrown

I second that ;) Have you checked the SMART-Attributes of your disks, are they still OK? But if they weren't, you wouldn't see that they're a bit more busy, you'd only feel it from bad performance.

Indeed I think you need to find out which processes create your I/O load, as it seems to be kind of a badly configured service/daemon which slows down your whole computer that way... It's probably a good idea to start with dstat and a wide screen :)

Stefan

Quoted 71 lines
Quoted 62 lines

cheers,
harald

well, I have only 2 hard drives and no space for more..

Am 16.08.2011 03:29, schrieb Roberto Spadim:

Quoted 37 lines

try raid10 far layout

2011/8/15 Harald Nikolisin <[email protected]
<mailto:[email protected]>>

    Since a long time I'm unhappy with the performance of my RAID-1 system.
    Investigation with atop and iostat unveils that the disk utilization is
    always on a certain level although nothing happens on the system. In the
    case of reading or writing files the utilization boosts always to 100%
    for a long time. Very ugly examples are "Firefox starting" or "zypper
    updates".
    That is snapshot of the output of iostat:


    Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
    avgrq-sz avgqu-sz   await  svctm  %util
    sda               0,00     0,00    0,00    7,33     0,00    43,33
    5,91     0,33   43,18  33,32  24,43
    sdb               0,00     0,00    0,00    7,33     0,00    43,33
    5,91     0,35   45,59  39,73  29,13
    md0               0,00     0,00    0,00    0,67     0,00     5,33
    8,00     0,00    0,00   0,00   0,00
    md1               0,00     0,00    0,00    0,33     0,00     5,33
    16,00     0,00    0,00   0,00   0,00
    md2               0,00     0,00    0,00    0,33     0,00     1,00
    3,00     0,00    0,00   0,00   0,00
    md3               0,00     0,00    0,00    0,00     0,00     0,00
    0,00     0,00    0,00   0,00   0,00
    md4               0,00     0,00    0,00    0,00     0,00     0,00
    0,00     0,00    0,00   0,00   0,00
    md5               0,00     0,00    0,00    0,33     0,00     0,67
    2,00     0,00    0,00   0,00   0,00

    I checked with mdadm if a resync happens or so, but this is not the
    case. The state says "active" on all RAID devices - btw. what is the
    difference to "clean" ?
Quoted 14 lines
    thanks for any hints,
     harald
    --
    To unsubscribe from this list: send the line "unsubscribe linux-raid" in
    the body of a message to [email protected]
    <mailto:[email protected]>
    More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Roberto Spadim
Spadim Technology / SPAEmpresarial
Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to NeilBrown
Quoted 20 lines · NeilBrown
Quoted 10 lines · On Thu, 18 Aug 2011 02:26:17 +0200 Harald Nikolisin <[email protected]> wrote:

hi,

I didn't want to complain in general about SW RAID-1 performance. I simply think something is wrong with my setup and I have currently no idea how to improve.

The basic questions (where I did not find an answer, neither in FAQ's nor in forum discussions) are.
a) Is it normal that the hard drives show an permanent utilization (around 20%) without any noticeable actions on the computer?

No.  If the array is resyncing or recovering then you would expect utilization for as many hours as it takes - but that would show in /proc/mdstat.

Quoted 1 line

b) Should (as long as no resync happens) the state of mdadm active or clean?

Quoted 7 lines

If anything has been written to the device in the last 200msec (including e.g. access time updates) then expect it to be 'active'.
If nothing has been written for 200msecc or more, then expect it to be clean.

If you crash while it is active, a resync is needed. If you crash while it is clean, no resync is needed. If you don't crash at all .... that is best :-)

I think this info should be wikified if not yet.

btw, I've experimented a bit on my /boot array (it doesn't being updated, checked with iostat ), and:

root@m2:~# for i in {1..5};do mdname="md0"; echo "iteration $i";
(mdadm --detail /dev/$mdname|grep 'State ';cat
/sys/block/$mdname/md/array_state;grep "$mdname :" /proc/mdstat);sleep
1;done
iteration 1
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 2
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 3
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 4
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 5
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]

so, mdadm --detail & array_state shows array is "clean", while /proc/mdstat shows array is "active" (no reads/writes happen).

Some value is lieing or being misunderdstanded by me...

Quoted 73 lines
Quoted 63 lines

cheers,
  harald

well, I have only 2 hard drives and no space for more..

Am 16.08.2011 03:29, schrieb Roberto Spadim:

Quoted 48 lines

try raid10 far layout

2011/8/15 Harald Nikolisin <[email protected]
<mailto:[email protected]>>

    Since a long time I'm unhappy with the performance of my RAID-1 system. Investigation with atop and iostat unveils that the disk utilization is always on a certain level although nothing happens on the system. In the case of reading or writing files the utilization boosts always to 100% for a long time. Very ugly examples are "Firefox starting" or "zypper updates".
    That is snapshot of the output of iostat:

    Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0,00     0,00    0,00    7,33     0,00    43,33 5,91     0,33   43,18  33,32  24,43
    sdb               0,00     0,00    0,00    7,33     0,00    43,33 5,91     0,35   45,59  39,73  29,13
    md0               0,00     0,00    0,00    0,67     0,00     5,33 8,00     0,00    0,00   0,00   0,00
    md1               0,00     0,00    0,00    0,33     0,00     5,33 16,00     0,00    0,00   0,00   0,00
    md2               0,00     0,00    0,00    0,33     0,00     1,00 3,00     0,00    0,00   0,00   0,00
    md3               0,00     0,00    0,00    0,00     0,00     0,00 0,00     0,00    0,00   0,00   0,00
    md4               0,00     0,00    0,00    0,00     0,00     0,00 0,00     0,00    0,00   0,00   0,00
    md5               0,00     0,00    0,00    0,33     0,00     0,67 2,00     0,00    0,00   0,00   0,00

    I checked with mdadm if a resync happens or so, but this is not the case. The state says "active" on all RAID devices - btw. what is the difference to "clean" ?

    thanks for any hints,
     harald
    --
    To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] <mailto:[email protected]>
    More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 3 lines

--
Roberto Spadim
Spadim Technology / SPAEmpresarial

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 6 lines
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw
Quoted 58 lines · On Thu, 18 Aug 2011 17:44:19 +0400 CoolCold <[email protected]> wrote:
Quoted 20 lines · NeilBrown
Quoted 10 lines · On Thu, 18 Aug 2011 02:26:17 +0200 Harald Nikolisin <[email protected]> wrote:

hi,

I didn't want to complain in general about SW RAID-1 performance. I simply think something is wrong with my setup and I have currently no idea how to improve.

The basic questions (where I did not find an answer, neither in FAQ's nor in forum discussions) are.
a) Is it normal that the hard drives show an permanent utilization (around 20%) without any noticeable actions on the computer?

No.  If the array is resyncing or recovering then you would expect utilization for as many hours as it takes - but that would show in /proc/mdstat.

Quoted 1 line

b) Should (as long as no resync happens) the state of mdadm active or clean?

Quoted 7 lines

If anything has been written to the device in the last 200msec (including e.g. access time updates) then expect it to be 'active'.
If nothing has been written for 200msecc or more, then expect it to be clean.

If you crash while it is active, a resync is needed. If you crash while it is clean, no resync is needed. If you don't crash at all .... that is best :-)

I think this info should be wikified if not yet.

btw, I've experimented a bit on my /boot array (it doesn't being updated, checked with iostat ), and:

root@m2:~# for i in {1..5};do mdname="md0"; echo "iteration $i";
(mdadm --detail /dev/$mdname|grep 'State ';cat
/sys/block/$mdname/md/array_state;grep "$mdname :" /proc/mdstat);sleep
1;done
iteration 1
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 2
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 3
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 4
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
iteration 5
          State : clean
clean
md0 : active raid1 sda1[0] sdb1[1]
Quoted 4 lines

so, mdadm --detail & array_state shows array is "clean", while /proc/mdstat shows array is "active" (no reads/writes happen).

Some value is lieing or being misunderdstanded by me...

In mdstat you have 'active' or 'inactive'. You cannot access an array at all until it is active. If you are assembling an array bit by bit with "mdadm -I", it will be inactive until all the devices appear. Then it will be active.

In mdadm "State :" you have 'active' or 'clean'. as described above. It used to be 'dirty' or 'clean' but people were confused by having 'dirty' arrays in normal operation. So I changed it to 'active' and now it confuses a different set of people. You just can't win can you :-)

NeilBrown

Quoted 69 lines
Quoted 69 lines
Quoted 68 lines

cheers,
  harald

well, I have only 2 hard drives and no space for more..

Am 16.08.2011 03:29, schrieb Roberto Spadim:

Quoted 48 lines

try raid10 far layout

2011/8/15 Harald Nikolisin <[email protected]
<mailto:[email protected]>>

    Since a long time I'm unhappy with the performance of my RAID-1 system. Investigation with atop and iostat unveils that the disk utilization is always on a certain level although nothing happens on the system. In the case of reading or writing files the utilization boosts always to 100% for a long time. Very ugly examples are "Firefox starting" or "zypper updates".
    That is snapshot of the output of iostat:

    Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0,00     0,00    0,00    7,33     0,00    43,33 5,91     0,33   43,18  33,32  24,43
    sdb               0,00     0,00    0,00    7,33     0,00    43,33 5,91     0,35   45,59  39,73  29,13
    md0               0,00     0,00    0,00    0,67     0,00     5,33 8,00     0,00    0,00   0,00   0,00
    md1               0,00     0,00    0,00    0,33     0,00     5,33 16,00     0,00    0,00   0,00   0,00
    md2               0,00     0,00    0,00    0,33     0,00     1,00 3,00     0,00    0,00   0,00   0,00
    md3               0,00     0,00    0,00    0,00     0,00     0,00 0,00     0,00    0,00   0,00   0,00
    md4               0,00     0,00    0,00    0,00     0,00     0,00 0,00     0,00    0,00   0,00   0,00
    md5               0,00     0,00    0,00    0,33     0,00     0,67 2,00     0,00    0,00   0,00   0,00

    I checked with mdadm if a resync happens or so, but this is not the case. The state says "active" on all RAID devices - btw. what is the difference to "clean" ?

    thanks for any hints,
     harald
    --
    To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] <mailto:[email protected]>
    More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 3 lines

--
Roberto Spadim
Spadim Technology / SPAEmpresarial

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 8 lines
Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
Quoted 6 lines · NeilBrown

..
In mdstat you have 'active' or 'inactive'. You cannot access an array at all until it is active. If you are assembling an array bit by bit with "mdadm -I", it will be inactive until all the devices appear. Then it will be active.

Quoted 6 lines

In mdadm "State :" you have 'active' or 'clean'. as described above. It used to be 'dirty' or 'clean' but people were confused by having 'dirty' arrays in normal operation. So I changed it to 'active' and now it confuses a different set of people. You just can't win can you :-)

NeilBrown

mdstat:
"Enabled" or "Disabled" perhaps?

That matches what most commercial hardware RAID interfaces use.

Signature · 3 lines
Cheers,
Maurice Hilarius
eMail: /[email protected]/
#6 lore raw
Quoted 15 lines · maurice
Quoted 14 lines · NeilBrown

..
In mdstat you have 'active' or 'inactive'. You cannot access an array at all
until it is active. If you are assembling an array bit by bit with "mdadm -I", it will be inactive until all the devices appear. Then it will be active.

In mdadm "State :" you have 'active' or 'clean'. as described above. It used
to be 'dirty' or 'clean' but people were confused by having 'dirty' arrays in
normal operation. So I changed it to 'active' and now it confuses a different set of people. You just can't win can you :-)

Quoted 6 lines
Quoted 1 line

NeilBrown

mdstat:
"Enabled" or "Disabled" perhaps?

That matches what most commercial hardware RAID interfaces use.

Does it? It sounds more like an administrative action than a current status. I would have thought "online" or "offline" - unless that means something else somewhere else.

And for mdadm state: how about "busy" and "idle"? Hmm maybe just "busy" instead of "active" or "dirty"; we don't want to start an array with --assume-idle...

Cheers,

John.