Mdadm monitor sometimes doesn't send events.

DISCUSSION Arkadiusz Bubała 2 participants 4 messages Apr 29, 2013 – May 2, 2013 MD RAID

lore

Share & tools

lore

Arkadiusz Bubała

author

2013-04-29 08:52 UTC #0 lore raw

Hello,
We've got a problem that sometimes mdadm monitor doesn't send events. Actions are as follows:

start:
create new RAID 5 array
wait until it's rebuilding
stop created array
go to start

Only first time we get "NewArray" event then when new array is created only "RebuildStarted" event appears. After our analysis we found out that when we stop an array we get "DeviceDisapeared" event and when we create a new array we get "RebuildFinished" and then "RebuildStarted" event. Sometimes those two events don't show up. The problem is in mdstat_read function:

else if (ent->percent == -1 && w[0] >= '0' && w[0] <= '9' && w[l-1] == '%') {
    ent->percent = atoi(w);
}

atoi(w) sometimes returns 0 instead -1. It causes that mse->percent = 0 in:

            if (mse &&
                mse->percent == -1 &&
                st->percent >= 0)
                alert("RebuildFinished", dev, NULL, mailaddr, mailfrom,

alert_cmd, dosyslog);

and above conditions aren't met. Then in:

            if (mse)
                st->percent = mse->percent;

st->percent becomes 0 and next conditions where "RebuildStarted" event should be sent also aren't met:

            if (mse &&
                st->percent == -1 &&
                mse->percent >= 0)
                alert("RebuildStarted", dev, NULL, mailaddr, mailfrom,

alert_cmd, dosyslog);

Probably it is fine because rebuild already started and this is what monitor reads from /proc/mdstat. Shouldn't it return "NewArray" event instead those two "RebuildFinished" and "RebuildStarted" in this situation?

We use mdadm version 2.5.6 and monitor runs as a daemon with 10s interval and scan option.

Do we need to restart mdadm monitor when new disk is added to the system (hotplug)? Do we need to generate config file in such situation or running mdadm with option --scan is sufficient?

Signature · 9 lines

Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

NeilBrown

2013-04-30 01:48 UTC #1 lore raw

Quoted 10 lines · On Mon, 29 Apr 2013 10:52:57 +0200 Arkadiusz Bubała…

Hello,
We've got a problem that sometimes mdadm monitor doesn't send events. Actions are as follows:
start:
create new RAID 5 array
wait until it's rebuilding
stop created array
go to start

Quoted 5 lines

Only first time we get "NewArray" event then when new array is created only "RebuildStarted" event appears. After our analysis we found out that when we stop an array we get "DeviceDisapeared" event and when we create a new array we get "RebuildFinished" and then "RebuildStarted" event. Sometimes those two events don't show up.

The missing NewArray messages was fixed in June last year by commit 0f760384ebc92d81a988c1501e4a493ef064ad66

You are using a very old mdadm. Maybe you should try 3.2.6

I'm less sure about the RebuildStarted/RebuildFinished oddities but there is a good chance they are fixed is the latest release too.

NeilBrown

Quoted 39 lines

The problem is in mdstat_read function:
else if (ent->percent == -1 && w[0] >= '0' && w[0] <= '9' && w[l-1] ==
'%') {
     ent->percent = atoi(w);
}
atoi(w) sometimes returns 0 instead -1. It causes that mse->percent = 0 in:
             if (mse &&
                 mse->percent == -1 &&
                 st->percent >= 0)
                 alert("RebuildFinished", dev, NULL, mailaddr, mailfrom, 
alert_cmd, dosyslog);
and above conditions aren't met. Then in:
             if (mse)
                 st->percent = mse->percent;
st->percent becomes 0 and next conditions where "RebuildStarted" event should be sent also aren't met:
             if (mse &&
                 st->percent == -1 &&
                 mse->percent >= 0)
                 alert("RebuildStarted", dev, NULL, mailaddr, mailfrom, 
alert_cmd, dosyslog);
Probably it is fine because rebuild already started and this is what monitor reads from /proc/mdstat. Shouldn't it return "NewArray" event instead those two "RebuildFinished" and "RebuildStarted" in this situation?
We use mdadm version 2.5.6 and monitor runs as a daemon with 10s interval and scan option.
Do we need to restart mdadm monitor when new disk is added to the system (hotplug)? Do we need to generate config file in such situation or running mdadm with option --scan is sufficient?

Arkadiusz Bubała

author

2013-05-02 08:34 UTC #2 lore raw

Quoted 23 lines · NeilBrown

Quoted 10 lines · On Mon, 29 Apr 2013 10:52:57 +0200 Arkadiusz Bubała…
Hello,
We've got a problem that sometimes mdadm monitor doesn't send events. Actions are as follows:
start:
create new RAID 5 array
wait until it's rebuilding
stop created array
go to start
Quoted 6 lines
Only first time we get "NewArray" event then when new array is created only "RebuildStarted" event appears. After our analysis we found out that when we stop an array we get "DeviceDisapeared" event and when we create a new array we get "RebuildFinished" and then "RebuildStarted" event. Sometimes those two events don't show up.
The missing NewArray messages was fixed in June last year by commit 0f760384ebc92d81a988c1501e4a493ef064ad66

Quoted 8 lines

You are using a very old mdadm. Maybe you should try 3.2.6
I'm less sure about the RebuildStarted/RebuildFinished oddities but there is a good chance they are fixed is the latest release too.
NeilBrown

Thank you,
I'm testing this on 3.2.6 and it seems to work well. Events don't dissapear. We thought that lack of "NewArray" event may be intended.

Could you answer these questions, please:

Do we need to restart mdadm monitor when new disk is added to the system (hotplug)?

Do we need to generate config file in such situation or running mdadm with option --scan is sufficient?

Signature · 9 lines

Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

NeilBrown

2013-05-02 10:30 UTC #3 lore raw

Quoted 32 lines · On Thu, 02 May 2013 10:34:58 +0200 Arkadiusz Bubała…

Quoted 23 lines · NeilBrown
Quoted 10 lines · On Mon, 29 Apr 2013 10:52:57 +0200 Arkadiusz Bubała…
Hello,
We've got a problem that sometimes mdadm monitor doesn't send events. Actions are as follows:
start:
create new RAID 5 array
wait until it's rebuilding
stop created array
go to start
Quoted 6 lines
Only first time we get "NewArray" event then when new array is created only "RebuildStarted" event appears. After our analysis we found out that when we stop an array we get "DeviceDisapeared" event and when we create a new array we get "RebuildFinished" and then "RebuildStarted" event. Sometimes those two events don't show up.
The missing NewArray messages was fixed in June last year by commit 0f760384ebc92d81a988c1501e4a493ef064ad66
Quoted 8 lines
You are using a very old mdadm. Maybe you should try 3.2.6
I'm less sure about the RebuildStarted/RebuildFinished oddities but there is a good chance they are fixed is the latest release too.
NeilBrown

Quoted 8 lines

Thank you,
I'm testing this on 3.2.6 and it seems to work well. Events don't dissapear. We thought that lack of "NewArray" event may be intended.
Could you answer these questions, please:
Do we need to restart mdadm monitor when new disk is added to the system (hotplug)?

No.

Quoted 4 lines

Do we need to generate config file in such situation or running mdadm with option --scan is sufficient?

--scan should be sufficient.

NeilBrown