Skip to contentSkip to conversation

More ddf container woes

DISCUSSION Albert Pauw 2 participants 7 messages Mar 23, 2011 – Aug 14, 2012 MD RAID

lore
Share & tools
lore

Albert Pauw

author
#0 lore raw

Hi Neil,

I noticed on your 3.1.5 announcement that there were ddf fixes as well.

I tried the stuff I mentioned before (see below), but those issues weren't fixed.

I hope you will have some time to look into this.

Regards,

Albert Pauw

-------- Original Message --------

Hi Neil,

I updated to the git version (devel) and tried my "old" tricks:

- Create a container with 5 disks
- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online

- Failed one disk in md0

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Failed one disk in md1

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Added a new spare disk to the container

mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding

this looks good, but although the container has six disks, the lastly failed disk is missing, mdadm -E /dev/md127 only shows five disks (including the rebuilding one).

This time however, only one of the failed raid sets is rebuilding, so that fix is ok.

Here is another scenario with strange implications:

- Created a container with 6 disks

mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online

- Removed one of the disks, as I only needed 5

This time mdadm -e /dev/md127 shows six physical disks, one of which has no device

- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online, except the "empty entry" which stays
Global-Spare/Online

- I fail two disks, one in each raid array

mdadm -E /dev/md127 shows these two disks as active/Offline, Failed

- I add back the disk I removed earlier, it should fit into the empty slot of mdadm -E

mdadm -E /dev/md127 shows something very strange, namely
-> All disks are set to Global-Spare/Online
-> All device files are removed from the slots in mdadm -E, except the newly added one,
which shows the correct device

Albert

#1 lore raw
Quoted 3 lines · On Wed, 23 Mar 2011 20:18:24 +0100 Albert Pauw <[email protected]> wrote:

Hi Neil,

I noticed on your 3.1.5 announcement that there were ddf fixes as well.

Yes, they are mainly the fixes that you had helped me find and so already knew about.

Quoted 5 lines

I tried the stuff I mentioned before (see below), but those issues weren't fixed.

I hope you will have some time to look into this.

Hopefully, but probably not for a few weeks - I need to take a break from mdadm and catch up with some other stuff.

I'll keep this message to refer back to when I do get back to mdadm.

Thanks for your help,

NeilBrown

Quoted 73 lines

Regards,

Albert Pauw

-------- Original Message --------

Hi Neil,

I updated to the git version (devel) and tried my "old" tricks:

- Create a container with 5 disks
- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online

- Failed one disk in md0

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Failed one disk in md1

mdadm -E /dev/md127 shows this disk as active/Offline, Failed

- Added a new spare disk to the container

mdadm -E /dev/md127 shows this new disk as active/Online, Rebuilding

this looks good, but although the container has six disks, the lastly failed disk is missing, mdadm -E /dev/md127 only shows five disks (including the rebuilding one).

This time however, only one of the failed raid sets is rebuilding, so that fix is ok.

Here is another scenario with strange implications:

- Created a container with 6 disks

mdadm -E /dev/md127 shows all 6 disks as Global-Spare/Online

- Removed one of the disks, as I only needed 5

This time mdadm -e /dev/md127 shows six physical disks, one of which has no device

- Created two raid sets (raid 1 md0 and raid 5 md1) in this container

mdadm -E /dev/md127 shows all disks active/Online, except the "empty entry" which stays
Global-Spare/Online

- I fail two disks, one in each raid array

mdadm -E /dev/md127 shows these two disks as active/Offline, Failed

- I add back the disk I removed earlier, it should fit into the empty slot of mdadm -E

mdadm -E /dev/md127 shows something very strange, namely
-> All disks are set to Global-Spare/Online
-> All device files are removed from the slots in mdadm -E, except the newly added one,
which shows the correct device

Albert

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#2 lore raw

Version 3.2.5 and ddf issues (bugreport)

Hi Neil,

After a hiatus of 1.5 year (busy with all sorts) I am back and tried the ddf code to see how things improved.

I build a VM Centos 6.3 system with 6 extra 1GB disks for testing.
I found several issues in the standard installed 3.2.3 version of mdadm relating to ddf, but installed the 3.2.5 version in order to work with recent code.

However, while version 3.2.3 is able to create a ddf container with raidsets in it, I found a problem with the 3.2.5 version.

After initially creating the container:

mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]

which worked, I created a raid (1 or 5 it doesn't matter in this case) in it:

mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127

However, it stays on resync=PENDING and readonly, and doesn't get build.

So I tried to set it to readwrite:

mdadm --readwrite /dev/md0

Unfortunately, it stays on readonly and doesn't get build.

As said before, this did work in 3.2.3.

Are you already on this problem?

Kind regards,

Albert

P.S. My idea was, because the ddf code is not that much used, to create a test/validate script for ddf.

#3 lore raw

Re: Version 3.2.5 and ddf issues (bugreport)

Quoted 4 lines · On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <[email protected]> wrote:

Hi Neil,

After a hiatus of 1.5 year (busy with all sorts) I am back and tried the ddf code to see how things improved.

Thanks!

Quoted 22 lines

I build a VM Centos 6.3 system with 6 extra 1GB disks for testing. I found several issues in the standard installed 3.2.3 version of mdadm relating to ddf, but installed the
3.2.5 version in order to work with recent code.

However, while version 3.2.3 is able to create a ddf container with raidsets in it, I found a problem with the 3.2.5 version.

After initially creating the container:

mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]

which worked, I created a raid (1 or 5 it doesn't matter in this case) in it:

mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127

However, it stays on resync=PENDING and readonly, and doesn't get build.

So I tried to set it to readwrite:

Quoted 7 lines

mdadm --readwrite /dev/md0

Unfortunately, it stays on readonly and doesn't get build.

As said before, this did work in 3.2.3.

Are you already on this problem?

It sounds like a problem with 'mdmon'. mdmon needs to be running before the array can become read-write. mdadm should start mdmon automatically but maybe it isn't. Maybe it cannot find mdmon?

could you check if mdadm is running? If it isn't run
mdmon /dev/md127 &
and see if it starts working.

Quoted 8 lines

Kind regards,

Albert

P.S. My idea was, because the ddf code is not that much used, to create a test/validate script for ddf.

There is one 'ddf' test in my test suite and I think it is working. I'm happy to add more tests if you suggest or write some.

NeilBrown

author
#4 lore raw

Re: Version 3.2.5 and ddf issues (bugreport)

Quoted 35 lines · NeilBrown
Quoted 4 lines · On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <[email protected]> wrote:

Hi Neil,

After a hiatus of 1.5 year (busy with all sorts) I am back and tried the ddf code to see how things improved.

Thanks!

Quoted 27 lines

I build a VM Centos 6.3 system with 6 extra 1GB disks for testing. I found several issues in the standard installed 3.2.3 version of mdadm relating to ddf, but installed the
3.2.5 version in order to work with recent code.

However, while version 3.2.3 is able to create a ddf container with raidsets in it, I found a problem with the 3.2.5 version.

After initially creating the container:

mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]

which worked, I created a raid (1 or 5 it doesn't matter in this case) in it:

mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127

However, it stays on resync=PENDING and readonly, and doesn't get build.

So I tried to set it to readwrite:

mdadm --readwrite /dev/md0

Unfortunately, it stays on readonly and doesn't get build.

As said before, this did work in 3.2.3.

Quoted 8 lines
Quoted 1 line

Are you already on this problem?

It sounds like a problem with 'mdmon'. mdmon needs to be running before the array can become read-write. mdadm should start mdmon automatically but maybe it isn't. Maybe it cannot find mdmon?

could you check if mdadm is running? If it isn't run

    mdmon /dev/md127 &

and see if it starts working.

Hi Neil,

thanks for your reply. Yes, mdmon wasn't running. Couldn't get it running with a recompiled 3.2.5, the standard one which came with Centos (3.2.3) works fine, I assume the made some changes to the code? Anyway, I moved to my own laptop, running Fedora 16 and pulled mdadm frm git and recompiled. That works. I also used loop devices as disks.

Here is the first of my findings:

I created a container with six disks, disk 1-2 is a raid 1 device, disk 3-6 are a raid 6 device.

Here is the table shown at the end of the mdadm -E command for the container:

Physical Disks : 6

      Number    RefNo      Size       Device      Type/State
         0    06a5f547    479232K /dev/loop2      active/Online
         1    47564acc    479232K /dev/loop3      active/Online
         2    bf30692c    479232K /dev/loop5      active/Online
         3    275d02f5    479232K /dev/loop4      active/Online
         4    b0916b3f    479232K /dev/loop6      active/Online
         5    65956a72    479232K /dev/loop1      active/Online

I now fail a disk (disk 0) and I get:

Physical Disks : 6

      Number    RefNo      Size       Device      Type/State
         0    06a5f547    479232K /dev/loop2      active/Online
         1    47564acc    479232K /dev/loop3      active/Online
         2    bf30692c    479232K /dev/loop5      active/Online
         3    275d02f5    479232K /dev/loop4      active/Online
         4    b0916b3f    479232K /dev/loop6      active/Online
         5    65956a72    479232K /dev/loop1      active/Offline, Failed

Then I removed the disk from the container:

Physical Disks : 6

      Number    RefNo      Size       Device      Type/State
         0    06a5f547    479232K /dev/loop2      active/Online
         1    47564acc    479232K /dev/loop3      active/Online
         2    bf30692c    479232K /dev/loop5      active/Online
         3    275d02f5    479232K /dev/loop4      active/Online
         4    b0916b3f    479232K /dev/loop6      active/Online
         5    65956a72    479232K                 active/Offline,

Failed, Missing

Notice the active/Offline status, is this correct?

I added the disk back into the container, NO zero-superblock:

Physical Disks : 6

      Number    RefNo      Size       Device      Type/State
         0    06a5f547    479232K /dev/loop2      active/Online
         1    47564acc    479232K /dev/loop3      active/Online
         2    bf30692c    479232K /dev/loop5      active/Online
         3    275d02f5    479232K /dev/loop4      active/Online
         4    b0916b3f    479232K /dev/loop6      active/Online
         5    65956a72    479232K /dev/loop1      active/Offline,

Failed, Missing

It stays active/Offline (this is now correct I assume), Failed (again correct if had failed before), but also still missing.

I remove the disk again, do a zero-superblock and add it again:

Physical Disks : 6

      Number    RefNo      Size       Device      Type/State
         0    06a5f547    479232K /dev/loop2      active/Online
         1    47564acc    479232K /dev/loop3      active/Online
         2    bf30692c    479232K /dev/loop5      active/Online
         3    275d02f5    479232K /dev/loop4      active/Online
         4    b0916b3f    479232K /dev/loop6      active/Online
         5    ede51ba3    479232K /dev/loop1      active/Online, Rebuilding

This is correct, the disk is seen as a new disk and rebuilding starts.

Regards,

Albert

#5 lore raw

Re: Version 3.2.5 and ddf issues (bugreport)

Quoted 120 lines · On Tue, 31 Jul 2012 10:46:26 +0200 Albert Pauw <[email protected]> wrote:
Quoted 35 lines · NeilBrown
Quoted 4 lines · On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <[email protected]> wrote:

Hi Neil,

After a hiatus of 1.5 year (busy with all sorts) I am back and tried the ddf code to see how things improved.

Thanks!

Quoted 27 lines

I build a VM Centos 6.3 system with 6 extra 1GB disks for testing. I found several issues in the standard installed 3.2.3 version of mdadm relating to ddf, but installed the
3.2.5 version in order to work with recent code.

However, while version 3.2.3 is able to create a ddf container with raidsets in it, I found a problem with the 3.2.5 version.

After initially creating the container:

mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]

which worked, I created a raid (1 or 5 it doesn't matter in this case) in it:

mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127

However, it stays on resync=PENDING and readonly, and doesn't get build.

So I tried to set it to readwrite:

mdadm --readwrite /dev/md0

Unfortunately, it stays on readonly and doesn't get build.

As said before, this did work in 3.2.3.

Quoted 8 lines
Quoted 1 line

Are you already on this problem?

It sounds like a problem with 'mdmon'. mdmon needs to be running before the array can become read-write. mdadm should start mdmon automatically but maybe it isn't. Maybe it cannot find mdmon?

could you check if mdadm is running? If it isn't run

    mdmon /dev/md127 &

and see if it starts working.

Hi Neil,

thanks for your reply. Yes, mdmon wasn't running. Couldn't get it running with a recompiled 3.2.5, the standard one which came with Centos (3.2.3) works fine, I assume the made some changes to the code? Anyway, I moved to my own laptop, running Fedora 16 and pulled mdadm frm git and recompiled. That works. I also used loop devices as disks.

Here is the first of my findings:

I created a container with six disks, disk 1-2 is a raid 1 device, disk 3-6 are a raid 6 device.

Here is the table shown at the end of the mdadm -E command for the container:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Online

I now fail a disk (disk 0) and I get:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Offline, Failed

Then I removed the disk from the container:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K                 active/Offline,

Failed, Missing

Notice the active/Offline status, is this correct?

I added the disk back into the container, NO zero-superblock:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Offline,

Failed, Missing

It stays active/Offline (this is now correct I assume), Failed (again correct if had failed before), but also still missing.

I remove the disk again, do a zero-superblock and add it again:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    ede51ba3    479232K /dev/loop1      active/Online, Rebuilding
Quoted 6 lines

This is correct, the disk is seen as a new disk and rebuilding starts.

Regards,

Albert

Hi Albert,

thanks for this and your other reports.
I won't be able to look at them for a while, but hopefully will get back to you some time next week.

NeilBrown

#6 lore raw

Re: Version 3.2.5 and ddf issues (bugreport)

In reply to Albert Pauw
Quoted 90 lines · On Tue, 31 Jul 2012 10:46:26 +0200 Albert Pauw <[email protected]> wrote:
Quoted 35 lines · NeilBrown
Quoted 4 lines · On Sat, 28 Jul 2012 13:46:06 +0200 Albert Pauw <[email protected]> wrote:

Hi Neil,

After a hiatus of 1.5 year (busy with all sorts) I am back and tried the ddf code to see how things improved.

Thanks!

Quoted 27 lines

I build a VM Centos 6.3 system with 6 extra 1GB disks for testing. I found several issues in the standard installed 3.2.3 version of mdadm relating to ddf, but installed the
3.2.5 version in order to work with recent code.

However, while version 3.2.3 is able to create a ddf container with raidsets in it, I found a problem with the 3.2.5 version.

After initially creating the container:

mdadm -C /dev/md127 -e ddf -l container /dev/sd[b-g]

which worked, I created a raid (1 or 5 it doesn't matter in this case) in it:

mdadm -C /dev/md0 -l raid5 -n 3 /dev/md127

However, it stays on resync=PENDING and readonly, and doesn't get build.

So I tried to set it to readwrite:

mdadm --readwrite /dev/md0

Unfortunately, it stays on readonly and doesn't get build.

As said before, this did work in 3.2.3.

Quoted 8 lines
Quoted 1 line

Are you already on this problem?

It sounds like a problem with 'mdmon'. mdmon needs to be running before the array can become read-write. mdadm should start mdmon automatically but maybe it isn't. Maybe it cannot find mdmon?

could you check if mdadm is running? If it isn't run

    mdmon /dev/md127 &

and see if it starts working.

Hi Neil,

thanks for your reply. Yes, mdmon wasn't running. Couldn't get it running with a recompiled 3.2.5, the standard one which came with Centos (3.2.3) works fine, I assume the made some changes to the code? Anyway, I moved to my own laptop, running Fedora 16 and pulled mdadm frm git and recompiled. That works. I also used loop devices as disks.

Here is the first of my findings:

I created a container with six disks, disk 1-2 is a raid 1 device, disk 3-6 are a raid 6 device.

Here is the table shown at the end of the mdadm -E command for the container:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Online

I now fail a disk (disk 0) and I get:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Offline, Failed

Then I removed the disk from the container:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K                 active/Offline,
Quoted 3 lines

Failed, Missing

Notice the active/Offline status, is this correct?

To be honest, I don't know. The DDF spec doesn't really go into that sort of detail, or at least I didn't find it.
Given that the device is Missing, it hardly seems to matter whether it is Active or Spare or Foreign or Legacy.
I guess if it re-appears we want to know what it was ... maybe.

Quoted 11 lines

I added the disk back into the container, NO zero-superblock:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    65956a72    479232K /dev/loop1      active/Offline,
Quoted 4 lines

Failed, Missing

It stays active/Offline (this is now correct I assume), Failed (again correct if had failed before), but also still missing.

I found why this happens. When I added code to support incremental assembly of DDF arrays, I broke the ability to hot-add a device which happened to have reasonably good looking metadata on it. The best approach for now is to --zero the device first. I'll push out a patch which does just that.

Quoted 12 lines

I remove the disk again, do a zero-superblock and add it again:

Physical Disks : 6

       Number    RefNo      Size       Device      Type/State
          0    06a5f547    479232K /dev/loop2      active/Online
          1    47564acc    479232K /dev/loop3      active/Online
          2    bf30692c    479232K /dev/loop5      active/Online
          3    275d02f5    479232K /dev/loop4      active/Online
          4    b0916b3f    479232K /dev/loop6      active/Online
          5    ede51ba3    479232K /dev/loop1      active/Online, Rebuilding
Quoted 6 lines

This is correct, the disk is seen as a new disk and rebuilding starts.

Regards,

Albert

Manage.c +1 -0
index f83af65..7f27f74 100644
@@ -786,6 +786,7 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
 			return -1;
 		}
 
+		Kill(dv->devname, NULL, 0, -1, 0);
 		dfd = dev_open(dv->devname, O_RDWR | O_EXCL|O_DIRECT);
 		if (mdmon_running(tst->container_dev))
 			tst->update_tail = &tst->updates;

Thanks,
NeilBrown