Skip to contentSkip to conversation

Partitioned arrays initially missing from /proc/partitions

DISCUSSION Mike Accetta 5 participants 15 messages Dec 1, 2006 – May 7, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

In setting up a partitioned array as the boot disk and using a nash initrd to find the root file system by volume label, I see a delay in the appearance of the /dev/md_d0p partitions in /proc/partitions. When the mdadm --assemble command completes, only /dev/md_d0 is visible. Since the raid partitions are not visible after the assemble, the volume label search will not consult them in looking for the root volume and the boot gets aborted. When I run a similar assemble command while up multi-user in a friendlier debug environment I see the same effect and observe that pretty much any access of /dev/md_d0 has the side effect of then making the /dev/md_d0p partitions visible in /proc/partitions.

I tried a few experiments changing the --assemble code in mdadm. If I open() and close() /dev/md_d0 after assembly *before* closing the file descriptor which the assemble step used to assemble the array, there is no effect. Even doing a BLKRRPART ioctl call on the assembly fd or the newly opened fd have no effect. The kernel prints "unknown partition" diagnostics on the console. However, if the assembly fd is first close()'d, a simple open() of /dev/md_d0 and immediate close() of that fd has the side effect of making the /dev/md_d0p partitions visible and one sees the console disk partitioning confirmation from the kernel as well.

Adding the open()/close() after assembly within mdadm solves my problem, but I thought I'd raise the issue on the list as it seems there is a bug somewhere. I see in the kernel md driver that the RUN_ARRAY ioctl() calls do_md_run() which calls md_probe() which calls add_disk() and I gather that this would normally have the side effect of making the partitions visible. However, my experiments at user level seem to imply that the array isn't completely usable until the assembly file descriptor is closed, even on return from the ioctl(), and hence the kernel add_disk() isn't having the desired partitioning side effect at the point it is being invoked.

This is all with kernel 2.6.18 and mdadm 2.3.1

Signature · 4 lines
Mike Accetta

ECI Telecom Ltd.
Data Networking Division (previously Laurel Networks)
#1 lore raw

Hi Neil

I think this is a bug.

Essentially if I create an auto=part md device then I get md_d0p? partitions. If I stop the array and just re-assemble, I don't.

It looks like the same (?) problem as Mike (see below - Mike do you have a patch?) but I'm on 2.6.20.7 with mdadm v2.5.6

FWIW I upgraded from 2.6.16 where it worked (but used in-kernel detection which isn't working in 2.6.20 for some reason but I don't mind).

Here's a simple sequence of commands:

teak:~# mdadm --stop /dev/md_d0
mdadm: stopped /dev/md_d0
teak:~# mdadm --create /dev/md_d0 -l5 -n5 --bitmap=internal -e1.2 --auto=part
--name media --force /dev/sde1 /dev/sdc1 /dev/sdd1 missing /dev/sdf1
mdadm: /dev/sde1 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
mdadm: /dev/sdc1 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
mdadm: /dev/sdd1 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
mdadm: /dev/sdf1 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
Continue creating array? y
mdadm: array /dev/md_d0 started.
teak:~# grep md /proc/partitions
 254     0 1250241792 md_d0
 254     1 1250144138 md_d0p1
 254     2      97652 md_d0p2
teak:~# mdadm --stop /dev/md_d0
mdadm: stopped /dev/md_d0
teak:~# mdadm --assemble /dev/md_d0 --auto=part  /dev/sde1 /dev/sdc1 /dev/sdd1
/dev/sdf1
mdadm: /dev/md_d0 has been started with 4 drives (out of 5).
teak:~# grep md /proc/partitions
 254     0 1250241792 md_d0

If I then run cfdisk it finds the partition table. I write this and get:

teak:~# cfdisk /dev/md_d0

Disk has been changed.

WARNING: If you have created or modified any
DOS 6.x partitions, please see the cfdisk manual
page for additional information.

teak:~# grep md /proc/partitions
 254     0 1250241792 md_d0
 254     1 1250144138 md_d0p1
 254     2      97652 md_d0p2

and the syslog:

Apr 23 15:13:13 localhost kernel: md: md_d0 stopped.
Apr 23 15:13:13 localhost kernel: md: unbind<sde1>
Apr 23 15:13:13 localhost kernel: md: export_rdev(sde1)
Apr 23 15:13:13 localhost kernel: md: unbind<sdf1>
Apr 23 15:13:13 localhost kernel: md: export_rdev(sdf1)
Apr 23 15:13:13 localhost kernel: md: unbind<sdd1>
Apr 23 15:13:13 localhost kernel: md: export_rdev(sdd1)
Apr 23 15:13:13 localhost kernel: md: unbind<sdc1>
Apr 23 15:13:13 localhost kernel: md: export_rdev(sdc1)
Apr 23 15:13:13 localhost mdadm: DeviceDisappeared event detected on md device
/dev/md_d0
Apr 23 15:13:36 localhost kernel: md: bind<sde1>
Apr 23 15:13:36 localhost kernel: md: bind<sdc1>
Apr 23 15:13:36 localhost kernel: md: bind<sdd1>
Apr 23 15:13:36 localhost kernel: md: bind<sdf1>
Apr 23 15:13:36 localhost kernel: raid5: device sdf1 operational as raid disk 4
Apr 23 15:13:36 localhost kernel: raid5: device sdd1 operational as raid disk 2
Apr 23 15:13:36 localhost kernel: raid5: device sdc1 operational as raid disk 1
Apr 23 15:13:36 localhost kernel: raid5: device sde1 operational as raid disk 0
Apr 23 15:13:36 localhost kernel: raid5: allocated 5236kB for md_d0
Apr 23 15:13:36 localhost kernel: raid5: raid level 5 set md_d0 active with 4
out of 5 devices, algorithm 2
Apr 23 15:13:36 localhost kernel: RAID5 conf printout:
Apr 23 15:13:36 localhost kernel:  --- rd:5 wd:4
Apr 23 15:13:36 localhost kernel:  disk 0, o:1, dev:sde1
Apr 23 15:13:36 localhost kernel:  disk 1, o:1, dev:sdc1
Apr 23 15:13:36 localhost kernel:  disk 2, o:1, dev:sdd1
Apr 23 15:13:36 localhost kernel:  disk 4, o:1, dev:sdf1
Apr 23 15:13:36 localhost kernel: md_d0: bitmap initialized from disk: read 1/1
pages, set 19078 bits, status: 0
Apr 23 15:13:36 localhost kernel: created bitmap (10 pages) for device md_d0
Apr 23 15:13:36 localhost kernel:  md_d0: p1 p2
Apr 23 15:13:54 localhost kernel: md: md_d0 stopped.
Apr 23 15:13:54 localhost kernel: md: unbind<sdf1>
Apr 23 15:13:54 localhost kernel: md: export_rdev(sdf1)
Apr 23 15:13:54 localhost kernel: md: unbind<sdd1>
Apr 23 15:13:54 localhost kernel: md: export_rdev(sdd1)
Apr 23 15:13:54 localhost kernel: md: unbind<sdc1>
Apr 23 15:13:54 localhost kernel: md: export_rdev(sdc1)
Apr 23 15:13:54 localhost kernel: md: unbind<sde1>
Apr 23 15:13:54 localhost kernel: md: export_rdev(sde1)
Apr 23 15:13:54 localhost mdadm: DeviceDisappeared event detected on md device
/dev/md_d0
Apr 23 15:14:04 localhost kernel: md: md_d0 stopped.
Apr 23 15:14:04 localhost kernel: md: bind<sdc1>
Apr 23 15:14:04 localhost kernel: md: bind<sdd1>
Apr 23 15:14:04 localhost kernel: md: bind<sdf1>
Apr 23 15:14:04 localhost kernel: md: bind<sde1>
Apr 23 15:14:04 localhost kernel: raid5: device sde1 operational as raid disk 0
Apr 23 15:14:04 localhost kernel: raid5: device sdf1 operational as raid disk 4
Apr 23 15:14:04 localhost kernel: raid5: device sdd1 operational as raid disk 2
Apr 23 15:14:04 localhost kernel: raid5: device sdc1 operational as raid disk 1
Apr 23 15:14:04 localhost kernel: raid5: allocated 5236kB for md_d0
Apr 23 15:14:04 localhost kernel: raid5: raid level 5 set md_d0 active with 4
out of 5 devices, algorithm 2
Apr 23 15:14:04 localhost kernel: RAID5 conf printout:
Apr 23 15:14:04 localhost kernel:  --- rd:5 wd:4
Apr 23 15:14:04 localhost kernel:  disk 0, o:1, dev:sde1
Apr 23 15:14:04 localhost kernel:  disk 1, o:1, dev:sdc1
Apr 23 15:14:04 localhost kernel:  disk 2, o:1, dev:sdd1
Apr 23 15:14:04 localhost kernel:  disk 4, o:1, dev:sdf1
Apr 23 15:14:04 localhost kernel: md_d0: bitmap initialized from disk: read 1/1
pages, set 0 bits, status: 0
Apr 23 15:14:04 localhost kernel: created bitmap (10 pages) for device md_d0
Apr 23 15:14:04 localhost kernel:  md_d0: unknown partition table

after cfdisk write:

Apr 23 15:33:00 localhost kernel:  md_d0: p1 p2
Quoted 34 lines · Back in dec 2006,Mike Accetta wrote:

In setting up a partitioned array as the boot disk and using a nash initrd to find the root file system by volume label, I see a delay in the appearance of the /dev/md_d0p partitions in /proc/partitions. When the mdadm --assemble command completes, only /dev/md_d0 is visible. Since the raid partitions are not visible after the assemble, the volume label search will not consult them in looking for the root volume and the boot gets aborted. When I run a similar assemble command while up multi-user in a friendlier debug environment I see the same effect and observe that pretty much any access of /dev/md_d0 has the side effect of then making the /dev/md_d0p partitions visible in /proc/partitions.

I tried a few experiments changing the --assemble code in mdadm. If I open() and close() /dev/md_d0 after assembly *before* closing the file descriptor which the assemble step used to assemble the array, there is no effect. Even doing a BLKRRPART ioctl call on the assembly fd or the newly opened fd have no effect. The kernel prints "unknown partition" diagnostics on the console. However, if the assembly fd is first close()'d, a simple open() of /dev/md_d0 and immediate close() of that fd has the side effect of making the /dev/md_d0p partitions visible and one sees the console disk partitioning confirmation from the kernel as well.

Adding the open()/close() after assembly within mdadm solves my problem, but I thought I'd raise the issue on the list as it seems there is a bug somewhere. I see in the kernel md driver that the RUN_ARRAY ioctl() calls do_md_run() which calls md_probe() which calls add_disk() and I gather that this would normally have the side effect of making the partitions visible. However, my experiments at user level seem to imply that the array isn't completely usable until the assembly file descriptor is closed, even on return from the ioctl(), and hence the kernel add_disk() isn't having the desired partitioning side effect at the point it is being invoked.

This is all with kernel 2.6.18 and mdadm 2.3.1

#2 lore raw

David Greaves writes:

...

Quoted 2 lines

It looks like the same (?) problem as Mike (see below - Mike do you have a patch?) but I'm on 2.6.20.7 with mdadm v2.5.6

...

We have since started assembling the array from the initrd using --homehost and --auto-update-homehost which takes a different path through the code, and in this path the kernel figures out there are partitions on the array before mdadm exists.

For the previous code path, we had been ruuning with the patch I described in my original post which I've included below. I'd guess that the bug is actually in the kernel code and I looked at it briefly but couldn't figure out how things all fit together well enough to come up with a patch there. The user level patch is a bit of a hack and there may be other code paths that also need a similar patch. I only made this patch in the assembly code path we were executing at the time.

==== BUILD/mdadm/mdadm.c#2 (text) - BUILD/mdadm/mdadm.c#3 (text) ==== content @@ -983,6 +983,10 @@

                                                               NULL,
                                                               readonly, runstop, NULL, verbose-quiet, force);
                                        close(mdfd);

+ mdfd = open(array_list->devname, O_RDONLY);
+ if (mdfd >= 0) {
+ close(mdfd);
+ }

                                }
                }
                break;

--
Mike Accetta

ECI Telecom Ltd.
Data Networking Division (previously Laurel Networks)

#3 lore raw

This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.

I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.

Note that you can still access the partitions even though they do not appear in /proc/partitions. Any attempt to access and of them will make them all appear in /proc/partitions. But I understand there is sometimes value in seeing them before accessing them.

NeilBrown

#4 lore raw
Quoted 9 lines · Neil Brown

This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.

I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.

Quoted 6 lines

Note that you can still access the partitions even though they do not appear in /proc/partitions. Any attempt to access and of them will make them all appear in /proc/partitions. But I understand there is sometimes value in seeing them before accessing them.

NeilBrown

Um. Are you sure?
The reason I noticed is that I couldn't mount them until they appeared; see these cut'n'pastes from my terminal history:

teak:~# mount /media/
mount: /dev/md_d0p1 is not a valid block device
teak:~# mount /dev/md_d0p1 /media
mount: you must specify the filesystem type
teak:~# xfs_repair -ln /dev/md_d0p2 /dev/md_d0p1
Usage: xfs_repair [-nLvV] [-o subopt[=value]] [-l logdev] [-r rtdev] devname
teak:~# ll /dev/md*
brw-rw---- 1 root disk 254, 0 2007-04-23 15:44 /dev/md_d0
brw-rw---- 1 root disk 254, 1 2007-04-23 14:46 /dev/md_d0p1
brw-rw---- 1 root disk 254, 2 2007-04-23 14:46 /dev/md_d0p2
brw-rw---- 1 root disk 254, 3 2007-04-23 15:44 /dev/md_d0p3
brw-rw---- 1 root disk 254, 4 2007-04-23 15:44 /dev/md_d0p4

/dev/md:
total 0

teak:~# /etc/init.d/mdadm-raid stop
Stopping MD array md_d0...done (stopped).
teak:~# /etc/init.d/mdadm-raid start
Assembling MD array md_d0...done (degraded [4/5]).
Generating udev events for MD arrays...done.
teak:~# cfdisk /dev/md_d0
teak:~# mount /dev/md_d0p1
mount: /dev/md_d0p1 is not a valid block device

and so on...

Notice the cfdisk command above. I did this to check the on-array table (it was good). I assume cfdisk opens the array - but the partitions were still not there afterwards. I did not do a 'Write' from in cfdisk this time.

I wouldn't be so concerned at a cosmetic thing in /proc/partitions - the problem is that I can't mount my array after doing an assemble and I have to --create each time - not the nicest solution.

Oh, I'm using udev FWIW.

David

#13 lore raw
In reply to Mike Accetta
Quoted 11 lines · Mike Accetta

David Greaves writes:

...

Quoted 2 lines

It looks like the same (?) problem as Mike (see below - Mike do you have a patch?) but I'm on 2.6.20.7 with mdadm v2.5.6

...

We have since started assembling the array from the initrd using --homehost and --auto-update-homehost which takes a different path through the code, and in this path the kernel figures out there are partitions on the array before mdadm exists.

Just tried that - doesn't work :)

Quoted 10 lines

For the previous code path, we had been ruuning with the patch I described in my original post which I've included below. I'd guess that the bug is actually in the kernel code and I looked at it briefly but couldn't figure out how things all fit together well enough to come up with a patch there. The user level patch is a bit of a hack and there may be other code paths that also need a similar patch. I only made this patch in the assembly code path we were executing at the time.

==== BUILD/mdadm/mdadm.c#2 (text) - BUILD/mdadm/mdadm.c#3 (text) ==== content

Review line 992

Quoted 10 lines
(quoted diff) +4 -0
@@ -983,6 +983,10 @@
                                                               NULL,
                                                               readonly, runstop, NULL, verbose-quiet, force);
                                        close(mdfd);
+                                       mdfd = open(array_list->devname, O_RDONLY); 
+                                       if (mdfd >= 0) {
+                                           close(mdfd);
+                                       }
                                }
                }
                break;

Thanks Mike

But this doesn't work for me either :(

I changed array_list to devlist inline with 2.6.9 and it compiles and runs OK.

teak:~# mdadm --stop /dev/md_d0
mdadm: stopped /dev/md_d0
teak:~# /everything/devel/mdadm/mdadm-2.5.6/mdadm --assemble /dev/md_d0
/dev/sd[bcdef]1
mdadm: With Fudge.
mdadm: /dev/md_d0 has been started with 5 drives.
mdadm: Fudging partition creation.
teak:~# mount /media
mount: /dev/md_d0p1 is not a valid block device
teak:~#

I also wrote a small c program to call the RAID_AUTORUN ioctl - that didn't work either because I'd compiled RAID as a module so the ioctl isn't defined.

currently recompiling the kernel to allow autorun...

David

#14 lore raw
Quoted 1 line · David Greaves

currently recompiling the kernel to allow autorun...

Which of course won't work because I'm on 1.2 superblocks:
md: Autodetecting RAID arrays.
md: invalid raid superblock magic on sdb1
md: sdb1 has invalid sb, not importing!
md: invalid raid superblock magic on sdc1
md: sdc1 has invalid sb, not importing!
md: invalid raid superblock magic on sdd1
md: sdd1 has invalid sb, not importing!
md: invalid raid superblock magic on sde1
md: sde1 has invalid sb, not importing!
md: invalid raid superblock magic on sdf1
md: sdf1 has invalid sb, not importing!
md: autorun ...
md: ... autorun DONE.

David

PS Dropped Mike from cc since I doubt he's too interested :)

#7 lore raw
In reply to Neil Brown
Quoted 9 lines · Neil Brown

This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.

I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.

Quoted 6 lines

Note that you can still access the partitions even though they do not appear in /proc/partitions. Any attempt to access and of them will make them all appear in /proc/partitions. But I understand there is sometimes value in seeing them before accessing them.

NeilBrown

For anyone else who is in this boat and doesn't fancy finding somewhere in mdadm
to hack, here's a simple program that issues the BLKRRPART ioctl.
This re-reads the block device partition table and 'works for me'.

I think partx -a would do the same job but for some reason partx isn't in utils-linux for Debian...

Neil, isn't it easy to just do this after an assemble?

David

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <stdlib.h>
#include <fcntl.h>
#include <stdio.h>

#include <linux/raid/md_u.h>
#include <linux/major.h>
#include </usr/include/linux/fs.h>

int main(int argc, char *argv[])

{
    int fd;

	if (argc != 2)
       fprintf(stderr, "Usage: %s <md device>\n", argv[0]);

	
    if ((fd = open(argv[1], O_RDONLY)) == -1) {
       fprintf(stderr, "Can't open md device %s\n", argv[1]);
       return -1;
    }

	if (ioctl(fd,  BLKRRPART, NULL) != 0) {
       fprintf(stderr, "ioctl failed\n");
        close (fd);
        return -1;
    }

    close (fd);

    return 0;
}
#5 lore raw
In reply to David Greaves
Quoted 10 lines · On Tuesday April 24, [email protected] wrote:
Quoted 9 lines · Neil Brown

This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.

I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.

Quoted 8 lines
Quoted 6 lines

Note that you can still access the partitions even though they do not appear in /proc/partitions. Any attempt to access and of them will make them all appear in /proc/partitions. But I understand there is sometimes value in seeing them before accessing them.

NeilBrown

Um. Are you sure?

"Works for me".

What happens if you
blockdev --rereadpt /dev/md_d0
?? It probably works then.
It sounds like someone is deliberately removing all the partition info.

Can you try this patch and see if it reports anyone calling '2' on md_d0 ??

block/ioctl.c +3 -0
diff .prev/block/ioctl.c ./block/ioctl.c
@@ -17,6 +17,7 @@ static int blkpg_ioctl(struct block_devi
 	long long start, length;
 	int part;
 	int i;
+	char b[BDEVNAME_SIZE];
 
 	if (!capable(CAP_SYS_ADMIN))
 		return -EACCES;
@@ -30,6 +31,8 @@ static int blkpg_ioctl(struct block_devi
 	part = p.pno;
 	if (part <= 0 || part >= disk->minors)
 		return -EINVAL;
+	printk("blkpg_ioctl: %s called %d on %s\n",
+	       current->comm, a.op, bdevname(bdev, b));
 	switch (a.op) {
 		case BLKPG_ADD_PARTITION:
 			start = p.start >> 9;
#8 lore raw
In reply to David Greaves
Quoted 15 lines · On Tuesday April 24, [email protected] wrote:
Quoted 14 lines · Neil Brown

This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.

I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.

Note that you can still access the partitions even though they do not appear in /proc/partitions. Any attempt to access and of them will make them all appear in /proc/partitions. But I understand there is sometimes value in seeing them before accessing them.

Quoted 5 lines
Quoted 1 line

NeilBrown

For anyone else who is in this boat and doesn't fancy finding somewhere in mdadm
to hack, here's a simple program that issues the BLKRRPART ioctl.
This re-reads the block device partition table and 'works for me'.

blockdev --rereadpt /dev/md_d0
does the same thing.

Quoted 5 lines

I think partx -a would do the same job but for some reason partx isn't in utils-linux for Debian...

Neil, isn't it easy to just do this after an assemble?

Yes, but it should not be needed, and I'd like to understand why it is.
One of the last things do_md_run does is
mddev->changed = 1;

When you next open /dev/md_d0, md_open is called which calls check_disk_change().
This will call into md_fops->md_media_changed which will return the value of mddev->changed, which will be '1'.
So check_disk_change will then call md_fops->revalidate_disk which will set mddev->changed to 0, and will then set bd_invalidated to 1 (as bd_disk->minors > 1 (being 64)).

md_open will then return into do_open (in fs/block_dev.c) and because bd_invalidated is true, it will call rescan_partitions and the partitions will appear.

Hmmm... there is room for a race there. If some other process opens /dev/md_d0 before mdadm gets to close it, it will call rescan_partitions before first calling bd_set_size to update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..

I guess udev must be opening the block device at exactly the wrong time.

I can simulate this by holding /dev/md_d0 open while assembling the array. If I do that, the partitions don't get created. Yuck.

Maybe I could call bd_set_size in md_open before calling check_disk_change..

Yep, this patch seems to fix it. Could you confirm?

Thanks,

NeilBrown

drivers/md/md.c +2 -0
diff .prev/drivers/md/md.c ./drivers/md/md.c
@@ -4485,6 +4485,8 @@ static int md_open(struct inode *inode, 
 	mddev_get(mddev);
 	mddev_unlock(mddev);
 
+	if (mddev->changed)
+		bd_set_size(inode->i_bdev, mddev->array_size << 1);
 	check_disk_change(inode->i_bdev);
  out:
 	return err;
#6 lore raw
In reply to Neil Brown
Quoted 16 lines · Neil Brown
Quoted 15 lines · On Tuesday April 24, [email protected] wrote:
Quoted 14 lines · Neil Brown

This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.

I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.

Note that you can still access the partitions even though they do not appear in /proc/partitions. Any attempt to access and of them will make them all appear in /proc/partitions. But I understand there is sometimes value in seeing them before accessing them.

Quoted 4 lines
Quoted 2 lines
Quoted 1 line

NeilBrown

Um. Are you sure?

"Works for me".

Lucky you ;)

Quoted 3 lines

What happens if you
blockdev --rereadpt /dev/md_d0
?? It probably works then.

Well, that's probably the same as my BLKRRPART ioctl so I guess yes. [confirmed - yes, but blockdev seems to do it twice - I get 2 kernel messages]

Quoted 2 lines

It sounds like someone is deliberately removing all the partition info.

Gremlins?

Quoted 2 lines

Can you try this patch and see if it reports anyone calling '2' on md_d0 ??

Nope, not being called at all.

teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1
mdadm: /dev/md_d0 has been started with 5 drives.

dmesg:
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1
md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0

teak:~# mount /media
mount: special device /dev/md_d0p1 does not exist

no dmesg

teak:~# blockdev --rereadpt /dev/md_d0
dmesg:
 md_d0: p1 p2
 md_d0: p1 p2

did I mention 2.6.20.7 and mdadm v2.5.6 and udev

I'd be happy if I've done something wrong...

anyway, more config data...

teak:~# mdadm --detail /dev/md_d0
/dev/md_d0:
        Version : 01.02.03
  Creation Time : Mon Apr 23 15:13:35 2007
     Raid Level : raid5
     Array Size : 1250241792 (1192.32 GiB 1280.25 GB)
    Device Size : 625120896 (298.08 GiB 320.06 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0
    Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Tue Apr 24 12:49:26 2007
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Name : media
UUID : f7835ba6:e38b6feb:c0cd2e2d:3079db59
Events : 25292

Number Major Minor RaidDevice State
0 8 65 0 active sync /dev/sde1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
5 8 17 3 active sync /dev/sdb1
4 8 81 4 active sync /dev/sdf1

teak:~# cat /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md_d0 auto=part level=raid5 num-devices=5
UUID=f7835ba6:e38b6feb:c0cd2e2d:3079db59
MAILADDR [email protected]

David

#9 lore raw
In reply to Neil Brown
Quoted 30 lines · Neil Brown
Quoted 1 line · On Tuesday April 24, [email protected] wrote:

Neil, isn't it easy to just do this after an assemble?

Yes, but it should not be needed, and I'd like to understand why it is.
One of the last things do_md_run does is
mddev->changed = 1;

When you next open /dev/md_d0, md_open is called which calls check_disk_change().
This will call into md_fops->md_media_changed which will return the value of mddev->changed, which will be '1'.
So check_disk_change will then call md_fops->revalidate_disk which will set mddev->changed to 0, and will then set bd_invalidated to 1 (as bd_disk->minors > 1 (being 64)).

md_open will then return into do_open (in fs/block_dev.c) and because bd_invalidated is true, it will call rescan_partitions and the partitions will appear.

Hmmm... there is room for a race there. If some other process opens /dev/md_d0 before mdadm gets to close it, it will call rescan_partitions before first calling bd_set_size to update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..

I guess udev must be opening the block device at exactly the wrong time.

Quoted 8 lines

I can simulate this by holding /dev/md_d0 open while assembling the array. If I do that, the partitions don't get created. Yuck.

Maybe I could call bd_set_size in md_open before calling check_disk_change..

Yep, this patch seems to fix it. Could you confirm?

almost...

teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1
mdadm: /dev/md_d0 has been started with 5 drives.
teak:~# mount /media
teak:~# umount /media
teak:~# mdadm --stop /dev/md_d0
mdadm: stopped /dev/md_d0
teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1
mdadm: /dev/md_d0 has been started with 5 drives.
teak:~# mount /media
mount: No such file or directory
teak:~# mount /media
teak:~#
(second mount succeeds second time around)

md: md_d0 stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1
md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0
md_d0: p1 p2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystem md_d0p1
Ending clean XFS mount for filesystem: md_d0p1
md: md_d0 stopped.
md: unbind<sde1>
md: export_rdev(sde1)
md: unbind<sdf1>
md: export_rdev(sdf1)
md: unbind<sdb1>
md: export_rdev(sdb1)
md: unbind<sdd1>
md: export_rdev(sdd1)
md: unbind<sdc1>
md: export_rdev(sdc1)
md: md_d0 stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1
md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0
md_d0: p1 p2
XFS: Invalid device [/dev/md_d0p2], error=-2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystem md_d0p1
Ending clean XFS mount for filesystem: md_d0p1

#12 lore raw
In reply to Neil Brown
Quoted 14 lines · Neil Brown

Yes, but it should not be needed, and I'd like to understand why it is.
One of the last things do_md_run does is
mddev->changed = 1;

When you next open /dev/md_d0, md_open is called which calls check_disk_change().
This will call into md_fops->md_media_changed which will return the value of mddev->changed, which will be '1'.
So check_disk_change will then call md_fops->revalidate_disk which will set mddev->changed to 0, and will then set bd_invalidated to 1 (as bd_disk->minors > 1 (being 64)).

Quoted 3 lines

md_open will then return into do_open (in fs/block_dev.c) and because bd_invalidated is true, it will call rescan_partitions and the partitions will appear.

Yuck. The md stack should populate the partition information on device creation *without* needing someone to open the resulting device. That you can tweak mdadm to open the device after creation is fine, but unless no other program is allowed to use the ioctls to start devices, and unless this is a documented part of the API, waiting until second open to populate the device info is just flat wrong. It breaks all sorts of expectations people have regarding things like mount by label, etc.

Quoted 40 lines

Hmmm... there is room for a race there. If some other process opens /dev/md_d0 before mdadm gets to close it, it will call rescan_partitions before first calling bd_set_size to update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..

I guess udev must be opening the block device at exactly the wrong time.

I can simulate this by holding /dev/md_d0 open while assembling the array. If I do that, the partitions don't get created. Yuck.

Maybe I could call bd_set_size in md_open before calling check_disk_change..

Yep, this patch seems to fix it. Could you confirm?

Thanks,

NeilBrown

diff .prev/drivers/md/md.c ./drivers/md/md.c

drivers/md/md.c +2 -1
@@ -4485,6 +4485,8 @@ static int md_open(struct inode *inode, 
 	mddev_get(mddev);
 	mddev_unlock(mddev);
 
+	if (mddev->changed)
+		bd_set_size(inode->i_bdev, mddev->array_size << 1);
 	check_disk_change(inode->i_bdev);
  out:
 	return err;
 
-

To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 5 lines
Doug Ledford <[email protected]>
http://people.redhat.com/dledford

Infiniband specific RPMs can be found at
http://people.redhat.com/dledford/Infiniband
#10 lore raw
In reply to David Greaves

Hi Neil

Just wondering what the status is here - do you need any more from me or is it on your stack?

The patch helped but didn't cure.
After a clean boot it mounted correctly first try. Then I unmounted, stopped and re-assembled the array. The next mount failed.
The subsequent mount succeeded.

How do other block devices initialise their partitions on 'discovery'?

David

Quoted 126 lines · David Greaves
Quoted 29 lines · Neil Brown
Quoted 1 line · On Tuesday April 24, [email protected] wrote:

Neil, isn't it easy to just do this after an assemble?

Yes, but it should not be needed, and I'd like to understand why it is.
One of the last things do_md_run does is
mddev->changed = 1;

When you next open /dev/md_d0, md_open is called which calls check_disk_change().
This will call into md_fops->md_media_changed which will return the value of mddev->changed, which will be '1'.
So check_disk_change will then call md_fops->revalidate_disk which will set mddev->changed to 0, and will then set bd_invalidated to 1 (as bd_disk->minors > 1 (being 64)).

md_open will then return into do_open (in fs/block_dev.c) and because bd_invalidated is true, it will call rescan_partitions and the partitions will appear.

Hmmm... there is room for a race there. If some other process opens /dev/md_d0 before mdadm gets to close it, it will call rescan_partitions before first calling bd_set_size to update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..

I guess udev must be opening the block device at exactly the wrong time.

Quoted 8 lines

I can simulate this by holding /dev/md_d0 open while assembling the array. If I do that, the partitions don't get created. Yuck.

Maybe I could call bd_set_size in md_open before calling check_disk_change..

Yep, this patch seems to fix it. Could you confirm?

almost...

teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1
mdadm: /dev/md_d0 has been started with 5 drives.
teak:~# mount /media
teak:~# umount /media
teak:~# mdadm --stop /dev/md_d0
mdadm: stopped /dev/md_d0
teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1
mdadm: /dev/md_d0 has been started with 5 drives.
teak:~# mount /media
mount: No such file or directory
teak:~# mount /media
teak:~#
(second mount succeeds second time around)

md: md_d0 stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1
md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0
md_d0: p1 p2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystem md_d0p1
Ending clean XFS mount for filesystem: md_d0p1
md: md_d0 stopped.
md: unbind<sde1>
md: export_rdev(sde1)
md: unbind<sdf1>
md: export_rdev(sdf1)
md: unbind<sdb1>
md: export_rdev(sdb1)
md: unbind<sdd1>
md: export_rdev(sdd1)
md: unbind<sdc1>
md: export_rdev(sdc1)
md: md_d0 stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1
md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0
md_d0: p1 p2
XFS: Invalid device [/dev/md_d0p2], error=-2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystem md_d0p1
Ending clean XFS mount for filesystem: md_d0p1

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
Quoted 10 lines · On Monday May 7, [email protected] wrote:

Hi Neil

Just wondering what the status is here - do you need any more from me or is it on your stack?

The patch helped but didn't cure.
After a clean boot it mounted correctly first try. Then I unmounted, stopped and re-assembled the array. The next mount failed.
The subsequent mount succeeded.

I just wrote the following patch. I think it does what you want. Let me know how it goes.

Quoted 3 lines

How do other block devices initialise their partitions on 'discovery'?

They don't call add_disk() until the disk actually exists. md has to call add_disk before the array exists, so that it can be created.

NeilBrown

-----------------------------------
Improve partition detection in md array.

md currently uses ->media_changed to make sure rescan_partitions is call on md array after they are assembled.

However that doesn't happen until the array is opened, which is later than some people would like.

So use blkdev_ioctl to do the rescan immediately that the array has been assembled.

This means we can remove all the ->change infrastructure as it was only used to trigger a partition rescan.

### Diffstat output

 ./drivers/md/md.c           |   26 ++++++++------------------
 ./drivers/md/raid1.c        |    1 -
 ./drivers/md/raid5.c        |    2 --
 ./include/linux/raid/md_k.h |    6 +++---
 4 files changed, 11 insertions(+), 24 deletions(-)
drivers/md/md.c +8 -18
diff .prev/drivers/md/md.c ./drivers/md/md.c
@@ -3104,6 +3104,7 @@ static int do_md_run(mddev_t * mddev)
 	struct gendisk *disk;
 	struct mdk_personality *pers;
 	char b[BDEVNAME_SIZE];
+	struct block_device *bdev;
 
 	if (list_empty(&mddev->disks))
 		/* cannot run an array with no devices.. */
@@ -3331,7 +3332,13 @@ static int do_md_run(mddev_t * mddev)
 	md_wakeup_thread(mddev->thread);
 	md_wakeup_thread(mddev->sync_thread); /* possibly kick off a reshape */
 
-	mddev->changed = 1;
+	bdev = bdget_disk(mddev->gendisk, 0);
+	if (bdev) {
+		bd_set_size(bdev, mddev->array_size << 1);
+		blkdev_ioctl(bdev->bd_inode, NULL, BLKRRPART, 0);
+		bdput(bdev);
+	}
+
 	md_new_event(mddev);
 	kobject_uevent(&mddev->gendisk->kobj, KOBJ_CHANGE);
 	return 0;
@@ -3453,7 +3460,6 @@ static int do_md_stop(mddev_t * mddev, i
 			mddev->pers = NULL;
 
 			set_capacity(disk, 0);
-			mddev->changed = 1;
 
 			if (mddev->ro)
 				mddev->ro = 0;
@@ -4593,20 +4599,6 @@ static int md_release(struct inode *inod
 	return 0;
 }
 
-static int md_media_changed(struct gendisk *disk)
-{
-	mddev_t *mddev = disk->private_data;
-
-	return mddev->changed;
-}
-
-static int md_revalidate(struct gendisk *disk)
-{
-	mddev_t *mddev = disk->private_data;
-
-	mddev->changed = 0;
-	return 0;
-}
 static struct block_device_operations md_fops =
 {
 	.owner		= THIS_MODULE,
@@ -4614,8 +4606,6 @@ static struct block_device_operations md
 	.release	= md_release,
 	.ioctl		= md_ioctl,
 	.getgeo		= md_getgeo,
-	.media_changed	= md_media_changed,
-	.revalidate_disk= md_revalidate,
 };
 
 static int md_thread(void * arg)
drivers/md/raid1.c +0 -1
diff .prev/drivers/md/raid1.c ./drivers/md/raid1.c
@@ -2063,7 +2063,6 @@ static int raid1_resize(mddev_t *mddev, 
 	 */
 	mddev->array_size = sectors>>1;
 	set_capacity(mddev->gendisk, mddev->array_size << 1);
-	mddev->changed = 1;
 	if (mddev->array_size > mddev->size && mddev->recovery_cp == MaxSector) {
 		mddev->recovery_cp = mddev->size << 1;
 		set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
drivers/md/raid5.c +0 -2
diff .prev/drivers/md/raid5.c ./drivers/md/raid5.c
@@ -4514,7 +4514,6 @@ static int raid5_resize(mddev_t *mddev, 
 	sectors &= ~((sector_t)mddev->chunk_size/512 - 1);
 	mddev->array_size = (sectors * (mddev->raid_disks-conf->max_degraded))>>1;
 	set_capacity(mddev->gendisk, mddev->array_size << 1);
-	mddev->changed = 1;
 	if (sectors/2  > mddev->size && mddev->recovery_cp == MaxSector) {
 		mddev->recovery_cp = mddev->size << 1;
 		set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
@@ -4649,7 +4648,6 @@ static void end_reshape(raid5_conf_t *co
 		conf->mddev->array_size = conf->mddev->size *
 			(conf->raid_disks - conf->max_degraded);
 		set_capacity(conf->mddev->gendisk, conf->mddev->array_size << 1);
-		conf->mddev->changed = 1;
 
 		bdev = bdget_disk(conf->mddev->gendisk, 0);
 		if (bdev) {
include/linux/raid/md_k.h +3 -3
diff .prev/include/linux/raid/md_k.h ./include/linux/raid/md_k.h
@@ -201,9 +201,9 @@ struct mddev_s
 	struct mutex			reconfig_mutex;
 	atomic_t			active;
 
-	int				changed;	/* true if we might need to reread partition info */
-	int				degraded;	/* whether md should consider
-							 * adding a spare
+	int				degraded;	/* whether md should
+							 * consider adding a
+							 * spare
 							 */
 	int				barriers_work;	/* initialised to true, cleared as soon
 							 * as a barrier request to slave