Partitioned arrays initially missing from /proc/partitions
Subsystems (2)
In setting up a partitioned array as the boot disk and using a nash initrd to find the root file system by volume label, I see a delay in the appearance of the /dev/md_d0p partitions in /proc/partitions. When the mdadm --assemble command completes, only /dev/md_d0 is visible. Since the raid partitions are not visible after the assemble, the volume label search will not consult them in looking for the root volume and the boot gets aborted. When I run a similar assemble command while up multi-user in a friendlier debug environment I see the same effect and observe that pretty much any access of /dev/md_d0 has the side effect of then making the /dev/md_d0p partitions visible in /proc/partitions.
I tried a few experiments changing the --assemble code in mdadm. If I open() and close() /dev/md_d0 after assembly *before* closing the file descriptor which the assemble step used to assemble the array, there is no effect. Even doing a BLKRRPART ioctl call on the assembly fd or the newly opened fd have no effect. The kernel prints "unknown partition" diagnostics on the console. However, if the assembly fd is first close()'d, a simple open() of /dev/md_d0 and immediate close() of that fd has the side effect of making the /dev/md_d0p partitions visible and one sees the console disk partitioning confirmation from the kernel as well.
Adding the open()/close() after assembly within mdadm solves my problem, but I thought I'd raise the issue on the list as it seems there is a bug somewhere. I see in the kernel md driver that the RUN_ARRAY ioctl() calls do_md_run() which calls md_probe() which calls add_disk() and I gather that this would normally have the side effect of making the partitions visible. However, my experiments at user level seem to imply that the array isn't completely usable until the assembly file descriptor is closed, even on return from the ioctl(), and hence the kernel add_disk() isn't having the desired partitioning side effect at the point it is being invoked.
This is all with kernel 2.6.18 and mdadm 2.3.1
Signature · 4 lines
Mike Accetta ECI Telecom Ltd. Data Networking Division (previously Laurel Networks)
Hi Neil
I think this is a bug.
Essentially if I create an auto=part md device then I get md_d0p? partitions. If I stop the array and just re-assemble, I don't.
It looks like the same (?) problem as Mike (see below - Mike do you have a patch?) but I'm on 2.6.20.7 with mdadm v2.5.6
FWIW I upgraded from 2.6.16 where it worked (but used in-kernel detection which isn't working in 2.6.20 for some reason but I don't mind).
Here's a simple sequence of commands:
teak:~# mdadm --stop /dev/md_d0 mdadm: stopped /dev/md_d0
teak:~# mdadm --create /dev/md_d0 -l5 -n5 --bitmap=internal -e1.2 --auto=part
--name media --force /dev/sde1 /dev/sdc1 /dev/sdd1 missing /dev/sdf1
mdadm: /dev/sde1 appears to be part of a raid array:
level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
mdadm: /dev/sdc1 appears to be part of a raid array:
level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
mdadm: /dev/sdd1 appears to be part of a raid array:
level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
mdadm: /dev/sdf1 appears to be part of a raid array:
level=raid5 devices=5 ctime=Mon Apr 23 15:02:13 2007
Continue creating array? y
mdadm: array /dev/md_d0 started.teak:~# grep md /proc/partitions 254 0 1250241792 md_d0 254 1 1250144138 md_d0p1 254 2 97652 md_d0p2
teak:~# mdadm --stop /dev/md_d0 mdadm: stopped /dev/md_d0
teak:~# mdadm --assemble /dev/md_d0 --auto=part /dev/sde1 /dev/sdc1 /dev/sdd1 /dev/sdf1 mdadm: /dev/md_d0 has been started with 4 drives (out of 5).
teak:~# grep md /proc/partitions 254 0 1250241792 md_d0
If I then run cfdisk it finds the partition table. I write this and get:
teak:~# cfdisk /dev/md_d0
Disk has been changed.
WARNING: If you have created or modified any
DOS 6.x partitions, please see the cfdisk manual
page for additional information.
teak:~# grep md /proc/partitions 254 0 1250241792 md_d0 254 1 1250144138 md_d0p1 254 2 97652 md_d0p2
and the syslog:
Apr 23 15:13:13 localhost kernel: md: md_d0 stopped. Apr 23 15:13:13 localhost kernel: md: unbind<sde1> Apr 23 15:13:13 localhost kernel: md: export_rdev(sde1) Apr 23 15:13:13 localhost kernel: md: unbind<sdf1> Apr 23 15:13:13 localhost kernel: md: export_rdev(sdf1) Apr 23 15:13:13 localhost kernel: md: unbind<sdd1> Apr 23 15:13:13 localhost kernel: md: export_rdev(sdd1) Apr 23 15:13:13 localhost kernel: md: unbind<sdc1> Apr 23 15:13:13 localhost kernel: md: export_rdev(sdc1) Apr 23 15:13:13 localhost mdadm: DeviceDisappeared event detected on md device /dev/md_d0 Apr 23 15:13:36 localhost kernel: md: bind<sde1> Apr 23 15:13:36 localhost kernel: md: bind<sdc1> Apr 23 15:13:36 localhost kernel: md: bind<sdd1> Apr 23 15:13:36 localhost kernel: md: bind<sdf1> Apr 23 15:13:36 localhost kernel: raid5: device sdf1 operational as raid disk 4 Apr 23 15:13:36 localhost kernel: raid5: device sdd1 operational as raid disk 2 Apr 23 15:13:36 localhost kernel: raid5: device sdc1 operational as raid disk 1 Apr 23 15:13:36 localhost kernel: raid5: device sde1 operational as raid disk 0 Apr 23 15:13:36 localhost kernel: raid5: allocated 5236kB for md_d0 Apr 23 15:13:36 localhost kernel: raid5: raid level 5 set md_d0 active with 4 out of 5 devices, algorithm 2 Apr 23 15:13:36 localhost kernel: RAID5 conf printout: Apr 23 15:13:36 localhost kernel: --- rd:5 wd:4 Apr 23 15:13:36 localhost kernel: disk 0, o:1, dev:sde1 Apr 23 15:13:36 localhost kernel: disk 1, o:1, dev:sdc1 Apr 23 15:13:36 localhost kernel: disk 2, o:1, dev:sdd1 Apr 23 15:13:36 localhost kernel: disk 4, o:1, dev:sdf1 Apr 23 15:13:36 localhost kernel: md_d0: bitmap initialized from disk: read 1/1 pages, set 19078 bits, status: 0 Apr 23 15:13:36 localhost kernel: created bitmap (10 pages) for device md_d0 Apr 23 15:13:36 localhost kernel: md_d0: p1 p2 Apr 23 15:13:54 localhost kernel: md: md_d0 stopped. Apr 23 15:13:54 localhost kernel: md: unbind<sdf1> Apr 23 15:13:54 localhost kernel: md: export_rdev(sdf1) Apr 23 15:13:54 localhost kernel: md: unbind<sdd1> Apr 23 15:13:54 localhost kernel: md: export_rdev(sdd1) Apr 23 15:13:54 localhost kernel: md: unbind<sdc1> Apr 23 15:13:54 localhost kernel: md: export_rdev(sdc1) Apr 23 15:13:54 localhost kernel: md: unbind<sde1> Apr 23 15:13:54 localhost kernel: md: export_rdev(sde1) Apr 23 15:13:54 localhost mdadm: DeviceDisappeared event detected on md device /dev/md_d0 Apr 23 15:14:04 localhost kernel: md: md_d0 stopped. Apr 23 15:14:04 localhost kernel: md: bind<sdc1> Apr 23 15:14:04 localhost kernel: md: bind<sdd1> Apr 23 15:14:04 localhost kernel: md: bind<sdf1> Apr 23 15:14:04 localhost kernel: md: bind<sde1> Apr 23 15:14:04 localhost kernel: raid5: device sde1 operational as raid disk 0 Apr 23 15:14:04 localhost kernel: raid5: device sdf1 operational as raid disk 4 Apr 23 15:14:04 localhost kernel: raid5: device sdd1 operational as raid disk 2 Apr 23 15:14:04 localhost kernel: raid5: device sdc1 operational as raid disk 1 Apr 23 15:14:04 localhost kernel: raid5: allocated 5236kB for md_d0 Apr 23 15:14:04 localhost kernel: raid5: raid level 5 set md_d0 active with 4 out of 5 devices, algorithm 2 Apr 23 15:14:04 localhost kernel: RAID5 conf printout: Apr 23 15:14:04 localhost kernel: --- rd:5 wd:4 Apr 23 15:14:04 localhost kernel: disk 0, o:1, dev:sde1 Apr 23 15:14:04 localhost kernel: disk 1, o:1, dev:sdc1 Apr 23 15:14:04 localhost kernel: disk 2, o:1, dev:sdd1 Apr 23 15:14:04 localhost kernel: disk 4, o:1, dev:sdf1 Apr 23 15:14:04 localhost kernel: md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 Apr 23 15:14:04 localhost kernel: created bitmap (10 pages) for device md_d0 Apr 23 15:14:04 localhost kernel: md_d0: unknown partition table
after cfdisk write:
Apr 23 15:33:00 localhost kernel: md_d0: p1 p2
Quoted 34 lines · Back in dec 2006,Mike Accetta wrote:
In setting up a partitioned array as the boot disk and using a nash initrd to find the root file system by volume label, I see a delay in the appearance of the
/dev/md_d0ppartitions in/proc/partitions. When the mdadm --assemble command completes, only/dev/md_d0is visible. Since the raid partitions are not visible after the assemble, the volume label search will not consult them in looking for the root volume and the boot gets aborted. When I run a similar assemble command while up multi-user in a friendlier debug environment I see the same effect and observe that pretty much any access of/dev/md_d0has the side effect of then making the/dev/md_d0ppartitions visible in/proc/partitions.I tried a few experiments changing the --assemble code in mdadm. If I
open()andclose()/dev/md_d0after assembly *before* closing the file descriptor which the assemble step used to assemble the array, there is no effect. Even doing a BLKRRPART ioctl call on the assembly fd or the newly opened fd have no effect. The kernel prints "unknown partition" diagnostics on the console. However, if the assembly fd is first close()'d, a simpleopen()of/dev/md_d0and immediateclose()of that fd has the side effect of making the/dev/md_d0ppartitions visible and one sees the console disk partitioning confirmation from the kernel as well.Adding the open()/close() after assembly within mdadm solves my problem, but I thought I'd raise the issue on the list as it seems there is a bug somewhere. I see in the kernel md driver that the
RUN_ARRAYioctl()callsdo_md_run()which callsmd_probe()which callsadd_disk()and I gather that this would normally have the side effect of making the partitions visible. However, my experiments at user level seem to imply that the array isn't completely usable until the assembly file descriptor is closed, even on return from theioctl(), and hence the kerneladd_disk()isn't having the desired partitioning side effect at the point it is being invoked.This is all with kernel 2.6.18 and mdadm 2.3.1
David Greaves writes:
...
Quoted 2 lines
It looks like the same (?) problem as Mike (see below - Mike do you have a patch?) but I'm on 2.6.20.7 with mdadm v2.5.6
...
We have since started assembling the array from the initrd using --homehost and --auto-update-homehost which takes a different path through the code, and in this path the kernel figures out there are partitions on the array before mdadm exists.
For the previous code path, we had been ruuning with the patch I described in my original post which I've included below. I'd guess that the bug is actually in the kernel code and I looked at it briefly but couldn't figure out how things all fit together well enough to come up with a patch there. The user level patch is a bit of a hack and there may be other code paths that also need a similar patch. I only made this patch in the assembly code path we were executing at the time.
==== BUILD/mdadm/mdadm.c#2 (text) - BUILD/mdadm/mdadm.c#3 (text) ==== content @@ -983,6 +983,10 @@
NULL,
readonly, runstop, NULL, verbose-quiet, force);
close(mdfd);+ mdfd = open(array_list->devname, O_RDONLY);
+ if (mdfd >= 0) {
+ close(mdfd);
+ }
}
}
break;--
Mike Accetta
ECI Telecom Ltd.
Data Networking Division (previously Laurel Networks)
This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.
I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.
Note that you can still access the partitions even though they do not appear in /proc/partitions. Any attempt to access and of them will make them all appear in /proc/partitions. But I understand there is sometimes value in seeing them before accessing them.
NeilBrown
Quoted 9 lines · Neil Brown
This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.
Quoted 6 lines
Note that you can still access the partitions even though they do not appear in
/proc/partitions. Any attempt to access and of them will make them all appear in/proc/partitions. But I understand there is sometimes value in seeing them before accessing them.NeilBrown
Um. Are you sure?
The reason I noticed is that I couldn't mount them until they appeared; see these cut'n'pastes from my terminal history:
teak:~# mount /media/ mount: /dev/md_d0p1 is not a valid block device
teak:~# mount /dev/md_d0p1 /media mount: you must specify the filesystem type
teak:~# xfs_repair -ln /dev/md_d0p2 /dev/md_d0p1 Usage: xfs_repair [-nLvV] [-o subopt[=value]] [-l logdev] [-r rtdev] devname teak:~# ll /dev/md* brw-rw---- 1 root disk 254, 0 2007-04-23 15:44 /dev/md_d0 brw-rw---- 1 root disk 254, 1 2007-04-23 14:46 /dev/md_d0p1 brw-rw---- 1 root disk 254, 2 2007-04-23 14:46 /dev/md_d0p2 brw-rw---- 1 root disk 254, 3 2007-04-23 15:44 /dev/md_d0p3 brw-rw---- 1 root disk 254, 4 2007-04-23 15:44 /dev/md_d0p4
/dev/md:
total 0
teak:~# /etc/init.d/mdadm-raid stop Stopping MD array md_d0...done (stopped). teak:~# /etc/init.d/mdadm-raid start Assembling MD array md_d0...done (degraded [4/5]). Generating udev events for MD arrays...done. teak:~# cfdisk /dev/md_d0
teak:~# mount /dev/md_d0p1 mount: /dev/md_d0p1 is not a valid block device
and so on...
Notice the cfdisk command above. I did this to check the on-array table (it was good). I assume cfdisk opens the array - but the partitions were still not there afterwards. I did not do a 'Write' from in cfdisk this time.
I wouldn't be so concerned at a cosmetic thing in /proc/partitions - the problem is that I can't mount my array after doing an assemble and I have to --create each time - not the nicest solution.
Oh, I'm using udev FWIW.
David
Quoted 11 lines · Mike Accetta
David Greaves writes:
...
Quoted 2 lines
It looks like the same (?) problem as Mike (see below - Mike do you have a patch?) but I'm on 2.6.20.7 with mdadm v2.5.6
...
We have since started assembling the array from the initrd using --homehost and --auto-update-homehost which takes a different path through the code, and in this path the kernel figures out there are partitions on the array before mdadm exists.
Just tried that - doesn't work :)
Quoted 10 lines
For the previous code path, we had been ruuning with the patch I described in my original post which I've included below. I'd guess that the bug is actually in the kernel code and I looked at it briefly but couldn't figure out how things all fit together well enough to come up with a patch there. The user level patch is a bit of a hack and there may be other code paths that also need a similar patch. I only made this patch in the assembly code path we were executing at the time.
==== BUILD/mdadm/mdadm.c#2 (text) - BUILD/mdadm/mdadm.c#3 (text) ==== content
Review line 992
Quoted 10 lines
(quoted diff) +4
-0@@ -983,6 +983,10 @@ NULL, readonly, runstop, NULL, verbose-quiet, force); close(mdfd); + mdfd = open(array_list->devname, O_RDONLY); + if (mdfd >= 0) { + close(mdfd); + } } } break;
Thanks Mike
But this doesn't work for me either :(
I changed array_list to devlist inline with 2.6.9 and it compiles and runs OK.
teak:~# mdadm --stop /dev/md_d0 mdadm: stopped /dev/md_d0 teak:~# /everything/devel/mdadm/mdadm-2.5.6/mdadm --assemble /dev/md_d0 /dev/sd[bcdef]1 mdadm: With Fudge. mdadm: /dev/md_d0 has been started with 5 drives. mdadm: Fudging partition creation. teak:~# mount /media mount: /dev/md_d0p1 is not a valid block device teak:~#
I also wrote a small c program to call the RAID_AUTORUN ioctl - that didn't work either because I'd compiled RAID as a module so the ioctl isn't defined.
currently recompiling the kernel to allow autorun...
David
Quoted 1 line · David Greaves
currently recompiling the kernel to allow autorun...
Which of course won't work because I'm on 1.2 superblocks:
md: Autodetecting RAID arrays.
md: invalid raid superblock magic on sdb1
md: sdb1 has invalid sb, not importing!
md: invalid raid superblock magic on sdc1
md: sdc1 has invalid sb, not importing!
md: invalid raid superblock magic on sdd1
md: sdd1 has invalid sb, not importing!
md: invalid raid superblock magic on sde1
md: sde1 has invalid sb, not importing!
md: invalid raid superblock magic on sdf1
md: sdf1 has invalid sb, not importing!
md: autorun ...
md: ... autorun DONE.
David
PS Dropped Mike from cc since I doubt he's too interested :)
Quoted 9 lines · Neil Brown
This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.
Quoted 6 lines
Note that you can still access the partitions even though they do not appear in
/proc/partitions. Any attempt to access and of them will make them all appear in/proc/partitions. But I understand there is sometimes value in seeing them before accessing them.NeilBrown
For anyone else who is in this boat and doesn't fancy finding somewhere in mdadm
to hack, here's a simple program that issues the BLKRRPART ioctl.
This re-reads the block device partition table and 'works for me'.
I think partx -a would do the same job but for some reason partx isn't in utils-linux for Debian...
Neil, isn't it easy to just do this after an assemble?
David
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <stdlib.h>
#include <fcntl.h>
#include <stdio.h>
#include <linux/raid/md_u.h>
#include <linux/major.h>
#include </usr/include/linux/fs.h>
int main(int argc, char *argv[])
{
int fd;
if (argc != 2)
fprintf(stderr, "Usage: %s <md device>\n", argv[0]);
if ((fd = open(argv[1], O_RDONLY)) == -1) {
fprintf(stderr, "Can't open md device %s\n", argv[1]);
return -1;
}
if (ioctl(fd, BLKRRPART, NULL) != 0) {
fprintf(stderr, "ioctl failed\n");
close (fd);
return -1;
}
close (fd);
return 0;
}Quoted 10 lines · On Tuesday April 24, [email protected] wrote:
Quoted 9 lines · Neil Brown
This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.
Quoted 8 lines
Quoted 6 lines
Note that you can still access the partitions even though they do not appear in
/proc/partitions. Any attempt to access and of them will make them all appear in/proc/partitions. But I understand there is sometimes value in seeing them before accessing them.NeilBrown
Um. Are you sure?
"Works for me".
What happens if you
blockdev --rereadpt /dev/md_d0
?? It probably works then.
It sounds like someone is deliberately removing all the partition info.
Can you try this patch and see if it reports anyone calling '2' on md_d0 ??
block/ioctl.c +3 -0
@@ -17,6 +17,7 @@ static int blkpg_ioctl(struct block_devilong long start, length;int part;int i; +char b[BDEVNAME_SIZE ];if (!capable (CAP_SYS_ADMIN ))return -EACCES ;
@@ -30,6 +31,8 @@ static int blkpg_ioctl(struct block_devi part = p.pno;if (part <=0 || part >= disk->minors)return -EINVAL ; +printk ("blkpg_ioctl: %s called %d on %s\n" , + current->comm, a.op,bdevname (bdev, b));switch (a.op) {case BLKPG_ADD_PARTITION : start = p.start >>9 ;
Quoted 15 lines · On Tuesday April 24, [email protected] wrote:
Quoted 14 lines · Neil Brown
This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.
Note that you can still access the partitions even though they do not appear in
/proc/partitions. Any attempt to access and of them will make them all appear in/proc/partitions. But I understand there is sometimes value in seeing them before accessing them.
Quoted 5 lines
Quoted 1 line
NeilBrown
For anyone else who is in this boat and doesn't fancy finding somewhere in mdadm
to hack, here's a simple program that issues the BLKRRPART ioctl.
This re-reads the block device partition table and 'works for me'.
blockdev --rereadpt /dev/md_d0
does the same thing.
Quoted 5 lines
I think partx -a would do the same job but for some reason partx isn't in utils-linux for Debian...
Neil, isn't it easy to just do this after an assemble?
Yes, but it should not be needed, and I'd like to understand why it is.
One of the last things do_md_run does is
mddev->changed = 1;
When you next open /dev/md_d0, md_open is called which calls check_disk_change().
This will call into md_fops->md_media_changed which will return the value of mddev->changed, which will be '1'.
So check_disk_change will then call md_fops->revalidate_disk which will set mddev->changed to 0, and will then set bd_invalidated to 1 (as bd_disk->minors > 1 (being 64)).
md_open will then return into do_open (in fs/block_dev.c) and because bd_invalidated is true, it will call rescan_partitions and the partitions will appear.
Hmmm... there is room for a race there. If some other process opens /dev/md_d0 before mdadm gets to close it, it will call rescan_partitions before first calling bd_set_size to update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..
I guess udev must be opening the block device at exactly the wrong time.
I can simulate this by holding /dev/md_d0 open while assembling the array. If I do that, the partitions don't get created. Yuck.
Maybe I could call bd_set_size in md_open before calling check_disk_change..
Yep, this patch seems to fix it. Could you confirm?
Thanks,
NeilBrown
drivers/md/md.c +2 -0
@@ -4485,6 +4485,8 @@ static int md_open(struct inode *inode,mddev_get (mddev);mddev_unlock (mddev); +if (mddev->changed) +bd_set_size (inode->i_bdev, mddev->array_size <<1 );check_disk_change (inode->i_bdev); out:return err;
Quoted 16 lines · Neil Brown
Quoted 15 lines · On Tuesday April 24, [email protected] wrote:
Quoted 14 lines · Neil Brown
This problem is very hard to solve inside the kernel.
The partitions will not be visible until the array is opened *after* it has been created. Making the partitions visible before that would be possible, but would be very easy.I think the best solution is Mike's solution which is to simply open/close the array after it has been assembled. I will make sure this is in the next release of mdadm.
Note that you can still access the partitions even though they do not appear in
/proc/partitions. Any attempt to access and of them will make them all appear in/proc/partitions. But I understand there is sometimes value in seeing them before accessing them.
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
NeilBrown
Um. Are you sure?
"Works for me".
Lucky you ;)
Quoted 3 lines
What happens if you
blockdev --rereadpt/dev/md_d0
?? It probably works then.
Well, that's probably the same as my BLKRRPART ioctl so I guess yes. [confirmed - yes, but blockdev seems to do it twice - I get 2 kernel messages]
Quoted 2 lines
It sounds like someone is deliberately removing all the partition info.
Gremlins?
Quoted 2 lines
Can you try this patch and see if it reports anyone calling '2' on
md_d0??
Nope, not being called at all.
teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1 mdadm: /dev/md_d0 has been started with 5 drives.
dmesg:
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0
teak:~# mount /media mount: special device /dev/md_d0p1 does not exist
no dmesg
teak:~# blockdev --rereadpt /dev/md_d0 dmesg: md_d0: p1 p2 md_d0: p1 p2
did I mention 2.6.20.7 and mdadm v2.5.6 and udev
I'd be happy if I've done something wrong...
anyway, more config data...
teak:~# mdadm --detail /dev/md_d0
/dev/md_d0:
Version : 01.02.03
Creation Time : Mon Apr 23 15:13:35 2007
Raid Level : raid5
Array Size : 1250241792 (1192.32 GiB 1280.25 GB)
Device Size : 625120896 (298.08 GiB 320.06 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistentIntent Bitmap : Internal
Update Time : Tue Apr 24 12:49:26 2007
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : media
UUID : f7835ba6:e38b6feb:c0cd2e2d:3079db59
Events : 25292
Number Major Minor RaidDevice State
0 8 65 0 active sync /dev/sde1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
5 8 17 3 active sync /dev/sdb1
4 8 81 4 active sync /dev/sdf1
teak:~# cat /etc/mdadm/mdadm.conf DEVICE partitions ARRAY /dev/md_d0 auto=part level=raid5 num-devices=5 UUID=f7835ba6:e38b6feb:c0cd2e2d:3079db59 MAILADDR [email protected]
David
Quoted 30 lines · Neil Brown
Quoted 1 line · On Tuesday April 24, [email protected] wrote:
Neil, isn't it easy to just do this after an assemble?
Yes, but it should not be needed, and I'd like to understand why it is.
One of the last thingsdo_md_rundoes is
mddev->changed= 1;When you next open
/dev/md_d0,md_openis called which callscheck_disk_change().
This will call intomd_fops->md_media_changedwhich will return the value ofmddev->changed, which will be '1'.
Socheck_disk_changewill then callmd_fops->revalidate_diskwhich will setmddev->changedto 0, and will then setbd_invalidatedto 1 (asbd_disk->minors> 1 (being 64)).
md_openwill then return intodo_open(infs/block_dev.c) and becausebd_invalidatedis true, it will callrescan_partitionsand the partitions will appear.Hmmm... there is room for a race there. If some other process opens
/dev/md_d0before mdadm gets to close it, it will callrescan_partitionsbefore first callingbd_set_sizeto update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..I guess udev must be opening the block device at exactly the wrong time.
Quoted 8 lines
I can simulate this by holding
/dev/md_d0open while assembling the array. If I do that, the partitions don't get created. Yuck.Maybe I could call
bd_set_sizeinmd_openbefore callingcheck_disk_change..Yep, this patch seems to fix it. Could you confirm?
almost...
teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1 mdadm: /dev/md_d0 has been started with 5 drives. teak:~# mount /media teak:~# umount /media teak:~# mdadm --stop /dev/md_d0 mdadm: stopped /dev/md_d0 teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1 mdadm: /dev/md_d0 has been started with 5 drives. teak:~# mount /media mount: No such file or directory teak:~# mount /media teak:~# (second mount succeeds second time around)
md: md_d0 stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0
md_d0: p1 p2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystem md_d0p1
Ending clean XFS mount for filesystem: md_d0p1
md: md_d0 stopped.
md: unbind<sde1>
md: export_rdev(sde1)
md: unbind<sdf1>
md: export_rdev(sdf1)
md: unbind<sdb1>
md: export_rdev(sdb1)
md: unbind<sdd1>
md: export_rdev(sdd1)
md: unbind<sdc1>
md: export_rdev(sdc1)
md: md_d0 stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB for md_d0
raid5: raid level 5 set md_d0 active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for device md_d0
md_d0: p1 p2
XFS: Invalid device [/dev/md_d0p2], error=-2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystem md_d0p1
Ending clean XFS mount for filesystem: md_d0p1
Quoted 14 lines · Neil Brown
Yes, but it should not be needed, and I'd like to understand why it is.
One of the last thingsdo_md_rundoes is
mddev->changed= 1;When you next open
/dev/md_d0,md_openis called which callscheck_disk_change().
This will call intomd_fops->md_media_changedwhich will return the value ofmddev->changed, which will be '1'.
Socheck_disk_changewill then callmd_fops->revalidate_diskwhich will setmddev->changedto 0, and will then setbd_invalidatedto 1 (asbd_disk->minors> 1 (being 64)).
Quoted 3 lines
md_openwill then return intodo_open(infs/block_dev.c) and becausebd_invalidatedis true, it will callrescan_partitionsand the partitions will appear.
Yuck. The md stack should populate the partition information on device creation *without* needing someone to open the resulting device. That you can tweak mdadm to open the device after creation is fine, but unless no other program is allowed to use the ioctls to start devices, and unless this is a documented part of the API, waiting until second open to populate the device info is just flat wrong. It breaks all sorts of expectations people have regarding things like mount by label, etc.
Quoted 40 lines
Hmmm... there is room for a race there. If some other process opens
/dev/md_d0before mdadm gets to close it, it will callrescan_partitionsbefore first callingbd_set_sizeto update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..I guess udev must be opening the block device at exactly the wrong time.
I can simulate this by holding
/dev/md_d0open while assembling the array. If I do that, the partitions don't get created. Yuck.Maybe I could call
bd_set_sizeinmd_openbefore callingcheck_disk_change..Yep, this patch seems to fix it. Could you confirm?
Thanks,
NeilBrown
diff
.prev/drivers/md/md.c./drivers/md/md.cdrivers/md/md.c +2
-1@@ -4485,6 +4485,8 @@ static int md_open(struct inode *inode,mddev_get (mddev);mddev_unlock (mddev); +if (mddev->changed) +bd_set_size (inode->i_bdev, mddev->array_size <<1 );check_disk_change (inode->i_bdev); out:return err; -To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Doug Ledford <[email protected]> http://people.redhat.com/dledford Infiniband specific RPMs can be found at http://people.redhat.com/dledford/Infiniband
Hi Neil
Just wondering what the status is here - do you need any more from me or is it on your stack?
The patch helped but didn't cure.
After a clean boot it mounted correctly first try. Then I unmounted, stopped and re-assembled the array. The next mount failed.
The subsequent mount succeeded.
How do other block devices initialise their partitions on 'discovery'?
David
Quoted 126 lines · David Greaves
Quoted 29 lines · Neil Brown
Quoted 1 line · On Tuesday April 24, [email protected] wrote:
Neil, isn't it easy to just do this after an assemble?
Yes, but it should not be needed, and I'd like to understand why it is.
One of the last thingsdo_md_rundoes is
mddev->changed= 1;When you next open
/dev/md_d0,md_openis called which callscheck_disk_change().
This will call intomd_fops->md_media_changedwhich will return the value ofmddev->changed, which will be '1'.
Socheck_disk_changewill then callmd_fops->revalidate_diskwhich will setmddev->changedto 0, and will then setbd_invalidatedto 1 (asbd_disk->minors> 1 (being 64)).
md_openwill then return intodo_open(infs/block_dev.c) and becausebd_invalidatedis true, it will callrescan_partitionsand the partitions will appear.Hmmm... there is room for a race there. If some other process opens
/dev/md_d0before mdadm gets to close it, it will callrescan_partitionsbefore first callingbd_set_sizeto update the size of the bdev. So when we try to read the partition table, it will appear to be reading past the EOF, and will not actually read anything..I guess udev must be opening the block device at exactly the wrong time.
Quoted 8 lines
I can simulate this by holding
/dev/md_d0open while assembling the array. If I do that, the partitions don't get created. Yuck.Maybe I could call
bd_set_sizeinmd_openbefore callingcheck_disk_change..Yep, this patch seems to fix it. Could you confirm?
almost...
teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1 mdadm: /dev/md_d0 has been started with 5 drives. teak:~# mount /media teak:~# umount /media teak:~# mdadm --stop /dev/md_d0 mdadm: stopped /dev/md_d0 teak:~# mdadm --assemble /dev/md_d0 --auto=parts /dev/sd[bcdef]1 mdadm: /dev/md_d0 has been started with 5 drives. teak:~# mount /media mount: No such file or directory teak:~# mount /media teak:~# (second mount succeeds second time around)md:
md_d0stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB formd_d0
raid5: raid level 5 setmd_d0active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for devicemd_d0
md_d0: p1 p2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystemmd_d0p1
Ending clean XFS mount for filesystem:md_d0p1
md:md_d0stopped.
md: unbind<sde1>
md:export_rdev(sde1)
md: unbind<sdf1>
md:export_rdev(sdf1)
md: unbind<sdb1>
md:export_rdev(sdb1)
md: unbind<sdd1>
md:export_rdev(sdd1)
md: unbind<sdc1>
md:export_rdev(sdc1)
md:md_d0stopped.
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sde1>
raid5: device sde1 operational as raid disk 0
raid5: device sdf1 operational as raid disk 4
raid5: device sdb1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: allocated 5236kB formd_d0
raid5: raid level 5 setmd_d0active with 5 out of 5 devices, algorithm 2 RAID5 conf printout:
--- rd:5 wd:5
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdd1
disk 3, o:1, dev:sdb1
disk 4, o:1, dev:sdf1md_d0: bitmap initialized from disk: read 1/1 pages, set 0 bits, status: 0 created bitmap (10 pages) for devicemd_d0
md_d0: p1 p2
XFS: Invalid device [/dev/md_d0p2], error=-2
Filesystem "md_d0p1": Disabling barriers, not supported with external log device XFS mounting filesystemmd_d0p1
Ending clean XFS mount for filesystem:md_d0p1-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · On Monday May 7, [email protected] wrote:
Hi Neil
Just wondering what the status is here - do you need any more from me or is it on your stack?
The patch helped but didn't cure.
After a clean boot it mounted correctly first try. Then I unmounted, stopped and re-assembled the array. The next mount failed.
The subsequent mount succeeded.
I just wrote the following patch. I think it does what you want. Let me know how it goes.
Quoted 3 lines
How do other block devices initialise their partitions on 'discovery'?
They don't call add_disk() until the disk actually exists. md has to call add_disk before the array exists, so that it can be created.
NeilBrown
-----------------------------------
Improve partition detection in md array.
md currently uses ->media_changed to make sure rescan_partitions is call on md array after they are assembled.
However that doesn't happen until the array is opened, which is later than some people would like.
So use blkdev_ioctl to do the rescan immediately that the array has been assembled.
This means we can remove all the ->change infrastructure as it was only used to trigger a partition rescan.
### Diffstat output
./drivers/md/md.c | 26 ++++++++------------------ ./drivers/md/raid1.c | 1 - ./drivers/md/raid5.c | 2 -- ./include/linux/raid/md_k.h | 6 +++--- 4 files changed, 11 insertions(+), 24 deletions(-)
drivers/md/md.c +8 -18
@@ -3104,6 +3104,7 @@ static int do_md_run(mddev_t * mddev)struct gendisk *disk;struct mdk_personality *pers;char b[BDEVNAME_SIZE ]; +struct block_device *bdev;if (list_empty (&mddev->disks))/* cannot run an array with no devices.. */
@@ -3331,7 +3332,13 @@ static int do_md_run(mddev_t * mddev)md_wakeup_thread (mddev->thread);md_wakeup_thread (mddev->sync_thread);/* possibly kick off a reshape */ - mddev->changed =1 ; + bdev =bdget_disk (mddev->gendisk,0 ); +if (bdev) { +bd_set_size (bdev, mddev->array_size <<1 ); +blkdev_ioctl (bdev->bd_inode,NULL ,BLKRRPART ,0 ); +bdput (bdev); + } +md_new_event (mddev);kobject_uevent (&mddev->gendisk->kobj,KOBJ_CHANGE );return 0 ;
@@ -3453,7 +3460,6 @@ static int do_md_stop(mddev_t * mddev, i mddev->pers =NULL ;set_capacity (disk,0 ); - mddev->changed =1 ;if (mddev->ro) mddev->ro =0 ;
@@ -4593,20 +4599,6 @@ static int md_release(struct inode *inodreturn 0 ; } -static int md_media_changed (struct gendisk *disk) -{ -mddev_t *mddev = disk->private_data; - -return mddev->changed; -} - -static int md_revalidate (struct gendisk *disk) -{ -mddev_t *mddev = disk->private_data; - - mddev->changed =0 ; -return 0 ; -}static struct block_device_operations md_fops = { .owner =THIS_MODULE ,
@@ -4614,8 +4606,6 @@ static struct block_device_operations md .release = md_release, .ioctl = md_ioctl, .getgeo = md_getgeo, - .media_changed = md_media_changed, - .revalidate_disk= md_revalidate, };static int md_thread (void * arg)
drivers/md/raid1.c +0 -1
@@ -2063,7 +2063,6 @@ static int raid1_resize(mddev_t *mddev,*/ mddev->array_size = sectors>>1 ;set_capacity (mddev->gendisk, mddev->array_size <<1 ); - mddev->changed =1 ;if (mddev->array_size > mddev->size && mddev->recovery_cp == MaxSector) { mddev->recovery_cp = mddev->size <<1 ;set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);
drivers/md/raid5.c +0 -2
@@ -4514,7 +4514,6 @@ static int raid5_resize(mddev_t *mddev, sectors &= ~((sector_t )mddev->chunk_size/512 -1 ); mddev->array_size = (sectors * (mddev->raid_disks-conf->max_degraded))>>1 ;set_capacity (mddev->gendisk, mddev->array_size <<1 ); - mddev->changed =1 ;if (sectors/2 > mddev->size && mddev->recovery_cp == MaxSector) { mddev->recovery_cp = mddev->size <<1 ;set_bit (MD_RECOVERY_NEEDED , &mddev->recovery);
@@ -4649,7 +4648,6 @@ static void end_reshape(raid5_conf_t *co conf->mddev->array_size = conf->mddev->size * (conf->raid_disks - conf->max_degraded);set_capacity (conf->mddev->gendisk, conf->mddev->array_size <<1 ); - conf->mddev->changed =1 ; bdev =bdget_disk (conf->mddev->gendisk,0 );if (bdev) {
include/linux/raid/md_k.h +3 -3
@@ -201,9 +201,9 @@ struct mddev_sstruct mutex reconfig_mutex;atomic_t active; -int changed;/* true if we might need to reread partition info */ -int degraded;/* whether md should consider -* adding a spare +int degraded;/* whether md should +* consider adding a +* spare */ int barriers_work;/* initialised to true, cleared as soon * as a barrier request to slave