mdraid rootfs support
cover mdraid rootfs support
This series is a first take at dracut support for an mdraid rootfs. It includes considerations for the new external metadata formats supported in the latest development branch of mdadm:
git://neil.brown.name/mdadm devel-3.0
This is an RFC because it is not clear to me that a single call to "udevadm settle" is enough to guarantee discovery of all storage devices ahead of raid assembly. A cursory test with (4) ahci attached drives was successful.
Regards,
Dan
---
Dan Williams (3):
add more disk id helpers to udevexe
raid: external and internal metadata support
gen-mod-lists: create lists of modules that may talk to a root device
dracut | 13 ++++++++++--- gen-mod-lists | 34 ++++++++++++++++++++++++++++++++++ init | 10 ++++++++++ 3 files changed, 54 insertions(+), 3 deletions(-) create mode 100755 gen-mod-lists
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
RFC PATCH 1/3 gen-mod-lists: create lists of modules that may talk to a root device
In reply to a message not in the archive[email protected]:
The idea is that we don't want to include every single module, but we want to include every module that might define a block device to boot from, or a network device to network boot from. Having it in the upstream kernel would be helpful, although how it's generated now is obviously a hack.
Doing it at runtime in dracut would work, but would be obviously slow.
This is a temporary hack to duplicate this functionality from the Fedora kernel srpm in dracut.
Also added "raid" modules.
--- gen-mod-lists | 34 ++++++++++++++++++++++++++++++++++ 1 files changed, 34 insertions(+), 0 deletions(-) create mode 100755 gen-mod-lists
gen-mod-lists new file +34 -0
@@ -0,0 +1,34 @@ +#!/bin/bash + +# Copied from from kernel.spec (kernel-2.6.27.12-78.2.8.fc9.src.rpm) +# Creates /lib/modules/$KernelVer/modules.{block,networking,raid} + +KernelVer=$1 +[ -n $KernelVer ] && KernelVer=$(uname -r) + +if [ ! -d /lib/modules/$KernelVer ]; then + echo "error: could not find /lib/modules/$KernelVer" + exit 1 +fi + +find /lib/modules/$KernelVer -name "*.ko" -type f >modnames + +# Generate a list of modules for block and networking. + +fgrep /drivers/ modnames | xargs --no-run-if-empty nm -upA | +sed -n 's,^.*/\([^/]*\.ko\): *U \(.*\)$,\1 \2,p' > drivers.undef + +collect_modules_list() +{ + sed -r -n -e "s/^([^ ]+) \\.?($2)\$/\\1/p" drivers.undef | + LC_ALL=C sort -u > /lib/modules/$KernelVer/modules.$1 +} + +collect_modules_list networking \ + 'register_netdev|ieee80211_register_hw|usbnet_probe' +collect_modules_list block \ + 'ata_scsi_ioctl|scsi_add_host|blk_init_queue|register_mtd_blktrans|scsi_esp_register' + +# mdraid modules, could be made part of 'block' +collect_modules_list raid \ + 'register_md_personality'
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
RFC PATCH 2/3 raid: external and internal metadata support
In reply to a message not in the archiveExternal metadata support implies that metadata events are handled by a userspace daemon. This daemon, mdmon, needs to be started ahead of the rootfs being mounted to handle the raid volume dirty bit. Even if the rootfs is mounted read-only the rootdev may still be written by filesystem journal-playback operations.
After the rootfs is mounted to /sysroot, mdmon is restarted in the new namespace. The command "mdmon /proc/mdstat /sysroot" tells mdmon to terminate any instances in the current namespace and then launch new instances, chroot(2) to /sysroot, per container device found in /proc/mdstat.
--- dracut | 11 +++++++++-- init | 10 ++++++++++ 2 files changed, 19 insertions(+), 2 deletions(-)
dracut +9 -2
@@ -69,13 +69,14 @@ initdir=$(mktemp -d -t initramfs.XXXXXX) exe="/bin/bash /bin/mount /bin/mknod /bin/mkdir /sbin/modprobe /sbin/udevd /sbin/udevadm /sbin/nash /bin/kill /sbin/pidof /bin/sleep /bin/echo /usr/sbin/chroot" lvmexe="/sbin/lvm" cryptexe="/sbin/cryptsetup" +raidexe="/sbin/mdadm /sbin/mdmon" # and some things that are nice for debugging debugexe="/bin/ls /bin/cat /bin/ln /bin/ps /bin/grep /bin/more" # udev things we care about udevexe="/lib/udev/vol_id /lib/udev/console_init" # install base files -for binary in $exe $debugexe $udevexe $lvmexe $cryptexe ; do +for binary in $exe $debugexe $udevexe $lvmexe $cryptexe $raidexe ; do inst $binary $initdir done
@@ -152,7 +153,7 @@ cp $switchroot $initdir/sbin/switch_root mkdir -p $initdir/etc $initdir/proc $initdir/sys $initdir/sysroot $initdir/dev/pts # FIXME: hard-coded module list of doom. -[ -z "$modules" ] && modules="=ata =block =drm dm-crypt aes sha256 cbc" +[ -z "$modules" ] && modules="=ata =block =drm =raid dm-crypt aes sha256 cbc" mkdir -p $initdir/lib/modules/$kernel # expand out module deps, etc
@@ -171,6 +172,12 @@ if [ -x /usr/libexec/plymouth/plymouth-populate-initrd ]; then /usr/libexec/plymouth/plymouth-populate-initrd -t "$initdir" || : fi +# raid +# mdadm.conf allows mdadm to disambiguate foreign arrays for some metadata types +# check /etc and /etc/mdadm (/etc wins if both are present) +[ -f /etc/mdadm/mdadm.conf ] && inst /etc/mdadm/mdadm.conf "$initdir" /etc/mdadm.conf +[ -f /etc/mdadm.conf ] && inst /etc/mdadm.conf "$initdir" + pushd $initdir >/dev/null find . |cpio -H newc -o |gzip -9 > $outfile popd >/dev/null
init +10 -0
@@ -46,6 +46,13 @@ mknod /dev/tty1 c 4 1 /sbin/udevd --daemon /sbin/udevadm trigger +# start any defined raid arrays +# we settle before assembling to hopefully prevent prematurely degrading arrays +if [ -f /etc/mdadm.conf ]; then + /sbin/udevadm settle + /sbin/mdadm -Asc /etc/mdadm.conf +fi + # mount the rootfs NEWROOT="/sysroot"
@@ -110,6 +117,9 @@ kill `pidof udevd` [ -x /bin/plymouth ] && /bin/plymouth --newroot=$NEWROOT +# switch any mdmon instances to newroot +[ -f /etc/mdadm.conf ] && /sbin/mdmon /proc/mdstat $NEWROOT + # FIXME: nash die die die exec /sbin/switch_root # davej doesn't like initrd bugs
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
RFC PATCH 3/3 add more disk id helpers to udevexe
In reply to a message not in the archiveAllow udev to create /dev/disk/by-id links
--- dracut | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
dracut +1 -1
@@ -73,7 +73,7 @@ raidexe="/sbin/mdadm /sbin/mdmon" # and some things that are nice for debugging debugexe="/bin/ls /bin/cat /bin/ln /bin/ps /bin/grep /bin/more" # udev things we care about -udevexe="/lib/udev/vol_id /lib/udev/console_init" +udevexe="/lib/udev/*_id /lib/udev/console_init" # install base files for binary in $exe $debugexe $udevexe $lvmexe $cryptexe $raidexe ; do
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 4 lines · Dan Williams
index 706127f..0294502 100755
init mknod /dev/tty1 c 4 1 · line 55
Quoted 10 lines
init +7
-0@@ -46,6 +46,13 @@ mknod /dev/tty1 c 4 1 /sbin/udevd --daemon /sbin/udevadm trigger +# start any defined raid arrays +# we settle before assembling to hopefully prevent prematurely degrading arrays +if [ -f /etc/mdadm.conf ]; then + /sbin/udevadm settle + /sbin/mdadm -Asc /etc/mdadm.conf +fi +
RAID arrays should be getting started by udev rules, not by explicit calls to mdadm in /init. Yes, this means having proper integration with udev for your kernel pieces. But this ends up helping everything as it will also let us lose the multiple redundant calls to things like mdadm (and lvm, etc) throughout the boot process which should just be occurring as devices show up.
init kill `pidof udevd` · line 122
Quoted 3 lines
init +3
-0+# switch any mdmon instances to newroot +[ -f /etc/mdadm.conf ] && /sbin/mdmon /proc/mdstat $NEWROOT +
Is there a real need for mdmon to start prior to being in the real rootfs?
Jeremy
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 3 lines
Is there a real need for mdmon to start prior to being in the real rootfs?
mdmon is needed to change raid array to RW mode, so as long as rootfs is mounted RO, mdmon can be started in real rootfs.-- To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 5 lines · Jacek
Quoted 3 lines
Is there a real need for mdmon to start prior to being in the real rootfs?
mdmon is needed to change raid array to RW mode, so as long as rootfs is mounted RO, mdmon can be started in real rootfs.--
No, that is what I originally thought until I tried to mount an xfs filesystem that had been uncleanly shutdown. Even if the rootfs is mounted read-only the backing device needs to be read-write to recover the journal.
--
Dan
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 4 lines · Jacek
Quoted 2 lines
Is there a real need for mdmon to start prior to being in the real rootfs?
mdmon is needed to change raid array to RW mode, so as long as rootfs is mounted RO, mdmon can be started in real rootfs.
So, for one particular specific type of block device, you need a daemon to switch it writable. Every other type of block device can handle this without separate tooling. I'm not seeing how this is an improvement.
Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 5 lines · Bill Nottingham
So, for one particular specific type of block device, you need a daemon to switch it writable. Every other type of block device can handle this without separate tooling. I'm not seeing how this is an improvement.
It is not just setting writable, mdmon is also there to clear the bit when writes have quiesced. Raid devices have always been special in that they need to manage a dirty bit in their metadata to determine if a resync needs to be performed after a dirty shutdown. With hardware raid or pure kernel (MD metadata) raid this mechanism is hidden.
External metadata raid is akin to fuse filesystems. The kernel provides the generic infrastructure and a userspace daemon handles the implementation details.
The improvement is that with one kernel implementation we can support any number of metadata formats.
--
Dan
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 2 lines · Dan Williams
It is not just setting writable, mdmon is also there to clear the bit when writes have quiesced.
Let me just see if I understand this infrastructure correctly.
- device is set writable
- kernel tells userspace
- userspace frobs bit in superblock to say 'I want to be dirty!'
- userspace tells kernel
- kernel writes bit to disk
... stuff happens ...
- userspace tells kernel to unmount, or remount R/O
- kernel tells userspace "hey, i unmounted this" (userspace freaks out because the filesystem the daemon is running on just went away)
- userspace frobs bit in superblock to say 'This array is CLEAN!'
- userspace tells kernel
- kernel writes bit to disk
Is that really how it's supposed to work?
So, why isn't the ext* journal or filesystem unclean flag handled via a userspace file monitoring daemon, then?
Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 3 lines · Bill Nottingham
So, why isn't the ext* journal or filesystem unclean flag handled via a userspace file monitoring daemon, then?
Dan, Neil
Are any plans about rewrite mdmon in kernel-space?
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 6 lines · Jeremy Katz
RAID arrays should be getting started by udev rules, not by explicit calls to mdadm in /init. Yes, this means having proper integration with udev for your kernel pieces. But this ends up helping everything as it will also let us lose the multiple redundant calls to things like mdadm (and lvm, etc) throughout the boot process which should just be occurring as devices show up.
The trick is determining when a device has not shown up yet versus it will never show up... to prevent the array being marked degraded prematurely. Is there some mechanism for udev to broadcast "if you were waiting for more devices to show up don't hold your breath"? I.e. at the point where a call to "udevadm settle" would reasonably be expected to not find any pending events?
I am thinking something along the lines of:
<udev: add disk>
mdadm --incremental --no-degraded $dev
<udev: add disk>
mdadm --incremental --no-degraded $dev
<udev: probably no more devices>
mdadm --incremental $last_dev
--
Dan
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 8 lines · Bill Nottingham
Quoted 2 lines · Dan Williams
It is not just setting writable, mdmon is also there to clear the bit when writes have quiesced.
Let me just see if I understand this infrastructure correctly.
- device is set writable
- kernel tells userspace
...tells userspace that we want to transition the array from clean to dirty, yes
Quoted 1 line
- userspace frobs bit in superblock to say 'I want to be dirty!'
yes
Quoted 1 line
- userspace tells kernel
...yup array is dirty, start writing.
Quoted 6 lines
- kernel writes bit to disk
... stuff happens ...
- userspace tells kernel to unmount, or remount R/O
- kernel tells userspace "hey, i unmounted this" (userspace freaks out because the filesystem the daemon is running on just went away)
mdmon does not know or care if the *filesystem* is read-only. It is reading and writing /proc, /sys, and the raw disk devices.
Quoted 1 line
- userspace frobs bit in superblock to say 'This array is CLEAN!'
...not in this scenario no.
Quoted 4 lines
- userspace tells kernel
- kernel writes bit to diskIs that really how it's supposed to work?
You lost me at userspace freaks out, but that is the general flow.
Quoted 2 lines
So, why isn't the ext* journal or filesystem unclean flag handled via a userspace file monitoring daemon, then?
I'm not trying to be obtuse, but because it isn't. Put another way, consider what extra tools the initramfs would need if we wanted to support an ntfs-3g rootfs.
--
Dan
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 5 lines · Dan Williams
Quoted 3 lines
- kernel tells userspace "hey, i unmounted this" (userspace freaks out because the filesystem the daemon is running on just went away)
mdmon does not know or care if the *filesystem* is read-only. It is reading and writing /proc, /sys, and the raw disk devices.
Your daemon has to be running from somewhere. That tends to be reduced to the initramfs that you've already deleted and switchrooted away from (in which case, good luck on upgrades of your userspace tools - you're stuck with the version in the initramfs you booted from, even if your later userspace tools end up using some later protocol). You can't run it from the rootfs, because that's a chicken/egg scenario (or you'll switch to it, and then be unable to mark it clean, because you can't mark the array r/o, because the filesystem is r/w, which you can't undo because the daemon is running on it...)
Long-lived daemons running from the initramfs aren't really good. We don't run udev that way.
Quoted 2 lines
Quoted 1 line
- userspace frobs bit in superblock to say 'This array is CLEAN!'
...not in this scenario no.
Then when is the clean bit set?
Quoted 6 lines
Quoted 2 lines
So, why isn't the ext* journal or filesystem unclean flag handled via a userspace file monitoring daemon, then?
I'm not trying to be obtuse, but because it isn't. Put another way, consider what extra tools the initramfs would need if we wanted to support an ntfs-3g rootfs.
You're asking for *the exact same thing*... just RAID specific.
Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 4 lines · Bill Nottingham
Quoted 3 lines · Dan Williams
Quoted 3 lines
- kernel tells userspace "hey, i unmounted this" (userspace freaks out because the filesystem the daemon is running on just went away)
Quoted 8 lines
Quoted 2 lines
mdmon does not know or care if the *filesystem* is read-only. It is reading and writing /proc, /sys, and the raw disk devices.
Your daemon has to be running from somewhere. That tends to be reduced to the initramfs that you've already deleted and switchrooted away from (in which case, good luck on upgrades of your userspace tools - you're stuck with the version in the initramfs you booted from, even if your later userspace tools end up using some later protocol).
Actually no, your not necessarily stuck with the mdmon from boot. In a pinch you could "mdmon /proc/mdstat /". Worse case you need to re-dracut and reboot, but that is already more flexible than the metadata handled in kernel-space approach.
Quoted 5 lines
You can't run it from the rootfs, because that's a chicken/egg scenario (or you'll switch to it, and then be unable to mark it clean, because you can't mark the array r/o, because the filesystem is r/w, which you can't undo because the daemon is running on it...)
Array r/o is a separate issue from the raid metadata clean bit, see below.
Quoted 1 line
Long-lived daemons running from the initramfs aren't really good.
I agree, and I initially looked for ways to wait until the rootfs was available before launching mdmon... then I hit the xfs journal recovery case.
Quoted 1 line
We don't run udev that way.
At first glance it looks like plymouth is run this way, but I am probably mistaken. From dracut/init:
[ -x /bin/plymouth ] && /bin/plymouth --newroot=$NEWROOT
One might say "just set the dirty bit, terminate, and wait for the mdmon in the rootfs to take over". The problem is that a disk could fail in this window, and this event needs to be handled before the kernel does anything else to the array.
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
- userspace frobs bit in superblock to say 'This array is CLEAN!'
...not in this scenario no.
Then when is the clean bit set?
The clean bit can be set as soon as the parity data is in sync with the data on the other drives. We typically wait for some period of write-inactivity to avoid needlessly touching the metadata after every write.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
So, why isn't the ext* journal or filesystem unclean flag handled via a userspace file monitoring daemon, then?
I'm not trying to be obtuse, but because it isn't. Put another way, consider what extra tools the initramfs would need if we wanted to support an ntfs-3g rootfs.
You're asking for *the exact same thing*... just RAID specific.
The key difference being that there are performance reasons for handling filesystem metadata in the kernel. Raid metadata events are always in the slow path.
--
Dan
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 9 lines · Jacek Danecki
Quoted 3 lines · Bill Nottingham
So, why isn't the ext* journal or filesystem unclean flag handled via a userspace file monitoring daemon, then?
Dan, Neil
Are any plans about rewrite mdmon in kernel-space?
Definitely not.
There is more to this than the 'unclean' flag.
The really important task for mdmon (which hopefully it never has to perform...) is to record device failures (which is what RAID is really all about).
If a device fails while trying to write to it, we cannot allow that write to complete until the other devices have had that device failure recorded on them. Otherwise, following an unclean shutdown we might trust the data that is on that drive, which is now out-of-date.
The task of mdmon is to discover when there have been write error, record the device failure in the metadata, then allow the write to complete.
It has a number of other tasks as well, but that is the important one which means that it must always be running when the array is writable.
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 9 lines · NeilBrown
There is more to this than the 'unclean' flag.
The really important task for mdmon (which hopefully it never has to perform...) is to record device failures (which is what RAID is really all about).If a device fails while trying to write to it, we cannot allow that write to complete until the other devices have had that device failure recorded on them. Otherwise, following an unclean shutdown we might trust the data that is on that drive, which is now out-of-date.
OK, so:
1) kernel sends write request. If error....
2) <some error occurs>
3) kernel sends error to userspace
4) mdmon wakes up
5) mdmon decides where to record this
6) mdmon writes to super blocks
7) go to step one, hope you don't hit step 2 this time
This now means that reliable suspend and resume is completely impossible on RAID devices, just as it is on FUSE. You can't have waking up userspace be part of your write and sync process - you've just deadlocked at step 3/4.
Unless I've missed something here?
Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 2 lines · Dan Williams
Actually no, your not necessarily stuck with the mdmon from boot. In a pinch you could "mdmon
/proc/mdstat/".
Not really.
You state:
Quoted 4 lines
One might say "just set the dirty bit, terminate, and wait for the mdmon in the rootfs to take over". The problem is that a disk could fail in this window, and this event needs to be handled before the kernel does anything else to the array.
...
Quoted 4 lines
The clean bit can be set as soon as the parity data is in sync with the data on the other drives. We typically wait for some period of write-inactivity to avoid needlessly touching the metadata after every write.
You shut down the machine. After a while, you get to the point where you're getting ready to unmount the filesystem. Since mdmon's running on it (if you started it post boot), you have to kill it. After that point, there are going to be writes (a final sync, if nothing else, when you unmount the filesystem.) And you won't be able to set any RAID metadata flags then, as the daemon won't be running. So, doing a later run of "mdmon /proc/mdstat" doesn't fully protect you.
Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 18 lines · Bill Nottingham
Quoted 2 lines · Dan Williams
Actually no, your not necessarily stuck with the mdmon from boot. In a pinch you could "mdmon
/proc/mdstat/".Not really.
You state:
Quoted 4 lines
One might say "just set the dirty bit, terminate, and wait for the mdmon in the rootfs to take over". The problem is that a disk could fail in this window, and this event needs to be handled before the kernel does anything else to the array.
...
Quoted 4 lines
The clean bit can be set as soon as the parity data is in sync with the data on the other drives. We typically wait for some period of write-inactivity to avoid needlessly touching the metadata after every write.
Quoted 7 lines
You shut down the machine. After a while, you get to the point where you're getting ready to unmount the filesystem. Since mdmon's running on it (if you started it post boot), you have to kill it. After that point, there are going to be writes (a final sync, if nothing else, when you unmount the filesystem.) And you won't be able to set any RAID metadata flags then, as the daemon won't be running. So, doing a later run of "mdmon
/proc/mdstat" doesn't fully protect you.
???
Last time I checked, Linux would not unmount the root filesystem. It just remounts it 'read-only'.
Is that going to change?
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 18 lines · Bill Nottingham
Quoted 2 lines · Dan Williams
Actually no, your not necessarily stuck with the mdmon from boot. In a pinch you could "mdmon
/proc/mdstat/".Not really.
You state:
Quoted 4 lines
One might say "just set the dirty bit, terminate, and wait for the mdmon in the rootfs to take over". The problem is that a disk could fail in this window, and this event needs to be handled before the kernel does anything else to the array.
...
Quoted 4 lines
The clean bit can be set as soon as the parity data is in sync with the data on the other drives. We typically wait for some period of write-inactivity to avoid needlessly touching the metadata after every write.
Quoted 8 lines
You shut down the machine. After a while, you get to the point where you're getting ready to unmount the filesystem. Since mdmon's running on it (if you started it post boot), you have to kill it. After that point, there are going to be writes (a final sync, if nothing else, when you unmount the filesystem.) And you won't be able to set any RAID metadata flags then, as the daemon won't be running. So, doing a later run of "mdmon
/proc/mdstat" doesn't fully protect you.
mdmon needs some coordination with the shutdown scripts to be kept alive until the rootfs is marked readonly... actually up until the point where the rootdev can be marked readonly.
If you take a look at Debian's killall implementation it has provisions to exclude fuse and other critical userspace process from killall. A similar exclusion is needed for mdmon.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 11 lines · NeilBrown
Quoted 7 lines
You shut down the machine. After a while, you get to the point where you're getting ready to unmount the filesystem. Since mdmon's running on it (if you started it post boot), you have to kill it. After that point, there are going to be writes (a final sync, if nothing else, when you unmount the filesystem.) And you won't be able to set any RAID metadata flags then, as the daemon won't be running. So, doing a later run of "mdmon
/proc/mdstat" doesn't fully protect you.Last time I checked, Linux would not unmount the root filesystem. It just remounts it 'read-only'.
Is that going to change?
Yeah, I screwed up that part. However, it still syncs, and the mdmon process will still be dead.
Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to Bill NottinghamQuoted 10 lines · Bill Nottingham
OK, so:
1) kernel sends write request. If error....
2) <some error occurs>
3) kernel sends error to userspace
4) mdmon wakes up
5) mdmon decides where to record this
6) mdmon writes to super blocks
7) go to step one, hope you don't hit step 2 this time
Quoted 2 lines
This now means that reliable suspend and resume is completely impossible on RAID devices, just as it is on FUSE.
It's not clear from the context but I suppose you mean only FUSE root file systems (e.g. what Ubuntu/WUBI has on NTFS via NTFS-3G).
One of the solutions is to apply the same mechanism what swapfiles use. That avoids user space completely. The suspend information go to a dynamically (userspace involved) and a statically (no userspace involved) allocated space on the suspend device.
Regards, Szaka
--
NTFS-3G: http://ntfs-3g.org
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to Dan WilliamsQuoted 2 lines · Dan Williams
consider what extra tools the initramfs would need if we wanted to support an ntfs-3g rootfs.
The FUSE kernel module. Nothing else. Several distros do it.
Regards, Szaka
NTFS-3G: http://ntfs-3g.org
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 2/3] raid: external and internal metadata support
In reply to a message not in the archiveQuoted 26 lines · Dan Williams
Quoted 16 lines · Bill Nottingham
Quoted 2 lines · Dan Williams
Actually no, your not necessarily stuck with the mdmon from boot. In a pinch you could "mdmon
/proc/mdstat/".Not really.
You state:
Quoted 4 lines
One might say "just set the dirty bit, terminate, and wait for the mdmon in the rootfs to take over". The problem is that a disk could fail in this window, and this event needs to be handled before the kernel does anything else to the array.
...
Quoted 4 lines
The clean bit can be set as soon as the parity data is in sync with the data on the other drives. We typically wait for some period of write-inactivity to avoid needlessly touching the metadata after every write.
Quoted 8 lines
You shut down the machine. After a while, you get to the point where you're getting ready to unmount the filesystem. Since mdmon's running on it (if you started it post boot), you have to kill it. After that point, there are going to be writes (a final sync, if nothing else, when you unmount the filesystem.) And you won't be able to set any RAID metadata flags then, as the daemon won't be running. So, doing a later run of "mdmon
/proc/mdstat" doesn't fully protect you.
Quoted 7 lines
mdmon needs some coordination with the shutdown scripts to be kept alive until the rootfs is marked readonly... actually up until the point where the rootdev can be marked readonly.
If you take a look at Debian's killall implementation it has provisions to exclude fuse and other critical userspace process from killall. A similar exclusion is needed for mdmon.
It appears we have no solution for this yet in Fedora 12.
https://bugzilla.redhat.com/show_bug.cgi?id=496843
This bug has a similar request for network block devices that need a userspace process.
Warren Togami
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html