multipath device issue
Hi,
The multipath device node was not created during testing kdump on multipath target. dracut drop into emergency shell, I checked that 'multipathd -B' was running in background. Any idea?
Attached the full log of debug info.
Signature · 2 lines
Thanks Dave
Am 02.08.2012 07:09, schrieb Dave Young:
Quoted 8 lines
Hi,
The multipath device node was not created during testing kdump on multipath target. dracut drop into emergency shell, I checked that 'multipathd -B' was running in background. Any idea?
Attached the full log of debug info.
hmm.. I hope, this did not introduce a regression:
http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710e
Quoted 7 lines · Harald Hoyer
Am 02.08.2012 07:09, schrieb Dave Young:
Quoted 6 lines
Hi,
The multipath device node was not created during testing kdump on multipath target. dracut drop into emergency shell, I checked that 'multipathd -B' was running in background. Any idea?
Quoted 5 lines
Quoted 2 lines
Attached the full log of debug info.
hmm.. I hope, this did not introduce a regression:
http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710e
Revert this commit does not help in this case.
I tried manually run multipath in the emergency shell the /dev/mapper/mpathb will be created..
Signature · 2 lines
Thanks Dave
Am 02.08.2012 14:41, schrieb Dave Young:
Quoted 12 lines · Harald Hoyer
Quoted 11 lines
Am 02.08.2012 07:09, schrieb Dave Young:
Quoted 8 lines
Hi,
The multipath device node was not created during testing kdump on multipath target. dracut drop into emergency shell, I checked that 'multipathd -B' was running in background. Any idea?
Attached the full log of debug info.
Quoted 9 lines
Quoted 2 lines
hmm.. I hope, this did not introduce a regression:
http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710eRevert this commit does not help in this case.
I tried manually run
multipathin the emergency shell the/dev/mapper/mpathbwill be created..
oh.. can you retry with dracut-023? I think your version is missing this patch:
from
Quoted 23 lines · Harald Hoyer
Am 02.08.2012 14:41, schrieb Dave Young:
Quoted 12 lines · Harald Hoyer
Quoted 11 lines
Am 02.08.2012 07:09, schrieb Dave Young:
Quoted 8 lines
Hi,
The multipath device node was not created during testing kdump on multipath target. dracut drop into emergency shell, I checked that 'multipathd -B' was running in background. Any idea?
Attached the full log of debug info.
Quoted 9 lines
Quoted 2 lines
hmm.. I hope, this did not introduce a regression:
http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710eRevert this commit does not help in this case.
I tried manually run
multipathin the emergency shell the/dev/mapper/mpathbwill be created..
Quoted 7 lines
oh.. can you retry with dracut-023? I think your version is missing this patch:
from
I in fact use the dracut-023, but I found I have no 66-kpartx.rules, maybe I should upgrade udev as well.
Will have a test tomorrow.
Signature · 2 lines
Thanks Dave
Quoted 31 lines · Dave Young
Quoted 29 lines · Harald Hoyer
Am 02.08.2012 14:41, schrieb Dave Young:
Quoted 12 lines · Harald Hoyer
Quoted 11 lines
Am 02.08.2012 07:09, schrieb Dave Young:
Quoted 8 lines
Hi,
The multipath device node was not created during testing kdump on multipath target. dracut drop into emergency shell, I checked that 'multipathd -B' was running in background. Any idea?
Attached the full log of debug info.
Quoted 9 lines
Quoted 2 lines
hmm.. I hope, this did not introduce a regression:
http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710eRevert this commit does not help in this case.
I tried manually run
multipathin the emergency shell the/dev/mapper/mpathbwill be created..oh.. can you retry with dracut-023? I think your version is missing this patch:
from
Quoted 7 lines
Quoted 1 line
I in fact use the dracut-023, but I found I have no 66-kpartx.rules, maybe I should upgrade udev as well.
Will have a test tomorrow.
Hi, harald
It has nothing to do with the udev rules.
There's two issues:
1) 90multipath install multipath.sh to pre-trigger hook, but seems /dev/mapper/mpathX depends on /dev/dm-* population which will be created after udev trigger.
2) fstab-sys call wait_for_dev /dev/mapper/mpathb in initqueue, /dev/mapper/mpathb does not geting change to create, so dracut drop to emergency shell.
I'm not sure what's the right way to resolve this, move multipath.sh to pre-mount hook looks resolve the issue 1), how to resolve the issue 2)? comment out it works, but it looks not a right fix.
I'm not familiar with multipath, please correct me for anything wrong.
Signature · 2 lines
Thanks Dave
Am 03.08.2012 07:36, schrieb Dave Young:
Quoted 13 lines
Hi, harald
It has nothing to do with the udev rules.
There's two issues:
1) 90multipath installmultipath.shto pre-trigger hook, but seems/dev/mapper/mpathXdepends on/dev/dm-*population which will be created after udev trigger.2) fstab-sys call
wait_for_dev/dev/mapper/mpathbin initqueue,/dev/mapper/mpathbdoes not geting change to create, so dracut drop to emergency shell.
Quoted 6 lines
I'm not sure what's the right way to resolve this, move
multipath.shto pre-mount hook looks resolve the issue 1), how to resolve the issue 2)? comment out it works, but it looks not a right fix.I'm not familiar with multipath, please correct me for anything wrong.
I think it _has_ to do with the udev rules.
1. the multipathd daemon has to be started before udevd is started.
2. when udev is triggered, 62-multipath.rules should kick in and call "multipath" for every disk, which adds disks to the daemon, if they fit.
3. of course /etc/multipath.conf has to be configured correctly.
So, if you see /dev/mapper/mpathb be correctly created, when you start multpathd after all devices exist in /dev, then the 62-multipath.rules mechanism does not seem to work,
With the next systemd release "rd.udev.log-priority=debug" does work again and we might be able to debug this issue.
Quoted 18 lines · Harald Hoyer
Am 03.08.2012 07:36, schrieb Dave Young:
Quoted 17 lines
Hi, harald
It has nothing to do with the udev rules.
There's two issues:
1) 90multipath installmultipath.shto pre-trigger hook, but seems/dev/mapper/mpathXdepends on/dev/dm-*population which will be created after udev trigger.2) fstab-sys call
wait_for_dev/dev/mapper/mpathbin initqueue,/dev/mapper/mpathbdoes not geting change to create, so dracut drop to emergency shell.I'm not sure what's the right way to resolve this, move
multipath.shto pre-mount hook looks resolve the issue 1), how to resolve the issue 2)? comment out it works, but it looks not a right fix.
Quoted 5 lines
Quoted 2 lines
I'm not familiar with multipath, please correct me for anything wrong.
I think it _has_ to do with the udev rules.
What I should say is "It has nothing to do with the udev rules patches you mentioned previously" ;)
Quoted 6 lines
1. the multipathd daemon has to be started before udevd is started.
2. when udev is triggered, 62-multipath.rules should kick in and call "multipath" for every disk, which adds disks to the daemon, if they fit.
3. of course/etc/multipath.confhas to be configured correctly.
Quoted 8 lines
So, if you see
/dev/mapper/mpathbbe correctly created, when you start multpathd after all devices exist in /dev, then the 62-multipath.rules mechanism does not seem to work,With the next systemd release "rd.udev.log-priority=debug" does work again and we might be able to debug this issue.
Ok, will see if I can get something from the 62-multipath.rules. Thanks for the suggestion.
Signature · 2 lines
Thanks Dave
Quoted 7 lines
Quoted 6 lines
So, if you see
/dev/mapper/mpathbbe correctly created, when you start multpathd after all devices exist in /dev, then the 62-multipath.rules mechanism does not seem to work,
Quoted 8 lines
Quoted 2 lines
With the next systemd release "rd.udev.log-priority=debug" does work again and we might be able to debug this issue.
Ok, will see if I can get something from the 62-multipath.rules. Thanks for the suggestion.
Hi, here is the debug messages with latest systemd, 62-multipath fails at run 'socket:/org/kernel/dm/multipath_event':
---
Welcome to Fedora 17 (Beefy Miracle) dracut-023-2.fc18 (Initramfs)!
[ 0.799958] systemd[1]: No hostname configured. [ 0.801608] systemd[1]: Set hostname to <localhost>. [ 0.807834] systemd[1]: Cannot add dependency job for unit
systemd-journal-flush.service, ignoring: Unit systemd-journal-flush.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-journal-flush.service' for details.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Listening on udev Control Socket.
[ OK ] Listening on Journal Socket.
Starting Journal Service...
[ 0.825703] systemd-journald[44]: Fixed max_use=11.8M max_size=1.4M
min_size=64.0K keep_free=5.9M
[ 0.828702] systemd-journald[44]: Reserving 2695 entries in hash table.
[ 0.831147] systemd-journald[44]: Vacuuming...
[ OK ] Started Journal Service.
[ OK ] Reached target Sockets.
Starting Dracut cmdline hook...
Starting Setup Virtual Console...
[ OK ] Reached target Swap.
[ OK ] [ 0.850156] usb 1-1: new full-speed USB device number 2
using uhci_hcd
Reached target Local File Systems.
[ OK ] Started Setup Virtual Console.
[ OK ] Reached target System Initialization.dracut-cmdline[46]: Warning: Option 'KEYTABLE' is deprecated, use 'KEYMAP' instead.
dracut-cmdline[46]: Warning: Option 'SYSFONT' is deprecated, use 'FONT' instead.
[ OK ] Started Dracut cmdline hook.
Starting Dracut pre-udev hook...
[ OK ] Started Dracut pre-udev hook.
Starting udev Kernel Device Manager...
[ OK ] Started udev Kernel Device Manager.
Starting Dracut pre-trigger hook...
[ OK ] Reached target Basic System.
[ 0.950521] systemd-udevd[102]: starting version 188 [ 0.970090] device-mapper: multipath: version 1.3.0 loaded [ 1.003546] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 1.266732] usb 1-1: New USB device found, idVendor=0627, idProduct=0001 [ 1.269940] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5 [ 1.272323] usb 1-1: Product: QEMU USB Tablet [ 1.273889] usb 1-1: Manufacturer: QEMU 1.0 [ 1.275491] usb 1-1: SerialNumber: 42 [ 1.311638] input: QEMU 1.0 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input3 [ 1.315072] generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 1.0 QEMU USB Tablet] on usb-0000:00:01.2-1/input0 [ 1.344055] Refined TSC clocksource calibration: 2491.907 MHz. [ 1.349823] systemd-udevd[163]: failed to execute
'/usr/lib/udev/socket:/org/kernel/dm/multipath_event' 'socket:/org/kernel/dm/multipath_event': No such file or directory dracut-initqueue[144]: Warning: Could not boot. dracut-initqueue[144]: Warning: /dev/mapper/mpathb does not exist
Entering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs.
dracut:/#
Signature · 2 lines
Thanks Dave
Cc vivek
Quoted 86 lines · Dave Young
Quoted 7 lines
Quoted 6 lines
So, if you see
/dev/mapper/mpathbbe correctly created, when you start multpathd after all devices exist in /dev, then the 62-multipath.rules mechanism does not seem to work,Quoted 8 lines
Quoted 2 lines
With the next systemd release "rd.udev.log-priority=debug" does work again and we might be able to debug this issue.
Ok, will see if I can get something from the 62-multipath.rules. Thanks for the suggestion.
Hi, here is the debug messages with latest systemd, 62-multipath fails at run 'socket:/org/kernel/dm/multipath_event':
---
Welcome to Fedora 17 (Beefy Miracle) dracut-023-2.fc18 (Initramfs)![ 0.799958] systemd[1]: No hostname configured. [ 0.801608] systemd[1]: Set hostname to <localhost>. [ 0.807834] systemd[1]: Cannot add dependency job for unitsystemd-journal-flush.service, ignoring: Unit systemd-journal-flush.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-journal-flush.service' for details.
[ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ 0.825703] systemd-journald[44]: Fixed max_use=11.8M max_size=1.4M min_size=64.0K keep_free=5.9M [ 0.828702] systemd-journald[44]: Reserving 2695 entries in hash table. [ 0.831147] systemd-journald[44]: Vacuuming... [ OK ] Started Journal Service. [ OK ] Reached target Sockets. Starting Dracut cmdline hook... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] [ 0.850156] usb 1-1: new full-speed USB device number 2 using uhci_hcd Reached target Local File Systems. [ OK ] Started Setup Virtual Console. [ OK ] Reached target System Initialization.dracut-cmdline[46]: Warning: Option 'KEYTABLE' is deprecated, use 'KEYMAP' instead.
dracut-cmdline[46]: Warning: Option 'SYSFONT' is deprecated, use 'FONT' instead.
[ OK ] Started Dracut cmdline hook.
Starting Dracut pre-udev hook...
[ OK ] Started Dracut pre-udev hook.
Starting udev Kernel Device Manager...
[ OK ] Started udev Kernel Device Manager.
Starting Dracut pre-trigger hook...
[ OK ] Reached target Basic System.[ 0.950521] systemd-udevd[102]: starting version 188 [ 0.970090] device-mapper: multipath: version 1.3.0 loaded [ 1.003546] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 1.266732] usb 1-1: New USB device found, idVendor=0627, idProduct=0001 [ 1.269940] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5 [ 1.272323] usb 1-1: Product: QEMU USB Tablet [ 1.273889] usb 1-1: Manufacturer: QEMU 1.0 [ 1.275491] usb 1-1: SerialNumber: 42 [ 1.311638] input: QEMU 1.0 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input3 [ 1.315072] generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 1.0 QEMU USB Tablet] on usb-0000:00:01.2-1/input0 [ 1.344055] Refined TSC clocksource calibration: 2491.907 MHz. [ 1.349823] systemd-udevd[163]: failed to execute'/usr/lib/udev/socket:/org/kernel/dm/multipath_event' 'socket:/org/kernel/dm/multipath_event': No such file or directory dracut-initqueue[144]: Warning: Could not boot. dracut-initqueue[144]: Warning:
/dev/mapper/mpathbdoes not existEntering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs.
dracut:/#
Signature · 2 lines
Thanks Dave