Delay netroot mounting by 1 second

While testing in our production environment, I discovered
that the time between interface configuration and trying to mount the nfs root is too short if the dhcp-server provides an mtu. Having to set a mtu means that netroot executes 'ip link down', set's the mtu and the does
'ip link up' again.

This results in the mount error "No route to host", because
the network equipment has not had enough time to "shake hands". The patch below adds a sleep 1 after interface configuration, to give the equipment enough time.

Could anyone with a similar environment (and possibly with a netroot server reachable only via default route) test this with and without this patch to confirm?

Thank you,
Philippe

index f719d93..81b99f7 100755
@@ -88,7 +88,11 @@ for iface in $IFACES ; do
    . /tmp/net.$iface.up

done

-[ -e /tmp/net.$netif.gw ] && . /tmp/net.$netif.gw
+# Wait a bit until we continue to give the interface(s) time to
+# configure
+sleep 1
+
+[ -e /tmp/net.$netif.gw ] && . /tmp/net.$netif.gw [ -e /tmp/net.$netif.hostname ] && . /tmp/net.$netif.hostname
[ -e /tmp/net.$netif.resolv.conf ] && cp -f /tmp/net.$netif.resolv.conf /etc/resolv.conf

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Depending on your switch configuration, the time for the switch to start passing traffic again on link bounce couuld be anywhere from 0.1 seconds to 20 seconds...

Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quite correct. Up until now I've fared well with adding 1-5 second sleeps to my stuff. But yes it could even take longer. Any ideas how to solve this better?

Regards,
Philippe
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Post-boot, we (in Fedora) we check every half second until a configurable delay period (defualt: five seconds) has elapsed; if we don't have a link by then, we punt. Of course, you probably don't want configuration parameters in the initramfs.

Bill
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Any way to explicitly detect when the link is back?

Warren
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Disable spanning tree on all edge switch ports. :)

Or you can include ethtool and check every half second until a timeout to see if the link is back.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

With most switches, you'll get a link back almost immediately, but it will be in blocking mode and won't transmit your packets for 20-30 seconds.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Our test suite can simulate this by using attaching qemu's -net tap device to a bridge to talk to the other virtual machine instead of using -net socket mcast mode.

qemu -net tap[,vlan=n][,name=name][,fd=h][,ifname=name][,script=file][,downscript=dfile]

script=can be a script that tells it to attach to a particular bridge.

#!/bin/sh
export PATH=/sbin:/usr/sbin:$PATH
BRIDGEDEV=br0
ifconfig $1 0.0.0.0 up
brctl addif ${BRIDGEDEV} $1

brctl setfd <bridge> <time> sets the bridge’s ’bridge forward delay’ to <time> seconds.

Then you can use brctl to make the bridge's forward delay super long.

Warren Togami
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

<And a huge grin spreads over my face> Finally. I'm not alone anymore:-)
(I have been fighting the STP issue for years now.)

No, no, no. You don't want to add 'just' a sleep here because things don't work as expected. Certainly not in this case.

The problem here is that most programs don't / can't distinguish between an EHOSTUNREACH during startup (where we should retry as the link / connection might not be up yet) and an EHOSTUNREACH during normal operation (where we should return the error as this is a genuine error condition).

The only safe way here is to send a ping to the host port we're trying to connect to, and limit this with a timeout. And start up the command once the ping succeeds. Otherwise timeout and start error recovery.

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Yep, portfast is your friend if you have any control over your network...

Certainly we don't want to sleep if there is no reason to... why slow down the boot if the network admin has the switch nicely configured for us?

It may just be a matter of letting DHCP retry a sufficiency long time before we look at the results and try to boot. As long as we don't down the link after getting our information, the mount/login to the root device should proceed fairly quickly. If we do down the link, the blocking mode timer starts again and our very quick boot takes at least 40-60 seconds.

So, should we give dhclient 45-60 seconds to do its thing, and the admin can use ip= lines to disable interfaces we know aren't going to get a response?

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

What if he hasn't?

DHCP timeout already is 60 seconds. That's dhclient's default if you don't provide other options.

The problem only manifests itself it we have to correct the mtu after dhcp, which currently set's the interface down and up (paranoia I guess), or if we do static configuration.

For dhcp the solution could be easy: Either we ignore the mtu or don't down/up the interface to set it.

Disabling interfaces is not needed I think. If you know which interfaces are used/not used, you just tell dracut which interface(s) to use. All other interfaces will be ignored.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

That's where this part comes in:

Ok, then we just need to set the udev settle time to 60 seconds in the event of a netroot and avoid downing the link wherever possible. Given that timeout, the only problem cases are static configs and MTU changes on devices that cannot do that while UP'd.

We shouldn't ignore the MTU, and we generally have to down/up the interface to set it. Certainly, we should try to set it without downing the interface, and fall back to the down/set/up sequence if that fails. That would avoid further delay when good hardware/drivers allows it.

In the event of static config, or needing to down/up, I think it makes sense to try to ping the server for a period of time and see if we are able to communicate, before we try to get our root. You had even suggested something similar some time back with your original network scripts.

But that's not a 'sleep 60' just because we may happen be on a network with a long blocking state. Perhaps I misread what we're talking about and we're on the same page.

That's what I meant by disabling.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I'd prefer not use an absolute settle time but instead "test/wait" the queue. But that's a minor detail.

Hmmm... I'm not sure that a failing 'ip set eth0 mtu xy' is an indicator
that we have to down/set mtu/up the interface. But to be honest, the hardware/driver combinations I'm able to test on seem to be working without down/up.

Are there any other methods to decide if we can set the mtu on the fly?

Actually, if I'm not mistaken STP can have a maximum timeout of 90 seconds. Add some paranoia and you're at 100. :-)

But yes, sleep 60 is a really bad idea for impatient people like you or me.

On the other hand, I'd really prefer other solutions to "ping" (if there are any). Having used it myself doesn't mean I like it. First of all, needing another utility enlarges the initrd even more, there's a possible risk of having paranoid firewalls and we'd have to check the effective service-port which isn't possible with standard iputils-ping.

Regards,
Philippe
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

True, I'm just thinking that older udevs probably don't give us much options here.

As an aside, we should keep in mind that we shouldn't wait for all of the disks to become available if we don't have to -- I've got a few systems with quite a few LUNs to scan, but we only need one for the root. If it can find it and continue, then the rest can be done in parallel later.

Only by doing it. If it fails, it may be an indication that the MTU is invalid for the device, or that it cannot be changed while the device is up. If it is invalid, it fails then as well.

This method is a compromise; those with good hardware/drivers can use large MTUs without taking a penalty, and it still works for those less fortunate. Only the guys trying to set an invalid MTU pay a potentially needless time penalty, but I'm OK with that. :)

To check any of these, you'll need a separate utility. bash has /dev/tcp/IP/port redirection, but I doubt dash supports that. And even then you'll have some klunky trickery in the shell script to add a timeout to it. Nor will you be able to work with anything over UDP.

I think that checking an IP is reachable is sufficient to verify that traffic is flowing -- and we have choices there. We can use the server IP if it is local, and the gateway IP if not. If we're concerned about firewalls blocking ICMP, we could use 'arping' to send ARP requests. If there's a way to kick off an arp request with 'ip' or something in /proc or /sys, that'd avoid additional bloat, but I'm not aware of one. --
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

David Dillow wrote:
[snip]

Ha! You're brilliant. I completely forgot about ARP.

Including arping isn't such a bad idea. First of all, I tried using arping to solve the problem some years ago but busybox' arping didn't have the options required, iputils' arping does. Second, we should include it anyway since doing DAD in dhcp-script isn't such a bad idea.

Now, I'd use the gateway for detection if possible. But if it isn't available, how to guess the server? That logic currently resides in the root-handlers/-parsers and I'd prefer not to have the root-handler
do the checking.

Regards,
Philippe
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I've been thinking about this a bit, and I'd like to split the duties up a bit, as I'd like to know the server IPs needed for other reasons. My proposed split is that legacy handling -- ie nfsroot=blah to netroot=nfs:blah conversion -- is done as it is now, and have the root handler do syntax validation and stop, spit out the server IP(s), and do the mount, depending on options handed to it (--validate, --servers, and no options, respectively).

The sourced scripts could just call into their handler to do the final validation, and netroot would get rid of its loop as it can call the handler directly since it has the protocol field.

The only issue is handling the legacy NFS root-path options, everything else is expected to have nfs:/nfs4:/nbd:/iscsi: or what not. I don't think it would be the end of the world if the that handling lived in netroot itself, as it will be the only root-path legacy handler we have. --
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

David Dillow wrote:
[snip]

Hmmm... just so that I understand you correctly. Are you proposing to move the cmdline-parsers back into the root handlers and add arguments? If not maybe you could a more detailed example of your idea?

Uhhh... which loop are you talking about?

If possible, netroot should remain as generic as possible. You never know what the future holds. --
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes, something like that.

parse-nfsroot.sh, parse-nbdroot.sh, and parse-iscsiroot.sh would only exist to convert legacy command line arguments to the netroot={nfs,nbd,iscsi}: format. They would not validate the netroot= format.

As part of the command line parsing, the appropriate handler would be called for netroot to check its validity, such as:

/cmdline/98-check-netroot.sh:

if [ -n "$netroot" ]; then
[ -e /sbin/${netroot%%:*}-root ] || die "No root handler for protocol ${netroot%%:*}" /sbin/${netroot%%:*}-root --validate "$netroot"
case $? in
0) rootok=1 ;; # All's good
1) DHCPORSERVER=1 ;; # No server specified
*) die "Invalid argument to root handler ${netroot%%:*}" ;;
esac
fi

Checking the validity of the DHCP root-option becomes (modulo NFS legacy support):

[ -e /sbin/${rootpath%%:*}-root ] || die "No root handler for protocol ${rootpath%%:*}" /sbin/${rootpath%%:*}-root --validate "$rootpath" ||

	die "Invalid DHCP root-option for ${rootpath%%:*}"

Later, we can decide which IP to arping:

server=$(/sbin/${netroot%%:*}-root --server "$netroot") destip=$gateway
is_ip_local $server $myip $netmask && destip=$server

while ! arping ... $server; do

    # count here to break out, etc.
    sleep 1
done

The variable names will likely change a bit, and the messages will need to be massaged, but we have a need to decode the netroot= arguments three times -- once for validation, once for extracting the server, and once for the actual mounting. It makes sense to have the same parser for all three uses. I think it would also clean things up to separate the legacy handling from the supported mechanisms.

I haven't started coding this up; I wanted to try to get some consensus first. NFSv4 would get its own handler, which I think makes sense -- I'll probably break it out into it's own module anyway.

/sbin/netroot:
# Source netroot hooks before we start the handler source_all netroot

That would become the validity check above.

As a general design principle, I tend to agree with you. However, the only root-path we support that doesn't match the form protocol:arguments is the [serverip:]/path syntax of NFS. I think it makes sense to either pull that into netroot.

Of course, I just thought of an alternative solution -- maybe we could pass both netroot and root-path in to the root handler, and it can do the right thing. The initial cmdline parse would pass in "" for root-path, or perhaps find another way to indicate we don't have the root-path yet.

It's a wart, but I think we can get it cleaned up.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

[ Hannes, Victor, Bill: I'm loathe to cut cc's normally, but if this discussion doesn't interest you, let me know privately and I'll trim you from my future messages on it. ]

Hmm, that source_all is vestigial, I think.

This is the loop I'm thinking of:

# Figure out the handler for root=dhcp by recalling all netroot cmdline

# handlers
if [ "$netroot" = "dhcp" ] ; then
[snip]
    for f in ./cmdline/90*.sh; do
        [ -f "$f" ] && . "$f";
    done
fi

This could go away modulo the NFS issue I mentioned.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Just to be sure I'm coming across clearly -- I want us to work properly if the admin hasn't configured the network nicely for us, but I do not want us to slow down at all when he has.

My networks will be configured correctly, and I'm an impatient bastard. I don't want to wait if I don't have to. But not everyone will have the luxury of control of their network.
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

On a network where I don't have control over the switch (some Cisco model) and people connect laptops, I've seen random cases of DHCP failing. That's because the STP can take 60 seconds and DHCP gives up just when the link becomes usable... For the workstations connected on another network where STP is also enabled, I've experimented with various solutions, the one using ping to the gateway (with a timeout of 100 seconds) worked best and was simplest for me to maintain.

No, it's not paranoia. Maybe things have changed in the meantime, but some years ago when I was involved in the Linux network stack some drivers couldn't change the MTU on the fly due to internal limitations. The only possibility to change it would be to bring the interface down then up. Not all drivers are limited this way, though. Maybe it's possible to try to set the MTU, check whether it has succeeded and only if it failed bring the interface down then up; don't know anymore if a failed MTU setting would make ifconfig return something... maybe just reading the MTU again and compare with desired would work too.

Bogdan Costescu

IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8240, Fax: +49 6221 54 8850
E-mail: bogdan.costescu-hEciA7+sKtudPOQpRHQ53DeJuz7u0hKX@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe initramfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

You can say that loud. Over the years I've tried various solutions, from sleeping, to catching states via netlink, to guessing STP timeouts (on newer switches probability of RSTP-like behaviour is 95%) to your mentioned ping...

Sleep isn't a good solution, I agree there. But it's simple. Hence the RFT to see if someone can construct a real life case where it actually isn't enough.

Yes. If we know that 1) We're using the correct interface and
2) provided server data is correct. Otherwise we just unnecessary lengthen to boot process.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html