RFC: InfiniBand hotplug agent
I merged new feature for the InfiniBand stack in kernel 2.6.17. There will be /sys/class/infiniband/<dev>/node_desc files. This can be used by userspace to change the IB node description, which is retrieved by various network management stuff. Putting the hostname in the node description makes things a lot nicer, since it provides a link from the IB network view back to the real world.
Right now on my systems, I'm using the script below, which takes the output of hostname and adds it to the node description. I put this in /lib/udev/infiniband.agent along with a udev rule like
SUBSYSTEM="infiniband", RUN+="infiniband.agent"
to run it.
So I have two questions:
- Is this script reasonable? I'm by no means a hotplug expert, and I'd like to get critiques and suggestions for how to write a proper agent.
- What's the right way to get this into distributions? File wishlist bugs against udev with the rule and the script included?
Thanks,
Roland
Here's the script:
#!/bin/sh -e
#
# infiniband agent #
cd /lib/udev/
. ./hotplug.functions
if [ "$ACTION" = "add" ]; then
DESC_PATH="/sys/$DEVPATH/node_desc"
if [ -e "$DESC_PATH" ]; then
DESC="`hostname` (`cat $DESC_PATH`)"
echo -n $DESC > $DESC_PATH
fifi
exit 0
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 2 lines · Roland Dreier
- What's the right way to get this into distributions? File wishlist bugs against udev with the rule and the script included?
Yes.
Quoted 2 lines
cd
/lib/udev/
. ./hotplug.functions
I think that this is debian-specific, and you are not using it anyway.
Quoted 1 line
if [ "$ACTION" = "add" ]; then
Remove this if statement and use such a rule instead:
ACTION="add", SUBSYSTEM="infiniband", RUN+="infiniband.agent"
#!/bin/sh -e
DESC_PATH="/sys/$DEVPATH/node_desc"
# is this actually needed? can node_desc really not exist? [ -e "$DESC_PATH" ] || exit 0
# by using read instead of cat you save a fork+exec read OLD_DESC < $DESC_PATH DESC="`hostname` ($OLD_DESC)" echo -n $DESC > $DESC_PATH
But probably using hostname(1) in a RUN rule is not such a great idea, because at boot time the script will be run long before the system hostname will have been set by an init script.
Signature · 13 lines
ciao, Marco ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642 _______________________________________________ Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net [email protected] https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Thanks for the comments.
Marco> Remove this if statement and use such a rule instead:
Marco> ACTION="add", SUBSYSTEM="infiniband", RUN+="infiniband.agent"
Good point.
Marco> # is this actually needed? can node_desc really not exist? Marco> [ -e "$DESC_PATH" ] || exit 0
Certainly on kernels prior to 2.6.17 it won't be there...
Marco> # by using read instead of cat you save a fork+exec read
Another good point.
Marco> But probably using hostname(1) in a RUN rule is not such a Marco> great idea, because at boot time the script will be run Marco> long before the system hostname will have been set by an Marco> init script.
Hmm, that's a problem, and indeed if I let the driver be loaded on boot, I end up with "(none)" as the hostname (I usually blacklist my IB drivers, because I do a lot of driver development and it's a pain when the system crashes on boot because of bugs I wrote).
Do you think using /etc/hostname instead (on Debian at least) is a palatable solution? It seems this will end up being distro specific unfortunately... Here's what I just tested (along with your ACTION="add" suggestion), and it worked well even when the driver is loaded at boot time:
#!/bin/sh -e
DESC_PATH="/sys/$DEVPATH/node_desc"
[ -e "$DESC_PATH" ] || exit 0
read OLD_DESC < $DESC_PATH
read HOSTNAME < /etc/hostname
echo -n "$HOSTNAME ($OLD_DESC)" > $DESC_PATH
exit 0
Do you think this is OK for Debian? Obviously I'll have to come up with something else for Gentoo, Fedora, etc.
Thanks,
Roland
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 10 lines · Roland Dreier
Hmm, that's a problem, and indeed if I let the driver be loaded on boot, I end up with "(none)" as the hostname (I usually blacklist my IB drivers, because I do a lot of driver development and it's a pain when the system crashes on boot because of bugs I wrote).
Do you think using
/etc/hostnameinstead (on Debian at least) is a palatable solution? It seems this will end up being distro specific unfortunately... Here's what I just tested (along with your ACTION="add" suggestion), and it worked well even when the driver is loaded at boot time:
/etc is not even around at early boot time (think module loading from initramfs/initrd), which is what the "enterprise" distros do.
Quoted 9 lines
#!/bin/sh -e
DESC_PATH="/sys/$DEVPATH/node_desc"
[ -e "$DESC_PATH" ] || exit 0
read
OLD_DESC< $DESC_PATH
read HOSTNAME </etc/hostname
echo -n "$HOSTNAME ($OLD_DESC)" > $DESC_PATH
Why does your drivers care about the hostname? The kernel already knows this information anyway...
thanks,
greg k-h
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 4 lines · Marco d'Itri
But probably using hostname(1) in a RUN rule is not such a great idea, because at boot time the script will be run long before the system hostname will have been set by an init script.
I got sufficiently amused by the three or four things that failed because of this that I moved the hostname setting to almost the top of the boot sequence in Ubuntu, with no ill effects.
It's a tiny operation, after all.
Scott
Signature · 2 lines
Scott James Remnant [email protected]
Greg> Why does your drivers care about the hostname? The kernel Greg> already knows this information anyway...
As I explained before, this script puts the hostname in the InfiniBand "node description," which is retrieved by various IB network management tools. It makes the output of these tools much easier to read, since you can see things like "node1 (CA)" and "node2 (CA)" in the network management tools instead of just two copies of "CA".
Yes, I could modify the driver to look at system_utsname and pick a way to format that. But that suffers from the problem of putting policy in the kernel (who picks the format for node descriptions?), although that's minor. And we still have the problem of what to do before the hostname is set -- put "(none)" in our node description?
Anyway, do you think just having this node description setting in the kernel is the best way to go?
Thanks,
Roland
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 8 lines · Roland Dreier
Greg> Why does your drivers care about the hostname? The kernel Greg> already knows this information anyway...
As I explained before, this script puts the hostname in the InfiniBand "node description," which is retrieved by various IB network management tools. It makes the output of these tools much easier to read, since you can see things like "node1 (CA)" and "node2 (CA)" in the network management tools instead of just two copies of "CA".
Ick, the kernel is then printing that data back out somewhere else? I really don't want to know... :)
Quoted 8 lines
Yes, I could modify the driver to look at
system_utsnameand pick a way to format that. But that suffers from the problem of putting policy in the kernel (who picks the format for node descriptions?), although that's minor. And we still have the problem of what to do before the hostname is set -- put "(none)" in our node description?Anyway, do you think just having this node description setting in the kernel is the best way to go?
I think that your proposed solution will not work on the current round of "enterprise" distros, so I don't think it is a real "solution".
As for a proposed one, I have no idea, sorry.
greg k-h
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel