if/how to dictate IB device name per PCI BDF
Hi Roland,
We got a report that on a system with multiple (say two) ConnectX HCAs, its possible for the order of device probing to be different across simple reboots, that is sometimes the device with PCI BDF X is probed 1st and gets to be IB device mlx4_0 and some other-timesthe device with BDF Y gets to be mlx4_0 and X becomes mlx4_1, and so on, which for some reason creates a hassle for them.
I don't fully understand how the PCI scan order can change between different reboots, but have the feeling its possible (in black box manner, the reports claims that).
Thinking about this a bit, will it be possible to provide a udev rulethat can dictate to the IB core what name / suffix digit to assign for a device with certain BDF?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Or Gerlitz
Thinking about this a bit, will it be possible to provide a udev rulethat can dictate to the IB core what name / suffix digit to assign for a device with certain BDF?
FWIW, I reflected on this once (I belive it was for the netlink discussion).. Allowing for a device to be renamed creates a serious problem for RDMA since there isn't a stable 'if-index' like way for software to refer to it.
The udev rules all rely on a means to rename the device once the kernel has created it.
For this reason, all the software I've built uses the port GID to ID the resource for bind/etc and strongly discourages use of the RDMA device name.
The ipoib devices should already be renamable via udev rules, though has anyone checked that udev doesn't have a problem with the long 'MAC'?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · On Fri, 12 Oct 2012 16:43:32 -0600 Jason Gunthorpe…
Quoted 3 lines · Or Gerlitz
Thinking about this a bit, will it be possible to provide a udev rulethat can dictate to the IB core what name / suffix digit to assign for a device with certain BDF?
FWIW, I reflected on this once (I belive it was for the netlink discussion).. Allowing for a device to be renamed creates a serious problem for RDMA since there isn't a stable 'if-index' like way for software to refer to it.
Quoted 6 lines
The udev rules all rely on a means to rename the device once the kernel has created it.
For this reason, all the software I've built uses the port GID to ID the resource for bind/etc and strongly discourages use of the RDMA device name.
IMHO, from a user perspective using GID's or GUIDs is just too hard. While the port GID is unique across reboots in practice with 1000's of nodes in a cluster names are much more useful and, at least here at LLNL, nobody refers to HCA's by GUID. They refer to them by hostname/card name. This is especially true when HCA's get swapped out for maintenance.
Furthermore, how would you propose MPI selects a card when the job is launched on different nodes, chosen by a scheduler, from run to run?
Unfortunately, I don't know the details of the "serious problem" you describe above but it seems that a udev solution to renaming things would be worth trying to solve from a users perspective.
Ira
Quoted 9 lines
The ipoib devices should already be renamable via udev rules, though has anyone checked that udev doesn't have a problem with the long 'MAC'?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 9 lines
Ira Weiny Member of Technical Staff Lawrence Livermore National Lab 925-423-8008 [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 12 lines · Ira Weiny
Quoted 11 lines
FWIW, I reflected on this once (I belive it was for the netlink discussion).. Allowing for a device to be renamed creates a serious problem for RDMA since there isn't a stable 'if-index' like way for software to refer to it.
The udev rules all rely on a means to rename the device once the kernel has created it.
For this reason, all the software I've built uses the port GID to ID the resource for bind/etc and strongly discourages use of the RDMA device name.
Quoted 6 lines
IMHO, from a user perspective using GID's or GUIDs is just too hard. While the port GID is unique across reboots in practice with 1000's of nodes in a cluster names are much more useful and, at least here at LLNL, nobody refers to HCA's by GUID. They refer to them by hostname/card name. This is especially true when HCA's get swapped out for maintenance.
Textual names (ie a name service) for GID alleviate alot of that pain, but really, how often do you need to use device RDMA name??
Assuming GID port selection is available, the only case that requires the full GID is when you have multiple HCAs and ports are active on the *same subnet*.
Otherwise you can refer to the port by: port number, by GID prefix or by 'first found active port' without ever involving the rdma device name.
I never did it, but it did occur to me to support the IBoIB device name and IP addresses as a port specifier..
Quoted 3 lines
Furthermore, how would you propose MPI selects a card when the job is launched on different nodes, chosen by a scheduler, from run to run?
Three things spring to mind:
- Choose an active port at random that matches a set of subnet prefixes
- Choose an active port at random from a globally known list of job-acceptable GIDs
- Choose an active port #X that matches a set of subnet prefixes, from a random device
Relying on rdma device name for MPI wouldn't work in a heterogeneous environment where different nodes have a different 'correct' device name.
Quoted 3 lines
Unfortunately, I don't know the details of the "serious problem" you describe above but it seems that a udev solution to renaming things would be worth trying to solve from a users perspective.
Renaming the device changes the name of all the sysfs directories, so any 'handle' is invalidated at rename time. All udev can do for you is trigger the rename call, you still have to somewhat sanely enable renaming in the kernel.
And it is not entirely theoretical, with threaded startup and things like systemd, it is very plausible that the rename could hit in the middle of something like opensm starting up and create a flakey system. Particularly since we'd want to see opensm's startup triggered by directory creation in sysfs.
I think it would be great to be able to do the rename, but to do it properly things need to be more like the net stack - netlink for configuration/discovery, and a stable ifindex for use in all netlink messages.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html