Skip to contentSkip to conversation

ipath - char devices for diagnostics and lightweight subnet management

PATCH Roland Dreier 4 participants 27 messages Mar 9, 2006 – Mar 10, 2006 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Bryan> The ipath_sma.c file supports a lightweight userspace Bryan> subnet management agent (SMA). This is used in deployments Bryan> (such as HPC clusters) where a full Infiniband protocol Bryan> stack is not needed.

I've never understood what forces you to maintain two separate SMAs. Why can't you pick one of the two SMAs and use that unconditionally?

- R.

#2 lore raw
Quoted 2 lines · Roland Dreier

I've never understood what forces you to maintain two separate SMAs. Why can't you pick one of the two SMAs and use that unconditionally?

Three reasons.

* OpenSM wasn't usable when we wrote our SMA. We have customers using ours now, so we have to support it.
* Our SMA does some setup for the layered ethernet emulation driver.
* Our SMA works without an IB stack of any kind present.

<b

#3 lore raw
Quoted 4 lines

Three reasons.

* OpenSM wasn't usable when we wrote our SMA. We have customers using ours now, so we have to support it.

Huh? What does OpenSM working or not have to do with the SMA?

Quoted 3 lines

* Our SMA does some setup for the layered ethernet emulation driver.
* Our SMA works without an IB stack of any kind present.

That's fine. So then I guess the question is, why can't you use your SMA all the time?

And does that mean that the verbs SMA doesn't support ethernet emulation, so you can't use ethernet emulation and verbs at the same time?

- R.

#4 lore raw
Quoted 2 lines · Roland Dreier

That's fine. So then I guess the question is, why can't you use your SMA all the time?

We do. It coexists with OpenSM if OpenSM is present.

	<b
#5 lore raw

Roland> That's fine. So then I guess the question is, why can't Roland> you use your SMA all the time?

Bryan> We do. It coexists with OpenSM if OpenSM is present.

So can we kill the other SMA?

- R.

#6 lore raw
In reply to Bryan O'Sullivan

Quoting r. Bryan O'Sullivan <[email protected]>:

Quoted 11 lines

Subject: Re: [PATCH 9 of 20] ipath - char devices for diagnostics and lightweight subnet management

Quoted 2 lines · Roland Dreier

I've never understood what forces you to maintain two separate SMAs. Why can't you pick one of the two SMAs and use that unconditionally?

Three reasons.

* OpenSM wasn't usable when we wrote our SMA. We have customers using ours now, so we have to support it.

Presumably you mean the ib_mad SMA - OpenSM is not an SMA. I don't understand this. You don't talk to an SMA directly - its jobs is to receive and send management packets that the card gets from a subnet manager. So what do customers care which SMA implementation is used, as long as it formats the management packets correctly?

If you want to extend the SMA in non-standard ways, you can snoop and send management packets by loading ib_umad.

Quoted 2 lines

* Our SMA does some setup for the layered ethernet emulation driver.

You are doing this from userspace, right? But you can already send/receive MADs from userspace by loading ib_umad. What is there that is not sufficient for your purposes?

By the way, this might also be a clean way for you to get at the port info node info and port counters atomically like you wanted to in another thread:
post a management packet to the local device, get a packet back. No need for extra sysfs files.

Quoted 1 line

* Our SMA works without an IB stack of any kind present.

The stack is pretty slim, AFAIK it mostly consists of an SMA implementation. So where's the benefit in avoiding it? Just link against ib_mad, it will get loaded atomatically.

Signature · 2 lines
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies
#7 lore raw
Quoted 4 lines · Michael S. Tsirkin
Quoted 2 lines

* OpenSM wasn't usable when we wrote our SMA. We have customers using ours now, so we have to support it.

Presumably you mean the ib_mad SMA - OpenSM is not an SMA.

Yes, I already mentioned that I got my terms swapped in another message.

Quoted 3 lines

So what do customers care which SMA
implementation is used, as long as it formats the management packets correctly?

Many, perhaps most right now, of our customers don't have a full IB stack loaded. That's why we have this small userspace SMA.

	<b
#8 lore raw
In reply to a message not in the archive
Quoted 5 lines
(quoted diff) +5 -0
+static int ipath_sma_open(struct inode *in, struct file *fp)
+{
+	int s;
+
+	if (ipath_sma_alive) {

Review

Quoted 10 lines
(quoted diff) +10 -0
+		ipath_dbg("SMA already running (pid %u), failing\n",
+			  ipath_sma_alive);
+		return -EBUSY;
+	}
+
+	for (s = 0; s < atomic_read(&ipath_max); s++) {
+		struct ipath_devdata *dd = ipath_lookup(s);
+		/* we need at least one infinipath device to be initialized. */
+		if (dd && dd->ipath_flags & IPATH_INITTED) {
+			ipath_sma_alive = current->pid;

It seems there's a window here where two processes can both pass the if (ipath_sma_alive) test and then proceed to step on each other.

- R.

#9 lore raw
Quoted 2 lines · Roland Dreier

It seems there's a window here where two processes can both pass the if (ipath_sma_alive) test and then proceed to step on each other.

Yep, this is a real race, albeit incredibly unlikely. I just turned ipath_sma_alive into an atomic_t, and wrapped the open/close code in spinlocks.

	<b
#10 lore raw

Bryan> Yep, this is a real race, albeit incredibly unlikely. I Bryan> just turned ipath_sma_alive into an atomic_t, and wrapped Bryan> the open/close code in spinlocks.

How does making it an atomic_t help? I think you're only going to be using atomic_set() and atomic_read(), and atomic_t doesn't provide anything in that case.

- R.

#11 lore raw
In reply to a message not in the archive
Quoted 7 lines
(quoted diff) +7 -0
+static int ipath_sma_release(struct inode *in, struct file *fp)
+{
+	int s;
+
+	ipath_sma_alive = 0;
+	ipath_cdbg(SMA, "Closing SMA device\n");
+	for (s = 0; s < atomic_read(&ipath_max); s++) {

Review

Quoted 10 lines
(quoted diff) +10 -0
+		struct ipath_devdata *dd = ipath_lookup(s);
+
+		if (!dd || !(dd->ipath_flags & IPATH_INITTED))
+			continue;
+		*dd->ipath_statusp &= ~IPATH_STATUS_SMA;
+		if (dd->verbs_layer.l_flags & IPATH_VERBS_KERNEL_SMA)
+			*dd->ipath_statusp |= IPATH_STATUS_OIB_SMA;
+	}
+	return 0;
+}

Similarly what protects against another process opening the device right after the ipath_sma_alive = 0 setting, but before you do all the cleanup that's after that?

And what protects against a hot unplug of a device after the test of s against ipath_max?

- R.

#12 lore raw
Quoted 3 lines · Roland Dreier

Similarly what protects against another process opening the device right after the ipath_sma_alive = 0 setting, but before you do all the cleanup that's after that?

This is fixed by the stuff I just did in response to your earlier message.

Quoted 2 lines

And what protects against a hot unplug of a device after the test of s against ipath_max?

We don't support hotplugged devices at the moment. If you're asking whether an rmmod at the wrong time could cause something bad to happen, I don't *think* so.

	<b
#13 lore raw

Bryan> We don't support hotplugged devices at the moment. If Bryan> you're asking whether an rmmod at the wrong time could Bryan> cause something bad to happen, I don't *think* so.

How do you stop someone from hot plugging a PCIe device?

- R.

#14 lore raw
Quoted 3 lines · Roland Dreier

Bryan> We don't support hotplugged devices at the moment.

How do you stop someone from hot plugging a PCIe device?

You say "we don't support this yet" somewhere in big letters. The chips and cards support hotplug electrically and logically. We just haven't had time yet to do the driver support work, and won't for a while.

	<b
#15 lore raw
In reply to Bryan O'Sullivan
Quoted 6 lines · Bryan O'Sullivan
Quoted 3 lines · Roland Dreier

Similarly what protects against another process opening the device right after the ipath_sma_alive = 0 setting, but before you do all the cleanup that's after that?

Quoted 7 lines

This is fixed by the stuff I just did in response to your earlier message.

Quoted 2 lines

And what protects against a hot unplug of a device after the test of s against ipath_max?

We don't support hotplugged devices at the moment.

Why not? Your cards can't be placed in a machine that supports PCI Hotplug (or PCI-E hotplug)? You can't really tell users that (no matter how often I have wished I could...)

thanks,

greg k-h

#16 lore raw
Quoted 4 lines · Greg KH
Quoted 1 line

We don't support hotplugged devices at the moment.

Why not? Your cards can't be placed in a machine that supports PCI Hotplug (or PCI-E hotplug)?

No, the driver and userspace code doesn't support it yet. That's all.

Quoted 2 lines

You can't really tell users that (no matter
how often I have wished I could...)

I don't expect this to be a practical problem. We're planning to add hotplug support to the driver once we have some cycles free.

	<b
#17 lore raw
Quoted 7 lines · Bryan O'Sullivan
Quoted 4 lines · Greg KH
Quoted 1 line

We don't support hotplugged devices at the moment.

Why not? Your cards can't be placed in a machine that supports PCI Hotplug (or PCI-E hotplug)?

Quoted 7 lines

No, the driver and userspace code doesn't support it yet. That's all.

Quoted 2 lines

You can't really tell users that (no matter
how often I have wished I could...)

I don't expect this to be a practical problem. We're planning to add hotplug support to the driver once we have some cycles free.

Ugh, that means it's never going to be there.

All new PCI drivers have the requirement that they work properly in hotplug systems, as they should follow the PCI core api. If not, odds are they will not be accepted into the tree :(

thanks,

greg k-h

#18 lore raw
Quoted 8 lines · Greg KH
Quoted 2 lines

I don't expect this to be a practical problem. We're planning to add hotplug support to the driver once we have some cycles free.

Ugh, that means it's never going to be there.

All new PCI drivers have the requirement that they work properly in hotplug systems, as they should follow the PCI core api. If not, odds are they will not be accepted into the tree :(

Okay, maybe we're talking at cross purposes here. We do follow the PCI core API. We have a __devinit probe and __devexit remove routine, a MODULE_DEVICE_TABLE, the kernel generates hotplug events when a device is detected or the driver is unloaded, and so on.

I *assumed* that there was something more that we would need to do in order to support real hotplug of actual physical cards, but now that I look more closely, it doesn't appear that there is. At least, there's nothing in Documentation/pci.txt or LDD3 that indicates to me that we ought to be doing more.

Am I missing something?

	<b
#19 lore raw
Quoted 16 lines · Bryan O'Sullivan
Quoted 8 lines · Greg KH
Quoted 2 lines

I don't expect this to be a practical problem. We're planning to add hotplug support to the driver once we have some cycles free.

Ugh, that means it's never going to be there.

All new PCI drivers have the requirement that they work properly in hotplug systems, as they should follow the PCI core api. If not, odds are they will not be accepted into the tree :(

Okay, maybe we're talking at cross purposes here. We do follow the PCI core API. We have a __devinit probe and __devexit remove routine, a MODULE_DEVICE_TABLE, the kernel generates hotplug events when a device is detected or the driver is unloaded, and so on.

Quoted 7 lines

I *assumed* that there was something more that we would need to do in order to support real hotplug of actual physical cards, but now that I look more closely, it doesn't appear that there is. At least, there's nothing in Documentation/pci.txt or LDD3 that indicates to me that we ought to be doing more.

Am I missing something?

Nope, that's all that you need to do. Your driver will be notified that the device will be going away by calling the disconnect function. So great, nothing needs to be done :)

Oh, and you can test this out if you don't have a pci hotplug system by using the fakephp driver and disconnecting your device that way.

thanks,

greg k-h

#20 lore raw
Quoted 2 lines · Greg KH

Oh, and you can test this out if you don't have a pci hotplug system by using the fakephp driver and disconnecting your device that way.

That's good to know, thanks.

	<b
#21 lore raw
In reply to Bryan O'Sullivan

Bryan> I *assumed* that there was something more that we would Bryan> need to do in order to support real hotplug of actual Bryan> physical cards, but now that I look more closely, it Bryan> doesn't appear that there is. At least, there's nothing in Bryan> Documentation/pci.txt or LDD3 that indicates to me that we Bryan> ought to be doing more.

Bryan> Am I missing something?

No, the only problems are with the way the various pieces of your drivers refer to devices by index. There are obvious races such as

Quoted 5 lines
(quoted diff) +5 -0
+int __init ipath_verbs_init(void)
+{
+	int i;
+
+	number_of_devices = ipath_layer_get_num_of_dev();

Review

Quoted 10 lines
(quoted diff) +10 -0
+	i = number_of_devices * sizeof(struct ipath_ibdev *);
+	ipath_devices = kmalloc(i, GFP_ATOMIC);
+	if (ipath_devices == NULL)
+		return -ENOMEM;
+
+	for (i = 0; i < number_of_devices; i++) {
+		struct ipath_devdata *dd;
+		int ret = ipath_verbs_register(i, ipath_ib_piobufavail,
+					       ipath_ib_rcv, ipath_ib_timer,
+					       &dd);

suppose number_of_devices gets set to 5 but by the time you call ipath_verbs_register(5,...), the 5th device has been unplugged?

Also you only do this when the module is loaded, so you won't handle devices that are hot-plugged later. And I don't see anything that would handle hot unplug either.

Pretty much any use of ipath_max is probably broken by hot plug.

- R.

#22 lore raw
Quoted 2 lines · Roland Dreier

No, the only problems are with the way the various pieces of your drivers refer to devices by index.

OK. What's a safe way to iterate over the devices in the presence of hotplug, then? I assume it's list_for_each_mumble; I just don't know what mumble is :-)

Quoted 2 lines

Also you only do this when the module is loaded, so you won't handle devices that are hot-plugged later.

No, ipath_max is updated any time a probe routine is called.

Quoted 2 lines

And I don't see anything that
would handle hot unplug either.

What would this anything look like, if I were hoping for an example to emulate? There's nothing in LDD3 about this, so I'm kind of in the dark.

	<b
Signature · 1 line
Bryan O'Sullivan <[email protected]>
#23 lore raw
Quoted 8 lines · Bryan O'Sullivan
Quoted 2 lines · Roland Dreier

No, the only problems are with the way the various pieces of your drivers refer to devices by index.

OK. What's a safe way to iterate over the devices in the presence of hotplug, then? I assume it's list_for_each_mumble; I just don't know what mumble is :-)

You keep an internal list of devices, if you really need to do such a thing.

Quoted 11 lines
Quoted 2 lines

Also you only do this when the module is loaded, so you won't handle devices that are hot-plugged later.

No, ipath_max is updated any time a probe routine is called.

Quoted 2 lines

And I don't see anything that
would handle hot unplug either.

What would this anything look like, if I were hoping for an example to emulate? There's nothing in LDD3 about this, so I'm kind of in the dark.

It's just the "disconnect" PCI function being called, which can happen at any time.

thanks,

greg k-h

#24 lore raw
Quoted 2 lines · Greg KH

You keep an internal list of devices, if you really need to do such a thing.

OK. I do think we need it, because we have a dozen or so cases where we do "iterate over all of the known devices".

Quoted 2 lines

It's just the "disconnect" PCI function being called, which can happen at any time.

Thanks.

	<b
#25 lore raw
In reply to Bryan O'Sullivan

Bryan> OK. What's a safe way to iterate over the devices in the Bryan> presence of hotplug, then? I assume it's Bryan> list_for_each_mumble; I just don't know what mumble is :-)

You need something that takes a reference to each device while you're looking at it, like for_each_pci_dev(). But in general iterating through devices is usually the wrong thing to do, because devices can come and go in the middle of your loop. It's better to be driven by the add and remove callbacks.

Bryan> No, ipath_max is updated any time a probe routine is Bryan> called.

Yes, that's true. (BTW, what does making ipath_max an atomic_t get you? The updates are protected by a lock anyway). But I was talking about the code in ipath_verbs_init(), which is the only place you call ipath_verbs_register() that I could find. You make one pass through the devices that are present when ipath_verbs_init() is called at module load time, and any devices that get added later are missed.

Similarly, if a device is unplugged while the verbs module is loaded, there's no notification from the core driver of that, and you'll go ahead and do ipath_verbs_unregister() on a device that is long gone when you get to ipath_verbs_cleanup().

- R.

#26 lore raw
Quoted 6 lines · Roland Dreier

Bryan> OK. What's a safe way to iterate over the devices in the Bryan> presence of hotplug, then? I assume it's Bryan> list_for_each_mumble; I just don't know what mumble is :-)

You need something that takes a reference to each device while you're looking at it, like for_each_pci_dev().

OK, thanks.

It seems like the thing to do to be fully safe might be to have ipath_get() (just rename ipath_lookup) and ipath_put() functions. Embed a kref inside ipath_devdata, and have ipath_dev_get increment the ref count on both the ipath_devdata and the pci_dev.

Is that sane, or am I way off base?

Quoted 3 lines

But in general iterating
through devices is usually the wrong thing to do, because devices can come and go in the middle of your loop.

I understand that. In practice, though, I think this is not a good approach in many of the cases we're dealing with. Every use of ipath_max iterates over all devices for a reason.

For example, the diag open routine wants to find at least one device that's up. We could maintain a separate list of devices that are in the state that it needs, so that it can just get the first entry off that list or fail if the list is empty, but that's more complex than simply iterating over every device and checking each one.

Quoted 2 lines

(BTW, what does making ipath_max an atomic_t get you? The updates are protected by a lock anyway).

Probably not much. The motivation was to ensure that if it got incremented during an iteration, whoever was iterating would see the update in a timely fashion.

Quoted 5 lines

But I was talking
about the code in ipath_verbs_init(), which is the only place you call ipath_verbs_register() that I could find. You make one pass through the devices that are present when ipath_verbs_init() is called at module load time, and any devices that get added later are missed.

Yes, that code ought to be restructured.

Thanks for the helpful feedback.

	<b
#27 lore raw

Bryan> Probably not much. The motivation was to ensure that if it Bryan> got incremented during an iteration, whoever was iterating Bryan> would see the update in a timely fashion.

But as far as I can see, you never do atomic_inc() -- only atomic_set() under a spinlock.