Skip to contentSkip to conversation

pci logical vs. physical hotplug

DISCUSSION Alex Chiang 2 participants 2 messages Mar 31, 2009 – Apr 2, 2009

lore
Share & tools
lore
#0 lore raw

* Kenji Kaneshige <[email protected]>:

Quoted 4 lines

I confirmed this patch fix the kernel oops problem I reported.

Thank you for reviewing and testing.

Quoted 7 lines

By the way, /sys/bus/pci/slots/<slot> directory by acpiphp are remaining even after the parent bridge/bus of the slots are removed. At this point, acpiphp is working with struct pci_bus for the already disabled pci bus. I guess some operation against the files under /sys/bus/pci/slots/<slot> directory would cause something problems. So I think we also need something mechanism to unregister acpiphp slots when the parent bus is removed.

Yes, I've been thinking about this (and thank you for your other mail confirming the issue).

The logical hotplug and physical hotplug don't play very nicely with each other.

I think one of the core issues is that logical hotplug allows function level granularity while physical hotplug is naturally restricted to physical slot granularity, which includes an entire hierarchy, from host bus down to function.

If a user uses logical hotplug to take out a piece of the tree, what does that mean if it's part of the physical slot/device?

What should happen?

Take something like this:

[0000:2e-4f]----00.0-[0000:2f-4f]--+-02.0-[0000:30-3f]--+-00.0  Intel GigE
                                   |                    \-00.1  Intel GigE
                                   \-04.0-[0000:40-4f]--+-00.0  Intel GigE
                                                        \-00.1  Intel GigE

Assume that this is a quad-port NIC with a bridge in it, and that the physical slot is 0000:2e:00.0.

What should we do if the user does a logical hotplug on 0000:2f:04.0 and has acpiphp loaded?

If acpiphp tries to do anything to 0000:40: we'll probably get an oops.

But just because the user took out one piece of that tree doesn't mean that we should disable the entire slot. If that was the case, then he could just use the existing hotplug drivers.

I don't have a good answer for right now, other than, "don't try to mix logical and physical hotplug". I'm open to any ideas that you may have.

Thanks.

/ac

#1 lore raw
Quoted 16 lines · Alex Chiang

* Kenji Kaneshige <[email protected]>:

Quoted 4 lines

I confirmed this patch fix the kernel oops problem I reported.

Thank you for reviewing and testing.

Quoted 7 lines

By the way, /sys/bus/pci/slots/<slot> directory by acpiphp are remaining even after the parent bridge/bus of the slots are removed. At this point, acpiphp is working with struct pci_bus for the already disabled pci bus. I guess some operation against the files under /sys/bus/pci/slots/<slot> directory would cause something problems. So I think we also need something mechanism to unregister acpiphp slots when the parent bus is removed.

Quoted 3 lines

Yes, I've been thinking about this (and thank you for your other mail confirming the issue).

The pciehp (and probably also shpchp) doesn't have this problem because /sys/bus/pci/slots/<slot> directories are unregistered when the parent bridge of those slots is removed. This is done with PCI driver model. The acpiphp and pci_slot driver doesn't match the PCI driver model, so I think those drivers need something other notification mechanism of PCI device removal. I guess this kind of notification mechanism might already exist in the kernel to bind ACPI device and PCI device, though it's still just my guess.

Please note that I'm not saying that pciehp doesn't have any problem about the interaction with logical hotplug mechanism. I think there are still remaining issues, and we need to solve them one by one.

Quoted 29 lines

The logical hotplug and physical hotplug don't play very nicely with each other.

I think one of the core issues is that logical hotplug allows function level granularity while physical hotplug is naturally restricted to physical slot granularity, which includes an entire hierarchy, from host bus down to function.

If a user uses logical hotplug to take out a piece of the tree, what does that mean if it's part of the physical slot/device?

What should happen?

Take something like this:

[0000:2e-4f]----00.0-[0000:2f-4f]--+-02.0-[0000:30-3f]--+-00.0  Intel GigE
                                   |                    \-00.1  Intel GigE
                                   \-04.0-[0000:40-4f]--+-00.0  Intel GigE
                                                        \-00.1  Intel GigE

Assume that this is a quad-port NIC with a bridge in it, and that the physical slot is 0000:2e:00.0.

What should we do if the user does a logical hotplug on 0000:2f:04.0 and has acpiphp loaded?

If acpiphp tries to do anything to 0000:40: we'll probably get an oops.

Quoted 8 lines

But just because the user took out one piece of that tree doesn't mean that we should disable the entire slot. If that was the case, then he could just use the existing hotplug drivers.

I don't have a good answer for right now, other than, "don't try to mix logical and physical hotplug". I'm open to any ideas that you may have.

Physical hotplug is:

	physical hot-add = physical power on + logical hot-add;
	physical hot-removal = logical hot-removal + physical power off;

In the above case, some pieces of logical hot-removal has already been done. So I think what acpiphp need to do is remaining pieces of logical hot-removal of tree and power off.

Thanks,
Kenji Kaneshige