Skip to contentSkip to conversation

pciehp: Handle interrupts that happen during initialization.

PATCH Applied Eric W. Biederman 3 participants 19 messages Jan 29, 2009 – Feb 24, 2009 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pciehp: Handle interrupts that happen during initialization.

Move the enabling of interrupts after all of the data structures are setup so that we can safely run the interrupt handler as soon as it is registered.

---
 drivers/pci/hotplug/pciehp.h      |    2 ++
 drivers/pci/hotplug/pciehp_core.c |    7 +++++++
 drivers/pci/hotplug/pciehp_hpc.c  |   15 +++++++--------
 3 files changed, 16 insertions(+), 8 deletions(-)
drivers/pci/hotplug/pciehp.h +2 -0
index db85284..39ae375 100644
@@ -111,6 +111,7 @@ struct controller {
 	int cmd_busy;
 	unsigned int no_cmd_complete:1;
 	unsigned int link_active_reporting:1;
+	unsigned int notification_enabled:1;
 };
 
 #define INT_BUTTON_IGNORE		0
@@ -170,6 +171,7 @@ extern int pciehp_configure_device(struct slot *p_slot);
 extern int pciehp_unconfigure_device(struct slot *p_slot);
 extern void pciehp_queue_pushbutton_work(struct work_struct *work);
 struct controller *pcie_init(struct pcie_device *dev);
+int pcie_init_notification(struct controller *ctrl);
 int pciehp_enable_slot(struct slot *p_slot);
 int pciehp_disable_slot(struct slot *p_slot);
 int pcie_enable_notification(struct controller *ctrl);
drivers/pci/hotplug/pciehp_core.c +7 -0
index c248554..681e391 100644
@@ -434,6 +434,13 @@ static int pciehp_probe(struct pcie_device *dev, const struct pcie_port_service_
 		goto err_out_release_ctlr;
 	}
 
+	/* Enable events after we have setup the data structures */
+	rc = pcie_init_notification(ctrl);
+	if (rc) {
+		ctrl_err(ctrl, "Notification initialization failed\n");
+		goto err_out_release_ctlr;
+	}
+
 	/* Check if slot is occupied */
 	t_slot = pciehp_find_slot(ctrl, ctrl->slot_device_offset);
 	t_slot->hpc_ops->get_adapter_status(t_slot, &value);
drivers/pci/hotplug/pciehp_hpc.c +7 -8
index 71a8012..7a16c68 100644
@@ -934,7 +934,7 @@ static void pcie_disable_notification(struct controller *ctrl)
 		ctrl_warn(ctrl, "Cannot disable software notification\n");
 }
 
-static int pcie_init_notification(struct controller *ctrl)
+int pcie_init_notification(struct controller *ctrl)
 {
 	if (pciehp_request_irq(ctrl))
 		return -1;
@@ -942,13 +942,17 @@ static int pcie_init_notification(struct controller *ctrl)
 		pciehp_free_irq(ctrl);
 		return -1;
 	}
+	ctrl->notification_enabled = 1;
 	return 0;
 }
 
 static void pcie_shutdown_notification(struct controller *ctrl)
 {
-	pcie_disable_notification(ctrl);
-	pciehp_free_irq(ctrl);
+	if (ctrl->notification_enabled) {
+		pcie_disable_notification(ctrl);
+		pciehp_free_irq(ctrl);
+		ctrl->notification_enabled = 0;
+	}
 }
 
 static int pcie_init_slot(struct controller *ctrl)
@@ -1110,13 +1114,8 @@ struct controller *pcie_init(struct pcie_device *dev)
 	if (pcie_init_slot(ctrl))
 		goto abort_ctrl;
 
-	if (pcie_init_notification(ctrl))
-		goto abort_slot;
-
 	return ctrl;
 
-abort_slot:
-	pcie_cleanup_slot(ctrl);
 abort_ctrl:
 	kfree(ctrl);
 abort:
Signature · 1 line
1.5.6.6
#1 lore raw

Hi,

I don't understand what this patch is trying to fix. Could you please tell me more about the problem?

And I have a worry. With this patch, the hotplug operation files under /sys/bus/pci/slots/*/ directory will be created before the interrupt is initialized. If hotplug operation is initiated (power on/off, change attention indicator and so on) by user before interrupt is initialized (though I think it's rare), command completion event could not be handled.

Thanks,
Kenji Kaneshige

Quoted 96 lines · Eric W. Biederman

Move the enabling of interrupts after all of the data structures are setup so that we can safely run the interrupt handler as soon as it is registered.

---
 drivers/pci/hotplug/pciehp.h      |    2 ++
 drivers/pci/hotplug/pciehp_core.c |    7 +++++++
 drivers/pci/hotplug/pciehp_hpc.c  |   15 +++++++--------
 3 files changed, 16 insertions(+), 8 deletions(-)
drivers/pci/hotplug/pciehp.h +2 -0
index db85284..39ae375 100644
@@ -111,6 +111,7 @@ struct controller {
 	int cmd_busy;
 	unsigned int no_cmd_complete:1;
 	unsigned int link_active_reporting:1;
+	unsigned int notification_enabled:1;
 };
 
 #define INT_BUTTON_IGNORE		0
@@ -170,6 +171,7 @@ extern int pciehp_configure_device(struct slot *p_slot);
 extern int pciehp_unconfigure_device(struct slot *p_slot);
 extern void pciehp_queue_pushbutton_work(struct work_struct *work);
 struct controller *pcie_init(struct pcie_device *dev);
+int pcie_init_notification(struct controller *ctrl);
 int pciehp_enable_slot(struct slot *p_slot);
 int pciehp_disable_slot(struct slot *p_slot);
 int pcie_enable_notification(struct controller *ctrl);
drivers/pci/hotplug/pciehp_core.c +7 -0
index c248554..681e391 100644
@@ -434,6 +434,13 @@ static int pciehp_probe(struct pcie_device *dev, const struct pcie_port_service_
 		goto err_out_release_ctlr;
 	}
 
+	/* Enable events after we have setup the data structures */
+	rc = pcie_init_notification(ctrl);
+	if (rc) {
+		ctrl_err(ctrl, "Notification initialization failed\n");
+		goto err_out_release_ctlr;
+	}
+
 	/* Check if slot is occupied */
 	t_slot = pciehp_find_slot(ctrl, ctrl->slot_device_offset);
 	t_slot->hpc_ops->get_adapter_status(t_slot, &value);
drivers/pci/hotplug/pciehp_hpc.c +7 -8
index 71a8012..7a16c68 100644
@@ -934,7 +934,7 @@ static void pcie_disable_notification(struct controller *ctrl)
 		ctrl_warn(ctrl, "Cannot disable software notification\n");
 }
 
-static int pcie_init_notification(struct controller *ctrl)
+int pcie_init_notification(struct controller *ctrl)
 {
 	if (pciehp_request_irq(ctrl))
 		return -1;
@@ -942,13 +942,17 @@ static int pcie_init_notification(struct controller *ctrl)
 		pciehp_free_irq(ctrl);
 		return -1;
 	}
+	ctrl->notification_enabled = 1;
 	return 0;
 }
 
 static void pcie_shutdown_notification(struct controller *ctrl)
 {
-	pcie_disable_notification(ctrl);
-	pciehp_free_irq(ctrl);
+	if (ctrl->notification_enabled) {
+		pcie_disable_notification(ctrl);
+		pciehp_free_irq(ctrl);
+		ctrl->notification_enabled = 0;
+	}
 }
 
 static int pcie_init_slot(struct controller *ctrl)
@@ -1110,13 +1114,8 @@ struct controller *pcie_init(struct pcie_device *dev)
 	if (pcie_init_slot(ctrl))
 		goto abort_ctrl;
 
-	if (pcie_init_notification(ctrl))
-		goto abort_slot;
-
 	return ctrl;
 
-abort_slot:
-	pcie_cleanup_slot(ctrl);
 abort_ctrl:
 	kfree(ctrl);
 abort:
#2 lore raw

Any update here, Eric? Sounds like you're using hotplug in real environments with complex topologies (based on your earlier messages), so we're interested in what you're seeing here...

Thanks,
Jesse

Quoted 113 lines · Kenji Kaneshige

Hi,

I don't understand what this patch is trying to fix. Could you please tell me more about the problem?

And I have a worry. With this patch, the hotplug operation files under /sys/bus/pci/slots/*/ directory will be created before the interrupt is initialized. If hotplug operation is initiated (power on/off, change attention indicator and so on) by user before interrupt is initialized (though I think it's rare), command completion event could not be handled.

Thanks,
Kenji Kaneshige

Quoted 97 lines · Eric W. Biederman

Move the enabling of interrupts after all of the data structures are setup so that we can safely run the interrupt handler as soon as it is registered.

---
 drivers/pci/hotplug/pciehp.h      |    2 ++
 drivers/pci/hotplug/pciehp_core.c |    7 +++++++
 drivers/pci/hotplug/pciehp_hpc.c  |   15 +++++++--------
 3 files changed, 16 insertions(+), 8 deletions(-)
drivers/pci/hotplug/pciehp.h +2 -0
index db85284..39ae375 100644
@@ -111,6 +111,7 @@ struct controller {
 	int cmd_busy;
 	unsigned int no_cmd_complete:1;
 	unsigned int link_active_reporting:1;
+	unsigned int notification_enabled:1;
 };
 
 #define INT_BUTTON_IGNORE		0
@@ -170,6 +171,7 @@ extern int pciehp_configure_device(struct slot
 *p_slot); extern int pciehp_unconfigure_device(struct slot *p_slot);
 extern void pciehp_queue_pushbutton_work(struct work_struct *work);
 struct controller *pcie_init(struct pcie_device *dev);
+int pcie_init_notification(struct controller *ctrl);
 int pciehp_enable_slot(struct slot *p_slot);
 int pciehp_disable_slot(struct slot *p_slot);
 int pcie_enable_notification(struct controller *ctrl);
(quoted diff) +0 -0

b/drivers/pci/hotplug/pciehp_core.c index c248554..681e391 100644

drivers/pci/hotplug/pciehp_core.c +7 -0
@@ -434,6 +434,13 @@ static int pciehp_probe(struct pcie_device *dev,
 const struct pcie_port_service_ goto err_out_release_ctlr;
 	}
 
+	/* Enable events after we have setup the data structures */
+	rc = pcie_init_notification(ctrl);
+	if (rc) {
+		ctrl_err(ctrl, "Notification initialization failed\n");
+		goto err_out_release_ctlr;
+	}
+
 	/* Check if slot is occupied */
 	t_slot = pciehp_find_slot(ctrl, ctrl->slot_device_offset);
 	t_slot->hpc_ops->get_adapter_status(t_slot, &value);
(quoted diff) +0 -0

b/drivers/pci/hotplug/pciehp_hpc.c index 71a8012..7a16c68 100644

drivers/pci/hotplug/pciehp_hpc.c +0 -0
@@ -934,7 +934,7 @@ static void pcie_disable_notification(struct

controller *ctrl) ctrl_warn(ctrl, "Cannot disable software notification\n");
}

drivers/pci/hotplug/pciehp_hpc.c +7 -8
-static int pcie_init_notification(struct controller *ctrl)
+int pcie_init_notification(struct controller *ctrl)
 {
 	if (pciehp_request_irq(ctrl))
 		return -1;
@@ -942,13 +942,17 @@ static int pcie_init_notification(struct controller
 *ctrl) pciehp_free_irq(ctrl);
 		return -1;
 	}
+	ctrl->notification_enabled = 1;
 	return 0;
 }
 
 static void pcie_shutdown_notification(struct controller *ctrl)
 {
-	pcie_disable_notification(ctrl);
-	pciehp_free_irq(ctrl);
+	if (ctrl->notification_enabled) {
+		pcie_disable_notification(ctrl);
+		pciehp_free_irq(ctrl);
+		ctrl->notification_enabled = 0;
+	}
 }
 
 static int pcie_init_slot(struct controller *ctrl)
@@ -1110,13 +1114,8 @@ struct controller *pcie_init(struct pcie_device
 *dev) if (pcie_init_slot(ctrl))
 		goto abort_ctrl;
 
-	if (pcie_init_notification(ctrl))
-		goto abort_slot;
-
 	return ctrl;
 
-abort_slot:
-	pcie_cleanup_slot(ctrl);
 abort_ctrl:
 	kfree(ctrl);
 abort:
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#3 lore raw
Quoted 3 lines · Jesse Barnes

Any update here, Eric? Sounds like you're using hotplug in real environments with complex topologies (based on your earlier messages), so we're interested in what you're seeing here...

Yes.

Currently I have a test system that is a subset of what I'm worried about and will shortly have the real hardware, so my immediate goal is to get things working well enough so my internal users won't get blocked by bugs. Currently I only have the pcie hotplug and pcie hotplug surprise case. My basic topology is 16 hotplug slots into which I will be plugging in pci express switches with a couple of additional hotplug slots. As for the firmware, I will have it reserving bus numbers and mmio space on each of the first 16 slots and the rest is going to be up to the linux kernel. This is an embedded design so no ACPI is appears more pain than it is worth to implement.

I am also looking at the case of pcie switches with two upstream ports, and switching which cpu they are connected to at runtime. So in some cases I will have devices whose presence is detected but will not get link for hours or days, as opposed to the 20ms time limit in the pci express specification. Call it a necessary extension.

I need to revisit the pciehp driver but my first pass through it looked like every corner case appeared to get something wrong. So I have written myself a little 430 line replaces that handles the case that I currently care about. Part of what I was seeing before is that we don't clear pending events in the pciehp driver before we enable interrupts. So if booting the system has left some pending and you have CONFIG_DEBUG_SHIRQ enabled you get a nice oops because p_slot has not been initialized and so the interrupts can't be handled.

My little driver is at least good enough to let me start looking at other things. I have just found yesterday that if you mmap a resource in sysfs hot-remove doesn't complete. Sysfs issues seem to be the bane of my existence and I am currently working on a patch for that.

Currently the pcie port driver calls the remove methods of child drivers twice if it removed (say because you have hot unplugged a bridge). Which is one of the truly nasty bugs I saw when I was trying to bring up my test system, as things start access freed memory and all kinds of silly things happen.

After I get the worst of the problems handled I intend to do a thorough review and fix everything that I can see.

Eric

#4 lore raw

And on the big gotcha's I have found one more I am tracking.

I am seeing pci bridges with a NULL pointer for the subordinate bus. Earlier I had thought that this was a symptom of the double remove but I have been able to reproduce it without that.

On just a little bit deeper investigation it looks like the cases are dying are all coming when the nested bridge reappears.

Which is wrong on so many levels as I am toggle power to the outer slot, so the nested bridge should not even exist at that time. Ugh. More tracing to for me on that one.

watch -n1 lspci -t seems like a good way to exacerbate races, when performing hotplug tests.

Eric

#5 lore raw
Quoted 6 lines · [email protected] (Eric W. Biederman) writes:

And on the big gotcha's I have found one more I am tracking.

I am seeing pci bridges with a NULL pointer for the subordinate bus. Earlier I had thought that this was a symptom of the double remove but I have been able to reproduce it without that.

Quoted 6 lines

On just a little bit deeper investigation it looks like the cases are dying are all coming when the nested bridge reappears.

Which is wrong on so many levels as I am toggle power to the outer slot, so the nested bridge should not even exist at that time. Ugh. More tracing to for me on that one.

Ok. Got it. I was processing the interrupt for a device after it had been hot removed but before the device state had disappeared.

pcie_isr looks like it would be even worse in that situation. Looping forever if pciehp_readw(ctrl, PCIE_EXPSLTA) always succeed sand returns 0xffff.

That loop in there appears impossibly misguided. If the pending interrupt values change after you have received the interrupt another instance of the same interrupt should be pending so the loop should be completely unnecessary.

Eric

#6 lore raw
Quoted 18 lines · Eric W. Biederman
Quoted 6 lines · [email protected] (Eric W. Biederman) writes:

And on the big gotcha's I have found one more I am tracking.

I am seeing pci bridges with a NULL pointer for the subordinate bus. Earlier I had thought that this was a symptom of the double remove but I have been able to reproduce it without that.

Quoted 6 lines

On just a little bit deeper investigation it looks like the cases are dying are all coming when the nested bridge reappears.

Which is wrong on so many levels as I am toggle power to the outer slot, so the nested bridge should not even exist at that time. Ugh. More tracing to for me on that one.

Ok. Got it. I was processing the interrupt for a device after it had been hot removed but before the device state had disappeared.

Quoted 8 lines

pcie_isr looks like it would be even worse in that situation. Looping forever if pciehp_readw(ctrl, PCIE_EXPSLTA) always succeed sand returns 0xffff.

That loop in there appears impossibly misguided. If the pending interrupt values change after you have received the interrupt another instance of the same interrupt should be pending so the loop should be completely unnecessary.

For level-triggered interrupt, I think it's true.

But for edge-triggered interrupt, I don't think it's true. I think only one interrupt is generated if the first hotplug event occurs and the second hotplug event occurs before clearing the status of first hotplug event.

Thanks,
Kenji Kaneshige

#7 lore raw
Quoted 28 lines · Kenji Kaneshige
Quoted 18 lines · Eric W. Biederman
Quoted 6 lines · [email protected] (Eric W. Biederman) writes:

And on the big gotcha's I have found one more I am tracking.

I am seeing pci bridges with a NULL pointer for the subordinate bus. Earlier I had thought that this was a symptom of the double remove but I have been able to reproduce it without that.

Quoted 6 lines

On just a little bit deeper investigation it looks like the cases are dying are all coming when the nested bridge reappears.

Which is wrong on so many levels as I am toggle power to the outer slot, so the nested bridge should not even exist at that time. Ugh. More tracing to for me on that one.

Ok. Got it. I was processing the interrupt for a device after it had been hot removed but before the device state had disappeared.

Quoted 8 lines

pcie_isr looks like it would be even worse in that situation. Looping forever if pciehp_readw(ctrl, PCIE_EXPSLTA) always succeed sand returns 0xffff.

That loop in there appears impossibly misguided. If the pending interrupt values change after you have received the interrupt another instance of the same interrupt should be pending so the loop should be completely unnecessary.

Quoted 6 lines

For level-triggered interrupt, I think it's true.

But for edge-triggered interrupt, I don't think it's true. I think only one interrupt is generated if the first hotplug event occurs and the second hotplug event occurs before clearing the status of first hotplug event.

My test case is edge-triggered MSI's. The issue is that I get an interrupt from the card that I am unplugging, but by the time the interrupt handler is executed the card is physically absent, but the pci_dev structure is still present in the kernel.

Eric

#8 lore raw
Quoted 35 lines · Eric W. Biederman
Quoted 26 lines · Kenji Kaneshige
Quoted 17 lines · Eric W. Biederman
Quoted 6 lines · [email protected] (Eric W. Biederman) writes:

And on the big gotcha's I have found one more I am tracking.

I am seeing pci bridges with a NULL pointer for the subordinate bus. Earlier I had thought that this was a symptom of the double remove but I have been able to reproduce it without that.

Quoted 6 lines

On just a little bit deeper investigation it looks like the cases are dying are all coming when the nested bridge reappears.

Which is wrong on so many levels as I am toggle power to the outer slot, so the nested bridge should not even exist at that time. Ugh. More tracing to for me on that one.

Ok. Got it. I was processing the interrupt for a device after it had been hot removed but before the device state had disappeared.

Quoted 8 lines

pcie_isr looks like it would be even worse in that situation. Looping forever if pciehp_readw(ctrl, PCIE_EXPSLTA) always succeed sand returns 0xffff.

That loop in there appears impossibly misguided. If the pending interrupt values change after you have received the interrupt another instance of the same interrupt should be pending so the loop should be completely unnecessary.

Quoted 6 lines

For level-triggered interrupt, I think it's true.

But for edge-triggered interrupt, I don't think it's true. I think only one interrupt is generated if the first hotplug event occurs and the second hotplug event occurs before clearing the status of first hotplug event.

Quoted 5 lines

My test case is edge-triggered MSI's. The issue is that I get an interrupt from the card that I am unplugging, but by the time the interrupt handler is executed the card is physically absent, but the pci_dev structure is still present in the kernel.

Ok, I understood what is happening. Could you try the following patch? It is currently in Jesse's linux-next.

http://marc.info/?l=linux-pci&m=123364118418484&w=2

BTW, I don't think surprise removal is well tested.

Thanks,
Kenji Kaneshige

#9 lore raw
Quoted 6 lines · Kenji Kaneshige

Ok, I understood what is happening. Could you try the following patch? It is currently in Jesse's linux-next.

http://marc.info/?l=linux-pci&m=123364118418484&w=2

BTW, I don't think surprise removal is well tested.

That patch should guarantee that we don't loop forever, and if we are going to loop that looks like a reasonable way to handle it.

When I start working on what is the most maintainable way to implement merge my hotplug driver work I will come back and test this.

At the moment it appears that it will at least suffer from detecting a presence change event with a device showing up. Before pci structure for the device is removed. I seem to recall some dead locks on the pciehp work queue hotunplugging a hotplug driver as well.

Eric

#10 lore raw
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Kenji Kaneshige

Ok, I understood what is happening. Could you try the following patch? It is currently in Jesse's linux-next.

http://marc.info/?l=linux-pci&m=123364118418484&w=2

BTW, I don't think surprise removal is well tested.

That patch should guarantee that we don't loop forever, and if we are going to loop that looks like a reasonable way to handle it.

Quoted 7 lines

When I start working on what is the most maintainable way to implement merge my hotplug driver work I will come back and test this.

At the moment it appears that it will at least suffer from detecting a presence change event with a device showing up. Before pci structure for the device is removed. I seem to recall some dead locks on the pciehp work queue hotunplugging a hotplug driver as well.

So where does that leave us with your original patch; do you think we need something in 2.6.29 for this issue? Or are you planning on reworking things for .30?

Thanks,

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#11 lore raw
In reply to Eric W. Biederman
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Kenji Kaneshige

Ok, I understood what is happening. Could you try the following patch? It is currently in Jesse's linux-next.

http://marc.info/?l=linux-pci&m=123364118418484&w=2

BTW, I don't think surprise removal is well tested.

That patch should guarantee that we don't loop forever, and if we are going to loop that looks like a reasonable way to handle it.

Quoted 7 lines

When I start working on what is the most maintainable way to implement merge my hotplug driver work I will come back and test this.

At the moment it appears that it will at least suffer from detecting a presence change event with a device showing up. Before pci structure for the device is removed. I seem to recall some dead locks on the pciehp work queue hotunplugging a hotplug driver as well.

Ah looks like Kenji-san wants this one pushed; so unless I hear otherwise I'll queue it up in my for-linus branch for Linus to pull tomorrow or so.

Thanks,

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#12 lore raw
In reply to Eric W. Biederman
Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Jesse Barnes

Any update here, Eric? Sounds like you're using hotplug in real environments with complex topologies (based on your earlier messages), so we're interested in what you're seeing here...

Yes.

Quoted 11 lines

Currently I have a test system that is a subset of what I'm worried about and will shortly have the real hardware, so my immediate goal is to get things working well enough so my internal users won't get blocked by bugs. Currently I only have the pcie hotplug and pcie hotplug surprise case. My basic topology is 16 hotplug slots into which I will be plugging in pci express switches with a couple of additional hotplug slots. As for the firmware, I will have it reserving bus numbers and mmio space on each of the first 16 slots and the rest is going to be up to the linux kernel. This is an embedded design so no ACPI is appears more pain than it is worth to implement.

Very interesting. Can I ask you some questions?

- On hot-insertion of pci express switches with a additional hotplug slots, who initialize HwInit registers (for example, physical slot number field in the Slot Capabilities register)? OS, firmware, hardware or others?

- Bus numbers and MMIO space that needs to be reserved is depending on platform design. How do you tell kernel (or hotplug drivers) how many resources need to be reserved, in your current design?

Quoted 6 lines

I am also looking at the case of pcie switches with two upstream ports, and switching which cpu they are connected to at runtime. So in some cases I will have devices whose presence is detected but will not get link for hours or days, as opposed to the 20ms time limit in the pci express specification. Call it a necessary extension.

Quoted 9 lines

I need to revisit the pciehp driver but my first pass through it looked like every corner case appeared to get something wrong. So I have written myself a little 430 line replaces that handles the case that I currently care about. Part of what I was seeing before is that we don't clear pending events in the pciehp driver before we enable interrupts. So if booting the system has left some pending and you have CONFIG_DEBUG_SHIRQ enabled you get a nice oops because p_slot has not been initialized and so the interrupts can't be handled.

I've made a fix (c4635eb06af700820d658a163f06aff12e17cfb2) for a similar problem several months ago. With this fix, pciehp had been changed to initialize p_slot before installing interrupt service routine. So I still don't understand what is happening. Could you please tell me the details about "p_slot has not been initialized..."?

Thanks,
Kenji Kaneshige

#13 lore raw
Quoted 21 lines · Kenji Kaneshige
Quoted 8 lines · Eric W. Biederman
Quoted 3 lines · Jesse Barnes

Any update here, Eric? Sounds like you're using hotplug in real environments with complex topologies (based on your earlier messages), so we're interested in what you're seeing here...

Yes.

Quoted 11 lines

Currently I have a test system that is a subset of what I'm worried about and will shortly have the real hardware, so my immediate goal is to get things working well enough so my internal users won't get blocked by bugs. Currently I only have the pcie hotplug and pcie hotplug surprise case. My basic topology is 16 hotplug slots into which I will be plugging in pci express switches with a couple of additional hotplug slots. As for the firmware, I will have it reserving bus numbers and mmio space on each of the first 16 slots and the rest is going to be up to the linux kernel. This is an embedded design so no ACPI is appears more pain than it is worth to implement.

Quoted 6 lines

Very interesting. Can I ask you some questions?

- On hot-insertion of pci express switches with a additional hotplug slots, who initialize HwInit registers (for example, physical slot number field in the Slot Capabilities register)? OS, firmware, hardware or others?

It happens before the linux kernel gets to see it. Call it firmware.

Quoted 3 lines

- Bus numbers and MMIO space that needs to be reserved is depending on platform design. How do you tell kernel (or hotplug drivers) how many resources need to be reserved, in your current design?

So far it looks like I can get away without telling the kernel anything, and just perform reservations at the layer of the firmware on the primary board, and have the kernel see those reservations when it boots up, and just subdivide them.

I have some thoughts on how to do things better but I'm not at a point where it makes a difference right now.

Quoted 10 lines
Quoted 9 lines

I need to revisit the pciehp driver but my first pass through it looked like every corner case appeared to get something wrong. So I have written myself a little 430 line replaces that handles the case that I currently care about. Part of what I was seeing before is that we don't clear pending events in the pciehp driver before we enable interrupts. So if booting the system has left some pending and you have CONFIG_DEBUG_SHIRQ enabled you get a nice oops because p_slot has not been initialized and so the interrupts can't be handled.

Quoted 5 lines

I've made a fix (c4635eb06af700820d658a163f06aff12e17cfb2) for a similar problem several months ago. With this fix, pciehp had been changed to initialize p_slot before installing interrupt service routine. So I still don't understand what is happening. Could you please tell me the details about "p_slot has not been initialized..."?

kobject_name is not initialized, and slot_name(p_slot) calls hoptlug_slot_name which calls pci_slot_name which kobj_name. It looks like this problem was introduced in commit e1acb24f059defdaa0264e925f19cc21b0a3e592

Eric

#14 lore raw
Quoted 34 lines · Eric W. Biederman
Quoted 19 lines · Kenji Kaneshige
Quoted 7 lines · Eric W. Biederman
Quoted 3 lines · Jesse Barnes

Any update here, Eric? Sounds like you're using hotplug in real environments with complex topologies (based on your earlier messages), so we're interested in what you're seeing here...

Yes.

Quoted 11 lines

Currently I have a test system that is a subset of what I'm worried about and will shortly have the real hardware, so my immediate goal is to get things working well enough so my internal users won't get blocked by bugs. Currently I only have the pcie hotplug and pcie hotplug surprise case. My basic topology is 16 hotplug slots into which I will be plugging in pci express switches with a couple of additional hotplug slots. As for the firmware, I will have it reserving bus numbers and mmio space on each of the first 16 slots and the rest is going to be up to the linux kernel. This is an embedded design so no ACPI is appears more pain than it is worth to implement.

Quoted 6 lines

Very interesting. Can I ask you some questions?

- On hot-insertion of pci express switches with a additional hotplug slots, who initialize HwInit registers (for example, physical slot number field in the Slot Capabilities register)? OS, firmware, hardware or others?

It happens before the linux kernel gets to see it. Call it firmware.

Quoted 3 lines

- Bus numbers and MMIO space that needs to be reserved is depending on platform design. How do you tell kernel (or hotplug drivers) how many resources need to be reserved, in your current design?

Quoted 8 lines

So far it looks like I can get away without telling the kernel anything, and just perform reservations at the layer of the firmware on the primary board, and have the kernel see those reservations when it boots up, and just subdivide them.

I have some thoughts on how to do things better but I'm not at a point where it makes a difference right now.

In the current pciehp implementation, minimum resources enough to enable devices under the bridge are assigned when P2P bridge is hot-added. My concern is that enough resources are NOT assigned to the bridge if an additional slot is empty. As a result, hot-add adapter card on the additional slot won't work because of resource shortage.

Quoted 15 lines
Quoted 9 lines
Quoted 9 lines

I need to revisit the pciehp driver but my first pass through it looked like every corner case appeared to get something wrong. So I have written myself a little 430 line replaces that handles the case that I currently care about. Part of what I was seeing before is that we don't clear pending events in the pciehp driver before we enable interrupts. So if booting the system has left some pending and you have CONFIG_DEBUG_SHIRQ enabled you get a nice oops because p_slot has not been initialized and so the interrupts can't be handled.

Quoted 5 lines

I've made a fix (c4635eb06af700820d658a163f06aff12e17cfb2) for a similar problem several months ago. With this fix, pciehp had been changed to initialize p_slot before installing interrupt service routine. So I still don't understand what is happening. Could you please tell me the details about "p_slot has not been initialized..."?

Quoted 4 lines

kobject_name is not initialized, and slot_name(p_slot) calls hoptlug_slot_name which calls pci_slot_name which kobj_name. It looks like this problem was introduced in commit e1acb24f059defdaa0264e925f19cc21b0a3e592

Thank your for the information. I understood what is happening. This needs to be fixed. But, as I mentioned before, I think software notification mechanism should be initialized before sysfs entries are created. I'll consider alternative fix.

Thanks,
Kenji Kaneshige

#15 lore raw
Quoted 6 lines · Kenji Kaneshige

In the current pciehp implementation, minimum resources enough to enable devices under the bridge are assigned when P2P bridge is hot-added. My concern is that enough resources are NOT assigned to the bridge if an additional slot is empty. As a result, hot-add adapter card on the additional slot won't work because of resource shortage.

It is a good concern. Right now I know I won't need a bus number but you are quite right the mmio and iospace may be a problem.

My preliminary test case doesn't cover that so I will look and confirm it is a problem I need to address.

Quoted 9 lines
Quoted 4 lines

kobject_name is not initialized, and slot_name(p_slot) calls hoptlug_slot_name which calls pci_slot_name which kobj_name. It looks like this problem was introduced in commit e1acb24f059defdaa0264e925f19cc21b0a3e592

Thank your for the information. I understood what is happening. This needs to be fixed. But, as I mentioned before, I think software notification mechanism should be initialized before sysfs entries are created. I'll consider alternative fix.

Reasonable. I haven't had the need nor gotten brave enough to support those sysfs entries in my minimal driver.

Eric

Reviewed-by Tested-by
#16 lore raw
Quoted 19 lines · Eric W. Biederman
Quoted 6 lines · Kenji Kaneshige

In the current pciehp implementation, minimum resources enough to enable devices under the bridge are assigned when P2P bridge is hot-added. My concern is that enough resources are NOT assigned to the bridge if an additional slot is empty. As a result, hot-add adapter card on the additional slot won't work because of resource shortage.

It is a good concern. Right now I know I won't need a bus number but you are quite right the mmio and iospace may be a problem.

My preliminary test case doesn't cover that so I will look and confirm it is a problem I need to address.

Quoted 4 lines
Quoted 4 lines

kobject_name is not initialized, and slot_name(p_slot) calls hoptlug_slot_name which calls pci_slot_name which kobj_name. It looks like this problem was introduced in commit e1acb24f059defdaa0264e925f19cc21b0a3e592

Quoted 8 lines
Quoted 4 lines

Thank your for the information. I understood what is happening. This needs to be fixed. But, as I mentioned before, I think software notification mechanism should be initialized before sysfs entries are created. I'll consider alternative fix.

Reasonable. I haven't had the need nor gotten brave enough to support those sysfs entries in my minimal driver.

Eric-san, Jesse-san,

I tried to make alternative fix, but now I have a conclusion that Eric-san's patch is the best solution. The reasons are

- Eric-san's approach prevents the regression such as commit e1acb24f059defdaa0264e925f19cc21b0a3e592.

- My concern was that command completion might not be handled if we enable software notification mechanism after creating sysfs entries. But it is not true because pciehp uses polling approach if command is issued before enabling software notification (I made it, but I forgot it...). Just in case, I confirmed that hotplug operations (power on/off, attention indication on/off and so on) works even if I make pciehp_init_notification() NOP.

Jesse-san, could you consider applying Eric-san's patch?

Here are my

Note: I tested Eric-san's patch on Jesse-san's for-linus tree.

Thanks,
Kenji Kaneshige

#17 lore raw
In reply to Eric W. Biederman
Quoted 9 lines · [email protected] (Eric W. Biederman) writes:
Quoted 6 lines · Kenji Kaneshige

In the current pciehp implementation, minimum resources enough to enable devices under the bridge are assigned when P2P bridge is hot-added. My concern is that enough resources are NOT assigned to the bridge if an additional slot is empty. As a result, hot-add adapter card on the additional slot won't work because of resource shortage.

Quoted 5 lines

It is a good concern. Right now I know I won't need a bus number but you are quite right the mmio and iospace may be a problem.

My preliminary test case doesn't cover that so I will look and confirm it is a problem I need to address.

Tested. I do have a problem with not assigning resources to new bridges.

My plan to handle this currently is to do what the pci code does for cardbus without resources. To give them a fixed sized resource assignment without looking downstream from the bridge. The code looks simple, and we have existing practice so it should not be too bad to deal with.

Eric