Skip to contentSkip to conversation

pci-driver: Add driver load messages

PATCH In review Prarit Bhargava 3 participants 14 messages Jan 26, 2021 – Mar 5, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pci-driver: Add driver load messages

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

2) Storage and Network device vendors have relatively short lives for some of their hardware. Some devices may continue to function but are problematic due to out-of-date firmware or other issues. Maintaining a database of the hardware is out-of-the-question in the kernel as it would require constant updating. Outputting a message in the log would allow different OSes to determine if the problem hardware was truly supported or not.

Add optional driver load messages from the PCI core that indicates which driver was loaded, on which slot, and on which device.

Cc Myron Stowe <[email protected]>
Cc Jonathan Corbet <[email protected]>
Cc Bjorn Helgaas <[email protected]>
---
 Documentation/admin-guide/kernel-parameters.txt |  2 ++
 drivers/pci/pci-driver.c                        | 14 +++++++++++++-
 drivers/pci/pci.c                               |  2 ++
 drivers/pci/pci.h                               |  1 +
 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +2 -0
index 23f209c8c19a..32ecee6a4ef0 100644
@@ -3793,6 +3793,8 @@
 		nomio		[S390] Do not use MIO instructions.
 		norid		[S390] ignore the RID field and force use of
 				one PCI domain per PCI function
+		driver_load_messages
+				Output driver load status messages.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
drivers/pci/pci-driver.c +13 -1
index 8b587fc97f7b..35d5b6973578 100644
@@ -23,6 +23,8 @@
 #include "pci.h"
 #include "pcie/portdrv.h"
 
+bool driver_load_messages;
+
 struct pci_dynid {
 	struct list_head node;
 	struct pci_device_id id;
@@ -305,10 +307,20 @@ static long local_pci_probe(void *_ddi)
 	 */
 	pm_runtime_get_sync(dev);
 	pci_dev->driver = pci_drv;
+	if (driver_load_messages)
+		pci_info(pci_dev, "loading on device [%0x:%0x]\n",
+			 pci_dev->vendor, pci_dev->device);
 	rc = pci_drv->probe(pci_dev, ddi->id);
-	if (!rc)
+	if (!rc) {
+		if (driver_load_messages)
+			pci_info(pci_dev, "loaded on device [%0x:%0x]\n",
+				 pci_dev->vendor, pci_dev->device);
 		return rc;
+	}
 	if (rc < 0) {
+		if (driver_load_messages)
+			pci_info(pci_dev, "failed (%d) on device [%0x:%0x]\n",
+				 rc, pci_dev->vendor, pci_dev->device);
 		pci_dev->driver = NULL;
 		pm_runtime_put_sync(dev);
 		return rc;
drivers/pci/pci.c +2 -0
index e578d34095e9..c64b3b6e1e8d 100644
@@ -6547,6 +6547,8 @@ static int __init pci_setup(char *str)
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
 			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
 				disable_acs_redir_param = str + 18;
+			} else if (!strncmp(str, "driver_load_messages", 24)) {
+				driver_load_messages = true;
 			} else {
 				pr_err("PCI: Unknown option `%s'\n", str);
 			}
drivers/pci/pci.h +1 -0
index f86cae9aa1f4..db1218e188ac 100644
@@ -690,4 +690,5 @@ static inline int pci_acpi_program_hp_params(struct pci_dev *dev)
 extern const struct attribute_group aspm_ctrl_attr_group;
 #endif
 
+extern bool driver_load_messages;
 #endif /* DRIVERS_PCI_H */
Signature · 1 line
2.29.2
#1 lore raw
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

During run:
We have many other solutions to get debug prints during run, for example tracing, which is possible to toggle dynamically.

Right now, my laptop will print 34 prints on boot and endless amount during day-to-day usage.

➜ kernel git:(rdma-next) ✗ lspci |wc -l
34

Quoted 7 lines

2) Storage and Network device vendors have relatively short lives for some of their hardware. Some devices may continue to function but are problematic due to out-of-date firmware or other issues. Maintaining a database of the hardware is out-of-the-question in the kernel as it would require constant updating. Outputting a message in the log would allow different OSes to determine if the problem hardware was truly supported or not.

And rely on some dmesg output as a true source of supported/not supported and making this ABI which needs knob in command line. ?

Quoted 3 lines

Add optional driver load messages from the PCI core that indicates which driver was loaded, on which slot, and on which device.

Why don't you add simple pr_debug(..) without any knob? You will be able to enable/disable it through dynamic prints facility.

Thanks

#2 lore raw
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 6 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

Quoted 18 lines

During run:
We have many other solutions to get debug prints during run, for example tracing, which is possible to toggle dynamically.

Right now, my laptop will print 34 prints on boot and endless amount during day-to-day usage.

➜ kernel git:(rdma-next) ✗ lspci |wc -l
34

Quoted 7 lines

2) Storage and Network device vendors have relatively short lives for some of their hardware. Some devices may continue to function but are problematic due to out-of-date firmware or other issues. Maintaining a database of the hardware is out-of-the-question in the kernel as it would require constant updating. Outputting a message in the log would allow different OSes to determine if the problem hardware was truly supported or not.

Quoted 2 lines

And rely on some dmesg output as a true source of supported/not supported and making this ABI which needs knob in command line. ?

Yes. The console log being saved would work as a true source of load messages to be interpreted by an OS tool. But I see your point about the knob below...

Quoted 7 lines
Quoted 4 lines

Add optional driver load messages from the PCI core that indicates which driver was loaded, on which slot, and on which device.

Why don't you add simple pr_debug(..) without any knob? You will be able to enable/disable it through dynamic prints facility.

Good point. I'll wait for more feedback and submit a v2 with pr_debug.

P.

Quoted 2 lines

Thanks

#3 lore raw
Quoted 20 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 6 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

Quoted 2 lines

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 19 lines
Quoted 19 lines

During run:
We have many other solutions to get debug prints during run, for example tracing, which is possible to toggle dynamically.

Right now, my laptop will print 34 prints on boot and endless amount during day-to-day usage.

➜ kernel git:(rdma-next) ✗ lspci |wc -l
34

Quoted 7 lines

2) Storage and Network device vendors have relatively short lives for some of their hardware. Some devices may continue to function but are problematic due to out-of-date firmware or other issues. Maintaining a database of the hardware is out-of-the-question in the kernel as it would require constant updating. Outputting a message in the log would allow different OSes to determine if the problem hardware was truly supported or not.

Quoted 6 lines
Quoted 2 lines

And rely on some dmesg output as a true source of supported/not supported and making this ABI which needs knob in command line. ?

Yes. The console log being saved would work as a true source of load messages to be interpreted by an OS tool. But I see your point about the knob below...

You will need much more stronger claim than the above if you want to proceed ABI path through dmesg prints.

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

Add optional driver load messages from the PCI core that indicates which driver was loaded, on which slot, and on which device.

Why don't you add simple pr_debug(..) without any knob? You will be able to enable/disable it through dynamic prints facility.

Good point. I'll wait for more feedback and submit a v2 with pr_debug.

Just to be clear, none of this can be ABI and any kernel print can be changed or removed any minute without any announcement.

Thanks

Quoted 6 lines

P.

Quoted 2 lines

Thanks

#4 lore raw
Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Prarit Bhargava
Quoted 18 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

Quoted 7 lines
Quoted 2 lines

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

I still have no idea *WHICH* device it was that the panic occurred on.

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines

During run:
We have many other solutions to get debug prints during run, for example tracing, which is possible to toggle dynamically.

Right now, my laptop will print 34 prints on boot and endless amount during day-to-day usage.

➜ kernel git:(rdma-next) ✗ lspci |wc -l
34

Quoted 7 lines

2) Storage and Network device vendors have relatively short lives for some of their hardware. Some devices may continue to function but are problematic due to out-of-date firmware or other issues. Maintaining a database of the hardware is out-of-the-question in the kernel as it would require constant updating. Outputting a message in the log would allow different OSes to determine if the problem hardware was truly supported or not.

And rely on some dmesg output as a true source of supported/not supported and making this ABI which needs knob in command line. ?

Quoted 7 lines
Quoted 3 lines

Yes. The console log being saved would work as a true source of load messages to be interpreted by an OS tool. But I see your point about the knob below...

You will need much more stronger claim than the above if you want to proceed ABI path through dmesg prints.

See my answer below. I agree with you on the ABI statement.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Add optional driver load messages from the PCI core that indicates which driver was loaded, on which slot, and on which device.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Why don't you add simple pr_debug(..) without any knob? You will be able to enable/disable it through dynamic prints facility.

Good point. I'll wait for more feedback and submit a v2 with pr_debug.

Just to be clear, none of this can be ABI and any kernel print can be changed or removed any minute without any announcement.

Yes, that's absolutely the case and I agree with you that nothing can guarantee ABI of those pr_debug() statements. They are *debug* after all.

P.

Quoted 10 lines

Thanks

Quoted 6 lines

P.

Quoted 2 lines

Thanks

#5 lore raw
Quoted 27 lines · Prarit Bhargava
Quoted 24 lines · Leon Romanovsky
Quoted 22 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 6 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

Quoted 6 lines
Quoted 4 lines

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.

If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

Quoted 42 lines
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines

During run:
We have many other solutions to get debug prints during run, for example tracing, which is possible to toggle dynamically.

Right now, my laptop will print 34 prints on boot and endless amount during day-to-day usage.

➜ kernel git:(rdma-next) ✗ lspci |wc -l
34

Quoted 7 lines

2) Storage and Network device vendors have relatively short lives for some of their hardware. Some devices may continue to function but are problematic due to out-of-date firmware or other issues. Maintaining a database of the hardware is out-of-the-question in the kernel as it would require constant updating. Outputting a message in the log would allow different OSes to determine if the problem hardware was truly supported or not.

And rely on some dmesg output as a true source of supported/not supported and making this ABI which needs knob in command line. ?

Quoted 7 lines
Quoted 3 lines

Yes. The console log being saved would work as a true source of load messages to be interpreted by an OS tool. But I see your point about the knob below...

You will need much more stronger claim than the above if you want to proceed ABI path through dmesg prints.

See my answer below. I agree with you on the ABI statement.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

Add optional driver load messages from the PCI core that indicates which driver was loaded, on which slot, and on which device.

Why don't you add simple pr_debug(..) without any knob? You will be able to enable/disable it through dynamic prints facility.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Good point. I'll wait for more feedback and submit a v2 with pr_debug.

Just to be clear, none of this can be ABI and any kernel print can be changed or removed any minute without any announcement.

Yes, that's absolutely the case and I agree with you that nothing can guarantee ABI of those pr_debug() statements. They are *debug* after all.

You missed the point. ALL pr*() prints are not ABI, without relation to their level.

Thanks

Quoted 14 lines

P.

Quoted 10 lines

Thanks

Quoted 6 lines

P.

Quoted 2 lines

Thanks

#6 lore raw
Quoted 33 lines · Leon Romanovsky
Quoted 32 lines · Prarit Bhargava
Quoted 28 lines · Leon Romanovsky
Quoted 20 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 6 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

Quoted 2 lines

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 7 lines
Quoted 1 line

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Quoted 45 lines
Quoted 45 lines
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines

During run:
We have many other solutions to get debug prints during run, for example tracing, which is possible to toggle dynamically.

Right now, my laptop will print 34 prints on boot and endless amount during day-to-day usage.

➜ kernel git:(rdma-next) ✗ lspci |wc -l
34

Quoted 7 lines

2) Storage and Network device vendors have relatively short lives for some of their hardware. Some devices may continue to function but are problematic due to out-of-date firmware or other issues. Maintaining a database of the hardware is out-of-the-question in the kernel as it would require constant updating. Outputting a message in the log would allow different OSes to determine if the problem hardware was truly supported or not.

And rely on some dmesg output as a true source of supported/not supported and making this ABI which needs knob in command line. ?

Quoted 7 lines
Quoted 3 lines

Yes. The console log being saved would work as a true source of load messages to be interpreted by an OS tool. But I see your point about the knob below...

You will need much more stronger claim than the above if you want to proceed ABI path through dmesg prints.

See my answer below. I agree with you on the ABI statement.

Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

Add optional driver load messages from the PCI core that indicates which driver was loaded, on which slot, and on which device.

Why don't you add simple pr_debug(..) without any knob? You will be able to enable/disable it through dynamic prints facility.

Good point. I'll wait for more feedback and submit a v2 with pr_debug.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Just to be clear, none of this can be ABI and any kernel print can be changed or removed any minute without any announcement.

Yes, that's absolutely the case and I agree with you that nothing can guarantee ABI of those pr_debug() statements. They are *debug* after all.

You missed the point. ALL pr*() prints are not ABI, without relation to their level.

Yes, I understood that. I'm just emphasizing your ABI concern.

P.

Quoted 17 lines

Thanks

Quoted 14 lines

P.

Quoted 10 lines

Thanks

Quoted 6 lines

P.

Quoted 2 lines

Thanks

#7 lore raw

Hi Prarit,

Quoted 51 lines · Prarit Bhargava
Quoted 30 lines · Leon Romanovsky
Quoted 29 lines · Prarit Bhargava
Quoted 27 lines · Leon Romanovsky
Quoted 13 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 8 lines
Quoted 5 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 7 lines
Quoted 1 line

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

Quoted 5 lines

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Would a dev_panic() interface that identified the device and driver help with this?

For driver_load_messages, it doesn't seem necessarily PCI-specific. If we want a message like that, maybe it could be in driver_probe_device() or similar? There are already a few pr_debug() calls in that path. There are some enabled by initcall_debug that include the return value from the probe; would those be close to what you're looking for?

Bjorn

#8 lore raw
Quoted 14 lines · Bjorn Helgaas

Hi Prarit,

Quoted 11 lines · Prarit Bhargava
Quoted 10 lines · Leon Romanovsky
Quoted 9 lines · Prarit Bhargava
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Prarit Bhargava
Quoted 6 lines · Leon Romanovsky
Quoted 5 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe()

function

Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 11 lines
Quoted 1 line

has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

Quoted 6 lines
Quoted 4 lines

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

Quoted 7 lines
Quoted 7 lines

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader]

(rev 02)

Quoted 5 lines
Quoted 5 lines

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

Quoted 8 lines
Quoted 5 lines

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Would a dev_panic() interface that identified the device and driver help with this?

It would but,

Quoted 7 lines

For driver_load_messages, it doesn't seem necessarily PCI-specific. If we want a message like that, maybe it could be in driver_probe_device() or similar? There are already a few pr_debug() calls in that path. There are some enabled by initcall_debug that include the return value from the probe; would those be close to what you're looking for?

I think this drivers/base/dd.c:727 might suffice. Let me try some tests with that and get back to you.

Thanks for the pointers,

P.

Quoted 3 lines

Bjorn

#9 lore raw
In reply to Bjorn Helgaas
Quoted 54 lines · Bjorn Helgaas

Hi Prarit,

Quoted 51 lines · Prarit Bhargava
Quoted 30 lines · Leon Romanovsky
Quoted 29 lines · Prarit Bhargava
Quoted 27 lines · Leon Romanovsky
Quoted 13 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 8 lines
Quoted 5 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 7 lines
Quoted 1 line

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

Quoted 9 lines
Quoted 5 lines

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Would a dev_panic() interface that identified the device and driver help with this?

^^ the more I look at this problem, the more a dev_panic() that would output a device specific message at panic time is what I really need.

Quoted 6 lines

For driver_load_messages, it doesn't seem necessarily PCI-specific. If we want a message like that, maybe it could be in driver_probe_device() or similar? There are already a few pr_debug() calls in that path. There are some enabled by initcall_debug that include the return value from the probe; would those be close to what you're looking for?

I took a look at those, and unfortunately they do not meet my requirements. Ultimately, at panic time, I need to know that a driver was loaded on a device at a specific location in the PCI space.

The driver_probe_device() pr_debug calls tell me the location and the driver, but not anything to uniquely identify the device (ie, the PCI vendor and device IDs).

It sounds like you've had some thoughts about a dev_panic() implementation. Care to share them with me? I'm more than willing to implement it but just want to get your more experienced view of what is needed.

P.

Quoted 3 lines

Bjorn

#10 lore raw
Quoted 79 lines · Prarit Bhargava
Quoted 59 lines · Bjorn Helgaas
Quoted 51 lines · Prarit Bhargava
Quoted 30 lines · Leon Romanovsky
Quoted 29 lines · Prarit Bhargava
Quoted 27 lines · Leon Romanovsky
Quoted 13 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 8 lines
Quoted 5 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 7 lines
Quoted 1 line

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

Quoted 6 lines

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Quoted 2 lines

Would a dev_panic() interface that identified the device and driver help with this?

^^ the more I look at this problem, the more a dev_panic() that would output a device specific message at panic time is what I really need.

Quoted 6 lines

For driver_load_messages, it doesn't seem necessarily PCI-specific. If we want a message like that, maybe it could be in driver_probe_device() or similar? There are already a few pr_debug() calls in that path. There are some enabled by initcall_debug that include the return value from the probe; would those be close to what you're looking for?

I took a look at those, and unfortunately they do not meet my requirements. Ultimately, at panic time, I need to know that a driver was loaded on a device at a specific location in the PCI space.

Quoted 8 lines

The driver_probe_device() pr_debug calls tell me the location and the driver, but not anything to uniquely identify the device (ie, the PCI vendor and device IDs).

It sounds like you've had some thoughts about a dev_panic() implementation. Care to share them with me? I'm more than willing to implement it but just want to get your more experienced view of what is needed.

A dev_panic() might indeed be useful. It would be nice to capture the device information at the *first* crash instead of having to boot again with initcall_debug and try to reproduce the crash.

include/linux/dev_printk.h has a whole mess of #defines for dev_info() et al, and maybe that could be extended for dev_panic(). Or maybe the dev_WARN() definition at the bottom, which basically just pastes the driver and device info at the beginning of the string, would be better.

Even if you do add a dev_panic(), I would also take a look through drivers/base/dd.c and similar files to see whether the pr_warn/pr_debug/etc they do could be converted to dev_warn/dev_dbg/etc.

  $ git grep -A2 pr_ drivers/base | grep dev_name

finds several likely-looking candidates. Here are some past patches along that line:

64b3eaf3715d ("xenbus: Use dev_printk() when possible") a88a7b3eb076 ("vfio: Use dev_printk() when possible") 780da9e4f5bf ("iommu: Use dev_printk() when possible")

Bjorn

#11 lore raw
Quoted 61 lines · Bjorn Helgaas
Quoted 60 lines · Prarit Bhargava
Quoted 59 lines · Bjorn Helgaas
Quoted 57 lines · Prarit Bhargava
Quoted 30 lines · Leon Romanovsky
Quoted 29 lines · Prarit Bhargava
Quoted 27 lines · Leon Romanovsky
Quoted 13 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 8 lines
Quoted 5 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 7 lines
Quoted 1 line

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Would a dev_panic() interface that identified the device and driver help with this?

^^ the more I look at this problem, the more a dev_panic() that would output a device specific message at panic time is what I really need.

Bjorn,

I went down this road a bit and had a realization. The issue isn't with printing something at panic time, but the *data* that is output. Each PCI device is associated with a struct device. That device struct's name is output for dev_dbg(), etc., commands. The PCI subsystem sets the device struct name at drivers/pci/probe.c: 1799

	        dev_set_name(&dev->dev, "%04x:%02x:%02x.%d", pci_domain_nr(dev->bus),
                     dev->bus->number, PCI_SLOT(dev->devfn),
                     PCI_FUNC(dev->devfn));

My problem really is that the above information is insufficient when I (or a user) need to debug a system. The complexities of debugging multiple broken driver loads would be much easier if I didn't have to constantly add this output manually :).

Would you be okay with adding a *debug* parameter to expand the device name to include the vendor & device ID pair? FWIW, I'm somewhat against yet-another-kernel-option but that's really the information I need. I could then add dev_dbg() statements in the local_pci_probe() function.

Thoughts?

P.

#12 lore raw
Quoted 78 lines · Prarit Bhargava
Quoted 61 lines · Bjorn Helgaas
Quoted 60 lines · Prarit Bhargava
Quoted 59 lines · Bjorn Helgaas
Quoted 57 lines · Prarit Bhargava
Quoted 30 lines · Leon Romanovsky
Quoted 29 lines · Prarit Bhargava
Quoted 27 lines · Leon Romanovsky
Quoted 13 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 8 lines
Quoted 5 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 7 lines
Quoted 1 line

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Would a dev_panic() interface that identified the device and driver help with this?

^^ the more I look at this problem, the more a dev_panic() that would output a device specific message at panic time is what I really need.

I went down this road a bit and had a realization. The issue isn't with printing something at panic time, but the *data* that is output. Each PCI device is associated with a struct device. That device struct's name is output for dev_dbg(), etc., commands. The PCI subsystem sets the device struct name at drivers/pci/probe.c:
1799

Quoted 8 lines
	        dev_set_name(&dev->dev, "%04x:%02x:%02x.%d", pci_domain_nr(dev->bus),
                     dev->bus->number, PCI_SLOT(dev->devfn),
                     PCI_FUNC(dev->devfn));

My problem really is that the above information is insufficient when I (or a user) need to debug a system. The complexities of debugging multiple broken driver loads would be much easier if I didn't have to constantly add this output manually :).

This *should* already be in the dmesg log:

pci 0000:00:00.0: [8086:5910] type 00 class 0x060000 pci 0000:00:01.0: [8086:1901] type 01 class 0x060400 pci 0000:00:02.0: [8086:591b] type 00 class 0x030000

So if you had a dev_panic(), that message would include the bus/device/function number, and that would be enough to find the vendor/device ID from when the device was first enumerated.

Or are you saying you can't get the part of the dmesg log that contains those vendor/device IDs?

Quoted 5 lines

Would you be okay with adding a *debug* parameter to expand the device name to include the vendor & device ID pair? FWIW, I'm somewhat against yet-another-kernel-option but that's really the information I need. I could then add dev_dbg() statements in the local_pci_probe() function.

#13 lore raw
Quoted 94 lines · Bjorn Helgaas
Quoted 78 lines · Prarit Bhargava
Quoted 61 lines · Bjorn Helgaas
Quoted 60 lines · Prarit Bhargava
Quoted 59 lines · Bjorn Helgaas
Quoted 57 lines · Prarit Bhargava
Quoted 30 lines · Leon Romanovsky
Quoted 29 lines · Prarit Bhargava
Quoted 27 lines · Leon Romanovsky
Quoted 13 lines · Prarit Bhargava
Quoted 12 lines · Leon Romanovsky
Quoted 6 lines · Prarit Bhargava

There are two situations where driver load messages are helpful.

1) Some drivers silently load on devices and debugging driver or system failures in these cases is difficult. While some drivers (networking for example) may not completely initialize when the PCI driver probe() function has returned, it is still useful to have some idea of driver completion.

Sorry, probably it is me, but I don't understand this use case. Are you adding global to whole kernel command line boot argument to debug what and when?

Quoted 8 lines
Quoted 5 lines

During boot:
If device success, you will see it in /sys/bus/pci/[drivers|devices]/*. If device fails, you should get an error from that device (fix the device to return an error), or something immediately won't work and you won't see it in sysfs.

What if there is a panic during boot? There's no way to get to sysfs. That's the case where this is helpful.

How? If you have kernel panic, it means you have much more worse problem than not-supported device. If kernel panic was caused by the driver, you will see call trace related to it. If kernel panic was caused by something else, supported/not supported won't help here.

Quoted 7 lines
Quoted 1 line

I still have no idea *WHICH* device it was that the panic occurred on.

The kernel panic is printed from the driver. There is one driver loaded for all same PCI devices which are probed without relation to their number.>
If you have host with ten same cards, you will see one driver and this is where the problem and not in supported/not-supported device.

That's true, but you can also have different cards loading the same driver. See, for example, any PCI_IDs list in a driver.

For example,

10:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02) 20:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

Both load the megaraid driver and have different profiles within the driver. I have no idea which one actually panicked until removing one card.

It's MUCH worse when debugging new hardware and getting a panic from, for example, the uncore code which binds to a PCI mapped device. One device might work and the next one doesn't. And then you can multiply that by seeing *many* panics at once and trying to determine if the problem was on one specific socket, die, or core.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Would a dev_panic() interface that identified the device and driver help with this?

^^ the more I look at this problem, the more a dev_panic() that would output a device specific message at panic time is what I really need.

I went down this road a bit and had a realization. The issue isn't with printing something at panic time, but the *data* that is output. Each PCI device is associated with a struct device. That device struct's name is output for dev_dbg(), etc., commands. The PCI subsystem sets the device struct name at drivers/pci/probe.c:
1799

Quoted 8 lines
	        dev_set_name(&dev->dev, "%04x:%02x:%02x.%d", pci_domain_nr(dev->bus),
                     dev->bus->number, PCI_SLOT(dev->devfn),
                     PCI_FUNC(dev->devfn));

My problem really is that the above information is insufficient when I (or a user) need to debug a system. The complexities of debugging multiple broken driver loads would be much easier if I didn't have to constantly add this output manually :).

This *should* already be in the dmesg log:

pci 0000:00:00.0: [8086:5910] type 00 class 0x060000 pci 0000:00:01.0: [8086:1901] type 01 class 0x060400 pci 0000:00:02.0: [8086:591b] type 00 class 0x030000

Quoted 6 lines

So if you had a dev_panic(), that message would include the bus/device/function number, and that would be enough to find the vendor/device ID from when the device was first enumerated.

Or are you saying you can't get the part of the dmesg log that contains those vendor/device IDs?

/me hangs head in shame

I didn't notice that until now. :)

Uh thanks for the polite hit with a cluebat :) I *think* that will work. Let me try some additional driver failure tests.

P.

Quoted 7 lines
Quoted 6 lines

Would you be okay with adding a *debug* parameter to expand the device name to include the vendor & device ID pair? FWIW, I'm somewhat against yet-another-kernel-option but that's really the information I need. I could then add dev_dbg() statements in the local_pci_probe() function.