Skip to contentSkip to conversation

net-tg3: Initialize REG_BASE_ADDR at PCI config offset 120 to 0

PATCH In review Nat Gurumoorthy 4 participants 13 messages Dec 9, 2013 – Dec 12, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net-tg3: Initialize REG_BASE_ADDR at PCI config offset 120 to 0

The new tg3 driver leaves REG_BASE_ADDR (PCI config offset 120) uninitialized. From power on reset this register may have garbage in it. The Register Base Address register defines the device local address of a register. The data pointed to by this location is read or written using the Register Data register (PCI config offset 128). When REG_BASE_ADDR has garbage any read or write of Register Data Register (PCI 128) will cause the PCI bus to lock up. The TCO watchdog will fire and bring down the system.

---
 drivers/net/ethernet/broadcom/tg3.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/net/ethernet/broadcom/tg3.c +3 -0
index 369b736..9a904fd 100644
@@ -16499,6 +16499,9 @@ static int tg3_get_invariants(struct tg3 *tp, const struct pci_device_id *ent)
 	/* Clear this out for sanity. */
 	tw32(TG3PCI_MEM_WIN_BASE_ADDR, 0);
 
+	/* Clear TG3PCI_REG_BASE_ADDR to prevent hangs. */
+	tw32(TG3PCI_REG_BASE_ADDR, 0);
+
 	pci_read_config_dword(tp->pdev, TG3PCI_PCISTATE,
 			      &pci_state_reg);
 	if ((pci_state_reg & PCISTATE_CONV_PCI_MODE) == 0 &&
Signature · 1 line
1.8.5.1
#1 lore raw
Quoted 9 lines · Nat Gurumoorthy

The new tg3 driver leaves REG_BASE_ADDR (PCI config offset 120) uninitialized. From power on reset this register may have garbage in it. The Register Base Address register defines the device local address of a register. The data pointed to by this location is read or written using the Register Data register (PCI config offset 128). When REG_BASE_ADDR has garbage any read or write of Register Data Register (PCI 128) will cause the PCI bus to lock up. The TCO watchdog will fire and bring down the system.

Is this to prevent problem from other software that may be scanning the PCI config space? It won't help if this happens before the tg3 driver is loaded, right?

Thanks.

#2 lore raw

Michael,
We had crashes when the PCI config space got scanned via
/sys/devices/pcixxxx/....../config.

I agree that this fix will not help if the scan happens before the tg3 driver gets loaded.

Regards
Nat

Quoted 16 lines · Michael Chan
Quoted 9 lines · Nat Gurumoorthy

The new tg3 driver leaves REG_BASE_ADDR (PCI config offset 120) uninitialized. From power on reset this register may have garbage in it. The Register Base Address register defines the device local address of a register. The data pointed to by this location is read or written using the Register Data register (PCI config offset 128). When REG_BASE_ADDR has garbage any read or write of Register Data Register (PCI 128) will cause the PCI bus to lock up. The TCO watchdog will fire and bring down the system.

Is this to prevent problem from other software that may be scanning the PCI config space? It won't help if this happens before the tg3 driver is loaded, right?

Thanks.

Signature · 2 lines
Regards
Nat Gurumoorthy AB6SJ
#3 lore raw

Hello.

Quoted 6 lines · Natarajan Gurumoorthy

Michael,
We had crashes when the PCI config space got scanned via
/sys/devices/pcixxxx/....../config.

I agree that this fix will not help if the scan happens before the tg3 driver gets loaded.

Then perhaps a better place for such fixup would be a PCI quirk?

Quoted 19 lines

Regards
Nat

Quoted 15 lines · Michael Chan
Quoted 7 lines · Nat Gurumoorthy

The new tg3 driver leaves REG_BASE_ADDR (PCI config offset 120) uninitialized. From power on reset this register may have garbage in it. The Register Base Address register defines the device local address of a register. The data pointed to by this location is read or written using the Register Data register (PCI config offset 128). When REG_BASE_ADDR has garbage any read or write of Register Data Register (PCI 128) will cause the PCI bus to lock up. The TCO watchdog will fire and bring down the system.

Is this to prevent problem from other software that may be scanning the PCI config space? It won't help if this happens before the tg3 driver is loaded, right?

Thanks.

WBR, Sergei

#4 lore raw
Quoted 8 lines · Sergei Shtylyov
Quoted 5 lines

We had crashes when the PCI config space got scanned via
/sys/devices/pcixxxx/....../config.

I agree that this fix will not help if the scan happens before the tg3 driver gets loaded.

Then perhaps a better place for such fixup would be a PCI quirk?

Yes, I agree. Thanks.

#5 lore raw
Quoted 11 lines · Michael Chan
Quoted 8 lines · Sergei Shtylyov
Quoted 5 lines

We had crashes when the PCI config space got scanned via
/sys/devices/pcixxxx/....../config.

I agree that this fix will not help if the scan happens before the tg3 driver gets loaded.

Then perhaps a better place for such fixup would be a PCI quirk?

Yes, I agree. Thanks.

On second thought, I think your original patch should be sufficient and we don't need to add the PCI quirk to cover so many devices. The reason is that indirect access needs to be explicitly enabled in the MISC_HOST_CTRL (0x68) register. The default value for register 0x68 should have indirect access disabled.

Nat, does this match what you're seeing? Did you ever see any system crash before tg3 loads?

#6 lore raw

Michael,
The only time I see crashes is after the tg3 driver has been
loaded into the system. I our use case we are poking around /sys/devices/pcixxxx/......../config.
I guess you will incorporate the original patch into the driver
and we can abandon this patch.

Regards
Nat

Quoted 22 lines · Michael Chan
Quoted 11 lines · Michael Chan
Quoted 8 lines · Sergei Shtylyov
Quoted 5 lines

We had crashes when the PCI config space got scanned via
/sys/devices/pcixxxx/....../config.

I agree that this fix will not help if the scan happens before the tg3 driver gets loaded.

Then perhaps a better place for such fixup would be a PCI quirk?

Yes, I agree. Thanks.

On second thought, I think your original patch should be sufficient and we don't need to add the PCI quirk to cover so many devices. The reason is that indirect access needs to be explicitly enabled in the MISC_HOST_CTRL (0x68) register. The default value for register 0x68 should have indirect access disabled.

Nat, does this match what you're seeing? Did you ever see any system crash before tg3 loads?

Signature · 2 lines
Regards
Nat Gurumoorthy AB6SJ
Acked-by
#7 lore raw
Quoted 3 lines · Natarajan Gurumoorthy

The only time I see crashes is after the tg3 driver has been
loaded into the system. I our use case we are poking around /sys/devices/pcixxxx/......../config.

David, please apply this first patch.

maintainer
#8 lore raw
In reply to Michael Chan

From: Michael Chan <[email protected]>
Date: Mon, 9 Dec 2013 22:58:44 -0800

Quoted 13 lines · Michael Chan
Quoted 11 lines · Sergei Shtylyov
Quoted 8 lines
Quoted 5 lines

We had crashes when the PCI config space got scanned via
/sys/devices/pcixxxx/....../config.

I agree that this fix will not help if the scan happens before the tg3 driver gets loaded.

Then perhaps a better place for such fixup would be a PCI quirk?

Yes, I agree. Thanks.

Quoted 8 lines

On second thought, I think your original patch should be sufficient and we don't need to add the PCI quirk to cover so many devices. The reason is that indirect access needs to be explicitly enabled in the MISC_HOST_CTRL (0x68) register. The default value for register 0x68 should have indirect access disabled.

Nat, does this match what you're seeing? Did you ever see any system crash before tg3 loads?

What if the kernel is booted via kexec, and the driver in the kernel we are kexec'ing from left indirect access enabled in MISC_HOST_CTRL?

#9 lore raw
Quoted 2 lines · David Miller

What if the kernel is booted via kexec, and the driver in the kernel we are kexec'ing from left indirect access enabled in MISC_HOST_CTRL?

That should be ok. The driver will only use valid register offsets in indirect mode during run time, so register 0x78 should point to a valid register. If indirect mode is never used by the driver, it will point to zero with this patch. So register 0x78 will be valid (or zero) in the kexec'ed kernel.

maintainer
#10 lore raw

From: Michael Chan <[email protected]>
Date: Tue, 10 Dec 2013 10:49:39 -0800

Quoted 10 lines · David Miller
Quoted 3 lines

What if the kernel is booted via kexec, and the driver in the kernel we are kexec'ing from left indirect access enabled in MISC_HOST_CTRL?

That should be ok. The driver will only use valid register offsets in indirect mode during run time, so register 0x78 should point to a valid register. If indirect mode is never used by the driver, it will point to zero with this patch. So register 0x78 will be valid (or zero) in the kexec'ed kernel.

Ok, that may be true, but I'd like to consider the much larger issue at hand.

If the indirect mechanism is enabled, some of the offsets that may be in there might be value, but would be entirely undesirable to be read because the read has side effects.

What if the interrupt status register is what gets read if the user scans config space at just the wrong moment, and therefore an interrupt gets lost?

I understand that the patch we are discussing is a serious improvement from the current situation, so I will apply it and queue it up for -stable.

However I think we need to do something reasonable to prevent the kinds of situations I described above.

Thanks.

#11 lore raw
Quoted 11 lines · David Miller

Ok, that may be true, but I'd like to consider the much larger issue at hand.

If the indirect mechanism is enabled, some of the offsets that may be in there might be value, but would be entirely undesirable to be read because the read has side effects.

What if the interrupt status register is what gets read if the user scans config space at just the wrong moment, and therefore an interrupt gets lost?

I did a quick audit of the driver to see if we are seriously exposed to this problem. Only very old chips (5700 to 5703) use indirect register access as workarounds for various issues. I looked at the IRQ path which uses status block to determine if there is work to do. If status block contains no new data, we check the PCI_STATE register and the bits don't get cleared on read. MAC status register which is used to determine link status does not have bits that are cleared on read. So we should be ok.

We use the statistics block for counters on these older chips so there is no conflict. On newer chips, we read counters from registers which get cleared on read, but we never use indirect methods to read these counters.

On all new chips, indirect method is not used except to download firmware (on 57766 only). Userspace read will not interfere with firmware download.

So it looks to me that we are in good shape. Thanks.

maintainer
#12 lore raw

From: Michael Chan <[email protected]>
Date: Wed, 11 Dec 2013 15:21:18 -0800

Quoted 1 line

So it looks to me that we are in good shape. Thanks.

Thanks for checking this out, and providing such detailed analysis.