Skip to contentSkip to conversation

ath9k: SMP fixes

PATCH Superseded Luis R. Rodriguez 2 participants 7 messages Mar 12, 2009 – Mar 13, 2009 Networking drivers (wireless)

lore
Share & tools
lore
#0 lore raw

v6 cover ath9k: SMP fixes

This v6 removes the hotplug CPU stuff which although it technically would be correct is more cruft and highly unlikely. The down side is you will get serialization applied even if you only have one CPU but who cares, I doubt those nutty SGI guys are toying with ath9k in the lab.

Yesterday I started mucking with an alternative approach to serialization which worked. I added just 5 udelay()s in key places where we had a lot of consecutive IO read/writes issued and that resolved the issue at least for STA mode but it didn't do the trick for AP mode. Technically it should be possible to groom all hardware access routines to do the same but I'm tired of this issue and want to a fix merged today not a few weeks from now. Anyway if you are curious you can check out that patch replacement approach for serialization here:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/smp-udelay-fix.patch

Note that AP won't work though so if you have a lot of time in your hands and have an SMP box with PCI Atheros 11n and are seeing the hangs and want a neat alternative to the current serialization implementation try adding udelays like the ones in the patch in places where we do a lot of io read/writes. Send me the results :)

The compromise is to keep serialization conditional -- we do not want to do it for every read/write regardless of the type of card you have, the code overhead is not great and we maintain this anyway. Adding it for all cases would just not be optimal.

Luis R. Rodriguez (3):
ath9k: implement IO serialization
ath9k: AR9280 PCI devices must serialize IO as well
ath9k: remove dummy PCI "retry timeout" fix

 drivers/net/wireless/ath9k/ath9k.h |   34 ++++++++++++++++++++++++++++++++++
 drivers/net/wireless/ath9k/hw.c    |   28 +++++++++++++++++++++++++++-
 drivers/net/wireless/ath9k/hw.h    |    4 ++--
 drivers/net/wireless/ath9k/main.c  |    1 +
 drivers/net/wireless/ath9k/pci.c   |   18 ------------------
 5 files changed, 64 insertions(+), 21 deletions(-)
#1 lore raw

PATCH v6 1/3 ath9k: implement IO serialization

All 802.11n PCI devices (Cardbus, PCI, mini-PCI) require serialization of IO when on non-uniprocessor systems. PCI express devices not not require this.

This should fix our only last standing open ath9k kernel.org bugzilla bug report:

http://bugzilla.kernel.org/show_bug.cgi?id=12110

A port is probably required to older kernels and I can work on that.

Cc stable at kernel.org
---
 drivers/net/wireless/ath9k/ath9k.h |   34 ++++++++++++++++++++++++++++++++++
 drivers/net/wireless/ath9k/hw.c    |   19 +++++++++++++++++++
 drivers/net/wireless/ath9k/hw.h    |    4 ++--
 drivers/net/wireless/ath9k/main.c  |    1 +
 4 files changed, 56 insertions(+), 2 deletions(-)
drivers/net/wireless/ath9k/ath9k.h +34 -0
index f0b105a..050304e 100644
@@ -579,6 +579,7 @@ struct ath_softc {
 	void __iomem *mem;
 	int irq;
 	spinlock_t sc_resetlock;
+	spinlock_t sc_serial_rw;
 	struct mutex mutex;
 
 	u8 curbssid[ETH_ALEN];
@@ -724,4 +725,37 @@ void ath9k_wiphy_pause_all_forced(struct ath_softc *sc,
 bool ath9k_wiphy_scanning(struct ath_softc *sc);
 void ath9k_wiphy_work(struct work_struct *work);
 
+/*
+ * Read and write, they both share the same lock. We do this to serialize
+ * reads and writes on Atheros 802.11n PCI devices only. This is required
+ * as the FIFO on these devices can only accept sanely 2 requests. After
+ * that the device goes bananas. Serializing the reads/writes prevents this
+ * from happening.
+ */
+
+static inline void ath9k_iowrite32(struct ath_hw *ah, u32 reg_offset, u32 val)
+{
+	if (ah->config.serialize_regmode == SER_REG_MODE_ON) {
+		unsigned long flags;
+		spin_lock_irqsave(&ah->ah_sc->sc_serial_rw, flags);
+		iowrite32(val, ah->ah_sc->mem + reg_offset);
+		spin_unlock_irqrestore(&ah->ah_sc->sc_serial_rw, flags);
+	} else
+		iowrite32(val, ah->ah_sc->mem + reg_offset);
+}
+
+static inline unsigned int ath9k_ioread32(struct ath_hw *ah, u32 reg_offset)
+{
+	u32 val;
+	if (ah->config.serialize_regmode == SER_REG_MODE_ON) {
+		unsigned long flags;
+		spin_lock_irqsave(&ah->ah_sc->sc_serial_rw, flags);
+		val = ioread32(ah->ah_sc->mem + reg_offset);
+		spin_unlock_irqrestore(&ah->ah_sc->sc_serial_rw, flags);
+	} else
+		val = ioread32(ah->ah_sc->mem + reg_offset);
+	return val;
+}
+
+
 #endif /* ATH9K_H */
drivers/net/wireless/ath9k/hw.c +19 -0
index ea550cc..50a44a0 100644
@@ -391,6 +391,25 @@ static void ath9k_hw_set_defaults(struct ath_hw *ah)
 	}
 
 	ah->config.intr_mitigation = 1;
+
+	/*
+	 * We need this for PCI devices only (Cardbus, PCI, miniPCI)
+	 * _and_ if on non-uniprocessor systems (Multiprocessor/HT).
+	 * This means we use it for all AR5416 devices, and the few
+	 * minor PCI AR9280 devices out there.
+	 *
+	 * Serialization is required because these devices do not handle
+	 * well the case of two concurrent reads/writes due to the latency
+	 * involved. During one read/write another read/write can be issued
+	 * on another CPU while the previous read/write may still be working
+	 * on our hardware, if we hit this case the hardware poops in a loop.
+	 * We prevent this by serializing reads and writes.
+	 *
+	 * This issue is not present on PCI-Express devices or pre-AR5416
+	 * devices (legacy, 802.11abg).
+	 */
+	if (num_possible_cpus() > 1)
+		ah->config.serialize_regmode = SER_REG_MODE_AUTO;
 }
 
 static struct ath_hw *ath9k_hw_newstate(u16 devid, struct ath_softc *sc,
drivers/net/wireless/ath9k/hw.h +2 -2
index 89936a0..dc681f0 100644
@@ -42,8 +42,8 @@
 #define AR5416_MAGIC		0x19641014
 
 /* Register read/write primitives */
-#define REG_WRITE(_ah, _reg, _val) iowrite32(_val, _ah->ah_sc->mem + _reg)
-#define REG_READ(_ah, _reg) ioread32(_ah->ah_sc->mem + _reg)
+#define REG_WRITE(_ah, _reg, _val) ath9k_iowrite32((_ah), (_reg), (_val))
+#define REG_READ(_ah, _reg) ath9k_ioread32((_ah), (_reg))
 
 #define SM(_v, _f)  (((_v) << _f##_S) & _f)
 #define MS(_v, _f)  (((_v) & _f) >> _f##_S)
drivers/net/wireless/ath9k/main.c +1 -0
index a2f5af6..8db75f6 100644
@@ -1370,6 +1370,7 @@ static int ath_init(u16 devid, struct ath_softc *sc)
 
 	spin_lock_init(&sc->wiphy_lock);
 	spin_lock_init(&sc->sc_resetlock);
+	spin_lock_init(&sc->sc_serial_rw);
 	mutex_init(&sc->mutex);
 	tasklet_init(&sc->intr_tq, ath9k_tasklet, (unsigned long)sc);
 	tasklet_init(&sc->bcon_tasklet, ath_beacon_tasklet,
Signature · 1 line
1.6.0.6
#2 lore raw

PATCH v6 2/3 ath9k: AR9280 PCI devices must serialize IO as well

In reply to Luis R. Rodriguez
Cc stable at kernel.org
---
 drivers/net/wireless/ath9k/hw.c |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)
drivers/net/wireless/ath9k/hw.c +8 -1
index 50a44a0..51d3a4e 100644
@@ -628,8 +628,15 @@ static struct ath_hw *ath9k_hw_do_attach(u16 devid, struct ath_softc *sc,
 		goto bad;
 	}
 
+	/*
+	 * All PCI devices should be put here.
+	 * XXX: remove ah->is_pciexpress and use pdev->is_pcie, then
+	 * we can just check for !pdev->is_pcie here, but
+	 * consideration must be taken for handling AHB as well.
+	 */
 	if (ah->config.serialize_regmode == SER_REG_MODE_AUTO) {
-		if (ah->hw_version.macVersion == AR_SREV_VERSION_5416_PCI) {
+		if (ah->hw_version.macVersion == AR_SREV_VERSION_5416_PCI ||
+		    (AR_SREV_9280(ah) && !ah->is_pciexpress)) {
 			ah->config.serialize_regmode =
 				SER_REG_MODE_ON;
 		} else {
Signature · 1 line
1.6.0.6
#3 lore raw

PATCH v6 3/3 ath9k: remove dummy PCI "retry timeout" fix

In reply to Luis R. Rodriguez

Remove the PCI retry timeout code as that was just taken from ipw2100 due to historical reasons but in reality its a no-op, additionally its simply incorrect as each PCI devices has its own custom PCI configuration space on PCI config space >= 0x40. Not to mention we were trying to write 0 to a place that already has 0 on it.

---
 drivers/net/wireless/ath9k/pci.c |   18 ------------------
 1 files changed, 0 insertions(+), 18 deletions(-)
drivers/net/wireless/ath9k/pci.c +0 -18
index 9a58baa..53572d9 100644
@@ -87,7 +87,6 @@ static int ath_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	struct ath_softc *sc;
 	struct ieee80211_hw *hw;
 	u8 csz;
-	u32 val;
 	int ret = 0;
 	struct ath_hw *ah;
 
@@ -134,14 +133,6 @@ static int ath_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	pci_set_master(pdev);
 
-	/*
-	 * Disable the RETRY_TIMEOUT register (0x41) to keep
-	 * PCI Tx retries from interfering with C3 CPU state.
-	 */
-	pci_read_config_dword(pdev, 0x40, &val);
-	if ((val & 0x0000ff00) != 0)
-		pci_write_config_dword(pdev, 0x40, val & 0xffff00ff);
-
 	ret = pci_request_region(pdev, 0, "ath9k");
 	if (ret) {
 		dev_err(&pdev->dev, "PCI memory region reserve error\n");
@@ -253,21 +244,12 @@ static int ath_pci_resume(struct pci_dev *pdev)
 	struct ieee80211_hw *hw = pci_get_drvdata(pdev);
 	struct ath_wiphy *aphy = hw->priv;
 	struct ath_softc *sc = aphy->sc;
-	u32 val;
 	int err;
 
 	err = pci_enable_device(pdev);
 	if (err)
 		return err;
 	pci_restore_state(pdev);
-	/*
-	 * Suspend/Resume resets the PCI configuration space, so we have to
-	 * re-disable the RETRY_TIMEOUT register (0x41) to keep
-	 * PCI Tx retries from interfering with C3 CPU state
-	 */
-	pci_read_config_dword(pdev, 0x40, &val);
-	if ((val & 0x0000ff00) != 0)
-		pci_write_config_dword(pdev, 0x40, val & 0xffff00ff);
 
 	/* Enable LED */
 	ath9k_hw_cfg_output(sc->sc_ah, ATH_LED_PIN,
Signature · 1 line
1.6.0.6
#4 lore raw

v6 cover ath9k: SMP fixes

In reply to Luis R. Rodriguez
Quoted 34 lines · Luis R. Rodriguez

This v6 removes the hotplug CPU stuff which although it technically would be correct is more cruft and highly unlikely. The down side is you will get serialization applied even if you only have one CPU but who cares, I doubt those nutty SGI guys are toying with ath9k in the lab.

Yesterday I started mucking with an alternative approach to serialization which worked. I added just 5 udelay()s in key places where we had a lot of consecutive IO read/writes issued and that resolved the issue at least for STA mode but it didn't do the trick for AP mode. Technically it should be possible to groom all hardware access routines to do the same but I'm tired of this issue and want to a fix merged today not a few weeks from now. Anyway if you are curious you can check out that patch replacement approach for serialization here:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/smp-udelay-fix.patch

Note that AP won't work though so if you have a lot of time in your hands and have an SMP box with PCI Atheros 11n and are seeing the hangs and want a neat alternative to the current serialization implementation try adding udelays like the ones in the patch in places where we do a lot of io read/writes. Send me the results :)

The compromise is to keep serialization conditional -- we do not want to do it for every read/write regardless of the type of card you have, the code overhead is not great and we maintain this anyway. Adding it for all cases would just not be optimal.

Luis R. Rodriguez (3):
?ath9k: implement IO serialization
?ath9k: AR9280 PCI devices must serialize IO as well ?ath9k: remove dummy PCI "retry timeout" fix

Quoted 6 lines
?drivers/net/wireless/ath9k/ath9k.h | ? 34 ++++++++++++++++++++++++++++++++++
?drivers/net/wireless/ath9k/hw.c ? ?| ? 28 +++++++++++++++++++++++++++-
?drivers/net/wireless/ath9k/hw.h ? ?| ? ?4 ++--
?drivers/net/wireless/ath9k/main.c ?| ? ?1 +
?drivers/net/wireless/ath9k/pci.c ? | ? 18 ------------------

?5 files changed, 64 insertions(+), 21 deletions(-)

John, Greg,

these patches require a port down to the other stable kernels, I've taken the time to port only the necessary patches (the first two) to fix the SMP hang on each stable kernel and test them accordingly:

* 2.6.27
* 2.6.28
* 2.6.29

Once merged into wireless-testing the following patches can be used to push into Linus' tree:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.29/

Once merged into stable these can be used to merge into 2.6.27 and 2.6.28:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.27/
http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.28/

Just replace:

This is a port of commit SHA1 <BLEH>

With the respective SHA1.

I can also simply resend the ports for 27 and 28 once merged into Linus' tree. Whatever makes it easier for you.

Luis

#5 lore raw

v6 cover ath9k: SMP fixes

Quoted 63 lines · Luis R. Rodriguez
Quoted 34 lines · Luis R. Rodriguez

This v6 removes the hotplug CPU stuff which although it technically would be correct is more cruft and highly unlikely. The down side is you will get serialization applied even if you only have one CPU but who cares, I doubt those nutty SGI guys are toying with ath9k in the lab.

Yesterday I started mucking with an alternative approach to serialization which worked. I added just 5 udelay()s in key places where we had a lot of consecutive IO read/writes issued and that resolved the issue at least for STA mode but it didn't do the trick for AP mode. Technically it should be possible to groom all hardware access routines to do the same but I'm tired of this issue and want to a fix merged today not a few weeks from now. Anyway if you are curious you can check out that patch replacement approach for serialization here:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/smp-udelay-fix.patch

Note that AP won't work though so if you have a lot of time in your hands and have an SMP box with PCI Atheros 11n and are seeing the hangs and want a neat alternative to the current serialization implementation try adding udelays like the ones in the patch in places where we do a lot of io read/writes. Send me the results :)

The compromise is to keep serialization conditional -- we do not want to do it for every read/write regardless of the type of card you have, the code overhead is not great and we maintain this anyway. Adding it for all cases would just not be optimal.

Luis R. Rodriguez (3):
?ath9k: implement IO serialization
?ath9k: AR9280 PCI devices must serialize IO as well ?ath9k: remove dummy PCI "retry timeout" fix

Quoted 6 lines
?drivers/net/wireless/ath9k/ath9k.h | ? 34 ++++++++++++++++++++++++++++++++++
?drivers/net/wireless/ath9k/hw.c ? ?| ? 28 +++++++++++++++++++++++++++-
?drivers/net/wireless/ath9k/hw.h ? ?| ? ?4 ++--
?drivers/net/wireless/ath9k/main.c ?| ? ?1 +
?drivers/net/wireless/ath9k/pci.c ? | ? 18 ------------------

?5 files changed, 64 insertions(+), 21 deletions(-)

John, Greg,

these patches require a port down to the other stable kernels, I've taken the time to port only the necessary patches (the first two) to fix the SMP hang on each stable kernel and test them accordingly:

* 2.6.27
* 2.6.28
* 2.6.29

Once merged into wireless-testing the following patches can be used to push into Linus' tree:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.29/

Once merged into stable these can be used to merge into 2.6.27 and 2.6.28:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.27/
http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.28/

Quoted 8 lines

Just replace:

This is a port of commit SHA1 <BLEH>

With the respective SHA1.

I can also simply resend the ports for 27 and 28 once merged into Linus' tree. Whatever makes it easier for you.

Please, when the patches go into Linus's tree, send stable at kernel.org the backported patches, with the proper git commit id in them.

Otherwise it's just too much confusing work for me and I'll easily mess it up. The number of -stable patches is increasing and I need all the help I can get in handling this in a sane manner :)

thanks,

greg k-h

#6 lore raw

v6 cover ath9k: SMP fixes

Quoted 73 lines · Greg KH
Quoted 63 lines · Luis R. Rodriguez
Quoted 34 lines · Luis R. Rodriguez

This v6 removes the hotplug CPU stuff which although it technically would be correct is more cruft and highly unlikely. The down side is you will get serialization applied even if you only have one CPU but who cares, I doubt those nutty SGI guys are toying with ath9k in the lab.

Yesterday I started mucking with an alternative approach to serialization which worked. I added just 5 udelay()s in key places where we had a lot of consecutive IO read/writes issued and that resolved the issue at least for STA mode but it didn't do the trick for AP mode. Technically it should be possible to groom all hardware access routines to do the same but I'm tired of this issue and want to a fix merged today not a few weeks from now. Anyway if you are curious you can check out that patch replacement approach for serialization here:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/smp-udelay-fix.patch

Note that AP won't work though so if you have a lot of time in your hands and have an SMP box with PCI Atheros 11n and are seeing the hangs and want a neat alternative to the current serialization implementation try adding udelays like the ones in the patch in places where we do a lot of io read/writes. Send me the results :)

The compromise is to keep serialization conditional -- we do not want to do it for every read/write regardless of the type of card you have, the code overhead is not great and we maintain this anyway. Adding it for all cases would just not be optimal.

Luis R. Rodriguez (3):
ath9k: implement IO serialization
ath9k: AR9280 PCI devices must serialize IO as well
ath9k: remove dummy PCI "retry timeout" fix

Quoted 6 lines
 drivers/net/wireless/ath9k/ath9k.h |   34 ++++++++++++++++++++++++++++++++++
 drivers/net/wireless/ath9k/hw.c    |   28 +++++++++++++++++++++++++++-
 drivers/net/wireless/ath9k/hw.h    |    4 ++--
 drivers/net/wireless/ath9k/main.c  |    1 +
 drivers/net/wireless/ath9k/pci.c   |   18 ------------------
 5 files changed, 64 insertions(+), 21 deletions(-)

John, Greg,

these patches require a port down to the other stable kernels, I've taken the time to port only the necessary patches (the first two) to fix the SMP hang on each stable kernel and test them accordingly:

* 2.6.27
* 2.6.28
* 2.6.29

Once merged into wireless-testing the following patches can be used to push into Linus' tree:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.29/

Once merged into stable these can be used to merge into 2.6.27 and 2.6.28:

http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.27/
http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.28/

Quoted 8 lines

Just replace:

This is a port of commit SHA1 <BLEH>

With the respective SHA1.

I can also simply resend the ports for 27 and 28 once merged into Linus' tree. Whatever makes it easier for you.

Quoted 6 lines

Please, when the patches go into Linus's tree, send stable at kernel.org the backported patches, with the proper git commit id in them.

Otherwise it's just too much confusing work for me and I'll easily mess it up. The number of -stable patches is increasing and I need all the help I can get in handling this in a sane manner :)

Sure will do thanks,

Luis