ath5k/ath9k: Fix 64 bits TSF reads

According to tests, both TSF lower and upper registers kept counting, so the higher part could have been updated after the lower part has been read, as shown in the following log where the upper part is read first and the lower part next.

tsf = {00000003-fffffffd}
tsf = {00000003-00000001}
tsf = {00000004-0000000b}

This patch corrects this by checking that the upper part has not been changed while the lower part was read. It has been tested in an IBSS network where artifical IBSS merges have been done in order to trigger hundreds of rollover for the TSF lower part.

It follows the logic mentionned by Derek, with only 2 register reads needed at each additional steps instead of 3 (the minimum number of register reads is still 3).

---
 drivers/net/wireless/ath/ath5k/pcu.c |   31 +++++++++++++++++++++++++++++--
 drivers/net/wireless/ath/ath9k/hw.c  |   19 +++++++++++++++----
 2 files changed, 44 insertions(+), 6 deletions(-)

index 710870e..35ac13e 100644
@@ -496,6 +496,8 @@ void ath5k_hw_set_rx_filter(struct ath5k_hw *ah, u32 filter)
 * Beacon control *
 \****************/
 
+#define ATH5K_MAX_TSF_READ 10
+
 /**
  * ath5k_hw_get_tsf64 - Get the full 64bit TSF
  *

@@ -505,10 +507,35 @@ void ath5k_hw_set_rx_filter(struct ath5k_hw *ah, u32 filter)
  */
 u64 ath5k_hw_get_tsf64(struct ath5k_hw *ah)
 {
-	u64 tsf = ath5k_hw_reg_read(ah, AR5K_TSF_U32);
+	u32 tsf_lower, tsf_upper1, tsf_upper2;
+	int i;
+
+	/*
+	 * While reading TSF upper and then lower part, the clock is still
+	 * counting (or jumping in case of IBSS merge) so we might get
+	 * inconsistent values. To avoid this, we read the upper part again
+	 * and check it has not been changed. We make the hypothesis that a
+	 * maximum of 3 changes can happens in a row (we use 10 as a safe
+	 * value).
+	 * 
+	 * Impact on performance is pretty small, since in most cases, only
+	 * 3 register reads are needed.
+	 */
+
+	tsf_upper1 = ath5k_hw_reg_read(ah, AR5K_TSF_U32);
+	for (i = 0; i < ATH5K_MAX_TSF_READ; i++) {
+		tsf_lower = ath5k_hw_reg_read(ah, AR5K_TSF_L32);
+		tsf_upper2 = ath5k_hw_reg_read(ah, AR5K_TSF_U32);
+		if (tsf_upper2 == tsf_upper1)
+			break;
+		tsf_upper1 = tsf_upper2;
+	}
+
+	WARN_ON( i == ATH5K_MAX_TSF_READ );
+
 	ATH5K_TRACE(ah->ah_sc);
 
-	return ath5k_hw_reg_read(ah, AR5K_TSF_L32) | (tsf << 32);
+	return (((u64)tsf_upper1 << 32) | tsf_lower);
 }
 
 /**

index 0945452..370e5ed 100644
@@ -3606,14 +3606,25 @@ void ath9k_hw_write_associd(struct ath_hw *ah)
 }
 EXPORT_SYMBOL(ath9k_hw_write_associd);
 
+#define ATH9K_MAX_TSF_READ 10
+
 u64 ath9k_hw_gettsf64(struct ath_hw *ah)
 {
-	u64 tsf;
+	u32 tsf_lower, tsf_upper1, tsf_upper2;
+	int i;
+
+	tsf_upper1 = REG_READ(ah, AR_TSF_U32);
+	for (i = 0; i < ATH9K_MAX_TSF_READ; i++) {
+		tsf_lower = REG_READ(ah, AR_TSF_L32);
+		tsf_upper2 = REG_READ(ah, AR_TSF_U32);
+		if (tsf_upper2 == tsf_upper1)
+			break;
+		tsf_upper1 = tsf_upper2;
+	}
 
-	tsf = REG_READ(ah, AR_TSF_U32);
-	tsf = (tsf << 32) | REG_READ(ah, AR_TSF_L32);
+	WARN_ON( i == ATH9K_MAX_TSF_READ );
 
-	return tsf;
+	return (((u64)tsf_upper1 << 32) | tsf_lower);
 }
 EXPORT_SYMBOL(ath9k_hw_gettsf64);
 

1.5.6.5

I would prefer an approach whereas tsf_upper2 or tsf_upper1 is chosen based on whether tsf_lower is more or less than 0x80000000 if (tsf_upper2 - tsf_upper1) is 1. If the difference is not 0 or 1, either the hardware is broken or the kernel was stuck for so long (71 minutes!) that getting the exact tsf should be the least worry. That's when WARN_ON would be appropriate.

The problem with overengineered code is that it doesn't break when it's better to break and expose the problem :-)

But it's just a suggestion, not a NACK. It's better to have some fix than no fix at all.

Regards,
Pavel Roskin

Pavel Roskin a ?crit :

Hello Pavel,

You are considering rollover here, but the TSF can make big jumps as well (in case of IBSS merges). This later case can only be handled by the above code, to my knowledge. I am seeking consistency first and optimization next, not the opposite.

We can even consider the case where only the lower TSF has made a small jump before reading the higher part and the lower part. However, such case cannot be distinguished from a normal case and the resulting value will be consistent anyway (since the higher part has not changed).

In the case where tsf_upper2 = tsf_upper1 + 1, this can happen when a rollover occurs. It could also happens in case of IBSS merge, in which case, your logic won't work. Let's take an example:

real TSF 0x00000001-ffffffc0 => tsf_upper1 = 1 --- rollover ---
real TSF 0x00000001-ffffffc8 => tsf_lower = 0xffffffc8 real TSF 0x00000002-00000008 => tsf_upper2 = 2

In this case, we assume that TSF = 0x00000001-ffffffc8 (since 0xffffffc8

real TSF 0x00000001-10000009 => tsf_upper1 = 1 --- HW IBSS merge ---
real TSF 0x00000002-ffffffc8 => tsf_lower = 0xffffffc8 real TSF 0x00000002-ffffffd0 => tsf_upper2 = 2

In this case, we assume that TSF = 0x00000001-ffffffc8 ... which is wrong!

Did I miss something?

Regards,
Benoit

Quoting Benoit PAPILLAULT <[email protected]>:

OK, I didn't realize that. Then I'm fine with your patch. Sorry for the noise.

Regards,
Pavel Roskin

Hi,
The original code was wrong, and there must have been occasions when the
TSF was read incorrectly. Those occasions were infrequent, and would have show up in large networks that were operational on the month timescale.

Benoit's code tries 10 times to read the TSF, looking for when two consecutive upper TSF values that are the same. I can think of no physical scenario that would cause the 10 consecutive reads to not terminate.

If this fails to happen, then the TSF counter on the radio board is busted. If the TSF counter (or the reading of the TSF counter) is busted, then you have a bad situation, and something seriously wrong is happening. We need to know about this - so the kernel warnings are good.

Yes, but the problem with underengineered code is that it doesn't break, and the users of the code are blissfully unaware of serious problems.

I would not call this overengineered. I would call this the appropriate level of peer review to get stable code that is acceptably reliable.

Benoit's patch is good.
ACK.

Derek.
========================================================================

Derek Smithies Ph.D.
IndraNet Technologies Ltd.
Email: derek at indranet.co.nz
ph +64 3 365 6485
Web: http://www.indranet-technologies.com/