cfg80211: Fix support for flushing old scan results

__cfg80211_bss_expire function was incorrectly used to flush the BSS entries from the previous scan results, causing NL80211_SCAN_FLAG_FLUSH flag to have no effect.

Quoted 3 lines · Tim Kourt

__cfg80211_bss_expire function was incorrectly used to flush the BSS entries from the previous scan results, causing NL80211_SCAN_FLAG_FLUSH flag to have no effect.

Hmm. I guess I'm not convinced - what's the bug?

We flush anything that's older than our start, so that should work just fine?

Hi Johannes,

Quoted 10 lines · Johannes Berg

Quoted 3 lines · Tim Kourt

__cfg80211_bss_expire function was incorrectly used to flush the BSS entries from the previous scan results, causing NL80211_SCAN_FLAG_FLUSH flag to have no effect.

Hmm. I guess I'm not convinced - what's the bug?

We flush anything that's older than our start, so that should work just fine?

Just FYI, there's definitely something funny with the scanning code:

denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush BSS 10:c3:7b:54:74:d4(on wlp2s0)

     last seen: 274.815s [boottime]
     freq: 5765
     beacon interval: 100 TUs
     signal: -35.00 dBm
     last seen: 349 ms ago
     Information elements from Probe Response frame:
     SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

Then if I try:
denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush ssid myssid BSS 10:c3:7b:54:74:d4(on wlp2s0)

     last seen: 319.667s [boottime]
     freq: 5765
     beacon interval: 100 TUs
     signal: -42.00 dBm
     last seen: 350 ms ago
     Information elements from Probe Response frame:
     SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

....
BSS 10:c3:7b:54:74:d4(on wlp2s0)
last seen: 319.662s [boottime]
freq: 5765
beacon interval: 100 TUs
signal: -37.00 dBm
last seen: 355 ms ago
Information elements from Probe Response frame:
SSID: myssid

Shouldn't the second scan give a single result from that one BSS?

Quoted 47 lines · Denis Kenzior

Hi Johannes,

Quoted 10 lines · Johannes Berg

Quoted 3 lines · Tim Kourt

__cfg80211_bss_expire function was incorrectly used to flush the BSS entries from the previous scan results, causing NL80211_SCAN_FLAG_FLUSH flag to have no effect.

Hmm. I guess I'm not convinced - what's the bug?

We flush anything that's older than our start, so that should work just fine?

Just FYI, there's definitely something funny with the scanning code:

denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush BSS 10:c3:7b:54:74:d4(on wlp2s0)
     last seen: 274.815s [boottime]
     freq: 5765
     beacon interval: 100 TUs
     signal: -35.00 dBm
     last seen: 349 ms ago
     Information elements from Probe Response frame:
     SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

Then if I try:
denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush ssid myssid BSS 10:c3:7b:54:74:d4(on wlp2s0)
     last seen: 319.667s [boottime]
     freq: 5765
     beacon interval: 100 TUs
     signal: -42.00 dBm
     last seen: 350 ms ago
     Information elements from Probe Response frame:
     SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00
....
BSS 10:c3:7b:54:74:d4(on wlp2s0)
     last seen: 319.662s [boottime]
     freq: 5765
     beacon interval: 100 TUs
     signal: -37.00 dBm
     last seen: 355 ms ago
     Information elements from Probe Response frame:
     SSID: myssid

Quoted 1 line

Shouldn't the second scan give a single result from that one BSS?

Looking at the 'last seen' values it does look ok. Both results have the same BSSID, but the first one shows the broadcast ssid (or so it seems).

Neither iw nor nl80211 on the kernel side add the broadcast ssid. So question is what device are you using and does it use mac80211 software

scanning or hardware scanning. I did not dive into mac80211 to see if the broadcast ssid is added there.

Hi Arend,

Quoted 51 lines · Arend van Spriel

Quoted 49 lines · Denis Kenzior

Hi Johannes,

Quoted 11 lines · Johannes Berg

Quoted 4 lines · Tim Kourt

__cfg80211_bss_expire function was incorrectly used to flush the BSS entries from the previous scan results, causing NL80211_SCAN_FLAG_FLUSH
flag to have no effect.

Hmm. I guess I'm not convinced - what's the bug?

We flush anything that's older than our start, so that should work just fine?

Just FYI, there's definitely something funny with the scanning code:

denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush BSS 10:c3:7b:54:74:d4(on wlp2s0)

     last seen: 274.815s [boottime]
     freq: 5765
     beacon interval: 100 TUs
     signal: -35.00 dBm
     last seen: 349 ms ago
     Information elements from Probe Response frame:
     SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

Then if I try:
denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush ssid myssid BSS 10:c3:7b:54:74:d4(on wlp2s0)

     last seen: 319.667s [boottime]
     freq: 5765
     beacon interval: 100 TUs
     signal: -42.00 dBm
     last seen: 350 ms ago
     Information elements from Probe Response frame:
     SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

....
BSS 10:c3:7b:54:74:d4(on wlp2s0)
last seen: 319.662s [boottime]
freq: 5765
beacon interval: 100 TUs
signal: -37.00 dBm
last seen: 355 ms ago
Information elements from Probe Response frame:
SSID: myssid

Shouldn't the second scan give a single result from that one BSS?

Quoted 3 lines

Looking at the 'last seen' values it does look ok. Both results have the same BSSID, but the first one shows the broadcast ssid (or so it seems).

Are you saying the first result is from the Beacon and the other is from the Probe Response? Then why are the 'Information elements from Probe Response frame' the way they are?

Quoted 2 lines

Neither iw nor nl80211 on the kernel side add the broadcast ssid. So question is what device are you using and does it use mac80211 software

Intel 7260. We're seeing the same results with hwsim as well though. This was just a quick test to illustrate.

Quoted 2 lines

scanning or hardware scanning. I did not dive into mac80211 to see if the broadcast ssid is added there.

By the way, if you're interested. The same tests with a Broadcom based device wouldn't even find the hidden network. It would always come back with a single 'x00' SSID regardless of whether I added 'ssid myssid' at the end.

Quoted 3 lines

Are you saying the first result is from the Beacon and the other is from the Probe Response?  Then why are the 'Information elements from Probe Response frame' the way they are?

Nope. I am not saying that. I am saying that there are two probe requests being sent. One with broadcast ssid, ie. ssid_len == 0, and with ssid 'myssid'. But it is speculation without a sniffer capture.

Hi Arend,

Quoted 4 lines

Quoted 3 lines

Are you saying the first result is from the Beacon and the other is from the Probe Response? Then why are the 'Information elements from Probe Response frame' the way they are?

Quoted 3 lines

Nope. I am not saying that. I am saying that there are two probe requests being sent. One with broadcast ssid, ie. ssid_len == 0, and with ssid 'myssid'. But it is speculation without a sniffer capture.

Ah I see what you mean now. No, we traced this down to hostapd itself and it was receiving a single Probe Request with the ssid set and replying to it per spec. So I'm pretty confident this scenario isn't what is happening. Let me try to get some actual packet captures...

Quoted 6 lines · Denis Kenzior

Hi Arend,

Quoted 4 lines

Quoted 3 lines

Are you saying the first result is from the Beacon and the other is from the Probe Response? Then why are the 'Information elements from Probe Response frame' the way they are?

Quoted 8 lines

Quoted 3 lines

Nope. I am not saying that. I am saying that there are two probe requests being sent. One with broadcast ssid, ie. ssid_len == 0, and with ssid 'myssid'. But it is speculation without a sniffer capture.

Ah I see what you mean now. No, we traced this down to hostapd itself and it was receiving a single Probe Request with the ssid set and replying to it per spec. So I'm pretty confident this scenario isn't what is happening. Let me try to get some actual packet captures...

Was "myssid" the real SSID, or did you hide that from us and it was really 9 characters long in the original?

If it was really 9 characters long I could imagine that there's a different bug with a beacon with all-zero-bytes having been received (and getting stuck into the probe response buffer for some reason), and then you *should* see both entries.

Quoted 11 lines · Johannes Berg

Quoted 10 lines · Denis Kenzior

Hi Arend,

Quoted 7 lines

Quoted 3 lines

Are you saying the first result is from the Beacon and the other is from the Probe Response? Then why are the 'Information elements from Probe Response frame' the way they are?

Nope. I am not saying that. I am saying that there are two probe requests being sent. One with broadcast ssid, ie. ssid_len == 0, and with ssid 'myssid'. But it is speculation without a sniffer capture.

Quoted 7 lines

Quoted 4 lines

Ah I see what you mean now. No, we traced this down to hostapd itself and it was receiving a single Probe Request with the ssid set and replying to it per spec. So I'm pretty confident this scenario isn't what is happening. Let me try to get some actual packet captures...

Was "myssid" the real SSID, or did you hide that from us and it was really 9 characters long in the original?

Quoted 5 lines

If it was really 9 characters long I could imagine that there's a different bug with a beacon with all-zero-bytes having been received (and getting stuck into the probe response buffer for some reason), and then you *should* see both entries.

Or perhaps there's a bug with how we link the results between hidden/non-hidden, but it seems to me that hostapd would never have responded with a probe response with zeroed bytes.

Just FYI, there's definitely something funny with the scanning code:

denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush BSS 10:c3:7b:54:74:d4(on wlp2s0)

	last seen: 274.815s [boottime]
	freq: 5765
	beacon interval: 100 TUs
	signal: -35.00 dBm
	last seen: 349 ms ago
	Information elements from Probe Response frame:
	SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

Then if I try:
denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush ssid myssid BSS 10:c3:7b:54:74:d4(on wlp2s0)

	last seen: 319.667s [boottime]
	freq: 5765
	beacon interval: 100 TUs
	signal: -42.00 dBm
	last seen: 350 ms ago
	Information elements from Probe Response frame:
	SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00
....
BSS 10:c3:7b:54:74:d4(on wlp2s0)
	last seen: 319.662s [boottime]
	freq: 5765
	beacon interval: 100 TUs
	signal: -37.00 dBm
	last seen: 355 ms ago
	Information elements from Probe Response frame:
	SSID: myssid

Shouldn't the second scan give a single result from that one BSS?

Hi Denis,

Quoted 11 lines

Just FYI, there's definitely something funny with the scanning code:

denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush BSS 10:c3:7b:54:74:d4(on wlp2s0)

	last seen: 274.815s [boottime]
	freq: 5765
	beacon interval: 100 TUs
	signal: -35.00 dBm
	last seen: 349 ms ago
	Information elements from Probe Response frame:
	SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

This is already rather strange to start with. Can you provide a sniffer capture of this situation?

Thing is - the all-zero-bytes there points to using hidden SSID with a length of 9 characters, BUT
* "myssid" is just 6 characters long - or did you edit that?

* normally the zeroed-out SSID isn't transmitted in a *probe response* but only in beacons

denkenz@iwd-test ~ $ sudo iw dev wlp2s0 scan flush BSS 10:c3:7b:54:74:d4(on wlp2s0)

	last seen: 274.815s [boottime]
	freq: 5765
	beacon interval: 100 TUs
	signal: -35.00 dBm
	last seen: 349 ms ago
	Information elements from Probe Response frame:
	SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

I think I finally figured out what's going on. It's a mix between strange 'iw' behaviour, and strange backward-compatibility behaviour in cfg80211.

If you do this again and give the scan dump command explicitly with -b added, like

sudo iw dev wlp2s0 scan passive
iw dev wlp2s0 scan dump -b

then you'll likely see

BSS 10:c3:7b:54:74:d4(on wlp2s0)
last seen: 274.815s [boottime]
freq: 5765
beacon interval: 100 TUs
signal: -35.00 dBm
last seen: 349 ms ago
Information elements from Probe Response frame:
SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00

	[... ht/vht ...]
  	Information elements from Beacon frame:
  	SSID: \x00\x00\x00\x00\x00\x00\x00\x00\x00
	[... ht/vht ...]

So there are two things going on:

1) In iw, we limit the amount of information printed, but in that case still print "Information elements from Probe Response frame", if beacon IEs are also present in the BSS information.

2) Originally, the kernel didn't distinguish between probe response and beacon IE attributes, but only had "IEs" in general. When we later did want to distinguish, we changed nl80211 to unconditionally put some IEs into the "IEs", if received from beacon put them into "beacon IEs", but to avoid further duplication didn't introduce a separate "probe response IEs" attribute. You can see this in nl80211_send_bss():

         /* this pointer prefers to be pointed to probe response data
          * but is always valid
          */
         ies = rcu_dereference(res->ies);

         if (ies) {
                 if (nla_put_u64_64bit(msg, NL80211_BSS_TSF, ies->tsf,
                                       NL80211_BSS_PAD))
                         goto fail_unlock_rcu;
                 if (ies->len && nla_put(msg, NL80211_BSS_INFORMATION_ELEMENTS,
                                         ies->len, ies->data))
                         goto fail_unlock_rcu;
         }



         /* and this pointer is always (unless driver didn't know) beacon data */
         ies = rcu_dereference(res->beacon_ies);
         if (ies && ies->from_beacon) {
                 if (nla_put_u64_64bit(msg, NL80211_BSS_BEACON_TSF, ies->tsf,
                                       NL80211_BSS_PAD))
                         goto fail_unlock_rcu;
                 if (ies->len && nla_put(msg, NL80211_BSS_BEACON_IES,
                                         ies->len, ies->data))
                         goto fail_unlock_rcu;
         }

Ultimately, this is also lossy - so later I added NL80211_BSS_PRESP_DATA that indicates "these were *really* received by probe response.

I guess I should change iw in the following way:

https://p.sipsolutions.net/6958d16d00f955c7.txt

which will mean it will not print "probe response IEs" in a false positive way.

Obviously on kernels that don't set NL80211_BSS_PRESP_DATA it won't print it out even if the beacon/probe response were both received with different content, but it can't know if the kernel doesn't tell it.

So I need to absorb all of this some more, but I'm still wondering why we are seeing two separate scan entries (with hidden & plain ssid) when we requested a flush? Is there a way to force the kernel to only show us the probe responses.

It would seem that even with the flush flag set, there could still be spurious beacons getting in causing these results to appear in the scan result set, right?

Hi,

Quoted 4 lines

So I need to absorb all of this some more, but I'm still wondering why we are seeing two separate scan entries (with hidden & plain ssid) when we requested a flush? Is there a way to force the kernel to only show us the probe responses.

Oh. I didn't even think about this part, but that's just a consequence of having a hidden SSID. We need one entry to track the beacon, and then we add another entry for each hidden SSID it may have. Some APs implement multiple SSIDs that way (old Cisco equipment, IIRC, though then with zero-length SSID instead of zero-bytes), so you can't lump them together into a single BSS entry.

Quoted 3 lines

It would seem that even with the flush flag set, there could still be spurious beacons getting in causing these results to appear in the scan result set, right?

In general anything can, there's no filtering. So you might do a flush scan for a specific SSID and still get 20 (different) results, if 20 beacons happened to be received while you were scanning.

Okay, so we need to use NL80211_BSS_PRESP_DATA if we want to filter out scan results that are coming from beacons, right?

So what's the practical use of the flush flag? Or is that something that was meant to be 'for-testing-only'?

On May 22, 2018, at 3:52 PM, Johannes Berg <[email protected]> =

=20
On Tue, 2018-05-22 at 15:49 -0500, Denis Kenzior wrote:
=20

Quoted 1 line

Okay, so we need to use NL80211_BSS_PRESP_DATA if we want to filter =

Quoted 1 line

scan results that are coming from beacons, right?

=20
You could do that, yes. In non-hidden cases you get the beacon/probe response data combined, in hidden cases you may get the beacon data separately and the (second, third, ...) entry with probe response data will come with beacon data too.
=20
I'm not really sure what value that would have though. In general, you might want to not display hidden SSIDs as such or at all in the UI.

=20

Quoted 3 lines

So what's the practical use of the flush flag? Or is that something=20=

that was meant to be 'for-testing-only'?

=20
I think you misunderstand? The value is that it ensures that nothing =

present in the list that was received *before* the scan. =20

Quoted 6 lines

Quoted 2 lines

So what's the practical use of the flush flag? Or is that something that was meant to be 'for-testing-only'?

I think you misunderstand? The value is that it ensures that nothing is present in the list that was received *before* the scan.

We misunderstood how it was supposed to work, but no, I get how things work now.

I’m just curious what other potential uses this flag might have.

But in theory, I think you could've received the beacon with hidden SSID *before* the scan, yet it might be present in the scan results if the new scan caused the probe response to be associated with that scan.

Quoted 2 lines

I’m just curious what other potential uses this flag might have.

Well, basically, ensure that your scan data is up-to-date?

I think mostly it's because there are scenarios where the AP is expected to vary the probe response data, so you need to know if you
* have a new probe response, or
* didn't receive a new probe response, or
* the AP erroneously didn't change the new probe response.

Hi Johannes,

Quoted 3 lines

But in theory, I think you could've received the beacon with hidden SSID *before* the scan, yet it might be present in the scan results if the new scan caused the probe response to be associated with that scan.

Right, your explanation was helpful, thanks. It still seems completely weird and redundant that we get two separate entries though. The second entry with the probe response data still carries the beacon info (as you point out). Should the pure-beacon one be filtered?

Right, so thinking out loud here. Would it be useful to tell GET SCAN to only return entries with actual probe response data? Combined with the flush flag it seems like a much better fit for the cases you point out.

Quoted 6 lines · Denis Kenzior

Hi Johannes,

Quoted 3 lines

But in theory, I think you could've received the beacon with hidden SSID *before* the scan, yet it might be present in the scan results if the new scan caused the probe response to be associated with that scan.

Quoted 4 lines

Right, your explanation was helpful, thanks. It still seems completely weird and redundant that we get two separate entries though. The second entry with the probe response data still carries the beacon info (as you point out). Should the pure-beacon one be filtered?

I'm not sure. It still indicates that a hidden SSID was found, and in general even a real SSID on the same BSSID doesn't indicate that this was the only hidden SSID ...

Quoted 4 lines

Right, so thinking out loud here. Would it be useful to tell GET SCAN to only return entries with actual probe response data? Combined with the flush flag it seems like a much better fit for the cases you point out.

I don't really see much point in doing filtering in the kernel. It wouldn't doesn't hurt, but just trades off more kernel code for less transferred data - and that's mostly in this particular corner case, so not really an efficiency problem?

And if it wasn't a hidden SSID, then you probably do want to know about the non-hidden SSIDs that you picked up along the way. In fact, this will become crucial with OCE, since that results in cases where you don't even send a probe request if you've picked up certain things during the scan passively.

Quoted 3 lines

I'm not sure. It still indicates that a hidden SSID was found, and in general even a real SSID on the same BSSID doesn't indicate that this was the only hidden SSID ...

Right, but you still get that info conveyed through the Beacon IEs elements on the second/third/etc entry. So it still seems redundant to include the pure beacon one?

Also, given that you have to ask for the SSID you want specifically, what practical purpose does it serve to know that this wasn't the only hidden SSID? I mean you can see that hidden SSIDs are out there, run an active scan for the ones you can use. If none are there, you can just ignore that bssid...