Skip to contentSkip to conversation

netdev: Add logging for CQM messages

PATCH Superseded Michael Johnson 2 participants 7 messages Aug 1, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 netdev: Add logging for CQM messages

Add extra logging around CQM events to help track wifi status.

The signal strength logging matches wpa_supplicant's which looks like this:

CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-60 noise=-96 txrate=6000

---
 src/netdev.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
src/netdev.c +20 -2
index 5a6a7b70..f9acd639 100644
@@ -1092,12 +1092,26 @@ static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev)
 					rssi_event = (uint32_t *) data;
 					break;
 
+				case NL80211_ATTR_CQM_PKT_LOSS_EVENT:
+					if (len != 4)
+						continue;
+
+					l_warn("Packets lost event: %d",
+							*(uint32_t *) data);
+					break;
+
+				case NL80211_ATTR_CQM_BEACON_LOSS_EVENT:
+					l_warn("Beacon lost event");
+					break;
+
 				case NL80211_ATTR_CQM_RSSI_LEVEL:
 					if (len != 4)
 						continue;
 
 					rssi_val = (int32_t *) data;
 					break;
+				default:
+					l_debug("Unknown CQM event: %d", type);
 				}
 			}
 
@@ -1106,10 +1120,14 @@ static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev)
 	}
 
 	if (rssi_event) {
-		if (rssi_val)
+		if (rssi_val) {
+			l_info("Signal change event (above=%d signal=%d)",
+							*rssi_event, *rssi_val);
 			netdev_cqm_event_rssi_value(netdev, *rssi_val);
-		else
+		} else {
+			l_info("Signal change event (above=%d)", *rssi_event);
 			netdev_cqm_event_rssi_threshold(netdev, *rssi_event);
+		}
 	}
 }
 
Signature · 1 line
2.25.1
#1 lore raw

PATCH 2/2 station: Log scan results during a roam

Provides useful information on why a roam might have failed, such as failing to find the BSS or the BSS being ranked lower, and why that might be.

The output format is the same as station_add_seen_bss for consistency.

---
 src/station.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
src/station.c +12 -1
index 02f21c76..e39e9d4b 100644
@@ -2286,6 +2286,7 @@ static bool station_roam_scan_notify(int err, struct l_queue *bss_list,
 	struct station *station = userdata;
 	struct network *network = station->connected_network;
 	struct handshake_state *hs = netdev_get_handshake(station->netdev);
+	struct scan_bss *current_bss = station->connected_bss;
 	struct scan_bss *bss;
 	struct scan_bss *best_bss = NULL;
 	double best_bss_rank = 0.0;
@@ -2317,9 +2318,20 @@ static bool station_roam_scan_notify(int err, struct l_queue *bss_list,
 	 * for BSSes that are within the FT Mobility Domain so as to favor
 	 * Fast Roaming, if it is supported.
 	 */
+	l_debug("Current BSS '%s' with SSID: %s",
+		util_address_to_string(current_bss->addr),
+		util_ssid_to_utf8(current_bss->ssid_len, current_bss->ssid));
 
 	while ((bss = l_queue_pop_head(bss_list))) {
 		double rank;
+		uint32_t kbps100 = DIV_ROUND_CLOSEST(bss->data_rate, 100000);
+
+		l_debug("Processing BSS '%s' with SSID: %s, freq: %u, rank: %u,"
+				" strength: %i, data_rate: %u.%u",
+				util_address_to_string(bss->addr),
+				util_ssid_to_utf8(bss->ssid_len, bss->ssid),
+				bss->frequency, bss->rank, bss->signal_strength,
+				kbps100 / 10, kbps100 % 10);
 
 		/* Skip the BSS we are connected to if doing an AP roam */
 		if (station->ap_directed_roaming && !memcmp(bss->addr,
@@ -2327,7 +2339,6 @@ static bool station_roam_scan_notify(int err, struct l_queue *bss_list,
 			goto next;
 
 		/* Skip result if it is not part of the ESS */
-
 		if (bss->ssid_len != hs->ssid_len ||
 				memcmp(bss->ssid, hs->ssid, hs->ssid_len))
 			goto next;
Signature · 1 line
2.25.1
#2 lore raw
In reply to Michael Johnson

Hi Michael,

Quoted 10 lines · Michael Johnson

Add extra logging around CQM events to help track wifi status.

The signal strength logging matches wpa_supplicant's which looks like this:

CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-60 noise=-96 txrate=6000

---
  src/netdev.c | 22 ++++++++++++++++++++--
  1 file changed, 20 insertions(+), 2 deletions(-)

<snip>

src/netdev.c static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev) · line 1124

Quoted 7 lines
src/netdev.c +2 -1
@@ -1106,10 +1120,14 @@ static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev)
  	}
 
  	if (rssi_event) {
-		if (rssi_val)
+		if (rssi_val) {
+			l_info("Signal change event (above=%d signal=%d)",

I'm not sure 'above=%d' is going to work. rssi_event is only set on THRESHOLD_EVENT_LOW/THRESHOLD_EVENT_HIGH. If RSSI_LEVEL is being reported, that pointer is still NULL.

src/netdev.c static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev) · line 1126

Quoted 2 lines
src/netdev.c +1 -0
+							*rssi_event, *rssi_val);
  			netdev_cqm_event_rssi_value(netdev, *rssi_val);

Hmm, I don't think wpa_s uses RSSI_LEVEL stuff in the kernel, does it? The reason I ask is that we do have UIs that setup RSSI level reporting for displaying connection link quality (in number of bars for example). Since such events are quite common, I think you're going to get quite a bit of spam in the logs. Are you only interested in cases where the roaming threshold is reached?
If so, you may need to modify netdev_cqm_event_rssi_value() instead.

src/netdev.c static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev) · line 1133

Quoted 8 lines
src/netdev.c +3 -1
-		else
+		} else {
+			l_info("Signal change event (above=%d)", *rssi_event);
  			netdev_cqm_event_rssi_threshold(netdev, *rssi_event);
+		}
  	}
  }
 

Patch 2 looks fine btw, so I applied that one.

Regards,
-Denis

#3 lore raw

Hi Denis,

Quoted 11 lines · Denis Kenzior

src/netdev.c static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev) · line 1124

Quoted 7 lines
src/netdev.c +2 -1
@@ -1106,10 +1120,14 @@ static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev)
      }
 
      if (rssi_event) {
-             if (rssi_val)
+             if (rssi_val) {
+                     l_info("Signal change event (above=%d signal=%d)",

I'm not sure 'above=%d' is going to work. rssi_event is only set on THRESHOLD_EVENT_LOW/THRESHOLD_EVENT_HIGH. If RSSI_LEVEL is being reported, that pointer is still NULL.

Should the top level if (rssi_event) { not cover that case? I believe it checks both
pointers before the info statement. I didn't add any output in the case of an RSSI_LEVEL but no THRESHOLD_EVENT_* as I haven't seen this actually occurring so assumed iwd didn't set up non-threshold events?

Quoted 10 lines

src/netdev.c static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev) · line 1126

Quoted 3 lines
src/netdev.c +1 -0
+                                                     *rssi_event, *rssi_val);
                      netdev_cqm_event_rssi_value(netdev, *rssi_val);

Hmm, I don't think wpa_s uses RSSI_LEVEL stuff in the kernel, does it? The reason I ask is that we do have UIs that setup RSSI level reporting for displaying connection link quality (in number of bars for example). Since such events are quite common, I think you're going to get quite a bit of spam in the logs. Are you only interested in cases where the roaming threshold is reached?
If so, you may need to modify netdev_cqm_event_rssi_value() instead.

Sorry, I should have included more reasoning behind the change.

In my use case the client is running headless and so the logs are really the only
insight into why the network might have degraded to the point of causing issues. For example, if the client stops receiving data, it is useful to know what the rough
signal level is that caused it as that might imply a dead zone or interference. All
debugging is done remotely as well so we can't ever debug the network while it is currently in a bad state hence the log part.

The useful data isn't just the threshold but the general signal level at the time of
an incident. If all is working then we would have roamed before the signal gets bad enough to cause issues (it follows that we don't really care about _good_ signal values so maybe we could reduce spam there?)

As for the general spamminess, does the UI RSSI reporting cause more events than running normally? I believe I was only seeing events when the signal had changed more than the hysteresis value in netdev_build_cmd_cqm_rssi_update (5 dBm). This doesn't seem too frequent (for me at least) when the client is mostly static. When it is moving it is generally useful information.

For wpa_supplicant, I'm not sure the exact mechanism it uses but it does report almost identical output at a fairly high frequency if the client is moving e.g.

```
May 08 08:56:04 p3-2019 wpa_supplicant[842]: wlp2s0:
CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-62 noise=-93 txrate=6000
May 08 08:56:14 p3-2019 wpa_supplicant[842]: wlp2s0:
CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-76 noise=-93 txrate=6000
May 08 08:56:27 p3-2019 wpa_supplicant[842]: wlp2s0:
CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-79 noise=-94 txrate=6000
```

Finally, the reason I made this info level was in the hope that I could remove the -d flag from iwd which does produce a lot of spam in the logs. This isn't really feasible at the moment as there isn't enough info logging to track down any issues that might occur on production systems where changes are harder to make. If the info level becomes to verbose, maybe iwd could set the default logging level to warn to reduce spam?

Quoted 10 lines

src/netdev.c static void netdev_cqm_event(struct l_genl_msg *msg, struct netdev *netdev) · line 1133

Quoted 9 lines
src/netdev.c +3 -1
-             else
+             } else {
+                     l_info("Signal change event (above=%d)", *rssi_event);
                      netdev_cqm_event_rssi_threshold(netdev, *rssi_event);
+             }
      }
  }
 
Quoted 4 lines

Patch 2 looks fine btw, so I applied that one.

Regards,
-Denis

Regards,
Michael

#4 lore raw

Hi Michael,

Quoted 9 lines
Quoted 3 lines

I'm not sure 'above=%d' is going to work. rssi_event is only set on THRESHOLD_EVENT_LOW/THRESHOLD_EVENT_HIGH. If RSSI_LEVEL is being reported, that pointer is still NULL.

Should the top level if (rssi_event) { not cover that case? I believe it checks both
pointers before the info statement. I didn't add any output in the case of an RSSI_LEVEL but no THRESHOLD_EVENT_* as I haven't seen this actually occurring so assumed iwd didn't set up non-threshold events?

Ah you're right. I guess the kernel does send the threshold event attribute as well. I'm really not sure why actually? This seems useless since it depends on knowing the currently used high/low thresholds, and only the kernel knows those.

Quoted 13 lines
Quoted 10 lines
Quoted 3 lines
+                                                     *rssi_event, *rssi_val);
                       netdev_cqm_event_rssi_value(netdev, *rssi_val);

Hmm, I don't think wpa_s uses RSSI_LEVEL stuff in the kernel, does it? The reason I ask is that we do have UIs that setup RSSI level reporting for displaying connection link quality (in number of bars for example). Since such events are quite common, I think you're going to get quite a bit of spam in the logs. Are you only interested in cases where the roaming threshold is reached?
If so, you may need to modify netdev_cqm_event_rssi_value() instead.

Sorry, I should have included more reasoning behind the change.

Quoted 9 lines

In my use case the client is running headless and so the logs are really the only
insight into why the network might have degraded to the point of causing issues. For example, if the client stops receiving data, it is useful to know what the rough
signal level is that caused it as that might imply a dead zone or interference. All
debugging is done remotely as well so we can't ever debug the network while it is currently in a bad state hence the log part.

I wonder if we should report this over D-Bus instead? I mean you can always setup your own SignalLevelAgent, or we could even add more reporting via StationDiagnostic interface?

Quoted 7 lines

The useful data isn't just the threshold but the general signal level at the time of
an incident. If all is working then we would have roamed before the signal gets bad enough to cause issues (it follows that we don't really care about _good_ signal values so maybe we could reduce spam there?)

Quoted 5 lines

As for the general spamminess, does the UI RSSI reporting cause more events than running normally? I believe I was only seeing events when the signal had changed more than the hysteresis value in netdev_build_cmd_cqm_rssi_update (5 dBm). This doesn't seem too frequent (for me at least) when the client is mostly static. When it is moving it is generally useful information.

It would. By default we program only the low signal threshold value. So you should only see signal changes around that value. If an rssi agent is used, then multiple thresholds are programmed, so you'd see rssi events even when the signal is very good.

Quoted 12 lines

For wpa_supplicant, I'm not sure the exact mechanism it uses but it does report almost identical output at a fairly high frequency if the client is moving e.g.

```
May 08 08:56:04 p3-2019 wpa_supplicant[842]: wlp2s0:
CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-62 noise=-93 txrate=6000
May 08 08:56:14 p3-2019 wpa_supplicant[842]: wlp2s0:
CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-76 noise=-93 txrate=6000
May 08 08:56:27 p3-2019 wpa_supplicant[842]: wlp2s0:
CTRL-EVENT-SIGNAL-CHANGE above=0 signal=-79 noise=-94 txrate=6000
```

Right, but I think it sets a single CQM threshold, not multiple like we (might) do. Also, wpa_s is very spammy since it is really (only) meant for testing wifi implementations in a lab ;)

Quoted 6 lines

Finally, the reason I made this info level was in the hope that I could remove the -d flag from iwd which does produce a lot of spam in the logs. This isn't really feasible at the moment as there isn't enough info logging to track down any issues that might occur on production systems where changes are harder to make. If the info level becomes to verbose, maybe iwd could set the default logging level to warn to reduce spam?

You can use a glob match for the -d flag. Something like:
#iwd -d '*Signal change event*'

So you can make the rssi messages as 'debug', and have iwd only print those you want. Multiple globs are supported too, separate them by ':' or ','.

Regards,
-Denis

#5 lore raw

Hi Denis,

Quoted 3 lines · Denis Kenzior

I wonder if we should report this over D-Bus instead? I mean you can always setup your own SignalLevelAgent, or we could even add more reporting via StationDiagnostic interface?

I actually do record the signal level separately but it's stored in an annoying binary format so access is more painful :(. This was just to make things easier and reduce the jump from wpa_supplicant to iwd for anyone on plain ubuntu etc. I can understand why you don't want the spam though.

Quoted 3 lines

Right, but I think it sets a single CQM threshold, not multiple like we (might) do. Also, wpa_s is very spammy since it is really (only) meant for testing wifi implementations in a lab ;)

haha fair. Of course, it's still the default on most linux systems ;).

Quoted 8 lines
Quoted 2 lines

changes are harder to make. If the info level becomes to verbose, maybe iwd could set the default logging level to warn to reduce spam?

You can use a glob match for the -d flag. Something like:
#iwd -d '*Signal change event*'

So you can make the rssi messages as 'debug', and have iwd only print those you want. Multiple globs are supported too, separate them by ':' or ','.

That's a smart idea although as a user i feel like you only ever know what messages are important after the fact.

Quoted 3 lines

Regards,
-Denis

What would you like to do with the patch? I'm happy to drop it if its too spammy or we can lower the level to debug? Or make it only print when the threshold changes.
Are you happy with the packet loss and beacon loss events or are they too much as well?

Regards,
Michael

#6 lore raw

Hi Michael,

Quoted 7 lines · Michael Johnson

Hi Denis,

Quoted 3 lines · Denis Kenzior

I wonder if we should report this over D-Bus instead? I mean you can always setup your own SignalLevelAgent, or we could even add more reporting via StationDiagnostic interface?

Quoted 5 lines

I actually do record the signal level separately but it's stored in an annoying binary format so access is more painful :(. This was just to make things easier and reduce the jump from wpa_supplicant to iwd for anyone on plain ubuntu etc. I can understand why you don't want the spam though.

I don't mind the spam in debug mode, that's what it is for. But not sure they really belong at info/warn levels. I think we can consider putting in roaming related messages at the info level since that type of information might be generally useful.

Quoted 7 lines
Quoted 4 lines

Right, but I think it sets a single CQM threshold, not multiple like we (might) do. Also, wpa_s is very spammy since it is really (only) meant for testing wifi implementations in a lab ;)

haha fair. Of course, it's still the default on most linux systems ;).

And we're working to change that.

Quoted 4 lines

What would you like to do with the patch? I'm happy to drop it if its too spammy or we can lower the level to debug? Or make it only print when the threshold changes.

I don't mind these at debug level, if you want to start with that.

Quoted 2 lines

Are you happy with the packet loss and beacon loss events or are they too much as well?

As I mentioned before, kernel will generally disconnect us as soon as the beacon loss event happens (depends on the STA capability if I recall correctly). Not sure how really useful this is, but since beacon loss goes via dmesg at 'info' level, I think it would be fine for iwd to do so as well.

The packet loss one I'm not sure. Certaintly not 'warn'. I think I'd start with 'debug' since this event isn't being reacted to (yet).

Regards,
-Denis