Skip to contentSkip to conversation

iwd 1.21 stuck on connecting state

DISCUSSION 54d22547-afc0-4701-b1fc-5a2a2e411646 at simplelogin.co 3 participants 17 messages Jan 13, 2022 – Feb 3, 2022

lore
Share & tools
lore

54d22547-afc0-4701-b1fc-5a2a2e411646 at simplelogin.co

#0 lore raw

Hi,

I've recently started noticing on my laptop that iwd gets stuck on 'connecting' state when I boot a **second time** right after poweroff. The fix is quite easy. Just disconnect and connect again from iwctl.

I recall having a similar issue not long ago. Maybe a regression was introduced. For the record, router and connection settings are the same as before.

Arch Linux with kernel 5.16-zen1.

Thanks!

#1 lore raw

Hi,

Quoted 5 lines · On Thu, 2022-01-13 at 11:27 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…

Hi,

I've recently started noticing on my laptop that iwd gets stuck on 'connecting' state when I boot a **second time** right after poweroff. The fix is quite easy. Just disconnect and connect again from iwctl.

If its repeatable could you get an IWD log with debugging enabled, as well as an iwmon trace?

https://iwd.wiki.kernel.org/debugging

Quoted 11 lines

I recall having a similar issue not long ago. Maybe a regression was introduced. For the record, router and connection settings are the same as before.

Arch Linux with kernel 5.16-zen1.

Thanks!
_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org

#2 lore raw

Hi James,

I've gone through debugging and this is what I have: https://termbin.com/y9vq

iwd.service

# /usr/lib/systemd/system/iwd.service
[Unit]
Description=Wireless service
After=network-pre.target
Before=network.target
Wants=network.target

[Service]
Type=dbus
BusName=net.connman.iwd
ExecStart=/usr/lib/iwd/iwd
NotifyAccess=main
LimitNPROC=1
Restart=on-failure
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE PrivateTmp=true
NoNewPrivileges=true
DevicePolicy=closed
DeviceAllow=/dev/rfkill rw
ProtectHome=yes
ProtectSystem=strict
ProtectControlGroups=yes
ProtectKernelModules=yes
ConfigurationDirectory=iwd
StateDirectory=iwd
StateDirectoryMode=0700

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/iwd.service.d/override.conf
[Service]
RuntimeDirectory=resolvconf
ReadWritePaths=/etc/resolv.conf

/etc/iwd/main.conf

[General]
EnableNetworkConfiguration=true
AddressRandomization=once

[Network]
NameResolvingService=resolvconf

I've also tried to collect iwmon logs but they are huge, like almost 2 MB. Should I filter some of the info as the wiki shows (--nortnl --nowiphy --noscan).

Thanks!

#3 lore raw

Hi,

Since you have EnableNetworkConfiguration=true IWD is performing DHCP and waiting for that to succeed. I see further you tried again and got an IP so for some reason your DHCP server was not acting properly in the 'stuck in connecting case'.

You could try setting IWD_DHCP_DEBUG=1 as an environment variable and try again. That may give more insight into what DHCP is waiting for.

Thanks,
James

Quoted 59 lines · On Fri, 2022-01-14 at 20:08 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…

Hi James,

I've gone through debugging and this is what I have:
https://termbin.com/y9vq

iwd.service

# /usr/lib/systemd/system/iwd.service
[Unit]
Description=Wireless service
After=network-pre.target
Before=network.target
Wants=network.target

[Service]
Type=dbus
BusName=net.connman.iwd
ExecStart=/usr/lib/iwd/iwd
NotifyAccess=main
LimitNPROC=1
Restart=on-failure
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_NET_BIND_SERVICE PrivateTmp=true
NoNewPrivileges=true
DevicePolicy=closed
DeviceAllow=/dev/rfkill rw
ProtectHome=yes
ProtectSystem=strict
ProtectControlGroups=yes
ProtectKernelModules=yes
ConfigurationDirectory=iwd
StateDirectory=iwd
StateDirectoryMode=0700

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/iwd.service.d/override.conf
[Service]
RuntimeDirectory=resolvconf
ReadWritePaths=/etc/resolv.conf

/etc/iwd/main.conf

[General]
EnableNetworkConfiguration=true
AddressRandomization=once

[Network]
NameResolvingService=resolvconf

I've also tried to collect iwmon logs but they are huge, like almost 2 MB. Should I filter some of the info as the wiki shows (--nortnl -- nowiphy --noscan).

Thanks!
_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org

#4 lore raw

These are the logs with IWD_DHCP_DEBUG=1.

https://termbin.com/yn9k

On first attempt I didn't manage to get a connection due to DHCP, but after stopping/starting it worked.

#5 lore raw

Hi,

It looks like the DHCP server sent some packet that we did not accept. There isn't enough debug printing to know what exactly so we'll need to get a network capture PCAP and see what data it sent. You can use tcpdump/wireshark to get this. Just ensure you see DHCP packets in the capture.

Something like:

tcpdump -i <iface> port 67 or port 68 -s 0 -w dhcp.pcap

Thanks,
James

Quoted 9 lines · On Tue, 2022-01-18 at 11:43 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…

These are the logs with IWD_DHCP_DEBUG=1.

https://termbin.com/yn9k

On first attempt I didn't manage to get a connection due to DHCP, but after stopping/starting it worked.
_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org

#6 lore raw

Hi again,

I managed to trigger the problem while dumping tcp packets. Here is the result:

# ~ tcpdump -i wlan0 port 67 or port 68 -s 0 -w dhcp.pcap
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), snapshot length 262144 bytes ^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

# ~ tcpdump -r dhcp.pcap
reading from file dhcp.pcap, link-type EN10MB (Ethernet), snapshot length 262144
18:33:36.492009 IP _gateway.bootps > 192.168.1.35.bootpc: BOOTP/DHCP, Reply, length 284
18:33:36.492212 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:33:38.606330 IP _gateway.bootps > 192.168.1.36.bootpc: BOOTP/DHCP, Reply, length 284
18:33:40.021633 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:33:48.860117 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:05.360324 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:14.092945 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:14.705123 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:16.217430 IP _gateway.bootps > host.bootpc: BOOTP/DHCP, Reply, length 284
18:34:16.217613 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:16.313729 IP _gateway.bootps > host.bootpc: BOOTP/DHCP, Reply, length 284
18:34:16.320877 IP _gateway.bootps > host.bootpc: BOOTP/DHCP, Reply, length 284

So I stopped / started iwd.service and checked that connection was lost (no ping). Then started dumping packets while iwctl was constantly reporting 'connecting' state. From there I just disconnected and connected again successfully and aborted tcpdump.

#7 lore raw
Quoted 41 lines · On Thu, 2022-01-27 at 17:51 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…

Hi again,

I managed to trigger the problem while dumping tcp packets. Here is the result:

# ~ tcpdump -i wlan0 port 67 or port 68 -s 0 -w dhcp.pcap
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

# ~ tcpdump -r dhcp.pcap
reading from file dhcp.pcap, link-type EN10MB (Ethernet), snapshot length 262144
18:33:36.492009 IP _gateway.bootps > 192.168.1.35.bootpc: BOOTP/DHCP, Reply, length 284
18:33:36.492212 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:33:38.606330 IP _gateway.bootps > 192.168.1.36.bootpc: BOOTP/DHCP, Reply, length 284
18:33:40.021633 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:33:48.860117 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:05.360324 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:14.092945 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:14.705123 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:16.217430 IP _gateway.bootps > host.bootpc: BOOTP/DHCP, Reply, length 284
18:34:16.217613 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx (oui Unknown), length 300 18:34:16.313729 IP _gateway.bootps > host.bootpc: BOOTP/DHCP, Reply, length 284
18:34:16.320877 IP _gateway.bootps > host.bootpc: BOOTP/DHCP, Reply, length 284

Quoted 4 lines

So I stopped / started iwd.service and checked that connection was lost (no ping). Then started dumping packets while iwctl was constantly reporting 'connecting' state. From there I just disconnected and connected again successfully and aborted tcpdump.

Could you also attach the actual pcap file for inspection? Something in the packet is not validating but we have no way of knowing without seeing the actual packet contents.

Thanks,
James

Quoted 3 lines

_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org

#9 lore raw
Quoted 4 lines · On Sat, 2022-01-29 at 20:03 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…

Sorry, didn't know that. I reproduced the problem again and uploaded the file here: https://transfer.sh/get/jiEvOg/dhcp.pcap

Hope this helps!

Thanks for getting this. For some reason your DHCP server is sending two offers, and IWD is ignoring the second:

- DHCP offers 192.168.1.33
- IWD responds with a request for .33
- 2 seconds go by and DHCP offers 192.168.1.35
- IWD ignores offer as its already waiting for a reply from its original request
- IWD continues to retransmit requests for .35 but the server ignores them.

It seems the server wants .35 as this is what works later when (I assume) you restarted IWD.

A quick fix could be that we automatically restart DHCP after some timeout (same effect as you restarting IWD). The proper fix would be to handle this multiple offer case.

Out of curiosity what kind of network hardware are you using? Are you just using an off the shelf router or some custom openwrt/hostapd setup?

Thanks,
James

Quoted 3 lines

_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org

#10 lore raw

Hi James,

Quoted 8 lines

- DHCP offers 192.168.1.33
- IWD responds with a request for .33
- 2 seconds go by and DHCP offers 192.168.1.35
- IWD ignores offer as its already waiting for a reply from its original request
- IWD continues to retransmit requests for .35 but the server ignores them.

Quoted 6 lines

It seems the server wants .35 as this is what works later when (I assume) you restarted IWD.

A quick fix could be that we automatically restart DHCP after some timeout (same effect as you restarting IWD). The proper fix would be to handle this multiple offer case.

The second .35 offer seems to come from the same DHCP server as the first one. So we may be able to handle this case by assuming the current offer supersedes the last one we have from this server and restart the request.

Regards,
-Denis

#11 lore raw
In reply to James Prestwood
Quoted 3 lines

Out of curiosity what kind of network hardware are you using? Are you just using an off the shelf router or some custom openwrt/hostapd setup?

This is a router from my Internet provider. It's been with me for some years now but it's not the first time I experience a problem similar to this one with iwd.

Please take a look at this other issue from September 2021 also posted by me [1], just in case they both could share a root cause. Responses are not displayed on a single thread but they are easy to find on that month.

Thanks.
--
[1] https://lists.01.org/hyperkitty/list/iwd(a)lists.01.org/thread/K6FJL6B7UL3Z5OUPP4OEPFZPN6YNHNYC/

#12 lore raw
In reply to 54d22547-afc0-4701-b1fc-5a2a2e411646 at simplelogin.co

Hi,

Quoted 4 lines · On Sat, 2022-01-29 at 20:03 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…

Sorry, didn't know that. I reproduced the problem again and uploaded the file here: https://transfer.sh/get/jiEvOg/dhcp.pcap

Hope this helps!

Below is a patch for ELL [1]. If its possible for you to apply it and rebuild both IWD/ELL manually to test that would be great. The issue should be solved but you will see an additional print when you normally would see IWD hang and never get connected:

"Server sent another offer, using it instead"

Note: IWD_DHCP_DEBUG has to be enabled to see this.

If you are able to get a log showing this behavior we can mark this as fixed, otherwise we can wait till this patch is in a release and see if it fixed the issue then. Thanks for your help so far.

[1]
https://lists.01.org/hyperkitty/list/ell(a)lists.01.org/thread/DJHXIOGGOCGGFUD2E3N2XVXCUAO2X2UP/

Quoted 3 lines

_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org

#13 lore raw

Hi.

Quoted 4 lines

Below is a patch for ELL [1]. If its possible for you to apply it and rebuild both IWD/ELL manually to test that would be great. The issue should be solved but you will see an additional print when you normally would see IWD hang and never get connected:

I'm trying to apply the patch but I got several errors. I'm just copying and pasting it on a new file.

This is how I'm applying it: patch -d ell -p1 -i ../dhcp.patch

First it complained about being malformed:

patching file ell/dhcp.c
patch: **** malformed patch at line 72: void *userdata,

Just fixed it putting lines 71 and 72 together. Then all 3 hunks failed:

patching file ell/dhcp.c
Hunk #1 FAILED at 757.
Hunk #2 FAILED at 766.
Hunk #3 FAILED at 832.
3 out of 3 hunks FAILED -- saving rejects to file ell/dhcp.c.rej

I'm not used diff patching so probably I'm doing something wrong.

Quoted 2 lines

[1]
https://lists.01.org/hyperkitty/list/ell(a)lists.01.org/thread/DJHXIOGGOCGG...

#14 lore raw

Hi,

This actually got merged an hour or two ago:

https://git.kernel.org/pub/scm/libs/ell/ell.git/commit/?id=b02c2034d8f05b4d731196de51a31cf1ef282e1f

Quoted 35 lines · On Thu, 2022-02-03 at 16:05 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…

Hi.

Quoted 5 lines

Below is a patch for ELL [1]. If its possible for you to apply it and rebuild both IWD/ELL manually to test that would be great. The issue should be solved but you will see an additional print when you normally
would see IWD hang and never get connected:

I'm trying to apply the patch but I got several errors. I'm just copying and pasting it on a new file.

This is how I'm applying it: patch -d ell -p1 -i ../dhcp.patch

First it complained about being malformed:

patching file ell/dhcp.c
patch: **** malformed patch at line 72: void *userdata,

Just fixed it putting lines 71 and 72 together. Then all 3 hunks failed:

patching file ell/dhcp.c
Hunk #1 FAILED at 757.
Hunk #2 FAILED at 766.
Hunk #3 FAILED at 832.
3 out of 3 hunks FAILED -- saving rejects to file ell/dhcp.c.rej

I'm not used diff patching so probably I'm doing something wrong.

Quoted 3 lines

[1]
https://lists.01.org/hyperkitty/list/ell(a)lists.01.org/thread/DJHXIOGGOCGG
...

_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org

#15 lore raw
Quoted 1 line

This actually got merged an hour or two ago

I noticed right after posting. Classic hehe.

Issue seems solved. I get connectivity and according to logs iwd is discarding previous DHCP requests:

[DHCPv4] dhcp_client_send_discover:346
[DHCPv4] l_dhcp_client_start:1210 Entering state: DHCP_STATE_SELECTING [DHCPv4] dhcp_client_timeout_resend:526
[DHCPv4] dhcp_client_send_discover:346
[DHCPv4] dhcp_client_rx_message:832
[DHCPv4] dhcp_client_receive_offer:762
[DHCPv4] dhcp_client_handle_offer:805 Entering state: DHCP_STATE_REQUESTING [DHCPv4] dhcp_client_send_request:396
[DHCPv4] dhcp_client_rx_message:832
[DHCPv4] dhcp_client_receive_offer:762
[DHCPv4] dhcp_client_receive_offer:786 Server sent another offer, using it instead [DHCPv4] dhcp_client_handle_offer:805 Entering state: DHCP_STATE_REQUESTING [DHCPv4] dhcp_client_send_request:396
[DHCPv4] dhcp_client_rx_message:832
[DHCPv4] dhcp_client_receive_ack:684
[DHCPv4] dhcp_client_rx_message:905 Entering state: DHCP_STATE_BOUND [DHCPv4] dhcp_client_rx_message:939 T1 expiring in 21600784 ms
[DHCPv4] l_acd_start:447 Skipping probes and sending announcements
[DHCPv4] announce_wait_timeout:166 No conflicts found for 192.168.1.39, announcing address [DHCPv4] acd_send_packet:146 sending packet with target IP 192.168.1.39

Would you create a release soon?

Thanks!

#16 lore raw
Quoted 7 lines · On Thu, 2022-02-03 at 20:10 +0000, 54d22547-afc0-4701-b1fc-5a2a2e411646(a)simpl…
Quoted 1 line

This actually got merged an hour or two ago

I noticed right after posting. Classic hehe.

Issue seems solved. I get connectivity and according to logs iwd is discarding previous DHCP requests:

Quoted 14 lines

[DHCPv4] dhcp_client_send_discover:346
[DHCPv4] l_dhcp_client_start:1210 Entering state:
DHCP_STATE_SELECTING
[DHCPv4] dhcp_client_timeout_resend:526
[DHCPv4] dhcp_client_send_discover:346
[DHCPv4] dhcp_client_rx_message:832
[DHCPv4] dhcp_client_receive_offer:762
[DHCPv4] dhcp_client_handle_offer:805 Entering state:
DHCP_STATE_REQUESTING
[DHCPv4] dhcp_client_send_request:396
[DHCPv4] dhcp_client_rx_message:832
[DHCPv4] dhcp_client_receive_offer:762
[DHCPv4] dhcp_client_receive_offer:786 Server sent another offer, using it instead

Perfect, glad to see it worked.

Quoted 13 lines

[DHCPv4] dhcp_client_handle_offer:805 Entering state:
DHCP_STATE_REQUESTING
[DHCPv4] dhcp_client_send_request:396
[DHCPv4] dhcp_client_rx_message:832
[DHCPv4] dhcp_client_receive_ack:684
[DHCPv4] dhcp_client_rx_message:905 Entering state: DHCP_STATE_BOUND [DHCPv4] dhcp_client_rx_message:939 T1 expiring in 21600784 ms [DHCPv4] l_acd_start:447 Skipping probes and sending announcements [DHCPv4] announce_wait_timeout:166 No conflicts found for 192.168.1.39, announcing address
[DHCPv4] acd_send_packet:146 sending packet with target IP 192.168.1.39

Quoted 1 line

Would you create a release soon?

Yes, I think this is happening very soon.

Quoted 5 lines

Thanks!
_______________________________________________
iwd mailing list -- iwd(a)lists.01.org
To unsubscribe send an email to iwd-leave(a)lists.01.org