Skip to contentSkip to conversation

iwd 2.22 segfault

DISCUSSION Jules Maselbas 2 participants 4 messages Oct 3, 2024

lore
Share & tools
lore
#0 lore raw

Hi,

I am having a segfault in iwd 2.22, running on Alpine Linux (on edge).

I can reproduce the segfault by doing rc-service networking restart, dmesg gives this information:

iwd[4229]: segfault at a ip 00007f027ca94c6b sp 00007fffd6c75858 error 4 in ld-musl-x86_64.so.1[7f027ca44000+57000] likely on CPU 4 (core 2, socket 0)

Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48
...
iwd[24403]: segfault at a ip 00007fa91633ac6b sp 00007ffd1faaa028 error 4 in ld-musl-x86_64.so.1[7fa9162ea000+57000] likely on CPU 6 (core 3, socket 0)
Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48

This is not an issue in musl-libc, but a call to memcpy with a bad address,
we can see that the source address is 0xa (10) which is also the offset of aa field in the netdev->handshake struct which makes me think that handshake is null when netdev_rssi_poll is called.

Here is a backtrace when iwd segfault:
(gdb) bt

#0  memcpy () at src/string/x86_64/memcpy.s:22
#1  0x00005555555fc0dc in memcpy (__od=<optimized out>, __os=0xa, __n=6) at /usr/include/fortify/string.h:55
#2  l_netlink_message_append (message=0x7ffff7f34050, type=type@entry=6, data=0xa, len=len@entry=6) at ell/netlink.c:841
#3  0x00005555555fd78f in l_genl_msg_append_attr (msg=msg@entry=0x7ffff7f34020, type=type@entry=6, len=len@entry=6, data=<optimized out>) at ell/genl.c:1518
#4  0x0000555555559080 in netdev_rssi_poll (timeout=<optimized out>, user_data=0x7ffff7f38dc0) at src/netdev.c:760
#5  0x00005555555f959e in timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:69
#6  timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:58
#7  0x00005555555f8a75 in l_main_iterate (timeout=<optimized out>) at ell/main.c:461
#8  0x00005555555f8b4e in l_main_run () at ell/main.c:508
#9  l_main_run () at ell/main.c:490

#10 0x00005555555f8d7f in l_main_run_with_signal (callback=callback@entry=0x5555555587f0 <signal_handler>, user_data=user_data@entry=0x0) at ell/main.c:630 #11 0x0000555555557bd0 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:614

I've also ran a git bisect which points to 154a29be0552f5a39e34301ebaf24623d64073da netdev: fall back to RSSI polling if SET_CQM fails as the first bad commit. I noticed the "rssi" word is also present in the stacktrace.

I am using the following wifi card:
03:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
driver: mt7921e
version: 6.6.53-0-lts
firmware-version: ____000000-20240716163327
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Cheers,
Jules

#1 lore raw

Hi Jules,

Quoted 29 lines · Jules Maselbas

Hi,

I am having a segfault in iwd 2.22, running on Alpine Linux (on edge).

I can reproduce the segfault by doing rc-service networking restart, dmesg gives this information:

iwd[4229]: segfault at a ip 00007f027ca94c6b sp 00007fffd6c75858 error 4 in ld-musl-x86_64.so.1[7f027ca44000+57000] likely on CPU 4 (core 2, socket 0)

Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48
...
iwd[24403]: segfault at a ip 00007fa91633ac6b sp 00007ffd1faaa028 error 4 in ld-musl-x86_64.so.1[7fa9162ea000+57000] likely on CPU 6 (core 3, socket 0)
Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48

This is not an issue in musl-libc, but a call to memcpy with a bad address,
we can see that the source address is 0xa (10) which is also the offset of aa field in the netdev->handshake struct which makes me think that handshake is null when netdev_rssi_poll is called.

Here is a backtrace when iwd segfault:
(gdb) bt

#0  memcpy () at src/string/x86_64/memcpy.s:22
#1  0x00005555555fc0dc in memcpy (__od=<optimized out>, __os=0xa, __n=6) at /usr/include/fortify/string.h:55
#2  l_netlink_message_append (message=0x7ffff7f34050, type=type@entry=6, data=0xa, len=len@entry=6) at ell/netlink.c:841
#3  0x00005555555fd78f in l_genl_msg_append_attr (msg=msg@entry=0x7ffff7f34020, type=type@entry=6, len=len@entry=6, data=<optimized out>) at ell/genl.c:1518
#4  0x0000555555559080 in netdev_rssi_poll (timeout=<optimized out>, user_data=0x7ffff7f38dc0) at src/netdev.c:760
#5  0x00005555555f959e in timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:69
#6  timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:58
#7  0x00005555555f8a75 in l_main_iterate (timeout=<optimized out>) at ell/main.c:461
#8  0x00005555555f8b4e in l_main_run () at ell/main.c:508
#9  l_main_run () at ell/main.c:490
Quoted 7 lines

#10 0x00005555555f8d7f in l_main_run_with_signal (callback=callback@entry=0x5555555587f0 <signal_handler>, user_data=user_data@entry=0x0) at ell/main.c:630 #11 0x0000555555557bd0 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:614

I've also ran a git bisect which points to 154a29be0552f5a39e34301ebaf24623d64073da netdev: fall back to RSSI polling if SET_CQM fails as the first bad commit. I noticed the "rssi" word is also present in the stacktrace.

Thanks for such detailed info, do you happen to have debug logs when this happens? I'm just trying to see the code path which leads to this. I can't seem to reproduce it but I suspect musl-libc is just different enough that its exposing the bug

Quoted 15 lines

I am using the following wifi card:
03:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
driver: mt7921e
version: 6.6.53-0-lts
firmware-version: ____000000-20240716163327
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Quoted 4 lines

Cheers,
Jules

Thanks,

James

#2 lore raw

Hi Jules,

Quoted 58 lines · James Prestwood

Hi Jules,

Quoted 33 lines · Jules Maselbas

Hi,

I am having a segfault in iwd 2.22, running on Alpine Linux (on edge).

I can reproduce the segfault by doing rc-service networking restart, dmesg gives this information:

iwd[4229]: segfault at a ip 00007f027ca94c6b sp 00007fffd6c75858 error 4 in ld-musl-x86_64.so.1[7f027ca44000+57000] likely on CPU 4 (core 2, socket 0)

Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48
...
iwd[24403]: segfault at a ip 00007fa91633ac6b sp 00007ffd1faaa028 error 4 in ld-musl-x86_64.so.1[7fa9162ea000+57000] likely on CPU 6 (core 3, socket 0)
Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48

This is not an issue in musl-libc, but a call to memcpy with a bad address,
we can see that the source address is 0xa (10) which is also the offset of aa field in the netdev->handshake struct which makes me think that handshake is null when netdev_rssi_poll is called.

Here is a backtrace when iwd segfault:
(gdb) bt
#0  memcpy () at src/string/x86_64/memcpy.s:22
#1  0x00005555555fc0dc in memcpy (__od=<optimized out>, __os=0xa, __n=6) at /usr/include/fortify/string.h:55
#2  l_netlink_message_append (message=0x7ffff7f34050, type=type@entry=6, data=0xa, len=len@entry=6) at ell/netlink.c:841
#3  0x00005555555fd78f in l_genl_msg_append_attr (msg=msg@entry=0x7ffff7f34020, type=type@entry=6, len=len@entry=6, data=<optimized out>) at ell/genl.c:1518 #4  0x0000555555559080 in netdev_rssi_poll (timeout=<optimized out>, user_data=0x7ffff7f38dc0) at src/netdev.c:760
#5  0x00005555555f959e in timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:69 #6  timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:58
#7  0x00005555555f8a75 in l_main_iterate (timeout=<optimized out>) at ell/main.c:461
#8  0x00005555555f8b4e in l_main_run () at ell/main.c:508
#9  l_main_run () at ell/main.c:490
#10 0x00005555555f8d7f in l_main_run_with_signal (callback=callback@entry=0x5555555587f0 <signal_handler>, user_data=user_data@entry=0x0) at ell/main.c:630 #11 0x0000555555557bd0 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:614

Quoted 3 lines

I've also ran a git bisect which points to 154a29be0552f5a39e34301ebaf24623d64073da netdev: fall back to RSSI polling if SET_CQM fails as the first bad commit. I noticed the "rssi" word is also present in the stacktrace.

Thanks for such detailed info, do you happen to have debug logs when this happens? I'm just trying to see the code path which leads to this. I can't seem to reproduce it but I suspect musl-libc is just different enough that its exposing the bug

Quoted 15 lines

I am using the following wifi card:
03:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
driver: mt7921e
version: 6.6.53-0-lts
firmware-version: ____000000-20240716163327
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Quoted 8 lines
Quoted 4 lines

Cheers,
Jules

Thanks,

James

I was finally able to reproduce it. Its completely timing dependent and I think if IWD gets restarted _just_ before the timer fires it will crash. The easiest way to reproduce it was to just disconnect with iwctl. Anyways, I sent a patch to the list which should fix it. If you have the ability to try it out to confirm that would be great!

Thanks,

James

#3 lore raw

Hi,

Quoted 5 lines · James Prestwood

I was finally able to reproduce it. Its completely timing dependent and I think if IWD gets restarted _just_ before the timer fires it will crash. The easiest way to reproduce it was to just disconnect with iwctl. Anyways, I sent a patch to the list which should fix it. If you have the ability to try it out to confirm that would be great!

It did fix the issue thanks!
I also was able to reproduced the issue with iwctl station wlan0 disconnect before applying the patch.