iwd 2.22 segfault
Hi,
I am having a segfault in iwd 2.22, running on Alpine Linux (on edge).
I can reproduce the segfault by doing rc-service networking restart, dmesg gives this information:
iwd[4229]: segfault at a ip 00007f027ca94c6b sp 00007fffd6c75858 error 4 in ld-musl-x86_64.so.1[7f027ca44000+57000] likely on CPU 4 (core 2, socket 0)
Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48 ... iwd[24403]: segfault at a ip 00007fa91633ac6b sp 00007ffd1faaa028 error 4 in ld-musl-x86_64.so.1[7fa9162ea000+57000] likely on CPU 6 (core 3, socket 0) Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48
This is not an issue in musl-libc, but a call to memcpy with a bad address,
we can see that the source address is 0xa (10) which is also the offset of aa field in the netdev->handshake struct which makes me think that handshake is null when netdev_rssi_poll is called.
Here is a backtrace when iwd segfault:
(gdb) bt
#0 memcpy () at src/string/x86_64/memcpy.s:22 #1 0x00005555555fc0dc in memcpy (__od=<optimized out>, __os=0xa, __n=6) at /usr/include/fortify/string.h:55 #2 l_netlink_message_append (message=0x7ffff7f34050, type=type@entry=6, data=0xa, len=len@entry=6) at ell/netlink.c:841 #3 0x00005555555fd78f in l_genl_msg_append_attr (msg=msg@entry=0x7ffff7f34020, type=type@entry=6, len=len@entry=6, data=<optimized out>) at ell/genl.c:1518 #4 0x0000555555559080 in netdev_rssi_poll (timeout=<optimized out>, user_data=0x7ffff7f38dc0) at src/netdev.c:760 #5 0x00005555555f959e in timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:69 #6 timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:58 #7 0x00005555555f8a75 in l_main_iterate (timeout=<optimized out>) at ell/main.c:461 #8 0x00005555555f8b4e in l_main_run () at ell/main.c:508 #9 l_main_run () at ell/main.c:490
#10 0x00005555555f8d7f in l_main_run_with_signal (callback=callback@entry=0x5555555587f0 <signal_handler>, user_data=user_data@entry=0x0) at ell/main.c:630 #11 0x0000555555557bd0 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:614
I've also ran a git bisect which points to 154a29be0552f5a39e34301ebaf24623d64073da netdev: fall back to RSSI polling if SET_CQM fails as the first bad commit. I noticed the "rssi" word is also present in the stacktrace.
I am using the following wifi card:
03:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
driver: mt7921e
version: 6.6.53-0-lts
firmware-version: ____000000-20240716163327
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Cheers,
Jules
Hi Jules,
Quoted 29 lines · Jules Maselbas
Hi,
I am having a segfault in iwd 2.22, running on Alpine Linux (on edge).
I can reproduce the segfault by doing
rc-service networking restart, dmesg gives this information:iwd[4229]: segfault at a ip
00007f027ca94c6bsp00007fffd6c75858error 4 in ld-musl-x86_64.so.1[7f027ca44000+57000] likely on CPU 4 (core 2, socket 0)Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48 ... iwd[24403]: segfault at a ip 00007fa91633ac6b sp 00007ffd1faaa028 error 4 in ld-musl-x86_64.so.1[7fa9162ea000+57000] likely on CPU 6 (core 3, socket 0) Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48This is not an issue in musl-libc, but a call to memcpy with a bad address,
we can see that the source address is0xa(10) which is also the offset ofaafield in thenetdev->handshakestruct which makes me think that handshake is null whennetdev_rssi_pollis called.Here is a backtrace when iwd segfault:
(gdb) bt#0 memcpy () at src/string/x86_64/memcpy.s:22 #1 0x00005555555fc0dc in memcpy (__od=<optimized out>, __os=0xa, __n=6) at /usr/include/fortify/string.h:55 #2 l_netlink_message_append (message=0x7ffff7f34050, type=type@entry=6, data=0xa, len=len@entry=6) at ell/netlink.c:841 #3 0x00005555555fd78f in l_genl_msg_append_attr (msg=msg@entry=0x7ffff7f34020, type=type@entry=6, len=len@entry=6, data=<optimized out>) at ell/genl.c:1518 #4 0x0000555555559080 in netdev_rssi_poll (timeout=<optimized out>, user_data=0x7ffff7f38dc0) at src/netdev.c:760 #5 0x00005555555f959e in timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:69 #6 timeout_callback (fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) at ell/timeout.c:58 #7 0x00005555555f8a75 in l_main_iterate (timeout=<optimized out>) at ell/main.c:461 #8 0x00005555555f8b4e in l_main_run () at ell/main.c:508 #9 l_main_run () at ell/main.c:490
Quoted 7 lines
#10
0x00005555555f8d7finl_main_run_with_signal(callback=callback@entry=0x5555555587f0 <signal_handler>, user_data=user_data@entry=0x0) atell/main.c:630#110x0000555555557bd0in main (argc=<optimized out>, argv=<optimized out>) atsrc/main.c:614I've also ran a git bisect which points to
154a29be0552f5a39e34301ebaf24623d64073danetdev: fall back to RSSI polling ifSET_CQMfails as the first bad commit. I noticed the "rssi" word is also present in the stacktrace.
Thanks for such detailed info, do you happen to have debug logs when this happens? I'm just trying to see the code path which leads to this. I can't seem to reproduce it but I suspect musl-libc is just different enough that its exposing the bug
Quoted 15 lines
I am using the following wifi card:
03:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
driver: mt7921e
version: 6.6.53-0-lts
firmware-version: ____000000-20240716163327
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Quoted 4 lines
Cheers,
Jules
Thanks,
James
Hi Jules,
Quoted 58 lines · James Prestwood
Hi Jules,
Quoted 33 lines · Jules Maselbas
Hi,
I am having a segfault in iwd 2.22, running on Alpine Linux (on edge).
I can reproduce the segfault by doing
rc-service networking restart, dmesg gives this information:iwd[4229]: segfault at a ip
00007f027ca94c6bsp00007fffd6c75858error 4 in ld-musl-x86_64.so.1[7f027ca44000+57000] likely on CPU 4 (core 2, socket 0)Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48 ... iwd[24403]: segfault at a ip 00007fa91633ac6b sp 00007ffd1faaa028 error 4 in ld-musl-x86_64.so.1[7fa9162ea000+57000] likely on CPU 6 (core 3, socket 0) Code: f8 48 83 fa 08 72 14 f7 c7 07 00 00 00 74 0c a4 48 ff ca f7 c7 07 00 00 00 75 f4 48 89 d1 48 c1 e9 03 f3 48 a5 83 e2 07 74 05 <a4> ff ca 75 fb c3 48 89 f8 48 29 f0 48 39 d0 0f 83 bf ff ff ff 48This is not an issue in musl-libc, but a call to memcpy with a bad address,
we can see that the source address is0xa(10) which is also the offset ofaafield in thenetdev->handshakestruct which makes me think that handshake is null whennetdev_rssi_pollis called.Here is a backtrace when iwd segfault:
(gdb) bt
#0 memcpy () atsrc/string/x86_64/memcpy.s:22
#10x00005555555fc0dcin memcpy (__od=<optimized out>, __os=0xa, __n=6) at/usr/include/fortify/string.h:55
#2l_netlink_message_append(message=0x7ffff7f34050, type=type@entry=6, data=0xa, len=len@entry=6) atell/netlink.c:841
#30x00005555555fd78finl_genl_msg_append_attr(msg=msg@entry=0x7ffff7f34020, type=type@entry=6, len=len@entry=6, data=<optimized out>) atell/genl.c:1518#40x0000555555559080innetdev_rssi_poll(timeout=<optimized out>, user_data=0x7ffff7f38dc0) atsrc/netdev.c:760
#50x00005555555f959eintimeout_callback(fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) atell/timeout.c:69#6timeout_callback(fd=<optimized out>, events=<optimized out>, user_data=0x7ffff7f363b0) atell/timeout.c:58
#70x00005555555f8a75inl_main_iterate(timeout=<optimized out>) atell/main.c:461
#80x00005555555f8b4einl_main_run() atell/main.c:508
#9l_main_run() atell/main.c:490
#100x00005555555f8d7finl_main_run_with_signal(callback=callback@entry=0x5555555587f0 <signal_handler>, user_data=user_data@entry=0x0) atell/main.c:630#110x0000555555557bd0in main (argc=<optimized out>, argv=<optimized out>) atsrc/main.c:614Quoted 3 lines
I've also ran a git bisect which points to
154a29be0552f5a39e34301ebaf24623d64073danetdev: fall back to RSSI polling ifSET_CQMfails as the first bad commit. I noticed the "rssi" word is also present in the stacktrace.Thanks for such detailed info, do you happen to have debug logs when this happens? I'm just trying to see the code path which leads to this. I can't seem to reproduce it but I suspect musl-libc is just different enough that its exposing the bug
Quoted 15 lines
I am using the following wifi card:
03:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
driver: mt7921e
version: 6.6.53-0-lts
firmware-version: ____000000-20240716163327
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Quoted 8 lines
Quoted 4 lines
Cheers,
JulesThanks,
James
I was finally able to reproduce it. Its completely timing dependent and I think if IWD gets restarted _just_ before the timer fires it will crash. The easiest way to reproduce it was to just disconnect with iwctl. Anyways, I sent a patch to the list which should fix it. If you have the ability to try it out to confirm that would be great!
Thanks,
James
Hi,
Quoted 5 lines · James Prestwood
I was finally able to reproduce it. Its completely timing dependent and I think if IWD gets restarted _just_ before the timer fires it will crash. The easiest way to reproduce it was to just disconnect with iwctl. Anyways, I sent a patch to the list which should fix it. If you have the ability to try it out to confirm that would be great!
It did fix the issue thanks!
I also was able to reproduced the issue with iwctl station wlan0 disconnect before applying the patch.