Skip to contentSkip to conversation

iwd 1.23 crashes in eapol_handle_pkt_1_of_4()

PATCH In review Toke Høiland-Jørgensen <toke at toke.dk> 3 participants 8 messages Feb 2, 2022 – Feb 3, 2022 1 message not in the archive

lore
Share & tools
lore

Toke Høiland-Jørgensen <toke at toke.dk>

#1 lore raw
In reply to a message not in the archive

Hi

Since updating to iwd 1.23 (from Arch packages), I've started seeing periodic crashes. I rebuilt iwd with debug symbols enabled and got the following backtrace out of 'coredumpctl gdb':

Core was generated by `/usr/lib/iwd/iwd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00005555f0ba8b3d in eapol_handle_ptk_1_of_4 (sm=sm(a)entry=0x5555f2b4a920, ek=0x5555f2b62588, ek(a)entry=0x16, unencrypted=unencrypted(a)entry=false) at src/eapol.c:1236
1236 handshake_event(sm->handshake,
(gdb) bt

#0  0x00005555f0ba8b3d in eapol_handle_ptk_1_of_4 (sm=sm(a)entry=0x5555f2b4a920, ek=0x5555f2b62588, ek(a)entry=0x16, unencrypted=unencrypted(a)entry=false) at src/eapol.c:1236
#1  0x00005555f0bab118 in eapol_key_handle (unencrypted=<optimized out>, frame=<optimized out>, sm=0x5555f2b4a920) at src/eapol.c:2343
#2  eapol_rx_packet (proto=<optimized out>, from=<optimized out>, frame=<optimized out>, unencrypted=<optimized out>, user_data=0x5555f2b4a920) at src/eapol.c:2665
#3  0x00005555f0bac497 in __eapol_rx_packet (ifindex=62, src=src(a)entry=0x5555f2b62574 "x\212 J\207\267", proto=proto(a)entry=34958, frame=frame(a)entry=0x5555f2b62588 "\002\003", 
    len=len(a)entry=121, noencrypt=noencrypt(a)entry=false) at src/eapol.c:3017
#4  0x00005555f0b8c617 in netdev_control_port_frame_event (netdev=0x5555f2b64450, msg=0x5555f2b62588) at src/netdev.c:5574
#5  netdev_unicast_notify (msg=msg(a)entry=0x5555f2b619a0, user_data=<optimized out>) at src/netdev.c:5613
#6  0x00007f60084c9a51 in dispatch_unicast_watches (msg=0x5555f2b619a0, id=<optimized out>, genl=0x5555f2b3fc80) at ell/genl.c:954
#7  process_unicast (nlmsg=0x7fff61abeac0, genl=0x5555f2b3fc80) at ell/genl.c:973
#8  received_data (io=<optimized out>, user_data=0x5555f2b3fc80) at ell/genl.c:1098
#9  0x00007f60084c61bd in io_callback (fd=<optimized out>, events=1, user_data=0x5555f2b3fd20) at ell/io.c:120

#10 0x00007f60084c536d in l_main_iterate (timeout=<optimized out>) at ell/main.c:478
#11 0x00007f60084c543e in l_main_run () at ell/main.c:525
#12 l_main_run () at ell/main.c:507
#13 0x00007f60084c5670 in l_main_run_with_signal (callback=callback(a)entry=0x5555f0b89150 <signal_handler>, user_data=user_data(a)entry=0x0) at ell/main.c:647 #14 0x00005555f0b886a4 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:532

Does anyone have any ideas? :)

Thanks!

-Toke

#2 lore raw
Quoted 4 lines · On 2 February 2022 13:05:56 CET Toke Høiland-Jørgensen wrote:

Since updating to iwd 1.23 (from Arch packages), I've started seeing periodic crashes.

Does anyone have any ideas? :)

According to https://bugs.debian.org/1004607, iwd 1.23 needs a newer version of ELL (version 0.48), so maybe you need to upgrade ELL too?

#3 lore raw
Quoted 8 lines · Diederik de Haas
Quoted 4 lines · On 2 February 2022 13:05:56 CET Toke Høiland-Jørgensen wrote:

Since updating to iwd 1.23 (from Arch packages), I've started seeing periodic crashes.

Does anyone have any ideas? :)

According to https://bugs.debian.org/1004607, iwd 1.23 needs a newer version of ELL (version 0.48), so maybe you need to upgrade ELL too?

Nope, that's not it - already running ell v0.48 :)

-Toke

#4 lore raw

PATCH 1/1 handshake: Do not crash if handshake is destroyed

In reply to Toke Høiland-Jørgensen <toke at toke.dk>

Commit 4d2176df2985 ("handshake: Allow event handler to free handshake") introduced a re-entrancy guard so that handshake_state objects that are destroyed as a result of the event do not cause a crash. It rightly used a temporary object to store the passed in handshake. Unfortunately this caused variable shadowing which resulted in crashes fixed by commit d22b174a7318 ("handshake: use _hs directly in handshake_event"). However, since the temporary was no longer used, this fix itself caused a crash:

#0 0x00005555f0ba8b3d in eapol_handle_ptk_1_of_4 (sm=sm(a)entry=0x5555f2b4a920, ek=0x5555f2b62588, ek(a)entry=0x16, unencrypted=unencrypted(a)entry=false) at src/eapol.c:1236
1236 handshake_event(sm->handshake,
(gdb) bt

 #0  0x00005555f0ba8b3d in eapol_handle_ptk_1_of_4 (sm=sm(a)entry=0x5555f2b4a920, ek=0x5555f2b62588, ek(a)entry=0x16, unencrypted=unencrypted(a)entry=false) at src/eapol.c:1236
 #1  0x00005555f0bab118 in eapol_key_handle (unencrypted=<optimized out>, frame=<optimized out>, sm=0x5555f2b4a920) at src/eapol.c:2343
 #2  eapol_rx_packet (proto=<optimized out>, from=<optimized out>, frame=<optimized out>, unencrypted=<optimized out>, user_data=0x5555f2b4a920) at src/eapol.c:2665
 #3  0x00005555f0bac497 in __eapol_rx_packet (ifindex=62, src=src(a)entry=0x5555f2b62574 "x\212 J\207\267", proto=proto(a)entry=34958, frame=frame(a)entry=0x5555f2b62588 "\002\003",
   len=len(a)entry=121, noencrypt=noencrypt(a)entry=false) at src/eapol.c:3017
 #4  0x00005555f0b8c617 in netdev_control_port_frame_event (netdev=0x5555f2b64450, msg=0x5555f2b62588) at src/netdev.c:5574
 #5  netdev_unicast_notify (msg=msg(a)entry=0x5555f2b619a0, user_data=<optimized out>) at src/netdev.c:5613
 #6  0x00007f60084c9a51 in dispatch_unicast_watches (msg=0x5555f2b619a0, id=<optimized out>, genl=0x5555f2b3fc80) at ell/genl.c:954
 #7  process_unicast (nlmsg=0x7fff61abeac0, genl=0x5555f2b3fc80) at ell/genl.c:973
 #8  received_data (io=<optimized out>, user_data=0x5555f2b3fc80) at ell/genl.c:1098
 #9  0x00007f60084c61bd in io_callback (fd=<optimized out>, events=1, user_data=0x5555f2b3fd20) at ell/io.c:120

#10 0x00007f60084c536d in l_main_iterate (timeout=<optimized out>) at ell/main.c:478
#11 0x00007f60084c543e in l_main_run () at ell/main.c:525
#12 l_main_run () at ell/main.c:507
#13 0x00007f60084c5670 in l_main_run_with_signal (callback=callback(a)entry=0x5555f0b89150 <signal_handler>, user_data=user_data(a)entry=0x0) at ell/main.c:647 #14 0x00005555f0b886a4 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:532

This happens when the driver does not support rekeying, which causes iwd to attempt a disconnect and re-connect. The disconnect action is taken during the event callback and destroys the underlying eapol state machine. Since a temporary isn't used, attempting to dereference sm->handshake results in a crash.

Fix this by introducing a UNIQUE_ID macro which should prevent shadowing and using a temporary variable as originally intended.

Fixes d22b174a7318 ("handshake: use _hs directly in handshake_event")
Fixes 4d2176df2985 ("handshake: Allow event handler to free handshake")
Reported-By Toke Høiland-Jørgensen <toke(a)toke.dk>
---
 src/handshake.h | 38 +++++++++++++++++++++-----------------
 src/util.h      |  4 ++++
 2 files changed, 25 insertions(+), 17 deletions(-)
src/handshake.h +21 -17
index 7136087a1a4a..7f597b06078b 100644
@@ -164,23 +164,27 @@ struct handshake_state {
 	handshake_event_func_t event_func;
 };
 
-#define handshake_event(_hs, event, ...)	\
-	(__extension__ ({	\
-		bool freed = false;	\
-	\
-		if ((_hs)->event_func && !(_hs)->in_event) {	\
-			(_hs)->in_event = true;	\
-			(_hs)->event_func((_hs), event, (_hs)->user_data,	\
-					##__VA_ARGS__); \
-	\
-			if (!(_hs)->in_event) {	\
-				handshake_state_free((_hs));	\
-				freed = true;	\
-			} else	\
-				(_hs)->in_event = false;	\
-		}	\
-		freed;	\
-	}))
+#define HSID(x) UNIQUE_ID(handshake_, x)
+
+#define handshake_event(_hs, event, ...)				\
+	({								\
+		bool HSID(freed) = false;				\
+		typeof(_hs) HSID(hs) = (_hs);				\
+									\
+		if (HSID(hs)->event_func && !HSID(hs)->in_event) {	\
+			HSID(hs)->in_event = true;			\
+			HSID(hs)->event_func(HSID(hs), (event),		\
+					HSID(hs)->user_data,		\
+					##__VA_ARGS__);			\
+									\
+			if (!HSID(hs)->in_event) {			\
+				handshake_state_free(HSID(hs));		\
+				HSID(freed) = true;			\
+			} else						\
+				HSID(hs)->in_event = false;		\
+		}							\
+		HSID(freed);						\
+	})
 
 void handshake_state_free(struct handshake_state *s);
 
src/util.h +4 -0
index b55a7ea7cd6b..75c29039b04a 100644
@@ -26,6 +26,10 @@
 #include <stdint.h>
 #include <unistd.h>
 
+#define ___PASTE(a, b) a ## b
+#define __PASTE(a, b) ___PASTE(a, b)
+#define UNIQUE_ID(x, id) __PASTE(__unique_prefix_, __PASTE(x, id))
+
 #define align_len(len, boundary) (((len)+(boundary)-1) & ~((boundary)-1))
 
 #define MAC "%02x:%02x:%02x:%02x:%02x:%02x"
Signature · 1 line
2.32.0
#6 lore raw
In reply to Toke Høiland-Jørgensen <toke at toke.dk>

Hi Toke,

Quoted 6 lines · Toke Høiland-Jørgensen

Hi

Since updating to iwd 1.23 (from Arch packages), I've started seeing periodic crashes. I rebuilt iwd with debug symbols enabled and got the following backtrace out of 'coredumpctl gdb':

*hangs his head in shame*. So I think we screwed up the same macro 2x now.

Can you please try the patch I just sent out?

Regards,
-Denis

#7 lore raw
Quoted 11 lines · Denis Kenzior

Hi Toke,

Quoted 6 lines · Toke Høiland-Jørgensen

Hi

Since updating to iwd 1.23 (from Arch packages), I've started seeing periodic crashes. I rebuilt iwd with debug symbols enabled and got the following backtrace out of 'coredumpctl gdb':

*hangs his head in shame*. So I think we screwed up the same macro 2x now.

Heh, yeah, did see there was a commit touching that macro, but didn't look closer.

Quoted 1 line

Can you please try the patch I just sent out?

Can do - thanks! Did a build with it incorporated and am running that now. Usually crashes once every hour (with striking regularity); will leave it running overnight and see if the crashes are gone by tomorrow :)

-Toke

Tested-by Toke Høiland-Jørgensen on v1 1/1
#5 lore raw

Re: [PATCH] handshake: Do not crash if handshake is destroyed

In reply to Denis Kenzior
Quoted 35 lines · Denis Kenzior

Commit 4d2176df2985 ("handshake: Allow event handler to free handshake") introduced a re-entrancy guard so that handshake_state objects that are destroyed as a result of the event do not cause a crash. It rightly used a temporary object to store the passed in handshake. Unfortunately this caused variable shadowing which resulted in crashes fixed by commit d22b174a7318 ("handshake: use _hs directly in handshake_event"). However, since the temporary was no longer used, this fix itself caused a crash:

#0 0x00005555f0ba8b3d in eapol_handle_ptk_1_of_4 (sm=sm(a)entry=0x5555f2b4a920, ek=0x5555f2b62588, ek(a)entry=0x16, unencrypted=unencrypted(a)entry=false) at src/eapol.c:1236
1236 handshake_event(sm->handshake,
(gdb) bt

 #0  0x00005555f0ba8b3d in eapol_handle_ptk_1_of_4 (sm=sm(a)entry=0x5555f2b4a920, ek=0x5555f2b62588, ek(a)entry=0x16, unencrypted=unencrypted(a)entry=false) at src/eapol.c:1236
 #1  0x00005555f0bab118 in eapol_key_handle (unencrypted=<optimized out>, frame=<optimized out>, sm=0x5555f2b4a920) at src/eapol.c:2343
 #2  eapol_rx_packet (proto=<optimized out>, from=<optimized out>, frame=<optimized out>, unencrypted=<optimized out>, user_data=0x5555f2b4a920) at src/eapol.c:2665
 #3  0x00005555f0bac497 in __eapol_rx_packet (ifindex=62, src=src(a)entry=0x5555f2b62574 "x\212 J\207\267", proto=proto(a)entry=34958, frame=frame(a)entry=0x5555f2b62588 "\002\003",
   len=len(a)entry=121, noencrypt=noencrypt(a)entry=false) at src/eapol.c:3017
 #4  0x00005555f0b8c617 in netdev_control_port_frame_event (netdev=0x5555f2b64450, msg=0x5555f2b62588) at src/netdev.c:5574
 #5  netdev_unicast_notify (msg=msg(a)entry=0x5555f2b619a0, user_data=<optimized out>) at src/netdev.c:5613
 #6  0x00007f60084c9a51 in dispatch_unicast_watches (msg=0x5555f2b619a0, id=<optimized out>, genl=0x5555f2b3fc80) at ell/genl.c:954
 #7  process_unicast (nlmsg=0x7fff61abeac0, genl=0x5555f2b3fc80) at ell/genl.c:973
 #8  received_data (io=<optimized out>, user_data=0x5555f2b3fc80) at ell/genl.c:1098
 #9  0x00007f60084c61bd in io_callback (fd=<optimized out>, events=1, user_data=0x5555f2b3fd20) at ell/io.c:120

#10 0x00007f60084c536d in l_main_iterate (timeout=<optimized out>) at ell/main.c:478
#11 0x00007f60084c543e in l_main_run () at ell/main.c:525
#12 l_main_run () at ell/main.c:507
#13 0x00007f60084c5670 in l_main_run_with_signal (callback=callback(a)entry=0x5555f0b89150 <signal_handler>, user_data=user_data(a)entry=0x0) at ell/main.c:647 #14 0x00005555f0b886a4 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:532

This happens when the driver does not support rekeying, which causes iwd to attempt a disconnect and re-connect. The disconnect action is taken during the event callback and destroys the underlying eapol state machine. Since a temporary isn't used, attempting to dereference sm->handshake results in a crash.

Quoted 6 lines

Fix this by introducing a UNIQUE_ID macro which should prevent shadowing and using a temporary variable as originally intended.

Fixes d22b174a7318 ("handshake: use _hs directly in handshake_event")
Fixes 4d2176df2985 ("handshake: Allow event handler to free handshake")
Reported-By Toke Høiland-Jørgensen <toke(a)toke.dk>

With this patch, the crashes are gone - thanks!

Tested-by Toke Høiland-Jørgensen <toke(a)toke.dk>
#8 lore raw
In reply to Toke Høiland-Jørgensen <toke at toke.dk>
Quoted 17 lines · Toke Høiland-Jørgensen
Quoted 11 lines · Denis Kenzior

Hi Toke,

Quoted 6 lines · Toke Høiland-Jørgensen

Hi

Since updating to iwd 1.23 (from Arch packages), I've started seeing periodic crashes. I rebuilt iwd with debug symbols enabled and got the following backtrace out of 'coredumpctl gdb':

*hangs his head in shame*. So I think we screwed up the same macro 2x now.

Heh, yeah, did see there was a commit touching that macro, but didn't look closer.

Quoted 6 lines
Quoted 1 line

Can you please try the patch I just sent out?

Can do - thanks! Did a build with it incorporated and am running that now. Usually crashes once every hour (with striking regularity); will leave it running overnight and see if the crashes are gone by tomorrow :)

Alright, so no crashes - thanks for the swift fix!

And just for future reference, in case anyone else runs into the same
issue: The crashes were initially taking the kernel down with them, which has now also been fixed:
https://lore.kernel.org/r/20220201130951.22093-1-johannes(a)sipsolutions.net

-Toke