Skip to contentSkip to conversation

wintun driver get stalled for few sec until another packet is being sent -- approach 2

PATCH New odedkatz 1 participant 2 messages Feb 24

lore
Share & tools
lore

odedkatz

author
#0 lore raw

cover wintun driver get stalled for few sec until another packet is being sent -- approach 2

## Problem
High-speed TCP uploads stall for 4-5 seconds when the sender and receiver are close (~1ms RTT). The root cause is a missed-wakeup race in the Receive ring's Alertable/Tail signaling protocol between the userspace API (WintunSendPacket) and the kernel0 consumer thread (TunProcessReceiveData).

On x86-64, WriteRelease/ReadAcquire provide acquire-release semantics but do not emit MFENCE — the only instruction that prevents store-load reordering (the one reordering x86 permits). Both cores can simultaneously observe stale values of the other's store, causing the userspace to skip SetEvent while the driver enters KeWaitForMultipleObjects indefinitely.

The driver sleeps until a TCP retransmission (RTO ~1s + exponential backoff) coincidentally wins the race, producing the observed 4-5 second stalls.

## Fix
Insert MemoryBarrier() between each store-load pair on both sides. This guarantees that at least one side always sees the other's store, eliminating the missed wakeup. The Alertable optimization (avoiding SetEvent syscalls when the driver is actively spinning) is fully preserved. Alexey Lapuka (1):
Fix missed-wakeup race in ring buffer Alertable signaling

 api/session.c   | 1 +
 driver/wintun.c | 1 +
 2 files changed, 2 insertions(+)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH 1/1 Fix missed-wakeup race in ring buffer Alertable signaling

From: Alexey Lapuka <[email protected]>

Add MemoryBarrier() between store-load pairs in the Dekker-style synchronization used by the Receive ring's Alertable/Tail protocol.

On x86-64, WriteRelease/ReadAcquire only prevent compiler reordering and provide acquire/release semantics, but do not emit MFENCE — the only instruction that prevents store-load reordering across cores. Without a full barrier, both the userspace producer and the kernel consumer can simultaneously read stale values:

     Userspace: STORE(Tail)  ...  LOAD(Alertable) -> sees FALSE (stale)
     Driver:    STORE(Alertable=TRUE) ... LOAD(Tail) -> sees old tail

The driver then enters KeWaitForMultipleObjects with no pending SetEvent, sleeping until a TCP retransmission (typically 4-5s later) re-triggers the send path and wins the race.

The fix adds MemoryBarrier() (MFENCE on x86) on both sides:
- api/session.c WintunSendPacket: between WriteULongRelease(Tail) and ReadAcquire(Alertable)
- driver/twintun.c TunProcessReceiveData: between WriteRelease(Alertable, TRUE) and ReadULongAcquire(Tail)

This guarantees that at least one side always observes the other's store, preventing the missed wakeup while preserving the Alertable optimization that avoids unnecessary SetEvent syscalls.

---
 api/session.c   | 1 +
 driver/wintun.c | 1 +
 2 files changed, 2 insertions(+)
api/session.c +1 -0
index ab96c64..13d5bca 100644
@@ -302,6 +302,7 @@ WintunSendPacket(TUN_SESSION *Session, const BYTE *Packet)
     if (Session->Descriptor.Receive.Ring->Tail != Session->Receive.TailRelease)
     {
         WriteULongRelease(&Session->Descriptor.Receive.Ring->Tail, Session->Receive.TailRelease);
+        MemoryBarrier();
         if (ReadAcquire(&Session->Descriptor.Receive.Ring->Alertable))
             SetEvent(Session->Descriptor.Receive.TailMoved);
     }
driver/wintun.c +1 -0
index 82e346b..72ba5d3 100644
@@ -481,6 +481,7 @@ TunProcessReceiveData(_Inout_ TUN_CTX *Ctx)
             if (RingHead == RingTail)
             {
                 WriteRelease(&Ring->Alertable, TRUE);
+                MemoryBarrier();
                 RingTail = ReadULongAcquire(&Ring->Tail);
                 if (RingHead == RingTail)
                 {
Signature · 1 line
2.43.0