Skip to contentSkip to conversation

net: forbid the reentrant RX

PATCH In review Jason Wang 6 participants 13 messages Jul 22, 2020 – Sep 3, 2020 Networking

lore
Share & tools
lore

Jason Wang

author
#0 lore raw

PATCH 1/2 net: forbid the reentrant RX

The memory API allows DMA into NIC's MMIO area. This means the NIC's RX routine must be reentrant. Instead of auditing all the NIC, we can simply detect the reentrancy and return early. The queue->delivering is set and cleared by qemu_net_queue_deliver() for other queue helpers to know whether the delivering in on going (NIC's receive is being called). We can check it and return early in qemu_net_queue_flush() to forbid reentrant RX.

---
 net/queue.c | 3 +++
 1 file changed, 3 insertions(+)
net/queue.c +3 -0
index 0164727e39..19e32c80fd 100644
@@ -250,6 +250,9 @@ void qemu_net_queue_purge(NetQueue *queue, NetClientState *from)
 
 bool qemu_net_queue_flush(NetQueue *queue)
 {
+    if (queue->delivering)
+        return false;
+
     while (!QTAILQ_EMPTY(&queue->packets)) {
         NetPacket *packet;
         int ret;
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 2/2 e1000e: make TX reentrant

In loopback mode, e1000e RX can DMA into TX doorbell which requires TX to be reentrant. This patch make e1000e's TX routine reentrant by introducing a per device boolean for recording whether or not a TX rountine is being called and return early.

---
 hw/net/e1000e_core.c | 8 ++++++++
 hw/net/e1000e_core.h | 1 +
 2 files changed, 9 insertions(+)
hw/net/e1000e_core.c +8 -0
index bcd186cac5..8126a644a5 100644
@@ -923,6 +923,12 @@ e1000e_start_xmit(E1000ECore *core, const E1000E_TxRing *txr)
         return;
     }
 
+    if (core->sending) {
+        return;
+    }
+
+    core->sending = true;
+
     while (!e1000e_ring_empty(core, txi)) {
         base = e1000e_ring_head_descr(core, txi);
 
@@ -940,6 +946,8 @@ e1000e_start_xmit(E1000ECore *core, const E1000E_TxRing *txr)
     if (!ide || !e1000e_intrmgr_delay_tx_causes(core, &cause)) {
         e1000e_set_interrupt_cause(core, cause);
     }
+
+    core->sending = false;
 }
 
 static bool
hw/net/e1000e_core.h +1 -0
index aee32f7e48..4679c1761f 100644
@@ -114,6 +114,7 @@ struct E1000Core {
     void (*owner_start_recv)(PCIDevice *d);
 
     uint32_t msi_causes_pending;
+    bool sending;
 };
 
 void
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH 2/2] e1000e: make TX reentrant

Quoted 6 lines · Jason Wang <[email protected]> 于2020年7月22日周三 下午4:58写道:

In loopback mode, e1000e RX can DMA into TX doorbell which requires TX to be reentrant. This patch make e1000e's TX routine reentrant by introducing a per device boolean for recording whether or not a TX rountine is being called and return early.

Could we introduce a per-queue 'sending' variable just like the RX. So we can do this in net core layer.

Thanks,
Li Qiang

Quoted 47 lines
---
 hw/net/e1000e_core.c | 8 ++++++++
 hw/net/e1000e_core.h | 1 +
 2 files changed, 9 insertions(+)
hw/net/e1000e_core.c +8 -0
index bcd186cac5..8126a644a5 100644
@@ -923,6 +923,12 @@ e1000e_start_xmit(E1000ECore *core, const E1000E_TxRing *txr)
         return;
     }
 
+    if (core->sending) {
+        return;
+    }
+
+    core->sending = true;
+
     while (!e1000e_ring_empty(core, txi)) {
         base = e1000e_ring_head_descr(core, txi);
 
@@ -940,6 +946,8 @@ e1000e_start_xmit(E1000ECore *core, const E1000E_TxRing *txr)
     if (!ide || !e1000e_intrmgr_delay_tx_causes(core, &cause)) {
         e1000e_set_interrupt_cause(core, cause);
     }
+
+    core->sending = false;
 }
 
 static bool
hw/net/e1000e_core.h +1 -1
index aee32f7e48..4679c1761f 100644
@@ -114,6 +114,7 @@ struct E1000Core {
     void (*owner_start_recv)(PCIDevice *d);
 
     uint32_t msi_causes_pending;
+    bool sending;
 };
 
 void
--

2.20.1

author
#3 lore raw

Re: [PATCH 2/2] e1000e: make TX reentrant

Quoted 8 lines · Li Qiang
Quoted 5 lines · Jason Wang <[email protected]> 于2020年7月22日周三 下午4:58写道:

In loopback mode, e1000e RX can DMA into TX doorbell which requires TX to be reentrant. This patch make e1000e's TX routine reentrant by introducing a per device boolean for recording whether or not a TX rountine is being called and return early.

Could we introduce a per-queue 'sending' variable just like the RX. So we can do this in net core layer.

It's kind of not easy since TX routine is called before the packet can reach network queue.

Thanks

Quoted 51 lines

Thanks,
Li Qiang

Quoted 47 lines
---
  hw/net/e1000e_core.c | 8 ++++++++
  hw/net/e1000e_core.h | 1 +
  2 files changed, 9 insertions(+)
hw/net/e1000e_core.c +8 -0
index bcd186cac5..8126a644a5 100644
@@ -923,6 +923,12 @@ e1000e_start_xmit(E1000ECore *core, const E1000E_TxRing *txr)
          return;
      }
 
+    if (core->sending) {
+        return;
+    }
+
+    core->sending = true;
+
      while (!e1000e_ring_empty(core, txi)) {
          base = e1000e_ring_head_descr(core, txi);
 
@@ -940,6 +946,8 @@ e1000e_start_xmit(E1000ECore *core, const E1000E_TxRing *txr)
      if (!ide || !e1000e_intrmgr_delay_tx_causes(core, &cause)) {
          e1000e_set_interrupt_cause(core, cause);
      }
+
+    core->sending = false;
  }
 
  static bool
hw/net/e1000e_core.h +1 -1
index aee32f7e48..4679c1761f 100644
@@ -114,6 +114,7 @@ struct E1000Core {
      void (*owner_start_recv)(PCIDevice *d);
 
      uint32_t msi_causes_pending;
+    bool sending;
  };
 
  void
--

2.20.1

author
#5 lore raw

Re: [PATCH 2/2] e1000e: make TX reentrant

Quoted 4 lines · Michael Tokarev

FWIW, this is not "making TX reentrant", it is about forbidding reentrancy instead :)

/mjt

Indeed, I will rename the title.

Thanks

Quoted 1 line
#6 lore raw

Re: [PATCH 2/2] e1000e: make TX reentrant

Quoted 9 lines · Jason Wang
Quoted 4 lines · Michael Tokarev

FWIW, this is not "making TX reentrant", it is about forbidding reentrancy instead :)

/mjt

Indeed, I will rename the title.

Please also include a comment explaining the purpose of the early return in the code.

Stefan

#7 lore raw

Re: [PATCH 2/2] e1000e: make TX reentrant

In reply to Jason Wang
Quoted 8 lines · Jason Wang

In loopback mode, e1000e RX can DMA into TX doorbell which requires TX to be reentrant. This patch make e1000e's TX routine reentrant by introducing a per device boolean for recording whether or not a TX rountine is being called and return early.

---

This feels like a sticking-plaster fix that's not really in the right place... It stops us from calling back into e1000e_start_xmit(), but it doesn't prevent a DMA request from touching other device registers that update state in the E100ECore struct that the transmit code is not expecting to change.

thanks
-- PMM

author
#8 lore raw

Re: [PATCH 2/2] e1000e: make TX reentrant

Quoted 6 lines · Peter Maydell
Quoted 5 lines · Jason Wang

In loopback mode, e1000e RX can DMA into TX doorbell which requires TX to be reentrant. This patch make e1000e's TX routine reentrant by introducing a per device boolean for recording whether or not a TX rountine is being called and return early.

Quoted 8 lines
Quoted 2 lines

---

This feels like a sticking-plaster fix that's not really in the right place... It stops us from calling back into e1000e_start_xmit(), but it doesn't prevent a DMA request from touching other device registers that update state in the E100ECore struct that the transmit code is not expecting to change.

Right, so we can track the mr owner and fail the memory access if there's another mr transaction in memory core: memory_region_dispatch_read() and memory_region_dispatch_write().

But what's more interesting is that some device uses bh for the doorbell which may require more thought...

Thanks

Quoted 4 lines

thanks
-- PMM

author
#9 lore raw
In reply to Jason Wang
Quoted 18 lines · Jason Wang

The memory API allows DMA into NIC's MMIO area. This means the NIC's RX routine must be reentrant. Instead of auditing all the NIC, we can simply detect the reentrancy and return early. The queue->delivering is set and cleared by qemu_net_queue_deliver() for other queue helpers to know whether the delivering in on going (NIC's receive is being called). We can check it and return early in qemu_net_queue_flush() to forbid reentrant RX.

---
  net/queue.c | 3 +++
  1 file changed, 3 insertions(+)
net/queue.c +0 -0
index 0164727e39..19e32c80fd 100644

net/queue.c void qemu_net_queue_purge(NetQueue *queue, NetClientState *from) · line 258

Quoted 9 lines
net/queue.c +3 -0
@@ -250,6 +250,9 @@ void qemu_net_queue_purge(NetQueue *queue, NetClientState *from)
 
  bool qemu_net_queue_flush(NetQueue *queue)
  {
+    if (queue->delivering)
+        return false;
+
      while (!QTAILQ_EMPTY(&queue->packets)) {
          NetPacket *packet;
          int ret;

Queued for rc2.

Thanks

#10 lore raw
Quoted 32 lines · Jason Wang
Quoted 18 lines · Jason Wang

The memory API allows DMA into NIC's MMIO area. This means the NIC's RX routine must be reentrant. Instead of auditing all the NIC, we can simply detect the reentrancy and return early. The queue->delivering is set and cleared by qemu_net_queue_deliver() for other queue helpers to know whether the delivering in on going (NIC's receive is being called). We can check it and return early in qemu_net_queue_flush() to forbid reentrant RX.

---
  net/queue.c | 3 +++
  1 file changed, 3 insertions(+)
net/queue.c +0 -0
index 0164727e39..19e32c80fd 100644

net/queue.c void qemu_net_queue_purge(NetQueue *queue, NetClientState *from) · line 257

Quoted 8 lines
net/queue.c +3 -0
@@ -250,6 +250,9 @@ void qemu_net_queue_purge(NetQueue *queue, NetClientState *from)
  bool qemu_net_queue_flush(NetQueue *queue)
  {
+    if (queue->delivering)
+        return false;
+
      while (!QTAILQ_EMPTY(&queue->packets)) {
          NetPacket *packet;
          int ret;

Queued for rc2.

Quoted 2 lines

Thanks

Hi Jason,
I don't think this ever made it in. Are there any remaining problems? Thanks
-Alex

author
#11 lore raw
Quoted 31 lines · Alexander Bulekov
Quoted 30 lines · Jason Wang
Quoted 18 lines · Jason Wang

The memory API allows DMA into NIC's MMIO area. This means the NIC's RX routine must be reentrant. Instead of auditing all the NIC, we can simply detect the reentrancy and return early. The queue->delivering is set and cleared by qemu_net_queue_deliver() for other queue helpers to know whether the delivering in on going (NIC's receive is being called). We can check it and return early in qemu_net_queue_flush() to forbid reentrant RX.

---
   net/queue.c | 3 +++
   1 file changed, 3 insertions(+)
net/queue.c +0 -0
index 0164727e39..19e32c80fd 100644

net/queue.c void qemu_net_queue_purge(NetQueue *queue, NetClientState *from) · line 257

Quoted 8 lines
net/queue.c +3 -0
@@ -250,6 +250,9 @@ void qemu_net_queue_purge(NetQueue *queue, NetClientState *from)
   bool qemu_net_queue_flush(NetQueue *queue)
   {
+    if (queue->delivering)
+        return false;
+
       while (!QTAILQ_EMPTY(&queue->packets)) {
           NetPacket *packet;
           int ret;

Queued for rc2.

Quoted 6 lines
Quoted 2 lines

Thanks

Hi Jason,
I don't think this ever made it in. Are there any remaining problems? Thanks
-Alex

Hi Alex:

It should have been merged:

https://git.qemu.org/?p=qemu.git;a=commit;h=22dc8663d9fc7baa22100544c600b6285a63c7a3

Thanks

Quoted 1 line
#12 lore raw
Quoted 41 lines · Jason Wang
Quoted 31 lines · Alexander Bulekov
Quoted 30 lines · Jason Wang
Quoted 18 lines · Jason Wang

The memory API allows DMA into NIC's MMIO area. This means the NIC's RX routine must be reentrant. Instead of auditing all the NIC, we can simply detect the reentrancy and return early. The queue->delivering is set and cleared by qemu_net_queue_deliver() for other queue helpers to know whether the delivering in on going (NIC's receive is being called). We can check it and return early in qemu_net_queue_flush() to forbid reentrant RX.

---
   net/queue.c | 3 +++
   1 file changed, 3 insertions(+)
net/queue.c +0 -0
index 0164727e39..19e32c80fd 100644

net/queue.c void qemu_net_queue_purge(NetQueue *queue, NetClientState *from) · line 257

Quoted 8 lines
net/queue.c +3 -0
@@ -250,6 +250,9 @@ void qemu_net_queue_purge(NetQueue *queue, NetClientState *from)
   bool qemu_net_queue_flush(NetQueue *queue)
   {
+    if (queue->delivering)
+        return false;
+
       while (!QTAILQ_EMPTY(&queue->packets)) {
           NetPacket *packet;
           int ret;

Queued for rc2.

Quoted 6 lines
Quoted 2 lines

Thanks

Hi Jason,
I don't think this ever made it in. Are there any remaining problems? Thanks
-Alex

Quoted 8 lines

Hi Alex:

It should have been merged:

https://git.qemu.org/?p=qemu.git;a=commit;h=22dc8663d9fc7baa22100544c600b6285a63c7a3

Thanks

Ah. I missed only 1/2 was queued. I guess the e1000 patch didn't make the cut..
Thanks
-Alex

Quoted 3 lines
Quoted 2 lines