Skip to contentSkip to conversation

hw/arm/raspi4b: fix watchdog and GENET bugs hit by modern distro kernels

PATCH In review Marcelo Manzo 2 participants 5 messages Jul 25 – Aug 11

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover hw/arm/raspi4b: fix watchdog and GENET bugs hit by modern distro kernels

Follow-up to "hw/arm/bcm2838_pcie: make PCI device enumeration actually work" [1]. That series (plus the GENET/RNG200/thermal work before it) got a real Raspberry Pi 4 SoC far enough to boot official, unmodified Raspberry Pi OS releases end to end under raspi4b -- not just the functional-test suite's minimal 2019-era kernel+initrd. Doing that surfaced two further bugs that only show up with a modern distro's systemd actually driving the hardware, both on paths the existing tests don't exercise:

1. hw/misc/bcm2835_powermgt: the RSTC register's write-config bits (0x30) being set to the "full reset" value (0x20) does not mean "reset now" on real hardware -- it arms the watchdog so a reset happens only if the WDOG countdown register is not refreshed before it expires. QEMU's model treated any such RSTC write as an immediate reset. This is dormant on lighter userspace, but Debian 13/Trixie's systemd (257) writes to RSTC during routine early-boot watchdog probing, causing an immediate spurious reset a few seconds into boot. Combined with -no-reboot this looks exactly like a QEMU crash from the outside: clean exit, no panic, no guest reboot message, the last log line being the RSTC/WDOG write itself. Fixed by actually implementing the watchdog as a QEMUTimer that only fires -- and only then requests a reset/shutdown -- after the real timeout computed from the WDOG register.

2. hw/net/bcm2838_genet: three related per-ring DMA bugs. RDMA_CTRL writes never updated any per-ring status at all (only TDMA_CTRL updated a single overall DISABLED bit); enabling a ring didn't flush packets queued while it was disabled; and the RX fallback path hardcoded the last/default ring with no check that it was actually active. Bullseye and Bookworm's kernels default to that last ring, so this was invisible with them, but Trixie's 6.18 kernel actively uses rings 0-4 and never enables the default ring, so every unfiltered packet was silently dropped -- no DHCP lease, no usable networking. Fixed by tracking a real per-ring enable bitmask for both directions, flushing queued packets when a ring is enabled, and falling back to scanning for the first actually- active ring instead of assuming the last one.

Testing: rather than only the functional-test kernel, I booted three official Raspberry Pi OS Lite arm64 releases end to end against the patched raspi4b machine -- Bullseye (5.15.61-v8+), Bookworm (6.12.25+rpt-rpi-v8), and Trixie (6.18.34+rpt-rpi-v8) -- each with a clean boot (no crash/reboot loop), a DHCP lease obtained over GENET, working SSH login, and an error log containing nothing beyond the expected "hardware QEMU doesn't emulate" noise (VideoCore/VCHI, camera/codec/audio probes, GPIO regulators, Bluetooth UART, EEPROM check) on all three. Before patch 1, Trixie reset within seconds of boot; before patch 2, Trixie booted but never got a network address. Bullseye and Bookworm were unaffected either way, which is exactly why these bugs went unnoticed until a newer kernel exercised these specific paths.

I did not add a functional test for either fix: reliably triggering them needs a real systemd doing real early-boot watchdog/network setup, which the existing test kernel/initrd (a minimal 2019 busybox environment) doesn't exercise, and downloading a full modern distro image isn't practical for the functional-test asset infrastructure. Happy to take suggestions on a lighter-weight way to cover this if one exists.

checkpatch.pl is clean (0 errors, 0 warnings) on both patches, and the series builds standalone from a clean worktree (--target-list=aarch64-softmmu).

[1] https://patchwork.kernel.org/project/qemu-devel/cover/[email protected]/

Marcelo Manzo (2):
hw/misc/bcm2835_powermgt: implement a real watchdog timer hw/net/bcm2838_genet: fix per-ring DMA status and RX ring selection

 hw/misc/bcm2835_powermgt.c         | 49 +++++++++++++++++++++++-------
 hw/net/bcm2838_genet.c             | 24 +++++++++++++--
 include/hw/misc/bcm2835_powermgt.h |  2 ++
 include/hw/net/bcm2838_genet.h     |  2 ++
 4 files changed, 63 insertions(+), 14 deletions(-)
Signature · 1 line
2.47.1
#1 lore raw

PATCH 1/2 hw/misc/bcm2835_powermgt: implement a real watchdog timer

The RSTC register's write-config bits (0x30) being set to the "full reset" value (0x20) does not mean "reset now" -- it arms the hardware watchdog so that a reset happens if the WDOG countdown register is not refreshed before it expires. The previous implementation treated any such RSTC write as an immediate reset, regardless of the WDOG value.

This is dormant on older/lighter userspace (nothing in Bullseye's default boot touches these registers this way), but modern systemd (observed with Debian 13/Trixie's systemd 257) writes to RSTC as part of routine early-boot watchdog probing. With the old code, this fires an immediate reset a few seconds into boot; combined with -no-reboot this looks exactly like a QEMU crash (clean exit, no panic, no guest reboot message) with the last log line being the RSTC/WDOG write.

Fix this by actually implementing the watchdog as a QEMUTimer: writes to RSTC/WDOG (re)compute the timeout from the WDOG register (in units of 1/65536 s, per the real hardware) and arm a timer for that many nanoseconds out; only when the timer actually fires do we request a system reset or shutdown, matching real hardware behavior. Clearing the write-config bits or the WDOG value disarms the timer, and reset disarms it too.

Verified against real Raspberry Pi OS images under the patched raspi4b machine: Bullseye (5.15) and Bookworm (6.12) never exercised this path either way; Trixie (6.18, systemd 257) no longer crashes at boot and reaches a working login/SSH state.

---
 hw/misc/bcm2835_powermgt.c         | 49 +++++++++++++++++++++++-------
 include/hw/misc/bcm2835_powermgt.h |  2 ++
 2 files changed, 40 insertions(+), 11 deletions(-)
hw/misc/bcm2835_powermgt.c +38 -11
index 3ec7abad0e..d90b9a0c3e 100644
@@ -19,10 +19,40 @@
 #define PASSWORD_MASK 0xff000000
 
 #define R_RSTC 0x1c
-#define V_RSTC_RESET 0x20
+#define V_RSTC_WRCFG_MASK 0x30
+#define V_RSTC_FULL_RESET 0x20
 #define R_RSTS 0x20
 #define V_RSTS_POWEROFF 0x555 /* Linux uses partition 63 to indicate halt. */
 #define R_WDOG 0x24
+#define V_WDOG_TIME_MASK 0xfffff
+#define WDOG_TICKS_PER_SECOND 65536
+
+static void bcm2835_powermgt_expire(void *opaque)
+{
+    BCM2835PowerMgtState *s = opaque;
+
+    if ((s->rsts & 0xfff) == V_RSTS_POWEROFF) {
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
+    } else {
+        qemu_system_reset_request(SHUTDOWN_CAUSE_GUEST_RESET);
+    }
+}
+
+static void bcm2835_powermgt_update_wdog(BCM2835PowerMgtState *s)
+{
+    uint64_t timeout_ns;
+
+    if ((s->rstc & V_RSTC_WRCFG_MASK) != V_RSTC_FULL_RESET ||
+        s->wdog == 0) {
+        timer_del(s->wdog_timer);
+        return;
+    }
+
+    timeout_ns = muldiv64(s->wdog, NANOSECONDS_PER_SECOND,
+                          WDOG_TICKS_PER_SECOND);
+    timer_mod(s->wdog_timer,
+              qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + timeout_ns);
+}
 
 static uint64_t bcm2835_powermgt_read(void *opaque, hwaddr offset,
                                       unsigned size)
@@ -70,13 +100,7 @@ static void bcm2835_powermgt_write(void *opaque, hwaddr offset,
     switch (offset) {
     case R_RSTC:
         s->rstc = value;
-        if (value & V_RSTC_RESET) {
-            if ((s->rsts & 0xfff) == V_RSTS_POWEROFF) {
-                qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
-            } else {
-                qemu_system_reset_request(SHUTDOWN_CAUSE_GUEST_RESET);
-            }
-        }
+        bcm2835_powermgt_update_wdog(s);
         break;
     case R_RSTS:
         qemu_log_mask(LOG_UNIMP,
@@ -84,9 +108,8 @@ static void bcm2835_powermgt_write(void *opaque, hwaddr offset,
         s->rsts = value;
         break;
     case R_WDOG:
-        qemu_log_mask(LOG_UNIMP,
-                      "bcm2835_powermgt_write: WDOG\n");
-        s->wdog = value;
+        s->wdog = value & V_WDOG_TIME_MASK;
+        bcm2835_powermgt_update_wdog(s);
         break;
 
     default:
@@ -113,6 +136,7 @@ static const VMStateDescription vmstate_bcm2835_powermgt = {
         VMSTATE_UINT32(rstc, BCM2835PowerMgtState),
         VMSTATE_UINT32(rsts, BCM2835PowerMgtState),
         VMSTATE_UINT32(wdog, BCM2835PowerMgtState),
+        VMSTATE_TIMER_PTR(wdog_timer, BCM2835PowerMgtState),
         VMSTATE_END_OF_LIST()
     }
 };
@@ -124,6 +148,8 @@ static void bcm2835_powermgt_init(Object *obj)
     memory_region_init_io(&s->iomem, obj, &bcm2835_powermgt_ops, s,
                           TYPE_BCM2835_POWERMGT, 0x200);
     sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->iomem);
+    s->wdog_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
+                                 bcm2835_powermgt_expire, s);
 }
 
 static void bcm2835_powermgt_reset(DeviceState *dev)
@@ -134,6 +160,7 @@ static void bcm2835_powermgt_reset(DeviceState *dev)
     s->rstc = 0x00000102;
     s->rsts = 0x00001000;
     s->wdog = 0x00000000;
+    timer_del(s->wdog_timer);
 }
 
 static void bcm2835_powermgt_class_init(ObjectClass *klass, const void *data)
include/hw/misc/bcm2835_powermgt.h +2 -0
index fb0740c01e..d1903a9cce 100644
@@ -12,6 +12,7 @@
 #define BCM2835_POWERMGT_H
 
 #include "hw/core/sysbus.h"
+#include "qemu/timer.h"
 #include "qom/object.h"
 
 #define TYPE_BCM2835_POWERMGT "bcm2835-powermgt"
@@ -24,6 +25,7 @@ struct BCM2835PowerMgtState {
     uint32_t rstc;
     uint32_t rsts;
     uint32_t wdog;
+    QEMUTimer *wdog_timer;
 };
 
 #endif
Signature · 1 line
2.47.1
#3 lore raw

PATCH 2/2 hw/net/bcm2838_genet: fix per-ring DMA status and RX ring selection

In reply to Marcelo Manzo

Three related GENET DMA-ring bugs, all hit by newer guest drivers that actually use multiple RX/TX rings instead of relying on the single default ring:

- BCM2838_GENET_RDMA_CTRL writes never updated the RDMA status register at all (only TDMA_CTRL updated a status field, and only a single overall DISABLED bit, not per-ring state). Track a proper per-ring enable bitmask (bits 0-17, one per ring plus the default ring) for both RDMA and TDMA, computed directly from the ctrl register's enable bits on every write.

- Enabling a ring via RDMA_CTRL didn't flush any packets that had been queued while the ring was disabled, so traffic arriving in the window before a ring was enabled was silently dropped instead of delivered once it came up. Call qemu_flush_queued_packets() when the EN bit is set.

- bcm2838_genet_receive()'s fallback path hardcoded ring BCM2838_GENET_DMA_RING_CNT - 1 (the last/default ring) whenever a packet didn't match a specific filter, with no check that this ring was actually active. Older guest kernels default to this ring, but newer ones (observed: Debian 13/Trixie's 6.18 kernel) actively use rings 0-4 and never enable the default ring at all, so every unfiltered packet was silently dropped. Fall back to scanning for the first actually-active ring instead of assuming the last one.

Reset now initializes both status registers to "all rings enabled" to match real hardware defaults.

Verified against real Raspberry Pi OS images under the patched raspi4b machine: Bullseye (5.15) and Bookworm (6.12) worked with the old code (they use the default ring); Trixie (6.18) did not get a working DHCP lease over GENET without this fix and does with it.

---
 hw/net/bcm2838_genet.c         | 24 +++++++++++++++++++++---
 include/hw/net/bcm2838_genet.h |  2 ++
 2 files changed, 23 insertions(+), 3 deletions(-)
hw/net/bcm2838_genet.c +21 -3
index 43583aba64..a9798073dc 100644
@@ -132,6 +132,8 @@ FIELD(GENET_DMA_STATUS, DISABLED,           0, 1)
 FIELD(GENET_DMA_STATUS, DESC_RAM_INIT_BUSY, 1, 1)
 FIELD(GENET_DMA_STATUS, RSVD_2_31,          2, 30)
 
+#define GENET_DMA_ENABLE_MASK ((1U << 18) - 1)
+
 REG32(GENET_RDMA_LENGTH_STATUS,             0)
 FIELD(GENET_RDMA_LENGTH_STATUS, OVERRUN,    0, 1)
 FIELD(GENET_RDMA_LENGTH_STATUS, CRC_ERROR,  1, 1)
@@ -622,10 +624,8 @@ static void bcm2838_genet_tdma(BCM2838GenetState *s, hwaddr offset,
         }
         break;
     case BCM2838_GENET_TDMA_CTRL:
+        s->regs.tdma.status = (~dma_ctrl) & GENET_DMA_ENABLE_MASK;
         if (exst_tdma_en != incm_tdma_en) {
-            s->regs.tdma.status = FIELD_DP32(s->regs.tdma.status,
-                                             GENET_DMA_STATUS,
-                                             DISABLED, !exst_tdma_en);
             trace_bcm2838_genet_tx_dma(incm_tdma_en == 1
                                        ? "enabled"
                                        : "disabled");
@@ -727,6 +727,9 @@ static void bcm2838_genet_write(void *opaque, hwaddr offset, uint64_t value,
             s->regs.intrl0.stat = FIELD_DP32(s->regs.intrl0.stat,
                                              GENET_INTRL_0, MDIO_DONE, 1);
             break;
+        case BCM2838_GENET_RDMA_CTRL:
+            s->regs.rdma.status = (~value) & GENET_DMA_ENABLE_MASK;
+            break;
         case BCM2838_GENET_TDMA_REGS
             ... BCM2838_GENET_TDMA_REGS + sizeof(BCM2838GenetRegsTdma) - 1:
             bcm2838_genet_tdma(s, offset, value);
@@ -736,6 +739,10 @@ static void bcm2838_genet_write(void *opaque, hwaddr offset, uint64_t value,
         }
 
         memcpy((uint8_t *)&s->regs + offset, &value, size);
+        if (offset == BCM2838_GENET_RDMA_CTRL &&
+            FIELD_EX32(value, GENET_DMA_CTRL, EN)) {
+            qemu_flush_queued_packets(ncs);
+        }
         bcm2838_genet_set_irq_default(s);
         bcm2838_genet_set_irq_prio(s);
     } else {
@@ -927,6 +934,15 @@ static ssize_t bcm2838_genet_receive(NetClientState *nc, const uint8_t *buf,
             ring_index = bcm2838_genet_filter2ring(s, filter_index);
         } else {
             ring_index = BCM2838_GENET_DMA_RING_CNT - 1;
+            if (!bcm2838_genet_rdma_ring_active(s, ring_index)) {
+                for (ring_index = 0;
+                     ring_index < BCM2838_GENET_DMA_RING_CNT - 1;
+                     ring_index++) {
+                    if (bcm2838_genet_rdma_ring_active(s, ring_index)) {
+                        break;
+                    }
+                }
+            }
         }
 
         if (size <= MAX_PACKET_SIZE) {
@@ -1063,6 +1079,8 @@ static void bcm2838_genet_reset(Object *obj, ResetType type)
                                       MAJOR_REV, BCM2838_GENET_REV_MAJOR);
     s->regs.sys.rev_ctrl = FIELD_DP32(s->regs.sys.rev_ctrl, GENET_SYS_REV_CTRL,
                                       MINOR_REV, BCM2838_GENET_REV_MINOR);
+    s->regs.rdma.status = GENET_DMA_ENABLE_MASK;
+    s->regs.tdma.status = GENET_DMA_ENABLE_MASK;
 
     trace_bcm2838_genet_reset("done");
 
include/hw/net/bcm2838_genet.h +2 -0
index 5944e0cef9..960042f6f4 100644
@@ -55,6 +55,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(BCM2838GenetState, BCM2838_GENET)
 #define BCM2838_GENET_TDMA_RINGS        BCM2838_GENET_TDMA_REG(rings)
 #define BCM2838_GENET_TDMA_RING_CFG     BCM2838_GENET_TDMA_REG(ring_cfg)
 #define BCM2838_GENET_TDMA_CTRL         BCM2838_GENET_TDMA_REG(ctrl)
+#define BCM2838_GENET_TDMA_STATUS       BCM2838_GENET_TDMA_REG(status)
 
 #define BCM2838_GENET_RDMA_REGS         offsetof(BCM2838GenetRegs, rdma)
 #define BCM2838_GENET_RDMA_REG(reg)     (BCM2838_GENET_RDMA_REGS \
@@ -62,6 +63,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(BCM2838GenetState, BCM2838_GENET)
 #define BCM2838_GENET_RDMA_RINGS        BCM2838_GENET_RDMA_REG(rings)
 #define BCM2838_GENET_RDMA_RING_CFG     BCM2838_GENET_RDMA_REG(ring_cfg)
 #define BCM2838_GENET_RDMA_CTRL         BCM2838_GENET_RDMA_REG(ctrl)
+#define BCM2838_GENET_RDMA_STATUS       BCM2838_GENET_RDMA_REG(status)
 
 #define BCM2838_GENET_TRING_REG(reg)    offsetof(BCM2838GenetTdmaRing, reg)
 #define BCM2838_GENET_TRING_WRITE_PTR BCM2838_GENET_TRING_REG(write_ptr)
Signature · 1 line
2.47.1
#2 lore raw

Re: [PATCH 1/2] hw/misc/bcm2835_powermgt: implement a real watchdog timer

In reply to Marcelo Manzo
Quoted 32 lines · Marcelo Manzo

The RSTC register's write-config bits (0x30) being set to the "full reset" value (0x20) does not mean "reset now" -- it arms the hardware watchdog so that a reset happens if the WDOG countdown register is not refreshed before it expires. The previous implementation treated any such RSTC write as an immediate reset, regardless of the WDOG value.

This is dormant on older/lighter userspace (nothing in Bullseye's default boot touches these registers this way), but modern systemd (observed with Debian 13/Trixie's systemd 257) writes to RSTC as part of routine early-boot watchdog probing. With the old code, this fires an immediate reset a few seconds into boot; combined with -no-reboot this looks exactly like a QEMU crash (clean exit, no panic, no guest reboot message) with the last log line being the RSTC/WDOG write.

Fix this by actually implementing the watchdog as a QEMUTimer: writes to RSTC/WDOG (re)compute the timeout from the WDOG register (in units of 1/65536 s, per the real hardware) and arm a timer for that many nanoseconds out; only when the timer actually fires do we request a system reset or shutdown, matching real hardware behavior. Clearing the write-config bits or the WDOG value disarms the timer, and reset disarms it too.

Verified against real Raspberry Pi OS images under the patched raspi4b machine: Bullseye (5.15) and Bookworm (6.12) never exercised this path either way; Trixie (6.18, systemd 257) no longer crashes at boot and reaches a working login/SSH state.

hw/misc/bcm2835_powermgt.c static const VMStateDescription vmstate_bcm2835_powermgt = { · line 140

Quoted 5 lines
hw/misc/bcm2835_powermgt.c +1 -0
@@ -113,6 +136,7 @@ static const VMStateDescription vmstate_bcm2835_powermgt = {
         VMSTATE_UINT32(rstc, BCM2835PowerMgtState),
         VMSTATE_UINT32(rsts, BCM2835PowerMgtState),
         VMSTATE_UINT32(wdog, BCM2835PowerMgtState),
+        VMSTATE_TIMER_PTR(wdog_timer, BCM2835PowerMgtState),
         VMSTATE_END_OF_LIST()

If we add a field to the vmstate we should also increment the version_id and minimum_version_id, and note in the commit message that this is a migration break for the affected machine types. (We're OK to break migration compat for raspi because it's not a versioned machine type.)

I couldn't find any h/w docs for the watchdog (is it one of the undocumented devices on this SoC?), but this all looks right and matches how the Linux driver uses this register.

Since the version bump was the only issue here, I've applied this patch to target-arm.next ready for when we re-open development for 11.2, with that tweak made.

thanks
-- PMM

#4 lore raw
In reply to Marcelo Manzo
Quoted 10 lines · Marcelo Manzo

Follow-up to "hw/arm/bcm2838_pcie: make PCI device enumeration actually work" [1]. That series (plus the GENET/RNG200/thermal work before it) got a real Raspberry Pi 4 SoC far enough to boot official, unmodified Raspberry Pi OS releases end to end under raspi4b -- not just the functional-test suite's minimal 2019-era kernel+initrd. Doing that surfaced two further bugs that only show up with a modern distro's systemd actually driving the hardware, both on paths the existing tests don't exercise:

Quoted 3 lines

Marcelo Manzo (2):
hw/misc/bcm2835_powermgt: implement a real watchdog timer hw/net/bcm2838_genet: fix per-ring DMA status and RX ring selection

Patch 2 is to a device that's not in upstream git yet, so we should fold the bugfix into the patches that add it, rather than adding it and then bugfixing it afterwards.

I've taken patch 1.

thanks
-- PMM