clocksource: Rewrite clocksource watchdog and related cleanups

The clocksource watchdog code has over time reached the state of an unpenetrable maze of duct tape and staples. The original design, which was made in the context of systems far smaller than today, is based on the assumption that the to be monitored clocksource (TSC) can be trivially compared against a known to be stable clocksource (HPET/ACPI-PM timer).

Over the years it turned out that this approach has major flaws:

- Long delays between watchdog invocations can result in wrap arounds of the reference clocksource

- Scalability of the reference clocksource readout can degrade on large multi-socket systems due to interconnect congestion

- Fixed margins based on magic assumptions, which do not work when the watchdog interval diverges too much.

This was addressed with various heuristics which degraded the accurracy of the watchdog to the point that it fails to detect actual TSC problems on older hardware which exposes slow inter CPU drifts due to firmware manipulating the TSC to hide SMI time.

To address this and bring back sanity to the watchdog, rewrite the code completely with a different approach:

1) Restrict the validation against a reference clocksource to the boot CPU, which is usually the CPU/Socket closest to the legacy block which contains the reference source (HPET/ACPI-PM timer). Validate that the reference readout is within a bound latency so that the actual comparison against the TSC stays within 500ppm as long as the clocks are stable.

2) Compare the TSCs of the other CPUs in a round robin fashion against the boot CPU in the same way the TSC synchronization on CPU hotplug works. This still can suffer from delayed reaction of the remote CPU to the SMP function call and the latency of the control variable cache line. But this latency is not affecting correctness. It only affects the accuracy. With low contention the readout latency is in the low nanoseconds range, which detects even slight skews between CPUs. Under high contention this becomes obviously less accurate, but still detects slow skews reliably as it solely relies on subsequent readouts being monotonically increasing. It just can take slightly longer to detect the issue.

3) Rewrite the watchdog test so it tests the various mechanisms one by one and validates the result against the expectation.

The series also cleans up random places which set the watchdog bits without actually ever using them. Right now the only architecture which uses the clocksource watchdog is x86.

The series is based on

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core

and also available from git:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git timers/clocksource

Thanks,

tglx

---
 Documentation/admin-guide/kernel-parameters.txt |    7 
 arch/mips/Kconfig                               |    1 
 arch/parisc/kernel/time.c                       |    5 
 arch/x86/include/asm/time.h                     |    1 
 arch/x86/kernel/hpet.c                          |    4 
 arch/x86/kernel/tsc.c                           |   61 --
 drivers/clocksource/Kconfig                     |    1 
 drivers/clocksource/acpi_pm.c                   |    4 
 include/linux/clocksource.h                     |   24 
 kernel/time/Kconfig                             |   12 
 kernel/time/clocksource-wdtest.c                |  268 ++++----
 kernel/time/clocksource.c                       |  731 ++++++++++++------------
 kernel/time/jiffies.c                           |    1 
 13 files changed, 536 insertions(+), 584 deletions(-)

PARISC does not enable the clocksource watchdog, so the VERIFY flags are pointless as they are not evaluated. Remove them from the clocksource.

---
 arch/parisc/kernel/time.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

@@ -193,12 +193,9 @@ static struct clocksource clocksource_cr
 	.read			= read_cr16,
 	.mask			= CLOCKSOURCE_MASK(BITS_PER_LONG),
 	.flags			= CLOCK_SOURCE_IS_CONTINUOUS |
-					CLOCK_SOURCE_VALID_FOR_HRES |
-					CLOCK_SOURCE_MUST_VERIFY |
-					CLOCK_SOURCE_VERIFY_PERCPU,
+				  CLOCK_SOURCE_VALID_FOR_HRES,
 };
 
-
 /*
  * timer interrupt and sched_clock() initialization
  */

Thanks!
Helge

MIPS selects CLOCKSOURCE_WATCHDOG, but none of the clocksources actually sets the MUST_VERIFY flag. So compiling the watchdog in is a pointless exercise. Remove the selects.

---
 arch/mips/Kconfig           |    1 -
 drivers/clocksource/Kconfig |    1 -
 2 files changed, 2 deletions(-)

@@ -1129,7 +1129,6 @@ config CSRC_IOASIC
 	bool
 
 config CSRC_R4K
-	select CLOCKSOURCE_WATCHDOG if CPU_FREQ
 	bool
 
 config CSRC_SB1250

@@ -595,7 +595,6 @@ config CLKSRC_VERSATILE
 config CLKSRC_MIPS_GIC
 	bool
 	depends on MIPS_GIC
-	select CLOCKSOURCE_WATCHDOG
 	select TIMER_OF
 
 config CLKSRC_PXA

Based on commit 385864280597 ("mips: csrc-r4k: Mark R4K timer as unstable
if CPU freq changes") which added some of this stuff shouldn't the flag be set instead?

At first glance the situation seems analogous to that of the x86 TSC:
just as the TSC the R4k timer is a CPU onchip free-running counter driven by the CPU clock (typically at half the rate).

Maciej

On x86 the CPU frequency can be modified by the BIOS behind the kernels back, so the watchdog is required to catch this situation. So R4K only needs the watchdog when there is some magic going on even when the kernel itself does not manage the CPU frequency.

Thanks,

tglx

Unconditionally setting the CLOCK_SOURCE_VALID_FOR_HRES for the real TSC clocksource is wrong as there is no guarantee that the early TSC was validated for high resolution mode.

Set the flag only when the early TSC was validated as otherwise the clocksource selection might enable high resolution mode with a TSC of unknown quality and possibly no way to back out once it is discovered to be unsuitable.

---
 arch/x86/kernel/tsc.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

@@ -1193,7 +1193,6 @@ static struct clocksource clocksource_ts
 	.read			= read_tsc,
 	.mask			= CLOCKSOURCE_MASK(64),
 	.flags			= CLOCK_SOURCE_IS_CONTINUOUS |
-				  CLOCK_SOURCE_VALID_FOR_HRES |
 				  CLOCK_SOURCE_MUST_VERIFY |
 				  CLOCK_SOURCE_VERIFY_PERCPU,
 	.id			= CSID_X86_TSC,

@@ -1403,6 +1402,15 @@ static void tsc_refine_calibration_work(
 		have_art = true;
 		clocksource_tsc.base = &art_base_clk;
 	}
+
+	/*
+	 * Transfer the valid for high resolution flag if it was set on the
+	 * early TSC already. That guarantees that there is no intermediate
+	 * clocksource selected once the early TSC is unregistered.
+	 */
+	if (clocksource_tsc_early.flags & CLOCK_SOURCE_VALID_FOR_HRES)
+		clocksource_tsc.flags |= CLOCK_SOURCE_VALID_FOR_HRES;
+
 	clocksource_register_khz(&clocksource_tsc, tsc_khz);
 unreg:
 	clocksource_unregister(&clocksource_tsc_early);

Using a non-continuous aka untrusted clocksource as a watchdog for another untrusted clocksource is equivalent to putting the fox in charge of the henhouse.

That's especially true with the jiffies clocksource which depends on interrupt delivery based on a periodic timer. Neither the frequency of that timer is trustworthy nor the kernel's ability to react on it in a timely manner and rearm it if it is not self rearming.

Just don't bother to deal with this. It's not worth the trouble and only relevant to museum piece hardware.

---
 kernel/time/clocksource.c |    7 +++++++
 1 file changed, 7 insertions(+)

@@ -651,6 +651,13 @@ static void clocksource_select_watchdog(
 		if (cs->flags & CLOCK_SOURCE_MUST_VERIFY)
 			continue;
 
+		/*
+		 * If it's not continuous, don't put the fox in charge of
+		 * the henhouse.
+		 */
+		if (!(cs->flags & CLOCK_SOURCE_IS_CONTINUOUS))
+			continue;
+
 		/* Skip current if we were requested for a fallback. */
 		if (fallback && cs == old_wd)
 			continue;

The clocksource watchdog code has over time reached the state of an unpenetrable maze of duct tape and staples. The original design, which was made in the context of systems far smaller than today, is based on the assumption that the to be monitored clocksource (TSC) can be trivially compared against a known to be stable clocksource (HPET/ACPI-PM timer).

Over the years it turned out that this approach has major flaws:

- Long delays between watchdog invocations can result in wrap arounds of the reference clocksource

- Scalability of the reference clocksource readout can degrade on large multi-socket systems due to interconnect congestion

This was addressed with various heuristics which degraded the accurracy of the watchdog to the point that it fails to detect actual TSC problems on older hardware which exposes slow inter CPU drifts due to firmware manipulating the TSC to hide SMI time.

To address this and bring back sanity to the watchdog, rewrite the code completely with a different approach:

1) Restrict the validation against a reference clocksource to the boot CPU, which is usually the CPU/Socket closest to the legacy block which contains the reference source (HPET/ACPI-PM timer). Validate that the reference readout is within a bound latency so that the actual comparison against the TSC stays within 500ppm as long as the clocks are stable.

2) Compare the TSCs of the other CPUs in a round robin fashion against the boot CPU in the same way the TSC synchronization on CPU hotplug works. This still can suffer from delayed reaction of the remote CPU to the SMP function call and the latency of the control variable cache line. But this latency is not affecting correctness. It only affects the accuracy. With low contention the readout latency is in the low nanoseconds range, which detects even slight skews between CPUs. Under high contention this becomes obviously less accurate, but still detects slow skews reliably as it solely relies on subsequent readouts being monotonically increasing. It just can take slightly longer to detect the issue.

3) Rewrite the watchdog test so it tests the various mechanisms one by one and validating the result against the expectation.

---
 Documentation/admin-guide/kernel-parameters.txt |    7 
 arch/x86/include/asm/time.h                     |    1 
 arch/x86/kernel/hpet.c                          |    4 
 arch/x86/kernel/tsc.c                           |   51 -
 drivers/clocksource/acpi_pm.c                   |    4 
 include/linux/clocksource.h                     |   24 
 kernel/time/Kconfig                             |   12 
 kernel/time/clocksource-wdtest.c                |  274 ++++-----
 kernel/time/clocksource.c                       |  726 +++++++++++-------------
 kernel/time/jiffies.c                           |    1 
 10 files changed, 523 insertions(+), 581 deletions(-)

@@ -7852,12 +7852,7 @@ Kernel parameters
 			(HPET or PM timer) on systems whose TSC frequency was
 			obtained from HW or FW using either an MSR or CPUID(0x15).
 			Warn if the difference is more than 500 ppm.
-			[x86] watchdog: Use TSC as the watchdog clocksource with
-			which to check other HW timers (HPET or PM timer), but
-			only on systems where TSC has been deemed trustworthy.
-			This will be suppressed by an earlier tsc=nowatchdog and
-			can be overridden by a later tsc=nowatchdog.  A console
-			message will flag any such suppression or overriding.
+			[x86] watchdog: Enforce the clocksource watchdog on TSC
 
 	tsc_early_khz=  [X86,EARLY] Skip early TSC calibration and use the given
 			value instead. Useful when the early TSC frequency discovery

@@ -7,7 +7,6 @@
 
 extern void hpet_time_init(void);
 extern bool pit_timer_init(void);
-extern bool tsc_clocksource_watchdog_disabled(void);
 
 extern struct clock_event_device *global_clock_event;
 

@@ -854,7 +854,7 @@ static struct clocksource clocksource_hp
 	.rating		= 250,
 	.read		= read_hpet,
 	.mask		= HPET_MASK,
-	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_CALIBRATED,
 	.resume		= hpet_resume_counter,
 };
 

@@ -1082,8 +1082,6 @@ int __init hpet_enable(void)
 	if (!hpet_counting())
 		goto out_nohpet;
 
-	if (tsc_clocksource_watchdog_disabled())
-		clocksource_hpet.flags |= CLOCK_SOURCE_MUST_VERIFY;
 	clocksource_register_hz(&clocksource_hpet, (u32)hpet_freq);
 
 	if (id & HPET_ID_LEGSUP) {

@@ -98,7 +98,7 @@ static struct clocksource clocksource_ac
 	.rating		= 200,
 	.read		= acpi_pm_read,
 	.mask		= (u64)ACPI_PM_MASK,
-	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_CALIBRATED,
 	.suspend	= acpi_pm_suspend,
 	.resume		= acpi_pm_resume,
 };

@@ -243,8 +243,6 @@ static int __init init_acpi_pm_clocksour
 		return -ENODEV;
 	}
 
-	if (tsc_clocksource_watchdog_disabled())
-		clocksource_acpi_pm.flags |= CLOCK_SOURCE_MUST_VERIFY;
 	return clocksource_register_hz(&clocksource_acpi_pm, PMTMR_TICKS_PER_SEC);
 }
 

@@ -44,8 +44,6 @@ struct module;
  * @shift:		Cycle to nanosecond divisor (power of two)
  * @max_idle_ns:	Maximum idle time permitted by the clocksource (nsecs)
  * @maxadj:		Maximum adjustment value to mult (~11%)
- * @uncertainty_margin:	Maximum uncertainty in nanoseconds per half second.
- *			Zero says to use default WATCHDOG_THRESHOLD.
  * @archdata:		Optional arch-specific data
  * @max_cycles:		Maximum safe cycle value which won't overflow on
  *			multiplication

@@ -105,7 +103,6 @@ struct clocksource {
 	u32			shift;
 	u64			max_idle_ns;
 	u32			maxadj;
-	u32			uncertainty_margin;
 #ifdef CONFIG_ARCH_CLOCKSOURCE_DATA
 	struct arch_clocksource_data archdata;
 #endif

@@ -133,6 +130,7 @@ struct clocksource {
 	struct list_head	wd_list;
 	u64			cs_last;
 	u64			wd_last;
+	unsigned int		wd_cpu;
 #endif
 	struct module		*owner;
 };

@@ -142,13 +140,16 @@ struct clocksource {
  */
 #define CLOCK_SOURCE_IS_CONTINUOUS		0x01
 #define CLOCK_SOURCE_MUST_VERIFY		0x02
+#define CLOCK_SOURCE_CALIBRATED			0x04
 
 #define CLOCK_SOURCE_WATCHDOG			0x10
 #define CLOCK_SOURCE_VALID_FOR_HRES		0x20
 #define CLOCK_SOURCE_UNSTABLE			0x40
 #define CLOCK_SOURCE_SUSPEND_NONSTOP		0x80
 #define CLOCK_SOURCE_RESELECT			0x100
-#define CLOCK_SOURCE_VERIFY_PERCPU		0x200
+#define CLOCK_SOURCE_WDTEST			0x200
+#define CLOCK_SOURCE_WDTEST_PERCPU		0x400
+
 /* simplify initialization of mask field */
 #define CLOCKSOURCE_MASK(bits) GENMASK_ULL((bits) - 1, 0)
 

@@ -298,21 +299,6 @@ static inline void timer_probe(void) {}
 #define TIMER_ACPI_DECLARE(name, table_id, fn)		\
 	ACPI_DECLARE_PROBE_ENTRY(timer, name, table_id, 0, NULL, 0, fn)
 
-static inline unsigned int clocksource_get_max_watchdog_retry(void)
-{
-	/*
-	 * When system is in the boot phase or under heavy workload, there
-	 * can be random big latencies during the clocksource/watchdog
-	 * read, so allow retries to filter the noise latency. As the
-	 * latency's frequency and maximum value goes up with the number of
-	 * CPUs, scale the number of retries with the number of online
-	 * CPUs.
-	 */
-	return (ilog2(num_online_cpus()) / 2) + 1;
-}
-
-void clocksource_verify_percpu(struct clocksource *cs);
-
 /**
  * struct clocksource_base - hardware abstraction for clock on which a clocksource
  *			is based

@@ -196,18 +196,6 @@ config HIGH_RES_TIMERS
 	  hardware is not capable then this option only increases
 	  the size of the kernel image.
 
-config CLOCKSOURCE_WATCHDOG_MAX_SKEW_US
-	int "Clocksource watchdog maximum allowable skew (in microseconds)"
-	depends on CLOCKSOURCE_WATCHDOG
-	range 50 1000
-	default 125
-	help
-	  Specify the maximum amount of allowable watchdog skew in
-	  microseconds before reporting the clocksource to be unstable.
-	  The default is based on a half-second clocksource watchdog
-	  interval and NTP's maximum frequency drift of 500 parts
-	  per million.	If the clocksource is good enough for NTP,
-	  it is good enough for the clocksource watchdog!
 endif
 
 config POSIX_AUX_CLOCKS

@@ -32,7 +32,6 @@ static u64 jiffies_read(struct clocksour
 static struct clocksource clocksource_jiffies = {
 	.name			= "jiffies",
 	.rating			= 1, /* lowest valid rating*/
-	.uncertainty_margin	= 32 * NSEC_PER_MSEC,
 	.read			= jiffies_read,
 	.mask			= CLOCKSOURCE_MASK(32),
 	.mult			= TICK_NSEC << JIFFIES_SHIFT, /* details above */

Great work Thomas!

...

On my 1920 thread BullSequana SH160 test system (16 sockets with Numascale UPI Node Controller), I find this approach is intrinsically robust against system latency.

On x86, I agree iterating at a per-thread level is needed rather than one thread per NUMA node, since the TSC_ADJUST architectural MSR is per-core and we want detection completeness.

On other architectures, completeness could be traded off for lower overhead if it is guaranteed each processor thread uses the same clock value, though this is actually is what the clocksource watchdog seeks to validate, so agreed on the current approach there too.

[ 1873.419375] clocksource: Watchdog remote CPU 1807 read timed out
[ 1900.419375] clocksource: Watchdog remote CPU 1861 read timed out
[ 1925.924374] clocksource: Watchdog remote CPU 1912 read timed out
[ 1937.420453] clocksource: Watchdog remote CPU 15 read timed out
[ 1937.925028] clocksource: Watchdog remote CPU 16 read timed out
[ 1949.073317] workqueue: drm_fb_helper_damage_work hogged CPU for

[ 1954.924464] clocksource: Watchdog remote CPU 50 read timed out
[ 2032.923596] clocksource: Watchdog remote CPU 206 read timed out
[ 2042.924367] clocksource: Watchdog remote CPU 226 read timed out
[ 2066.420624] clocksource: Watchdog remote CPU 273 read timed out
[ 2072.924015] clocksource: Watchdog remote CPU 286 read timed out
[ 2115.602465] workqueue: drm_fb_helper_damage_work hogged CPU for

[ 2139.924153] clocksource: Watchdog remote CPU 420 read timed out
[ 2143.419690] clocksource: Watchdog remote CPU 427 read timed out
[ 2147.420587] clocksource: Watchdog remote CPU 435 read timed out
[ 2160.924251] clocksource: Watchdog remote CPU 462 read timed out
[ 2165.419843] clocksource: Watchdog remote CPU 471 read timed out
[ 2170.442815] clocksource: Watchdog remote CPU 481 read timed out
[ 2221.420468] clocksource: Watchdog remote CPU 583 read timed out

--
Daniel J Blueman

Thank you!

It's per thread not per core.

But that aside the TSC_ADJUST integrity is already self monitored independent of the watchdog. See tsc_verify_tsc_adjust(). So we might get away with a per socket check as all threads of a socket are fed by the same ART (Always Running Timer) and the main concern is that the ARTs of sockets drift apart especially on systems with more than four sockets.

x86 is the only one which actually utilizes the watchdog.

Right. Idle is definitely an issue when the remote CPU is in a deep C-state.

My concern is that the control CPU might spin there for a millisecond with interrupts disabled, which is not really desired especially not on RT systems.

Something like the untested below delta patch should work.

Thanks,

tglx
---

@@ -7,6 +7,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/delay.h>
 #include <linux/device.h>
 #include <linux/clocksource.h>
 #include <linux/init.h>

@@ -124,7 +125,8 @@ static atomic_t watchdog_reset_pending;
 #define WATCHDOG_INTERVAL_NS		(WATCHDOG_INTERVAL * (NSEC_PER_SEC / HZ))
 
 /* Maximum time between two watchdog readouts */
-#define WATCHDOG_READOUT_MAX_NS		(50 * NSEC_PER_USEC)
+#define WATCHDOG_READOUT_MAX_US		50
+#define WATCHDOG_READOUT_MAX_NS		(WATCHDOG_READOUT_MAX_US * NSEC_PER_USEC)
 
 /* Shift values to calculate the approximate $N ppm of a given delta. */
 #define SHIFT_500PPM			11

@@ -136,6 +138,9 @@ static atomic_t watchdog_reset_pending;
 /* Five reads local and remote for inter CPU skew detection */
 #define WATCHDOG_REMOTE_MAX_SEQ		10
 
+/* Number of attempts to synchronize with a remote CPU */
+#define WATCHDOG_REMOTE_RETRIES		10
+
 static inline void clocksource_watchdog_lock(unsigned long *flags)
 {
 	spin_lock_irqsave(&watchdog_lock, *flags);

@@ -336,22 +341,17 @@ static void watchdog_check_skew_remote(v
 	atomic_dec(&wd->remote_inprogress);
 }
 
-static void watchdog_check_cpu_skew(struct clocksource *cs)
+static inline bool wd_csd_locked(struct watchdog_cpu_data *wd)
 {
-	unsigned int cpu = cpumask_next_wrap(watchdog_data.curr_cpu, cpu_online_mask);
-	struct watchdog_cpu_data *wd;
-
-	watchdog_data.curr_cpu = cpu;
-	/* Skip the current CPU. Handles num_online_cpus() == 1 as well */
-	if (cpu == smp_processor_id())
-		return;
+	return READ_ONCE(wd->csd.node.u_flags) & CSD_FLAG_LOCK;
+}
 
-	/* Don't interfere with the test mechanics */
-	if ((cs->flags & CLOCK_SOURCE_WDTEST) && !(cs->flags & CLOCK_SOURCE_WDTEST_PERCPU))
-		return;
+static void __watchdog_check_cpu_skew(struct clocksource *cs, unsigned int cpu)
+{
+	struct watchdog_cpu_data *wd;
 
 	wd = per_cpu_ptr(&watchdog_cpu_data, cpu);
-	if (atomic_read(&wd->remote_inprogress)) {
+	if (atomic_read(&wd->remote_inprogress) || wd_csd_locked(wd)) {
 		watchdog_data.result = WD_CPU_TIMEOUT;
 		return;
 	}

@@ -377,6 +377,29 @@ static void watchdog_check_cpu_skew(stru
 	}
 }
 
+static void watchdog_check_cpu_skew(struct clocksource *cs)
+{
+	unsigned int cpu = cpumask_next_wrap(watchdog_data.curr_cpu, cpu_online_mask);
+
+	watchdog_data.curr_cpu = cpu;
+	/* Skip the current CPU. Handles num_online_cpus() == 1 as well */
+	if (cpu == smp_processor_id())
+		return;
+
+	/* Don't interfere with the test mechanics */
+	if ((cs->flags & CLOCK_SOURCE_WDTEST) && !(cs->flags & CLOCK_SOURCE_WDTEST_PERCPU))
+		return;
+
+	for (int i = 0; i < WATCHDOG_REMOTE_RETRIES; i++) {
+		__watchdog_check_cpu_skew(cs, cpu);
+
+		if (watchdog_data.result != WD_CPU_TIMEOUT)
+			return;
+
+		udelay(WATCHDOG_READOUT_MAX_US);
+	}
+}
+
 static bool watchdog_check_freq(struct clocksource *cs, bool reset_pending)
 {
 	unsigned int ppm_shift = SHIFT_4000PPM;

Good step forward! We can also reduce remote cacheline invalidation by putting 'seq' into the cacheline after 'cpu_ts' by reordering:

struct watchdog_cpu_data {
       atomic_t                remote_inprogress;
       struct clocksource      *cs;
       enum wd_result          result;
       u64 cpu_ts[2];
       call_single_data_t      csd;
       atomic_t        seq; /* Keep in second cacheline to elide

unnecessary invalidation */
};

and reordering the inner loop:

        for (int seq = local + 2; seq < WATCHDOG_REMOTE_MAX_SEQ; seq += 2) {
                if (!watchdog_wait_seq(wd, start, seq))
                        return;

               /* Capture local timestamp before possible non-local
coherency overhead */
               now = cs->read(cs);

               /* Store local timestamp before reading remote to limit
coherency stalls */
               wd->cpu_ts[local] = now;
               prev = wd->cpu_ts[remote];
               delta = (now - prev) & cs->mask;

                if (delta > cs->max_raw_delta) {
                        watchdog_set_result(wd, WD_CPU_SKEWED);

With that said, with your latest change on the 1920 thread setup, WATCHDOG_READOUT_MAX_US 1000 is still needed to avoid timeouts during the previous adverse workload, however some timeouts are still seen during massive parallel process teardowns.

To limit overhead, perhaps it is sufficient to set the timeout to 100us, avoid retries (as the hardware thread may continue to be busy and will be rechecked later anyway), and log timeouts at the debug level if at all.

Thanks,
Dan
--
Daniel J Blueman

Good point.

Something like the below should work even with 50us. I left the print at INFO level for now. We can either change it to pr_info_once() or to debug as you said.

Thanks,

tglx
---

@@ -7950,12 +7950,7 @@ Kernel parameters
 			(HPET or PM timer) on systems whose TSC frequency was
 			obtained from HW or FW using either an MSR or CPUID(0x15).
 			Warn if the difference is more than 500 ppm.
-			[x86] watchdog: Use TSC as the watchdog clocksource with
-			which to check other HW timers (HPET or PM timer), but
-			only on systems where TSC has been deemed trustworthy.
-			This will be suppressed by an earlier tsc=nowatchdog and
-			can be overridden by a later tsc=nowatchdog.  A console
-			message will flag any such suppression or overriding.
+			[x86] watchdog: Enforce the clocksource watchdog on TSC
 
 	tsc_early_khz=  [X86,EARLY] Skip early TSC calibration and use the given
 			value instead. Useful when the early TSC frequency discovery

@@ -7,7 +7,6 @@
 
 extern void hpet_time_init(void);
 extern bool pit_timer_init(void);
-extern bool tsc_clocksource_watchdog_disabled(void);
 
 extern struct clock_event_device *global_clock_event;
 

@@ -854,7 +854,7 @@ static struct clocksource clocksource_hp
 	.rating		= 250,
 	.read		= read_hpet,
 	.mask		= HPET_MASK,
-	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_CALIBRATED,
 	.resume		= hpet_resume_counter,
 };
 

@@ -1082,8 +1082,6 @@ int __init hpet_enable(void)
 	if (!hpet_counting())
 		goto out_nohpet;
 
-	if (tsc_clocksource_watchdog_disabled())
-		clocksource_hpet.flags |= CLOCK_SOURCE_MUST_VERIFY;
 	clocksource_register_hz(&clocksource_hpet, (u32)hpet_freq);
 
 	if (id & HPET_ID_LEGSUP) {

@@ -98,7 +98,7 @@ static struct clocksource clocksource_ac
 	.rating		= 200,
 	.read		= acpi_pm_read,
 	.mask		= (u64)ACPI_PM_MASK,
-	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_CALIBRATED,
 	.suspend	= acpi_pm_suspend,
 	.resume		= acpi_pm_resume,
 };

@@ -243,8 +243,6 @@ static int __init init_acpi_pm_clocksour
 		return -ENODEV;
 	}
 
-	if (tsc_clocksource_watchdog_disabled())
-		clocksource_acpi_pm.flags |= CLOCK_SOURCE_MUST_VERIFY;
 	return clocksource_register_hz(&clocksource_acpi_pm, PMTMR_TICKS_PER_SEC);
 }
 

@@ -44,8 +44,6 @@ struct module;
  * @shift:		Cycle to nanosecond divisor (power of two)
  * @max_idle_ns:	Maximum idle time permitted by the clocksource (nsecs)
  * @maxadj:		Maximum adjustment value to mult (~11%)
- * @uncertainty_margin:	Maximum uncertainty in nanoseconds per half second.
- *			Zero says to use default WATCHDOG_THRESHOLD.
  * @archdata:		Optional arch-specific data
  * @max_cycles:		Maximum safe cycle value which won't overflow on
  *			multiplication

@@ -105,7 +103,6 @@ struct clocksource {
 	u32			shift;
 	u64			max_idle_ns;
 	u32			maxadj;
-	u32			uncertainty_margin;
 #ifdef CONFIG_ARCH_CLOCKSOURCE_DATA
 	struct arch_clocksource_data archdata;
 #endif

@@ -133,6 +130,7 @@ struct clocksource {
 	struct list_head	wd_list;
 	u64			cs_last;
 	u64			wd_last;
+	unsigned int		wd_cpu;
 #endif
 	struct module		*owner;
 };

@@ -142,13 +140,16 @@ struct clocksource {
  */
 #define CLOCK_SOURCE_IS_CONTINUOUS		0x01
 #define CLOCK_SOURCE_MUST_VERIFY		0x02
+#define CLOCK_SOURCE_CALIBRATED			0x04
 
 #define CLOCK_SOURCE_WATCHDOG			0x10
 #define CLOCK_SOURCE_VALID_FOR_HRES		0x20
 #define CLOCK_SOURCE_UNSTABLE			0x40
 #define CLOCK_SOURCE_SUSPEND_NONSTOP		0x80
 #define CLOCK_SOURCE_RESELECT			0x100
-#define CLOCK_SOURCE_VERIFY_PERCPU		0x200
+#define CLOCK_SOURCE_WDTEST			0x200
+#define CLOCK_SOURCE_WDTEST_PERCPU		0x400
+
 /* simplify initialization of mask field */
 #define CLOCKSOURCE_MASK(bits) GENMASK_ULL((bits) - 1, 0)
 

@@ -298,21 +299,6 @@ static inline void timer_probe(void) {}
 #define TIMER_ACPI_DECLARE(name, table_id, fn)		\
 	ACPI_DECLARE_PROBE_ENTRY(timer, name, table_id, 0, NULL, 0, fn)
 
-static inline unsigned int clocksource_get_max_watchdog_retry(void)
-{
-	/*
-	 * When system is in the boot phase or under heavy workload, there
-	 * can be random big latencies during the clocksource/watchdog
-	 * read, so allow retries to filter the noise latency. As the
-	 * latency's frequency and maximum value goes up with the number of
-	 * CPUs, scale the number of retries with the number of online
-	 * CPUs.
-	 */
-	return (ilog2(num_online_cpus()) / 2) + 1;
-}
-
-void clocksource_verify_percpu(struct clocksource *cs);
-
 /**
  * struct clocksource_base - hardware abstraction for clock on which a clocksource
  *			is based

@@ -196,18 +196,6 @@ config HIGH_RES_TIMERS
 	  hardware is not capable then this option only increases
 	  the size of the kernel image.
 
-config CLOCKSOURCE_WATCHDOG_MAX_SKEW_US
-	int "Clocksource watchdog maximum allowable skew (in microseconds)"
-	depends on CLOCKSOURCE_WATCHDOG
-	range 50 1000
-	default 125
-	help
-	  Specify the maximum amount of allowable watchdog skew in
-	  microseconds before reporting the clocksource to be unstable.
-	  The default is based on a half-second clocksource watchdog
-	  interval and NTP's maximum frequency drift of 500 parts
-	  per million.	If the clocksource is good enough for NTP,
-	  it is good enough for the clocksource watchdog!
 endif
 
 config POSIX_AUX_CLOCKS

@@ -32,7 +32,6 @@ static u64 jiffies_read(struct clocksour
 static struct clocksource clocksource_jiffies = {
 	.name			= "jiffies",
 	.rating			= 1, /* lowest valid rating*/
-	.uncertainty_margin	= 32 * NSEC_PER_MSEC,
 	.read			= jiffies_read,
 	.mask			= CLOCKSOURCE_MASK(32),
 	.mult			= TICK_NSEC << JIFFIES_SHIFT, /* details above */

Daniel!

Any chance you can give this a test ride on that 1920 thread monstrosity?

Thanks,

tglx

My apologies for the delays!

Comparing the execution time of the existing mainline clocksource_watchdog() to the proposed approach, there isn't significant additional overhead [1], which is excellent.

With that said, on the 16 socket (1920 thread) setup, we see most remote calls end up timing out with WATCHDOG_READOUT_MAX_US at 50, leading to excessive logging. pr_info_once() would be a good approach to avoid the spam, however I still feel we should use a higher (250-500us?) timeout to keep the mechanism effective.

I also feel if a remote hardware thread is seen to timeout, retrying has a high likelyhood of timing out also, so it may be cheaper in the bigger picture to not retry. Sensitivity could be increased by walking threads in socket order (S0T0 ... S15T0 S0T1 ... S15T1 ...). These two items are my only concerns.

Thanks and great work,
Dan

-- [1]

# bpftrace -e 'kprobe:clocksource_watchdog { @start[tid] = nsecs; }
kretprobe:clocksource_watchdog /@start[tid]/ {
  @lat_us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);}'

Mainline idle:

[16, 32)       2.9% |@                                                   |
[32, 64)      96.0% |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[64, 128)      0.8% |                                                    |
[128, 256)     0.4% |                                                    |

Proposed idle:

[32, 64)       0.0% |                                                    |
[64, 128)      0.1% |                                                    |
[128, 256)    99.6% |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[256, 512)     0.2% |                                                    |

Mainline loaded:

[8, 16)        0.2% |                                                    |
[16, 32)      82.7% |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[32, 64)      17.0% |@@@@@@@@@@                                          |
[64, 128)      0.1% |                                                    |
[128, 256)     0.0% |                                                    |

Proposed loaded:

[16, 32)       6.2% |@@@                                                 |
[32, 64)      93.8% |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[64, 128)      0.0% |                                                    |

Daniel J Blueman

Right. So I ditched the immediate retry and replaced the hard coded timeout with a runtime calculated one when NUMA is enabled. It's a reasonable assumption that insanely big systems have properly initialized SLIT/SRAT tables. So we can just [ab]use the node distance to determine the timeout for a remote CPU.

Thanks,

tglx
---
From: Thomas Gleixner <[email protected]>
Subject: clocksource: Rewrite watchdog code completely
Date: Sat, 24 Jan 2026 00:18:01 +0100

From: Thomas Gleixner <[email protected]>

The clocksource watchdog code has over time reached the state of an unpenetrable maze of duct tape and staples. The original design, which was made in the context of systems far smaller than today, is based on the assumption that the to be monitored clocksource (TSC) can be trivially compared against a known to be stable clocksource (HPET/ACPI-PM timer).

Over the years it turned out that this approach has major flaws:

- Long delays between watchdog invocations can result in wrap arounds of the reference clocksource

- Scalability of the reference clocksource readout can degrade on large multi-socket systems due to interconnect congestion

This was addressed with various heuristics which degraded the accurracy of the watchdog to the point that it fails to detect actual TSC problems on older hardware which exposes slow inter CPU drifts due to firmware manipulating the TSC to hide SMI time.

To address this and bring back sanity to the watchdog, rewrite the code completely with a different approach:

1) Restrict the validation against a reference clocksource to the boot CPU, which is usually the CPU/Socket closest to the legacy block which contains the reference source (HPET/ACPI-PM timer). Validate that the reference readout is within a bound latency so that the actual comparison against the TSC stays within 500ppm as long as the clocks are stable.

2) Compare the TSCs of the other CPUs in a round robin fashion against the boot CPU in the same way the TSC synchronization on CPU hotplug works. This still can suffer from delayed reaction of the remote CPU to the SMP function call and the latency of the control variable cache line. But this latency is not affecting correctness. It only affects the accuracy. With low contention the readout latency is in the low nanoseconds range, which detects even slight skews between CPUs. Under high contention this becomes obviously less accurate, but still detects slow skews reliably as it solely relies on subsequent readouts being monotonically increasing. It just can take slightly longer to detect the issue.

3) Rewrite the watchdog test so it tests the various mechanisms one by one and validating the result against the expectation.

---
V2: Make it more cache line friendly and tweak it further for insane big
machines - Daniel

---
 Documentation/admin-guide/kernel-parameters.txt |    7 
 arch/x86/include/asm/time.h                     |    1 
 arch/x86/kernel/hpet.c                          |    4 
 arch/x86/kernel/tsc.c                           |   49 -
 drivers/clocksource/acpi_pm.c                   |    4 
 include/linux/clocksource.h                     |   28 
 kernel/time/Kconfig                             |   12 
 kernel/time/clocksource-wdtest.c                |  274 ++++----
 kernel/time/clocksource.c                       |  788 ++++++++++++------------
 kernel/time/jiffies.c                           |    1 
 10 files changed, 583 insertions(+), 585 deletions(-)

@@ -7950,12 +7950,7 @@ Kernel parameters
 			(HPET or PM timer) on systems whose TSC frequency was
 			obtained from HW or FW using either an MSR or CPUID(0x15).
 			Warn if the difference is more than 500 ppm.
-			[x86] watchdog: Use TSC as the watchdog clocksource with
-			which to check other HW timers (HPET or PM timer), but
-			only on systems where TSC has been deemed trustworthy.
-			This will be suppressed by an earlier tsc=nowatchdog and
-			can be overridden by a later tsc=nowatchdog.  A console
-			message will flag any such suppression or overriding.
+			[x86] watchdog: Enforce the clocksource watchdog on TSC
 
 	tsc_early_khz=  [X86,EARLY] Skip early TSC calibration and use the given
 			value instead. Useful when the early TSC frequency discovery

@@ -7,7 +7,6 @@
 
 extern void hpet_time_init(void);
 extern bool pit_timer_init(void);
-extern bool tsc_clocksource_watchdog_disabled(void);
 
 extern struct clock_event_device *global_clock_event;
 

@@ -854,7 +854,7 @@ static struct clocksource clocksource_hp
 	.rating		= 250,
 	.read		= read_hpet,
 	.mask		= HPET_MASK,
-	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_CALIBRATED,
 	.resume		= hpet_resume_counter,
 };
 

@@ -1082,8 +1082,6 @@ int __init hpet_enable(void)
 	if (!hpet_counting())
 		goto out_nohpet;
 
-	if (tsc_clocksource_watchdog_disabled())
-		clocksource_hpet.flags |= CLOCK_SOURCE_MUST_VERIFY;
 	clocksource_register_hz(&clocksource_hpet, (u32)hpet_freq);
 
 	if (id & HPET_ID_LEGSUP) {

@@ -98,7 +98,7 @@ static struct clocksource clocksource_ac
 	.rating		= 200,
 	.read		= acpi_pm_read,
 	.mask		= (u64)ACPI_PM_MASK,
-	.flags		= CLOCK_SOURCE_IS_CONTINUOUS,
+	.flags		= CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_CALIBRATED,
 	.suspend	= acpi_pm_suspend,
 	.resume		= acpi_pm_resume,
 };

@@ -243,8 +243,6 @@ static int __init init_acpi_pm_clocksour
 		return -ENODEV;
 	}
 
-	if (tsc_clocksource_watchdog_disabled())
-		clocksource_acpi_pm.flags |= CLOCK_SOURCE_MUST_VERIFY;
 	return clocksource_register_hz(&clocksource_acpi_pm, PMTMR_TICKS_PER_SEC);
 }
 

@@ -44,8 +44,6 @@ struct module;
  * @shift:		Cycle to nanosecond divisor (power of two)
  * @max_idle_ns:	Maximum idle time permitted by the clocksource (nsecs)
  * @maxadj:		Maximum adjustment value to mult (~11%)
- * @uncertainty_margin:	Maximum uncertainty in nanoseconds per half second.
- *			Zero says to use default WATCHDOG_THRESHOLD.
  * @archdata:		Optional arch-specific data
  * @max_cycles:		Maximum safe cycle value which won't overflow on
  *			multiplication

@@ -105,7 +103,6 @@ struct clocksource {
 	u32			shift;
 	u64			max_idle_ns;
 	u32			maxadj;
-	u32			uncertainty_margin;
 #ifdef CONFIG_ARCH_CLOCKSOURCE_DATA
 	struct arch_clocksource_data archdata;
 #endif

@@ -133,6 +130,7 @@ struct clocksource {
 	struct list_head	wd_list;
 	u64			cs_last;
 	u64			wd_last;
+	unsigned int		wd_cpu;
 #endif
 	struct module		*owner;
 };

@@ -142,15 +140,18 @@ struct clocksource {
  */
 #define CLOCK_SOURCE_IS_CONTINUOUS		0x01
 #define CLOCK_SOURCE_MUST_VERIFY		0x02
+#define CLOCK_SOURCE_CALIBRATED			0x04
 
 #define CLOCK_SOURCE_WATCHDOG			0x10
 #define CLOCK_SOURCE_VALID_FOR_HRES		0x20
 #define CLOCK_SOURCE_UNSTABLE			0x40
 #define CLOCK_SOURCE_SUSPEND_NONSTOP		0x80
 #define CLOCK_SOURCE_RESELECT			0x100
-#define CLOCK_SOURCE_VERIFY_PERCPU		0x200
-#define CLOCK_SOURCE_CAN_INLINE_READ		0x400
-#define CLOCK_SOURCE_HAS_COUPLED_CLOCK_EVENT	0x800
+#define CLOCK_SOURCE_CAN_INLINE_READ		0x200
+#define CLOCK_SOURCE_HAS_COUPLED_CLOCK_EVENT	0x400
+
+#define CLOCK_SOURCE_WDTEST			0x800
+#define CLOCK_SOURCE_WDTEST_PERCPU		0x1000
 
 /* simplify initialization of mask field */
 #define CLOCKSOURCE_MASK(bits) GENMASK_ULL((bits) - 1, 0)

@@ -301,21 +302,6 @@ static inline void timer_probe(void) {}
 #define TIMER_ACPI_DECLARE(name, table_id, fn)		\
 	ACPI_DECLARE_PROBE_ENTRY(timer, name, table_id, 0, NULL, 0, fn)
 
-static inline unsigned int clocksource_get_max_watchdog_retry(void)
-{
-	/*
-	 * When system is in the boot phase or under heavy workload, there
-	 * can be random big latencies during the clocksource/watchdog
-	 * read, so allow retries to filter the noise latency. As the
-	 * latency's frequency and maximum value goes up with the number of
-	 * CPUs, scale the number of retries with the number of online
-	 * CPUs.
-	 */
-	return (ilog2(num_online_cpus()) / 2) + 1;
-}
-
-void clocksource_verify_percpu(struct clocksource *cs);
-
 /**
  * struct clocksource_base - hardware abstraction for clock on which a clocksource
  *			is based

@@ -212,18 +212,6 @@ config HIGH_RES_TIMERS
 	  hardware is not capable then this option only increases
 	  the size of the kernel image.
 
-config CLOCKSOURCE_WATCHDOG_MAX_SKEW_US
-	int "Clocksource watchdog maximum allowable skew (in microseconds)"
-	depends on CLOCKSOURCE_WATCHDOG
-	range 50 1000
-	default 125
-	help
-	  Specify the maximum amount of allowable watchdog skew in
-	  microseconds before reporting the clocksource to be unstable.
-	  The default is based on a half-second clocksource watchdog
-	  interval and NTP's maximum frequency drift of 500 parts
-	  per million.	If the clocksource is good enough for NTP,
-	  it is good enough for the clocksource watchdog!
 endif
 
 config POSIX_AUX_CLOCKS

@@ -32,7 +32,6 @@ static u64 jiffies_read(struct clocksour
 static struct clocksource clocksource_jiffies = {
 	.name			= "jiffies",
 	.rating			= 1, /* lowest valid rating*/
-	.uncertainty_margin	= 32 * NSEC_PER_MSEC,
 	.read			= jiffies_read,
 	.mask			= CLOCKSOURCE_MASK(32),
 	.mult			= TICK_NSEC << JIFFIES_SHIFT, /* details above */

The node distance matches well: changing "if (wd_seq > WATCHDOG_READOUT_MAX_NS) {" to use WATCHDOG_NUMA_MAX_TIMEOUT_NS, this presented no regressions on 16 sockets/1920 threads during idle and a "stress-ng --msyncmany 0" all thread adverse workload for 2h.

Thanks and great work Thomas!
Dan
--
Daniel J Blueman

I assume you wanted to say Tested/Acked/Reviewed-by or a combination of that. Please clarify.

Thank you for your help and suggestions!

Thomas

Thanks,

Dan
--
Daniel J Blueman

The UEFI picks the boot CPU so the kernel does not have control over that. On the other hand, I think the CPU that is connected to the southbridge chip (by DMI or PCIe) will be selected in the majority of UEFI implementations. Even if the boot CPU had to use the inter-processor link the readout latency should often pass the 50 microsecond threshold. This is a histogram of the hpet-tsc-hpet readout latency (in nanoseconds) as measured by the old clocksource watchdog (reads carried out from all CPUs on the machine):

wd_delay Duration Distribution
wd_delay Duration Average: 7822 +- 9413 (min 2875, max 77916)

        Range      Count
    0 -  5000       2766 (73.10%)
 5000 - 10000        383 (10.12%)
20000 - 25000        402 (10.62%)
25000 - 30000         94 ( 2.48%)
30000 - 35000         49 ( 1.29%)
35000 - 40000         35 ( 0.92%)
40000 - 45000         21 ( 0.55%)
45000 - 50000         14 ( 0.37%)
50000 - 55000          7 ( 0.18%)
55000 - 60000          3 ( 0.08%)
60000 - 65000          4 ( 0.11%)
65000 - 70000          2 ( 0.05%)
70000 - 75000          1 ( 0.03%)
75000 - 80000          3 ( 0.08%)

Total count: 3,784

The machine has 8 NUMA nodes with 960x Intel Xeon Platinum 8490H. The machine was running:
stress-ng -t 30m --cpu 480 --switch 520
It definitely does not represent effect of any arbitrary workload on the inter-processor link but it is a data point.

There is one issue: What if the reference clocksource itself experiences time skew? I have seen a case like this with the sgi_rtc clocksource. I created a debugging kernel with the HPET as a second watchdog (not affecting the decisions by the watchdog) and got this result:

The intervals measured by the TSC and the HPET match very well; the sgi_rtc is off. Even the new implementation of the clocksource watchdog would be susceptible to the reference clocksource experiencing time skew. I think the clocksource watchdog needs to make the assumption that the reference clocksource is right, and the onus should be on hardware developers to make sure the reference clocksource is accurate. In reality, one has to resort to disabling the reference clocksource experiencing time skew or, at least, decreasing the rating of that clocksource.

Jiri Wiesner
SUSE Labs

Picking a remote node CPU would be insane, but yes BIOSes are insane by definition.

Yes, we have to make the assumption that the watchdog clocksource is actually stable and accurate. If the sgi_rtc is un-reliable, then it should be rated down. AFAICT it is per blade and I have no idea how synchronized it is accross blades.

Indeed.

That was the design goal of that rewrite. Glad you like it.

Thanks,

tglx

Yes, I was mostly just being paranoid. Please include:

in the next version of the patchset.

Jiri Wiesner
SUSE Labs

On Sat, Jan 24, 2026 at 12:18:01AM +0100, Thomas Gleixner wrote:

Patch 2 and 4 should say "Don't" in the subject title. :)

"impenetrable"

"accuracy"

But those are just nitpicks.

The overall thing sounds good to me, I haven't done any indepth review but from a cursory look, it looks nice.

Testing looks good too on one machine so

I say, let's queue it now so that it can get wider testing.

Thx.

Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

I'll wait for Daniel to report back from his 1920 CPU monstrosity and do that then.

Thanks

tglx

Hi,

We have been seeing intermittent early-boot guest hangs in our CI for some time. Guests stop making serial progress during the first few seconds after SMP bring-up - no panic, no oops, no other error messages the vm just hangs.

The issue appears strongly tied to high vCPU count: we reproduce with smp=255 and have not seen it on smaller guests in the same CI matrix.

We bisected the guest kernel between a known-good v7.0 and known-bad v7.1-rc1 using git bisect + KVM reboot stress (5 guests, smp=255, 500 parallel reboots).

First bad: 763aacf86f1baefb134c70813aa8c72d1675d738 "clocksource:
Rewrite watchdog code completely"
Good parent: 1432f9d4e8aa2d7585b678bdd0b740597af00d6e "clocksource:
Don't use non-continuous clocksources as watchdog" (= 763aacf8^, immediate parent)

Environment

Hypervisor: QEMU/KVM on bare metal (Intel and AMD hosts)

Guests: 5 VMs each 255 vCPUs, 44GB RAM
Guest kernel: 7.0.0-rc1+ at 763aacf8 (bad); parent at 1432f9d4 (good) [base: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v7.1-rc1]

Boot path: direct kernel boot (-kernel bzImage), serial console only
Workload: boot + serial login + reboot loop

Boot progresses through SMP bring-up and into early initcalls Serial then goes silent ~1–2s after boot
The exact stuck initcall varies by timing (e.g. sha1_mod_init, activate_jump_labels, sock_init) — not always the same function:

Workaround for this is to boot Guest with kernel cmdline:
tsc=nowatchdog. With that, the same CI setup completed 1000 reboot iterations on both Intel and AMD hosts with zero hangs.

I have attached below logs:

amd-vm4-qemu-cmd.txt
amd-vm4-serial-excerpt.log
amd-vm4-crash-excerpt.txt
guest_kconfig-excerpt Relevant guest .config options (clocksource/TSC/SMP/KVM guest). Full .config available on request.

Intel host shows the same hang pattern (sha1_mod_init path); full serial/crash and untrimmed logs available on request as they are big in size.

Happy to share any other information or help test a fix.

Thanks,
Srikanth Aithal <[email protected]>

qemu-system-x86_64 \
-name \
vm4 \
-blockdev \
node-name=file_ovmf_code,driver=file,filename=OvmfX64/OVMF_CODE.fd,auto-read-only=on,discard=unmap \
-blockdev \
node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
-blockdev \
node-name=file_ovmf_vars,driver=file,filename=vm4_26_04-server_vm4_qcow2_filesystem_VARS.raw,auto-read-only=on,discard=unmap \
-blockdev \
node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
-machine \
q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars,memory-backend=mem-machine_mem \
-device \
pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device \
pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \
-device \
qemu-xhci,id=usb1,bus=pcie-pci-bridge-0,addr=0x1 \
-nodefaults \
-vga \
none \
-object \
memory-backend-ram,size=45056M,id=mem-machine_mem \
-m \
45056 \
-smp \
255,maxcpus=255,cores=255,threads=1,sockets=1 \
-cpu \
EPYC,+svm,+x2apic \
-chardev \
socket,id=qmp,path=/tmp/repro_2ea6jhk9/vm4/qmp.sock,server=on,wait=off \
-mon \
chardev=qmp,mode=control \
-chardev \
socket,id=serial0,path=/tmp/repro_2ea6jhk9/vm4/serial.sock,server=on,wait=off \
-device \
isa-serial,id=serial0,chardev=serial0 \
-chardev \
socket,id=seabios,path=/tmp/repro_2ea6jhk9/vm4/seabios.sock,server=on,wait=off \
-device \
isa-debugcon,chardev=seabios,iobase=0x402 \
-device \
pvpanic,ioport=0x505 \
-device \
pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device \
virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-1,addr=0x0,disable-legacy=on,iommu_platform=on \
-blockdev \
node-name=file_image1,driver=file,filename=26.04-server_vm4.qcow2 \
-blockdev \
node-name=drive_image1,driver=qcow2,file=file_image1 \
-device \
scsi-hd,id=image1,drive=drive_image1 \
-kernel \
bzImage \
-append \
root=/dev/sda1 rw console=ttyS0,115200n8 earlyprintk=ttyS0,115200 net.ifnames=0 biosdevname=0 movable_node swiotlb=65536 initcall_debug loglevel=8 ignore_loglevel dyndbg="file kernel/time/clocksource.c +p; file kernel/time/timekeeping.c +p; file arch/x86/kernel/tsc.c +p" \
-nographic \
-rtc \
base=utc,clock=host \
-boot \
menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device \
pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x2,chassis=4 \
-S \
-pidfile \
/tmp/repro_2ea6jhk9/vm4/qemu.pid

2026-09-17 08:10:45: [    0.000000] Linux version 7.0.0-rc1+ (root@xxxx) (gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, GNU ld (GNU Binutils for Ubuntu) 2.46) #4 SMP PREEMPT_DYNAMIC Thu Sep 17 07:03:37 UTC 2026
2026-09-17 08:10:45: [    0.000000] Command line: root=/dev/sda1 rw console=ttyS0,115200n8 earlyprintk=ttyS0,115200 net.ifnames=0 biosdevname=0 movable_node swiotlb=65536 initcall_debug loglevel=8 ignore_loglevel dyndbg="file kernel/time/clocksource.c +p; file kernel/time/timekeeping.c +p; file arch/x86/kernel/tsc.c +p"
2026-09-17 08:10:45: [    0.000000] BIOS-provided physical RAM map:
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [gap 0x00000000000a0000-0x00000000000fffff]
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000007fffff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000800000-0x0000000000807fff]  ACPI NVS
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000808000-0x000000000080afff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000000080b000-0x000000000080bfff]  ACPI NVS
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000000080c000-0x0000000000811fff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000812000-0x0000000000812fff]  device reserved
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000813000-0x0000000000813fff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000814000-0x00000000008fffff]  ACPI NVS
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000000900000-0x000000007bf02fff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007bf03000-0x000000007bf04fff]  ACPI data
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007bf05000-0x000000007dc52fff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007dc53000-0x000000007dd5efff]  device reserved
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007dd5f000-0x000000007e6ecfff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007e6ed000-0x000000007e96cfff]  device reserved
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007e96d000-0x000000007e97efff]  ACPI data
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007e97f000-0x000000007e9fefff]  ACPI NVS
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007e9ff000-0x000000007f703fff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007f704000-0x000000007f787fff]  device reserved
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000007f788000-0x000000007fffffff]  ACPI NVS
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [gap 0x0000000080000000-0x00000000dfffffff]
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff]  device reserved
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [gap 0x00000000f0000000-0x00000000feffbfff]
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff]  device reserved
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [gap 0x00000000ff000000-0x00000000ffbfffff]
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x00000000ffc00000-0x00000000ffffffff]  device reserved
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000b7fffffff]  System RAM
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [gap 0x0000000b80000000-0x000000fcffffffff]
2026-09-17 08:10:45: [    0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff]  device reserved
2026-09-17 08:10:45: [    0.000000] printk: legacy bootconsole [earlyser0] enabled
2026-09-17 08:10:45: [    0.000000] printk: debug: ignoring loglevel setting.
2026-09-17 08:10:45: [    0.000000] NX (Execute Disable) protection: active
2026-09-17 08:10:45: [    0.000000] APIC: Static calls initialized
2026-09-17 08:10:45: [    0.000000] efi: EFI v2.7 by EDK II
2026-09-17 08:10:45: [    0.000000] efi: SMBIOS=0x7e73f000 SMBIOS 3.0=0x7e73d000 ACPI=0x7e97e000 ACPI 2.0=0x7e97e014 MEMATTR=0x7bf3d298 RNG=0x7e96e018 
2026-09-17 08:10:45: [    0.000000] random: crng init done
2026-09-17 08:10:45: [    0.000000] efi: Remove mem102: MMIO range=[0xffc00000-0xffffffff] (4MB) from e820 map
2026-09-17 08:10:45: [    0.000000] e820: remove [mem 0xffc00000-0xffffffff] device reserved
2026-09-17 08:10:45: [    0.000000] SMBIOS 3.0.0 present.
2026-09-17 08:10:45: [    0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
2026-09-17 08:10:45: [    0.000000] DMI: Memory slots populated: 1/1
2026-09-17 08:10:45: [    0.000000] Hypervisor detected: KVM
2026-09-17 08:10:45: [    0.000000] last_pfn = 0x7f704 max_arch_pfn = 0x400000000
2026-09-17 08:10:45: [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
2026-09-17 08:10:45: [    0.000001] kvm-clock: using sched offset of 920028785112 cycles
2026-09-17 08:10:45: [    0.001255] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
2026-09-17 08:10:45: [    0.005166] tsc: Detected 2396.400 MHz processor
2026-09-17 08:10:45: [    0.006386] e820: update [mem 0x00000000-0x00000fff] System RAM ==> device reserved
2026-09-17 08:10:45: [    0.008198] e820: remove [mem 0x000a0000-0x000fffff] System RAM
2026-09-17 08:10:45: [    0.009619] last_pfn = 0xb80000 max_arch_pfn = 0x400000000
2026-09-17 08:10:45: [    0.010953] MTRR map: 4 entries (2 fixed + 2 variable; max 18), built from 8 variable MTRRs
2026-09-17 08:10:45: [    0.012932] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
2026-09-17 08:10:45: Memory KASLR using RDRAND RDTSC...
2026-09-17 08:10:45: [    0.015369] last_pfn = 0x7f704 max_arch_pfn = 0x400000000
2026-09-17 08:10:45: [    0.019852] Using GB pages for direct mapping
2026-09-17 08:10:45: [    0.021399] Secure boot disabled
2026-09-17 08:10:45: [    0.022160] ACPI: Early table checksum verification disabled
2026-09-17 08:10:45: [    0.023498] ACPI: RSDP 0x000000007E97E014 000024 (v02 BOCHS )
2026-09-17 08:10:45: [    0.024854] ACPI: XSDT 0x000000007E97D0E8 00004C (v01 BOCHS  BXPC     00000001      01000013)
2026-09-17 08:10:45: [    0.026884] ACPI: FACP 0x000000007E973000 0000F4 (v03 BOCHS  BXPC     00000001 BXPC 00000001)
2026-09-17 08:10:45: [    0.028907] ACPI: DSDT 0x000000007E974000 00860C (v01 BOCHS  BXPC     00000001 BXPC 00000001)
2026-09-17 08:10:45: [    0.030930] ACPI: FACS 0x000000007E9DD000 000040
2026-09-17 08:10:45: [    0.032028] ACPI: APIC 0x000000007E972000 000868 (v03 BOCHS  BXPC     00000001 BXPC 00000001)
2026-09-17 08:10:45: [    0.034075] ACPI: HPET 0x000000007E971000 000038 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
2026-09-17 08:10:45: [    0.036089] ACPI: MCFG 0x000000007E970000 00003C (v01 BOCHS  BXPC     00000001 BXPC 00000001)
2026-09-17 08:10:45: [    0.038122] ACPI: WAET 0x000000007E96F000 000028 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
2026-09-17 08:10:45: [    0.040137] ACPI: Reserving FACP table memory at [mem 0x7e973000-0x7e9730f3]
2026-09-17 08:10:45: [    0.041800] ACPI: Reserving DSDT table memory at [mem 0x7e974000-0x7e97c60b]
2026-09-17 08:10:45: [    0.043470] ACPI: Reserving FACS table memory at [mem 0x7e9dd000-0x7e9dd03f]
2026-09-17 08:10:45: [    0.045132] ACPI: Reserving APIC table memory at [mem 0x7e972000-0x7e972867]
2026-09-17 08:10:45: [    0.046806] ACPI: Reserving HPET table memory at [mem 0x7e971000-0x7e971037]
2026-09-17 08:10:45: [    0.048485] ACPI: Reserving MCFG table memory at [mem 0x7e970000-0x7e97003b]
2026-09-17 08:10:45: [    0.050156] ACPI: Reserving WAET table memory at [mem 0x7e96f000-0x7e96f027]
2026-09-17 08:10:45: [    0.052279] No NUMA configuration found
2026-09-17 08:10:45: [    0.053199] Faking a node at [mem 0x0000000000000000-0x0000000b7fffffff]
2026-09-17 08:10:45: [    0.057251] NODE_DATA(0) allocated [mem 0xb7fffb4c0-0xb7fffffff]
2026-09-17 08:10:45: [    0.061411] ACPI: PM-Timer IO Port: 0x608
2026-09-17 08:10:45: [    0.062398] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
2026-09-17 08:10:45: [    0.063781] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
2026-09-17 08:10:45: [    0.065404] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
2026-09-17 08:10:45: [    0.066902] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
2026-09-17 08:10:45: [    0.068459] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
2026-09-17 08:10:45: [    0.070037] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
2026-09-17 08:10:45: [    0.071628] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
2026-09-17 08:10:45: [    0.073245] ACPI: Using ACPI (MADT) for SMP configuration information
2026-09-17 08:10:45: [    0.074768] ACPI: HPET id: 0x8086a201 base: 0xfed00000
2026-09-17 08:10:45: [    0.075991] CPU topo: Max. logical packages:   1
2026-09-17 08:10:45: [    0.077102] CPU topo: Max. logical dies:       1
2026-09-17 08:10:45: [    0.078236] CPU topo: Max. dies per package:   1
2026-09-17 08:10:45: [    0.079361] CPU topo: Max. threads per core:   1
2026-09-17 08:10:45: [    0.080485] CPU topo: Num. cores per package:   255
2026-09-17 08:10:45: [    0.081659] CPU topo: Num. threads per package: 255
2026-09-17 08:10:45: [    0.082813] CPU topo: Allowing 255 present CPUs plus 0 hotplug CPUs
2026-09-17 08:10:45: [    0.084341] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()
2026-09-17 08:10:45: [    0.086026] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
2026-09-17 08:10:45: [    0.087839] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000fffff]
2026-09-17 08:10:45: [    0.089644] PM: hibernation: Registered nosave memory: [mem 0x00800000-0x00807fff]
2026-09-17 08:10:45: [    0.091457] PM: hibernation: Registered nosave memory: [mem 0x0080b000-0x0080bfff]
2026-09-17 08:10:45: [    0.093252] PM: hibernation: Registered nosave memory: [mem 0x00812000-0x00812fff]
2026-09-17 08:10:45: [    0.095052] PM: hibernation: Registered nosave memory: [mem 0x00814000-0x008fffff]
2026-09-17 08:10:45: [    0.096853] PM: hibernation: Registered nosave memory: [mem 0x7bf03000-0x7bf04fff]
2026-09-17 08:10:45: [    0.098646] PM: hibernation: Registered nosave memory: [mem 0x7dc53000-0x7dd5efff]
2026-09-17 08:10:45: [    0.100463] PM: hibernation: Registered nosave memory: [mem 0x7e6ed000-0x7e9fefff]
2026-09-17 08:10:45: [    0.102335] PM: hibernation: Registered nosave memory: [mem 0x7f704000-0xffffffff]
2026-09-17 08:10:45: [    0.104170] [gap 0x80000000-0xdfffffff] available for PCI devices
2026-09-17 08:10:45: [    0.105674] Booting paravirtualized kernel on KVM
2026-09-17 08:10:45: [    0.106811] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
2026-09-17 08:10:45: [    0.114839] Zone ranges:
2026-09-17 08:10:45: [    0.115461]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
2026-09-17 08:10:45: [    0.116974]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
2026-09-17 08:10:45: [    0.118486]   Normal   [mem 0x0000000100000000-0x0000000b7fffffff]
2026-09-17 08:10:45: [    0.119991]   Device   empty
2026-09-17 08:10:45: [    0.120687] Movable zone start for each node
2026-09-17 08:10:45: [    0.121727] Early memory node ranges
2026-09-17 08:10:45: [    0.122598]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
2026-09-17 08:10:45: [    0.124130]   node   0: [mem 0x0000000000100000-0x00000000007fffff]
2026-09-17 08:10:45: [    0.125732]   node   0: [mem 0x0000000000808000-0x000000000080afff]
2026-09-17 08:10:45: [    0.127263]   node   0: [mem 0x000000000080c000-0x0000000000811fff]
2026-09-17 08:10:45: [    0.128788]   node   0: [mem 0x0000000000813000-0x0000000000813fff]
2026-09-17 08:10:45: [    0.130309]   node   0: [mem 0x0000000000900000-0x000000007bf02fff]
2026-09-17 08:10:45: [    0.131832]   node   0: [mem 0x000000007bf05000-0x000000007dc52fff]
2026-09-17 08:10:45: [    0.133364]   node   0: [mem 0x000000007dd5f000-0x000000007e6ecfff]
2026-09-17 08:10:45: [    0.134888]   node   0: [mem 0x000000007e9ff000-0x000000007f703fff]
2026-09-17 08:10:45: [    0.136402]   node   0: [mem 0x0000000100000000-0x0000000b7fffffff]
2026-09-17 08:10:45: [    0.137937] Initmem setup node 0 [mem 0x0000000000001000-0x0000000b7fffffff]
2026-09-17 08:10:45: [    0.139660] On node 0, zone DMA: 1 pages in unavailable ranges
2026-09-17 08:10:45: [    0.141078] On node 0, zone DMA: 96 pages in unavailable ranges
2026-09-17 08:10:45: [    0.142525] On node 0, zone DMA: 8 pages in unavailable ranges
2026-09-17 08:10:45: [    0.143936] On node 0, zone DMA: 1 pages in unavailable ranges
2026-09-17 08:10:45: [    0.145368] On node 0, zone DMA: 1 pages in unavailable ranges
2026-09-17 08:10:45: [    0.146790] On node 0, zone DMA: 236 pages in unavailable ranges
2026-09-17 08:10:45: [    0.149608] On node 0, zone DMA32: 2 pages in unavailable ranges
2026-09-17 08:10:45: [    0.151088] On node 0, zone DMA32: 268 pages in unavailable ranges
2026-09-17 08:10:45: [    0.152594] On node 0, zone DMA32: 786 pages in unavailable ranges
2026-09-17 08:10:46: [    0.184046] On node 0, zone Normal: 2300 pages in unavailable ranges
2026-09-17 08:10:46: [    0.185614] setup_percpu: NR_CPUS:512 nr_cpumask_bits:255 nr_cpu_ids:255 nr_node_ids:1
2026-09-17 08:10:46: [    0.192043] percpu: Embedded 58 pages/cpu s200704 r8192 d28672 u262144
2026-09-17 08:10:46: [    0.193667] pcpu-alloc: s200704 r8192 d28672 u262144 alloc=1*2097152
2026-09-17 08:10:46: [    0.195206] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 
2026-09-17 08:10:46: [    0.196581] pcpu-alloc: [0] 008 009 010 011 012 013 014 015 
2026-09-17 08:10:46: [    0.197972] pcpu-alloc: [0] 016 017 018 019 020 021 022 023 
2026-09-17 08:10:46: [    0.199341] pcpu-alloc: [0] 024 025 026 027 028 029 030 031 
2026-09-17 08:10:46: [    0.200727] pcpu-alloc: [0] 032 033 034 035 036 037 038 039 
2026-09-17 08:10:46: [    0.202105] pcpu-alloc: [0] 040 041 042 043 044 045 046 047 
2026-09-17 08:10:46: [    0.203458] pcpu-alloc: [0] 048 049 050 051 052 053 054 055 
2026-09-17 08:10:46: [    0.204815] pcpu-alloc: [0] 056 057 058 059 060 061 062 063 
2026-09-17 08:10:46: [    0.206168] pcpu-alloc: [0] 064 065 066 067 068 069 070 071 
2026-09-17 08:10:46: [    0.207511] pcpu-alloc: [0] 072 073 074 075 076 077 078 079 
2026-09-17 08:10:46: [    0.208857] pcpu-alloc: [0] 080 081 082 083 084 085 086 087 
2026-09-17 08:10:46: [    0.210213] pcpu-alloc: [0] 088 089 090 091 092 093 094 095 
2026-09-17 08:10:46: [    0.211546] pcpu-alloc: [0] 096 097 098 099 100 101 102 103 
2026-09-17 08:10:46: [    0.212894] pcpu-alloc: [0] 104 105 106 107 108 109 110 111 
2026-09-17 08:10:46: [    0.214246] pcpu-alloc: [0] 112 113 114 115 116 117 118 119 
2026-09-17 08:10:46: [    0.215582] pcpu-alloc: [0] 120 121 122 123 124 125 126 127 
2026-09-17 08:10:46: [    0.216924] pcpu-alloc: [0] 128 129 130 131 132 133 134 135 
2026-09-17 08:10:46: [    0.218274] pcpu-alloc: [0] 136 137 138 139 140 141 142 143 
2026-09-17 08:10:46: [    0.219608] pcpu-alloc: [0] 144 145 146 147 148 149 150 151 
2026-09-17 08:10:46: [    0.220940] pcpu-alloc: [0] 152 153 154 155 156 157 158 159 
2026-09-17 08:10:46: [    0.222309] pcpu-alloc: [0] 160 161 162 163 164 165 166 167 
2026-09-17 08:10:46: [    0.223651] pcpu-alloc: [0] 168 169 170 171 172 173 174 175 
2026-09-17 08:10:46: [    0.225012] pcpu-alloc: [0] 176 177 178 179 180 181 182 183 
2026-09-17 08:10:46: [    0.226373] pcpu-alloc: [0] 184 185 186 187 188 189 190 191 
2026-09-17 08:10:46: [    0.227724] pcpu-alloc: [0] 192 193 194 195 196 197 198 199 
2026-09-17 08:10:46: [    0.229060] pcpu-alloc: [0] 200 201 202 203 204 205 206 207 
2026-09-17 08:10:46: [    0.232337] pcpu-alloc: [0] 208 209 210 211 212 213 214 215 
2026-09-17 08:10:46: [    0.234729] pcpu-alloc: [0] 216 217 218 219 220 221 222 223 
2026-09-17 08:10:46: [    0.236091] pcpu-alloc: [0] 224 225 226 227 228 229 230 231 
2026-09-17 08:10:46: [    0.237444] pcpu-alloc: [0] 232 233 234 235 236 237 238 239 
2026-09-17 08:10:46: [    0.238798] pcpu-alloc: [0] 240 241 242 243 244 245 246 247 
2026-09-17 08:10:46: [    0.240136] pcpu-alloc: [0] 248 249 250 251 252 253 254 --- 
2026-09-17 08:10:46: [    0.241529] Kernel command line: root=/dev/sda1 rw console=ttyS0,115200n8 earlyprintk=ttyS0,115200 net.ifnames=0 biosdevname=0 movable_node swiotlb=65536 initcall_debug loglevel=8 ignore_loglevel dyndbg="file kernel/time/clocksource.c +p; file kernel/time/timekeeping.c +p; file arch/x86/kernel/tsc.c +p"
2026-09-17 08:10:46: [    0.247994] Unknown kernel command line parameters "biosdevname=0", will be passed to user space.
2026-09-17 08:10:46: [    0.250120] printk: log_buf_len individual max cpu contribution: 4096 bytes
2026-09-17 08:10:46: [    0.251783] printk: log_buf_len total cpu_extra contributions: 1040384 bytes
2026-09-17 08:10:46: [    0.253482] printk: log_buf_len min size: 262144 bytes
2026-09-17 08:10:46: [    0.255119] printk: log buffer data + meta data: 2097152 + 7340032 = 9437184 bytes
2026-09-17 08:10:46: [    0.256926] printk: early log buf free: 249872(95%)
2026-09-17 08:10:46: [    0.260244] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
2026-09-17 08:10:46: [    0.263329] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
2026-09-17 08:10:46: [    0.268262] software IO TLB: area num 256.
2026-09-17 08:10:46: [    0.278241] Fallback order for Node 0: 0 
2026-09-17 08:10:46: [    0.278267] Built 1 zonelists, mobility grouping on.  Total pages: 11530637
2026-09-17 08:10:46: [    0.280934] Policy zone: Normal
2026-09-17 08:10:46: [    0.281723] mem auto-init: stack:off, heap alloc:off, heap free:off
2026-09-17 08:10:46: [    0.318824] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=255, Nodes=1
2026-09-17 08:10:46: Poking KASLR using RDRAND RDTSC...
2026-09-17 08:10:46: [    0.321554] Dynamic Preempt: full
2026-09-17 08:10:46: [    0.322915] rcu: Preemptible hierarchical RCU implementation.
2026-09-17 08:10:46: [    0.324317] rcu: 	RCU event tracing is enabled.
2026-09-17 08:10:46: [    0.325426] rcu: 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=255.
2026-09-17 08:10:46: [    0.327103] 	Trampoline variant of Tasks RCU enabled.
2026-09-17 08:10:46: [    0.328325] 	Tracing variant of Tasks RCU enabled.
2026-09-17 08:10:46: [    0.329512] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
2026-09-17 08:10:46: [    0.331373] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=255
2026-09-17 08:10:46: [    0.333137] RCU Tasks: Setting shift to 8 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=255.
2026-09-17 08:10:46: [    0.338379] NR_IRQS: 33024, nr_irqs: 2464, preallocated irqs: 16
2026-09-17 08:10:46: [    0.340153] rcu: srcu_init: Setting srcu_struct sizes to big.
2026-09-17 08:10:46: [    0.341871] entering initcall level: console
2026-09-17 08:10:46: [    0.342885] calling  con_init+0x0/0x220 @ 0
2026-09-17 08:10:46: [    0.343923] Console: colour dummy device 80x25
2026-09-17 08:10:46: [    0.345000] initcall con_init+0x0/0x220 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.346463] calling  hvc_console_init+0x0/0x20 @ 0
2026-09-17 08:10:46: [    0.347637] initcall hvc_console_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.349262] calling  univ8250_console_init+0x0/0x30 @ 0
2026-09-17 08:10:46: [    0.350593] printk: legacy console [ttyS0] enabled
2026-09-17 08:10:46: [    0.350593] printk: legacy console [ttyS0] enabled
2026-09-17 08:10:46: [    0.352951] printk: legacy bootconsole [earlyser0] disabled
2026-09-17 08:10:46: [    0.352951] printk: legacy bootconsole [earlyser0] disabled
2026-09-17 08:10:46: [    0.355709] initcall univ8250_console_init+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.357554] ACPI: Core revision 20251212
2026-09-17 08:10:46: [    0.358785] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
2026-09-17 08:10:46: [    0.361314] APIC: Switch to symmetric I/O mode setup
2026-09-17 08:10:46: [    0.362965] x2apic enabled
2026-09-17 08:10:46: [    0.363992] APIC: Switched APIC routing to: physical x2apic
2026-09-17 08:10:46: [    0.366630] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
2026-09-17 08:10:46: [    0.368423] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x228aeee0e7a, max_idle_ns: 440795314296 ns
2026-09-17 08:10:46: [    0.371125] Calibrating delay loop (skipped) preset value.. 4792.80 BogoMIPS (lpj=2396400)
2026-09-17 08:10:46: [    0.373194] AMD Zen1 DIV0 bug detected. Disable SMT for full protection.
2026-09-17 08:10:46: [    0.374230] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
2026-09-17 08:10:46: [    0.375125] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
2026-09-17 08:10:46: [    0.376128] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
2026-09-17 08:10:46: [    0.377126] Speculative Store Bypass: Vulnerable
2026-09-17 08:10:46: [    0.378126] Spectre V2 : Mitigation: Retpolines
2026-09-17 08:10:46: [    0.379122] RETBleed: Mitigation: untrained return thunk
2026-09-17 08:10:46: [    0.379125] Speculative Return Stack Overflow: Mitigation: SMT disabled
2026-09-17 08:10:46: [    0.380125] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
2026-09-17 08:10:46: [    0.381125] Spectre V2 : Spectre v2 / SpectreRSB: Filling RSB on context switch and VMEXIT
2026-09-17 08:10:46: [    0.382125] active return thunk: retbleed_return_thunk
2026-09-17 08:10:46: [    0.383144] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
2026-09-17 08:10:46: [    0.384125] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
2026-09-17 08:10:46: [    0.385125] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
2026-09-17 08:10:46: [    0.386125] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
2026-09-17 08:10:46: [    0.387125] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format.
2026-09-17 08:10:46: [    0.408878] Freeing SMP alternatives memory: 60K
2026-09-17 08:10:46: [    0.409127] pid_max: default: 261120 minimum: 2040
2026-09-17 08:10:46: [    0.411188] SELinux:  Initializing.
2026-09-17 08:10:46: [    0.412471] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
2026-09-17 08:10:46: [    0.413162] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
2026-09-17 08:10:46: [    0.414271] VFS: Finished mounting rootfs on nullfs
2026-09-17 08:10:46: [    0.515919] smpboot: CPU0: AMD EPYC Processor (family: 0x17, model: 0x1, stepping: 0x2)
2026-09-17 08:10:46: [    0.516545] entering initcall level: early
2026-09-17 08:10:46: [    0.517126] calling  init_hw_perf_events+0x0/0x6b0 @ 1
2026-09-17 08:10:46: [    0.518125] Performance Events: PMU not available due to virtualization, using software events only.
2026-09-17 08:10:46: [    0.519126] initcall init_hw_perf_events+0x0/0x6b0 returned 0 after 1000 usecs
2026-09-17 08:10:46: [    0.520126] calling  bts_init+0x0/0x100 @ 1
2026-09-17 08:10:46: [    0.521125] initcall bts_init+0x0/0x100 returned -19 after 0 usecs
2026-09-17 08:10:46: [    0.522126] calling  do_init_real_mode+0x0/0x20 @ 1
2026-09-17 08:10:46: [    0.523305] initcall do_init_real_mode+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.524126] calling  init_sigframe_size+0x0/0x40 @ 1
2026-09-17 08:10:46: [    0.525126] signal: max sigframe size: 1776
2026-09-17 08:10:46: [    0.526129] initcall init_sigframe_size+0x0/0x40 returned 0 after 1000 usecs
2026-09-17 08:10:46: [    0.527126] calling  trace_init_perf_perm_irq_work_exit+0x0/0x20 @ 1
2026-09-17 08:10:46: [    0.528126] initcall trace_init_perf_perm_irq_work_exit+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.529126] calling  cache_ap_register+0x0/0x60 @ 1
2026-09-17 08:10:46: [    0.530126] initcall cache_ap_register+0x0/0x60 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.531126] calling  bp_init_aperfmperf+0x0/0x270 @ 1
2026-09-17 08:10:46: [    0.532126] initcall bp_init_aperfmperf+0x0/0x270 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.533125] calling  save_builtin_microcode+0x0/0xb0 @ 1
2026-09-17 08:10:46: [    0.534125] initcall save_builtin_microcode+0x0/0xb0 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.535126] calling  save_microcode_in_initrd+0x0/0x170 @ 1
2026-09-17 08:10:46: [    0.536126] initcall save_microcode_in_initrd+0x0/0x170 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.537125] calling  register_nmi_cpu_backtrace_handler+0x0/0x20 @ 1
2026-09-17 08:10:46: [    0.538126] initcall register_nmi_cpu_backtrace_handler+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.539127] calling  kvm_setup_vsyscall_timeinfo+0x0/0x150 @ 1
2026-09-17 08:10:46: [    0.540130] initcall kvm_setup_vsyscall_timeinfo+0x0/0x150 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.541126] calling  spawn_ksoftirqd+0x0/0x50 @ 1
2026-09-17 08:10:46: [    0.542181] initcall spawn_ksoftirqd+0x0/0x50 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.543125] calling  init_signal_sysctls+0x0/0x50 @ 1
2026-09-17 08:10:46: [    0.544131] initcall init_signal_sysctls+0x0/0x50 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.545126] calling  init_umh_sysctls+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.546133] initcall init_umh_sysctls+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.547125] calling  kthreads_init+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.548133] initcall kthreads_init+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.549125] calling  migration_init+0x0/0x40 @ 1
2026-09-17 08:10:46: [    0.550126] initcall migration_init+0x0/0x40 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.551126] calling  printk_set_kthreads_ready+0x0/0x40 @ 1
2026-09-17 08:10:46: [    0.552126] initcall printk_set_kthreads_ready+0x0/0x40 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.553127] calling  srcu_bootup_announce+0x0/0x80 @ 1
2026-09-17 08:10:46: [    0.554125] rcu: Hierarchical SRCU implementation.
2026-09-17 08:10:46: [    0.555125] rcu: 	Max phase no-delay instances is 400.
2026-09-17 08:10:46: [    0.556126] initcall srcu_bootup_announce+0x0/0x80 returned 0 after 2000 usecs
2026-09-17 08:10:46: [    0.557126] calling  rcu_spawn_gp_kthread+0x0/0x210 @ 1
2026-09-17 08:10:46: [    0.558197] initcall rcu_spawn_gp_kthread+0x0/0x210 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.559126] calling  check_cpu_stall_init+0x0/0x20 @ 1
2026-09-17 08:10:46: [    0.561126] initcall check_cpu_stall_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.562126] calling  rcu_sysrq_init+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.563126] initcall rcu_sysrq_init+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.564125] calling  trace_init_flags_sys_enter+0x0/0x10 @ 1
2026-09-17 08:10:46: [    0.565126] initcall trace_init_flags_sys_enter+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.566127] calling  trace_init_flags_sys_exit+0x0/0x10 @ 1
2026-09-17 08:10:46: [    0.567126] initcall trace_init_flags_sys_exit+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.568125] calling  tmigr_init+0x0/0x170 @ 1
2026-09-17 08:10:46: [    0.569125] Timer migration: 3 hierarchy levels; 8 children per group; 3 crossnode level
2026-09-17 08:10:46: [    0.570135] initcall tmigr_init+0x0/0x170 returned 0 after 1000 usecs
2026-09-17 08:10:46: [    0.571126] calling  cpu_stop_init+0x0/0x80 @ 1
2026-09-17 08:10:46: [    0.572174] initcall cpu_stop_init+0x0/0x80 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.573126] calling  init_kprobes+0x0/0x190 @ 1
2026-09-17 08:10:46: [    0.575279] initcall init_kprobes+0x0/0x190 returned 0 after 1000 usecs
2026-09-17 08:10:46: [    0.576126] calling  init_trace_printk+0x0/0x10 @ 1
2026-09-17 08:10:46: [    0.577126] initcall init_trace_printk+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.578126] calling  event_trace_enable_again+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.579127] initcall event_trace_enable_again+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.580127] calling  irq_work_init_threads+0x0/0x10 @ 1
2026-09-17 08:10:46: [    0.581126] initcall irq_work_init_threads+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.582126] calling  static_call_init+0x0/0x90 @ 1
2026-09-17 08:10:46: [    0.589153] initcall static_call_init+0x0/0x90 returned 0 after 6000 usecs
2026-09-17 08:10:46: [    0.590126] calling  jump_label_init_module+0x0/0x10 @ 1
2026-09-17 08:10:46: [    0.591126] initcall jump_label_init_module+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.592127] calling  init_zero_pfn+0x0/0x40 @ 1
2026-09-17 08:10:46: [    0.593125] initcall init_zero_pfn+0x0/0x40 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.594126] calling  init_fs_inode_sysctls+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.595129] initcall init_fs_inode_sysctls+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.596125] calling  init_fs_locks_sysctls+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.597127] initcall init_fs_locks_sysctls+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.598125] calling  init_fs_sysctls+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.599128] initcall init_fs_sysctls+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.600126] calling  init_security_keys_sysctls+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.601129] initcall init_security_keys_sysctls+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.602127] calling  security_initcall_early+0x0/0x90 @ 1
2026-09-17 08:10:46: [    0.603125] initcall security_initcall_early+0x0/0x90 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.604125] calling  dynamic_debug_init+0x0/0x240 @ 1
2026-09-17 08:10:46: [    0.605559] initcall dynamic_debug_init+0x0/0x240 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.606126] calling  efi_memreserve_root_init+0x0/0x30 @ 1
2026-09-17 08:10:46: [    0.607125] initcall efi_memreserve_root_init+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.608125] calling  efi_earlycon_remap_fb+0x0/0x60 @ 1
2026-09-17 08:10:46: [    0.609125] initcall efi_earlycon_remap_fb+0x0/0x60 returned 0 after 0 usecs
2026-09-17 08:10:46: [    0.612327] smp: Bringing up secondary CPUs ...
2026-09-17 08:10:46: [    0.613274] smpboot: x86: Booting SMP configuration:
2026-09-17 08:10:46: [    0.614136] .... node  #0, CPUs:          #1   #2   #3   #4   #5   #6   #7   #8   #9  #10  #11  #12  #13  #14  #15  #16  #17  #18  #19  #20  #21  #22  #23  #24  #25  #26  #27  #28  #29  #30  #31  #32  #33  #34  #35  #36  #37  #38  #39  #40  #41  #42  #43  #44  #45  #46  #47  #48  #49  #50  #51  #52  #53  #54  #55  #56  #57  #58  #59  #60  #61  #62  #63  #64  #65  #66  #67  #68  #69  #70  #71  #72  #73  #74  #75  #76  #77  #78  #79  #80  #81  #82  #83  #84  #85  #86  #87  #88  #89  #90  #91  #92  #93  #94  #95  #96  #97  #98  #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 #120 #121 #122 #123 #124 #125 #126 #127 #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141 #142 #143 #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156 #157 #158 #159 #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171 #172 #173 #174 #175 #176 #177 #178 #179 #180 #181 #182 #183 #184 #185 #186 #187 #188 #189 #190 #191 #192 #193 #194 #195 #196 #197 #198
2026-09-17 08:10:48: [    0.757376]  #199 #200 #201 #202 #203 #204 #205 #206 #207 #208 #209 #210 #211 #212 #213 #214 #215 #216 #217 #218 #219 #220 #221 #222 #223 #224 #225 #226 #227 #228 #229 #230 #231 #232 #233 #234 #235 #236 #237 #238 #239 #240 #241 #242 #243 #244 #245 #246 #247 #248 #249 #250 #251 #252 #253 #254
2026-09-17 08:10:48: [    1.735451] smp: Brought up 1 node, 255 CPUs
2026-09-17 08:10:48: [    1.737128] smpboot: Total of 255 processors activated (1222164.00 BogoMIPS)
2026-09-17 08:10:48: [    1.752261] Memory: 44986640K/46122548K available (20879K kernel code, 3364K rwdata, 8700K rodata, 3008K init, 3568K bss, 1071956K reserved, 0K cma-reserved)
2026-09-17 08:10:48: [    1.754302] devtmpfs: initialized
2026-09-17 08:10:48: [    1.755281] x86/mm: Memory block size: 128MB
2026-09-17 08:10:48: [    1.764050] entering initcall level: pure
2026-09-17 08:10:48: [    1.764134] calling  setup_split_lock_delayed_work+0x0/0x80 @ 1
2026-09-17 08:10:48: [    1.765163] initcall setup_split_lock_delayed_work+0x0/0x80 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.766126] calling  ipc_ns_init+0x0/0x70 @ 1
2026-09-17 08:10:48: [    1.767131] initcall ipc_ns_init+0x0/0x70 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.768125] calling  mmap_min_addr_init+0x0/0x40 @ 1
2026-09-17 08:10:48: [    1.769130] initcall mmap_min_addr_init+0x0/0x40 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.770125] calling  security_initcall_pure+0x0/0x90 @ 1
2026-09-17 08:10:48: [    1.771126] initcall security_initcall_pure+0x0/0x90 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.772125] calling  pci_realloc_setup_params+0x0/0x70 @ 1
2026-09-17 08:10:48: [    1.773126] initcall pci_realloc_setup_params+0x0/0x70 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.774125] calling  inet_frag_wq_init+0x0/0x40 @ 1
2026-09-17 08:10:48: [    1.775241] initcall inet_frag_wq_init+0x0/0x40 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.776161] entering initcall level: core
2026-09-17 08:10:48: [    1.777129] calling  e820__register_nvs_regions+0x0/0x40 @ 1
2026-09-17 08:10:48: [    1.778127] ACPI: PM: Registering ACPI NVS region [mem 0x00800000-0x00807fff] (32768 bytes)
2026-09-17 08:10:48: [    1.779126] ACPI: PM: Registering ACPI NVS region [mem 0x0080b000-0x0080bfff] (4096 bytes)
2026-09-17 08:10:48: [    1.780125] ACPI: PM: Registering ACPI NVS region [mem 0x00814000-0x008fffff] (966656 bytes)
2026-09-17 08:10:48: [    1.781141] ACPI: PM: Registering ACPI NVS region [mem 0x7e97f000-0x7e9fefff] (524288 bytes)
2026-09-17 08:10:48: [    1.782131] ACPI: PM: Registering ACPI NVS region [mem 0x7f788000-0x7fffffff] (8880128 bytes)
2026-09-17 08:10:48: [    1.783202] initcall e820__register_nvs_regions+0x0/0x40 returned 0 after 5000 usecs
2026-09-17 08:10:48: [    1.784129] calling  cpufreq_register_tsc_scaling+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.785127] initcall cpufreq_register_tsc_scaling+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.786126] calling  reboot_init+0x0/0x40 @ 1
2026-09-17 08:10:48: [    1.787129] initcall reboot_init+0x0/0x40 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.788125] calling  init_lapic_sysfs+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.789126] initcall init_lapic_sysfs+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.790127] calling  alloc_frozen_cpus+0x0/0x10 @ 1
2026-09-17 08:10:48: [    1.791125] initcall alloc_frozen_cpus+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.792125] calling  cpu_hotplug_pm_sync_init+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.793127] initcall cpu_hotplug_pm_sync_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.794126] calling  wq_sysfs_init+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.795166] initcall wq_sysfs_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.796126] calling  ksysfs_init+0x0/0xa0 @ 1
2026-09-17 08:10:48: [    1.797140] initcall ksysfs_init+0x0/0xa0 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.798126] calling  schedutil_gov_init+0x0/0x10 @ 1
2026-09-17 08:10:48: [    1.799126] initcall schedutil_gov_init+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.800132] calling  pm_init+0x0/0xb0 @ 1
2026-09-17 08:10:48: [    1.801247] initcall pm_init+0x0/0xb0 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.802126] calling  pm_disk_init+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.803131] initcall pm_disk_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.804125] calling  swsusp_header_init+0x0/0x50 @ 1
2026-09-17 08:10:48: [    1.805128] initcall swsusp_header_init+0x0/0x50 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.806126] calling  rcu_set_runtime_mode+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.807131] initcall rcu_set_runtime_mode+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.808125] calling  rcu_init_tasks_generic+0x0/0xd0 @ 1
2026-09-17 08:10:48: [    1.809180] initcall rcu_init_tasks_generic+0x0/0xd0 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.810127] calling  init_jiffies_clocksource+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.811126] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
2026-09-17 08:10:48: [    1.812127] initcall init_jiffies_clocksource+0x0/0x20 returned 0 after 1000 usecs
2026-09-17 08:10:48: [    1.813126] calling  posixtimer_init+0x0/0xf0 @ 1
2026-09-17 08:10:48: [    1.814133] posixtimers hash table entries: 131072 (order: 9, 2097152 bytes, linear)
2026-09-17 08:10:48: [    1.815264] initcall posixtimer_init+0x0/0xf0 returned 0 after 1000 usecs
2026-09-17 08:10:48: [    1.816127] calling  futex_init+0x0/0x190 @ 1
2026-09-17 08:10:48: [    1.818178] futex hash table entries: 65536 (4194304 bytes on 1 NUMA nodes, total 4096 KiB, linear).
2026-09-17 08:10:48: [    1.819125] initcall futex_init+0x0/0x190 returned 0 after 2000 usecs
2026-09-17 08:10:48: [    1.820125] calling  cgroup_wq_init+0x0/0x80 @ 1
2026-09-17 08:10:48: [    1.822154] initcall cgroup_wq_init+0x0/0x80 returned 0 after 1000 usecs
2026-09-17 08:10:48: [    1.823125] calling  cgroup1_wq_init+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.824175] initcall cgroup1_wq_init+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.825125] calling  trace_events_eprobe_init_early+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.826125] initcall trace_events_eprobe_init_early+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.827125] calling  init_kprobe_trace_early+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.828127] initcall init_kprobe_trace_early+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.829125] calling  init_events_core_sysctls+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.830132] initcall init_events_core_sysctls+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.831126] calling  init_callchain_sysctls+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.832128] initcall init_callchain_sysctls+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.833127] calling  fsnotify_init+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.834187] initcall fsnotify_init+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.835127] calling  filelock_init+0x0/0x120 @ 1
2026-09-17 08:10:48: [    1.836200] initcall filelock_init+0x0/0x120 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.837125] calling  init_misc_binfmt+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.838128] initcall init_misc_binfmt+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.839125] calling  init_script_binfmt+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.840126] initcall init_script_binfmt+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.841125] calling  init_elf_binfmt+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.842125] initcall init_elf_binfmt+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.843125] calling  init_compat_elf_binfmt+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.845125] initcall init_compat_elf_binfmt+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.846125] calling  configfs_init+0x0/0xd0 @ 1
2026-09-17 08:10:48: [    1.847172] initcall configfs_init+0x0/0xd0 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.848125] calling  debugfs_init+0x0/0x100 @ 1
2026-09-17 08:10:48: [    1.850164] initcall debugfs_init+0x0/0x100 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.851127] calling  tracefs_init+0x0/0xb0 @ 1
2026-09-17 08:10:48: [    1.852156] initcall tracefs_init+0x0/0xb0 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.853125] calling  security_initcall_core+0x0/0x90 @ 1
2026-09-17 08:10:48: [    1.854126] initcall security_initcall_core+0x0/0x90 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.855125] calling  virtio_init+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.856143] initcall virtio_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.857125] calling  iommu_init+0x0/0x30 @ 1
2026-09-17 08:10:48: [    1.858129] initcall iommu_init+0x0/0x30 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.859126] calling  component_debug_init+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.860186] initcall component_debug_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.861131] calling  early_resume_init+0x0/0xd0 @ 1
2026-09-17 08:10:48: [    1.862243] PM: RTC time: 08:10:36, date: 2026-09-17
2026-09-17 08:10:48: [    1.863126] initcall early_resume_init+0x0/0xd0 returned 0 after 1000 usecs
2026-09-17 08:10:48: [    1.864125] calling  cpufreq_core_init+0x0/0x90 @ 1
2026-09-17 08:10:48: [    1.865135] initcall cpufreq_core_init+0x0/0x90 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.866125] calling  cpufreq_gov_performance_init+0x0/0x10 @ 1
2026-09-17 08:10:48: [    1.867126] initcall cpufreq_gov_performance_init+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.868125] calling  cpufreq_gov_userspace_init+0x0/0x10 @ 1
2026-09-17 08:10:48: [    1.869126] initcall cpufreq_gov_userspace_init+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.870126] calling  CPU_FREQ_GOV_ONDEMAND_init+0x0/0x10 @ 1
2026-09-17 08:10:48: [    1.871125] initcall CPU_FREQ_GOV_ONDEMAND_init+0x0/0x10 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.872125] calling  cpuidle_init+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.873133] initcall cpuidle_init+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.874126] calling  unaccepted_memory_init_kdump+0x0/0x20 @ 1
2026-09-17 08:10:48: [    1.875125] initcall unaccepted_memory_init_kdump+0x0/0x20 returned 0 after 0 usecs
2026-09-17 08:10:48: [    1.876125] calling  sock_init+0x0/0xe0 @ 1

CONFIG_CLOCKSOURCE_WATCHDOG=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y CONFIG_GENERIC_CLOCKEVENTS_BROADCAST_IDLE=y CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y CONFIG_GENERIC_CLOCKEVENTS_COUPLED=y CONFIG_GENERIC_CLOCKEVENTS_COUPLED_INLINE=y CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF=y
# CONFIG_BPF_SYSCALL is not set
# CONFIG_BPF_JIT is not set CONFIG_PREEMPT_BUILD=y
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_LAZY is not set
# CONFIG_PREEMPT_RT is not set CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y CONFIG_PREEMPT_DYNAMIC=y CONFIG_PREEMPT_RCU=y
# CONFIG_PRINTK_INDEX is not set CONFIG_NET_NS=y
CONFIG_PRINTK=y
CONFIG_SMP=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set CONFIG_KVM_GUEST=y
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set CONFIG_PARAVIRT_CLOCK=y
CONFIG_X86_TSC=y
# CONFIG_MAXSMP is not set CONFIG_NR_CPUS_RANGE_BEGIN=2 CONFIG_NR_CPUS_RANGE_END=512 CONFIG_NR_CPUS_DEFAULT=64 CONFIG_NR_CPUS=512
CONFIG_X86_LOCAL_APIC=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_RANDOMIZE_BASE=y CONFIG_RANDOMIZE_MEMORY=y CONFIG_RANDOMIZE_MEMORY_PHYSICAL_PADDING=0xa CONFIG_KVM_GUEST_MEMFD=y CONFIG_JUMP_LABEL=y
# CONFIG_STATIC_CALL_SELFTEST is not set CONFIG_RANDOMIZE_KSTACK_OFFSET=y
# CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT is not set CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_NET=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
CONFIG_NET_XGRESS=y
CONFIG_NET_CRC32C=y
# CONFIG_NET_KEY is not set CONFIG_NET_HANDSHAKE=y
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set CONFIG_NET_IP_TUNNEL=y
# CONFIG_NET_IPVTI is not set
# CONFIG_NET_FOU is not set CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y CONFIG_NET_PTP_CLASSIFY=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set CONFIG_NETFILTER=y CONFIG_NETFILTER_ADVANCED=y CONFIG_NETFILTER_INGRESS=y CONFIG_NETFILTER_EGRESS=y CONFIG_NETFILTER_SKIP_EGRESS=y CONFIG_NETFILTER_NETLINK=y CONFIG_NETFILTER_FAMILY_BRIDGE=y CONFIG_NETFILTER_FAMILY_ARP=y CONFIG_NETFILTER_NETLINK_HOOK=y CONFIG_NETFILTER_NETLINK_ACCT=y CONFIG_NETFILTER_NETLINK_QUEUE=y CONFIG_NETFILTER_NETLINK_LOG=y CONFIG_NETFILTER_NETLINK_OSF=y CONFIG_NETFILTER_CONNCOUNT=y CONFIG_NETFILTER_NETLINK_GLUE_CT=y CONFIG_NETFILTER_SYNPROXY=y CONFIG_NETFILTER_XTABLES=y CONFIG_NETFILTER_XTABLES_COMPAT=y
# CONFIG_NETFILTER_XTABLES_LEGACY is not set CONFIG_NETFILTER_XT_MARK=y CONFIG_NETFILTER_XT_CONNMARK=y CONFIG_NETFILTER_XT_TARGET_AUDIT=y CONFIG_NETFILTER_XT_TARGET_CHECKSUM=y CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y CONFIG_NETFILTER_XT_TARGET_CONNMARK=y CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
# CONFIG_NETFILTER_XT_TARGET_CT is not set CONFIG_NETFILTER_XT_TARGET_DSCP=y CONFIG_NETFILTER_XT_TARGET_HL=y CONFIG_NETFILTER_XT_TARGET_HMARK=y CONFIG_NETFILTER_XT_TARGET_IDLETIMER=y CONFIG_NETFILTER_XT_TARGET_LED=y CONFIG_NETFILTER_XT_TARGET_LOG=y CONFIG_NETFILTER_XT_TARGET_MARK=y CONFIG_NETFILTER_XT_NAT=y CONFIG_NETFILTER_XT_TARGET_NETMAP=y CONFIG_NETFILTER_XT_TARGET_NFLOG=y CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y CONFIG_NETFILTER_XT_TARGET_RATEEST=y CONFIG_NETFILTER_XT_TARGET_REDIRECT=y CONFIG_NETFILTER_XT_TARGET_MASQUERADE=y CONFIG_NETFILTER_XT_TARGET_TEE=y CONFIG_NETFILTER_XT_TARGET_TPROXY=y CONFIG_NETFILTER_XT_TARGET_SECMARK=y CONFIG_NETFILTER_XT_TARGET_TCPMSS=y CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=y CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y CONFIG_NETFILTER_XT_MATCH_BPF=y CONFIG_NETFILTER_XT_MATCH_CGROUP=y CONFIG_NETFILTER_XT_MATCH_CLUSTER=y CONFIG_NETFILTER_XT_MATCH_COMMENT=y CONFIG_NETFILTER_XT_MATCH_CONNBYTES=y CONFIG_NETFILTER_XT_MATCH_CONNLABEL=y CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=y CONFIG_NETFILTER_XT_MATCH_CONNMARK=y CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y CONFIG_NETFILTER_XT_MATCH_CPU=y CONFIG_NETFILTER_XT_MATCH_DCCP=y CONFIG_NETFILTER_XT_MATCH_DEVGROUP=y CONFIG_NETFILTER_XT_MATCH_DSCP=y CONFIG_NETFILTER_XT_MATCH_ECN=y CONFIG_NETFILTER_XT_MATCH_ESP=y CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y CONFIG_NETFILTER_XT_MATCH_HELPER=y CONFIG_NETFILTER_XT_MATCH_HL=y CONFIG_NETFILTER_XT_MATCH_IPCOMP=y CONFIG_NETFILTER_XT_MATCH_IPRANGE=y CONFIG_NETFILTER_XT_MATCH_L2TP=y CONFIG_NETFILTER_XT_MATCH_LENGTH=y CONFIG_NETFILTER_XT_MATCH_LIMIT=y CONFIG_NETFILTER_XT_MATCH_MAC=y CONFIG_NETFILTER_XT_MATCH_MARK=y CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y CONFIG_NETFILTER_XT_MATCH_NFACCT=y CONFIG_NETFILTER_XT_MATCH_OSF=y CONFIG_NETFILTER_XT_MATCH_OWNER=y CONFIG_NETFILTER_XT_MATCH_POLICY=y CONFIG_NETFILTER_XT_MATCH_PHYSDEV=y CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y CONFIG_NETFILTER_XT_MATCH_QUOTA=y CONFIG_NETFILTER_XT_MATCH_RATEEST=y CONFIG_NETFILTER_XT_MATCH_REALM=y CONFIG_NETFILTER_XT_MATCH_RECENT=y CONFIG_NETFILTER_XT_MATCH_SCTP=y CONFIG_NETFILTER_XT_MATCH_SOCKET=y CONFIG_NETFILTER_XT_MATCH_STATE=y CONFIG_NETFILTER_XT_MATCH_STATISTIC=y CONFIG_NETFILTER_XT_MATCH_STRING=y CONFIG_NETFILTER_XT_MATCH_TCPMSS=y CONFIG_NETFILTER_XT_MATCH_TIME=y CONFIG_NETFILTER_XT_MATCH_U32=y
# CONFIG_NET_DSA is not set CONFIG_NET_SCHED=y
# CONFIG_NET_SCH_HTB is not set
# CONFIG_NET_SCH_HFSC is not set
# CONFIG_NET_SCH_PRIO is not set
# CONFIG_NET_SCH_MULTIQ is not set
# CONFIG_NET_SCH_RED is not set
# CONFIG_NET_SCH_SFB is not set
# CONFIG_NET_SCH_SFQ is not set
# CONFIG_NET_SCH_TEQL is not set
# CONFIG_NET_SCH_TBF is not set
# CONFIG_NET_SCH_CBS is not set
# CONFIG_NET_SCH_ETF is not set
# CONFIG_NET_SCH_TAPRIO is not set
# CONFIG_NET_SCH_GRED is not set
# CONFIG_NET_SCH_NETEM is not set
# CONFIG_NET_SCH_DRR is not set
# CONFIG_NET_SCH_MQPRIO is not set
# CONFIG_NET_SCH_SKBPRIO is not set
# CONFIG_NET_SCH_CHOKE is not set
# CONFIG_NET_SCH_QFQ is not set
# CONFIG_NET_SCH_CODEL is not set
# CONFIG_NET_SCH_FQ_CODEL is not set
# CONFIG_NET_SCH_CAKE is not set
# CONFIG_NET_SCH_FQ is not set
# CONFIG_NET_SCH_HHF is not set
# CONFIG_NET_SCH_PIE is not set
# CONFIG_NET_SCH_INGRESS is not set
# CONFIG_NET_SCH_PLUG is not set
# CONFIG_NET_SCH_ETS is not set
# CONFIG_NET_SCH_DUALPI2 is not set
# CONFIG_NET_SCH_DEFAULT is not set CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
# CONFIG_NET_CLS_ROUTE4 is not set
# CONFIG_NET_CLS_FW is not set
# CONFIG_NET_CLS_U32 is not set
# CONFIG_NET_CLS_FLOW is not set CONFIG_NET_CLS_CGROUP=y
# CONFIG_NET_CLS_BPF is not set
# CONFIG_NET_CLS_FLOWER is not set
# CONFIG_NET_CLS_MATCHALL is not set CONFIG_NET_EMATCH=y CONFIG_NET_EMATCH_STACK=32
# CONFIG_NET_EMATCH_CMP is not set
# CONFIG_NET_EMATCH_NBYTE is not set
# CONFIG_NET_EMATCH_U32 is not set
# CONFIG_NET_EMATCH_META is not set
# CONFIG_NET_EMATCH_TEXT is not set
# CONFIG_NET_EMATCH_IPT is not set CONFIG_NET_CLS_ACT=y
# CONFIG_NET_ACT_POLICE is not set
# CONFIG_NET_ACT_GACT is not set
# CONFIG_NET_ACT_MIRRED is not set
# CONFIG_NET_ACT_SAMPLE is not set CONFIG_NET_ACT_NAT=y
# CONFIG_NET_ACT_PEDIT is not set
# CONFIG_NET_ACT_SIMP is not set
# CONFIG_NET_ACT_SKBEDIT is not set
# CONFIG_NET_ACT_CSUM is not set
# CONFIG_NET_ACT_MPLS is not set
# CONFIG_NET_ACT_VLAN is not set
# CONFIG_NET_ACT_BPF is not set
# CONFIG_NET_ACT_CONNMARK is not set
# CONFIG_NET_ACT_CTINFO is not set
# CONFIG_NET_ACT_SKBMOD is not set
# CONFIG_NET_ACT_IFE is not set
# CONFIG_NET_ACT_TUNNEL_KEY is not set
# CONFIG_NET_ACT_GATE is not set
# CONFIG_NET_TC_SKB_EXT is not set CONFIG_NET_SCH_FIFO=y
# CONFIG_NETLINK_DIAG is not set
# CONFIG_NET_NSH is not set
# CONFIG_NET_SWITCHDEV is not set
# CONFIG_NET_L3_MASTER_DEV is not set
# CONFIG_NET_NCSI is not set CONFIG_NET_RX_BUSY_POLL=y CONFIG_NET_FLOW_LIMIT=y
# CONFIG_NET_PKTGEN is not set
# CONFIG_NET_DROP_MONITOR is not set CONFIG_NET_9P=y
CONFIG_NET_9P_FD=y
CONFIG_NET_9P_VIRTIO=y
# CONFIG_NET_9P_DEBUG is not set
# CONFIG_NET_IFE is not set CONFIG_NET_SELFTESTS=y
CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
# CONFIG_NET_FC is not set
# CONFIG_NET_TEAM is not set CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_DYNAMIC is not set
# CONFIG_NETCONSOLE_EXTENDED_LOG is not set CONFIG_NETPOLL=y CONFIG_NET_POLL_CONTROLLER=y CONFIG_NET_VENDOR_3COM=y CONFIG_NET_VENDOR_ADAPTEC=y CONFIG_NET_VENDOR_AGERE=y CONFIG_NET_VENDOR_ALACRITECH=y CONFIG_NET_VENDOR_ALTEON=y CONFIG_NET_VENDOR_AMAZON=y CONFIG_NET_VENDOR_AMD=y CONFIG_NET_VENDOR_AQUANTIA=y CONFIG_NET_VENDOR_ARC=y CONFIG_NET_VENDOR_ASIX=y CONFIG_NET_VENDOR_ATHEROS=y CONFIG_NET_VENDOR_BROADCOM=y CONFIG_NET_VENDOR_CADENCE=y CONFIG_NET_VENDOR_CAVIUM=y CONFIG_NET_VENDOR_CHELSIO=y CONFIG_NET_VENDOR_CISCO=y CONFIG_NET_VENDOR_CORTINA=y CONFIG_NET_VENDOR_DAVICOM=y CONFIG_NET_VENDOR_DEC=y
CONFIG_NET_TULIP=y CONFIG_NET_VENDOR_DLINK=y CONFIG_NET_VENDOR_EMULEX=y CONFIG_NET_VENDOR_ENGLEDER=y CONFIG_NET_VENDOR_EZCHIP=y CONFIG_NET_VENDOR_FUJITSU=y CONFIG_NET_VENDOR_FUNGIBLE=y CONFIG_NET_VENDOR_GOOGLE=y CONFIG_NET_VENDOR_HISILICON=y CONFIG_NET_VENDOR_HUAWEI=y CONFIG_NET_VENDOR_I825XX=y CONFIG_NET_VENDOR_INTEL=y CONFIG_NET_VENDOR_LITEX=y CONFIG_NET_VENDOR_MARVELL=y CONFIG_NET_VENDOR_MELLANOX=y CONFIG_NET_VENDOR_META=y CONFIG_NET_VENDOR_MICREL=y CONFIG_NET_VENDOR_MICROCHIP=y CONFIG_NET_VENDOR_MICROSEMI=y CONFIG_NET_VENDOR_MICROSOFT=y CONFIG_NET_VENDOR_MUCSE=y CONFIG_NET_VENDOR_MYRI=y CONFIG_NET_VENDOR_NI=y CONFIG_NET_VENDOR_NATSEMI=y CONFIG_NET_VENDOR_NETRONOME=y CONFIG_NET_VENDOR_8390=y CONFIG_NET_VENDOR_NVIDIA=y CONFIG_NET_VENDOR_OKI=y CONFIG_NET_VENDOR_PACKET_ENGINES=y CONFIG_NET_VENDOR_PENSANDO=y CONFIG_NET_VENDOR_QLOGIC=y
# CONFIG_NETXEN_NIC is not set CONFIG_NET_VENDOR_BROCADE=y CONFIG_NET_VENDOR_QUALCOMM=y CONFIG_NET_VENDOR_RDC=y CONFIG_NET_VENDOR_REALTEK=y CONFIG_NET_VENDOR_RENESAS=y CONFIG_NET_VENDOR_ROCKER=y CONFIG_NET_VENDOR_SAMSUNG=y CONFIG_NET_VENDOR_SEEQ=y CONFIG_NET_VENDOR_SILAN=y CONFIG_NET_VENDOR_SIS=y CONFIG_NET_VENDOR_SOLARFLARE=y CONFIG_NET_VENDOR_SMSC=y CONFIG_NET_VENDOR_SOCIONEXT=y CONFIG_NET_VENDOR_STMICRO=y CONFIG_NET_VENDOR_SUN=y CONFIG_NET_VENDOR_SYNOPSYS=y CONFIG_NET_VENDOR_TEHUTI=y CONFIG_NET_VENDOR_TI=y CONFIG_NET_VENDOR_VERTEXCOM=y CONFIG_NET_VENDOR_VIA=y CONFIG_NET_VENDOR_WANGXUN=y CONFIG_NET_VENDOR_WIZNET=y CONFIG_NET_VENDOR_XILINX=y CONFIG_NET_VENDOR_XIRCOM=y
# CONFIG_NETDEVSIM is not set CONFIG_NET_FAILOVER=y CONFIG_NETFS_SUPPORT=y
# CONFIG_NETFS_STATS is not set
# CONFIG_NETFS_DEBUG is not set CONFIG_NETWORK_FILESYSTEMS=y CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_SHA3=y
CONFIG_PRINTK_TIME=y
# CONFIG_PRINTK_CALLER is not set CONFIG_DYNAMIC_DEBUG=y CONFIG_DYNAMIC_DEBUG_CORE=y CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_NONE is not set CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y # CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
# CONFIG_DEBUG_INFO_REDUCED is not set CONFIG_DEBUG_INFO_COMPRESSED_NONE=y
# CONFIG_DEBUG_INFO_COMPRESSED_ZLIB is not set # CONFIG_DEBUG_INFO_COMPRESSED_ZSTD is not set # CONFIG_DEBUG_INFO_SPLIT is not set
# CONFIG_NET_DEV_REFCNT_TRACKER is not set # CONFIG_NET_NS_REFCNT_TRACKER is not set
# CONFIG_PREEMPT_TRACER is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set

      KERNEL: /home/VT_BUILD/linux/vmlinux
    DUMPFILE: /home/VT_BUILD/debug_dumps/vm4-20260917-082127.kdump  [PARTIAL DUMP]
        CPUS: 255
        DATE: Thu Jan  1 00:00:00 UTC 1970
      UPTIME: 00:00:01
LOAD AVERAGE: 0.00, 0.00, 0.00
       TASKS: 1596
    NODENAME: (none)
     RELEASE: 7.0.0-rc1+
     VERSION: #4 SMP PREEMPT_DYNAMIC Thu Sep 17 07:03:37 UTC 2026
     MACHINE: x86_64  (2396 Mhz)
      MEMORY: 44 GB
       PANIC: ""
         PID: 0
     COMMAND: "swapper/0"
        TASK: ffffffff88a0d940  (1 of 255)  [THREAD_INFO: ffffffff88a0d940]
         CPU: 0
       STATE: TASK_RUNNING 
     WARNING: panic task not found

crash> sys

      KERNEL: /home/VT_BUILD/linux/vmlinux
    DUMPFILE: /home/VT_BUILD/debug_dumps/vm4-20260917-082127.kdump  [PARTIAL DUMP]
        CPUS: 255
        DATE: Thu Jan  1 00:00:00 UTC 1970
      UPTIME: 00:00:01
LOAD AVERAGE: 0.00, 0.00, 0.00
       TASKS: 1596
    NODENAME: (none)
     RELEASE: 7.0.0-rc1+
     VERSION: #4 SMP PREEMPT_DYNAMIC Thu Sep 17 07:03:37 UTC 2026
     MACHINE: x86_64  (2396 Mhz)
      MEMORY: 44 GB
       PANIC: ""

crash> log | tail -80

[    1.793127] initcall cpu_hotplug_pm_sync_init+0x0/0x20 returned 0 after 0 usecs
[    1.794126] calling  wq_sysfs_init+0x0/0x20 @ 1
[    1.795166] initcall wq_sysfs_init+0x0/0x20 returned 0 after 0 usecs
[    1.796126] calling  ksysfs_init+0x0/0xa0 @ 1
[    1.797140] initcall ksysfs_init+0x0/0xa0 returned 0 after 0 usecs
[    1.798126] calling  schedutil_gov_init+0x0/0x10 @ 1
[    1.799126] initcall schedutil_gov_init+0x0/0x10 returned 0 after 0 usecs
[    1.800132] calling  pm_init+0x0/0xb0 @ 1
[    1.801247] initcall pm_init+0x0/0xb0 returned 0 after 0 usecs
[    1.802126] calling  pm_disk_init+0x0/0x20 @ 1
[    1.803131] initcall pm_disk_init+0x0/0x20 returned 0 after 0 usecs
[    1.804125] calling  swsusp_header_init+0x0/0x50 @ 1
[    1.805128] initcall swsusp_header_init+0x0/0x50 returned 0 after 0 usecs
[    1.806126] calling  rcu_set_runtime_mode+0x0/0x20 @ 1
[    1.807131] initcall rcu_set_runtime_mode+0x0/0x20 returned 0 after 0 usecs
[    1.808125] calling  rcu_init_tasks_generic+0x0/0xd0 @ 1
[    1.809180] initcall rcu_init_tasks_generic+0x0/0xd0 returned 0 after 0 usecs
[    1.810127] calling  init_jiffies_clocksource+0x0/0x20 @ 1
[    1.811126] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    1.812127] initcall init_jiffies_clocksource+0x0/0x20 returned 0 after 1000 usecs
[    1.813126] calling  posixtimer_init+0x0/0xf0 @ 1
[    1.814133] posixtimers hash table entries: 131072 (order: 9, 2097152 bytes, linear)
[    1.815264] initcall posixtimer_init+0x0/0xf0 returned 0 after 1000 usecs
[    1.816127] calling  futex_init+0x0/0x190 @ 1
[    1.818178] futex hash table entries: 65536 (4194304 bytes on 1 NUMA nodes, total 4096 KiB, linear).
[    1.819125] initcall futex_init+0x0/0x190 returned 0 after 2000 usecs
[    1.820125] calling  cgroup_wq_init+0x0/0x80 @ 1
[    1.822154] initcall cgroup_wq_init+0x0/0x80 returned 0 after 1000 usecs
[    1.823125] calling  cgroup1_wq_init+0x0/0x30 @ 1
[    1.824175] initcall cgroup1_wq_init+0x0/0x30 returned 0 after 0 usecs
[    1.825125] calling  trace_events_eprobe_init_early+0x0/0x30 @ 1
[    1.826125] initcall trace_events_eprobe_init_early+0x0/0x30 returned 0 after 0 usecs
[    1.827125] calling  init_kprobe_trace_early+0x0/0x30 @ 1
[    1.828127] initcall init_kprobe_trace_early+0x0/0x30 returned 0 after 0 usecs
[    1.829125] calling  init_events_core_sysctls+0x0/0x30 @ 1
[    1.830132] initcall init_events_core_sysctls+0x0/0x30 returned 0 after 0 usecs
[    1.831126] calling  init_callchain_sysctls+0x0/0x30 @ 1
[    1.832128] initcall init_callchain_sysctls+0x0/0x30 returned 0 after 0 usecs
[    1.833127] calling  fsnotify_init+0x0/0x30 @ 1
[    1.834187] initcall fsnotify_init+0x0/0x30 returned 0 after 0 usecs
[    1.835127] calling  filelock_init+0x0/0x120 @ 1
[    1.836200] initcall filelock_init+0x0/0x120 returned 0 after 0 usecs
[    1.837125] calling  init_misc_binfmt+0x0/0x30 @ 1
[    1.838128] initcall init_misc_binfmt+0x0/0x30 returned 0 after 0 usecs
[    1.839125] calling  init_script_binfmt+0x0/0x20 @ 1
[    1.840126] initcall init_script_binfmt+0x0/0x20 returned 0 after 0 usecs
[    1.841125] calling  init_elf_binfmt+0x0/0x20 @ 1
[    1.842125] initcall init_elf_binfmt+0x0/0x20 returned 0 after 0 usecs
[    1.843125] calling  init_compat_elf_binfmt+0x0/0x20 @ 1
[    1.845125] initcall init_compat_elf_binfmt+0x0/0x20 returned 0 after 0 usecs
[    1.846125] calling  configfs_init+0x0/0xd0 @ 1
[    1.847172] initcall configfs_init+0x0/0xd0 returned 0 after 0 usecs
[    1.848125] calling  debugfs_init+0x0/0x100 @ 1
[    1.850164] initcall debugfs_init+0x0/0x100 returned 0 after 0 usecs
[    1.851127] calling  tracefs_init+0x0/0xb0 @ 1
[    1.852156] initcall tracefs_init+0x0/0xb0 returned 0 after 0 usecs
[    1.853125] calling  security_initcall_core+0x0/0x90 @ 1
[    1.854126] initcall security_initcall_core+0x0/0x90 returned 0 after 0 usecs
[    1.855125] calling  virtio_init+0x0/0x20 @ 1
[    1.856143] initcall virtio_init+0x0/0x20 returned 0 after 0 usecs
[    1.857125] calling  iommu_init+0x0/0x30 @ 1
[    1.858129] initcall iommu_init+0x0/0x30 returned 0 after 0 usecs
[    1.859126] calling  component_debug_init+0x0/0x20 @ 1
[    1.860186] initcall component_debug_init+0x0/0x20 returned 0 after 0 usecs
[    1.861131] calling  early_resume_init+0x0/0xd0 @ 1
[    1.862243] PM: RTC time: 08:10:36, date: 2026-09-17
[    1.863126] initcall early_resume_init+0x0/0xd0 returned 0 after 1000 usecs
[    1.864125] calling  cpufreq_core_init+0x0/0x90 @ 1
[    1.865135] initcall cpufreq_core_init+0x0/0x90 returned 0 after 0 usecs
[    1.866125] calling  cpufreq_gov_performance_init+0x0/0x10 @ 1
[    1.867126] initcall cpufreq_gov_performance_init+0x0/0x10 returned 0 after 0 usecs
[    1.868125] calling  cpufreq_gov_userspace_init+0x0/0x10 @ 1
[    1.869126] initcall cpufreq_gov_userspace_init+0x0/0x10 returned 0 after 0 usecs
[    1.870126] calling  CPU_FREQ_GOV_ONDEMAND_init+0x0/0x10 @ 1
[    1.871125] initcall CPU_FREQ_GOV_ONDEMAND_init+0x0/0x10 returned 0 after 0 usecs
[    1.872125] calling  cpuidle_init+0x0/0x20 @ 1
[    1.873133] initcall cpuidle_init+0x0/0x20 returned 0 after 0 usecs
[    1.874126] calling  unaccepted_memory_init_kdump+0x0/0x20 @ 1
[    1.875125] initcall unaccepted_memory_init_kdump+0x0/0x20 returned 0 after 0 usecs
[    1.876125] calling  sock_init+0x0/0xe0 @ 1

crash> runq
CPU 0 RUNQUEUE: ffff89bd8fe2dbc0
CURRENT: PID: 1 TASK: ffff89b34110b240 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff89bd8fe2ddc0
[no tasks queued]
CFS RB_ROOT: ffff89bd8fe2dc78
[no tasks queued]

... [runq for CPUs 1-254 omitted; idle swapper tasks] ...

crash> bt 1
PID: 1 TASK: ffff89b34110b240 CPU: 0 COMMAND: "swapper/0"

    [exception RIP: watchdog_check_skew+107]
    RIP: ffffffff86e74c9b  RSP: ffffcdf6c0003e38  RFLAGS: 00000002
    RAX: 0000000000000ffd  RBX: ffffffff88a50d00  RCX: 0000000000000008
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: ffffffff88b65a40
    RBP: 000000005a1f4a00   R8: ffff89bd8feaec40   R9: 0000000000000002
    R10: 0000000000000000  R11: 0000000002ab2262  R12: ffff89bd8feaec80
    R13: 0000000000000002  R14: 0000000000000004  R15: ffff89bd8feaec40
    CS: 0010  SS: 0018

#0 [ffffcdf6c0003e70] clocksource_watchdog at ffffffff86e754a2 #1 [ffffcdf6c0003ec0] call_timer_fn at ffffffff86e6b335
#2 [ffffcdf6c0003ee8] __run_timers at ffffffff86e6b89c
#3 [ffffcdf6c0003f78] run_timer_base at ffffffff86e6b981
#4 [ffffcdf6c0003f88] run_timer_softirq at ffffffff86e6bba7 #5 [ffffcdf6c0003f90] handle_softirqs at ffffffff86db039b
#6 [ffffcdf6c0003fe8] __irq_exit_rcu at ffffffff86db06ec
#7 [ffffcdf6c0003ff0] sysvec_apic_timer_interrupt at ffffffff87e4f339
--- <IRQ stack> ---
#8 [ffffcdf6c00239b8] asm_sysvec_apic_timer_interrupt at ffffffff86a00dc6

    [exception RIP: smp_call_function_many_cond+662]
    RIP: ffffffff86e906d6  RSP: ffffcdf6c0023a68  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: 000000000000008f  RCX: 0000000000000000
    RDX: ffff89bd921eed00  RSI: 0000000000000080  RDI: 000000000000000f
    RBP: ffff89bd8fe2ed40   R8: ffff89bd921f39a0   R9: 000000000000008f
    R10: ffff89bd8fe2ed88  R11: 0000000000000003  R12: 0000000000000000
    R13: ffffffff896759a0  R14: 0000000000200000  R15: 000000000000008f
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018

#9 [ffffcdf6c0023ae8] on_each_cpu_cond_mask at ffffffff86e90a4d
#10 [ffffcdf6c0023b18] __change_page_attr_set_clr at ffffffff86d9b161 #11 [ffffcdf6c0023bf8] __change_page_attr_set_clr at ffffffff86d9baae #12 [ffffcdf6c0023cd8] change_page_attr_set_clr at ffffffff86d9bc8a #13 [ffffcdf6c0023d68] set_memory_ro at ffffffff86d9c9bf #14 [ffffcdf6c0023d80] bpf_prog_select_runtime at ffffffff86f0feac #15 [ffffcdf6c0023db8] bpf_prepare_filter at ffffffff87a58111 #16 [ffffcdf6c0023e08] bpf_prog_create at ffffffff87a58322 #17 [ffffcdf6c0023e30] ptp_classifier_init at ffffffff893ef991 #18 [ffffcdf6c0023e50] sock_init at ffffffff893eefd8
#19 [ffffcdf6c0023e90] do_one_initcall at ffffffff86c7f9ed #20 [ffffcdf6c0023f00] kernel_init_freeable at ffffffff8938523c #21 [ffffcdf6c0023f28] kernel_init at ffffffff87e51dd1
#22 [ffffcdf6c0023f30] ret_from_fork at ffffffff86d5895f #23 [ffffcdf6c0023f50] ret_from_fork_asm at ffffffff86c8585a

    RIP: 0000000000000000  RSP: 0000000000000000  RFLAGS: 841f0f2e66
    RAX: 0000000000000000  RBX: 1f0f2e6600000000  RCX: 2e66000000000084
    RDX: 0000000000841f0f  RSI: 000000841f0f2e66  RDI: 00841f0f2e660000
    RBP: 00841f0f2e660000   R8: 00841f0f2e660000   R9: 000000841f0f2e66
    R10: 0000000000841f0f  R11: 2e66000000000084  R12: 000000841f0f2e66
    R13: 0000000000841f0f  R14: 2e66000000000084  R15: 1f0f2e6600000000
    ORIG_RAX: 1f0f2e6600000000  CS: 841f0f  SS: 401f0f00000000

WARNING: possibly bogus exception frame

crash> dis watchdog_check_skew
0xffffffff86e74c30 <watchdog_check_skew>: push %r15
0xffffffff86e74c32 <watchdog_check_skew+2>: mov %rdi,%r15
0xffffffff86e74c35 <watchdog_check_skew+5>: push %r14
0xffffffff86e74c37 <watchdog_check_skew+7>: movslq %esi,%r14

0xffffffff86e74c3a <watchdog_check_skew+10>:	push   %r13
0xffffffff86e74c3c <watchdog_check_skew+12>:	mov    %r14,%r13
0xffffffff86e74c3f <watchdog_check_skew+15>:	push   %r12
0xffffffff86e74c41 <watchdog_check_skew+17>:	lea    0x40(%r15),%r12
0xffffffff86e74c45 <watchdog_check_skew+21>:	push   %rbp
0xffffffff86e74c46 <watchdog_check_skew+22>:	push   %rbx
0xffffffff86e74c47 <watchdog_check_skew+23>:	sub    $0x8,%rsp
0xffffffff86e74c4b <watchdog_check_skew+27>:	call   0xffffffff86e70d40 <ktime_get_raw>
0xffffffff86e74c50 <watchdog_check_skew+32>:	mov    0x38(%r15),%rbx
0xffffffff86e74c54 <watchdog_check_skew+36>:	lea    0x1(%r14),%edx
0xffffffff86e74c58 <watchdog_check_skew+40>:	add    $0x4,%r14
0xffffffff86e74c5c <watchdog_check_skew+44>:	mov    %edx,(%rsp)
0xffffffff86e74c5f <watchdog_check_skew+47>:	mov    %rax,%rbp
0xffffffff86e74c62 <watchdog_check_skew+50>:	mov    %rbx,%rdi
0xffffffff86e74c65 <watchdog_check_skew+53>:	mov    (%rbx),%rax
0xffffffff86e74c68 <watchdog_check_skew+56>:	call   0xffffffff87e5e440 <__pi___x86_indirect_thunk_rax>
0xffffffff86e74c6d <watchdog_check_skew+61>:	mov    %rax,0x8(%r15,%r14,8)
0xffffffff86e74c72 <watchdog_check_skew+66>:	lock incl 0x40(%r15)
0xffffffff86e74c77 <watchdog_check_skew+71>:	mov    (%rsp),%edx
0xffffffff86e74c7a <watchdog_check_skew+74>:	add    $0x2,%r13d
0xffffffff86e74c7e <watchdog_check_skew+78>:	and    $0x1,%edx
0xffffffff86e74c81 <watchdog_check_skew+81>:	lea    0x4(%rdx),%rax
0xffffffff86e74c85 <watchdog_check_skew+85>:	mov    %rax,(%rsp)
0xffffffff86e74c89 <watchdog_check_skew+89>:	mov    0x40(%r15),%edx
0xffffffff86e74c8d <watchdog_check_skew+93>:	xor    %eax,%eax
0xffffffff86e74c8f <watchdog_check_skew+95>:	cmp    %edx,%r13d
0xffffffff86e74c92 <watchdog_check_skew+98>:	jg     0xffffffff86e74ca4 <watchdog_check_skew+116>
0xffffffff86e74c94 <watchdog_check_skew+100>:	jmp    0xffffffff86e74cdb <watchdog_check_skew+171>
0xffffffff86e74c96 <watchdog_check_skew+102>:	add    $0x1,%eax
0xffffffff86e74c99 <watchdog_check_skew+105>:	pause
0xffffffff86e74c9b <watchdog_check_skew+107>:	mov    0x40(%r15),%edx
0xffffffff86e74c9f <watchdog_check_skew+111>:	cmp    %r13d,%edx
0xffffffff86e74ca2 <watchdog_check_skew+114>:	jge    0xffffffff86e74cdb <watchdog_check_skew+171>
0xffffffff86e74ca4 <watchdog_check_skew+116>:	mov    0x24(%r15),%edx
0xffffffff86e74ca8 <watchdog_check_skew+120>:	test   %edx,%edx
0xffffffff86e74caa <watchdog_check_skew+122>:	jne    0xffffffff86e74d11 <watchdog_check_skew+225>
0xffffffff86e74cac <watchdog_check_skew+124>:	cmp    $0x1388,%eax
0xffffffff86e74cb1 <watchdog_check_skew+129>:	jne    0xffffffff86e74c96 <watchdog_check_skew+102>
0xffffffff86e74cb3 <watchdog_check_skew+131>:	call   0xffffffff86e70d40 <ktime_get_raw>
0xffffffff86e74cb8 <watchdog_check_skew+136>:	sub    %rbp,%rax
0xffffffff86e74cbb <watchdog_check_skew+139>:	cmp    0x48(%r15),%rax
0xffffffff86e74cbf <watchdog_check_skew+143>:	jae    0xffffffff86e74d21 <watchdog_check_skew+241>
0xffffffff86e74cc1 <watchdog_check_skew+145>:	pause
0xffffffff86e74cc3 <watchdog_check_skew+147>:	mov    0x40(%r15),%eax
0xffffffff86e74cc7 <watchdog_check_skew+151>:	cmp    %r13d,%eax
0xffffffff86e74cca <watchdog_check_skew+154>:	jge    0xffffffff86e74cdb <watchdog_check_skew+171>
0xffffffff86e74ccc <watchdog_check_skew+156>:	mov    0x24(%r15),%eax
0xffffffff86e74cd0 <watchdog_check_skew+160>:	test   %eax,%eax
0xffffffff86e74cd2 <watchdog_check_skew+162>:	jne    0xffffffff86e74d11 <watchdog_check_skew+225>
0xffffffff86e74cd4 <watchdog_check_skew+164>:	mov    $0x1,%eax
0xffffffff86e74cd9 <watchdog_check_skew+169>:	jmp    0xffffffff86e74c96 <watchdog_check_skew+102>
0xffffffff86e74cdb <watchdog_check_skew+171>:	mov    (%rbx),%rax
0xffffffff86e74cde <watchdog_check_skew+174>:	mov    %rbx,%rdi
0xffffffff86e74ce1 <watchdog_check_skew+177>:	call   0xffffffff87e5e440 <__pi___x86_indirect_thunk_rax>
0xffffffff86e74ce6 <watchdog_check_skew+182>:	mov    (%rsp),%rcx
0xffffffff86e74cea <watchdog_check_skew+186>:	mov    %rax,0x8(%r15,%r14,8)
0xffffffff86e74cef <watchdog_check_skew+191>:	sub    0x8(%r15,%rcx,8),%rax
0xffffffff86e74cf4 <watchdog_check_skew+196>:	and    0x8(%rbx),%rax
0xffffffff86e74cf8 <watchdog_check_skew+200>:	cmp    %rax,0x30(%rbx)
0xffffffff86e74cfc <watchdog_check_skew+204>:	jb     0xffffffff86e74d73 <watchdog_check_skew+323>
0xffffffff86e74cfe <watchdog_check_skew+206>:	lock incl (%r12)
0xffffffff86e74d03 <watchdog_check_skew+211>:	add    $0x2,%r13d
0xffffffff86e74d07 <watchdog_check_skew+215>:	cmp    $0x9,%r13d
0xffffffff86e74d0b <watchdog_check_skew+219>:	jle    0xffffffff86e74c89 <watchdog_check_skew+89>
0xffffffff86e74d11 <watchdog_check_skew+225>:	pop    %rax
0xffffffff86e74d12 <watchdog_check_skew+226>:	pop    %rbx
0xffffffff86e74d13 <watchdog_check_skew+227>:	pop    %rbp
0xffffffff86e74d14 <watchdog_check_skew+228>:	pop    %r12
0xffffffff86e74d16 <watchdog_check_skew+230>:	pop    %r13
0xffffffff86e74d18 <watchdog_check_skew+232>:	pop    %r14
0xffffffff86e74d1a <watchdog_check_skew+234>:	pop    %r15
0xffffffff86e74d1c <watchdog_check_skew+236>:	jmp    0xffffffff87e5ef40 <retbleed_return_thunk>
0xffffffff86e74d21 <watchdog_check_skew+241>:	mov    $0xffffffff896e5500,%rdi
0xffffffff86e74d28 <watchdog_check_skew+248>:	call   0xffffffff87e5dd70 <_raw_spin_lock>
0xffffffff86e74d2d <watchdog_check_skew+253>:	mov    0x24(%r15),%ecx
0xffffffff86e74d31 <watchdog_check_skew+257>:	test   %ecx,%ecx
0xffffffff86e74d33 <watchdog_check_skew+259>:	je     0xffffffff86e74d61 <watchdog_check_skew+305>
0xffffffff86e74d35 <watchdog_check_skew+261>:	movabs $0xffffffff896e54ff,%rax
0xffffffff86e74d3f <watchdog_check_skew+271>:	cmp    $0xffffffffffffefff,%rax
0xffffffff86e74d45 <watchdog_check_skew+277>:	ja     0xffffffff86e74d11 <watchdog_check_skew+225>
0xffffffff86e74d47 <watchdog_check_skew+279>:	add    $0x8,%rsp
0xffffffff86e74d4b <watchdog_check_skew+283>:	mov    $0xffffffff896e5500,%rdi
0xffffffff86e74d52 <watchdog_check_skew+290>:	pop    %rbx
0xffffffff86e74d53 <watchdog_check_skew+291>:	pop    %rbp
0xffffffff86e74d54 <watchdog_check_skew+292>:	pop    %r12
0xffffffff86e74d56 <watchdog_check_skew+294>:	pop    %r13
0xffffffff86e74d58 <watchdog_check_skew+296>:	pop    %r14
0xffffffff86e74d5a <watchdog_check_skew+298>:	pop    %r15
0xffffffff86e74d5c <watchdog_check_skew+300>:	jmp    0xffffffff87e5dbf0 <_raw_spin_unlock>
0xffffffff86e74d61 <watchdog_check_skew+305>:	movl   $0xa,0x40(%r15)
0xffffffff86e74d69 <watchdog_check_skew+313>:	movl   $0x5,0x24(%r15)
0xffffffff86e74d71 <watchdog_check_skew+321>:	jmp    0xffffffff86e74d35 <watchdog_check_skew+261>
0xffffffff86e74d73 <watchdog_check_skew+323>:	mov    $0xffffffff896e5500,%rdi
0xffffffff86e74d7a <watchdog_check_skew+330>:	call   0xffffffff87e5dd70 <_raw_spin_lock>
0xffffffff86e74d7f <watchdog_check_skew+335>:	mov    0x24(%r15),%edx
0xffffffff86e74d83 <watchdog_check_skew+339>:	test   %edx,%edx
0xffffffff86e74d85 <watchdog_check_skew+341>:	jne    0xffffffff86e74d35 <watchdog_check_skew+261>
0xffffffff86e74d87 <watchdog_check_skew+343>:	movl   $0xa,0x40(%r15)
0xffffffff86e74d8f <watchdog_check_skew+351>:	movl   $0x6,0x24(%r15)
0xffffffff86e74d97 <watchdog_check_skew+359>:	jmp    0xffffffff86e74d35 <watchdog_check_skew+261>

Please enable CONFIG_CSD_LOCK_WAIT_DEBUG and run it again. That might tell us something more about what's going on.

Thx.

Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

I recreated the issue with CONFIG_CSD_LOCK_WAIT_DEBUG=y / DEFAULT=y (with and without csdlock_debug=1) and still get no csd wait dumps.