Skip to contentSkip to conversation

Mysterious RTC hangs on x86_64 - fixed, sort of

PATCH Zachary Amsden 3 participants 5 messages May 2, 2007 – May 3, 2007

lore
Share & tools
lore
#0 lore raw

With this patch, /sbin/hwclock no longer hangs my AMD64 machine when run after reaching multiuser. What I don't understand is why. I have the RTC based sound sequencer timer as a module, but not loaded, and the error message I added to indicate broken rtc control does not fire.

So why is it that if I stop taking the rtc_task_lock and issuing the callbacks which should never be held or exist that my system no longer hard freezes?

Zach

char/rtc.c 2007-05-03 15:27:49.000000000 -0700 renamed from tmp/a 2007-05-03 15:36:07.451256181 -0700 +5 -4
@@ -265,10 +265,10 @@
        spin_unlock (&rtc_lock);
 
        /* Now do the rest of the actions */
-       spin_lock(&rtc_task_lock);
-       if (rtc_callback)
-               rtc_callback->func(rtc_callback->private_data);
-       spin_unlock(&rtc_task_lock);
+/*     spin_lock(&rtc_task_lock); */
+//     if (rtc_callback)
+//             rtc_callback->func(rtc_callback->private_data);
+/*     spin_unlock(&rtc_task_lock); */
        wake_up_interruptible(&rtc_wait);       
 
        kill_fasync (&rtc_async_queue, SIGIO, POLL_IN);
@@ -811,6 +811,7 @@
 
 int rtc_register(rtc_task_t *task)
 {
+       printk(KERN_ERR "rtc_register is busted\n");
 #ifndef RTC_IRQ
        return -EIO;
 #else

CONFIG_HPET_EMULATE_RTC=y
CONFIG_RTC=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_SND_RTCTIMER=m
CONFIG_SND_SEQ_RTCTIMER_DEFAULT=y
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
CONFIG_RTC_DEBUG=y
# RTC interfaces
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y

#1 lore raw
Quoted 14 lines · Zachary Amsden

With this patch, /sbin/hwclock no longer hangs my AMD64 machine when run after reaching multiuser. What I don't understand is why. I have the RTC based sound sequencer timer as a module, but not loaded, and the error message I added to indicate broken rtc control does not fire.

So why is it that if I stop taking the rtc_task_lock and issuing the callbacks which should never be held or exist that my system no longer hard freezes?

char/rtc.c 2007-05-03 15:27:49.000000000 -0700 renamed from tmp/a 2007-05-03 15:36:07.451256181 -0700 +0 -0
@@ -265,10 +265,10 @@
        spin_unlock (&rtc_lock);
 

char/rtc.c 2007-05-03 15:27:49.000000000 -0700 line 272

Quoted 10 lines
char/rtc.c 2007-05-03 15:27:49.000000000 -0700 renamed from tmp/a 2007-05-03 15:36:07.451256181 -0700 +4 -4
        /* Now do the rest of the actions */
-       spin_lock(&rtc_task_lock);
-       if (rtc_callback)
-               rtc_callback->func(rtc_callback->private_data);
-       spin_unlock(&rtc_task_lock);
+/*     spin_lock(&rtc_task_lock); */
+//     if (rtc_callback)
+//             rtc_callback->func(rtc_callback->private_data);
+/*     spin_unlock(&rtc_task_lock); */
        wake_up_interruptible(&rtc_wait);       

Try leaving the spinlocks and just disabling the callbacks. And maybe enable spinlock debugging...

Quoted 2 lines

CONFIG_HPET_EMULATE_RTC=y

Did you try without that?

#2 lore raw

Chuck Ebbert wrote:

Well, turns out this is a heisenbug. Which is good, since it means the nop patch didn't change anything.

Quoted 3 lines

Try leaving the spinlocks and just disabling the callbacks. And maybe enable spinlock debugging...

I tried removing all the spinlocks inside the interrupt handler. Seemed to work fine for a while, but still hung (at worst, it looks missing locks means we might screw up and read / write the wrong CMOS register, not hang or crash).

So I took down 2nd CPU with hotplug (did not yet try UP kernel though). It took a longer time, but still hung. Seems not to be a spinlock problem, but I'll turn on debugging anyway.

Quoted 6 lines
Quoted 3 lines

CONFIG_HPET_EMULATE_RTC=y

Did you try without that?

Will do. That looks much more suspicious like. I thought I killed it already, but had only got this:

# CONFIG_HPET_RTC_IRQ is not set

If that still crashes, I'll try running cmos access in a loop in userspace to see if maybe the port I/O is tickling a chipset bug (the only other report I know of is on same chipset, nVidia MCP51). Maybe SMM handler is accessing CMOS or something wacked out. <laughs hysterical... stops laughing when the theory actually sounds plausible>. Stuck in SMM is not good for CPU thermal throttling ... hopefully Turion's don't reach nuclear emission point.

Would also explain maybe why NMI watchdog doesn't seem to notice anything wrong.

Thanks,
Zach

#3 lore raw
In reply to Chuck Ebbert
Quoted 7 lines · Chuck Ebbert
Quoted 4 lines

CONFIG_HPET_EMULATE_RTC=y

Did you try without that?

Just did. Still hangs same way; strace shows /sbin/hwclock dying after hundreds of RTC_RD_TIME. And now /proc/interrupts shows no rtc interrupts being generated (expected, I gues). Seems to take longer to crash, but this is a heisenbug.

Enough crashing for today. Strangest thing is the NMI watchdog does not fire...

Zach

#4 lore raw
Quoted 3 lines

Enough crashing for today. Strangest thing is the NMI watchdog does not fire...

It's disabled now by default.

-Andi