Fix Hard Lockup in ktime_get_ts64 with debuggers

This patch fixes a hard system lockup that occurs when a system has been held by a kernel debugger such as kgdb, kdb, or mdb on a multiprocessor system with all processors suspended then released after 20-30 minutes of debugging.

SUMMARY

An SMP multiprocessor capable kernel debugger must stop all processors on a target system and trigger them to be held inside a NMI polling routine while tracing and debugging active events on a target system. This is typically accomplished by triggering an NMI interrupt via an IPI call to each processor then directing the processors to enter a suspend code loop from the NMI handlers. While the processors are held suspended, the system hardware timer will continue to run as well as other hardware components. Upon exiting the debugger console, the processors are released and the system is expected to resume normal operations. In the case that causes this bug, the Linux timekeeper subsystem sees a very large delta value for the time in nanoseconds elapsed then attempts to process this large number. Since the function macro timespec_add_ns calls an inline macro function that manually subtracts the values longhand rather than use the processor to perform the divide operation, this results in the hard lockup detector causing a panic because the function is looping and hung for about 4 minutes on average calling this longhand divide macro __iter_div_u64_rem.

CONDITIONS THAT RESULT IN THIS LOCKUP

Holding the system processors suspended in the NMI handlers for 20 minutes to several hours while examining and debugging a target system will trigger this bug upon exit from the debugger console on either kgdb, kdb, or mdb. It appears to be caused by the delta shift of the counters in the hardware clock in ns being calculated and passed to the function to update the timespec. The function being replaced is little more than a cycle hog on the system and needs to be replaced with something that does not trigger the hard lockup detector if it sees a really big number for nanoseconds.

---
 include/linux/math64.h | 17 +++++++++++++++++
 include/linux/time.h   |  2 +-
 2 files changed, 18 insertions(+), 1 deletion(-)

index 6e8b5b2..6769a3d 100644
@@ -22,6 +22,15 @@ static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder)
 }
 
 /**
+ * div_s64_rem64 - unsigned 64bit divide with 32bit divisor with 64bit remainder
+ */
+static inline u64 div_u64_rem64(u64 dividend, u32 divisor, u64 *remainder)
+{
+	*remainder = dividend % divisor;
+	return dividend / divisor;
+}
+
+/**
  * div_s64_rem - signed 64bit divide with 32bit divisor with remainder
  */
 static inline s64 div_s64_rem(s64 dividend, s32 divisor, s32 *remainder)

@@ -68,6 +77,14 @@ static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder)
 }
 #endif
 
+#ifndef div_u64_rem64
+static inline u64 div_u64_rem64(u64 dividend, u32 divisor, u64 *remainder)
+{
+	*remainder = do_div(dividend, divisor);
+	return dividend;
+}
+#endif
+
 #ifndef div_s64_rem
 extern s64 div_s64_rem(s64 dividend, s32 divisor, s32 *remainder);
 #endif

index 297f09f..e9e1fbc 100644
@@ -258,7 +258,7 @@ extern struct timeval ns_to_timeval(const s64 nsec);
  */
 static __always_inline void timespec_add_ns(struct timespec *a, u64 ns)
 {
-	a->tv_sec += __iter_div_u64_rem(a->tv_nsec + ns, NSEC_PER_SEC, &ns);
+	a->tv_sec += div_u64_rem64(a->tv_nsec + ns, NSEC_PER_SEC, &ns);
 	a->tv_nsec = ns;
 }
 

1.8.3.1

I just finished a 4 hour regression test of this patch and please drop
it. It does fix the symptom but not the illness. It worked on i386
and x86_64 for my first test but failed the second test. I did a 4 hour debugging session and the problem just shows up another way.

This patch gets rid of the hard lockup but it results in the gnome gui and systemd eventually locking up after entering and exiting the debugger and holding the system suspended for several hours. This tells me that whatever is causing that value to get so large with the clock skew is affecting other areas of the timekeeping logic.

I will keep working on it since I need this to work. kgdb also experiences the same problem.

So drop this patch and I'll keep looking for the sickness rather than treat the symtoms. If you guys have any ideas let me know.

Jeff