hangcheck-timer module

[ Feh, forgot to attach the damned file. ]

Folks,
Attached is a module, hangcheck-timer. It is used to detect
when the system goes out to lunch for a period of time, such as when a driver like qla2x00 udelays a bunch.
The module sets a timer. When the timer goes off, it then uses
the TSC (warning: portability needed) to determine how much real time has passed.
On a normal system, the real elapsed time will be almost
identical to the expected timer duration. However, if a device decided to udelay for 60 seconds (or some other circumstance), the module takes notice. If the margin of error passes a threshold, the machine is rebooted.
The module is currently used in a cluster environment. After
some time out to lunch, the rest of the cluster will have given up on a machine. If the machine suddenly comes back and assumes it is still "live", bad things can happen.
We can also see use for this in a debugging sense, for kernel
hangs as well as driver code. That's why I'm proposing it for general inclusion.
Comments? Thoughts?

Joel

Building:
The module should happily build against most 2.4 kernels. The
usual module building compile line:

	gcc  -I /scratch/jlbec/kernel/linux-2.4.20-rc2/include \
		-DMODULE -D__KERNEL__ -DLINUX  -c -o hangcheck-timer.o \
		hangcheck-timer.c

Running:
Load the module with insmod. There are two options.
"hangcheck_tick=<seconds>" specifies the timer timeout, and "hangcheck_margin=<seconds" specifies the margin of error.

"Friends may come and go, but enemies accumulate." 
        - Thomas Jones

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: [email protected]
Phone: (650) 506-8127

/*
* hangcheck-timer.c
*
* Test driver for a little io fencing timer.
*
* Copyright (C) 2002 Oracle Corporation. All rights reserved. *
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public
* License version 2 as published by the Free Software Foundation. *
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have recieved a copy of the GNU General Public
* License along with this program; if not, write to the
* Free Software Foundation, Inc., 59 Temple Place - Suite 330,
* Boston, MA 021110-1307, USA.
*/

#include <linux/module.h>
#include <linux/config.h>
#include <linux/types.h>
#include <linux/kernel.h>
#include <linux/fs.h>
#include <linux/mm.h>
#include <linux/reboot.h>
#include <linux/smp_lock.h>
#include <linux/init.h>
#include <asm/uaccess.h>

/* #include "hangcheck-timer.h" */

#define DEFAULT_IOFENCE_MARGIN 60 /* Default fudge factor, in seconds */
#define DEFAULT_IOFENCE_TICK 180 /* Default timer timeout, in seconds */

static int hangcheck_tick = DEFAULT_IOFENCE_TICK; static int hangcheck_margin = DEFAULT_IOFENCE_MARGIN;

MODULE_PARM(hangcheck_tick,"i");
MODULE_PARM_DESC(hangcheck_tick, "Timer delay.");
MODULE_PARM(hangcheck_margin,"i");
MODULE_PARM_DESC(hangcheck_margin, "If the hangcheck timer has been delayed more than hangcheck_margin seconds, the machine will reboot."); MODULE_LICENSE("GPL");

static void hangcheck_fire(unsigned long);

/* Last time scheduled */
static unsigned long long hangcheck_tsc, hangcheck_tsc_margin;

static struct timer_list hangcheck_ticktock = {
function: hangcheck_fire,
};

static void hangcheck_fire(unsigned long data)

{
	unsigned long long cur_tsc, tsc_diff;

	rdtscll(cur_tsc);

	if (cur_tsc > hangcheck_tsc)
		tsc_diff = cur_tsc - hangcheck_tsc;
	else
		tsc_diff = (cur_tsc + (~0ULL - hangcheck_tsc)); /* or something */

#if 0
	printk(KERN_CRIT "tsc_diff = %lu.%lu, predicted diff is %lu.%lu.\n",
	       (unsigned long) ((tsc_diff >> 32) & 0xFFFFFFFFULL),
	       (unsigned long) (tsc_diff & 0xFFFFFFFFULL),
	       (unsigned long) ((hangcheck_tsc_margin >> 32) & 0xFFFFFFFFULL),
	       (unsigned long) (hangcheck_tsc_margin & 0xFFFFFFFFULL));
	printk(KERN_CRIT "hangcheck_margin = %lu, HZ = %lu, current_cpu_data.loops_per_jiffy = %lu.\n", hangcheck_margin, HZ, current_cpu_data.loops_per_jiffy);
#endif
	
	if (tsc_diff > hangcheck_tsc_margin) {
		printk(KERN_CRIT "hangcheck is restarting the machine.\n");
		machine_restart(NULL);
	}
	mod_timer(&hangcheck_ticktock, jiffies + (hangcheck_tick*HZ));
	rdtscll(hangcheck_tsc);
}  /* hangcheck_fire() */

static int __init hangcheck_init(void)

{
        version_hash_print();
	printk("Starting hangcheck timer (tick is %d seconds, margin is %d seconds).\n",
	       hangcheck_tick, hangcheck_margin);

	hangcheck_tsc_margin = (unsigned long long)(hangcheck_margin + hangcheck_tick) * (unsigned long long)HZ * (unsigned long long)current_cpu_data.loops_per_jiffy;

	rdtscll(hangcheck_tsc);
	mod_timer(&hangcheck_ticktock, jiffies + (hangcheck_tick*HZ));

	return 0;
}  /* hangcheck_init() */

static void __exit hangcheck_exit(void)

{
	printk("Stopping hangcheck timer.\n");

	lock_kernel();
	del_timer(&hangcheck_ticktock);
	unlock_kernel();
}  /* hangcheck_exit() */

module_init(hangcheck_init);
module_exit(hangcheck_exit);

ISTR get_cycles() being around, which should be defined for other arches.

Bill

:-)

Would it make it more sense for other machines to "kill" offending machine (cut power or press reset)?

				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

There is no solution that is general and inexpensive. STONITH
is as close as it gets, and we don't have support for that. On other platforms where the shared disk is on FC, the device driver supports fencing nodes from the switch.
That said, this module isn't exclusively useful to a
cluster+shared disk environment. If it were, I couldn't see generic inclusion. This code is useful in many other situations.

Joel

Life's Little Instruction Book #313

	"Never underestimate the power of love."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: [email protected]
Phone: (650) 506-8127