Skip to contentSkip to conversation

sysrq: don't hold the sysrq_key_table_lock during the handler

RFC PATCH In review Xiaotian Feng 4 participants 9 messages Jul 26, 2010 – Jul 27, 2010

Subsystems (11)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 sysrq: don't hold the sysrq_key_table_lock during the handler

sysrq_key_table_lock is used to protect the sysrq_key_table, make sure we get/replace the right operation for the sysrq. But in __handle_sysrq, kernel will hold this lock and disable irqs until we finished op_p->handler(). This may cause false positive watchdog alert when we're doing "show-task-states" on a system with many tasks.

Cc Ingo Molnar <[email protected]>
Cc Dmitry Torokhov <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Neil Horman <[email protected]>
Cc "David S. Miller" <[email protected]>
---
 drivers/char/sysrq.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/char/sysrq.c +3 -1
index 878ac0c..0856e2e 100644
@@ -520,9 +520,11 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		if (!check_mask || sysrq_on_mask(op_p->enable_mask)) {
 			printk("%s\n", op_p->action_msg);
 			console_loglevel = orig_log_level;
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 			op_p->handler(key, tty);
 		} else {
 			printk("This sysrq operation is disabled.\n");
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 		}
 	} else {
 		printk("HELP : ");
@@ -541,8 +543,8 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		}
 		printk("\n");
 		console_loglevel = orig_log_level;
+		spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 	}
-	spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 }
 
 void handle_sysrq(int key, struct tty_struct *tty)
Signature · 1 line
1.7.2
#1 lore raw
Quoted 46 lines · Xiaotian Feng

sysrq_key_table_lock is used to protect the sysrq_key_table, make sure we get/replace the right operation for the sysrq. But in __handle_sysrq, kernel will hold this lock and disable irqs until we finished op_p->handler(). This may cause false positive watchdog alert when we're doing "show-task-states" on a system with many tasks.

Cc Ingo Molnar <[email protected]>
Cc Dmitry Torokhov <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Neil Horman <[email protected]>
Cc "David S. Miller" <[email protected]>
---
 drivers/char/sysrq.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/char/sysrq.c +3 -1
index 878ac0c..0856e2e 100644
@@ -520,9 +520,11 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		if (!check_mask || sysrq_on_mask(op_p->enable_mask)) {
 			printk("%s\n", op_p->action_msg);
 			console_loglevel = orig_log_level;
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 			op_p->handler(key, tty);
 		} else {
 			printk("This sysrq operation is disabled.\n");
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 		}
 	} else {
 		printk("HELP : ");
@@ -541,8 +543,8 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		}
 		printk("\n");
 		console_loglevel = orig_log_level;
+		spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 	}
-	spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 }
 
 void handle_sysrq(int key, struct tty_struct *tty)
Signature · 1 line
1.7.2

This creates the possibility of a race in the handler. Not that it happens often, but sysrq keys can be registered and unregistered dynamically. If that lock isn't held while we call the keys handler, the code implementing that handler can live in a module that gets removed while its executing, leading to an oops, etc. I think the better solution would be to use an rcu lock here. Neil

#2 lore raw
Quoted 42 lines · Neil Horman
Quoted 41 lines · Xiaotian Feng

sysrq_key_table_lock is used to protect the sysrq_key_table, make sure we get/replace the right operation for the sysrq. But in __handle_sysrq, kernel will hold this lock and disable irqs until we finished op_p->handler(). This may cause false positive watchdog alert when we're doing "show-task-states" on a system with many tasks.

Cc Ingo Molnar <[email protected]>
Cc Dmitry Torokhov <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Neil Horman <[email protected]>
Cc "David S. Miller" <[email protected]>
---
 drivers/char/sysrq.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/char/sysrq.c +3 -1
index 878ac0c..0856e2e 100644
@@ -520,9 +520,11 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		if (!check_mask || sysrq_on_mask(op_p->enable_mask)) {
 			printk("%s\n", op_p->action_msg);
 			console_loglevel = orig_log_level;
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 			op_p->handler(key, tty);
 		} else {
 			printk("This sysrq operation is disabled.\n");
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 		}
 	} else {
 		printk("HELP : ");
@@ -541,8 +543,8 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		}
 		printk("\n");
 		console_loglevel = orig_log_level;
+		spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 	}
-	spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 }
 
Quoted 11 lines
Quoted 5 lines
drivers/char/sysrq.c +0 -0
 void handle_sysrq(int key, struct tty_struct *tty)
Signature · 1 line
1.7.2

This creates the possibility of a race in the handler. Not that it happens often, but sysrq keys can be registered and unregistered dynamically. If that lock isn't held while we call the keys handler, the code implementing that handler can live in a module that gets removed while its executing, leading to an oops, etc. I think the better solution would be to use an rcu lock here.

I'd simply changed spinlock to a mutex.

Signature · 1 line
Dmitry
#3 lore raw
Quoted 55 lines · Dmitry Torokhov
Quoted 42 lines · Neil Horman
Quoted 41 lines · Xiaotian Feng

sysrq_key_table_lock is used to protect the sysrq_key_table, make sure we get/replace the right operation for the sysrq. But in __handle_sysrq, kernel will hold this lock and disable irqs until we finished op_p->handler(). This may cause false positive watchdog alert when we're doing "show-task-states" on a system with many tasks.

Cc Ingo Molnar <[email protected]>
Cc Dmitry Torokhov <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Neil Horman <[email protected]>
Cc "David S. Miller" <[email protected]>
---
 drivers/char/sysrq.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/char/sysrq.c +3 -1
index 878ac0c..0856e2e 100644
@@ -520,9 +520,11 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		if (!check_mask || sysrq_on_mask(op_p->enable_mask)) {
 			printk("%s\n", op_p->action_msg);
 			console_loglevel = orig_log_level;
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 			op_p->handler(key, tty);
 		} else {
 			printk("This sysrq operation is disabled.\n");
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 		}
 	} else {
 		printk("HELP : ");
@@ -541,8 +543,8 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		}
 		printk("\n");
 		console_loglevel = orig_log_level;
+		spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 	}
-	spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 }
 
Quoted 11 lines
Quoted 5 lines
drivers/char/sysrq.c +0 -0
 void handle_sysrq(int key, struct tty_struct *tty)
Signature · 1 line
1.7.2

This creates the possibility of a race in the handler. Not that it happens often, but sysrq keys can be registered and unregistered dynamically. If that lock isn't held while we call the keys handler, the code implementing that handler can live in a module that gets removed while its executing, leading to an oops, etc. I think the better solution would be to use an rcu lock here.

Quoted 2 lines

I'd simply changed spinlock to a mutex.

I don't think you can do that safely in this path, as sysrqs will be looked up in both process (echo t > /proc/sysrq-trigger) context and in interrupt (alt-sysrq-t) context. If a mutex is locked and you try to take it in interrupt context, you get a sleeping-in-interrupt panic IIRC

Neil

Quoted 3 lines
Signature · 1 line
Dmitry
#4 lore raw
Quoted 56 lines · Neil Horman
Quoted 55 lines · Dmitry Torokhov
Quoted 53 lines · Neil Horman
Quoted 46 lines · Xiaotian Feng

sysrq_key_table_lock is used to protect the sysrq_key_table, make sure we get/replace the right operation for the sysrq. But in __handle_sysrq, kernel will hold this lock and disable irqs until we finished op_p->handler(). This may cause false positive watchdog alert when we're doing "show-task-states" on a system with many tasks.

Cc Ingo Molnar <[email protected]>
Cc Dmitry Torokhov <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Neil Horman <[email protected]>
Cc "David S. Miller" <[email protected]>
---
 drivers/char/sysrq.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/char/sysrq.c +3 -1
index 878ac0c..0856e2e 100644
@@ -520,9 +520,11 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		if (!check_mask || sysrq_on_mask(op_p->enable_mask)) {
 			printk("%s\n", op_p->action_msg);
 			console_loglevel = orig_log_level;
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 			op_p->handler(key, tty);
 		} else {
 			printk("This sysrq operation is disabled.\n");
+			spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 		}
 	} else {
 		printk("HELP : ");
@@ -541,8 +543,8 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		}
 		printk("\n");
 		console_loglevel = orig_log_level;
+		spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 	}
-	spin_unlock_irqrestore(&sysrq_key_table_lock, flags);
 }
 
 void handle_sysrq(int key, struct tty_struct *tty)
Signature · 1 line
1.7.2

This creates the possibility of a race in the handler. Not that it happens often, but sysrq keys can be registered and unregistered dynamically. If that lock isn't held while we call the keys handler, the code implementing that handler can live in a module that gets removed while its executing, leading to an oops, etc. I think the better solution would be to use an rcu lock here.

Quoted 7 lines
Quoted 2 lines

I'd simply changed spinlock to a mutex.

I don't think you can do that safely in this path, as sysrqs will be looked up in both process (echo t > /proc/sysrq-trigger) context and in interrupt (alt-sysrq-t) context. If a mutex is locked and you try to take it in interrupt context, you get a sleeping-in-interrupt panic IIRC

Yes, indeed. But then even RCU will not really help us since keyboard driver will have inpterrupts disabled anyways.

Signature · 1 line
Dmitry
#5 lore raw
Quoted 14 lines · Dmitry Torokhov
Quoted 13 lines · Neil Horman
Quoted 12 lines · Dmitry Torokhov
Quoted 8 lines · Neil Horman

On Mon, Jul 26, 2010 at 05:54:02PM +0800, Xiaotian Feng wrote:
<snip>

This creates the possibility of a race in the handler. Not that it happens often, but sysrq keys can be registered and unregistered dynamically. If that lock isn't held while we call the keys handler, the code implementing that handler can live in a module that gets removed while its executing, leading to an oops, etc. I think the better solution would be to use an rcu lock here.

I'd simply changed spinlock to a mutex.

Quoted 9 lines
Quoted 5 lines

I don't think you can do that safely in this path, as sysrqs will be looked up in both process (echo t > /proc/sysrq-trigger) context and in interrupt (alt-sysrq-t) context. If a mutex is locked and you try to take it in interrupt context, you get a sleeping-in-interrupt panic IIRC

Yes, indeed. But then even RCU will not really help us since keyboard driver will have inpterrupts disabled anyways.

Hm, thats true. I suppose the right thing to do then is grab a reference on any sysrq implemented within code that might be considered transient before releasing the lock. I've not tested this patch out, but it should do what we need, in that it allows us to release the lock without having to worry about the op list changing underneath us, or having the module with the handler code dissappear

 arch/arm/kernel/etm.c           |    1 +
 arch/powerpc/xmon/xmon.c        |    1 +
 arch/sparc/kernel/process_64.c  |    1 +
 drivers/char/sysrq.c            |   19 ++++++++++++++++++-
 drivers/gpu/drm/drm_fb_helper.c |    1 +
 drivers/net/ibm_newemac/debug.c |    1 +
 include/linux/sysrq.h           |    1 +
 kernel/debug/debug_core.c       |    1 +
 kernel/power/poweroff.c         |    1 +
 9 files changed, 26 insertions(+), 1 deletion(-)
arch/arm/kernel/etm.c +1 -0
index 8277539..fb7c393 100644
@@ -240,6 +240,7 @@ static struct sysrq_key_op sysrq_etm_op = {
 	.handler = sysrq_etm_dump,
 	.help_msg = "ETM buffer dump",
 	.action_msg = "etm",
+	.module = THIS_MODULE,
 };
 
 static int etb_open(struct inode *inode, struct file *file)
arch/powerpc/xmon/xmon.c +1 -0
index 8bad7d5..b9b7aee 100644
@@ -2740,6 +2740,7 @@ static struct sysrq_key_op sysrq_xmon_op =
 	.handler =	sysrq_handle_xmon,
 	.help_msg =	"Xmon",
 	.action_msg =	"Entering xmon",
+	.module =	THIS_MODULE,
 };
 
 static int __init setup_xmon_sysrq(void)
arch/sparc/kernel/process_64.c +1 -0
index dbe81a3..f5a2efc 100644
@@ -312,6 +312,7 @@ static struct sysrq_key_op sparc_globalreg_op = {
 	.handler	= sysrq_handle_globreg,
 	.help_msg	= "Globalregs",
 	.action_msg	= "Show Global CPU Regs",
+	.module		= THIS_MODULE,
 };
 
 static int __init sparc_globreg_init(void)
drivers/char/sysrq.c +18 -1
index 878ac0c..3dd4034 100644
@@ -520,7 +520,24 @@ void __handle_sysrq(int key, struct tty_struct *tty, int check_mask)
 		if (!check_mask || sysrq_on_mask(op_p->enable_mask)) {
 			printk("%s\n", op_p->action_msg);
 			console_loglevel = orig_log_level;
-			op_p->handler(key, tty);
+
+			/*
+			 * We want to run the handler any time we can get
+			 * a reference on the module, or anytime we don't
+			 * have any module to get.  In both of these cases
+			 * its safe to do a module_put, as NULLS are skipped
+			 * there.
+			 */
+			if ((try_module_get(op_p->module) == 0) ||
+			    (!op_p->module)) {
+				spin_unlock_irqrestore(&sysrq_key_table_lock,
+						       flags);
+				op_p->handler(key, tty);
+				module_put(op_p->module);
+			} else
+				printk("Could not lock this sysrq key\n");
+
+			return;
 		} else {
 			printk("This sysrq operation is disabled.\n");
 		}
drivers/gpu/drm/drm_fb_helper.c +1 -0
index 7196620..623e701 100644
@@ -304,6 +304,7 @@ static struct sysrq_key_op sysrq_drm_fb_helper_restore_op = {
 	.handler = drm_fb_helper_sysrq,
 	.help_msg = "force-fb(V)",
 	.action_msg = "Restore framebuffer console",
+	.module	= THIS_MODULE,
 };
 #else
 static struct sysrq_key_op sysrq_drm_fb_helper_restore_op = { };
drivers/net/ibm_newemac/debug.c +1 -0
index 3995faf..b82aa35 100644
@@ -247,6 +247,7 @@ static struct sysrq_key_op emac_sysrq_op = {
 	.handler = emac_sysrq_handler,
 	.help_msg = "emaC",
 	.action_msg = "Show EMAC(s) status",
+	.module	= THIS_MODULE,
 };
 
 int __init emac_init_debug(void)
include/linux/sysrq.h +1 -0
index 609e8ca..4f219de 100644
@@ -35,6 +35,7 @@ struct sysrq_key_op {
 	char *help_msg;
 	char *action_msg;
 	int enable_mask;
+	struct module *module;
 };
 
 #ifdef CONFIG_MAGIC_SYSRQ
kernel/debug/debug_core.c +1 -0
index 8bc5eef..6b64063 100644
@@ -761,6 +761,7 @@ static struct sysrq_key_op sysrq_dbg_op = {
 	.handler	= sysrq_handle_dbg,
 	.help_msg	= "debug(G)",
 	.action_msg	= "DEBUG",
+	.module		= THIS_MODULE,
 };
 #endif
 
kernel/power/poweroff.c +1 -0
index e8b3370..58df039 100644
@@ -35,6 +35,7 @@ static struct sysrq_key_op	sysrq_poweroff_op = {
 	.help_msg       = "powerOff",
 	.action_msg     = "Power Off",
 	.enable_mask	= SYSRQ_ENABLE_BOOT,
+	.module		= THIS_MODULE,
 };
 
 static int pm_sysrq_init(void)
Quoted 8 lines
Signature · 6 lines
Dmitry
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#6 lore raw
Quoted 25 lines · Neil Horman
Quoted 14 lines · Dmitry Torokhov
Quoted 13 lines · Neil Horman
Quoted 12 lines · Dmitry Torokhov
Quoted 8 lines · Neil Horman

On Mon, Jul 26, 2010 at 05:54:02PM +0800, Xiaotian Feng wrote:
<snip>

This creates the possibility of a race in the handler. Not that it happens often, but sysrq keys can be registered and unregistered dynamically. If that lock isn't held while we call the keys handler, the code implementing that handler can live in a module that gets removed while its executing, leading to an oops, etc. I think the better solution would be to use an rcu lock here.

I'd simply changed spinlock to a mutex.

Quoted 9 lines
Quoted 5 lines

I don't think you can do that safely in this path, as sysrqs will be looked up in both process (echo t > /proc/sysrq-trigger) context and in interrupt (alt-sysrq-t) context. If a mutex is locked and you try to take it in interrupt context, you get a sleeping-in-interrupt panic IIRC

Yes, indeed. But then even RCU will not really help us since keyboard driver will have inpterrupts disabled anyways.

Quoted 7 lines

Hm, thats true. I suppose the right thing to do then is grab a reference on any sysrq implemented within code that might be considered transient before releasing the lock. I've not tested this patch out, but it should do what we need, in that it allows us to release the lock without having to worry about the op list changing underneath us, or having the module with the handler code dissappear

That would only help if you also offload execution to a workqueue (which may not be desirable in all cases) since keyboard driver^H^H input core still calls into SysRq code holding [another] spinlock with interrupts disabled.

Signature · 1 line
Dmitry
#7 lore raw
Quoted 34 lines · Dmitry Torokhov
Quoted 25 lines · Neil Horman
Quoted 14 lines · Dmitry Torokhov
Quoted 13 lines · Neil Horman
Quoted 12 lines · Dmitry Torokhov
Quoted 8 lines · Neil Horman

On Mon, Jul 26, 2010 at 05:54:02PM +0800, Xiaotian Feng wrote:
<snip>

This creates the possibility of a race in the handler. Not that it happens often, but sysrq keys can be registered and unregistered dynamically. If that lock isn't held while we call the keys handler, the code implementing that handler can live in a module that gets removed while its executing, leading to an oops, etc. I think the better solution would be to use an rcu lock here.

I'd simply changed spinlock to a mutex.

Quoted 9 lines
Quoted 5 lines

I don't think you can do that safely in this path, as sysrqs will be looked up in both process (echo t > /proc/sysrq-trigger) context and in interrupt (alt-sysrq-t) context. If a mutex is locked and you try to take it in interrupt context, you get a sleeping-in-interrupt panic IIRC

Yes, indeed. But then even RCU will not really help us since keyboard driver will have inpterrupts disabled anyways.

Quoted 7 lines

Hm, thats true. I suppose the right thing to do then is grab a reference on any sysrq implemented within code that might be considered transient before releasing the lock. I've not tested this patch out, but it should do what we need, in that it allows us to release the lock without having to worry about the op list changing underneath us, or having the module with the handler code dissappear

Quoted 5 lines

That would only help if you also offload execution to a workqueue (which may not be desirable in all cases) since keyboard driver^H^H input core still calls into SysRq code holding [another] spinlock with interrupts disabled.

Um, no, I don't think so. The concern that I had with the patch was that after you unlock that spinlock, a module which previously registered a sysrq handler could be removed during its execution leaving it executing in unknown memory. By doing a successful try_module_get we prevent the module remove code from deleting a module from the kernel, avoiding that condition until the execution of the requested sysrq handler completes. Offloading execution of the handler to a workqueue does nothing here, unless you see another problem, independent of the one I was addressing.

I suppose there is a possibiliy that the o_op value could change after we unlock the lock, but we could manage that by copying the pointer (although I don't think its needed unless some module tries to unregister sysrq handlers outside of the module_exit routine it has.

Neil

Quoted 3 lines
Signature · 1 line
Dmitry
#8 lore raw
In reply to Xiaotian Feng
Quoted 6 lines · On Mon, 26 Jul 2010 17:54:02 +0800 Xiaotian Feng <[email protected]> wrote:

sysrq_key_table_lock is used to protect the sysrq_key_table, make sure we get/replace the right operation for the sysrq. But in __handle_sysrq, kernel will hold this lock and disable irqs until we finished op_p->handler(). This may cause false positive watchdog alert when we're doing "show-task-states" on a system with many tasks.

It would be better to find a suitable point in an inner loop and add an appropriately-commented touch_nmi_watchdog().

That way the problem gets fixed for all irqs-off callers, not just one of them.