Skip to contentSkip to conversation

ipmi: fix potential deadlock on &kcs_bmc->lock

PATCH In review Chengfeng Ye 3 participants 7 messages Jun 27, 2023 – Jul 19, 2023 IPMI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ipmi: fix potential deadlock on &kcs_bmc->lock

As kcs_bmc_handle_event() is executed inside both a timer and a hardirq, it should disable irq before lock acquisition otherwise deadlock could happen if the timmer is preemtped by the irq.

Possible deadlock scenario:
aspeed_kcs_check_obe() (timer)

    -> kcs_bmc_handle_event()
    -> spin_lock(&kcs_bmc->lock)
        <irq interruption>
        -> aspeed_kcs_irq()
        -> kcs_bmc_handle_event()
        -> spin_lock(&kcs_bmc->lock) (deadlock here)

This flaw was found using an experimental static analysis tool we are developing for irq-related deadlock.

The tentative patch fix the potential deadlock by spin_lock_irqsave()

---
 drivers/char/ipmi/kcs_bmc.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/char/ipmi/kcs_bmc.c +3 -2
index 03d02a848f3a..8b1161d5194a 100644
@@ -56,12 +56,13 @@ irqreturn_t kcs_bmc_handle_event(struct kcs_bmc_device *kcs_bmc)
 {
 	struct kcs_bmc_client *client;
 	irqreturn_t rc = IRQ_NONE;
+	unsigned long flags;
 
-	spin_lock(&kcs_bmc->lock);
+	spin_lock_irqsave(&kcs_bmc->lock, flags);
 	client = kcs_bmc->client;
 	if (client)
 		rc = client->ops->event(client);
-	spin_unlock(&kcs_bmc->lock);
+	spin_unlock_irqrestore(&kcs_bmc->lock, flags);
 
 	return rc;
 }
Signature · 1 line
2.17.1
#1 lore raw

Indeed, this looks like an issue.

Andrew, any opinions on this? The attached patch will work, the other option would be to disable interrupts when calling kcs_bmc_handle_event() in the timer handler. But then you have to worry about RT.

-corey

Quoted 46 lines · Chengfeng Ye

As kcs_bmc_handle_event() is executed inside both a timer and a hardirq, it should disable irq before lock acquisition otherwise deadlock could happen if the timmer is preemtped by the irq.

Possible deadlock scenario:
aspeed_kcs_check_obe() (timer)

    -> kcs_bmc_handle_event()
    -> spin_lock(&kcs_bmc->lock)
        <irq interruption>
        -> aspeed_kcs_irq()
        -> kcs_bmc_handle_event()
        -> spin_lock(&kcs_bmc->lock) (deadlock here)

This flaw was found using an experimental static analysis tool we are developing for irq-related deadlock.

The tentative patch fix the potential deadlock by spin_lock_irqsave()

---
 drivers/char/ipmi/kcs_bmc.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/char/ipmi/kcs_bmc.c +3 -2
index 03d02a848f3a..8b1161d5194a 100644
@@ -56,12 +56,13 @@ irqreturn_t kcs_bmc_handle_event(struct kcs_bmc_device *kcs_bmc)
 {
 	struct kcs_bmc_client *client;
 	irqreturn_t rc = IRQ_NONE;
+	unsigned long flags;
 
-	spin_lock(&kcs_bmc->lock);
+	spin_lock_irqsave(&kcs_bmc->lock, flags);
 	client = kcs_bmc->client;
 	if (client)
 		rc = client->ops->event(client);
-	spin_unlock(&kcs_bmc->lock);
+	spin_unlock_irqrestore(&kcs_bmc->lock, flags);
 
 	return rc;
 }
Signature · 1 line
2.17.1
#2 lore raw

Hi Corey, Chengfeng,

Quoted 6 lines · Corey Minyard

Indeed, this looks like an issue.

Andrew, any opinions on this? The attached patch will work, the other option would be to disable interrupts when calling kcs_bmc_handle_event() in the timer handler. But then you have to worry about RT.

Right, I think we'd do best to not over-complicate things. spin_lock_irq{save,restore}() are the intuitive choice to me.

I'll follow up with R-b/T-b tags once I've booted the patch and done some testing.

Andrew

Quoted 50 lines

-corey

Quoted 46 lines · Chengfeng Ye

As kcs_bmc_handle_event() is executed inside both a timer and a hardirq, it should disable irq before lock acquisition otherwise deadlock could happen if the timmer is preemtped by the irq.

Possible deadlock scenario:
aspeed_kcs_check_obe() (timer)

    -> kcs_bmc_handle_event()
    -> spin_lock(&kcs_bmc->lock)
        <irq interruption>
        -> aspeed_kcs_irq()
        -> kcs_bmc_handle_event()
        -> spin_lock(&kcs_bmc->lock) (deadlock here)

This flaw was found using an experimental static analysis tool we are developing for irq-related deadlock.

The tentative patch fix the potential deadlock by spin_lock_irqsave()

---
 drivers/char/ipmi/kcs_bmc.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/char/ipmi/kcs_bmc.c +3 -2
index 03d02a848f3a..8b1161d5194a 100644
@@ -56,12 +56,13 @@ irqreturn_t kcs_bmc_handle_event(struct kcs_bmc_device *kcs_bmc)
 {
 	struct kcs_bmc_client *client;
 	irqreturn_t rc = IRQ_NONE;
+	unsigned long flags;
 
-	spin_lock(&kcs_bmc->lock);
+	spin_lock_irqsave(&kcs_bmc->lock, flags);
 	client = kcs_bmc->client;
 	if (client)
 		rc = client->ops->event(client);
-	spin_unlock(&kcs_bmc->lock);
+	spin_unlock_irqrestore(&kcs_bmc->lock, flags);
 
 	return rc;
 }
Signature · 1 line
2.17.1
#3 lore raw
Quoted 10 lines · Andrew Jeffery

Hi Corey, Chengfeng,

Quoted 6 lines · Corey Minyard

Indeed, this looks like an issue.

Andrew, any opinions on this? The attached patch will work, the other option would be to disable interrupts when calling kcs_bmc_handle_event() in the timer handler. But then you have to worry about RT.

Quoted 5 lines

Right, I think we'd do best to not over-complicate things. spin_lock_irq{save,restore}() are the intuitive choice to me.

I'll follow up with R-b/T-b tags once I've booted the patch and done some testing.

Thanks. This is in my for-next tree, I'd like to get this in the merge window, which I believe ends this Sunday.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This flaw was found using an experimental static analysis tool we are developing for irq-related deadlock.

Will this tool be available for general use? It's obviously quite handy.

-corey

#4 lore raw

Thanks so much for the reply!

Quoted 2 lines

Will this tool be available for general use? It's obviously quite handy.

And also thanks for your interest! I am still optimizing the tool, so far it still can report some false positives in some cases and require a certain effort of manual checking. Would let you know once I finish my work and open-source it later!

Best Regards,
Chengfeng

Reviewed-by Tested-by
#5 lore raw
In reply to Corey Minyard
Quoted 11 lines · Corey Minyard
Quoted 10 lines · Andrew Jeffery

Hi Corey, Chengfeng,

Quoted 6 lines · Corey Minyard

Indeed, this looks like an issue.

Andrew, any opinions on this? The attached patch will work, the other option would be to disable interrupts when calling kcs_bmc_handle_event() in the timer handler. But then you have to worry about RT.

Quoted 9 lines
Quoted 5 lines

Right, I think we'd do best to not over-complicate things. spin_lock_irq{save,restore}() are the intuitive choice to me.

I'll follow up with R-b/T-b tags once I've booted the patch and done some testing.

Thanks. This is in my for-next tree, I'd like to get this in the merge window, which I believe ends this Sunday.

I've successfully booted an IBM Rainier machine with this patch applied. Rainier is a Power10-based platform managed by an AST2600 BMC. The boot process makes heavy use of one of the KCS devices as part of an MCTP transport binding implementation between the BMC and host firmware.

Thanks,

Andrew