Skip to contentSkip to conversation

iommu/vt-d: fix race between free_irte() and get_irte()

PATCH In review Greg Edwards 2 participants 8 messages Jul 22, 2014 – Jul 31, 2014 IOMMU

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 iommu/vt-d: fix race between free_irte() and get_irte()

get_irte() can race with free_irte() and dereference a NULL iommu pointer.

---
 drivers/iommu/intel_irq_remapping.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/iommu/intel_irq_remapping.c +6 -0
index 9b17489..2d67e6d 100644
@@ -70,6 +70,12 @@ static int get_irte(int irq, struct irte *entry)
 
 	raw_spin_lock_irqsave(&irq_2_ir_lock, flags);
 
+	/* ensure we're not racing with free_irte() */
+	if (unlikely(!irq_iommu->iommu)) {
+		raw_spin_unlock_irqrestore(&irq_2_ir_lock, flags);
+		return -1;
+	}
+
 	index = irq_iommu->irte_index + irq_iommu->sub_handle;
 	*entry = *(irq_iommu->iommu->ir_table->base + index);
 
Signature · 1 line
1.9.3
maintainer
#1 lore raw
Quoted 2 lines · Greg Edwards

get_irte() can race with free_irte() and dereference a NULL iommu pointer.

Have you seen any real occurance of this race? Get_irte is called in the set_affinity path, how can that race with the irq being freed?

Joerg

#2 lore raw
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Greg Edwards

get_irte() can race with free_irte() and dereference a NULL iommu pointer.

Have you seen any real occurance of this race? Get_irte is called in the set_affinity path, how can that race with the irq being freed?

Yes, that's how we hit it. A process was setting the CPU affinity while QEMU was releasing the IRQ. We have a CI stress test that turned this up.

Greg

maintainer
#3 lore raw
Quoted 11 lines · Greg Edwards
Quoted 6 lines · Joerg Roedel
Quoted 2 lines · Greg Edwards

get_irte() can race with free_irte() and dereference a NULL iommu pointer.

Have you seen any real occurance of this race? Get_irte is called in the set_affinity path, how can that race with the irq being freed?

Yes, that's how we hit it. A process was setting the CPU affinity while QEMU was releasing the IRQ. We have a CI stress test that turned this up.

Can you update the commit message with the details of how this race can be triggered, ideally with a stack-trace of a real crash you triggered because of this issue?

Thanks,

Joerg

#4 lore raw

PATCH v2 1/1 iommu/vt-d: race setting IRQ CPU affinity while freeing IRQ

A user process setting the CPU affinity of an IRQ for a KVM direct-assigned device via /proc/irq/<IRQ#>/smp_affinity can race with the IRQ being released by QEMU, resulting in a NULL iommu pointer dereference in get_irte().

---

Dropped the Cc: for stable since this likely wouldn't ever be seen in the real world. We saw it on an automated CI stress test.

 drivers/iommu/intel_irq_remapping.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/iommu/intel_irq_remapping.c +5 -0
index 9b17489..d926676 100644
@@ -70,6 +70,11 @@ static int get_irte(int irq, struct irte *entry)
 
 	raw_spin_lock_irqsave(&irq_2_ir_lock, flags);
 
+	if (unlikely(!irq_iommu->iommu)) {
+		raw_spin_unlock_irqrestore(&irq_2_ir_lock, flags);
+		return -1;
+	}
+
 	index = irq_iommu->irte_index + irq_iommu->sub_handle;
 	*entry = *(irq_iommu->iommu->ir_table->base + index);
 
Signature · 1 line
1.9.3
maintainer
#5 lore raw

Re: [PATCH v2] iommu/vt-d: race setting IRQ CPU affinity while freeing IRQ

Quoted 4 lines · Greg Edwards

A user process setting the CPU affinity of an IRQ for a KVM direct-assigned device via /proc/irq/<IRQ#>/smp_affinity can race with the IRQ being released by QEMU, resulting in a NULL iommu pointer dereference in get_irte().

Maybe I wasn't clear enough, what I am missing is a panic message with a backtrace from the NULL pointer deref you are seeing in the commit message.

Also I am still wondering why it is possible to set affinity from userspace while the irq is about to be freed. Shouldn't the proc files are already unregistered when the irq is freed?

Joerg

#6 lore raw

Re: [PATCH v2] iommu/vt-d: race setting IRQ CPU affinity while freeing IRQ

Quoted 9 lines · Joerg Roedel
Quoted 4 lines · Greg Edwards

A user process setting the CPU affinity of an IRQ for a KVM direct-assigned device via /proc/irq/<IRQ#>/smp_affinity can race with the IRQ being released by QEMU, resulting in a NULL iommu pointer dereference in get_irte().

Maybe I wasn't clear enough, what I am missing is a panic message with a backtrace from the NULL pointer deref you are seeing in the commit message.

Sure, sorry I didn't send that along previously. We originally saw this on a 3.9 kernel, then recently on a 3.14 stable kernel. I hadn't reproduced it on current tip yet. Here it is for Linus' tree as of this morning:

[ 6638.327851] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
[ 6638.335955] IP: [<ffffffff8190a652>] intel_ioapic_set_affinity+0x82/0x1b0
[ 6638.343012] PGD 99172e067 PUD 1026979067 PMD 0 
[ 6638.347858] Oops: 0000 [#1] SMP 
[ 6638.351386] Modules linked in:
[ 6638.354718] CPU: 1 PID: 3354 Comm: affin Not tainted 3.16.0-rc7-00007-g31dab71 #1
[ 6638.362480] Hardware name: Supermicro SYS-F617R2-RT+/X9DRFR, BIOS 3.0a 01/29/2014
[ 6638.370259] task: ffff881025b0e720 ti: ffff88099173c000 task.ti: ffff88099173c000
[ 6638.378063] RIP: 0010:[<ffffffff8190a652>]  [<ffffffff8190a652>] intel_ioapic_set_affinity+0x82/0x1b0
[ 6638.387649] RSP: 0018:ffff88099173fdb0  EFLAGS: 00010046
[ 6638.393314] RAX: 0000000000000082 RBX: ffff880a36294600 RCX: 0000000000000082
[ 6638.400824] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8266af00
[ 6638.408341] RBP: ffff88099173fdf8 R08: 0000000000000000 R09: ffff88103ec00490
[ 6638.415867] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88099173fe90
[ 6638.423402] R13: 000000000000005f R14: ffff880faa38fe80 R15: ffff880faa38fe80
[ 6638.430942] FS:  00007f7161f05740(0000) GS:ffff88107fc40000(0000) knlGS:0000000000000000
[ 6638.439456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6638.445640] CR2: 0000000000000090 CR3: 000000099140d000 CR4: 00000000001427e0
[ 6638.453227] Stack:
[ 6638.455699]  ffffffff81c44740 ffff88099173fdc8 ffffffff00000000 00000000c991fd3b
[ 6638.463660]  ffff880a36294600 ffff88099173fe90 ffff88099173fe90 0000000000000000
[ 6638.471623]  0000000000000286 ffff88099173fe08 ffffffff8190aac5 ffff88099173fe28
[ 6638.479605] Call Trace:
[ 6638.482562]  [<ffffffff8190aac5>] set_remapped_irq_affinity+0x25/0x40
[ 6638.489534]  [<ffffffff811322dc>] irq_do_set_affinity+0x1c/0x50
[ 6638.495985]  [<ffffffff81132458>] irq_set_affinity_locked+0x98/0xd0
[ 6638.502781]  [<ffffffff811324d6>] __irq_set_affinity+0x46/0x70
[ 6638.509142]  [<ffffffff811362dc>] write_irq_affinity.isra.6+0xdc/0x100
[ 6638.516198]  [<ffffffff8113631c>] irq_affinity_list_proc_write+0x1c/0x20
[ 6638.523440]  [<ffffffff8129f30d>] proc_reg_write+0x3d/0x80
[ 6638.529474]  [<ffffffff812384a7>] vfs_write+0xb7/0x1f0
[ 6638.535165]  [<ffffffff81243619>] ? putname+0x29/0x40
[ 6638.540771]  [<ffffffff812390c5>] SyS_write+0x55/0xd0
[ 6638.546386]  [<ffffffff81adc729>] system_call_fastpath+0x16/0x1b
[ 6638.552970] Code: ff 48 85 d2 74 68 4c 8b 7a 30 4d 85 ff 74 5f 48 c7 c7 00 af 66 82 e8 9e 1b 1d 00 49 8b 57 20 41 0f b7 77 28 48 c7 c7 00 af 66 82 <48> 8b 8a 90 00 00 00 41 0f b7 57 2a 01 f2 48 89 c6 48 63 d2 48 
[ 6638.574199] RIP  [<ffffffff8190a652>] intel_ioapic_set_affinity+0x82/0x1b0
[ 6638.581735]  RSP <ffff88099173fdb0>
[ 6638.585872] CR2: 0000000000000090
Quoted 3 lines

Also I am still wondering why it is possible to set affinity from userspace while the irq is about to be freed. Shouldn't the proc files are already unregistered when the irq is freed?

The QEMU thread is doing a KVM_DEASSIGN_DEV_IRQ ioctl. It looks like the call chain is:

kvm_vm_ioctl_assigned_device
kvm_vm_ioctl_deassign_dev_irq
kvm_deassign_irq
deassign_host_irq
pci_disable_msix
free_msi_irqs
arch_teardown_msi_irqs
default_teardown_msi_irqs
arch_teardown_msi_irq
native_teardown_msi_irq
irq_free_hwirq
irq_free_hwirqs

irq_free_hwirqs() does:

460 for (i = from, j = cnt; j > 0; i++, j--) {
461 irq_set_status_flags(i, _IRQ_NOREQUEST | _IRQ_NOPROBE);
462 arch_teardown_hwirq(i); <----------- QEMU is down this path to free_irte()
463 }
464 irq_free_descs(from, cnt); <------------ proc files unregistered down this path

I guess the question is if it would be safe to change that ordering? I don't know.

Greg

maintainer
#7 lore raw

Re: [PATCH v2] iommu/vt-d: race setting IRQ CPU affinity while freeing IRQ

Quoted 30 lines · Greg Edwards
[ 6638.327851] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
[ 6638.335955] IP: [<ffffffff8190a652>] intel_ioapic_set_affinity+0x82/0x1b0
[ 6638.343012] PGD 99172e067 PUD 1026979067 PMD 0 
[ 6638.347858] Oops: 0000 [#1] SMP 
[ 6638.351386] Modules linked in:
[ 6638.354718] CPU: 1 PID: 3354 Comm: affin Not tainted 3.16.0-rc7-00007-g31dab71 #1
[ 6638.362480] Hardware name: Supermicro SYS-F617R2-RT+/X9DRFR, BIOS 3.0a 01/29/2014
[ 6638.370259] task: ffff881025b0e720 ti: ffff88099173c000 task.ti: ffff88099173c000
[ 6638.378063] RIP: 0010:[<ffffffff8190a652>]  [<ffffffff8190a652>] intel_ioapic_set_affinity+0x82/0x1b0
[ 6638.387649] RSP: 0018:ffff88099173fdb0  EFLAGS: 00010046
[ 6638.393314] RAX: 0000000000000082 RBX: ffff880a36294600 RCX: 0000000000000082
[ 6638.400824] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8266af00
[ 6638.408341] RBP: ffff88099173fdf8 R08: 0000000000000000 R09: ffff88103ec00490
[ 6638.415867] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88099173fe90
[ 6638.423402] R13: 000000000000005f R14: ffff880faa38fe80 R15: ffff880faa38fe80
[ 6638.430942] FS:  00007f7161f05740(0000) GS:ffff88107fc40000(0000) knlGS:0000000000000000
[ 6638.439456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6638.445640] CR2: 0000000000000090 CR3: 000000099140d000 CR4: 00000000001427e0
[ 6638.453227] Stack:
[ 6638.455699]  ffffffff81c44740 ffff88099173fdc8 ffffffff00000000 00000000c991fd3b
[ 6638.463660]  ffff880a36294600 ffff88099173fe90 ffff88099173fe90 0000000000000000
[ 6638.471623]  0000000000000286 ffff88099173fe08 ffffffff8190aac5 ffff88099173fe28
[ 6638.479605] Call Trace:
[ 6638.482562]  [<ffffffff8190aac5>] set_remapped_irq_affinity+0x25/0x40
[ 6638.489534]  [<ffffffff811322dc>] irq_do_set_affinity+0x1c/0x50
[ 6638.495985]  [<ffffffff81132458>] irq_set_affinity_locked+0x98/0xd0
[ 6638.502781]  [<ffffffff811324d6>] __irq_set_affinity+0x46/0x70
[ 6638.509142]  [<ffffffff811362dc>] write_irq_affinity.isra.6+0xdc/0x100
[ 6638.516198]  [<ffffffff8113631c>] irq_affinity_list_proc_write+0x1c/0x20
[ 6638.523440]  [<ffffffff8129f30d>] proc_reg_write+0x3d/0x80
Quoted 8 lines
[ 6638.529474]  [<ffffffff812384a7>] vfs_write+0xb7/0x1f0
[ 6638.535165]  [<ffffffff81243619>] ? putname+0x29/0x40
[ 6638.540771]  [<ffffffff812390c5>] SyS_write+0x55/0xd0
[ 6638.546386]  [<ffffffff81adc729>] system_call_fastpath+0x16/0x1b
[ 6638.552970] Code: ff 48 85 d2 74 68 4c 8b 7a 30 4d 85 ff 74 5f 48 c7 c7 00 af 66 82 e8 9e 1b 1d 00 49 8b 57 20 41 0f b7 77 28 48 c7 c7 00 af 66 82 <48> 8b 8a 90 00 00 00 41 0f b7 57 2a 01 f2 48 89 c6 48 63 d2 48 
[ 6638.574199] RIP  [<ffffffff8190a652>] intel_ioapic_set_affinity+0x82/0x1b0
[ 6638.581735]  RSP <ffff88099173fdb0>
[ 6638.585872] CR2: 0000000000000090

Okay, I added this to the commit message and applied the patch.

Quoted 16 lines

The QEMU thread is doing a KVM_DEASSIGN_DEV_IRQ ioctl. It looks like the call chain is:

kvm_vm_ioctl_assigned_device
kvm_vm_ioctl_deassign_dev_irq
kvm_deassign_irq
deassign_host_irq
pci_disable_msix
free_msi_irqs
arch_teardown_msi_irqs
default_teardown_msi_irqs
arch_teardown_msi_irq
native_teardown_msi_irq
irq_free_hwirq
irq_free_hwirqs

Quoted 7 lines

irq_free_hwirqs() does:

460 for (i = from, j = cnt; j > 0; i++, j--) {
461 irq_set_status_flags(i, _IRQ_NOREQUEST | _IRQ_NOPROBE);
462 arch_teardown_hwirq(i); <----------- QEMU is down this path to free_irte()
463 }
464 irq_free_descs(from, cnt); <------------ proc files unregistered down this path

I see, so it makes sense to fix it in the driver for now.

Joerg