Skip to contentSkip to conversation

vhost-vdpa: Fix invalid irq bypass unregister

RFC PATCH In review Dragos Tatulea 2 participants 10 messages Aug 1, 2024 – Aug 8, 2024

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 vhost-vdpa: Fix invalid irq bypass unregister

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.
3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().
4) The original vq tries to unregister it's producer which it has already unlinked in step 1. irq_bypass_unregister_producer() will go ahead and unlink the producer once again. That happens because:
a) The producer has a token.
b) An element with that token is found. But that element comes from step 3.

I see 3 ways to fix this:
1) Fix the vhost-vdpa part. What this patch does. vfio has a different workflow.
2) Set the token to NULL directly in irq_bypass_unregister_producer() after unlinking the producer. But that makes the API asymmetrical.
3) Make irq_bypass_unregister_producer() also compare the pointer elements not just the tokens and do the unlink only on match.

Any thoughts?

Oops: general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP
CPU: 8 PID: 5190 Comm: qemu-system-x86 Not tainted 6.10.0-rc7+ #6
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:irq_bypass_unregister_producer+0xa5/0xd0
RSP: 0018:ffffc900034d7e50 EFLAGS: 00010246
RAX: dead000000000122 RBX: ffff888353d12718 RCX: ffff88810336a000
RDX: dead000000000100 RSI: ffffffff829243a0 RDI: 0000000000000000
RBP: ffff888353c42000 R08: ffff888104882738 R09: ffff88810336a000
R10: ffff888448ab2050 R11: 0000000000000000 R12: ffff888353d126a0
R13: 0000000000000004 R14: 0000000000000055 R15: 0000000000000004
FS:  00007f9df9403c80(0000) GS:ffff88852cc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562dffc6b568 CR3: 000000012efbb006 CR4: 0000000000772ef0
PKRU: 55555554
Call Trace:
 <TASK>
 ? die_addr+0x36/0x90
 ? exc_general_protection+0x1a8/0x390
 ? asm_exc_general_protection+0x26/0x30
 ? irq_bypass_unregister_producer+0xa5/0xd0
 vhost_vdpa_setup_vq_irq+0x5a/0xc0 [vhost_vdpa]
 vhost_vdpa_unlocked_ioctl+0xdcd/0xe00 [vhost_vdpa]
 ? vhost_vdpa_config_cb+0x30/0x30 [vhost_vdpa]
 __x64_sys_ioctl+0x90/0xc0
 do_syscall_64+0x4f/0x110
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f9df930774f
RSP: 002b:00007ffc55013080 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000562dfe134d20 RCX: 00007f9df930774f
RDX: 00007ffc55013200 RSI: 000000004008af21 RDI: 0000000000000011
RBP: 00007ffc55013200 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000562dfe134360
R13: 0000562dfe134d20 R14: 0000000000000000 R15: 00007f9df801e190
---
 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +1 -0
index 478cd46a49ed..d4a7a3918d86 100644
@@ -226,6 +226,7 @@ static void vhost_vdpa_unsetup_vq_irq(struct vhost_vdpa *v, u16 qid)
 	struct vhost_virtqueue *vq = &v->vqs[qid];
 
 	irq_bypass_unregister_producer(&vq->call_ctx.producer);
+	vq->call_ctx.producer.token = NULL;
 }
 
 static int _compat_vdpa_reset(struct vhost_vdpa *v)
Signature · 1 line
2.45.2
#1 lore raw
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Quoted 69 lines

4) The original vq tries to unregister it's producer which it has already unlinked in step 1. irq_bypass_unregister_producer() will go ahead and unlink the producer once again. That happens because:
a) The producer has a token.
b) An element with that token is found. But that element comes from step 3.

I see 3 ways to fix this:
1) Fix the vhost-vdpa part. What this patch does. vfio has a different workflow.
2) Set the token to NULL directly in irq_bypass_unregister_producer() after unlinking the producer. But that makes the API asymmetrical.
3) Make irq_bypass_unregister_producer() also compare the pointer elements not just the tokens and do the unlink only on match.

Any thoughts?

Oops: general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP
CPU: 8 PID: 5190 Comm: qemu-system-x86 Not tainted 6.10.0-rc7+ #6
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:irq_bypass_unregister_producer+0xa5/0xd0
RSP: 0018:ffffc900034d7e50 EFLAGS: 00010246
RAX: dead000000000122 RBX: ffff888353d12718 RCX: ffff88810336a000
RDX: dead000000000100 RSI: ffffffff829243a0 RDI: 0000000000000000
RBP: ffff888353c42000 R08: ffff888104882738 R09: ffff88810336a000
R10: ffff888448ab2050 R11: 0000000000000000 R12: ffff888353d126a0
R13: 0000000000000004 R14: 0000000000000055 R15: 0000000000000004
FS:  00007f9df9403c80(0000) GS:ffff88852cc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562dffc6b568 CR3: 000000012efbb006 CR4: 0000000000772ef0
PKRU: 55555554
Call Trace:
 <TASK>
 ? die_addr+0x36/0x90
 ? exc_general_protection+0x1a8/0x390
 ? asm_exc_general_protection+0x26/0x30
 ? irq_bypass_unregister_producer+0xa5/0xd0
 vhost_vdpa_setup_vq_irq+0x5a/0xc0 [vhost_vdpa]
 vhost_vdpa_unlocked_ioctl+0xdcd/0xe00 [vhost_vdpa]
 ? vhost_vdpa_config_cb+0x30/0x30 [vhost_vdpa]
 __x64_sys_ioctl+0x90/0xc0
 do_syscall_64+0x4f/0x110
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f9df930774f
RSP: 002b:00007ffc55013080 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000562dfe134d20 RCX: 00007f9df930774f
RDX: 00007ffc55013200 RSI: 000000004008af21 RDI: 0000000000000011
RBP: 00007ffc55013200 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000562dfe134360
R13: 0000562dfe134d20 R14: 0000000000000000 R15: 00007f9df801e190
---
 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +1 -1
index 478cd46a49ed..d4a7a3918d86 100644
@@ -226,6 +226,7 @@ static void vhost_vdpa_unsetup_vq_irq(struct vhost_vdpa *v, u16 qid)
        struct vhost_virtqueue *vq = &v->vqs[qid];
 
        irq_bypass_unregister_producer(&vq->call_ctx.producer);
+       vq->call_ctx.producer.token = NULL;
 }
 
 static int _compat_vdpa_reset(struct vhost_vdpa *v)
--
Quoted 2 lines

2.45.2

Thanks

#2 lore raw
Quoted 8 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Quoted 9 lines

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Quoted 63 lines
Quoted 63 lines

4) The original vq tries to unregister it's producer which it has already unlinked in step 1. irq_bypass_unregister_producer() will go ahead and unlink the producer once again. That happens because:
a) The producer has a token.
b) An element with that token is found. But that element comes from step 3.

I see 3 ways to fix this:
1) Fix the vhost-vdpa part. What this patch does. vfio has a different workflow.
2) Set the token to NULL directly in irq_bypass_unregister_producer() after unlinking the producer. But that makes the API asymmetrical.
3) Make irq_bypass_unregister_producer() also compare the pointer elements not just the tokens and do the unlink only on match.

Any thoughts?

Oops: general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP
CPU: 8 PID: 5190 Comm: qemu-system-x86 Not tainted 6.10.0-rc7+ #6
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:irq_bypass_unregister_producer+0xa5/0xd0
RSP: 0018:ffffc900034d7e50 EFLAGS: 00010246
RAX: dead000000000122 RBX: ffff888353d12718 RCX: ffff88810336a000
RDX: dead000000000100 RSI: ffffffff829243a0 RDI: 0000000000000000
RBP: ffff888353c42000 R08: ffff888104882738 R09: ffff88810336a000
R10: ffff888448ab2050 R11: 0000000000000000 R12: ffff888353d126a0
R13: 0000000000000004 R14: 0000000000000055 R15: 0000000000000004
FS:  00007f9df9403c80(0000) GS:ffff88852cc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562dffc6b568 CR3: 000000012efbb006 CR4: 0000000000772ef0
PKRU: 55555554
Call Trace:
 <TASK>
 ? die_addr+0x36/0x90
 ? exc_general_protection+0x1a8/0x390
 ? asm_exc_general_protection+0x26/0x30
 ? irq_bypass_unregister_producer+0xa5/0xd0
 vhost_vdpa_setup_vq_irq+0x5a/0xc0 [vhost_vdpa]
 vhost_vdpa_unlocked_ioctl+0xdcd/0xe00 [vhost_vdpa]
 ? vhost_vdpa_config_cb+0x30/0x30 [vhost_vdpa]
 __x64_sys_ioctl+0x90/0xc0
 do_syscall_64+0x4f/0x110
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f9df930774f
RSP: 002b:00007ffc55013080 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000562dfe134d20 RCX: 00007f9df930774f
RDX: 00007ffc55013200 RSI: 000000004008af21 RDI: 0000000000000011
RBP: 00007ffc55013200 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000562dfe134360
R13: 0000562dfe134d20 R14: 0000000000000000 R15: 00007f9df801e190
---
 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +0 -0
index 478cd46a49ed..d4a7a3918d86 100644
@@ -226,6 +226,7 @@ static void vhost_vdpa_unsetup_vq_irq(struct vhost_vdpa *v, u16 qid)
        struct vhost_virtqueue *vq = &v->vqs[qid];
 
Quoted 9 lines
Quoted 8 lines
drivers/vhost/vdpa.c +1 -1
        irq_bypass_unregister_producer(&vq->call_ctx.producer);
+       vq->call_ctx.producer.token = NULL;
 }
 
 static int _compat_vdpa_reset(struct vhost_vdpa *v)
--

2.45.2

Thanks

#3 lore raw
Quoted 17 lines · Dragos Tatulea
Quoted 15 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Quoted 7 lines
Quoted 2 lines

Or did you mean when a single eventfd is shared among different vqs?

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Just to make sure I understand the issue. The eventfd_ctx should be still valid until a new VHOST_SET_VRING_CALL().

I may miss something but the only way to assign exactly the same eventfd_ctx value to another vq is where the guest tries to share the MSI-X vector among virtqueues, then qemu will use a single eventfd as the callback for multiple virtqueues. If this is true:

For bypass registering, only the first registering can succeed as the following registering will fail because the irq bypass manager already had exactly the same producer token.
For registering, all unregistering can succeed:

1) the first unregistering will do the real job that unregister the token
2) the following unregistering will do nothing by iterating the producer token list without finding a match one

Maybe you can show me the userspace behaviour (ioctls) when you see this?

Thanks

Quoted 75 lines
Quoted 64 lines
Quoted 64 lines

4) The original vq tries to unregister it's producer which it has already unlinked in step 1. irq_bypass_unregister_producer() will go ahead and unlink the producer once again. That happens because:
a) The producer has a token.
b) An element with that token is found. But that element comes from step 3.

I see 3 ways to fix this:
1) Fix the vhost-vdpa part. What this patch does. vfio has a different workflow.
2) Set the token to NULL directly in irq_bypass_unregister_producer() after unlinking the producer. But that makes the API asymmetrical.
3) Make irq_bypass_unregister_producer() also compare the pointer elements not just the tokens and do the unlink only on match.

Any thoughts?

Oops: general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP
CPU: 8 PID: 5190 Comm: qemu-system-x86 Not tainted 6.10.0-rc7+ #6
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:irq_bypass_unregister_producer+0xa5/0xd0
RSP: 0018:ffffc900034d7e50 EFLAGS: 00010246
RAX: dead000000000122 RBX: ffff888353d12718 RCX: ffff88810336a000
RDX: dead000000000100 RSI: ffffffff829243a0 RDI: 0000000000000000
RBP: ffff888353c42000 R08: ffff888104882738 R09: ffff88810336a000
R10: ffff888448ab2050 R11: 0000000000000000 R12: ffff888353d126a0
R13: 0000000000000004 R14: 0000000000000055 R15: 0000000000000004
FS:  00007f9df9403c80(0000) GS:ffff88852cc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562dffc6b568 CR3: 000000012efbb006 CR4: 0000000000772ef0
PKRU: 55555554
Call Trace:
 <TASK>
 ? die_addr+0x36/0x90
 ? exc_general_protection+0x1a8/0x390
 ? asm_exc_general_protection+0x26/0x30
 ? irq_bypass_unregister_producer+0xa5/0xd0
 vhost_vdpa_setup_vq_irq+0x5a/0xc0 [vhost_vdpa]
 vhost_vdpa_unlocked_ioctl+0xdcd/0xe00 [vhost_vdpa]
 ? vhost_vdpa_config_cb+0x30/0x30 [vhost_vdpa]
 __x64_sys_ioctl+0x90/0xc0
 do_syscall_64+0x4f/0x110
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f9df930774f
RSP: 002b:00007ffc55013080 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000562dfe134d20 RCX: 00007f9df930774f
RDX: 00007ffc55013200 RSI: 000000004008af21 RDI: 0000000000000011
RBP: 00007ffc55013200 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000562dfe134360
R13: 0000562dfe134d20 R14: 0000000000000000 R15: 00007f9df801e190
---
 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +0 -0
index 478cd46a49ed..d4a7a3918d86 100644
@@ -226,6 +226,7 @@ static void vhost_vdpa_unsetup_vq_irq(struct vhost_vdpa *v, u16 qid)
        struct vhost_virtqueue *vq = &v->vqs[qid];
 
Quoted 9 lines
Quoted 8 lines
drivers/vhost/vdpa.c +1 -1
        irq_bypass_unregister_producer(&vq->call_ctx.producer);
+       vq->call_ctx.producer.token = NULL;
 }
 
 static int _compat_vdpa_reset(struct vhost_vdpa *v)
--

2.45.2

Thanks

#4 lore raw
Quoted 20 lines · Jason Wang
Quoted 19 lines · Dragos Tatulea
Quoted 17 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Quoted 9 lines
Quoted 5 lines

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Just to make sure I understand the issue. The eventfd_ctx should be still valid until a new VHOST_SET_VRING_CALL().

I think it's not about the validity of the eventfd_ctx. More about the lingering ctx value of the producer after vhost_vdpa_unsetup_vq_irq(). That value is the eventfd ctx, but it could be anything else really...

Quoted 5 lines

I may miss something but the only way to assign exactly the same eventfd_ctx value to another vq is where the guest tries to share the MSI-X vector among virtqueues, then qemu will use a single eventfd as the callback for multiple virtqueues. If this is true:

I don't think this is the case. I see the issue happening when running qemu vdpa live migration tests on the same host. From a vdpa device it's basically a device starting on a VM over and over.

Quoted 11 lines

For bypass registering, only the first registering can succeed as the following registering will fail because the irq bypass manager already had exactly the same producer token.
For registering, all unregistering can succeed:

1) the first unregistering will do the real job that unregister the token
2) the following unregistering will do nothing by iterating the producer token list without finding a match one

Maybe you can show me the userspace behaviour (ioctls) when you see this?

Sure, what would you need? qemu traces?

Thanks,
Dragos

Quoted 78 lines

Thanks

Quoted 75 lines
Quoted 64 lines
Quoted 64 lines

4) The original vq tries to unregister it's producer which it has already unlinked in step 1. irq_bypass_unregister_producer() will go ahead and unlink the producer once again. That happens because:
a) The producer has a token.
b) An element with that token is found. But that element comes from step 3.

I see 3 ways to fix this:
1) Fix the vhost-vdpa part. What this patch does. vfio has a different workflow.
2) Set the token to NULL directly in irq_bypass_unregister_producer() after unlinking the producer. But that makes the API asymmetrical.
3) Make irq_bypass_unregister_producer() also compare the pointer elements not just the tokens and do the unlink only on match.

Any thoughts?

Oops: general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP
CPU: 8 PID: 5190 Comm: qemu-system-x86 Not tainted 6.10.0-rc7+ #6
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:irq_bypass_unregister_producer+0xa5/0xd0
RSP: 0018:ffffc900034d7e50 EFLAGS: 00010246
RAX: dead000000000122 RBX: ffff888353d12718 RCX: ffff88810336a000
RDX: dead000000000100 RSI: ffffffff829243a0 RDI: 0000000000000000
RBP: ffff888353c42000 R08: ffff888104882738 R09: ffff88810336a000
R10: ffff888448ab2050 R11: 0000000000000000 R12: ffff888353d126a0
R13: 0000000000000004 R14: 0000000000000055 R15: 0000000000000004
FS:  00007f9df9403c80(0000) GS:ffff88852cc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562dffc6b568 CR3: 000000012efbb006 CR4: 0000000000772ef0
PKRU: 55555554
Call Trace:
 <TASK>
 ? die_addr+0x36/0x90
 ? exc_general_protection+0x1a8/0x390
 ? asm_exc_general_protection+0x26/0x30
 ? irq_bypass_unregister_producer+0xa5/0xd0
 vhost_vdpa_setup_vq_irq+0x5a/0xc0 [vhost_vdpa]
 vhost_vdpa_unlocked_ioctl+0xdcd/0xe00 [vhost_vdpa]
 ? vhost_vdpa_config_cb+0x30/0x30 [vhost_vdpa]
 __x64_sys_ioctl+0x90/0xc0
 do_syscall_64+0x4f/0x110
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f9df930774f
RSP: 002b:00007ffc55013080 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000562dfe134d20 RCX: 00007f9df930774f
RDX: 00007ffc55013200 RSI: 000000004008af21 RDI: 0000000000000011
RBP: 00007ffc55013200 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000562dfe134360
R13: 0000562dfe134d20 R14: 0000000000000000 R15: 00007f9df801e190
---
 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +0 -0
index 478cd46a49ed..d4a7a3918d86 100644
@@ -226,6 +226,7 @@ static void vhost_vdpa_unsetup_vq_irq(struct vhost_vdpa *v, u16 qid)
        struct vhost_virtqueue *vq = &v->vqs[qid];
 
Quoted 9 lines
Quoted 8 lines
drivers/vhost/vdpa.c +1 -1
        irq_bypass_unregister_producer(&vq->call_ctx.producer);
+       vq->call_ctx.producer.token = NULL;
 }
 
 static int _compat_vdpa_reset(struct vhost_vdpa *v)
--

2.45.2

Thanks

#5 lore raw
Quoted 28 lines · Dragos Tatulea
Quoted 26 lines · Jason Wang
Quoted 24 lines · Dragos Tatulea
Quoted 8 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Quoted 9 lines

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Quoted 5 lines
Quoted 3 lines

Just to make sure I understand the issue. The eventfd_ctx should be still valid until a new VHOST_SET_VRING_CALL().

I think it's not about the validity of the eventfd_ctx. More about the lingering ctx value of the producer after vhost_vdpa_unsetup_vq_irq().

Probably, but

Quoted 1 line

That value is the eventfd ctx, but it could be anything else really...

I mean we hold a refcnt of the eventfd so it should be valid until the next set_vring_call() or vhost_dev_cleanup().

But I do spot some possible issue:

1) We swap and assign new ctx in vhost_vring_ioctl():

                swap(ctx, vq->call_ctx.ctx);

2) and old ctx will be put there as well:

                if (!IS_ERR_OR_NULL(ctx))
                        eventfd_ctx_put(ctx);

3) but in vdpa, we try to unregister the producer with the new token:

static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
void __user *argp)
{
...

        r = vhost_vring_ioctl(&v->vdev, cmd, argp);
...
        switch (cmd) {
...
        case VHOST_SET_VRING_CALL:
                if (vq->call_ctx.ctx) {
                        cb.callback = vhost_vdpa_virtqueue_cb;
                        cb.private = vq;
                        cb.trigger = vq->call_ctx.ctx;
                } else {
                        cb.callback = NULL;
                        cb.private = NULL;
                        cb.trigger = NULL;
                }
                ops->set_vq_cb(vdpa, idx, &cb);
                vhost_vdpa_setup_vq_irq(v, idx);

in vhost_vdpa_setup_vq_irq() we had:

        irq_bypass_unregister_producer(&vq->call_ctx.producer);

here the producer->token still points to the old one...

Is this what you have seen?

Quoted 16 lines
Quoted 7 lines

I may miss something but the only way to assign exactly the same eventfd_ctx value to another vq is where the guest tries to share the MSI-X vector among virtqueues, then qemu will use a single eventfd as the callback for multiple virtqueues. If this is true:

I don't think this is the case. I see the issue happening when running qemu vdpa live migration tests on the same host. From a vdpa device it's basically a device starting on a VM over and over.

Quoted 5 lines

For bypass registering, only the first registering can succeed as the following registering will fail because the irq bypass manager already had exactly the same producer token.
For registering, all unregistering can succeed:

Quoted 7 lines
Quoted 6 lines

1) the first unregistering will do the real job that unregister the token
2) the following unregistering will do nothing by iterating the producer token list without finding a match one

Maybe you can show me the userspace behaviour (ioctls) when you see this?

Sure, what would you need? qemu traces?

Yes, that would be helpful.

Thanks

Quoted 83 lines

Thanks,
Dragos

Quoted 78 lines

Thanks

Quoted 75 lines
Quoted 64 lines
Quoted 64 lines

4) The original vq tries to unregister it's producer which it has already unlinked in step 1. irq_bypass_unregister_producer() will go ahead and unlink the producer once again. That happens because:
a) The producer has a token.
b) An element with that token is found. But that element comes from step 3.

I see 3 ways to fix this:
1) Fix the vhost-vdpa part. What this patch does. vfio has a different workflow.
2) Set the token to NULL directly in irq_bypass_unregister_producer() after unlinking the producer. But that makes the API asymmetrical.
3) Make irq_bypass_unregister_producer() also compare the pointer elements not just the tokens and do the unlink only on match.

Any thoughts?

Oops: general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP
CPU: 8 PID: 5190 Comm: qemu-system-x86 Not tainted 6.10.0-rc7+ #6
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:irq_bypass_unregister_producer+0xa5/0xd0
RSP: 0018:ffffc900034d7e50 EFLAGS: 00010246
RAX: dead000000000122 RBX: ffff888353d12718 RCX: ffff88810336a000
RDX: dead000000000100 RSI: ffffffff829243a0 RDI: 0000000000000000
RBP: ffff888353c42000 R08: ffff888104882738 R09: ffff88810336a000
R10: ffff888448ab2050 R11: 0000000000000000 R12: ffff888353d126a0
R13: 0000000000000004 R14: 0000000000000055 R15: 0000000000000004
FS:  00007f9df9403c80(0000) GS:ffff88852cc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562dffc6b568 CR3: 000000012efbb006 CR4: 0000000000772ef0
PKRU: 55555554
Call Trace:
 <TASK>
 ? die_addr+0x36/0x90
 ? exc_general_protection+0x1a8/0x390
 ? asm_exc_general_protection+0x26/0x30
 ? irq_bypass_unregister_producer+0xa5/0xd0
 vhost_vdpa_setup_vq_irq+0x5a/0xc0 [vhost_vdpa]
 vhost_vdpa_unlocked_ioctl+0xdcd/0xe00 [vhost_vdpa]
 ? vhost_vdpa_config_cb+0x30/0x30 [vhost_vdpa]
 __x64_sys_ioctl+0x90/0xc0
 do_syscall_64+0x4f/0x110
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f9df930774f
RSP: 002b:00007ffc55013080 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000562dfe134d20 RCX: 00007f9df930774f
RDX: 00007ffc55013200 RSI: 000000004008af21 RDI: 0000000000000011
RBP: 00007ffc55013200 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000562dfe134360
R13: 0000562dfe134d20 R14: 0000000000000000 R15: 00007f9df801e190
---
 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +0 -0
index 478cd46a49ed..d4a7a3918d86 100644
@@ -226,6 +226,7 @@ static void vhost_vdpa_unsetup_vq_irq(struct vhost_vdpa *v, u16 qid)
        struct vhost_virtqueue *vq = &v->vqs[qid];
 
Quoted 9 lines
Quoted 8 lines
drivers/vhost/vdpa.c +1 -1
        irq_bypass_unregister_producer(&vq->call_ctx.producer);
+       vq->call_ctx.producer.token = NULL;
 }
 
 static int _compat_vdpa_reset(struct vhost_vdpa *v)
--

2.45.2

Thanks

#6 lore raw

(Re-sending. I messed up the previous message, sorry about that.)

Quoted 76 lines · Jason Wang
Quoted 28 lines · Dragos Tatulea
Quoted 26 lines · Jason Wang
Quoted 24 lines · Dragos Tatulea
Quoted 8 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Quoted 9 lines

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Quoted 5 lines
Quoted 3 lines

Just to make sure I understand the issue. The eventfd_ctx should be still valid until a new VHOST_SET_VRING_CALL().

I think it's not about the validity of the eventfd_ctx. More about the lingering ctx value of the producer after vhost_vdpa_unsetup_vq_irq().

Probably, but

Quoted 1 line

That value is the eventfd ctx, but it could be anything else really...

I mean we hold a refcnt of the eventfd so it should be valid until the next set_vring_call() or vhost_dev_cleanup().

But I do spot some possible issue:

1) We swap and assign new ctx in vhost_vring_ioctl():

                swap(ctx, vq->call_ctx.ctx);

2) and old ctx will be put there as well:

                if (!IS_ERR_OR_NULL(ctx))
                        eventfd_ctx_put(ctx);

3) but in vdpa, we try to unregister the producer with the new token:

static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
void __user *argp)
{
...

        r = vhost_vring_ioctl(&v->vdev, cmd, argp);
...
        switch (cmd) {
...
        case VHOST_SET_VRING_CALL:
                if (vq->call_ctx.ctx) {
                        cb.callback = vhost_vdpa_virtqueue_cb;
                        cb.private = vq;
                        cb.trigger = vq->call_ctx.ctx;
                } else {
                        cb.callback = NULL;
                        cb.private = NULL;
                        cb.trigger = NULL;
                }
                ops->set_vq_cb(vdpa, idx, &cb);
                vhost_vdpa_setup_vq_irq(v, idx);
Quoted 7 lines

in vhost_vdpa_setup_vq_irq() we had:

        irq_bypass_unregister_producer(&vq->call_ctx.producer);

here the producer->token still points to the old one...

Is this what you have seen?

Yup. That is the issue. The unregister already happened at vhost_vdpa_unsetup_vq_irq(). So this second unregister will work on an already unregistered element due to the token still being set.

Quoted 21 lines
Quoted 21 lines
Quoted 8 lines

I may miss something but the only way to assign exactly the same eventfd_ctx value to another vq is where the guest tries to share the MSI-X vector among virtqueues, then qemu will use a single eventfd as the callback for multiple virtqueues. If this is true:

I don't think this is the case. I see the issue happening when running qemu vdpa live migration tests on the same host. From a vdpa device it's basically a device starting on a VM over and over.

Quoted 9 lines

For bypass registering, only the first registering can succeed as the following registering will fail because the irq bypass manager already had exactly the same producer token.
For registering, all unregistering can succeed:

1) the first unregistering will do the real job that unregister the token
2) the following unregistering will do nothing by iterating the producer token list without finding a match one

Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Maybe you can show me the userspace behaviour (ioctls) when you see this?

Sure, what would you need? qemu traces?

Yes, that would be helpful.

Will try to get them.

Thanks,
Dragos

Quoted 86 lines

Thanks

Quoted 83 lines

Thanks,
Dragos

Quoted 78 lines

Thanks

Quoted 75 lines
Quoted 64 lines
Quoted 64 lines

4) The original vq tries to unregister it's producer which it has already unlinked in step 1. irq_bypass_unregister_producer() will go ahead and unlink the producer once again. That happens because:
a) The producer has a token.
b) An element with that token is found. But that element comes from step 3.

I see 3 ways to fix this:
1) Fix the vhost-vdpa part. What this patch does. vfio has a different workflow.
2) Set the token to NULL directly in irq_bypass_unregister_producer() after unlinking the producer. But that makes the API asymmetrical.
3) Make irq_bypass_unregister_producer() also compare the pointer elements not just the tokens and do the unlink only on match.

Any thoughts?

Oops: general protection fault, probably for non-canonical address 0xdead000000000108: 0000 [#1] SMP
CPU: 8 PID: 5190 Comm: qemu-system-x86 Not tainted 6.10.0-rc7+ #6
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:irq_bypass_unregister_producer+0xa5/0xd0
RSP: 0018:ffffc900034d7e50 EFLAGS: 00010246
RAX: dead000000000122 RBX: ffff888353d12718 RCX: ffff88810336a000
RDX: dead000000000100 RSI: ffffffff829243a0 RDI: 0000000000000000
RBP: ffff888353c42000 R08: ffff888104882738 R09: ffff88810336a000
R10: ffff888448ab2050 R11: 0000000000000000 R12: ffff888353d126a0
R13: 0000000000000004 R14: 0000000000000055 R15: 0000000000000004
FS:  00007f9df9403c80(0000) GS:ffff88852cc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562dffc6b568 CR3: 000000012efbb006 CR4: 0000000000772ef0
PKRU: 55555554
Call Trace:
 <TASK>
 ? die_addr+0x36/0x90
 ? exc_general_protection+0x1a8/0x390
 ? asm_exc_general_protection+0x26/0x30
 ? irq_bypass_unregister_producer+0xa5/0xd0
 vhost_vdpa_setup_vq_irq+0x5a/0xc0 [vhost_vdpa]
 vhost_vdpa_unlocked_ioctl+0xdcd/0xe00 [vhost_vdpa]
 ? vhost_vdpa_config_cb+0x30/0x30 [vhost_vdpa]
 __x64_sys_ioctl+0x90/0xc0
 do_syscall_64+0x4f/0x110
 entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7f9df930774f
RSP: 002b:00007ffc55013080 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000562dfe134d20 RCX: 00007f9df930774f
RDX: 00007ffc55013200 RSI: 000000004008af21 RDI: 0000000000000011
RBP: 00007ffc55013200 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000562dfe134360
R13: 0000562dfe134d20 R14: 0000000000000000 R15: 00007f9df801e190
---
 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +0 -0
index 478cd46a49ed..d4a7a3918d86 100644
@@ -226,6 +226,7 @@ static void vhost_vdpa_unsetup_vq_irq(struct vhost_vdpa *v, u16 qid)
        struct vhost_virtqueue *vq = &v->vqs[qid];
 
Quoted 9 lines
Quoted 8 lines
drivers/vhost/vdpa.c +1 -1
        irq_bypass_unregister_producer(&vq->call_ctx.producer);
+       vq->call_ctx.producer.token = NULL;
 }
 
 static int _compat_vdpa_reset(struct vhost_vdpa *v)
--

2.45.2

Thanks

#7 lore raw
Quoted 112 lines · Dragos Tatulea

(Re-sending. I messed up the previous message, sorry about that.)

Quoted 76 lines · Jason Wang
Quoted 28 lines · Dragos Tatulea
Quoted 26 lines · Jason Wang
Quoted 24 lines · Dragos Tatulea
Quoted 8 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Quoted 9 lines

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Quoted 5 lines
Quoted 3 lines

Just to make sure I understand the issue. The eventfd_ctx should be still valid until a new VHOST_SET_VRING_CALL().

I think it's not about the validity of the eventfd_ctx. More about the lingering ctx value of the producer after vhost_vdpa_unsetup_vq_irq().

Probably, but

Quoted 1 line

That value is the eventfd ctx, but it could be anything else really...

I mean we hold a refcnt of the eventfd so it should be valid until the next set_vring_call() or vhost_dev_cleanup().

But I do spot some possible issue:

1) We swap and assign new ctx in vhost_vring_ioctl():

                swap(ctx, vq->call_ctx.ctx);

2) and old ctx will be put there as well:

                if (!IS_ERR_OR_NULL(ctx))
                        eventfd_ctx_put(ctx);

3) but in vdpa, we try to unregister the producer with the new token:

static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
void __user *argp)
{
...

        r = vhost_vring_ioctl(&v->vdev, cmd, argp);
...
        switch (cmd) {
...
        case VHOST_SET_VRING_CALL:
                if (vq->call_ctx.ctx) {
                        cb.callback = vhost_vdpa_virtqueue_cb;
                        cb.private = vq;
                        cb.trigger = vq->call_ctx.ctx;
                } else {
                        cb.callback = NULL;
                        cb.private = NULL;
                        cb.trigger = NULL;
                }
                ops->set_vq_cb(vdpa, idx, &cb);
                vhost_vdpa_setup_vq_irq(v, idx);
Quoted 7 lines

in vhost_vdpa_setup_vq_irq() we had:

        irq_bypass_unregister_producer(&vq->call_ctx.producer);

here the producer->token still points to the old one...

Is this what you have seen?

Yup. That is the issue. The unregister already happened at vhost_vdpa_unsetup_vq_irq(). So this second unregister will work on an already unregistered element due to the token still being set.

Quoted 21 lines
Quoted 21 lines
Quoted 8 lines

I may miss something but the only way to assign exactly the same eventfd_ctx value to another vq is where the guest tries to share the MSI-X vector among virtqueues, then qemu will use a single eventfd as the callback for multiple virtqueues. If this is true:

I don't think this is the case. I see the issue happening when running qemu vdpa live migration tests on the same host. From a vdpa device it's basically a device starting on a VM over and over.

Quoted 9 lines

For bypass registering, only the first registering can succeed as the following registering will fail because the irq bypass manager already had exactly the same producer token.
For registering, all unregistering can succeed:

1) the first unregistering will do the real job that unregister the token
2) the following unregistering will do nothing by iterating the producer token list without finding a match one

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Maybe you can show me the userspace behaviour (ioctls) when you see this?

Sure, what would you need? qemu traces?

Yes, that would be helpful.

Will try to get them.

As the traces are quite large (~5MB), I uploaded them in this location [0]. I used the following qemu traces:
--trace vhost_vdpa* --trace virtio_net_handle*

[0] https://drive.google.com/file/d/1XyXYyockJ_O7zMgI7vot6AxYjze9Ljju/view?usp=sharing

Thanks,
Dragos

#8 lore raw
Quoted 120 lines · Dragos Tatulea
Quoted 116 lines · Dragos Tatulea

(Re-sending. I messed up the previous message, sorry about that.)

Quoted 76 lines · Jason Wang
Quoted 28 lines · Dragos Tatulea
Quoted 26 lines · Jason Wang
Quoted 24 lines · Dragos Tatulea
Quoted 8 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Quoted 9 lines

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Quoted 5 lines
Quoted 3 lines

Just to make sure I understand the issue. The eventfd_ctx should be still valid until a new VHOST_SET_VRING_CALL().

I think it's not about the validity of the eventfd_ctx. More about the lingering ctx value of the producer after vhost_vdpa_unsetup_vq_irq().

Probably, but

Quoted 1 line

That value is the eventfd ctx, but it could be anything else really...

I mean we hold a refcnt of the eventfd so it should be valid until the next set_vring_call() or vhost_dev_cleanup().

But I do spot some possible issue:

1) We swap and assign new ctx in vhost_vring_ioctl():

                swap(ctx, vq->call_ctx.ctx);

2) and old ctx will be put there as well:

                if (!IS_ERR_OR_NULL(ctx))
                        eventfd_ctx_put(ctx);

3) but in vdpa, we try to unregister the producer with the new token:

static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
void __user *argp)
{
...

        r = vhost_vring_ioctl(&v->vdev, cmd, argp);
...
        switch (cmd) {
...
        case VHOST_SET_VRING_CALL:
                if (vq->call_ctx.ctx) {
                        cb.callback = vhost_vdpa_virtqueue_cb;
                        cb.private = vq;
                        cb.trigger = vq->call_ctx.ctx;
                } else {
                        cb.callback = NULL;
                        cb.private = NULL;
                        cb.trigger = NULL;
                }
                ops->set_vq_cb(vdpa, idx, &cb);
                vhost_vdpa_setup_vq_irq(v, idx);
Quoted 7 lines

in vhost_vdpa_setup_vq_irq() we had:

        irq_bypass_unregister_producer(&vq->call_ctx.producer);

here the producer->token still points to the old one...

Is this what you have seen?

Yup. That is the issue. The unregister already happened at vhost_vdpa_unsetup_vq_irq(). So this second unregister will work on an already unregistered element due to the token still being set.

Quoted 25 lines
Quoted 24 lines
Quoted 8 lines

I may miss something but the only way to assign exactly the same eventfd_ctx value to another vq is where the guest tries to share the MSI-X vector among virtqueues, then qemu will use a single eventfd as the callback for multiple virtqueues. If this is true:

I don't think this is the case. I see the issue happening when running qemu vdpa live migration tests on the same host. From a vdpa device it's basically a device starting on a VM over and over.

Quoted 11 lines

For bypass registering, only the first registering can succeed as the following registering will fail because the irq bypass manager already had exactly the same producer token.
For registering, all unregistering can succeed:

1) the first unregistering will do the real job that unregister the token
2) the following unregistering will do nothing by iterating the producer token list without finding a match one

Maybe you can show me the userspace behaviour (ioctls) when you see this?

Sure, what would you need? qemu traces?

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Yes, that would be helpful.

Will try to get them.

As the traces are quite large (~5MB), I uploaded them in this location [0]. I used the following qemu traces:
--trace vhost_vdpa* --trace virtio_net_handle*

[0] https://drive.google.com/file/d/1XyXYyockJ_O7zMgI7vot6AxYjze9Ljju/view?usp=sharing

Thanks for doing this.

So it looks not like a case of eventfd sharing:

"""
[email protected]:vhost_vdpa_iotlb_begin_batch vdpa:0x7f6f9cfb5190
fd: 17 msg_type: 2 type: 5 [email protected]:vhost_vdpa_set_vring_base dev: 0x55573cc9ca70
index: 6 num: 0 svq 1 [email protected]:vhost_vdpa_set_vring_kick dev: 0x55573cc9ca70
index: 6 fd: 237 [email protected]:vhost_vdpa_set_vring_call dev: 0x55573cc9ca70
index: 6 fd: 238
[email protected]:vhost_vdpa_dma_map vdpa:0x7f6f9cfb5190 fd: 17
msg_type: 2 asid: 1 iova: 0x13000 size: 0x2000 uaddr: 0x7f6f9da1a000
perm: 0x1 type: 2
[email protected]:vhost_vdpa_dma_map vdpa:0x7f6f9cfb5190 fd: 17
msg_type: 2 asid: 1 iova: 0x15000 size: 0x1000 uaddr: 0x7f6f9da19000
perm: 0x3 type: 2 [email protected]:vhost_vdpa_set_vring_addr dev: 0x55573cc9ca70
index: 6 flags: 0x0 desc_user_addr: 0x13000 used_user_addr: 0x15000
avail_user_addr: 0x14000 log_guest_\
addr: 0x0
[email protected]:vhost_vdpa_set_vring_base dev: 0x55573cc9ca70
index: 7 num: 0 svq 1 [email protected]:vhost_vdpa_set_vring_kick dev: 0x55573cc9ca70
index: 7 fd: 239 [email protected]:vhost_vdpa_set_vring_call dev: 0x55573cc9ca70
index: 7 fd: 240
"""

I think a more proper way is to unregister and clean the token before calling vhost_vring_ioctl() in the case of SET_VRING_KICK. Let me try to draft a patch and see.

Thanks

Quoted 4 lines

Thanks,
Dragos

#9 lore raw
Quoted 162 lines · Jason Wang
Quoted 120 lines · Dragos Tatulea
Quoted 116 lines · Dragos Tatulea

(Re-sending. I messed up the previous message, sorry about that.)

Quoted 76 lines · Jason Wang
Quoted 28 lines · Dragos Tatulea
Quoted 26 lines · Jason Wang
Quoted 24 lines · Dragos Tatulea
Quoted 8 lines · Jason Wang
Quoted 6 lines · Dragos Tatulea

The following workflow triggers the crash referenced below:

1) vhost_vdpa_unsetup_vq_irq() unregisters the irq bypass producer but the producer->token is still valid.
2) vq context gets released and reassigned to another vq.

Quoted 9 lines

Just to make sure I understand here, which structure is referred to as "vq context" here? I guess it's not call_ctx as it is a part of the vq itself.

Quoted 2 lines

3) That other vq registers it's producer with the same vq context pointer as token in vhost_vdpa_setup_vq_irq().

Or did you mean when a single eventfd is shared among different vqs?

Yes, that's what I mean: vq->call_ctx.ctx which is a eventfd_ctx.

But I don't think it's shared in this case, only that the old eventfd_ctx value is lingering in producer->token. And this old eventfd_ctx is assigned now to another vq.

Quoted 5 lines
Quoted 3 lines

Just to make sure I understand the issue. The eventfd_ctx should be still valid until a new VHOST_SET_VRING_CALL().

I think it's not about the validity of the eventfd_ctx. More about the lingering ctx value of the producer after vhost_vdpa_unsetup_vq_irq().

Probably, but

Quoted 1 line

That value is the eventfd ctx, but it could be anything else really...

I mean we hold a refcnt of the eventfd so it should be valid until the next set_vring_call() or vhost_dev_cleanup().

But I do spot some possible issue:

1) We swap and assign new ctx in vhost_vring_ioctl():

                swap(ctx, vq->call_ctx.ctx);

2) and old ctx will be put there as well:

                if (!IS_ERR_OR_NULL(ctx))
                        eventfd_ctx_put(ctx);

3) but in vdpa, we try to unregister the producer with the new token:

static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
void __user *argp)
{
...

        r = vhost_vring_ioctl(&v->vdev, cmd, argp);
...
        switch (cmd) {
...
        case VHOST_SET_VRING_CALL:
                if (vq->call_ctx.ctx) {
                        cb.callback = vhost_vdpa_virtqueue_cb;
                        cb.private = vq;
                        cb.trigger = vq->call_ctx.ctx;
                } else {
                        cb.callback = NULL;
                        cb.private = NULL;
                        cb.trigger = NULL;
                }
                ops->set_vq_cb(vdpa, idx, &cb);
                vhost_vdpa_setup_vq_irq(v, idx);
Quoted 7 lines

in vhost_vdpa_setup_vq_irq() we had:

        irq_bypass_unregister_producer(&vq->call_ctx.producer);

here the producer->token still points to the old one...

Is this what you have seen?

Yup. That is the issue. The unregister already happened at vhost_vdpa_unsetup_vq_irq(). So this second unregister will work on an already unregistered element due to the token still being set.

Quoted 25 lines
Quoted 24 lines
Quoted 8 lines

I may miss something but the only way to assign exactly the same eventfd_ctx value to another vq is where the guest tries to share the MSI-X vector among virtqueues, then qemu will use a single eventfd as the callback for multiple virtqueues. If this is true:

I don't think this is the case. I see the issue happening when running qemu vdpa live migration tests on the same host. From a vdpa device it's basically a device starting on a VM over and over.

Quoted 11 lines

For bypass registering, only the first registering can succeed as the following registering will fail because the irq bypass manager already had exactly the same producer token.
For registering, all unregistering can succeed:

1) the first unregistering will do the real job that unregister the token
2) the following unregistering will do nothing by iterating the producer token list without finding a match one

Maybe you can show me the userspace behaviour (ioctls) when you see this?

Sure, what would you need? qemu traces?

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Yes, that would be helpful.

Will try to get them.

As the traces are quite large (~5MB), I uploaded them in this location [0]. I used the following qemu traces:
--trace vhost_vdpa* --trace virtio_net_handle*

[0] https://drive.google.com/file/d/1XyXYyockJ_O7zMgI7vot6AxYjze9Ljju/view?usp=sharing

Thanks for doing this.

So it looks not like a case of eventfd sharing:

"""
[email protected]:vhost_vdpa_iotlb_begin_batch vdpa:0x7f6f9cfb5190
fd: 17 msg_type: 2 type: 5 [email protected]:vhost_vdpa_set_vring_base dev: 0x55573cc9ca70
index: 6 num: 0 svq 1 [email protected]:vhost_vdpa_set_vring_kick dev: 0x55573cc9ca70
index: 6 fd: 237 [email protected]:vhost_vdpa_set_vring_call dev: 0x55573cc9ca70
index: 6 fd: 238
[email protected]:vhost_vdpa_dma_map vdpa:0x7f6f9cfb5190 fd: 17
msg_type: 2 asid: 1 iova: 0x13000 size: 0x2000 uaddr: 0x7f6f9da1a000
perm: 0x1 type: 2
[email protected]:vhost_vdpa_dma_map vdpa:0x7f6f9cfb5190 fd: 17
msg_type: 2 asid: 1 iova: 0x15000 size: 0x1000 uaddr: 0x7f6f9da19000
perm: 0x3 type: 2 [email protected]:vhost_vdpa_set_vring_addr dev: 0x55573cc9ca70
index: 6 flags: 0x0 desc_user_addr: 0x13000 used_user_addr: 0x15000
avail_user_addr: 0x14000 log_guest_\
addr: 0x0
[email protected]:vhost_vdpa_set_vring_base dev: 0x55573cc9ca70
index: 7 num: 0 svq 1 [email protected]:vhost_vdpa_set_vring_kick dev: 0x55573cc9ca70
index: 7 fd: 239 [email protected]:vhost_vdpa_set_vring_call dev: 0x55573cc9ca70
index: 7 fd: 240
"""

Quoted 3 lines

I think a more proper way is to unregister and clean the token before calling vhost_vring_ioctl() in the case of SET_VRING_KICK. Let me try to draft a patch and see.

I've posted a RFC patch, please try to see if it works.

Thanks

Quoted 7 lines

Thanks

Quoted 4 lines

Thanks,
Dragos