x86/paravirt: Guard against invalid cpu # in pv_vcpu_is_preempted()
Subsystems (3)
PATCH 1/1 x86/paravirt: Guard against invalid cpu # in pv_vcpu_is_preempted()
It was found that passing an invalid cpu number to pv_vcpu_is_preempted() might panic the kernel in a VM guest. For example,
[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20
To guard against this kind of kernel panic, check is added to pv_vcpu_is_preempted() to make sure that no invalid cpu number will be used.
--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/include/asm/paravirt.h +6 -0
@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu) static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }
Signature · 1 line
2.18.1
Juergen Gross
maintainerQuoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }
Can this really happen without being a programming error?
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
Juergen
Juergen Gross
maintainerQuoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }
Can this really happen without being a programming error?
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
Juergen
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
Quoted 11 lines
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }Can this really happen without being a programming error?
This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.
Quoted 2 lines
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.
The race condition that I am looking at is as follows:
CPU 0 CPU 1
----- -----up_write:
owner = NULL;
<release-barrier>
count = 0;
<rcu-free task structure>
rwsem_can_spin_on_owner:
rcu_read_lock();
read owner;
:
vcpu_is_preempted(owner->cpu); :
rcu_read_unlock()
When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.
Alternatively, I can also do the cpu check before calling vcpu_is_preempted().
Cheers,
Longman
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)
Quoted 11 lines
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }Can this really happen without being a programming error?
This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.
Quoted 2 lines
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.
The race condition that I am looking at is as follows:
CPU 0 CPU 1
----- -----up_write:
owner = NULL;
<release-barrier>
count = 0;
<rcu-free task structure>
rwsem_can_spin_on_owner:
rcu_read_lock();
read owner;
:
vcpu_is_preempted(owner->cpu); :
rcu_read_unlock()
When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.
Alternatively, I can also do the cpu check before calling vcpu_is_preempted().
Cheers,
Longman
Juergen Gross
maintainerQuoted 70 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)Quoted 11 lines
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }Can this really happen without being a programming error?
This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.
Quoted 2 lines
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.
The race condition that I am looking at is as follows:
CPU 0 CPU 1
----- -----up_write:
owner = NULL;
<release-barrier>
count = 0;<rcu-free task structure>
rwsem_can_spin_on_owner:
rcu_read_lock();
read owner;
:
vcpu_is_preempted(owner->cpu); :
rcu_read_unlock()
Quoted 7 lines
When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.
Alternatively, I can also do the cpu check before calling
vcpu_is_preempted().
I think I'd prefer that.
Juergen
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Juergen Gross
maintainerQuoted 70 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)Quoted 11 lines
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }Can this really happen without being a programming error?
This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.
Quoted 2 lines
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.
The race condition that I am looking at is as follows:
CPU 0 CPU 1
----- -----up_write:
owner = NULL;
<release-barrier>
count = 0;<rcu-free task structure>
rwsem_can_spin_on_owner:
rcu_read_lock();
read owner;
:
vcpu_is_preempted(owner->cpu); :
rcu_read_unlock()
Quoted 7 lines
When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.
Alternatively, I can also do the cpu check before calling
vcpu_is_preempted().
I think I'd prefer that.
Juergen
Quoted 74 lines · Juergen Gross
Quoted 73 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)Quoted 11 lines
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }Can this really happen without being a programming error?
This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.
Quoted 2 lines
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.
The race condition that I am looking at is as follows:
CPU 0 CPU 1
----- -----up_write:
owner = NULL;
<release-barrier>
count = 0;<rcu-free task structure>
rwsem_can_spin_on_owner:
rcu_read_lock();
read owner;
:
vcpu_is_preempted(owner->cpu); :
rcu_read_unlock()When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.
Quoted 7 lines
Quoted 2 lines
Alternatively, I can also do the cpu check before calling
vcpu_is_preempted().I think I'd prefer that.
Juergen
It turns out that it may be caused by a software bug after all. You can ignore this patch for now.
Thanks,
Longman
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 74 lines · Juergen Gross
Quoted 73 lines · Waiman Long
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Waiman Long
It was found that passing an invalid cpu number to
pv_vcpu_is_preempted()might panic the kernel in a VM guest. For example,[ 2.531077] Oops: 0000 [#1] SMP PTI : [ 2.532545] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2.533321] RIP: 0010:__raw_callee_save___kvm_vcpu_is_preempted+0x0/0x20To guard against this kind of kernel panic, check is added to
pv_vcpu_is_preempted()to make sure that no invalid cpu number will be used.Signed-off-by Waiman Long <[email protected]>--- arch/x86/include/asm/paravirt.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/include/asm/paravirt.h +0
-0@@ -671,6 +671,12 @@ static __always_inline void pv_kick(int cpu)Quoted 11 lines
arch/x86/include/asm/paravirt.h static __always_inline void pv_kick(int cpu) · line 681
Quoted 10 lines
arch/x86/include/asm/paravirt.h +6
-0static __always_inline bool pv_vcpu_is_preempted(long cpu) { +/* +* Guard against invalid cpu number or the kernel might panic. +*/ +if (WARN_ON_ONCE ((unsigned long )cpu >= nr_cpu_ids)) +return false; +return PVOP_CALLEE1 (bool, lock.vcpu_is_preempted, cpu); }Can this really happen without being a programming error?
This shouldn't happen without a programming error, I think. In my case, it was caused by a race condition leading to use-after-free of the cpu number. However, my point is that error like that shouldn't cause the kernel to panic.
Quoted 2 lines
Basically you'd need to guard all percpu area accesses to foreign cpus this way. Why is this one special?
It depends. If out-of-bound access can only happen with obvious programming error, I don't think we need to guard against them. In this case, I am not totally sure if the race condition that I found may happen with existing code or not. To be prudent, I decide to send this patch out.
The race condition that I am looking at is as follows:
CPU 0 CPU 1
----- -----up_write:
owner = NULL;
<release-barrier>
count = 0;<rcu-free task structure>
rwsem_can_spin_on_owner:
rcu_read_lock();
read owner;
:
vcpu_is_preempted(owner->cpu); :
rcu_read_unlock()When I tried to merge the owner into the count (clear the owner after the barrier), I can reproduce the crash 100% when booting up the kernel in a VM guest. However, I am not sure if the configuration above is safe and is just very hard to reproduce.
Quoted 7 lines
Quoted 2 lines
Alternatively, I can also do the cpu check before calling
vcpu_is_preempted().I think I'd prefer that.
Juergen
It turns out that it may be caused by a software bug after all. You can ignore this patch for now.
Thanks,
Longman