x86/flush: use APIC ALLBUT destination shorthand when possible
PATCH 1/1 x86/flush: use APIC ALLBUT destination shorthand when possible
If the flush mask matches the mask of online CPUs use the APIC ALLBUT destination shorthand in order to send an IPI to all CPUs on the system except the current one. This can only be safely used when no CPU hotplug or unplug operations are taking place, no offline CPUs or those have been onlined and parked and finally when all CPUs in the system have been accounted for (ie: the number of CPUs doesn't exceed NR_CPUS and APIC IDs are below MAX_APICS).
This is specially beneficial when using the PV shim, since using the shorthand avoids performing an APIC register write (or multiple ones if using xAPIC mode) for each destination in the flush mask.
The lock time on a 32 vCPU guest using the shim without the shorthand is:
Global lock flush_lock: addr=ffff82d0804b21c0, lockval=f602f602, not locked
lock:228455938(79406065573135), block:205908580(556416605761539)
Average lock time: 347577ns
While the same guest using the shorthand:
Global lock flush_lock: addr=ffff82d0804b41c0, lockval=d9c4d9bc, cpu=12
lock:1890775(416719148054), block:1663958(2500161282949)
Average lock time: 220395ns
Approximately a 1/3 improvement in the lock time.
--- xen/arch/x86/acpi/boot.c | 1 + xen/arch/x86/mpparse.c | 5 +++++ xen/arch/x86/smp.c | 41 ++++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/smp.h | 2 ++ 4 files changed, 48 insertions(+), 1 deletion(-)
xen/arch/x86/acpi/boot.c +1 -0
@@ -103,6 +103,7 @@ acpi_parse_x2apic(struct acpi_subtable_header *header, const unsigned long end) processor->lapic_flags &ACPI_MADT_ENABLED ?KERN_WARNING "WARNING: " :KERN_INFO , processor->local_apic_id, processor->uid); + cpu_overflow = true;/* * Must not return an error here, to prevent * acpi_table_parse_entries() from terminating early.
xen/arch/x86/mpparse.c +5 -0
@@ -66,6 +66,9 @@ static unsigned int __initdata disabled_cpus;/* Bitmask of physically existing CPUs */ physid_mask_t phys_cpu_present_map; +/* Record whether CPUs haven't been added due to overflows. */ +bool __read_mostly cpu_overflow; +void __init set_nr_cpu_ids (unsigned int max_cpus) {unsigned int tot_cpus = num_processors + disabled_cpus;
@@ -160,6 +163,7 @@ static int MP_processor_info_x(struct mpc_config_processor *m,printk_once (XENLOG_WARNING "WARNING: NR_CPUS limit of %u reached - ignoring further processors\n" , nr_cpu_ids); + cpu_overflow = true;return -ENOSPC ; }
@@ -167,6 +171,7 @@ static int MP_processor_info_x(struct mpc_config_processor *m, && genapic.name == apic_default.name) {printk_once (XENLOG_WARNING "WARNING: CPUs limit of 8 reached - ignoring futher processors\n" ); + cpu_overflow = true;return -ENOSPC ; }
xen/arch/x86/smp.c +40 -1
66 lines not on this page · raw message
xen/include/asm-x86/smp.h +2 -0
@@ -84,6 +84,8 @@ extern cpumask_t **socket_cpumask;#define get_cpu_current (cpu) \ (get_cpu_info_from_stack((unsigned long)stack_base[cpu])->current_vcpu) +extern bool cpu_overflow; +#endif /* !__ASSEMBLY__ */ #endif
Signature · 7 lines
2.24.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 22 lines · Roger Pau Monne
If the flush mask matches the mask of online CPUs use the APIC ALLBUT destination shorthand in order to send an IPI to all CPUs on the system except the current one. This can only be safely used when no CPU hotplug or unplug operations are taking place, no offline CPUs or those have been onlined and parked and finally when all CPUs in the system have been accounted for (ie: the number of CPUs doesn't exceed
NR_CPUSand APIC IDs are belowMAX_APICS).This is specially beneficial when using the PV shim, since using the shorthand avoids performing an APIC register write (or multiple ones if using xAPIC mode) for each destination in the flush mask.
The lock time on a 32 vCPU guest using the shim without the shorthand is:
Global lock
flush_lock: addr=ffff82d0804b21c0, lockval=f602f602, not locked
lock:228455938(79406065573135), block:205908580(556416605761539)Average lock time: 347577ns
While the same guest using the shorthand:
Quoted 8 lines
Global lock
flush_lock: addr=ffff82d0804b41c0, lockval=d9c4d9bc, cpu=12
lock:1890775(416719148054), block:1663958(2500161282949)Average lock time: 220395ns
Approximately a 1/3 improvement in the lock time.
Signed-off-by Roger Pau Monné <[email protected]>
While these are good stats, I'm somewhat hesitant about hacking this in like this. For one, it is a substantial amount of ad-hoc logic in flush_area_mask()
Shorthand safety really should be part of the apic driver, not part of the TLB logic.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 30 lines · Andrew Cooper
Quoted 29 lines · Roger Pau Monne
If the flush mask matches the mask of online CPUs use the APIC ALLBUT destination shorthand in order to send an IPI to all CPUs on the system except the current one. This can only be safely used when no CPU hotplug or unplug operations are taking place, no offline CPUs or those have been onlined and parked and finally when all CPUs in the system have been accounted for (ie: the number of CPUs doesn't exceed
NR_CPUSand APIC IDs are belowMAX_APICS).This is specially beneficial when using the PV shim, since using the shorthand avoids performing an APIC register write (or multiple ones if using xAPIC mode) for each destination in the flush mask.
The lock time on a 32 vCPU guest using the shim without the shorthand is:
Global lock
flush_lock: addr=ffff82d0804b21c0, lockval=f602f602, not locked
lock:228455938(79406065573135), block:205908580(556416605761539)Average lock time: 347577ns
While the same guest using the shorthand:
Global lock
flush_lock: addr=ffff82d0804b41c0, lockval=d9c4d9bc, cpu=12
lock:1890775(416719148054), block:1663958(2500161282949)Average lock time: 220395ns
Approximately a 1/3 improvement in the lock time.
Quoted 8 lines
Quoted 1 line
Signed-off-by Roger Pau Monné <[email protected]>While these are good stats, I'm somewhat hesitant about hacking this in like this. For one, it is a substantial amount of ad-hoc logic in
flush_area_mask()Shorthand safety really should be part of the apic driver, not part of the TLB logic.
Yes, I've coded it this way because there are already similar APIC hooks (ie: send_IPI_self). I can merge the shorthand functionality with send_IPI_mask if that's preferred.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 11 lines · Roger Pau Monne
xen/arch/x86/smp.c +2
-0@@ -227,14 +233,47 @@ void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags)if ( (flags & ~FLUSH_ORDER_MASK ) && !cpumask_subset (mask,cpumask_of (cpu)) ) { +bool cpus_locked = false; +spin_lock (&flush_lock);cpumask_and (&flush_cpumask, mask, &cpu_online_map);cpumask_clear_cpu (cpu, &flush_cpumask); flush_va = va; flush_flags = flags;
xen/arch/x86/smp.c void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags) · line 251
Quoted 10 lines
xen/arch/x86/smp.c +9
-1-send_IPI_mask (&flush_cpumask,INVALIDATE_TLB_VECTOR ); + +/* +* Prevent any CPU hot{un}plug while sending the IPIs if we are to use +* a shorthand, also refuse to use a shorthand if not all CPUs are +* online or have been parked. +*/ + if ( system_state > SYS_STATE_smp_boot && !cpu_overflow && + (cpus_locked = get_cpu_maps()) && + (park_offline_cpus ||
Why is it relevant whether we park offline CPUs, or whether we've even brought up all of the ones a system has? An IPI, in particular a broadcast one, shouldn't have any issue getting delivered if some of the nominal recipients don't listen, should it? (The use of cpu_online_map that was already there above is a sign - but not a proof, as it may itself be buggy - that the set of online CPUs fluctuating behind this function's back ought to not be a problem.)
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with the get_cpu_maps() lock held? Even if you did and even if there are none, I think the function should then get a comment attached to the effect of this lock order inversion risk. (For example, it isn't obvious to me that no user of stop_machine() would ever want to do any kind of TLB flushing.)
Overall I wonder whether your goal couldn't be achieved without the extra locking and without the special conditions.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Roger Pau Monne
xen/arch/x86/smp.c +7
-1@@ -227,14 +233,47 @@ void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags)if ( (flags & ~FLUSH_ORDER_MASK ) && !cpumask_subset (mask,cpumask_of (cpu)) ) { +bool cpus_locked = false; +spin_lock (&flush_lock);cpumask_and (&flush_cpumask, mask, &cpu_online_map);cpumask_clear_cpu (cpu, &flush_cpumask); flush_va = va; flush_flags = flags; -send_IPI_mask (&flush_cpumask,INVALIDATE_TLB_VECTOR ); + +/* +* Prevent any CPU hot{un}plug while sending the IPIs if we are to use +* a shorthand, also refuse to use a shorthand if not all CPUs are +* online or have been parked.
Quoted 8 lines
xen/arch/x86/smp.c void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags) · line 251
Quoted 4 lines
xen/arch/x86/smp.c +4
-0+*/ + if ( system_state > SYS_STATE_smp_boot && !cpu_overflow && + (cpus_locked = get_cpu_maps()) && + (park_offline_cpus ||Why is it relevant whether we park offline CPUs, or whether we've even brought up all of the ones a system has? An IPI, in particular a broadcast one, shouldn't have any issue getting delivered if some of the nominal recipients don't listen, should it?
TGLX had a hard time time making shorthands work correctly on Linux. ISTR target CPUs in wait-for-SIPI may cause the source side of the IPI to hang indefinitely, or yield an APIC Send error.
Linux maintains a bitmap of "CPUs which have booted once and we know can safely handle IPIs", and doesn't permit any broadcast shorthands until this matches the cpus_available mask.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 23 lines · Jan Beulich
Quoted 11 lines · Roger Pau Monne
xen/arch/x86/smp.c +2
-0@@ -227,14 +233,47 @@ void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags)if ( (flags & ~FLUSH_ORDER_MASK ) && !cpumask_subset (mask,cpumask_of (cpu)) ) { +bool cpus_locked = false; +spin_lock (&flush_lock);cpumask_and (&flush_cpumask, mask, &cpu_online_map);cpumask_clear_cpu (cpu, &flush_cpumask); flush_va = va; flush_flags = flags;xen/arch/x86/smp.c void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags) · line 251
Quoted 10 lines
xen/arch/x86/smp.c +9
-1-send_IPI_mask (&flush_cpumask,INVALIDATE_TLB_VECTOR ); + +/* +* Prevent any CPU hot{un}plug while sending the IPIs if we are to use +* a shorthand, also refuse to use a shorthand if not all CPUs are +* online or have been parked. +*/ + if ( system_state > SYS_STATE_smp_boot && !cpu_overflow && + (cpus_locked = get_cpu_maps()) && + (park_offline_cpus ||
Quoted 7 lines
Why is it relevant whether we park offline CPUs, or whether we've even brought up all of the ones a system has? An IPI, in particular a broadcast one, shouldn't have any issue getting delivered if some of the nominal recipients don't listen, should it? (The use of
cpu_online_mapthat was already there above is a sign - but not a proof, as it may itself be buggy - that the set of online CPUs fluctuating behind this function's back ought to not be a problem.)
I've tried it myself, and if not all CPUs are onlined when the shorthand is used the box would just reboot. This matches the description at:
https://lwn.net/Articles/793065/
Of the Linux shorthand implementation.
Quoted 3 lines
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with the
get_cpu_maps()lock held?
The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
Quoted 8 lines
Even if
you did and even if there are none, I think the function should then get a comment attached to the effect of this lock order inversion risk. (For example, it isn't obvious to me that no user ofstop_machine()would ever want to do any kind of TLB flushing.)Overall I wonder whether your goal couldn't be achieved without the extra locking and without the special conditions.
Hm, this so far has worked fine on all the boxes that I've tried. I'm happy to change it to a simpler approach, but I think the conditions and locking are required for this to work correctly.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 8 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
On 24.12.2019 13:44, Roger Pau Monne wrote:
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with theget_cpu_maps()lock held?The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
When already held by the same CPU - sure. It being a recursive one (which I paid attention to when writing my earlier reply) doesn't make it (together with any other one) immune against ABBA deadlocks, though.
Quoted 6 lines
Quoted 6 lines
Even if
you did and even if there are none, I think the function should then get a comment attached to the effect of this lock order inversion risk. (For example, it isn't obvious to me that no user ofstop_machine()would ever want to do any kind of TLB flushing.)
Quoted 6 lines
Quoted 2 lines
Overall I wonder whether your goal couldn't be achieved without the extra locking and without the special conditions.
Hm, this so far has worked fine on all the boxes that I've tried. I'm happy to change it to a simpler approach, but I think the conditions and locking are required for this to work correctly.
Which might then indicate said pre-existing use of cpu_online_map to be a (latent?) problem.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
On 24.12.2019 13:44, Roger Pau Monne wrote:
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with theget_cpu_maps()lock held?
Quoted 7 lines
Quoted 2 lines
The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
When already held by the same CPU - sure. It being a recursive one (which I paid attention to when writing my earlier reply) doesn't make it (together with any other one) immune against ABBA deadlocks, though.
There's no possibility of a deadlock here because get_cpu_maps does a trylock, so if another cpu is holding the lock the flush will just fallback to not using the shorthand.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Even if
you did and even if there are none, I think the function should then get a comment attached to the effect of this lock order inversion risk. (For example, it isn't obvious to me that no user ofstop_machine()would ever want to do any kind of TLB flushing.)Overall I wonder whether your goal couldn't be achieved without the extra locking and without the special conditions.
Quoted 6 lines
Quoted 3 lines
Hm, this so far has worked fine on all the boxes that I've tried. I'm happy to change it to a simpler approach, but I think the conditions and locking are required for this to work correctly.
Which might then indicate said pre-existing use of
cpu_online_mapto be a (latent?) problem.
Hm, maybe it could be a problem when offlining a CPU. I assume this is not an issue so far because there are no global TLB flushes with a mask contaning a CPU that is being offlined.
Regarding the patch itself, do you think the shorthand logic should be added to send_IPI_mask, or would you rather only use the shorthand for the TLB flushes?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 11 lines · Roger Pau Monné
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
On 24.12.2019 13:44, Roger Pau Monne wrote:
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with theget_cpu_maps()lock held?The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
Quoted 8 lines
Quoted 4 lines
When already held by the same CPU - sure. It being a recursive one (which I paid attention to when writing my earlier reply) doesn't make it (together with any other one) immune against ABBA deadlocks, though.
There's no possibility of a deadlock here because
get_cpu_mapsdoes a trylock, so if another cpu is holding the lock the flush will just fallback to not using the shorthand.
Well, with the _exact_ arrangements (flush_lock used only in one place, and cpu_add_remove_lock only used in ways similar to how it is used now), there's no such risk, I agree. But there's nothing at all making sure this doesn't change. Hence, as said, at the very least this needs reasoning about in the description (or a code comment).
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Even if
you did and even if there are none, I think the function should then get a comment attached to the effect of this lock order inversion risk. (For example, it isn't obvious to me that no user ofstop_machine()would ever want to do any kind of TLB flushing.)Overall I wonder whether your goal couldn't be achieved without the extra locking and without the special conditions.
Quoted 6 lines
Quoted 3 lines
Hm, this so far has worked fine on all the boxes that I've tried. I'm happy to change it to a simpler approach, but I think the conditions and locking are required for this to work correctly.
Which might then indicate said pre-existing use of
cpu_online_mapto be a (latent?) problem.
Quoted 7 lines
Hm, maybe it could be a problem when offlining a CPU. I assume this is not an issue so far because there are no global TLB flushes with a mask contaning a CPU that is being offlined.
Regarding the patch itself, do you think the shorthand logic should be added to
send_IPI_mask, or would you rather only use the shorthand for the TLB flushes?
If it could be generalize to all IPI sending, this would of course seem preferable.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 21 lines · Jan Beulich
Quoted 11 lines · Roger Pau Monné
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
On 24.12.2019 13:44, Roger Pau Monne wrote:
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with theget_cpu_maps()lock held?The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
Quoted 8 lines
Quoted 4 lines
When already held by the same CPU - sure. It being a recursive one (which I paid attention to when writing my earlier reply) doesn't make it (together with any other one) immune against ABBA deadlocks, though.
There's no possibility of a deadlock here because
get_cpu_mapsdoes a trylock, so if another cpu is holding the lock the flush will just fallback to not using the shorthand.
Quoted 6 lines
Well, with the _exact_ arrangements (
flush_lockused only in one place, andcpu_add_remove_lockonly used in ways similar to how it is used now), there's no such risk, I agree. But there's nothing at all making sure this doesn't change. Hence, as said, at the very least this needs reasoning about in the description (or a code comment).
I'm afraid you will have to bear with me, but I'm still not sure how the addition of get_cpu_maps in flush_area_mask can lead to deadlocks. As said above get_cpu_maps does a trylock, which means that it will never deadlock, and that's the only way to lock cpu_add_remove_lock.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 29 lines · Roger Pau Monné
Quoted 21 lines · Jan Beulich
Quoted 11 lines · Roger Pau Monné
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
On 24.12.2019 13:44, Roger Pau Monne wrote:
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with theget_cpu_maps()lock held?The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
Quoted 8 lines
Quoted 4 lines
When already held by the same CPU - sure. It being a recursive one (which I paid attention to when writing my earlier reply) doesn't make it (together with any other one) immune against ABBA deadlocks, though.
There's no possibility of a deadlock here because
get_cpu_mapsdoes a trylock, so if another cpu is holding the lock the flush will just fallback to not using the shorthand.Quoted 6 lines
Well, with the _exact_ arrangements (
flush_lockused only in one place, andcpu_add_remove_lockonly used in ways similar to how it is used now), there's no such risk, I agree. But there's nothing at all making sure this doesn't change. Hence, as said, at the very least this needs reasoning about in the description (or a code comment).
Quoted 4 lines
I'm afraid you will have to bear with me, but I'm still not sure how the addition of
get_cpu_mapsinflush_area_maskcan lead to deadlocks. As said aboveget_cpu_mapsdoes a trylock, which means that it will never deadlock, and that's the only way to lockcpu_add_remove_lock.
That's why I said "cpu_add_remove_lock only used in ways similar to how it is used now" - any non-trylock addition would break the assumptions you make afaict. And you can't really expect people adding another (slightly different) use of an existing lock to be aware they now need to check whether this introduces deadlock scenarios on unrelated pre-existing code paths. Hence my request to at least discuss this in the description (raising awareness, and hence allowing reviewers to judge whether further precautionary measures should be taken).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 35 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 21 lines · Jan Beulich
Quoted 11 lines · Roger Pau Monné
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
On 24.12.2019 13:44, Roger Pau Monne wrote:
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with theget_cpu_maps()lock held?The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
Quoted 8 lines
Quoted 4 lines
When already held by the same CPU - sure. It being a recursive one (which I paid attention to when writing my earlier reply) doesn't make it (together with any other one) immune against ABBA deadlocks, though.
There's no possibility of a deadlock here because
get_cpu_mapsdoes a trylock, so if another cpu is holding the lock the flush will just fallback to not using the shorthand.Quoted 6 lines
Well, with the _exact_ arrangements (
flush_lockused only in one place, andcpu_add_remove_lockonly used in ways similar to how it is used now), there's no such risk, I agree. But there's nothing at all making sure this doesn't change. Hence, as said, at the very least this needs reasoning about in the description (or a code comment).Quoted 4 lines
I'm afraid you will have to bear with me, but I'm still not sure how the addition of
get_cpu_mapsinflush_area_maskcan lead to deadlocks. As said aboveget_cpu_mapsdoes a trylock, which means that it will never deadlock, and that's the only way to lockcpu_add_remove_lock.
Quoted 9 lines
That's why I said "
cpu_add_remove_lockonly used in ways similar to how it is used now" - any non-trylock addition would break the assumptions you make afaict. And you can't really expect people adding another (slightly different) use of an existing lock to be aware they now need to check whether this introduces deadlock scenarios on unrelated pre-existing code paths. Hence my request to at least discuss this in the description (raising awareness, and hence allowing reviewers to judge whether further precautionary measures should be taken).
Thanks for the clarification, I did indeed misunderstood your complain.
Regarding the generalization of the shorthand and integration into send_IPI_mask, I've found an issue related to locking. send_IPI_mask is called with both interrupts enabled and disabled, and so get_cpu_maps panics because of the lock checking.
I however don't think that such panic is accurate: the logic in check_lock specifically relates to the spinning done when picking a lock, but I would say the call to check_lock in _spin_trylock{_recursive} is not strictly needed, the scenario described in check_lock doesn't apply when using trylock.
So my question is whether you would be OK to remove the calls to check_lock in the trylock variants, or would you rather keep the shorthand usage limited to flush_area_mask.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 54 lines · Roger Pau Monné
Quoted 35 lines · Jan Beulich
Quoted 29 lines · Roger Pau Monné
Quoted 21 lines · Jan Beulich
Quoted 11 lines · Roger Pau Monné
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
On 24.12.2019 13:44, Roger Pau Monne wrote:
Further a question on lock nesting: Since the commit message doesn't say anything in this regard, did you check there are no TLB flush invocations with theget_cpu_maps()lock held?The CPU maps lock is a recursive one, so it should be fine to attempt a TLB flush with the lock already held.
Quoted 8 lines
Quoted 4 lines
When already held by the same CPU - sure. It being a recursive one (which I paid attention to when writing my earlier reply) doesn't make it (together with any other one) immune against ABBA deadlocks, though.
There's no possibility of a deadlock here because
get_cpu_mapsdoes a trylock, so if another cpu is holding the lock the flush will just fallback to not using the shorthand.Quoted 6 lines
Well, with the _exact_ arrangements (
flush_lockused only in one place, andcpu_add_remove_lockonly used in ways similar to how it is used now), there's no such risk, I agree. But there's nothing at all making sure this doesn't change. Hence, as said, at the very least this needs reasoning about in the description (or a code comment).Quoted 4 lines
I'm afraid you will have to bear with me, but I'm still not sure how the addition of
get_cpu_mapsinflush_area_maskcan lead to deadlocks. As said aboveget_cpu_mapsdoes a trylock, which means that it will never deadlock, and that's the only way to lockcpu_add_remove_lock.Quoted 9 lines
That's why I said "
cpu_add_remove_lockonly used in ways similar to how it is used now" - any non-trylock addition would break the assumptions you make afaict. And you can't really expect people adding another (slightly different) use of an existing lock to be aware they now need to check whether this introduces deadlock scenarios on unrelated pre-existing code paths. Hence my request to at least discuss this in the description (raising awareness, and hence allowing reviewers to judge whether further precautionary measures should be taken).Thanks for the clarification, I did indeed misunderstood your complain.
Regarding the generalization of the shorthand and integration into
send_IPI_mask, I've found an issue related to locking.send_IPI_maskis called with both interrupts enabled and disabled, and soget_cpu_mapspanics because of the lock checking.
Quoted 5 lines
I however don't think that such panic is accurate: the logic in
check_lockspecifically relates to the spinning done when picking a lock, but I would say the call tocheck_lockin _spin_trylock{_recursive} is not strictly needed, the scenario described incheck_lockdoesn't apply when using trylock.
Never mind, this is not actually true. You can still trigger the deadlock if you mix trylock with regular locking, so I guess I will leave the shorthand usage to flush_area_mask unless I can make all the callers of send_IPI_mask use the same irq status.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel