x86/mm: Some fixes about global ASID allocation
v2 cover x86/mm: Some fixes about global ASID allocation
During the backporting of the global ASID allocation, I found some issues with the ASID range size calculations, and I'm not sure if I misread them.
Firstly, it's just a mathematical calculation. When we describe a range [a,b] as a closed interval, the size of the range should be 'b - a + 1'. As noted in the comment, the range for global ASIDs is [TLB_NR_DYN_SIZE, MAX_ASID_AVAILABLE-1], and the size of the bitmap is also defined as 'MAX_ASID_AVAILABL', Therefore, the size of the range should be 'MAX_ASID_AVAILABLE - TLB_NR_DYN_SIZE'. However, 'global_asid_available' is assigned the value of 'MAX_ASID_AVAILABLE - TLB_NR_DYN_SIZE - 1', which means one ASID will never be allocated.
The macro 'MAX_ASID_AVAILABLE', as I understand it, is used to represent the maximum valid ASID. Thus the available ASID range described in the comments is [0,MAX_ASID_AVAILABLE], which is a close interval too. So the acutal size of the range is 'MAX_ASID_AVAILABLE + 1'. But it is incorrectly used as the size of the bitmap in global ASID allocation, leading to the maximum ASID being excluded from global ASID allocation.
As referenced in [1], there is an issue: when a smaller ASID is freed and last_global_asid is not equal to 'MAX_ASID_AVAILABLE-1', the allocation will fail because the ASID space cannot be reset. This can be fixed by resetting the ASID space when an ASID rollover occurs, regardless of the value of 'last_global_asid'.
Additionally, I found that when PTI is off but 'CONFIG_MITIGATION_PAGE_TABLE_ISOLATION' is enabled (which should be true for broadcast TLB-capable hardware), only half of the ASID space is available because 'MAX_ASID_AVAILABLE' is defined as 2046. I have made some changes to expand the entire ASID space for global ASID allocation when PTI is off, but I haven't covered this in this patchset. I am considering whether we can align ASID and PCID to make the code clearer as Rik said in [0].
Note, I have only conducted some basic tests (running the 4096 tlb_flush2_threads tests to trigger ASID rollover) on the AMD platform.
v1: https://lore.kernel.org/lkml/[email protected]
[0]: https://lore.kernel.org/lkml/[email protected]
v1 -> v2:
- Add a new patch to fix the allocation failure.
- Rewrite the commit message and add Reviwed-by tag.
Hou Wenlong (4):
x86/mm: Correct the actual size of available global ASID range x86/mm: Fix wrong judgement in allocate_global_asid() x86/mm: Correct the actual size of ASID range x86/mm: Reset global ASID space when ASID rollover
arch/x86/mm/tlb.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-)
base-commit: d613f96096e48b3646217f006bcccc6ff973c428 --
2.31.1
PATCH v2 1/4 x86/mm: Correct the actual size of available global ASID range
As noted in the comment, the available ASID range for global ASID allocation is '[TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1]', which is a close interval. The size of bitmap is defined as 'MAX_ASID_AVAILABLE', so the actual available size should be '(MAX_ASID_AVAILABLE-1) - TLB_NR_DYN_ASIDS + 1'; otherwise, one ASID will leak.
--- arch/x86/mm/tlb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/tlb.c +1 -1
@@ -286,7 +286,7 @@ static DEFINE_RAW_SPINLOCK(global_asid_lock);static u16 last_global_asid =MAX_ASID_AVAILABLE ; static DECLARE_BITMAP(global_asid_used, MAX_ASID_AVAILABLE); static DECLARE_BITMAP(global_asid_freed, MAX_ASID_AVAILABLE); -static int global_asid_available =MAX_ASID_AVAILABLE -TLB_NR_DYN_ASIDS -1 ; +static int global_asid_available =MAX_ASID_AVAILABLE -TLB_NR_DYN_ASIDS ;/* * When the search for a free ASID in the global ASID space reaches
Signature · 1 line
2.31.1
Re: [PATCH v2 1/4] x86/mm: Correct the actual size of available global ASID range
Quoted 10 lines · Hou Wenlong
As noted in the comment, the available ASID range for global ASID allocation is '[
TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1]', which is a close interval. The size of bitmap is defined as 'MAX_ASID_AVAILABLE',
so the actual available size should be '(MAX_ASID_AVAILABLE-1) -TLB_NR_DYN_ASIDS+ 1'; otherwise, one ASID will
leak.Signed-off-by Hou Wenlong <[email protected]>
Signature · 1 line
All Rights Reversed.
PATCH v2 2/4 x86/mm: Fix wrong judgement in allocate_global_asid()
In reply to Hou WenlongIn allocate_global_asid(), 'global_asid_available' cannot be zero, as it has already been checked in use_global_asid(). The wrong judgment could result in the invalid ASID 'MAX_ASID_AVAILABLE' being allocated, triggering the warning in kern_pcid(). Therefore, remove the check for 'global_asid_available', since when the allocation reaches 'MAX_ASID_AVAILABLE', it must return regardless of the value of 'global_asid_available'.
--- arch/x86/mm/tlb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/tlb.c +1 -1
@@ -326,7 +326,7 @@ static u16 allocate_global_asid(void) asid =find_next_zero_bit (global_asid_used,MAX_ASID_AVAILABLE , last_global_asid); -if (asid >=MAX_ASID_AVAILABLE && !global_asid_available) { +if (asid >=MAX_ASID_AVAILABLE ) {/* This should never happen. */ VM_WARN_ONCE (1 ,"Unable to allocate global ASID despite %d available\n" , global_asid_available);
Signature · 1 line
2.31.1
PATCH v2 3/4 x86/mm: Correct the actual size of ASID range
In reply to Hou WenlongAs noted in the comment, 'MAX_ASID_AVAILABLE' represents the maximum valid ASID, i.e., the valid ASID range is '[0, MAX_ASID_AVAILABLE]'. Therefore, the actual size of the ASID range should be 'MAX_ASID_AVAILABLE + 1', as it is zero-based. However, global ASID allocation uses this value as the size of the bitmap, which results in the maximum ASID being excluded from global ASID allocation. To address this issue, redefine the 'MAX_ASID_AVAILABLE' as the size of ASID range.
--- arch/x86/mm/tlb.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/x86/mm/tlb.c +4 -5
@@ -104,18 +104,17 @@#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS)/* -* ASIDs are zero-based: 0->MAX_AVAIL_ASID are valid. -1 below to account -* for them being zero-based. Another -1 is because PCID 0 is reserved for -* use by non-PCID-aware users. +* ASIDs are zero-based: 0->MAX_ASID_AVAILABLE-1 are valid. -1 is because +* PCID 0 is reserved for use by non-PCID-aware users. */ -#define MAX_ASID_AVAILABLE ((1 << CR3_AVAIL_PCID_BITS) - 2) +#define MAX_ASID_AVAILABLE ((1 << CR3_AVAIL_PCID_BITS) - 1)/* * Given @asid, compute kPCID */ static inline u16 kern_pcid (u16 asid) { -VM_WARN_ON_ONCE (asid >MAX_ASID_AVAILABLE ); +VM_WARN_ON_ONCE (asid >=MAX_ASID_AVAILABLE );#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION /*
Signature · 1 line
2.31.1
PATCH v2 4/4 x86/mm: Reset global ASID space when ASID rollover
In reply to Hou WenlongThe global ASID allocation always searches for free ASIDs starting from the last allocated ASID and resets the ASID space when the last allocated ASID reaches the edge of the available ASID space. However, this approach fails in scenarios where the maximum ASID is allocated first and a smaller ASID is freed later. In such cases, the 'last_global_asid' may never reach the edge of the available ASID space, leading to allocation failures even when global ASIDs are available.
For example, if all ASIDs are allocated in the first round and then a smaller ASID is freed, the ASID space can be reset because 'last_global_asid' equals 'MAX_ASID_AVAILABLE - 1'. The smaller ASID can then be allocated, and 'last_global_asid' is updated accordingly. However, when this smaller ASID is freed again, the allocation will fail since 'last_global_asid' no longer equals 'MAX_ASID_AVAILABLE - 1', preventing the ASID space from being reset. To address this issue, the global ASID space should be reset based on ASID rollover rather than depending on the value of 'last_global_asid'.
--- arch/x86/mm/tlb.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/mm/tlb.c +7 -5
@@ -316,16 +316,18 @@ static void reset_global_asid_space(void)static u16 allocate_global_asid (void ) {u16 asid; +bool reset = false;lockdep_assert_held (&global_asid_lock); -/* The previous allocation hit the edge of available address space */ -if (last_global_asid >=MAX_ASID_AVAILABLE -1 ) -reset_global_asid_space (); - +restart: asid =find_next_zero_bit (global_asid_used,MAX_ASID_AVAILABLE , last_global_asid); -if (asid >=MAX_ASID_AVAILABLE ) { +if (!reset) { +reset_global_asid_space (); + reset = true; + goto restart; + }/* This should never happen. */ VM_WARN_ONCE (1 ,"Unable to allocate global ASID despite %d available\n" , global_asid_available);
Signature · 1 line
2.31.1
Re: [PATCH v2 4/4] x86/mm: Reset global ASID space when ASID rollover
Quoted 23 lines · Hou Wenlong
The global ASID allocation always searches for free ASIDs starting from
the last allocated ASID and resets the ASID space when the last allocated ASID reaches the edge of the available ASID space. However, this approach fails in scenarios where the maximum ASID is allocated first and a smaller ASID is freed later. In such cases, the 'last_global_asid' may never reach the edge of the available ASID space,
leading to allocation failures even when global ASIDs are available.For example, if all ASIDs are allocated in the first round and then a smaller ASID is freed, the ASID space can be reset because '
last_global_asid' equals 'MAX_ASID_AVAILABLE- 1'. The smaller ASID can
then be allocated, and 'last_global_asid' is updated accordingly. However, when this smaller ASID is freed again, the allocation will fail
since 'last_global_asid' no longer equals 'MAX_ASID_AVAILABLE- 1', preventing the ASID space from being reset. To address this issue, the
global ASID space should be reset based on ASID rollover rather than depending on the value of 'last_global_asid'.
Quoted 3 lines
Fixes d504d1247e36 ("x86/mm: Add global ASID allocation helperfunctions")
Signed-off-by Hou Wenlong <[email protected]>
Signature · 1 line
All Rights Reversed.