Skip to contentSkip to conversation

x86/mm: Some fixes about global ASID allocation

PATCH In review Hou Wenlong 2 participants 7 messages Jan 20

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover x86/mm: Some fixes about global ASID allocation

During the backporting of the global ASID allocation, I found some issues with the ASID range size calculations, and I'm not sure if I misread them.

Firstly, it's just a mathematical calculation. When we describe a range [a,b] as a closed interval, the size of the range should be 'b - a + 1'. As noted in the comment, the range for global ASIDs is [TLB_NR_DYN_SIZE, MAX_ASID_AVAILABLE-1], and the size of the bitmap is also defined as 'MAX_ASID_AVAILABL', Therefore, the size of the range should be 'MAX_ASID_AVAILABLE - TLB_NR_DYN_SIZE'. However, 'global_asid_available' is assigned the value of 'MAX_ASID_AVAILABLE - TLB_NR_DYN_SIZE - 1', which means one ASID will never be allocated.

The macro 'MAX_ASID_AVAILABLE', as I understand it, is used to represent the maximum valid ASID. Thus the available ASID range described in the comments is [0,MAX_ASID_AVAILABLE], which is a close interval too. So the acutal size of the range is 'MAX_ASID_AVAILABLE + 1'. But it is incorrectly used as the size of the bitmap in global ASID allocation, leading to the maximum ASID being excluded from global ASID allocation.

As referenced in [1], there is an issue: when a smaller ASID is freed and last_global_asid is not equal to 'MAX_ASID_AVAILABLE-1', the allocation will fail because the ASID space cannot be reset. This can be fixed by resetting the ASID space when an ASID rollover occurs, regardless of the value of 'last_global_asid'.

Additionally, I found that when PTI is off but 'CONFIG_MITIGATION_PAGE_TABLE_ISOLATION' is enabled (which should be true for broadcast TLB-capable hardware), only half of the ASID space is available because 'MAX_ASID_AVAILABLE' is defined as 2046. I have made some changes to expand the entire ASID space for global ASID allocation when PTI is off, but I haven't covered this in this patchset. I am considering whether we can align ASID and PCID to make the code clearer as Rik said in [0].

Note, I have only conducted some basic tests (running the 4096 tlb_flush2_threads tests to trigger ASID rollover) on the AMD platform.

v1: https://lore.kernel.org/lkml/[email protected]

[0]: https://lore.kernel.org/lkml/[email protected]

v1 -> v2:
- Add a new patch to fix the allocation failure.
- Rewrite the commit message and add Reviwed-by tag.

Hou Wenlong (4):
x86/mm: Correct the actual size of available global ASID range x86/mm: Fix wrong judgement in allocate_global_asid() x86/mm: Correct the actual size of ASID range x86/mm: Reset global ASID space when ASID rollover

 arch/x86/mm/tlb.c | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)

base-commit: d613f96096e48b3646217f006bcccc6ff973c428 --
2.31.1

#3 lore raw

PATCH v2 2/4 x86/mm: Fix wrong judgement in allocate_global_asid()

In allocate_global_asid(), 'global_asid_available' cannot be zero, as it has already been checked in use_global_asid(). The wrong judgment could result in the invalid ASID 'MAX_ASID_AVAILABLE' being allocated, triggering the warning in kern_pcid(). Therefore, remove the check for 'global_asid_available', since when the allocation reaches 'MAX_ASID_AVAILABLE', it must return regardless of the value of 'global_asid_available'.

Fixes d504d1247e36 ("x86/mm: Add global ASID allocation helper functions")
---
 arch/x86/mm/tlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/tlb.c +1 -1
index 42b025e2f825..a1e217a382d1 100644
@@ -326,7 +326,7 @@ static u16 allocate_global_asid(void)
 
 	asid = find_next_zero_bit(global_asid_used, MAX_ASID_AVAILABLE, last_global_asid);
 
-	if (asid >= MAX_ASID_AVAILABLE && !global_asid_available) {
+	if (asid >= MAX_ASID_AVAILABLE) {
 		/* This should never happen. */
 		VM_WARN_ONCE(1, "Unable to allocate global ASID despite %d available\n",
 				global_asid_available);
Signature · 1 line
2.31.1
#5 lore raw

PATCH v2 4/4 x86/mm: Reset global ASID space when ASID rollover

In reply to Hou Wenlong

The global ASID allocation always searches for free ASIDs starting from the last allocated ASID and resets the ASID space when the last allocated ASID reaches the edge of the available ASID space. However, this approach fails in scenarios where the maximum ASID is allocated first and a smaller ASID is freed later. In such cases, the 'last_global_asid' may never reach the edge of the available ASID space, leading to allocation failures even when global ASIDs are available.

For example, if all ASIDs are allocated in the first round and then a smaller ASID is freed, the ASID space can be reset because 'last_global_asid' equals 'MAX_ASID_AVAILABLE - 1'. The smaller ASID can then be allocated, and 'last_global_asid' is updated accordingly. However, when this smaller ASID is freed again, the allocation will fail since 'last_global_asid' no longer equals 'MAX_ASID_AVAILABLE - 1', preventing the ASID space from being reset. To address this issue, the global ASID space should be reset based on ASID rollover rather than depending on the value of 'last_global_asid'.

Fixes d504d1247e36 ("x86/mm: Add global ASID allocation helper functions")
---
 arch/x86/mm/tlb.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/mm/tlb.c +7 -5
index a7dbf784efd9..0f98b78a48cc 100644
@@ -316,16 +316,18 @@ static void reset_global_asid_space(void)
 static u16 allocate_global_asid(void)
 {
 	u16 asid;
+	bool reset = false;
 
 	lockdep_assert_held(&global_asid_lock);
 
-	/* The previous allocation hit the edge of available address space */
-	if (last_global_asid >= MAX_ASID_AVAILABLE - 1)
-		reset_global_asid_space();
-
+restart:
 	asid = find_next_zero_bit(global_asid_used, MAX_ASID_AVAILABLE, last_global_asid);
-
 	if (asid >= MAX_ASID_AVAILABLE) {
+		if (!reset) {
+			reset_global_asid_space();
+			reset = true;
+			goto restart;
+		}
 		/* This should never happen. */
 		VM_WARN_ONCE(1, "Unable to allocate global ASID despite %d available\n",
 				global_asid_available);
Signature · 1 line
2.31.1
#1 lore raw

PATCH v2 1/4 x86/mm: Correct the actual size of available global ASID range

In reply to Hou Wenlong

As noted in the comment, the available ASID range for global ASID allocation is '[TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1]', which is a close interval. The size of bitmap is defined as 'MAX_ASID_AVAILABLE', so the actual available size should be '(MAX_ASID_AVAILABLE-1) - TLB_NR_DYN_ASIDS + 1'; otherwise, one ASID will leak.

---
 arch/x86/mm/tlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/tlb.c +1 -1
index 621e09d049cb..42b025e2f825 100644
@@ -286,7 +286,7 @@ static DEFINE_RAW_SPINLOCK(global_asid_lock);
 static u16 last_global_asid = MAX_ASID_AVAILABLE;
 static DECLARE_BITMAP(global_asid_used, MAX_ASID_AVAILABLE);
 static DECLARE_BITMAP(global_asid_freed, MAX_ASID_AVAILABLE);
-static int global_asid_available = MAX_ASID_AVAILABLE - TLB_NR_DYN_ASIDS - 1;
+static int global_asid_available = MAX_ASID_AVAILABLE - TLB_NR_DYN_ASIDS;
 
 /*
  * When the search for a free ASID in the global ASID space reaches
Signature · 1 line
2.31.1
#4 lore raw

PATCH v2 3/4 x86/mm: Correct the actual size of ASID range

In reply to Hou Wenlong

As noted in the comment, 'MAX_ASID_AVAILABLE' represents the maximum valid ASID, i.e., the valid ASID range is '[0, MAX_ASID_AVAILABLE]'. Therefore, the actual size of the ASID range should be 'MAX_ASID_AVAILABLE + 1', as it is zero-based. However, global ASID allocation uses this value as the size of the bitmap, which results in the maximum ASID being excluded from global ASID allocation. To address this issue, redefine the 'MAX_ASID_AVAILABLE' as the size of ASID range.

---
 arch/x86/mm/tlb.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
arch/x86/mm/tlb.c +4 -5
index a1e217a382d1..a7dbf784efd9 100644
@@ -104,18 +104,17 @@
 #define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS)
 
 /*
- * ASIDs are zero-based: 0->MAX_AVAIL_ASID are valid.  -1 below to account
- * for them being zero-based.  Another -1 is because PCID 0 is reserved for
- * use by non-PCID-aware users.
+ * ASIDs are zero-based: 0->MAX_ASID_AVAILABLE-1 are valid.  -1 is because
+ * PCID 0 is reserved for use by non-PCID-aware users.
  */
-#define MAX_ASID_AVAILABLE ((1 << CR3_AVAIL_PCID_BITS) - 2)
+#define MAX_ASID_AVAILABLE ((1 << CR3_AVAIL_PCID_BITS) - 1)
 
 /*
  * Given @asid, compute kPCID
  */
 static inline u16 kern_pcid(u16 asid)
 {
-	VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
+	VM_WARN_ON_ONCE(asid >= MAX_ASID_AVAILABLE);
 
 #ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION
 	/*
Signature · 1 line
2.31.1
Reviewed-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] x86/mm: Correct the actual size of available global ASID range

In reply to Hou Wenlong
Quoted 10 lines · Hou Wenlong

As noted in the comment, the available ASID range for global ASID allocation is '[TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1]', which is a close interval. The size of bitmap is defined as 'MAX_ASID_AVAILABLE',
so the actual available size should be '(MAX_ASID_AVAILABLE-1) - TLB_NR_DYN_ASIDS + 1'; otherwise, one ASID will
leak.

Signature · 1 line
All Rights Reversed.
Reviewed-by on v2 4/4
#6 lore raw

Re: [PATCH v2 4/4] x86/mm: Reset global ASID space when ASID rollover

In reply to Hou Wenlong
Quoted 23 lines · Hou Wenlong

The global ASID allocation always searches for free ASIDs starting from
the last allocated ASID and resets the ASID space when the last allocated ASID reaches the edge of the available ASID space. However, this approach fails in scenarios where the maximum ASID is allocated first and a smaller ASID is freed later. In such cases, the 'last_global_asid' may never reach the edge of the available ASID space,
leading to allocation failures even when global ASIDs are available.

For example, if all ASIDs are allocated in the first round and then a smaller ASID is freed, the ASID space can be reset because 'last_global_asid' equals 'MAX_ASID_AVAILABLE - 1'. The smaller ASID can
then be allocated, and 'last_global_asid' is updated accordingly. However, when this smaller ASID is freed again, the allocation will fail
since 'last_global_asid' no longer equals 'MAX_ASID_AVAILABLE - 1', preventing the ASID space from being reset. To address this issue, the
global ASID space should be reset based on ASID rollover rather than depending on the value of 'last_global_asid'.

Quoted 3 lines
Fixes d504d1247e36 ("x86/mm: Add global ASID allocation helper

functions")

Signature · 1 line
All Rights Reversed.