Skip to contentSkip to conversation

xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

PATCH Superseded Henry Wang 5 participants 20 messages Oct 13, 2022 – Oct 14, 2022

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Henry Wang

author
#0 lore raw

PATCH 1/1 xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M pool")

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
 xen/arch/arm/domain.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +14 -0
index 2c84e6dbbb..e40e2bcba1 100644
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
         BUG();
     }
 
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool here.
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);
+        spin_unlock(&d->arch.paging.lock);
+        goto fail;
+    }
+    spin_unlock(&d->arch.paging.lock);
+
     if ( (rc = domain_vgic_register(d, &count)) != 0 )
         goto fail;
 
Signature · 1 line
2.17.1
#1 lore raw

Hi Henry,

Quoted 2 lines · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables

Can you describe in the commit message what is the worst case scenario?

Quoted 18 lines

and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M pool")

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
  xen/arch/arm/domain.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +0 -0
index 2c84e6dbbb..e40e2bcba1 100644

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 745

Quoted 6 lines
xen/arch/arm/domain.c +3 -0
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
          BUG();
      }
 
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area

The wording suggests that this is only necessary for GICv2. But below this is done unconditionally. I am happy with this been done unconditionally, but I think this should be clarified here.

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 751

Quoted 6 lines
xen/arch/arm/domain.c +6 -0
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool here.
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);

Shouldn't this be done in p2m_fiinal_teardown() to cover so the pages will be freed anything after this call will fail (include in the caller domain_create())?

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 759

Quoted 8 lines
xen/arch/arm/domain.c +5 -0
+        spin_unlock(&d->arch.paging.lock);
+        goto fail;
+    }
+    spin_unlock(&d->arch.paging.lock);
+
      if ( (rc = domain_vgic_register(d, &count)) != 0 )
          goto fail;
 

Cheers,

Signature · 1 line
Julien Grall
author
#2 lore raw

Hi Julien,

Quoted 5 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Quoted 7 lines

Hi Henry,

Quoted 2 lines · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables

Can you describe in the commit message what is the worst case scenario?

The two pages will be consecutive but not necessarily in the same L3 page table so the worst case is 4 + 2, is that correct?

Quoted 23 lines
Quoted 6 lines

and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M

pool")

Quoted 16 lines

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
  xen/arch/arm/domain.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +0 -0
index 2c84e6dbbb..e40e2bcba1 100644
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
          BUG();
      }
 
Quoted 8 lines

xen/arch/arm/domain.c int arch_domain_create(struct domain *d,

Quoted 3 lines
xen/arch/arm/domain.c +3 -0
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2

area

The wording suggests that this is only necessary for GICv2. But below this is done unconditionally. I am happy with this been done unconditionally, but I think this should be clarified here.

Sure, I will add "For GICv3, the above-mentioned P2M mapping is not necessary, but since the allocated 16 pages here would not be lost, hence populate these pages unconditionally" if it is ok to you.

Quoted 4 lines

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 747

Quoted 3 lines
xen/arch/arm/domain.c +2 -0
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool

here.

Quoted 8 lines

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 751

Quoted 4 lines
xen/arch/arm/domain.c +4 -0
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);

Shouldn't this be done in p2m_fiinal_teardown() to cover so the pages will be freed anything after this call will fail (include in the caller domain_create())?

Hmm, yes, I will remove this p2m_set_allocation(d, 0, NULL); in v2.

Kind regards,
Henry

Quoted 14 lines

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 759

Quoted 9 lines
xen/arch/arm/domain.c +5 -0
+        spin_unlock(&d->arch.paging.lock);
+        goto fail;
+    }
+    spin_unlock(&d->arch.paging.lock);
+
      if ( (rc = domain_vgic_register(d, &count)) != 0 )
          goto fail;
 

Cheers,

--
Julien Grall

#3 lore raw
Quoted 1 line · Henry Wang

Hi Julien,

Hi Henry,

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Hi Henry,

Quoted 8 lines
Quoted 5 lines · Henry Wang
Quoted 2 lines

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables

Can you describe in the commit message what is the worst case scenario?

The two pages will be consecutive but not necessarily in the same L3 page table so the worst case is 4 + 2, is that correct?

So I agree that the worse case is 6. But I don't understand what you mean by '4 + 2' here.

Quoted 29 lines
Quoted 29 lines
Quoted 7 lines

and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M

pool")

Quoted 13 lines

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
   xen/arch/arm/domain.c | 14 ++++++++++++++
   1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +0 -0
index 2c84e6dbbb..e40e2bcba1 100644

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 745

Quoted 6 lines
xen/arch/arm/domain.c +3 -0
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
           BUG();
       }
 
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2

area

Quoted 7 lines
Quoted 3 lines

The wording suggests that this is only necessary for GICv2. But below this is done unconditionally. I am happy with this been done unconditionally, but I think this should be clarified here.

Sure, I will add "For GICv3, the above-mentioned P2M mapping is not necessary, but since the allocated 16 pages here would not be lost, hence populate these pages unconditionally" if it is ok to you.

Sounds good to me.

Quoted 10 lines
Quoted 10 lines

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 747

Quoted 4 lines
xen/arch/arm/domain.c +2 -0
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool

here.

Quoted 4 lines
xen/arch/arm/domain.c +4 -0
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);
Quoted 5 lines
Quoted 3 lines

Shouldn't this be done in p2m_fiinal_teardown() to cover so the pages will be freed anything after this call will fail (include in the caller domain_create())?

Hmm, yes, I will remove this p2m_set_allocation(d, 0, NULL); in v2.

Just to clarify, I meant that a call in p2m_final_teardown() *is* missing in p2m_final_teardown() (or wherever we decide to add).

This would make this one redundant.

Cheers,

Signature · 1 line
Julien Grall
#4 lore raw
Quoted 10 lines · Julien Grall
Quoted 9 lines · Henry Wang
Quoted 9 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Quoted 2 lines · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Can you describe in the commit message what is the worst case scenario?

The two pages will be consecutive but not necessarily in the same L3 page table so the worst case is 4 + 2, is that correct?

So I agree that the worse case is 6. But I don't understand what you mean by '4 + 2' here.

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Jan

author
#5 lore raw

Hi Jan

Quoted 20 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Quoted 4 lines · Julien Grall
Quoted 3 lines · Henry Wang
Quoted 3 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping

in

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

arch_domain_create()

Quoted 1 line · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2

area

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

when the domain is created. Considering the worst case of page tables

Can you describe in the commit message what is the worst case scenario?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The two pages will be consecutive but not necessarily in the same L3 page table so the worst case is 4 + 2, is that correct?

So I agree that the worse case is 6. But I don't understand what you mean by '4 + 2' here.

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Kind regards,
Henry

Quoted 2 lines

Jan

#6 lore raw
Quoted 23 lines · Henry Wang
Quoted 23 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Quoted 4 lines · Julien Grall
Quoted 3 lines · Henry Wang
Quoted 3 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping

in

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

arch_domain_create()

Quoted 1 line · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2

area

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

when the domain is created. Considering the worst case of page tables

Can you describe in the commit message what is the worst case scenario?

The two pages will be consecutive but not necessarily in the same L3 page table so the worst case is 4 + 2, is that correct?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

So I agree that the worse case is 6. But I don't understand what you mean by '4 + 2' here.

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Isn't that a (perhaps just minor) mistake?

Jan

author
#7 lore raw

Hi Jan,

Quoted 8 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 4 lines
Quoted 2 lines

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Isn't that a (perhaps just minor) mistake?

Not really, in the code review phase, the question of whether we include the root in the p2m pool was discussed and the conclusion at that time was not including this page for now, as this is supposed to require a lot of extra work/refactor. Probably there will be a series from my side to add the root to the pool, but at least not now.

Kind regards,
Henry

Quoted 2 lines

Jan

#8 lore raw

Hi,

Quoted 7 lines · Henry Wang
Quoted 7 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 4 lines
Quoted 2 lines

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Quoted 7 lines
Quoted 1 line

Isn't that a (perhaps just minor) mistake?

Not really, in the code review phase, the question of whether we include the root in the p2m pool was discussed and the conclusion at that time was not including this page for now, as this is supposed to require a lot of extra work/refactor. Probably there will be a series from my side to add the root to the pool, but at least not now.

The root page tables can be one of multiple concatenated pages (up to 8 pages). The P2M pool is allocating page by page and therefore wouldn't allow us to allocate contiguous pages.

Therefore, we need to handle the root differently. At which point it doesn't seem to be worth it to allocate it from the P2M pool.

Cheers,

Signature · 1 line
Julien Grall
#9 lore raw
Quoted 12 lines · Julien Grall

Hi,

Quoted 9 lines · Henry Wang
Quoted 8 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 4 lines
Quoted 2 lines

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Isn't that a (perhaps just minor) mistake?

Quoted 7 lines
Quoted 5 lines

Not really, in the code review phase, the question of whether we include the root in the p2m pool was discussed and the conclusion at that time was not including this page for now, as this is supposed to require a lot of extra work/refactor. Probably there will be a series from my side to add the root to the pool, but at least not now.

The root page tables can be one of multiple concatenated pages (up to 8 pages). The P2M pool is allocating page by page and therefore wouldn't allow us to allocate contiguous pages.

Sorry that I 'm asking this so late (I was just going through the thread) but why 8?

Quoted 5 lines

Therefore, we need to handle the root differently. At which point it doesn't seem to be worth it to allocate it from the P2M pool.

Cheers,

Signature · 1 line
Xenia
#10 lore raw

Hi Xenia,

Quoted 21 lines · Xenia Ragiadakou
Quoted 19 lines · Julien Grall

Hi,

Quoted 15 lines · Henry Wang
Quoted 9 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 5 lines
Quoted 3 lines

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N
- 1))?
Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Isn't that a (perhaps just minor) mistake?

Not really, in the code review phase, the question of whether we include the root in the p2m pool was discussed and the conclusion at that time was not including this page for now, as this is supposed to require a lot of extra work/refactor. Probably there will be a series from my side to add the root to the pool, but at least not now.

Quoted 3 lines
Quoted 1 line

The root page tables can be one of multiple concatenated pages (up to 8 pages). The P2M pool is allocating page by page and therefore wouldn't allow us to allocate contiguous pages.

Sorry that I 'm asking this so late (I was just going through the thread) but why 8?

Rather than providing an extra level of page-tables, the architecture allows you to provide multiple pages at the root level.

The number of concatenated pages depend on the maximum physical address and the start level. You can look at the table in setup_virt_paging() for more details.

Cheers,

Signature · 1 line
Julien Grall
#11 lore raw
Quoted 24 lines · Julien Grall

Hi Xenia,

Quoted 21 lines · Xenia Ragiadakou
Quoted 19 lines · Julien Grall

Hi,

Quoted 15 lines · Henry Wang
Quoted 9 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 5 lines
Quoted 3 lines

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N
- 1))?
Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Isn't that a (perhaps just minor) mistake?

Not really, in the code review phase, the question of whether we include the root in the p2m pool was discussed and the conclusion at that time was not including this page for now, as this is supposed to require a lot of extra work/refactor. Probably there will be a series from my side to add the root to the pool, but at least not now.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

The root page tables can be one of multiple concatenated pages (up to 8 pages). The P2M pool is allocating page by page and therefore wouldn't allow us to allocate contiguous pages.

Sorry that I 'm asking this so late (I was just going through the thread) but why 8?

Rather than providing an extra level of page-tables, the architecture allows you to provide multiple pages at the root level.

The number of concatenated pages depend on the maximum physical address and the start level. You can look at the table in setup_virt_paging() for more details.

I think you are referring to this:
[6] = { 52, 12/*12*/, 3, 3 },
Still I cannot understand why the maximum number of concatenated level 0 translation tables for t0sz 12 and 4KB granule is 8 and not 16?

Quoted 3 lines

Cheers,

Signature · 1 line
Xenia
#12 lore raw
Quoted 27 lines · Xenia Ragiadakou
Quoted 25 lines · Julien Grall

Hi Xenia,

Quoted 22 lines · Xenia Ragiadakou
Quoted 19 lines · Julien Grall

Hi,

Quoted 7 lines · Henry Wang
Quoted 7 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 4 lines
Quoted 2 lines

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Quoted 7 lines
Quoted 1 line

Isn't that a (perhaps just minor) mistake?

Not really, in the code review phase, the question of whether we include the root in the p2m pool was discussed and the conclusion at that time was not including this page for now, as this is supposed to require a lot of extra work/refactor. Probably there will be a series from my side to add the root to the pool, but at least not now.

The root page tables can be one of multiple concatenated pages (up to 8 pages). The P2M pool is allocating page by page and therefore wouldn't allow us to allocate contiguous pages.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Sorry that I 'm asking this so late (I was just going through the thread) but why 8?

Rather than providing an extra level of page-tables, the architecture allows you to provide multiple pages at the root level.

The number of concatenated pages depend on the maximum physical address and the start level. You can look at the table in setup_virt_paging() for more details.

I think you are referring to this:
[6] = { 52,      12/*12*/,  3,          3 },
Still I cannot understand why the maximum number of concatenated level 0 translation tables for t0sz 12 and 4KB granule is 8 and not 16?

Let me explain. Initially I got confused because according to the manual up to 16 translation tables can be concatenated. Then, looking at the code you have pointed out, I understood that you were referring to the maximum root order in the table below.

pa_range_info[] = {
        /* T0SZ minimum and SL0 maximum from ARM DDI 0487H.a Table D5-6 */
        /*      PA size, t0sz(min), root-order, sl0(max) */
        [0] = { 32,      32/*32*/,  0,          1 },
        [1] = { 36,      28/*28*/,  0,          1 },
        [2] = { 40,      24/*24*/,  1,          1 },
        [3] = { 42,      22/*22*/,  3,          1 },
        [4] = { 44,      20/*20*/,  0,          2 },
        [5] = { 48,      16/*16*/,  0,          2 },
        [6] = { 52,      12/*12*/,  3,          3 },
        [7] = { 0 }  /* Invalid */
    };

I think that the pa range info for the 52-bit pa range is wrong. IMO, it could be either
[6] = { 52, 12/*12*/, 0, 3 } i.e single level -1 table
or
[6] = { 52, 12/*12*/, 4, 2 } i.e 16 concatenated level 0 tables
Hope I am not totally out of context.

Quoted 4 lines
Quoted 3 lines

Cheers,

Signature · 1 line
Xenia
#13 lore raw

Hi Xenia,

Quoted 53 lines · Xenia Ragiadakou
Quoted 27 lines · Xenia Ragiadakou
Quoted 25 lines · Julien Grall

Hi Xenia,

Quoted 22 lines · Xenia Ragiadakou
Quoted 19 lines · Julien Grall

Hi,

Quoted 7 lines · Henry Wang
Quoted 7 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 4 lines
Quoted 2 lines

Assuming you have 4 (N) page table levels, isn't it 7 (1 + 2 * (N - 1))? Or is the root table not taken from the p2m pool?

Correct, on arm the root is not taken from the pool.

Quoted 7 lines
Quoted 1 line

Isn't that a (perhaps just minor) mistake?

Not really, in the code review phase, the question of whether we include the root in the p2m pool was discussed and the conclusion at that time was not including this page for now, as this is supposed to require a lot of extra work/refactor. Probably there will be a series from my side to add the root to the pool, but at least not now.

The root page tables can be one of multiple concatenated pages (up to 8 pages). The P2M pool is allocating page by page and therefore wouldn't allow us to allocate contiguous pages.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Sorry that I 'm asking this so late (I was just going through the thread) but why 8?

Rather than providing an extra level of page-tables, the architecture allows you to provide multiple pages at the root level.

The number of concatenated pages depend on the maximum physical address and the start level. You can look at the table in setup_virt_paging() for more details.

I think you are referring to this:
[6] = { 52,      12/*12*/,  3,          3 },
Still I cannot understand why the maximum number of concatenated level 0 translation tables for t0sz 12 and 4KB granule is 8 and not 16?

Let me explain. Initially I got confused because according to the manual up to 16 translation tables can be concatenated. Then, looking at the code you have pointed out, I understood that you were referring to the maximum root order in the table below.

pa_range_info[] = {
        /* T0SZ minimum and SL0 maximum from ARM DDI 0487H.a Table D5-6 */ /*      PA size, t0sz(min), root-order, sl0(max) */
        [0] = { 32,      32/*32*/,  0,          1 },
        [1] = { 36,      28/*28*/,  0,          1 },
        [2] = { 40,      24/*24*/,  1,          1 },
        [3] = { 42,      22/*22*/,  3,          1 },
        [4] = { 44,      20/*20*/,  0,          2 },
        [5] = { 48,      16/*16*/,  0,          2 },
        [6] = { 52,      12/*12*/,  3,          3 },
        [7] = { 0 }  /* Invalid */
    };

Quoted 1 line

I think that the pa range info for the 52-bit pa range is wrong.

I was about to say the same and note that in Xen we don't support 52-bits. So effectively those values are ignored for now.

Quoted 2 lines

IMO, it could be either
[6] = { 52,      12/*12*/,  0,          3 } i.e single level -1 table

I don't think Xen is ready to support -1 for the root level. So...

Quoted 2 lines

or > [6] = { 52,      12/*12*/,  4,          2 } i.e 16 concatenated level 0 tables

... this would be better. But this is not related to this patch directly. So any fix should be sent separately.

Cheers,

Signature · 1 line
Julien Grall
#14 lore raw
In reply to Julien Grall

Hi Henry,

Quoted 14 lines · Julien Grall
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines

+     * when the domain is created. Considering the worst case for page +     * tables and keep a buffer, populate 16 pages to the P2M pages pool

here.

Quoted 4 lines

+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 ) +    {
+        p2m_set_allocation(d, 0, NULL);

Shouldn't this be done in p2m_fiinal_teardown() to cover so the pages will be freed anything after this call will fail (include in the caller domain_create())?

Quoted 5 lines
Quoted 1 line

Hmm, yes, I will remove this p2m_set_allocation(d, 0, NULL); in v2.

Just to clarify, I meant that a call in p2m_final_teardown() *is* missing in p2m_final_teardown() (or wherever we decide to add).

This would make this one redundant.

While chatting with you on IRC, I realized that a call to p2m_set_allocation() will only freed unused P2M pages. If some of them are in the P2M then they would be skipped.

This means we also need to call p2m_teardown() (which would need to be optionally preemptible).

Cheers,

Signature · 1 line
Julien Grall
#15 lore raw
In reply to Henry Wang
Quoted 26 lines · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M pool")

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
 xen/arch/arm/domain.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +3 -0
index 2c84e6dbbb..e40e2bcba1 100644
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
         BUG();
     }
 
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 755

Quoted 10 lines
xen/arch/arm/domain.c +10 -0
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool here.
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);
+        spin_unlock(&d->arch.paging.lock);
+        goto fail;
+    }
+    spin_unlock(&d->arch.paging.lock);

Generally, this would be better written as

spin_lock();
if ( rc = p2m_set_allocation(16) )
    p2m_set_allocation(0)
spin_unlock();

if ( rc )
    goto fail;

to reduce the number of spin_unlock() calls and make the error paths more clear.  However...

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 759

Quoted 4 lines
xen/arch/arm/domain.c +1 -0
+
     if ( (rc = domain_vgic_register(d, &count)) != 0 )
         goto fail;
 

... you've got a problem on this error path, so the set allocation to 0 needs to be in the fail: path with suitable locking.

There are perhaps better ways of doing it in 4.15(?) and later, but not in earlier versions.  As this is a fix to a bug in a security patch, simplicity is generally the better approach.

~Andrew

author
#16 lore raw

Hi Andrew,

Quoted 53 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Quoted 7 lines · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M

pool")

Quoted 13 lines

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
 xen/arch/arm/domain.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +0 -0
index 2c84e6dbbb..e40e2bcba1 100644

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 745

Quoted 6 lines
xen/arch/arm/domain.c +3 -0
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
         BUG();
     }
 
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2

area

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 747

Quoted 2 lines
xen/arch/arm/domain.c +2 -0
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool

here.

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 755

Quoted 8 lines
xen/arch/arm/domain.c +8 -0
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);
+        spin_unlock(&d->arch.paging.lock);
+        goto fail;
+    }
+    spin_unlock(&d->arch.paging.lock);

Generally, this would be better written as

spin_lock();
if ( rc = p2m_set_allocation(16) )
    p2m_set_allocation(0)
spin_unlock();

Quoted 5 lines

if ( rc )
    goto fail;

to reduce the number of spin_unlock() calls and make the error paths more clear.  However...

I think in Arm's arch_domain_create(), all the error handling are the same style using:

if ( (rc = <function>) !=0 )

    goto fail;

and we need to keep them the same? But I think I will drop the p2m_set_allocation(d, 0, NULL); as the arch_domain_destroy(d) in

fail:

    d->is_dying = DOMDYING_dead;
    arch_domain_destroy(d);

will clean-up the pool.

Kind regards,
Henry

Quoted 14 lines

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 759

Quoted 5 lines
xen/arch/arm/domain.c +1 -0
+
     if ( (rc = domain_vgic_register(d, &count)) != 0 )
         goto fail;
 

... you've got a problem on this error path, so the set allocation to 0 needs to be in the fail: path with suitable locking.

There are perhaps better ways of doing it in 4.15(?) and later, but not in earlier versions.  As this is a fix to a bug in a security patch, simplicity is generally the better approach.

~Andrew

#17 lore raw
Quoted 61 lines · Henry Wang

Hi Andrew,

Quoted 53 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Quoted 7 lines · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M

pool")

Quoted 13 lines

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
  xen/arch/arm/domain.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +0 -0
index 2c84e6dbbb..e40e2bcba1 100644

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 745

Quoted 6 lines
xen/arch/arm/domain.c +3 -0
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
          BUG();
      }
 
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2

area

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 747

Quoted 2 lines
xen/arch/arm/domain.c +2 -0
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool

here.

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 755

Quoted 8 lines
xen/arch/arm/domain.c +8 -0
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);
+        spin_unlock(&d->arch.paging.lock);
+        goto fail;
+    }
+    spin_unlock(&d->arch.paging.lock);

Generally, this would be better written as

spin_lock();
if ( rc = p2m_set_allocation(16) )
    p2m_set_allocation(0)
spin_unlock();

Quoted 5 lines

if ( rc )
    goto fail;

to reduce the number of spin_unlock() calls and make the error paths more clear.  However...

Quoted 7 lines

I think in Arm's arch_domain_create(), all the error handling are the same style using:

if ( (rc = <function>) !=0 )

     goto fail;

and we need to keep them the same?

We don't have too. I agree with Andrew's point. How about the following:

spin_lock()
rc = ...
spin_unlock()

if ( rc )
goto fail;

Cheers,

Signature · 1 line
Julien Grall
author
#18 lore raw

Hi Julien,

Quoted 12 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH] xen/arm: p2m: Populate pages for GICv2 mapping in arch_domain_create()

Quoted 7 lines

I think in Arm's arch_domain_create(), all the error handling are the same style using:

if ( (rc = <function>) !=0 )

     goto fail;

and we need to keep them the same?

Quoted 8 lines

We don't have too. I agree with Andrew's point. How about the following:

spin_lock()
rc = ...
spin_unlock()

if ( rc )
goto fail;

Yep this is my current method of local v2. Thanks for the clarification. Working on the p2m_teardown part now...

Kind regards,
Henry

Quoted 5 lines

Cheers,

--
Julien Grall

#19 lore raw
In reply to Andrew Cooper

Hi Andrew,

Quoted 56 lines · Andrew Cooper
Quoted 26 lines · Henry Wang

Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area when the domain is created. Considering the worst case of page tables and keep a buffer, populate 16 pages as the default value to the P2M pages pool in arch_domain_create() at the domain creation stage to satisfy the GICv2 requirement.

Fixes cbea5a1149ca ("xen/arm: Allocate and free P2M pages from the P2M pool")

---
This should also be backported to 4.13, 4.14, 4.15 and 4.16.

---
  xen/arch/arm/domain.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
xen/arch/arm/domain.c +3 -0
index 2c84e6dbbb..e40e2bcba1 100644
@@ -740,6 +740,20 @@ int arch_domain_create(struct domain *d,
          BUG();
      }
 
+    spin_lock(&d->arch.paging.lock);
+    /*
+     * Hardware using GICv2 needs to create a P2M mapping of 8KB GICv2 area

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 755

Quoted 10 lines
xen/arch/arm/domain.c +10 -0
+     * when the domain is created. Considering the worst case for page
+     * tables and keep a buffer, populate 16 pages to the P2M pages pool here.
+     */
+    if ( (rc = p2m_set_allocation(d, 16, NULL)) != 0 )
+    {
+        p2m_set_allocation(d, 0, NULL);
+        spin_unlock(&d->arch.paging.lock);
+        goto fail;
+    }
+    spin_unlock(&d->arch.paging.lock);

Generally, this would be better written as

spin_lock();
if ( rc = p2m_set_allocation(16) )
    p2m_set_allocation(0)
spin_unlock();

if ( rc )
    goto fail;

to reduce the number of spin_unlock() calls and make the error paths more clear.  However...

xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 759

Quoted 4 lines
xen/arch/arm/domain.c +1 -0
+
      if ( (rc = domain_vgic_register(d, &count)) != 0 )
          goto fail;
 
Quoted 6 lines

... you've got a problem on this error path, so the set allocation to 0 needs to be in the fail: path with suitable locking.

There are perhaps better ways of doing it in 4.15(?) and later, but not in earlier versions.  As this is a fix to a bug in a security patch, simplicity is generally the better approach.

I guess you are referring to domain_teardown()? I think it may end up to be quite large because we would have to move other bits of the arch_domain_destroy() in domain_teardown().

It is also not clear whether part of domain_relinquish_memory() would need to be moved there as well.

So this sounds more like some work for post-4.17.

Cheers,

Signature · 1 line
Julien Grall