Skip to contentSkip to conversation

x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

PATCH In review Roger Pau Monne 3 participants 11 messages Sep 9, 2020 – Sep 10, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Fixes 81fd0d3ca4b2cd ('x86/hvm: simplify 'mmio_direct' check in epte_get_entry_emt()')

---
Cc: Paul Durrant <[email protected]>

---
 xen/arch/x86/hvm/mtrr.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/mtrr.c +3 -1
index fb051d59c3..33b1dd9052 100644
@@ -829,7 +829,9 @@ int epte_get_entry_emt(struct domain *d, unsigned long gfn, mfn_t mfn,
 
     for ( i = 0; i < (1ul << order); i++ )
     {
-        if ( is_special_page(mfn_to_page(mfn_add(mfn, i))) )
+        const struct page_info *page = mfn_to_page(mfn_add(mfn, i));
+
+        if ( is_special_page(page) && page_get_owner(page) != dom_io )
         {
             if ( order )
                 return -1;
Signature · 1 line
2.28.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 18 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 09 September 2020 15:51
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH] x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Quoted 8 lines

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Fixes 81fd0d3ca4b2cd ('x86/hvm: simplify 'mmio_direct' check in epte_get_entry_emt()')

---
Cc: Paul Durrant <[email protected]>

This looks like the right thing to do...

#2 lore raw
In reply to Roger Pau Monne
Quoted 6 lines · Roger Pau Monne

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

Quoted 7 lines

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Did you consider the alternative of not marking those pages as Xen heap ones? In particular when looking at it from this angle I consider it at least odd for non-RAM (or more precisely non-heap) pages to get marked this way. And I can't currently see anything requiring them to be marked as such - them being owned by DomIO is all that's needed as it seems.

Jan

#3 lore raw
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Quoted 8 lines
Quoted 3 lines

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Did you consider the alternative of not marking those pages as Xen heap ones? In particular when looking at it from this angle I consider it at least odd for non-RAM (or more precisely non-heap) pages to get marked this way.

I wasn't sure whether this could cause issues in other places of the code that would rely on this fact and such change seemed more risky IMO.

Quoted 3 lines

And I can't currently see anything
requiring them to be marked as such - them being owned by DomIO is all that's needed as it seems.

Should those pages then simply be assigned to dom_io and set the appropriate flags (PGC_allocated | 1), or should share_xen_page_with_guest be modified to not set the PGC_xen_heap flag?

I see that such addition was done in a2b4b8c2041, but I'm afraid I don't fully understand why share_xen_page_with_guest needs to mark pages as Xen heap.

Thanks, Roger.

#4 lore raw
Quoted 37 lines

-----Original Message-----
From: Roger Pau Monné <[email protected]>
Sent: 10 September 2020 11:35
To: Jan Beulich <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>
Subject: Re: [PATCH] x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

Quoted 11 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Quoted 8 lines
Quoted 3 lines

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Did you consider the alternative of not marking those pages as Xen heap ones? In particular when looking at it from this angle I consider it at least odd for non-RAM (or more precisely non-heap) pages to get marked this way.

I wasn't sure whether this could cause issues in other places of the code that would rely on this fact and such change seemed more risky IMO.

Quoted 3 lines

And I can't currently see anything
requiring them to be marked as such - them being owned by DomIO is all that's needed as it seems.

Quoted 9 lines

Should those pages then simply be assigned to dom_io and set the appropriate flags (PGC_allocated | 1), or should share_xen_page_with_guest be modified to not set the PGC_xen_heap flag?

I see that such addition was done in a2b4b8c2041, but I'm afraid I don't fully understand why share_xen_page_with_guest needs to mark pages as Xen heap.

In general they are not marked Xen heap then they will be considered domheap and will go through the normal free-ing path on domain destruction. Of course this doesn't apply for a system domain that never gets destroyed.

Paul

#5 lore raw
Quoted 42 lines · Paul Durrant
Quoted 42 lines

-----Original Message-----
From: Roger Pau Monné <[email protected]>
Sent: 10 September 2020 11:35
To: Jan Beulich <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>
Subject: Re: [PATCH] x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

Quoted 11 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Quoted 8 lines
Quoted 3 lines

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Did you consider the alternative of not marking those pages as Xen heap ones? In particular when looking at it from this angle I consider it at least odd for non-RAM (or more precisely non-heap) pages to get marked this way.

I wasn't sure whether this could cause issues in other places of the code that would rely on this fact and such change seemed more risky IMO.

Quoted 3 lines

And I can't currently see anything
requiring them to be marked as such - them being owned by DomIO is all that's needed as it seems.

Should those pages then simply be assigned to dom_io and set the appropriate flags (PGC_allocated | 1), or should share_xen_page_with_guest be modified to not set the PGC_xen_heap flag?

Quoted 6 lines
Quoted 4 lines

I see that such addition was done in a2b4b8c2041, but I'm afraid I don't fully understand why share_xen_page_with_guest needs to mark pages as Xen heap.

In general they are not marked Xen heap then they will be considered domheap and will go through the normal free-ing path on domain destruction. Of course this doesn't apply for a system domain that never gets destroyed.

Hm, OK, the original commit (a2b4b8c2041) mentions that marking them as Xen heap is done to signal that the virtual address for those is not needed (and not available?).

For the MMIO regions I'm not sure it matters much, since those are not assigned to the domain, but just mapped into it. The MMIO regions are not shared with the domain accessing them, but rather with dom_io.

It's still not clear to me what option would be better: modify share_xen_page_with_guest to not mark pages as Xen heap, or implement something different to assign MMIO pages to dom_io without setting the Xen heap flag.

Thanks, Roger.

#6 lore raw
Quoted 65 lines

-----Original Message-----
From: Roger Pau Monné <[email protected]>
Sent: 10 September 2020 12:06
To: [email protected]
Cc: 'Jan Beulich' <[email protected]>; [email protected]; 'Andrew Cooper' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: Re: [PATCH] x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Roger Pau Monné <[email protected]>
Sent: 10 September 2020 11:35
To: Jan Beulich <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Wei Liu

<[email protected]>;

Quoted 37 lines
Quoted 37 lines

Paul Durrant <[email protected]>
Subject: Re: [PATCH] x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

Quoted 11 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Quoted 8 lines
Quoted 3 lines

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Did you consider the alternative of not marking those pages as Xen heap ones? In particular when looking at it from this angle I consider it at least odd for non-RAM (or more precisely non-heap) pages to get marked this way.

I wasn't sure whether this could cause issues in other places of the code that would rely on this fact and such change seemed more risky IMO.

Quoted 3 lines

And I can't currently see anything
requiring them to be marked as such - them being owned by DomIO is all that's needed as it seems.

Should those pages then simply be assigned to dom_io and set the appropriate flags (PGC_allocated | 1), or should share_xen_page_with_guest be modified to not set the PGC_xen_heap flag?

Quoted 6 lines
Quoted 4 lines

I see that such addition was done in a2b4b8c2041, but I'm afraid I don't fully understand why share_xen_page_with_guest needs to mark pages as Xen heap.

In general they are not marked Xen heap then they will be considered domheap and will go through the

normal free-ing path on domain destruction. Of course this doesn't apply for a system domain that never gets destroyed.

Hm, OK, the original commit (a2b4b8c2041) mentions that marking them as Xen heap is done to signal that the virtual address for those is not needed (and not available?).

Quoted 8 lines

For the MMIO regions I'm not sure it matters much, since those are not assigned to the domain, but just mapped into it. The MMIO regions are not shared with the domain accessing them, but rather with dom_io.

It's still not clear to me what option would be better: modify share_xen_page_with_guest to not mark pages as Xen heap, or implement something different to assign MMIO pages to dom_io without setting the Xen heap flag.

Would using map_mmio_regions() work?

Paul

Quoted 2 lines

Thanks, Roger.

#7 lore raw
Quoted 69 lines · Paul Durrant
Quoted 69 lines

-----Original Message-----
From: Roger Pau Monné <[email protected]>
Sent: 10 September 2020 12:06
To: [email protected]
Cc: 'Jan Beulich' <[email protected]>; [email protected]; 'Andrew Cooper' <[email protected]>; 'Wei Liu' <[email protected]>
Subject: Re: [PATCH] x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Roger Pau Monné <[email protected]>
Sent: 10 September 2020 11:35
To: Jan Beulich <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Wei Liu

<[email protected]>;

Quoted 37 lines
Quoted 37 lines

Paul Durrant <[email protected]>
Subject: Re: [PATCH] x86/hvm: don't treat MMIO pages as special ones regarding cache attributes

Quoted 11 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Quoted 8 lines
Quoted 3 lines

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Did you consider the alternative of not marking those pages as Xen heap ones? In particular when looking at it from this angle I consider it at least odd for non-RAM (or more precisely non-heap) pages to get marked this way.

I wasn't sure whether this could cause issues in other places of the code that would rely on this fact and such change seemed more risky IMO.

Quoted 3 lines

And I can't currently see anything
requiring them to be marked as such - them being owned by DomIO is all that's needed as it seems.

Should those pages then simply be assigned to dom_io and set the appropriate flags (PGC_allocated | 1), or should share_xen_page_with_guest be modified to not set the PGC_xen_heap flag?

Quoted 6 lines
Quoted 4 lines

I see that such addition was done in a2b4b8c2041, but I'm afraid I don't fully understand why share_xen_page_with_guest needs to mark pages as Xen heap.

In general they are not marked Xen heap then they will be considered domheap and will go through the

normal free-ing path on domain destruction. Of course this doesn't apply for a system domain that never gets destroyed.

Hm, OK, the original commit (a2b4b8c2041) mentions that marking them as Xen heap is done to signal that the virtual address for those is not needed (and not available?).

For the MMIO regions I'm not sure it matters much, since those are not assigned to the domain, but just mapped into it. The MMIO regions are not shared with the domain accessing them, but rather with dom_io.

Quoted 6 lines
Quoted 4 lines

It's still not clear to me what option would be better: modify share_xen_page_with_guest to not mark pages as Xen heap, or implement something different to assign MMIO pages to dom_io without setting the Xen heap flag.

Would using map_mmio_regions() work?

If you mean using map_mmio_regions against dom_io, then no, that won't work because dom_io is not a translated domain so that would be a noop.

Also I don't think map_mmio_regions takes any reference or changes the ownership of MMIO pages, so even if it could be used against a non-translated domain it's likely not what we want.

Thanks, Roger.

#8 lore raw
In reply to Roger Pau Monné
Quoted 4 lines · Roger Pau Monné

It's still not clear to me what option would be better: modify share_xen_page_with_guest to not mark pages as Xen heap, or implement something different to assign MMIO pages to dom_io without setting the Xen heap flag.

static void __init share_io_page(struct page_info *page)

{
    set_gpfn_from_mfn(mfn_x(page_to_mfn(page)), INVALID_M2P_ENTRY);

    /* The incremented type count pins as writable. */
    page->u.inuse.type_info = PGT_writable_page | PGT_validated | 1;

    page_set_owner(page, dom_io);

    page->count_info |= PGC_allocated | 1;
}

is of course much shorter than share_xen_page_with_guest(), but I'm nevertheless uncertain whether simply making conditional the setting of PGC_xen_heap there isn't the easier route. Of course, not pointlessly acquiring and releasing a lock has its own appeal.

Jan

#9 lore raw
Quoted 18 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

It's still not clear to me what option would be better: modify share_xen_page_with_guest to not mark pages as Xen heap, or implement something different to assign MMIO pages to dom_io without setting the Xen heap flag.

static void __init share_io_page(struct page_info *page)

{
    set_gpfn_from_mfn(mfn_x(page_to_mfn(page)), INVALID_M2P_ENTRY);

    /* The incremented type count pins as writable. */
    page->u.inuse.type_info = PGT_writable_page | PGT_validated | 1;

    page_set_owner(page, dom_io);

    page->count_info |= PGC_allocated | 1;
}
Quoted 5 lines

is of course much shorter than share_xen_page_with_guest(), but I'm nevertheless uncertain whether simply making conditional the setting of PGC_xen_heap there isn't the easier route. Of course, not pointlessly acquiring and releasing a lock has its own appeal.

I've went over the existing is_special_page users and I think it's fine for MMIO regions to not be marked as special pages.

Will send a new patch.

Roger.

#10 lore raw
In reply to Roger Pau Monné
Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Jan Beulich
Quoted 13 lines · Roger Pau Monne

MMIO regions below the maximum address on the memory map can have a backing page struct that's shared with dom_io (see x86 arch_init_memory and it's usage of share_xen_page_with_guest), and thus also fulfill the is_special_page check because the page has the Xen heap bit set.

This is incorrect for MMIO regions when is_special_page is used by epte_get_entry_emt, as it will force direct MMIO regions mapped into the guest p2m to have the cache attributes set to write-back.

Add an extra check in epte_get_entry_emt in order to detect pages shared with dom_io (ie: MMIO regions) and don't force them to write-back cache type on that case.

Quoted 8 lines
Quoted 4 lines

Did you consider the alternative of not marking those pages as Xen heap ones? In particular when looking at it from this angle I consider it at least odd for non-RAM (or more precisely non-heap) pages to get marked this way.

I wasn't sure whether this could cause issues in other places of the code that would rely on this fact and such change seemed more risky IMO.

As said - I don't think there is, but I've not done a full audit.

Quoted 8 lines
Quoted 3 lines

And I can't currently see anything
requiring them to be marked as such - them being owned by DomIO is all that's needed as it seems.

Should those pages then simply be assigned to dom_io and set the appropriate flags (PGC_allocated | 1), or should share_xen_page_with_guest be modified to not set the PGC_xen_heap flag?

Either approach may be okay, I think. It would really depend on how little of share_xen_page_with_guest() suffices for the dom_io assignment.

Quoted 3 lines

I see that such addition was done in a2b4b8c2041, but I'm afraid I don't fully understand why share_xen_page_with_guest needs to mark pages as Xen heap.

I see Paul has already answer this part. You can't drop the setting of the flag, but you could make it dependent upon d != dom_io.

Jan