Skip to contentSkip to conversation

xen/riscv: introduce p2m functionality

PATCH Superseded Oleksii Kurochko 5 participants 161 messages Jun 10, 2025 – Jul 29, 2025 RISC-V Page 2 of 2

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#114 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

In reply to Oleksii Kurochko
Quoted 127 lines · Oleksii Kurochko
Quoted 62 lines · Jan Beulich
Quoted 53 lines · Oleksii Kurochko
Quoted 52 lines · Jan Beulich
Quoted 28 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich

xen/arch/riscv/p2m.c static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn) · line 348

Quoted 7 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +1 -0
@@ -345,6 +345,26 @@ static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn)
          return __map_domain_page(p2m->root + root_table_indx);
      }
 
+static int p2m_type_radix_set(struct p2m_domain *p2m, pte_t pte, p2m_type_t t)

See comments on the earlier patch regarding naming.

xen/arch/riscv/p2m.c static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn) · line 351

Quoted 3 lines
xen/arch/riscv/p2m.c +3 -0
+{
+    int rc;
+    gfn_t gfn = mfn_to_gfn(p2m->domain, mfn_from_pte(pte));

How does this work, when you record GFNs only for Xenheap pages?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think I don't understand what is an issue. Could you please provide some extra details?

Counter question: The mfn_to_gfn() you currently have is only a stub. It only works for 1:1 mapped domains. Can you show me the eventual final implementation of the function, making it possible to use it here?

At the moment, I planned to support only 1:1 mapped domains, so it is final implementation.

Isn't that on overly severe limitation?

I wouldn't say that it's a severe limitation, as it's just a matter of how

Quoted 8 lines
|mfn_to_gfn()| is implemented. When non-1:1 mapped domains are supported,
|mfn_to_gfn()| can be implemented differently, while the code where it’s called

will likely remain unchanged.

What I meant in my reply is that, for the current state and current limitations, this is the final implementation of|mfn_to_gfn()|. But that doesn't mean I don't see the value in, or the need for, non-1:1 mapped domains—it's just that this limitation simplifies development at the current stage of the RISC-V port.

Simplification is fine in some cases, but not supporting the "normal" way of domain construction looks like a pretty odd restriction. I'm also curious how you envision to implement mfn_to_gfn() then, suitable for generic use like the one here. Imo, current limitation or not, you simply want to avoid use of that function outside of the special gnttab case.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

In this context (not sure if I asked before): With this use of a radix tree, how do you intend to bound the amount of memory that a domain can use, by making Xen insert very many entries?

I didn’t think about that. I assumed it would be enough to set the amount of memory a guest domain can use by specifying|xen,domain-p2m-mem-mb| in the DTS, or using some predefined value if|xen,domain-p2m-mem-mb| isn’t explicitly set.

Which would require these allocations to come from that pool.

Yes, and it is true only for non-hardware domains with the current implementation.

???

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I meant that pool is used now only for non-hardware domains at the moment.

And how does this matter here? The memory required for the radix tree doesn't come from that pool anyway.

I thought that is possible to do that somehow, but looking at a code of radix-tree.c it seems like the only one way to allocate memroy for the radix tree isradix_tree_node_alloc() -> xzalloc(struct rcu_node).

Then it is needed to introduce radix_tree_node_allocate(domain)

Quoted 8 lines

That would be a possibility, but you may have seen that less than half a year ago we got rid of something along these lines. So it would require some pretty good justification to re-introduce.

Quoted 2 lines

or radix tree
can't be used at all for mentioned in the previous replies security reason, no?

(Very) careful use may still be possible. But the downside of using this (potentially long lookup times) would always remain.

Could you please clarify what do you mean here by "(Very) careful"?
I thought about an introduction of an amount of possible keys in radix tree and if this amount is 0 then stop domain. And it is also unclear what should be a value for this amount. Probably, you have better idea.

But generally your idea below ...

Quoted 42 lines
Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Also, it seems this would just lead to the issue you mentioned earlier: when the memory runs out,|domain_crash()| will be called or PTE will be zapped.

Or one domain exhausting memory would cause another domain to fail. A domain impacting just itself may be tolerable. But a domain affecting other domains isn't.

But it seems like this issue could happen in any implementation. It won't happen only if we will have only pre-populated pool for any domain type (hardware, control, guest domain) without ability to extend them or allocate extra pages from domheap in runtime. Otherwise, if extra pages allocation is allowed then we can't really do something with this issue.

Quoted 4 lines
Quoted 2 lines

But that's why I brought this up: You simply have to. Or, as indicated, the moment you mark Xen security-supported on RISC-V, there will be an XSA needed.

Why it isn't XSA for other architectures? At least, Arm then should have such XSA.

Does Arm use a radix tree for storing types? It uses one for mem-access, but it's not clear to me whether that's actually a supported feature.

Quoted 3 lines

I don't understand why x86 won't have the same issue. Memory is the limited and shared resource, so if one of the domain will use to much memory then it could happen that other domains won't have enough memory for its purpose...

Quoted 7 lines
Quoted 4 lines

The question is whether allocations are bounded. With this use of a radix tree, you give domains a way to have Xen allocate pretty much arbitrary amounts of memory to populate that tree. That unbounded-ness is the problem, not memory allocations in general.

Isn't radix tree key bounded to an amount of GFNs given for a domain? We can't have more keys then a max GFN number for a domain. So a potential amount of necessary memory for radix tree is also bounded to an amount of GFNs.

To some degree yes, hence why I said "pretty much arbitrary amounts". But recall that "amount of GFNs" is a fuzzy term; I think you mean to use it to describe the amount of memory pages given to the guest. GFNs can be used for other purposes, though. Guests could e.g. grant themselves access to their own memory, then map those grants at otherwise unused GFNs.

Quoted 3 lines

Anyway, IIUC I just can't use radix tree for p2m types at all, right?
If yes, does it make sense to borrow 2 bits from struct page_info->type_info as now it is used 9-bits for count of a frame?

struct page_info describes MFNs, when you want to describe GFNs. As you mentioned earlier, multiple GFNs can in principle map to the same MFN. You would force them to all have the same properties, which would be in direct conflict with e.g. the grant P2M types.

Quoted 6 lines

Just to mention one possible alternative to using radix trees: You could maintain a 2nd set of intermediate "page tables", just that leaf entries would hold meta data for the respective GFN. The memory for those "page tables" could come from the normal P2M pool (and allocation would thus only consume domain-specific resources). Of course in any model like this the question of lookup times (as mentioned above) would remain.

Quoted 8 lines

...looks like an optimal option.

The only thing I worry about is that it will require some code duplication (I will think how to re-use the current one code), as for example, when setting/getting metadata, TLB flushing isn’t needed at all as we aren't working with with real P2M page tables.
Agree that lookup won't be the best one, but nothing can be done with such models.

Probably, instead of having a second set of intermediate "page tables", we could just allocate two consecutive pages within the real P2M page tables for the intermediate page table. The first page would serve as the actual page table to which the intermediate page table points, and the second page would store metadata for each entry of the page table that the intermediate page table references.

As we are supporting only 1gb, 2mb and 4kb mappings we could do a little optimization and start allocate these consecutive pages only for PT levels which corresponds to 1gb, 2mb, 4kb mappings.

Does it make sense?

~ Oleksii

#115 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 145 lines · Oleksii Kurochko
Quoted 127 lines · Oleksii Kurochko
Quoted 62 lines · Jan Beulich
Quoted 53 lines · Oleksii Kurochko
Quoted 52 lines · Jan Beulich
Quoted 28 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich

xen/arch/riscv/p2m.c static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn) · line 348

Quoted 7 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +1 -0
@@ -345,6 +345,26 @@ static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn)
          return __map_domain_page(p2m->root + root_table_indx);
      }
 
+static int p2m_type_radix_set(struct p2m_domain *p2m, pte_t pte, p2m_type_t t)

See comments on the earlier patch regarding naming.

xen/arch/riscv/p2m.c static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn) · line 351

Quoted 3 lines
xen/arch/riscv/p2m.c +3 -0
+{
+    int rc;
+    gfn_t gfn = mfn_to_gfn(p2m->domain, mfn_from_pte(pte));

How does this work, when you record GFNs only for Xenheap pages?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think I don't understand what is an issue. Could you please provide some extra details?

Counter question: The mfn_to_gfn() you currently have is only a stub. It only works for 1:1 mapped domains. Can you show me the eventual final implementation of the function, making it possible to use it here?

At the moment, I planned to support only 1:1 mapped domains, so it is final implementation.

Isn't that on overly severe limitation?

I wouldn't say that it's a severe limitation, as it's just a matter of how

Quoted 8 lines
|mfn_to_gfn()| is implemented. When non-1:1 mapped domains are supported,
|mfn_to_gfn()| can be implemented differently, while the code where it’s called

will likely remain unchanged.

What I meant in my reply is that, for the current state and current limitations, this is the final implementation of|mfn_to_gfn()|. But that doesn't mean I don't see the value in, or the need for, non-1:1 mapped domains—it's just that this limitation simplifies development at the current stage of the RISC-V port.

Simplification is fine in some cases, but not supporting the "normal" way of domain construction looks like a pretty odd restriction. I'm also curious how you envision to implement mfn_to_gfn() then, suitable for generic use like the one here. Imo, current limitation or not, you simply want to avoid use of that function outside of the special gnttab case.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

In this context (not sure if I asked before): With this use of a radix tree, how do you intend to bound the amount of memory that a domain can use, by making Xen insert very many entries?

I didn’t think about that. I assumed it would be enough to set the amount of memory a guest domain can use by specifying|xen,domain-p2m-mem-mb| in the DTS, or using some predefined value if|xen,domain-p2m-mem-mb| isn’t explicitly set.

Which would require these allocations to come from that pool.

Yes, and it is true only for non-hardware domains with the current implementation.

???

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I meant that pool is used now only for non-hardware domains at the moment.

And how does this matter here? The memory required for the radix tree doesn't come from that pool anyway.

I thought that is possible to do that somehow, but looking at a code of radix-tree.c it seems like the only one way to allocate memroy for the radix tree isradix_tree_node_alloc() -> xzalloc(struct rcu_node).

Then it is needed to introduce radix_tree_node_allocate(domain)

Quoted 8 lines

That would be a possibility, but you may have seen that less than half a year ago we got rid of something along these lines. So it would require some pretty good justification to re-introduce.

Quoted 2 lines

or radix tree
can't be used at all for mentioned in the previous replies security reason, no?

(Very) careful use may still be possible. But the downside of using this (potentially long lookup times) would always remain.

Could you please clarify what do you mean here by "(Very) careful"?
I thought about an introduction of an amount of possible keys in radix tree and if this amount is 0 then stop domain. And it is also unclear what should be a value for this amount. Probably, you have better idea.

But generally your idea below ...

Quoted 42 lines
Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Also, it seems this would just lead to the issue you mentioned earlier: when the memory runs out,|domain_crash()| will be called or PTE will be zapped.

Or one domain exhausting memory would cause another domain to fail. A domain impacting just itself may be tolerable. But a domain affecting other domains isn't.

But it seems like this issue could happen in any implementation. It won't happen only if we will have only pre-populated pool for any domain type (hardware, control, guest domain) without ability to extend them or allocate extra pages from domheap in runtime. Otherwise, if extra pages allocation is allowed then we can't really do something with this issue.

Quoted 4 lines
Quoted 2 lines

But that's why I brought this up: You simply have to. Or, as indicated, the moment you mark Xen security-supported on RISC-V, there will be an XSA needed.

Why it isn't XSA for other architectures? At least, Arm then should have such XSA.

Does Arm use a radix tree for storing types? It uses one for mem-access, but it's not clear to me whether that's actually a supported feature.

Quoted 3 lines

I don't understand why x86 won't have the same issue. Memory is the limited and shared resource, so if one of the domain will use to much memory then it could happen that other domains won't have enough memory for its purpose...

Quoted 7 lines
Quoted 4 lines

The question is whether allocations are bounded. With this use of a radix tree, you give domains a way to have Xen allocate pretty much arbitrary amounts of memory to populate that tree. That unbounded-ness is the problem, not memory allocations in general.

Isn't radix tree key bounded to an amount of GFNs given for a domain? We can't have more keys then a max GFN number for a domain. So a potential amount of necessary memory for radix tree is also bounded to an amount of GFNs.

To some degree yes, hence why I said "pretty much arbitrary amounts". But recall that "amount of GFNs" is a fuzzy term; I think you mean to use it to describe the amount of memory pages given to the guest. GFNs can be used for other purposes, though. Guests could e.g. grant themselves access to their own memory, then map those grants at otherwise unused GFNs.

Quoted 3 lines

Anyway, IIUC I just can't use radix tree for p2m types at all, right?
If yes, does it make sense to borrow 2 bits from struct page_info->type_info as now it is used 9-bits for count of a frame?

struct page_info describes MFNs, when you want to describe GFNs. As you mentioned earlier, multiple GFNs can in principle map to the same MFN. You would force them to all have the same properties, which would be in direct conflict with e.g. the grant P2M types.

Quoted 6 lines

Just to mention one possible alternative to using radix trees: You could maintain a 2nd set of intermediate "page tables", just that leaf entries would hold meta data for the respective GFN. The memory for those "page tables" could come from the normal P2M pool (and allocation would thus only consume domain-specific resources). Of course in any model like this the question of lookup times (as mentioned above) would remain.

Quoted 8 lines

...looks like an optimal option.

The only thing I worry about is that it will require some code duplication (I will think how to re-use the current one code), as for example, when setting/getting metadata, TLB flushing isn’t needed at all as we aren't working with with real P2M page tables.
Agree that lookup won't be the best one, but nothing can be done with such models.

Probably, instead of having a second set of intermediate "page tables", we could just allocate two consecutive pages within the real P2M page tables for the intermediate page table. The first page would serve as the actual page table to which the intermediate page table points, and the second page would store metadata for each entry of the page table that the intermediate page table references.

Quoted 5 lines

As we are supporting only 1gb, 2mb and 4kb mappings we could do a little optimization and start allocate these consecutive pages only for PT levels which corresponds to 1gb, 2mb, 4kb mappings.

Does it make sense?

I was indeed entertaining this idea, but I couldn't conclude for myself if that would indeed be without any rough edges. Hence I didn't want to suggest such. For example, the need to have adjacent pairs of pages could result in a higher rate of allocation failures (while populating or re-sizing the P2M pool). This would be possible to avoid by still using entirely separate pages, and then merely linking them together via some unused struct page_info fields (the "normal" linking fields can't be used, afaict).

Jan

#116 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 148 lines · Jan Beulich
Quoted 142 lines · Oleksii Kurochko
Quoted 126 lines · Oleksii Kurochko
Quoted 62 lines · Jan Beulich
Quoted 53 lines · Oleksii Kurochko
Quoted 52 lines · Jan Beulich
Quoted 28 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich
Quoted 7 lines · Oleksii Kurochko

--- a/xen/arch/riscv/p2m.c
+++ b/xen/arch/riscv/p2m.c
@@ -345,6 +345,26 @@ static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn)
return __map_domain_page(p2m->root + root_table_indx);
}

+static int p2m_type_radix_set(struct p2m_domain *p2m, pte_t pte, p2m_type_t t)

See comments on the earlier patch regarding naming.

Quoted 3 lines

+{
+ int rc;
+ gfn_t gfn = mfn_to_gfn(p2m->domain, mfn_from_pte(pte));

How does this work, when you record GFNs only for Xenheap pages?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think I don't understand what is an issue. Could you please provide some extra details?

Counter question: The mfn_to_gfn() you currently have is only a stub. It only works for 1:1 mapped domains. Can you show me the eventual final implementation of the function, making it possible to use it here?

At the moment, I planned to support only 1:1 mapped domains, so it is final implementation.

Isn't that on overly severe limitation?

I wouldn't say that it's a severe limitation, as it's just a matter of how

Quoted 8 lines
|mfn_to_gfn()| is implemented. When non-1:1 mapped domains are supported,
|mfn_to_gfn()| can be implemented differently, while the code where it’s called

will likely remain unchanged.

What I meant in my reply is that, for the current state and current limitations, this is the final implementation of|mfn_to_gfn()|. But that doesn't mean I don't see the value in, or the need for, non-1:1 mapped domains—it's just that this limitation simplifies development at the current stage of the RISC-V port.

Simplification is fine in some cases, but not supporting the "normal" way of domain construction looks like a pretty odd restriction. I'm also curious how you envision to implement mfn_to_gfn() then, suitable for generic use like the one here. Imo, current limitation or not, you simply want to avoid use of that function outside of the special gnttab case.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

In this context (not sure if I asked before): With this use of a radix tree, how do you intend to bound the amount of memory that a domain can use, by making Xen insert very many entries?

I didn’t think about that. I assumed it would be enough to set the amount of memory a guest domain can use by specifying|xen,domain-p2m-mem-mb| in the DTS, or using some predefined value if|xen,domain-p2m-mem-mb| isn’t explicitly set.

Which would require these allocations to come from that pool.

Yes, and it is true only for non-hardware domains with the current implementation.

???

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I meant that pool is used now only for non-hardware domains at the moment.

And how does this matter here? The memory required for the radix tree doesn't come from that pool anyway.

I thought that is possible to do that somehow, but looking at a code of radix-tree.c it seems like the only one way to allocate memroy for the radix tree isradix_tree_node_alloc() -> xzalloc(struct rcu_node).

Then it is needed to introduce radix_tree_node_allocate(domain)

Quoted 8 lines

That would be a possibility, but you may have seen that less than half a year ago we got rid of something along these lines. So it would require some pretty good justification to re-introduce.

Quoted 2 lines

or radix tree
can't be used at all for mentioned in the previous replies security reason, no?

(Very) careful use may still be possible. But the downside of using this (potentially long lookup times) would always remain.

Could you please clarify what do you mean here by "(Very) careful"?
I thought about an introduction of an amount of possible keys in radix tree and if this amount is 0 then stop domain. And it is also unclear what should be a value for this amount. Probably, you have better idea.

But generally your idea below ...

Quoted 42 lines
Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Also, it seems this would just lead to the issue you mentioned earlier: when the memory runs out,|domain_crash()| will be called or PTE will be zapped.

Or one domain exhausting memory would cause another domain to fail. A domain impacting just itself may be tolerable. But a domain affecting other domains isn't.

But it seems like this issue could happen in any implementation. It won't happen only if we will have only pre-populated pool for any domain type (hardware, control, guest domain) without ability to extend them or allocate extra pages from domheap in runtime. Otherwise, if extra pages allocation is allowed then we can't really do something with this issue.

Quoted 4 lines
Quoted 2 lines

But that's why I brought this up: You simply have to. Or, as indicated, the moment you mark Xen security-supported on RISC-V, there will be an XSA needed.

Why it isn't XSA for other architectures? At least, Arm then should have such XSA.

Does Arm use a radix tree for storing types? It uses one for mem-access, but it's not clear to me whether that's actually a supported feature.

Quoted 3 lines

I don't understand why x86 won't have the same issue. Memory is the limited and shared resource, so if one of the domain will use to much memory then it could happen that other domains won't have enough memory for its purpose...

Quoted 7 lines
Quoted 4 lines

The question is whether allocations are bounded. With this use of a radix tree, you give domains a way to have Xen allocate pretty much arbitrary amounts of memory to populate that tree. That unbounded-ness is the problem, not memory allocations in general.

Isn't radix tree key bounded to an amount of GFNs given for a domain? We can't have more keys then a max GFN number for a domain. So a potential amount of necessary memory for radix tree is also bounded to an amount of GFNs.

To some degree yes, hence why I said "pretty much arbitrary amounts". But recall that "amount of GFNs" is a fuzzy term; I think you mean to use it to describe the amount of memory pages given to the guest. GFNs can be used for other purposes, though. Guests could e.g. grant themselves access to their own memory, then map those grants at otherwise unused GFNs.

Quoted 3 lines

Anyway, IIUC I just can't use radix tree for p2m types at all, right?
If yes, does it make sense to borrow 2 bits from struct page_info->type_info as now it is used 9-bits for count of a frame?

struct page_info describes MFNs, when you want to describe GFNs. As you mentioned earlier, multiple GFNs can in principle map to the same MFN. You would force them to all have the same properties, which would be in direct conflict with e.g. the grant P2M types.

Quoted 6 lines

Just to mention one possible alternative to using radix trees: You could maintain a 2nd set of intermediate "page tables", just that leaf entries would hold meta data for the respective GFN. The memory for those "page tables" could come from the normal P2M pool (and allocation would thus only consume domain-specific resources). Of course in any model like this the question of lookup times (as mentioned above) would remain.

Quoted 8 lines

...looks like an optimal option.

The only thing I worry about is that it will require some code duplication (I will think how to re-use the current one code), as for example, when setting/getting metadata, TLB flushing isn’t needed at all as we aren't working with with real P2M page tables.
Agree that lookup won't be the best one, but nothing can be done with such models.

Probably, instead of having a second set of intermediate "page tables", we could just allocate two consecutive pages within the real P2M page tables for the intermediate page table. The first page would serve as the actual page table to which the intermediate page table points, and the second page would store metadata for each entry of the page table that the intermediate page table references.

Quoted 5 lines

As we are supporting only 1gb, 2mb and 4kb mappings we could do a little optimization and start allocate these consecutive pages only for PT levels which corresponds to 1gb, 2mb, 4kb mappings.

Does it make sense?

Quoted 8 lines

I was indeed entertaining this idea, but I couldn't conclude for myself if that would indeed be without any rough edges. Hence I didn't want to suggest such. For example, the need to have adjacent pairs of pages could result in a higher rate of allocation failures (while populating or re-sizing the P2M pool). This would be possible to avoid by still using entirely separate pages, and then merely linking them together via some unused struct page_info fields (the "normal" linking fields can't be used, afaict).

I think that all the fields are used, so it will be needed to introduce new "struct page_list_entry metadata_list;".

Can't we introduce new PGT_METADATA type and then add metadata page to struct page_info->list and when a metadata will be needed just iterate through page_info->list and find a page with PGT_METADATA type?

~ Oleksii

#117 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 157 lines · Oleksii Kurochko
Quoted 147 lines · Jan Beulich
Quoted 141 lines · Oleksii Kurochko
Quoted 125 lines · Oleksii Kurochko
Quoted 62 lines · Jan Beulich
Quoted 53 lines · Oleksii Kurochko
Quoted 52 lines · Jan Beulich
Quoted 28 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich

On 10.06.2025 15:05, Oleksii Kurochko wrote:
--- a/xen/arch/riscv/p2m.c
+++ b/xen/arch/riscv/p2m.c
@@ -345,6 +345,26 @@ static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn)
return __map_domain_page(p2m->root + root_table_indx);
}

+static int p2m_type_radix_set(struct p2m_domain *p2m, pte_t pte, p2m_type_t t)
See comments on the earlier patch regarding naming.

+{
+ int rc;
+ gfn_t gfn = mfn_to_gfn(p2m->domain, mfn_from_pte(pte));
How does this work, when you record GFNs only for Xenheap pages?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think I don't understand what is an issue. Could you please provide some extra details?

Counter question: The mfn_to_gfn() you currently have is only a stub. It only works for 1:1 mapped domains. Can you show me the eventual final implementation of the function, making it possible to use it here?

At the moment, I planned to support only 1:1 mapped domains, so it is final implementation.

Isn't that on overly severe limitation?

I wouldn't say that it's a severe limitation, as it's just a matter of how

Quoted 8 lines
|mfn_to_gfn()| is implemented. When non-1:1 mapped domains are supported,
|mfn_to_gfn()| can be implemented differently, while the code where it’s called

will likely remain unchanged.

What I meant in my reply is that, for the current state and current limitations, this is the final implementation of|mfn_to_gfn()|. But that doesn't mean I don't see the value in, or the need for, non-1:1 mapped domains—it's just that this limitation simplifies development at the current stage of the RISC-V port.

Simplification is fine in some cases, but not supporting the "normal" way of domain construction looks like a pretty odd restriction. I'm also curious how you envision to implement mfn_to_gfn() then, suitable for generic use like the one here. Imo, current limitation or not, you simply want to avoid use of that function outside of the special gnttab case.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

In this context (not sure if I asked before): With this use of a radix tree,
how do you intend to bound the amount of memory that a domain can use, by
making Xen insert very many entries?

I didn’t think about that. I assumed it would be enough to set the amount of memory a guest domain can use by specifying|xen,domain-p2m-mem-mb| in the DTS, or using some predefined value if|xen,domain-p2m-mem-mb| isn’t explicitly set.

Which would require these allocations to come from that pool.

Yes, and it is true only for non-hardware domains with the current implementation.

???

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I meant that pool is used now only for non-hardware domains at the moment.

And how does this matter here? The memory required for the radix tree doesn't come from that pool anyway.

I thought that is possible to do that somehow, but looking at a code of radix-tree.c it seems like the only one way to allocate memroy for the radix tree isradix_tree_node_alloc() -> xzalloc(struct rcu_node).

Then it is needed to introduce radix_tree_node_allocate(domain)

Quoted 8 lines

That would be a possibility, but you may have seen that less than half a year ago we got rid of something along these lines. So it would require some pretty good justification to re-introduce.

Quoted 2 lines

or radix tree
can't be used at all for mentioned in the previous replies security reason, no?

(Very) careful use may still be possible. But the downside of using this (potentially long lookup times) would always remain.

Could you please clarify what do you mean here by "(Very) careful"?
I thought about an introduction of an amount of possible keys in radix tree and if this amount is 0 then stop domain. And it is also unclear what should be a value for this amount. Probably, you have better idea.

But generally your idea below ...

Quoted 42 lines
Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Also, it seems this would just lead to the issue you mentioned earlier: when the memory runs out,|domain_crash()| will be called or PTE will be zapped.

Or one domain exhausting memory would cause another domain to fail. A domain impacting just itself may be tolerable. But a domain affecting other domains isn't.

But it seems like this issue could happen in any implementation. It won't happen only if we will have only pre-populated pool for any domain type (hardware, control, guest domain) without ability to extend them or allocate extra pages from domheap in runtime. Otherwise, if extra pages allocation is allowed then we can't really do something with this issue.

Quoted 4 lines
Quoted 2 lines

But that's why I brought this up: You simply have to. Or, as indicated, the moment you mark Xen security-supported on RISC-V, there will be an XSA needed.

Why it isn't XSA for other architectures? At least, Arm then should have such XSA.

Does Arm use a radix tree for storing types? It uses one for mem-access, but it's not clear to me whether that's actually a supported feature.

Quoted 3 lines

I don't understand why x86 won't have the same issue. Memory is the limited and shared resource, so if one of the domain will use to much memory then it could happen that other domains won't have enough memory for its purpose...

Quoted 7 lines
Quoted 4 lines

The question is whether allocations are bounded. With this use of a radix tree, you give domains a way to have Xen allocate pretty much arbitrary amounts of memory to populate that tree. That unbounded-ness is the problem, not memory allocations in general.

Isn't radix tree key bounded to an amount of GFNs given for a domain? We can't have more keys then a max GFN number for a domain. So a potential amount of necessary memory for radix tree is also bounded to an amount of GFNs.

To some degree yes, hence why I said "pretty much arbitrary amounts". But recall that "amount of GFNs" is a fuzzy term; I think you mean to use it to describe the amount of memory pages given to the guest. GFNs can be used for other purposes, though. Guests could e.g. grant themselves access to their own memory, then map those grants at otherwise unused GFNs.

Quoted 3 lines

Anyway, IIUC I just can't use radix tree for p2m types at all, right?
If yes, does it make sense to borrow 2 bits from struct page_info->type_info as now it is used 9-bits for count of a frame?

struct page_info describes MFNs, when you want to describe GFNs. As you mentioned earlier, multiple GFNs can in principle map to the same MFN. You would force them to all have the same properties, which would be in direct conflict with e.g. the grant P2M types.

Quoted 6 lines

Just to mention one possible alternative to using radix trees: You could maintain a 2nd set of intermediate "page tables", just that leaf entries would hold meta data for the respective GFN. The memory for those "page tables" could come from the normal P2M pool (and allocation would thus only consume domain-specific resources). Of course in any model like this the question of lookup times (as mentioned above) would remain.

Quoted 8 lines

...looks like an optimal option.

The only thing I worry about is that it will require some code duplication (I will think how to re-use the current one code), as for example, when setting/getting metadata, TLB flushing isn’t needed at all as we aren't working with with real P2M page tables.
Agree that lookup won't be the best one, but nothing can be done with such models.

Probably, instead of having a second set of intermediate "page tables", we could just allocate two consecutive pages within the real P2M page tables for the intermediate page table. The first page would serve as the actual page table to which the intermediate page table points, and the second page would store metadata for each entry of the page table that the intermediate page table references.

Quoted 5 lines

As we are supporting only 1gb, 2mb and 4kb mappings we could do a little optimization and start allocate these consecutive pages only for PT levels which corresponds to 1gb, 2mb, 4kb mappings.

Does it make sense?

Quoted 8 lines

I was indeed entertaining this idea, but I couldn't conclude for myself if that would indeed be without any rough edges. Hence I didn't want to suggest such. For example, the need to have adjacent pairs of pages could result in a higher rate of allocation failures (while populating or re-sizing the P2M pool). This would be possible to avoid by still using entirely separate pages, and then merely linking them together via some unused struct page_info fields (the "normal" linking fields can't be used, afaict).

Quoted 2 lines

I think that all the fields are used, so it will be needed to introduce new "struct page_list_entry metadata_list;".

All the fields are used _somewhere_, sure. But once you have allocated a page (and that page isn't assigned to a domain), you control what the fields are used for. Or else enlisting pages on private lists wouldn't be legitimate either.

Quoted 3 lines

Can't we introduce new PGT_METADATA type and then add metadata page to struct page_info->list and when a metadata will be needed just iterate through page_info->list and find a page with PGT_METADATA type?

I'd be careful with the introduction of new page types. All handling of page types everywhere in the (affected part of the) code base would then need auditing.

Jan

#118 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 157 lines · Jan Beulich
Quoted 156 lines · Oleksii Kurochko
Quoted 155 lines · Jan Beulich
Quoted 141 lines · Oleksii Kurochko
Quoted 125 lines · Oleksii Kurochko
Quoted 62 lines · Jan Beulich
Quoted 53 lines · Oleksii Kurochko
Quoted 52 lines · Jan Beulich
Quoted 28 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko

On 7/1/25 5:08 PM, Jan Beulich wrote:
On 10.06.2025 15:05, Oleksii Kurochko wrote:
--- a/xen/arch/riscv/p2m.c
+++ b/xen/arch/riscv/p2m.c
@@ -345,6 +345,26 @@ static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn)
return __map_domain_page(p2m->root + root_table_indx);
}

+static int p2m_type_radix_set(struct p2m_domain *p2m, pte_t pte, p2m_type_t t)
See comments on the earlier patch regarding naming.

+{
+ int rc;
+ gfn_t gfn = mfn_to_gfn(p2m->domain, mfn_from_pte(pte));
How does this work, when you record GFNs only for Xenheap pages?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think I don't understand what is an issue. Could you please provide
some extra details?

Counter question: The mfn_to_gfn() you currently have is only a stub. It only works for 1:1 mapped domains. Can you show me the eventual final implementation of the function, making it possible to use it here?

At the moment, I planned to support only 1:1 mapped domains, so it is final implementation.

Isn't that on overly severe limitation?

I wouldn't say that it's a severe limitation, as it's just a matter of how

Quoted 8 lines
|mfn_to_gfn()| is implemented. When non-1:1 mapped domains are supported,
|mfn_to_gfn()| can be implemented differently, while the code where it’s called

will likely remain unchanged.

What I meant in my reply is that, for the current state and current limitations, this is the final implementation of|mfn_to_gfn()|. But that doesn't mean I don't see the value in, or the need for, non-1:1 mapped domains—it's just that this limitation simplifies development at the current stage of the RISC-V port.

Simplification is fine in some cases, but not supporting the "normal" way of domain construction looks like a pretty odd restriction. I'm also curious how you envision to implement mfn_to_gfn() then, suitable for generic use like the one here. Imo, current limitation or not, you simply want to avoid use of that function outside of the special gnttab case.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines

In this context (not sure if I asked before): With this use of a radix tree,
how do you intend to bound the amount of memory that a domain can use, by
making Xen insert very many entries?
I didn’t think about that. I assumed it would be enough to set the amount of
memory a guest domain can use by specifying|xen,domain-p2m-mem-mb| in the DTS,
or using some predefined value if|xen,domain-p2m-mem-mb| isn’t explicitly set.

Which would require these allocations to come from that pool.

Yes, and it is true only for non-hardware domains with the current implementation.

???

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I meant that pool is used now only for non-hardware domains at the moment.

And how does this matter here? The memory required for the radix tree doesn't come from that pool anyway.

I thought that is possible to do that somehow, but looking at a code of radix-tree.c it seems like the only one way to allocate memroy for the radix tree isradix_tree_node_alloc() -> xzalloc(struct rcu_node).

Then it is needed to introduce radix_tree_node_allocate(domain)

Quoted 8 lines

That would be a possibility, but you may have seen that less than half a year ago we got rid of something along these lines. So it would require some pretty good justification to re-introduce.

Quoted 2 lines

or radix tree
can't be used at all for mentioned in the previous replies security reason, no?

(Very) careful use may still be possible. But the downside of using this (potentially long lookup times) would always remain.

Could you please clarify what do you mean here by "(Very) careful"?
I thought about an introduction of an amount of possible keys in radix tree and if this amount is 0 then stop domain. And it is also unclear what should be a value for this amount. Probably, you have better idea.

But generally your idea below ...

Quoted 42 lines
Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Also, it seems this would just lead to the issue you mentioned earlier: when
the memory runs out,|domain_crash()| will be called or PTE will be zapped.

Or one domain exhausting memory would cause another domain to fail. A domain impacting just itself may be tolerable. But a domain affecting other domains isn't.

But it seems like this issue could happen in any implementation. It won't happen only if we will have only pre-populated pool for any domain type (hardware, control, guest domain) without ability to extend them or allocate extra pages from domheap in runtime. Otherwise, if extra pages allocation is allowed then we can't really do something with this issue.

Quoted 4 lines
Quoted 2 lines

But that's why I brought this up: You simply have to. Or, as indicated, the moment you mark Xen security-supported on RISC-V, there will be an XSA needed.

Why it isn't XSA for other architectures? At least, Arm then should have such XSA.

Does Arm use a radix tree for storing types? It uses one for mem-access, but it's not clear to me whether that's actually a supported feature.

Quoted 3 lines

I don't understand why x86 won't have the same issue. Memory is the limited and shared resource, so if one of the domain will use to much memory then it could happen that other domains won't have enough memory for its purpose...

Quoted 7 lines
Quoted 4 lines

The question is whether allocations are bounded. With this use of a radix tree, you give domains a way to have Xen allocate pretty much arbitrary amounts of memory to populate that tree. That unbounded-ness is the problem, not memory allocations in general.

Isn't radix tree key bounded to an amount of GFNs given for a domain? We can't have more keys then a max GFN number for a domain. So a potential amount of necessary memory for radix tree is also bounded to an amount of GFNs.

To some degree yes, hence why I said "pretty much arbitrary amounts". But recall that "amount of GFNs" is a fuzzy term; I think you mean to use it to describe the amount of memory pages given to the guest. GFNs can be used for other purposes, though. Guests could e.g. grant themselves access to their own memory, then map those grants at otherwise unused GFNs.

Quoted 3 lines

Anyway, IIUC I just can't use radix tree for p2m types at all, right?
If yes, does it make sense to borrow 2 bits from struct page_info->type_info as now it is used 9-bits for count of a frame?

struct page_info describes MFNs, when you want to describe GFNs. As you mentioned earlier, multiple GFNs can in principle map to the same MFN. You would force them to all have the same properties, which would be in direct conflict with e.g. the grant P2M types.

Quoted 6 lines

Just to mention one possible alternative to using radix trees: You could maintain a 2nd set of intermediate "page tables", just that leaf entries would hold meta data for the respective GFN. The memory for those "page tables" could come from the normal P2M pool (and allocation would thus only consume domain-specific resources). Of course in any model like this the question of lookup times (as mentioned above) would remain.

Quoted 8 lines

...looks like an optimal option.

The only thing I worry about is that it will require some code duplication (I will think how to re-use the current one code), as for example, when setting/getting metadata, TLB flushing isn’t needed at all as we aren't working with with real P2M page tables.
Agree that lookup won't be the best one, but nothing can be done with such models.

Probably, instead of having a second set of intermediate "page tables", we could just allocate two consecutive pages within the real P2M page tables for the intermediate page table. The first page would serve as the actual page table to which the intermediate page table points, and the second page would store metadata for each entry of the page table that the intermediate page table references.

Quoted 5 lines

As we are supporting only 1gb, 2mb and 4kb mappings we could do a little optimization and start allocate these consecutive pages only for PT levels which corresponds to 1gb, 2mb, 4kb mappings.

Does it make sense?

I was indeed entertaining this idea, but I couldn't conclude for myself if that would indeed be without any rough edges. Hence I didn't want to suggest such. For example, the need to have adjacent pairs of pages could result in a higher rate of allocation failures (while populating or re-sizing the P2M pool). This would be possible to avoid by still using entirely separate pages, and then merely linking them together via some unused struct page_info fields (the "normal" linking fields can't be used, afaict).

Quoted 5 lines
Quoted 2 lines

I think that all the fields are used, so it will be needed to introduce new "struct page_list_entry metadata_list;".

All the fields are used _somewhere_, sure. But once you have allocated a page (and that page isn't assigned to a domain), you control what the fields are used for.

I thought that the whole idea is to use domain's pages from P2M pool freelist, pages for which is allocated by alloc_domheap_page(d, MEMF_no_owner), so an allocated page is assigned to a domain.

I assume that I have in this case to take some pages for an intermediate page table from freelist P2M pool, set an owner domain to NULL (pg->inuse.domain = NULL).

Then in this case it isn't clear why pg->list can't be re-used to link several pages for intermediate page table purposes + metadata? Is it because pg->list can be not empty? In this case it isn't clear if I could use a page, which has threaded pages.

page_info->count_info can't be re-used as it will break put_page_*() connected stuff. And for similar reason page_info->v.{...} can't be re-used as then page_get_owner() will be broken.
And page_info->tlbflush_timestamp still need for a common alloc algo to handle when to do TLB flush.

So if I will add something to page_info->v.{...} or page_info->u.{...} then mentioned above functions can't be used anymore for such pages which are used for intermediate page tables.

Quoted 2 lines

Or else enlisting pages on private lists wouldn't be
legitimate either.

Hmm, but I still should have link several pages somehow.
Or you meant just to have a field which just store a physical address to metadata? (what still looks like a list)

Quoted 7 lines
Quoted 4 lines

Can't we introduce new PGT_METADATA type and then add metadata page to struct page_info->list and when a metadata will be needed just iterate through page_info->list and find a page with PGT_METADATA type?

I'd be careful with the introduction of new page types. All handling of page types everywhere in the (affected part of the) code base would then need auditing.

~ Oleksii

#119 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 161 lines · Oleksii Kurochko
Quoted 159 lines · Jan Beulich
Quoted 158 lines · Oleksii Kurochko
Quoted 147 lines · Jan Beulich
Quoted 141 lines · Oleksii Kurochko
Quoted 125 lines · Oleksii Kurochko
Quoted 62 lines · Jan Beulich
Quoted 53 lines · Oleksii Kurochko
Quoted 52 lines · Jan Beulich
Quoted 28 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich

On 15.07.2025 16:47, Oleksii Kurochko wrote:
On 7/1/25 5:08 PM, Jan Beulich wrote:
On 10.06.2025 15:05, Oleksii Kurochko wrote:
--- a/xen/arch/riscv/p2m.c
+++ b/xen/arch/riscv/p2m.c
@@ -345,6 +345,26 @@ static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn)
return __map_domain_page(p2m->root + root_table_indx);
}

+static int p2m_type_radix_set(struct p2m_domain *p2m, pte_t pte, p2m_type_t t)
See comments on the earlier patch regarding naming.

+{
+ int rc;
+ gfn_t gfn = mfn_to_gfn(p2m->domain, mfn_from_pte(pte));
How does this work, when you record GFNs only for Xenheap pages?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

I think I don't understand what is an issue. Could you please provide
some extra details?
Counter question: The mfn_to_gfn() you currently have is only a stub. It only
works for 1:1 mapped domains. Can you show me the eventual final implementation
of the function, making it possible to use it here?

At the moment, I planned to support only 1:1 mapped domains, so it is final implementation.

Isn't that on overly severe limitation?

I wouldn't say that it's a severe limitation, as it's just a matter of how

Quoted 8 lines
|mfn_to_gfn()| is implemented. When non-1:1 mapped domains are supported,
|mfn_to_gfn()| can be implemented differently, while the code where it’s called

will likely remain unchanged.

What I meant in my reply is that, for the current state and current limitations, this is the final implementation of|mfn_to_gfn()|. But that doesn't mean I don't see the value in, or the need for, non-1:1 mapped domains—it's just that this limitation simplifies development at the current stage of the RISC-V port.

Simplification is fine in some cases, but not supporting the "normal" way of domain construction looks like a pretty odd restriction. I'm also curious how you envision to implement mfn_to_gfn() then, suitable for generic use like the one here. Imo, current limitation or not, you simply want to avoid use of that function outside of the special gnttab case.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines

In this context (not sure if I asked before): With this use of a radix tree,
how do you intend to bound the amount of memory that a domain can use, by
making Xen insert very many entries?
I didn’t think about that. I assumed it would be enough to set the amount of
memory a guest domain can use by specifying|xen,domain-p2m-mem-mb| in the DTS,
or using some predefined value if|xen,domain-p2m-mem-mb| isn’t explicitly set.
Which would require these allocations to come from that pool.

Yes, and it is true only for non-hardware domains with the current implementation.

???

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I meant that pool is used now only for non-hardware domains at the moment.

And how does this matter here? The memory required for the radix tree doesn't come from that pool anyway.

I thought that is possible to do that somehow, but looking at a code of radix-tree.c it seems like the only one way to allocate memroy for the radix tree isradix_tree_node_alloc() -> xzalloc(struct rcu_node).

Then it is needed to introduce radix_tree_node_allocate(domain)

Quoted 8 lines

That would be a possibility, but you may have seen that less than half a year ago we got rid of something along these lines. So it would require some pretty good justification to re-introduce.

Quoted 2 lines

or radix tree
can't be used at all for mentioned in the previous replies security reason, no?

(Very) careful use may still be possible. But the downside of using this (potentially long lookup times) would always remain.

Could you please clarify what do you mean here by "(Very) careful"?
I thought about an introduction of an amount of possible keys in radix tree and if this amount is 0 then stop domain. And it is also unclear what should be a value for this amount. Probably, you have better idea.

But generally your idea below ...

Quoted 42 lines
Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

Also, it seems this would just lead to the issue you mentioned earlier: when
the memory runs out,|domain_crash()| will be called or PTE will be zapped.
Or one domain exhausting memory would cause another domain to fail. A domain
impacting just itself may be tolerable. But a domain affecting other domains
isn't.

But it seems like this issue could happen in any implementation. It won't happen only if we will have only pre-populated pool for any domain type (hardware, control, guest domain) without ability to extend them or allocate extra pages from domheap in runtime. Otherwise, if extra pages allocation is allowed then we can't really do something with this issue.

Quoted 4 lines
Quoted 2 lines

But that's why I brought this up: You simply have to. Or, as indicated, the moment you mark Xen security-supported on RISC-V, there will be an XSA needed.

Why it isn't XSA for other architectures? At least, Arm then should have such XSA.

Does Arm use a radix tree for storing types? It uses one for mem-access, but it's not clear to me whether that's actually a supported feature.

Quoted 3 lines

I don't understand why x86 won't have the same issue. Memory is the limited and shared resource, so if one of the domain will use to much memory then it could happen that other domains won't have enough memory for its purpose...

Quoted 7 lines
Quoted 4 lines

The question is whether allocations are bounded. With this use of a radix tree, you give domains a way to have Xen allocate pretty much arbitrary amounts of memory to populate that tree. That unbounded-ness is the problem, not memory allocations in general.

Isn't radix tree key bounded to an amount of GFNs given for a domain? We can't have more keys then a max GFN number for a domain. So a potential amount of necessary memory for radix tree is also bounded to an amount of GFNs.

To some degree yes, hence why I said "pretty much arbitrary amounts". But recall that "amount of GFNs" is a fuzzy term; I think you mean to use it to describe the amount of memory pages given to the guest. GFNs can be used for other purposes, though. Guests could e.g. grant themselves access to their own memory, then map those grants at otherwise unused GFNs.

Quoted 3 lines

Anyway, IIUC I just can't use radix tree for p2m types at all, right?
If yes, does it make sense to borrow 2 bits from struct page_info->type_info as now it is used 9-bits for count of a frame?

struct page_info describes MFNs, when you want to describe GFNs. As you mentioned earlier, multiple GFNs can in principle map to the same MFN. You would force them to all have the same properties, which would be in direct conflict with e.g. the grant P2M types.

Quoted 6 lines

Just to mention one possible alternative to using radix trees: You could maintain a 2nd set of intermediate "page tables", just that leaf entries would hold meta data for the respective GFN. The memory for those "page tables" could come from the normal P2M pool (and allocation would thus only consume domain-specific resources). Of course in any model like this the question of lookup times (as mentioned above) would remain.

Quoted 8 lines

...looks like an optimal option.

The only thing I worry about is that it will require some code duplication (I will think how to re-use the current one code), as for example, when setting/getting metadata, TLB flushing isn’t needed at all as we aren't working with with real P2M page tables.
Agree that lookup won't be the best one, but nothing can be done with such models.

Probably, instead of having a second set of intermediate "page tables", we could just allocate two consecutive pages within the real P2M page tables for the intermediate page table. The first page would serve as the actual page table to which the intermediate page table points, and the second page would store metadata for each entry of the page table that the intermediate page table references.

Quoted 5 lines

As we are supporting only 1gb, 2mb and 4kb mappings we could do a little optimization and start allocate these consecutive pages only for PT levels which corresponds to 1gb, 2mb, 4kb mappings.

Does it make sense?

Quoted 8 lines

I was indeed entertaining this idea, but I couldn't conclude for myself if that would indeed be without any rough edges. Hence I didn't want to suggest such. For example, the need to have adjacent pairs of pages could result in a higher rate of allocation failures (while populating or re-sizing the P2M pool). This would be possible to avoid by still using entirely separate pages, and then merely linking them together via some unused struct page_info fields (the "normal" linking fields can't be used, afaict).

I think that all the fields are used, so it will be needed to introduce new "struct page_list_entry metadata_list;".

Quoted 7 lines
Quoted 3 lines

All the fields are used _somewhere_, sure. But once you have allocated a page (and that page isn't assigned to a domain), you control what the fields are used for.

I thought that the whole idea is to use domain's pages from P2M pool freelist, pages for which is allocated by alloc_domheap_page(d, MEMF_no_owner), so an allocated page is assigned to a domain.

You did check what effect MEMF_no_owner has, didn't you? Such pages are _not_ assigned to the domain.

Quoted 6 lines

I assume that I have in this case to take some pages for an intermediate page table from freelist P2M pool, set an owner domain to NULL (pg->inuse.domain = NULL).

Then in this case it isn't clear why pg->list can't be re-used to link several pages for intermediate page table purposes + metadata? Is it because pg->list can be not empty? In this case it isn't clear if I could use a page, which has threaded pages.

Actually looks like I was mis-remembering. Pages removed from freelist indeed aren't put on any other list, so the linking fields are available for use. I guess I had x86 shadow code in mind, where the linking fields are further used.

Jan

#120 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 9 lines · Jan Beulich
Quoted 6 lines

I assume that I have in this case to take some pages for an intermediate page table from freelist P2M pool, set an owner domain to NULL (pg->inuse.domain = NULL).

Then in this case it isn't clear why pg->list can't be re-used to link several pages for intermediate page table purposes + metadata? Is it because pg->list can be not empty? In this case it isn't clear if I could use a page, which has threaded pages.

Actually looks like I was mis-remembering. Pages removed from freelist indeed aren't put on any other list, so the linking fields are available for use. I guess I had x86 shadow code in mind, where the linking fields are further used.

Perhaps, I misunderstood you about "linking fields", but it seems like I can't reuse struct page_info->list as it is used by page_list_add() which is called by p2m_alloc_page() to allocate page(s) for an intermediate page table:
static inline void
page_list_add(struct page_info *page, struct page_list_head *head)

   {
        list_add(&page->list, head);

}

    struct page_info * paging_alloc_page(struct domain *d)
    {
        struct page_info *pg;

        spin_lock(&d->arch.paging.lock);
        pg = page_list_remove_head(&d->arch.paging.freelist);
        spin_unlock(&d->arch.paging.lock);

        INIT_LIST_HEAD(&pg->list);

        return pg;
    }

    static struct page_info *p2m_alloc_page(struct domain *d)
    {
        struct page_info *pg = paging_alloc_page(d);

        if ( pg )
            page_list_add(pg, &p2m_get_hostp2m(d)->pages);

        return pg;
    }

So I have to reuse another field from struct page_info. It seems like it won't be an issue if to add a new struct page_list_entry metadata_list to 'union v':

    union {
        /* Page is in use */
        struct {
            /* Owner of this page (NULL if page is anonymous). */
            struct domain *domain;
        } inuse;

        /* Page is on a free list. */
        struct {
            /* Order-size of the free chunk this page is the head of. */
            unsigned int order;
        } free;
+
+       struct page_list_entry metadata_list;
    } v;

Am I missing something?

~ Oleksii

#121 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 61 lines · Oleksii Kurochko
Quoted 9 lines · Jan Beulich
Quoted 6 lines

I assume that I have in this case to take some pages for an intermediate page table from freelist P2M pool, set an owner domain to NULL (pg->inuse.domain = NULL).

Then in this case it isn't clear why pg->list can't be re-used to link several pages for intermediate page table purposes + metadata? Is it because pg->list can be not empty? In this case it isn't clear if I could use a page, which has threaded pages.

Actually looks like I was mis-remembering. Pages removed from freelist indeed aren't put on any other list, so the linking fields are available for use. I guess I had x86 shadow code in mind, where the linking fields are further used.

Perhaps, I misunderstood you about "linking fields", but it seems like I can't reuse struct page_info->list as it is used by page_list_add() which is called by p2m_alloc_page() to allocate page(s) for an intermediate page table:

    static inline void
    page_list_add(struct page_info *page, struct page_list_head *head)
    {
         list_add(&page->list, head);
    }

     struct page_info * paging_alloc_page(struct domain *d)
     {
         struct page_info *pg;

         spin_lock(&d->arch.paging.lock);
         pg = page_list_remove_head(&d->arch.paging.freelist);
         spin_unlock(&d->arch.paging.lock);

         INIT_LIST_HEAD(&pg->list);

         return pg;
     }

     static struct page_info *p2m_alloc_page(struct domain *d)
     {
         struct page_info *pg = paging_alloc_page(d);

         if ( pg )
             page_list_add(pg, &p2m_get_hostp2m(d)->pages);

         return pg;
     }

So I have to reuse another field from struct page_info. It seems like it won't be an issue if to add a new struct page_list_entry metadata_list to 'union v':

     union {
         /* Page is in use */
         struct {
             /* Owner of this page (NULL if page is anonymous). */
             struct domain *domain;
         } inuse;

         /* Page is on a free list. */
         struct {
             /* Order-size of the free chunk this page is the head of. */
             unsigned int order;
         } free;
(quoted diff) +2 -0
+
+       struct page_list_entry metadata_list;
     } v;
Quoted 1 line

Am I missing something?

Well, you're doubling the size of that union then, aren't you? As was mentioned quite some time ago, struct page_info needs quite a bit of care when you mean to add new fields there. Question is whether for the purpose here you actually need a doubly-linked list. A single pointer would be fine to put there.

Jan

#122 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 61 lines · Jan Beulich
Quoted 60 lines · Oleksii Kurochko
Quoted 9 lines · Jan Beulich
Quoted 6 lines

I assume that I have in this case to take some pages for an intermediate page table from freelist P2M pool, set an owner domain to NULL (pg->inuse.domain = NULL).

Then in this case it isn't clear why pg->list can't be re-used to link several pages for intermediate page table purposes + metadata? Is it because pg->list can be not empty? In this case it isn't clear if I could use a page, which has threaded pages.

Actually looks like I was mis-remembering. Pages removed from freelist indeed aren't put on any other list, so the linking fields are available for use. I guess I had x86 shadow code in mind, where the linking fields are further used.

Perhaps, I misunderstood you about "linking fields", but it seems like I can't reuse struct page_info->list as it is used by page_list_add() which is called by p2m_alloc_page() to allocate page(s) for an intermediate page table:

     static inline void
     page_list_add(struct page_info *page, struct page_list_head *head)
     {
          list_add(&page->list, head);
     }

      struct page_info * paging_alloc_page(struct domain *d)
      {
          struct page_info *pg;

          spin_lock(&d->arch.paging.lock);
          pg = page_list_remove_head(&d->arch.paging.freelist);
          spin_unlock(&d->arch.paging.lock);

          INIT_LIST_HEAD(&pg->list);

          return pg;
      }

      static struct page_info *p2m_alloc_page(struct domain *d)
      {
          struct page_info *pg = paging_alloc_page(d);

          if ( pg )
              page_list_add(pg, &p2m_get_hostp2m(d)->pages);

          return pg;
      }

So I have to reuse another field from struct page_info. It seems like it won't be an issue if to add a new struct page_list_entry metadata_list to 'union v':

      union {
          /* Page is in use */
          struct {
              /* Owner of this page (NULL if page is anonymous). */
              struct domain *domain;
          } inuse;

          /* Page is on a free list. */
          struct {
              /* Order-size of the free chunk this page is the head of. */
              unsigned int order;
          } free;
(quoted diff) +2 -0
+
+       struct page_list_entry metadata_list;
      } v;
Quoted 5 lines
Quoted 1 line

Am I missing something?

Well, you're doubling the size of that union then, aren't you? As was mentioned quite some time ago, struct page_info needs quite a bit of care when you mean to add new fields there. Question is whether for the purpose here you actually need a doubly-linked list. A single pointer would be fine to put there.

Agree, a single pointer will be more then enough.

I'm thinking if it is possible to do something with the case if someone will try to use:
#define page_get_owner(p) (p)->v.inuse.domain
for a page which was allocated for metadata storage. Shouldn't I have a separate list for such pages and a macro which will check if a page is in this list? Similar a list which we have for p2m pages in struct p2m_domain:

    ...
    /* Pages used to construct the p2m */
    struct page_list_head pages;
    ...

Of course, such pages are allocated by alloc_domheap_page(d, MEMF_no_owner), so there is no owner. But if someone will accidentally use this macro for such pages then it will be an issue as ->domain likely won't be a NULL anymore.

~ Oleksii

#123 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

Quoted 82 lines · Oleksii Kurochko
Quoted 61 lines · Jan Beulich
Quoted 60 lines · Oleksii Kurochko
Quoted 9 lines · Jan Beulich
Quoted 6 lines

I assume that I have in this case to take some pages for an intermediate page table from freelist P2M pool, set an owner domain to NULL (pg->inuse.domain = NULL).

Then in this case it isn't clear why pg->list can't be re-used to link several pages for intermediate page table purposes + metadata? Is it because pg->list can be not empty? In this case it isn't clear if I could use a page, which has threaded pages.

Actually looks like I was mis-remembering. Pages removed from freelist indeed aren't put on any other list, so the linking fields are available for use. I guess I had x86 shadow code in mind, where the linking fields are further used.

Perhaps, I misunderstood you about "linking fields", but it seems like I can't reuse struct page_info->list as it is used by page_list_add() which is called by p2m_alloc_page() to allocate page(s) for an intermediate page table:

     static inline void
     page_list_add(struct page_info *page, struct page_list_head *head)
     {
          list_add(&page->list, head);
     }

      struct page_info * paging_alloc_page(struct domain *d)
      {
          struct page_info *pg;

          spin_lock(&d->arch.paging.lock);
          pg = page_list_remove_head(&d->arch.paging.freelist);
          spin_unlock(&d->arch.paging.lock);

          INIT_LIST_HEAD(&pg->list);

          return pg;
      }

      static struct page_info *p2m_alloc_page(struct domain *d)
      {
          struct page_info *pg = paging_alloc_page(d);

          if ( pg )
              page_list_add(pg, &p2m_get_hostp2m(d)->pages);

          return pg;
      }

So I have to reuse another field from struct page_info. It seems like it won't be an issue if to add a new struct page_list_entry metadata_list to 'union v':

      union {
          /* Page is in use */
          struct {
              /* Owner of this page (NULL if page is anonymous). */
              struct domain *domain;
          } inuse;

          /* Page is on a free list. */
          struct {
              /* Order-size of the free chunk this page is the head of. */
              unsigned int order;
          } free;
(quoted diff) +2 -0
+
+       struct page_list_entry metadata_list;
      } v;
Quoted 5 lines
Quoted 1 line

Am I missing something?

Well, you're doubling the size of that union then, aren't you? As was mentioned quite some time ago, struct page_info needs quite a bit of care when you mean to add new fields there. Question is whether for the purpose here you actually need a doubly-linked list. A single pointer would be fine to put there.

Agree, a single pointer will be more then enough.

I'm thinking if it is possible to do something with the case if someone will try to use:

   #define page_get_owner(p)    (p)->v.inuse.domain
for a page which was allocated for metadata storage. Shouldn't I have a separate
list for such pages and a macro which will check if a page is in this list?
Similar a list which we have for p2m pages in struct p2m_domain:
     ...
     /* Pages used to construct the p2m */
     struct page_list_head pages;
     ...
Quoted 3 lines

Of course, such pages are allocated by alloc_domheap_page(d, MEMF_no_owner), so there is no owner. But if someone will accidentally use this macro for such pages then it will be an issue as ->domain likely won't be a NULL anymore.

It's the nature of using unions that such a risk exists. Take a look at x86'es structure, where several of the fields are re-purposed for shadow pages. It's something similar you'd do here, in the end.

Jan

#124 lore raw

Re: [PATCH v2 13/17] xen/riscv: Implement p2m_entry_from_mfn() and support PBMT configuration

In reply to Oleksii Kurochko
Quoted 69 lines · Oleksii Kurochko
Quoted 68 lines · Jan Beulich
Quoted 53 lines · Oleksii Kurochko
Quoted 52 lines · Jan Beulich
Quoted 28 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko
Quoted 14 lines · Jan Beulich

xen/arch/riscv/p2m.c static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn) · line 348

Quoted 7 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +1 -0
@@ -345,6 +345,26 @@ static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn)
          return __map_domain_page(p2m->root + root_table_indx);
      }
 
+static int p2m_type_radix_set(struct p2m_domain *p2m, pte_t pte, p2m_type_t t)

See comments on the earlier patch regarding naming.

xen/arch/riscv/p2m.c static pte_t *p2m_get_root_pointer(struct p2m_domain *p2m, gfn_t gfn) · line 351

Quoted 3 lines
xen/arch/riscv/p2m.c +3 -0
+{
+    int rc;
+    gfn_t gfn = mfn_to_gfn(p2m->domain, mfn_from_pte(pte));

How does this work, when you record GFNs only for Xenheap pages?

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think I don't understand what is an issue. Could you please provide some extra details?

Counter question: The mfn_to_gfn() you currently have is only a stub. It only works for 1:1 mapped domains. Can you show me the eventual final implementation of the function, making it possible to use it here?

At the moment, I planned to support only 1:1 mapped domains, so it is final implementation.

Isn't that on overly severe limitation?

I wouldn't say that it's a severe limitation, as it's just a matter of how

Quoted 8 lines
|mfn_to_gfn()| is implemented. When non-1:1 mapped domains are supported,
|mfn_to_gfn()| can be implemented differently, while the code where it’s called

will likely remain unchanged.

What I meant in my reply is that, for the current state and current limitations, this is the final implementation of|mfn_to_gfn()|. But that doesn't mean I don't see the value in, or the need for, non-1:1 mapped domains—it's just that this limitation simplifies development at the current stage of the RISC-V port.

Simplification is fine in some cases, but not supporting the "normal" way of domain construction looks like a pretty odd restriction. I'm also curious how you envision to implement mfn_to_gfn() then, suitable for generic use like the one here. Imo, current limitation or not, you simply want to avoid use of that function outside of the special gnttab case.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

In this context (not sure if I asked before): With this use of a radix tree, how do you intend to bound the amount of memory that a domain can use, by making Xen insert very many entries?

I didn’t think about that. I assumed it would be enough to set the amount of memory a guest domain can use by specifying|xen,domain-p2m-mem-mb| in the DTS, or using some predefined value if|xen,domain-p2m-mem-mb| isn’t explicitly set.

Which would require these allocations to come from that pool.

Yes, and it is true only for non-hardware domains with the current implementation.

???

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I meant that pool is used now only for non-hardware domains at the moment.

And how does this matter here? The memory required for the radix tree doesn't come from that pool anyway.

I thought that is possible to do that somehow, but looking at a code of radix-tree.c it seems like the only one way to allocate memroy for the radix tree isradix_tree_node_alloc() -> xzalloc(struct rcu_node).

Then it is needed to introduce radix_tree_node_allocate(domain)

That would be a possibility, but you may have seen that less than half a year ago we got rid of something along these lines. So it would require some pretty good justification to re-introduce.

Quoted 2 lines

or radix tree
can't be used at all for mentioned in the previous replies security reason, no?

Quoted 7 lines
Quoted 2 lines

(Very) careful use may still be possible. But the downside of using this (potentially long lookup times) would always remain.

Could you please clarify what do you mean here by "(Very) careful"?
I thought about an introduction of an amount of possible keys in radix tree and if this amount is 0 then stop domain. And it is also unclear what should be a value for this amount. Probably, you have better idea.

I had no particular idea in mind. I said "(very) careful" merely to clarify that whatever model is chosen, it would need to satisfy certain needs.

Jan

#125 lore raw

PATCH v2 14/17 xen/riscv: implement p2m_next_level()

In reply to Oleksii Kurochko

Implement the p2m_next_level() function, which enables traversal and dynamic allocation of intermediate levels (if necessary) in the RISC-V p2m (physical-to-machine) page table hierarchy.

To support this, the following helpers are introduced:
- p2me_is_mapping(): Determines whether a PTE represents a valid mapping.
- page_to_p2m_table(): Constructs non-leaf PTEs pointing to next-level page tables with correct attributes.
- p2m_alloc_page(): Allocates page table pages, supporting both hardware and guest domains.
- p2m_create_table(): Allocates and initializes a new page table page and installs it into the hierarchy.

---
Changes in V2:
- New patch. It was a part of a big patch "xen/riscv: implement p2m mapping functionality" which was splitted to smaller.
- s/p2m_is_mapping/p2me_is_mapping.

---
 xen/arch/riscv/p2m.c | 103 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 101 insertions(+), 2 deletions(-)
xen/arch/riscv/p2m.c +101 -2
index cba04acf38..87dd636b80 100644
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
     return p2m_type_radix_get(p2m, pte) != p2m_invalid;
 }
 
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}
+
 static inline bool p2me_is_superpage(struct p2m_domain *p2m, pte_t pte,
                                     unsigned int level)
 {
@@ -492,6 +503,70 @@ static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t,
     return e;
 }
 
+/* Generate table entry with correct attributes. */
+static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
+{
+    /*
+     * Since this function generates a table entry, according to "Encoding
+     * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
+     * to point to the next level of the page table.
+     * Therefore, to ensure that an entry is a page table entry,
+     * `p2m_access_n2rwx` is passed to `mfn_to_p2m_entry()` as the access value,
+     * which overrides whatever was passed as `p2m_type_t` and guarantees that
+     * the entry is a page table entry by setting r = w = x = 0.
+     */
+    return p2m_entry_from_mfn(p2m, page_to_mfn(page), p2m_ram_rw, p2m_access_n2rwx);
+}
+
+static struct page_info *p2m_alloc_page(struct domain *d)
+{
+    struct page_info *pg;
+
+    /*
+     * For hardware domain, there should be no limit in the number of pages that
+     * can be allocated, so that the kernel may take advantage of the extended
+     * regions. Hence, allocate p2m pages for hardware domains from heap.
+     */
+    if ( is_hardware_domain(d) )
+    {
+        pg = alloc_domheap_page(d, MEMF_no_owner);
+        if ( pg == NULL )
+            printk(XENLOG_G_ERR "Failed to allocate P2M pages for hwdom.\n");
+    }
+    else
+    {
+        spin_lock(&d->arch.paging.lock);
+        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
+        spin_unlock(&d->arch.paging.lock);
+    }
+
+    return pg;
+}
+
+/* Allocate a new page table page and hook it in via the given entry. */
+static int p2m_create_table(struct p2m_domain *p2m, pte_t *entry)
+{
+    struct page_info *page;
+    pte_t *p;
+
+    ASSERT(!p2me_is_valid(p2m, *entry));
+
+    page = p2m_alloc_page(p2m->domain);
+    if ( page == NULL )
+        return -ENOMEM;
+
+    page_list_add(page, &p2m->pages);
+
+    p = __map_domain_page(page);
+    clear_page(p);
+
+    unmap_domain_page(p);
+
+    p2m_write_pte(entry, page_to_p2m_table(p2m, page), p2m->clean_pte);
+
+    return 0;
+}
+
 #define GUEST_TABLE_MAP_NONE 0
 #define GUEST_TABLE_MAP_NOMEM 1
 #define GUEST_TABLE_SUPER_PAGE 2
@@ -516,9 +591,33 @@ static int p2m_next_level(struct p2m_domain *p2m, bool alloc_tbl,
                           unsigned int level, pte_t **table,
                           unsigned int offset)
 {
-    panic("%s: hasn't been implemented yet\n", __func__);
+    pte_t *entry;
+    int ret;
+    mfn_t mfn;
+
+    entry = *table + offset;
+
+    if ( !p2me_is_valid(p2m, *entry) )
+    {
+        if ( !alloc_tbl )
+            return GUEST_TABLE_MAP_NONE;
+
+        ret = p2m_create_table(p2m, entry);
+        if ( ret )
+            return GUEST_TABLE_MAP_NOMEM;
+    }
+
+    /* The function p2m_next_level() is never called at the last level */
+    ASSERT(level != 0);
+    if ( p2me_is_mapping(p2m, *entry) )
+        return GUEST_TABLE_SUPER_PAGE;
+
+    mfn = mfn_from_pte(*entry);
+
+    unmap_domain_page(*table);
+    *table = map_domain_page(mfn);
 
-    return GUEST_TABLE_MAP_NONE;
+    return GUEST_TABLE_NORMAL;
 }
 
 static void p2m_put_foreign_page(struct page_info *pg)
Signature · 1 line
2.49.0
#126 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
     return p2m_type_radix_get(p2m, pte) != p2m_invalid;
 }
 

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

Quoted 7 lines
xen/arch/riscv/p2m.c +3 -0
@@ -492,6 +503,70 @@ static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t,
     return e;
 }
 
+/* Generate table entry with correct attributes. */
+static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
+{

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 518

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+    /*
+     * Since this function generates a table entry, according to "Encoding
+     * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
+     * to point to the next level of the page table.
+     * Therefore, to ensure that an entry is a page table entry,
+     * `p2m_access_n2rwx` is passed to `mfn_to_p2m_entry()` as the access value,
+     * which overrides whatever was passed as `p2m_type_t` and guarantees that
+     * the entry is a page table entry by setting r = w = x = 0.
+     */
+    return p2m_entry_from_mfn(p2m, page_to_mfn(page), p2m_ram_rw, p2m_access_n2rwx);

Similarly P2M access shouldn't apply to intermediate page tables. (Moot with that, but (ab)using p2m_access_n2rwx would also look wrong: You did read what it means, didn't you?)

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+}
+
+static struct page_info *p2m_alloc_page(struct domain *d)
+{
+    struct page_info *pg;
+
+    /*

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 535

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+     * For hardware domain, there should be no limit in the number of pages that
+     * can be allocated, so that the kernel may take advantage of the extended
+     * regions. Hence, allocate p2m pages for hardware domains from heap.
+     */
+    if ( is_hardware_domain(d) )
+    {
+        pg = alloc_domheap_page(d, MEMF_no_owner);
+        if ( pg == NULL )
+            printk(XENLOG_G_ERR "Failed to allocate P2M pages for hwdom.\n");
+    }

The comment looks to have been taken verbatim from Arm. Whatever "extended regions" are, does the same concept even exist on RISC-V?

Also, special casing Dom0 like this has benefits, but also comes with a
pitfall: If the system's out of memory, allocations will fail. A pre- populated pool would avoid that (until exhausted, of course). If special- casing of Dom0 is needed, I wonder whether ...

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 541

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    else
+    {
+        spin_lock(&d->arch.paging.lock);
+        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
+        spin_unlock(&d->arch.paging.lock);
+    }

... going this path but with a Dom0-only fallback to general allocation wouldn't be the better route.

Quoted 11 lines
xen/arch/riscv/p2m.c +11 -0
+    return pg;
+}
+
+/* Allocate a new page table page and hook it in via the given entry. */
+static int p2m_create_table(struct p2m_domain *p2m, pte_t *entry)
+{
+    struct page_info *page;
+    pte_t *p;
+
+    ASSERT(!p2me_is_valid(p2m, *entry));
+

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 563

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+    page = p2m_alloc_page(p2m->domain);
+    if ( page == NULL )
+        return -ENOMEM;
+
+    page_list_add(page, &p2m->pages);
+
+    p = __map_domain_page(page);
+    clear_page(p);
+
+    unmap_domain_page(p);

clear_domain_page()? Or actually clear_and_clean_page()?

Quoted 13 lines
xen/arch/riscv/p2m.c +8 -1
@@ -516,9 +591,33 @@ static int p2m_next_level(struct p2m_domain *p2m, bool alloc_tbl,
                           unsigned int level, pte_t **table,
                           unsigned int offset)
 {
-    panic("%s: hasn't been implemented yet\n", __func__);
+    pte_t *entry;
+    int ret;
+    mfn_t mfn;
+
+    entry = *table + offset;
+
+    if ( !p2me_is_valid(p2m, *entry) )
+    {

xen/arch/riscv/p2m.c static int p2m_next_level(struct p2m_domain *p2m, bool alloc_tbl, · line 611

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+        if ( !alloc_tbl )
+            return GUEST_TABLE_MAP_NONE;
+
+        ret = p2m_create_table(p2m, entry);
+        if ( ret )
+            return GUEST_TABLE_MAP_NOMEM;
+    }
+
+    /* The function p2m_next_level() is never called at the last level */
+    ASSERT(level != 0);

Logically you would perhaps better do this ahead of trying to allocate a page table. Calls here with level == 0 are invalid in all cases aiui, not just when you make it here.

xen/arch/riscv/p2m.c static int p2m_next_level(struct p2m_domain *p2m, bool alloc_tbl, · line 618

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    if ( p2me_is_mapping(p2m, *entry) )
+        return GUEST_TABLE_SUPER_PAGE;
+
+    mfn = mfn_from_pte(*entry);
+
+    unmap_domain_page(*table);
+    *table = map_domain_page(mfn);

Just to mention it (may not need taking care of right away), there's an inefficiency here: In p2m_create_table() you map the page to clear it. Then you tear down that mapping, just to re-establish it here.

Jan

#127 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 7 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
      return p2m_type_radix_get(p2m, pte) != p2m_invalid;
  }
 
Quoted 12 lines

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

It doesn't matter whether it is an intermediate page table or a leaf PTE pointing to a page — PTE should be valid. Considering that in the current implementation it’s possible for PTE.v = 0 but P2M.v = 1, it is better to check P2M.v instead of PTE.v.

Quoted 5 lines
Quoted 5 lines
xen/arch/riscv/p2m.c +0 -0
@@ -492,6 +503,70 @@ static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t,
      return e;
  }
 
Quoted 16 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 518

Quoted 13 lines
xen/arch/riscv/p2m.c +13 -0
+/* Generate table entry with correct attributes. */
+static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
+{
+    /*
+     * Since this function generates a table entry, according to "Encoding
+     * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
+     * to point to the next level of the page table.
+     * Therefore, to ensure that an entry is a page table entry,
+     * `p2m_access_n2rwx` is passed to `mfn_to_p2m_entry()` as the access value,
+     * which overrides whatever was passed as `p2m_type_t` and guarantees that
+     * the entry is a page table entry by setting r = w = x = 0.
+     */
+    return p2m_entry_from_mfn(p2m, page_to_mfn(page), p2m_ram_rw, p2m_access_n2rwx);

Similarly P2M access shouldn't apply to intermediate page tables. (Moot with that, but (ab)using p2m_access_n2rwx would also look wrong: You did read what it means, didn't you?)

|p2m_access_n2rwx| was chosen not really because of the description mentioned near its declaration, but because it sets r=w=x=0, which RISC-V expects for a PTE that points to the next-level page table.

Generally, I agree that P2M access shouldn't be applied to intermediate page tables.

What I can suggest in this case is to use|p2m_access_rwx| instead of|p2m_access_n2rwx|,
which will ensure that the P2M access type isn't applied when|p2m_entry_from_mfn() |is called, and then, after calling|p2m_entry_from_mfn()|, simply set|PTE.r,w,x=0|. So this function will look like:

    /* Generate table entry with correct attributes. */
    static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
    {
        /*
        * p2m_ram_rw is chosen for a table entry as p2m table should be valid
        * from both P2M and hardware point of view.
        *
        * p2m_access_rwx is chosen to restrict access permissions, what mean
        * do not apply access permission for a table entry
        */
        pte_t pte = p2m_pte_from_mfn(p2m, page_to_mfn(page), _gfn(0), p2m_ram_rw,
                                    p2m_access_rwx);

        /*
        * Since this function generates a table entry, according to "Encoding
        * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
        * to point to the next level of the page table.
        */
        pte.pte &= ~PTE_ACCESS_MASK;

        return pte;
    }

Does this make sense? Or would it be better to keep the current version of |page_to_p2m_table()| and just improve the comment explaining why|p2m_access_n2rwx |is used for a table entry?

Quoted 12 lines
Quoted 12 lines
xen/arch/riscv/p2m.c +11 -0
+}
+
+static struct page_info *p2m_alloc_page(struct domain *d)
+{
+    struct page_info *pg;
+
+    /*
+     * For hardware domain, there should be no limit in the number of pages that
+     * can be allocated, so that the kernel may take advantage of the extended
+     * regions. Hence, allocate p2m pages for hardware domains from heap.
+     */
Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 535

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    if ( is_hardware_domain(d) )
+    {
+        pg = alloc_domheap_page(d, MEMF_no_owner);
+        if ( pg == NULL )
+            printk(XENLOG_G_ERR "Failed to allocate P2M pages for hwdom.\n");
+    }

The comment looks to have been taken verbatim from Arm. Whatever "extended regions" are, does the same concept even exist on RISC-V?

Initially, I missed that it’s used only for Arm. Since it was mentioned in |doc/misc/xen-command-line.pandoc|, I assumed it applied to all architectures. But now I see that it’s Arm-specific:: ### ext_regions (Arm)

Quoted 6 lines

Also, special casing Dom0 like this has benefits, but also comes with a
pitfall: If the system's out of memory, allocations will fail. A pre- populated pool would avoid that (until exhausted, of course). If special- casing of Dom0 is needed, I wonder whether ...

Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 541

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    else
+    {
+        spin_lock(&d->arch.paging.lock);
+        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
+        spin_unlock(&d->arch.paging.lock);
+    }

... going this path but with a Dom0-only fallback to general allocation wouldn't be the better route.

IIUC, then it should be something like:
static struct page_info *p2m_alloc_page(struct domain *d)

  {
      struct page_info *pg;
      
      spin_lock(&d->arch.paging.lock);
      pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
      spin_unlock(&d->arch.paging.lock);

      if ( !pg && is_hardware_domain(d) )
      {
            /* Need to allocate more memory from domheap */
            pg = alloc_domheap_page(d, MEMF_no_owner);
            if ( pg == NULL )
            {
                printk(XENLOG_ERR "Failed to allocate pages.\n");
                return pg;
            }
            ACCESS_ONCE(d->arch.paging.total_pages)++;
            page_list_add_tail(pg, &d->arch.paging.freelist);
      }
   
      return pg;
}

And basically use|d->arch.paging.freelist| for both dom0less and dom0 domains,
with the only difference being that in the case of Dom0,|d->arch.paging.freelist |could be extended.

Do I understand your idea correctly?

(
Probably, this is the reply you’re referring to:
https://lore.kernel.org/xen-devel/[email protected]/,
at the moment, I can't find a better one.
)

Quoted 15 lines
Quoted 15 lines
xen/arch/riscv/p2m.c +14 -0
+    return pg;
+}
+
+/* Allocate a new page table page and hook it in via the given entry. */
+static int p2m_create_table(struct p2m_domain *p2m, pte_t *entry)
+{
+    struct page_info *page;
+    pte_t *p;
+
+    ASSERT(!p2me_is_valid(p2m, *entry));
+
+    page = p2m_alloc_page(p2m->domain);
+    if ( page == NULL )
+        return -ENOMEM;
Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 563

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+
+    page_list_add(page, &p2m->pages);
+
+    p = __map_domain_page(page);
+    clear_page(p);
+
+    unmap_domain_page(p);

clear_domain_page()? Or actually clear_and_clean_page()?

Agree, clear_and_clean_page() would be better here.

Quoted 19 lines
Quoted 19 lines
xen/arch/riscv/p2m.c +13 -1
@@ -516,9 +591,33 @@ static int p2m_next_level(struct p2m_domain *p2m, bool alloc_tbl,
                            unsigned int level, pte_t **table,
                            unsigned int offset)
  {
-    panic("%s: hasn't been implemented yet\n", __func__);
+    pte_t *entry;
+    int ret;
+    mfn_t mfn;
+
+    entry = *table + offset;
+
+    if ( !p2me_is_valid(p2m, *entry) )
+    {
+        if ( !alloc_tbl )
+            return GUEST_TABLE_MAP_NONE;
+
+        ret = p2m_create_table(p2m, entry);
+        if ( ret )
Quoted 8 lines

xen/arch/riscv/p2m.c static int p2m_next_level(struct p2m_domain *p2m, bool alloc_tbl, · line 611

Quoted 5 lines
xen/arch/riscv/p2m.c +5 -0
+            return GUEST_TABLE_MAP_NOMEM;
+    }
+
+    /* The function p2m_next_level() is never called at the last level */
+    ASSERT(level != 0);

Logically you would perhaps better do this ahead of trying to allocate a page table. Calls here with level == 0 are invalid in all cases aiui, not just when you make it here.

It makes sense. I will move ASSERT() to the start of the function p2m_next_level().

Quoted 10 lines

xen/arch/riscv/p2m.c static int p2m_next_level(struct p2m_domain *p2m, bool alloc_tbl, · line 618

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    if ( p2me_is_mapping(p2m, *entry) )
+        return GUEST_TABLE_SUPER_PAGE;
+
+    mfn = mfn_from_pte(*entry);
+
+    unmap_domain_page(*table);
+    *table = map_domain_page(mfn);

Just to mention it (may not need taking care of right away), there's an inefficiency here: In p2m_create_table() you map the page to clear it. Then you tear down that mapping, just to re-establish it here.

I will add:
/*
* TODO: There's an inefficiency here:
* In p2m_create_table(), the page is mapped to clear it.
* Then that mapping is torn down in p2m_create_table(),
* only to be re-established here.
*/
*table = map_domain_page(mfn);

Thanks.

~ Oleksii

#128 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 21 lines · Oleksii Kurochko
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
      return p2m_type_radix_get(p2m, pte) != p2m_invalid;
  }
 
Quoted 12 lines

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

Quoted 4 lines

It doesn't matter whether it is an intermediate page table or a leaf PTE pointing to a page — PTE should be valid. Considering that in the current implementation it’s possible for PTE.v = 0 but P2M.v = 1, it is better to check P2M.v instead of PTE.v.

I'm confused by this reply. If you want to name 2nd level page table entries P2M - fine (but unhelpful). But then for any memory access there's only one of the two involved: A PTE (Xen accesses) or a P2M (guest accesses). Hence how can there be "PTE.v = 0 but P2M.v = 1"?

An intermediate page table entry is something Xen controls entirely. Hence it has no (guest induced) type.

Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
xen/arch/riscv/p2m.c +0 -0
@@ -492,6 +503,70 @@ static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t,
      return e;
  }
 
Quoted 16 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 518

Quoted 13 lines
xen/arch/riscv/p2m.c +13 -0
+/* Generate table entry with correct attributes. */
+static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
+{
+    /*
+     * Since this function generates a table entry, according to "Encoding
+     * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
+     * to point to the next level of the page table.
+     * Therefore, to ensure that an entry is a page table entry,
+     * `p2m_access_n2rwx` is passed to `mfn_to_p2m_entry()` as the access value,
+     * which overrides whatever was passed as `p2m_type_t` and guarantees that
+     * the entry is a page table entry by setting r = w = x = 0.
+     */
+    return p2m_entry_from_mfn(p2m, page_to_mfn(page), p2m_ram_rw, p2m_access_n2rwx);

Similarly P2M access shouldn't apply to intermediate page tables. (Moot with that, but (ab)using p2m_access_n2rwx would also look wrong: You did read what it means, didn't you?)

Quoted 7 lines

|p2m_access_n2rwx| was chosen not really because of the description mentioned near its declaration, but because it sets r=w=x=0, which RISC-V expects for a PTE that points to the next-level page table.

Generally, I agree that P2M access shouldn't be applied to intermediate page tables.

What I can suggest in this case is to use|p2m_access_rwx| instead of|p2m_access_n2rwx|,

No. p2m_access_* shouldn't come into play here at all. Period. Just like P2M types shouldn't. As per above - intermediate page tables are Xen internal constructs.

Quoted 25 lines

which will ensure that the P2M access type isn't applied when|p2m_entry_from_mfn() |is called, and then, after calling|p2m_entry_from_mfn()|, simply set|PTE.r,w,x=0|. So this function will look like:

     /* Generate table entry with correct attributes. */
     static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
     {
         /*
         * p2m_ram_rw is chosen for a table entry as p2m table should be valid
         * from both P2M and hardware point of view.
         *
         * p2m_access_rwx is chosen to restrict access permissions, what mean
         * do not apply access permission for a table entry
         */
         pte_t pte = p2m_pte_from_mfn(p2m, page_to_mfn(page), _gfn(0), p2m_ram_rw,
                                     p2m_access_rwx);

         /*
         * Since this function generates a table entry, according to "Encoding
         * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
         * to point to the next level of the page table.
         */
         pte.pte &= ~PTE_ACCESS_MASK;

         return pte;
     }
Quoted 2 lines

Does this make sense? Or would it be better to keep the current version of |page_to_p2m_table()| and just improve the comment explaining why|p2m_access_n2rwx |is used for a table entry?

No to both, as per above.

Quoted 62 lines
Quoted 9 lines
Quoted 9 lines
xen/arch/riscv/p2m.c +9 -0
+static struct page_info *p2m_alloc_page(struct domain *d)
+{
+    struct page_info *pg;
+
+    /*
+     * For hardware domain, there should be no limit in the number of pages that
+     * can be allocated, so that the kernel may take advantage of the extended
+     * regions. Hence, allocate p2m pages for hardware domains from heap.
+     */
Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 535

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    if ( is_hardware_domain(d) )
+    {
+        pg = alloc_domheap_page(d, MEMF_no_owner);
+        if ( pg == NULL )
+            printk(XENLOG_G_ERR "Failed to allocate P2M pages for hwdom.\n");
+    }

The comment looks to have been taken verbatim from Arm. Whatever "extended regions" are, does the same concept even exist on RISC-V?

Initially, I missed that it’s used only for Arm. Since it was mentioned in |doc/misc/xen-command-line.pandoc|, I assumed it applied to all architectures. But now I see that it’s Arm-specific:: ### ext_regions (Arm)

Quoted 6 lines

Also, special casing Dom0 like this has benefits, but also comes with a
pitfall: If the system's out of memory, allocations will fail. A pre- populated pool would avoid that (until exhausted, of course). If special- casing of Dom0 is needed, I wonder whether ...

Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 541

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    else
+    {
+        spin_lock(&d->arch.paging.lock);
+        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
+        spin_unlock(&d->arch.paging.lock);
+    }

... going this path but with a Dom0-only fallback to general allocation wouldn't be the better route.

IIUC, then it should be something like:

   static struct page_info *p2m_alloc_page(struct domain *d)
   {
       struct page_info *pg;
       
       spin_lock(&d->arch.paging.lock);
       pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
       spin_unlock(&d->arch.paging.lock);

       if ( !pg && is_hardware_domain(d) )
       {
             /* Need to allocate more memory from domheap */
             pg = alloc_domheap_page(d, MEMF_no_owner);
             if ( pg == NULL )
             {
                 printk(XENLOG_ERR "Failed to allocate pages.\n");
                 return pg;
             }
             ACCESS_ONCE(d->arch.paging.total_pages)++;
             page_list_add_tail(pg, &d->arch.paging.freelist);
       }
    
       return pg;
}
Quoted 4 lines

And basically use|d->arch.paging.freelist| for both dom0less and dom0 domains,
with the only difference being that in the case of Dom0,|d->arch.paging.freelist |could be extended.

Do I understand your idea correctly?

Broadly yes, but not in the details. For example, I don't think such a page allocated from the general heap would want appending to freelist. Commentary and alike also would want tidying.

And of course going forward, for split hardware and control domains the latter may want similar treatment.

Jan

#129 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 21 lines · Jan Beulich
Quoted 20 lines · Oleksii Kurochko
Quoted 19 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
       return p2m_type_radix_get(p2m, pte) != p2m_invalid;
   }
 

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

Quoted 8 lines
Quoted 4 lines

It doesn't matter whether it is an intermediate page table or a leaf PTE pointing to a page — PTE should be valid. Considering that in the current implementation it’s possible for PTE.v = 0 but P2M.v = 1, it is better to check P2M.v instead of PTE.v.

I'm confused by this reply. If you want to name 2nd level page table entries P2M - fine (but unhelpful). But then for any memory access there's only one of the two involved: A PTE (Xen accesses) or a P2M (guest accesses). Hence how can there be "PTE.v = 0 but P2M.v = 1"?

I think I understand your confusion, let me try to rephrase.

The reason for having both|p2m_is_valid()| and|pte_is_valid()| is that I want to have the ability to use the P2M PTE valid bit to track which pages were accessed by a vCPU, so that cleaning and invalidating RAM associated with the guest vCPU won't be too expensive, for example.
In this case, the P2M PTE valid bit will be set to 0, but the P2M PTE type bits will be set to something other than|p2m_invalid| (even for a table entries), so when an MMU fault occurs, we can properly resolve it.

So, if the P2M PTE type (what|p2m_is_valid()| checks) is set to|p2m_invalid|, it means that the valid bit (what|pte_is_valid()| checks) should be set to 0, so the P2M PTE is genuinely invalid.

It could also be the case that the P2M PTE type isn't|p2m_invalid (and P2M PTE valid will be intentionally set to 0 to have ability to track which pages were accessed for the reason I wrote above)|, and when MMU fault occurs we could properly handle it and set to 1 P2M PTE valid bit to 1...

Quoted 3 lines

An intermediate page table entry is something Xen controls entirely. Hence it has no (guest induced) type.

... And actually it is a reason why it is needed to set a type even for an intermediate page table entry.

I hope now it is a lit bit clearer what and why was done.

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
xen/arch/riscv/p2m.c +3 -0
@@ -492,6 +503,70 @@ static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t,
       return e;
   }
 
+/* Generate table entry with correct attributes. */
+static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
+{

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 518

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+    /*
+     * Since this function generates a table entry, according to "Encoding
+     * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
+     * to point to the next level of the page table.
+     * Therefore, to ensure that an entry is a page table entry,
+     * `p2m_access_n2rwx` is passed to `mfn_to_p2m_entry()` as the access value,
+     * which overrides whatever was passed as `p2m_type_t` and guarantees that
+     * the entry is a page table entry by setting r = w = x = 0.
+     */
+    return p2m_entry_from_mfn(p2m, page_to_mfn(page), p2m_ram_rw, p2m_access_n2rwx);

Similarly P2M access shouldn't apply to intermediate page tables. (Moot with that, but (ab)using p2m_access_n2rwx would also look wrong: You did read what it means, didn't you?)

Quoted 8 lines
Quoted 7 lines

|p2m_access_n2rwx| was chosen not really because of the description mentioned near its declaration, but because it sets r=w=x=0, which RISC-V expects for a PTE that points to the next-level page table.

Generally, I agree that P2M access shouldn't be applied to intermediate page tables.

What I can suggest in this case is to use|p2m_access_rwx| instead of|p2m_access_n2rwx|,

No. p2m_access_* shouldn't come into play here at all.

Okay, then it seems like I just can't explicitly re-use p2m_pte_from_mfn() in page_to_p2m_table() and have to open-code p2m_pte_from_mfn() or add another one argument is_table to decide if p2m_access_t and/or p2m_type_t should be applied.

Quoted 2 lines

Period. Just like P2M types
shouldn't. As per above - intermediate page tables are Xen internal constructs.

Look please at the explaining above why p2m types is needed despite of the fact that logically it isn't really needed.

Quoted 89 lines
Quoted 26 lines

which will ensure that the P2M access type isn't applied when|p2m_entry_from_mfn() |is called, and then, after calling|p2m_entry_from_mfn()|, simply set|PTE.r,w,x=0|. So this function will look like:

      /* Generate table entry with correct attributes. */
      static pte_t page_to_p2m_table(struct p2m_domain *p2m, struct page_info *page)
      {
          /*
          * p2m_ram_rw is chosen for a table entry as p2m table should be valid
          * from both P2M and hardware point of view.
          *
          * p2m_access_rwx is chosen to restrict access permissions, what mean
          * do not apply access permission for a table entry
          */
          pte_t pte = p2m_pte_from_mfn(p2m, page_to_mfn(page), _gfn(0), p2m_ram_rw,
                                      p2m_access_rwx);

          /*
          * Since this function generates a table entry, according to "Encoding
          * of PTE R/W/X fields," the entry's r, w, and x fields must be set to 0
          * to point to the next level of the page table.
          */
          pte.pte &= ~PTE_ACCESS_MASK;

          return pte;
      }
Quoted 2 lines

Does this make sense? Or would it be better to keep the current version of |page_to_p2m_table()| and just improve the comment explaining why|p2m_access_n2rwx |is used for a table entry?

No to both, as per above.

Quoted 59 lines
Quoted 9 lines
Quoted 9 lines
xen/arch/riscv/p2m.c +9 -0
+static struct page_info *p2m_alloc_page(struct domain *d)
+{
+    struct page_info *pg;
+
+    /*
+     * For hardware domain, there should be no limit in the number of pages that
+     * can be allocated, so that the kernel may take advantage of the extended
+     * regions. Hence, allocate p2m pages for hardware domains from heap.
+     */
Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 535

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    if ( is_hardware_domain(d) )
+    {
+        pg = alloc_domheap_page(d, MEMF_no_owner);
+        if ( pg == NULL )
+            printk(XENLOG_G_ERR "Failed to allocate P2M pages for hwdom.\n");
+    }

The comment looks to have been taken verbatim from Arm. Whatever "extended regions" are, does the same concept even exist on RISC-V?

Initially, I missed that it’s used only for Arm. Since it was mentioned in |doc/misc/xen-command-line.pandoc|, I assumed it applied to all architectures. But now I see that it’s Arm-specific:: ### ext_regions (Arm)

Quoted 5 lines

Also, special casing Dom0 like this has benefits, but also comes with a
pitfall: If the system's out of memory, allocations will fail. A pre- populated pool would avoid that (until exhausted, of course). If special- casing of Dom0 is needed, I wonder whether ...

Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 541

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    else
+    {
+        spin_lock(&d->arch.paging.lock);
+        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
+        spin_unlock(&d->arch.paging.lock);
+    }

... going this path but with a Dom0-only fallback to general allocation wouldn't be the better route.

IIUC, then it should be something like:

    static struct page_info *p2m_alloc_page(struct domain *d)
    {
        struct page_info *pg;
        
        spin_lock(&d->arch.paging.lock);
        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
        spin_unlock(&d->arch.paging.lock);

        if ( !pg && is_hardware_domain(d) )
        {
              /* Need to allocate more memory from domheap */
              pg = alloc_domheap_page(d, MEMF_no_owner);
              if ( pg == NULL )
              {
                  printk(XENLOG_ERR "Failed to allocate pages.\n");
                  return pg;
              }
              ACCESS_ONCE(d->arch.paging.total_pages)++;
              page_list_add_tail(pg, &d->arch.paging.freelist);
        }
     
        return pg;
}
Quoted 7 lines
Quoted 4 lines

And basically use|d->arch.paging.freelist| for both dom0less and dom0 domains,
with the only difference being that in the case of Dom0,|d->arch.paging.freelist |could be extended.

Do I understand your idea correctly?

Broadly yes, but not in the details. For example, I don't think such a page allocated from the general heap would want appending to freelist. Commentary and alike also would want tidying.

Could you please explain why it wouldn't want appending to freelist?

Quoted 3 lines

And of course going forward, for split hardware and control domains the latter may want similar treatment.

Could you please clarify what is the difference between hardware and control domains?
I thought that it is the same or is it for the case when we have dom0 (control domain) which runs domD (hardware domain) and guest domain?

Thanks.

~ Oleksii

#130 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 31 lines · Oleksii Kurochko
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Oleksii Kurochko
Quoted 19 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
       return p2m_type_radix_get(p2m, pte) != p2m_invalid;
   }
 

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

Quoted 8 lines
Quoted 4 lines

It doesn't matter whether it is an intermediate page table or a leaf PTE pointing to a page — PTE should be valid. Considering that in the current implementation it’s possible for PTE.v = 0 but P2M.v = 1, it is better to check P2M.v instead of PTE.v.

I'm confused by this reply. If you want to name 2nd level page table entries P2M - fine (but unhelpful). But then for any memory access there's only one of the two involved: A PTE (Xen accesses) or a P2M (guest accesses). Hence how can there be "PTE.v = 0 but P2M.v = 1"?

Quoted 6 lines

I think I understand your confusion, let me try to rephrase.

The reason for having both|p2m_is_valid()| and|pte_is_valid()| is that I want to have the ability to use the P2M PTE valid bit to track which pages were accessed by a vCPU, so that cleaning and invalidating RAM associated with the guest vCPU won't be too expensive, for example.

I don't know what you're talking about here.

Quoted 13 lines

In this case, the P2M PTE valid bit will be set to 0, but the P2M PTE type bits will be set to something other than|p2m_invalid| (even for a table entries), so when an MMU fault occurs, we can properly resolve it.

So, if the P2M PTE type (what|p2m_is_valid()| checks) is set to|p2m_invalid|, it means that the valid bit (what|pte_is_valid()| checks) should be set to 0, so the P2M PTE is genuinely invalid.

It could also be the case that the P2M PTE type isn't|p2m_invalid (and P2M PTE valid will be intentionally set to 0 to have ability to track which pages were accessed for the reason I wrote above)|, and when MMU fault occurs we could properly handle it and set to 1 P2M PTE valid bit to 1...

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

An intermediate page table entry is something Xen controls entirely. Hence it has no (guest induced) type.

... And actually it is a reason why it is needed to set a type even for an intermediate page table entry.

I hope now it is a lit bit clearer what and why was done.

Sadly not. I still don't see what use the P2M type in of an intermediate page table is going to be. It surely can't reliably describe all of the entries that page table holds. Intermediate page tables and leaf pages are just too different to share a concept like this, I think. That said, I'll be happy to be shown code demonstrating the contrary.

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 9 lines
xen/arch/riscv/p2m.c +9 -0
+static struct page_info *p2m_alloc_page(struct domain *d)
+{
+    struct page_info *pg;
+
+    /*
+     * For hardware domain, there should be no limit in the number of pages that
+     * can be allocated, so that the kernel may take advantage of the extended
+     * regions. Hence, allocate p2m pages for hardware domains from heap.
+     */
Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 535

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    if ( is_hardware_domain(d) )
+    {
+        pg = alloc_domheap_page(d, MEMF_no_owner);
+        if ( pg == NULL )
+            printk(XENLOG_G_ERR "Failed to allocate P2M pages for hwdom.\n");
+    }

The comment looks to have been taken verbatim from Arm. Whatever "extended regions" are, does the same concept even exist on RISC-V?

Initially, I missed that it’s used only for Arm. Since it was mentioned in |doc/misc/xen-command-line.pandoc|, I assumed it applied to all architectures. But now I see that it’s Arm-specific:: ### ext_regions (Arm)

Quoted 13 lines

Also, special casing Dom0 like this has benefits, but also comes with a
pitfall: If the system's out of memory, allocations will fail. A pre- populated pool would avoid that (until exhausted, of course). If special- casing of Dom0 is needed, I wonder whether ...

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 541

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    else
+    {
+        spin_lock(&d->arch.paging.lock);
+        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
+        spin_unlock(&d->arch.paging.lock);
+    }

... going this path but with a Dom0-only fallback to general allocation wouldn't be the better route.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

IIUC, then it should be something like:

    static struct page_info *p2m_alloc_page(struct domain *d)
    {
        struct page_info *pg;
        
        spin_lock(&d->arch.paging.lock);
        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);

Note this: Here you _remove_ from freelist, because you want to actually use the page. Then clearly ...

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
        spin_unlock(&d->arch.paging.lock);

        if ( !pg && is_hardware_domain(d) )
        {
              /* Need to allocate more memory from domheap */
              pg = alloc_domheap_page(d, MEMF_no_owner);
              if ( pg == NULL )
              {
                  printk(XENLOG_ERR "Failed to allocate pages.\n");
                  return pg;
              }
              ACCESS_ONCE(d->arch.paging.total_pages)++;
              page_list_add_tail(pg, &d->arch.paging.freelist);
        }
     
        return pg;
}

And basically use|d->arch.paging.freelist| for both dom0less and dom0 domains,
with the only difference being that in the case of Dom0,|d->arch.paging.freelist |could be extended.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Do I understand your idea correctly?

Broadly yes, but not in the details. For example, I don't think such a page allocated from the general heap would want appending to freelist. Commentary and alike also would want tidying.

Could you please explain why it wouldn't want appending to freelist?

... adding to freelist here is wrong: You want to use this separately allocated page, too. Else once it is freed it'll be added to freelist a 2nd time, leading to a corrupt list.

Quoted 7 lines
Quoted 2 lines

And of course going forward, for split hardware and control domains the latter may want similar treatment.

Could you please clarify what is the difference between hardware and control domains?
I thought that it is the same or is it for the case when we have dom0 (control domain) which runs domD (hardware domain) and guest domain?

That's the common case, yes, but conceptually the two can be separate. And if you've followed recent discussions on the list you would also have noticed that work is being done in that direction. (But this was really a forward-looking comment; I didn't mean to make you cover that case right away. Just wanted you to be aware.)

Jan

#131 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 31 lines · Jan Beulich
Quoted 30 lines · Oleksii Kurochko
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Oleksii Kurochko
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
        return p2m_type_radix_get(p2m, pte) != p2m_invalid;
    }
 
Quoted 12 lines

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

Quoted 4 lines

It doesn't matter whether it is an intermediate page table or a leaf PTE pointing to a page — PTE should be valid. Considering that in the current implementation it’s possible for PTE.v = 0 but P2M.v = 1, it is better to check P2M.v instead of PTE.v.

I'm confused by this reply. If you want to name 2nd level page table entries P2M - fine (but unhelpful). But then for any memory access there's only one of the two involved: A PTE (Xen accesses) or a P2M (guest accesses). Hence how can there be "PTE.v = 0 but P2M.v = 1"?

Quoted 7 lines
Quoted 6 lines

I think I understand your confusion, let me try to rephrase.

The reason for having both|p2m_is_valid()| and|pte_is_valid()| is that I want to have the ability to use the P2M PTE valid bit to track which pages were accessed by a vCPU, so that cleaning and invalidating RAM associated with the guest vCPU won't be too expensive, for example.

I don't know what you're talking about here.

https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/mmu/p2m.c#L1649

Quoted 18 lines
Quoted 18 lines

In this case, the P2M PTE valid bit will be set to 0, but the P2M PTE type bits will be set to something other than|p2m_invalid| (even for a table entries), so when an MMU fault occurs, we can properly resolve it.

So, if the P2M PTE type (what|p2m_is_valid()| checks) is set to|p2m_invalid|, it means that the valid bit (what|pte_is_valid()| checks) should be set to 0, so the P2M PTE is genuinely invalid.

It could also be the case that the P2M PTE type isn't|p2m_invalid (and P2M PTE valid will be intentionally set to 0 to have ability to track which pages were accessed for the reason I wrote above)|, and when MMU fault occurs we could properly handle it and set to 1 P2M PTE valid bit to 1...

Quoted 2 lines

An intermediate page table entry is something Xen controls entirely. Hence it has no (guest induced) type.

... And actually it is a reason why it is needed to set a type even for an intermediate page table entry.

Quoted 6 lines
Quoted 1 line

I hope now it is a lit bit clearer what and why was done.

Sadly not. I still don't see what use the P2M type in of an intermediate page table is going to be. It surely can't reliably describe all of the entries that page table holds. Intermediate page tables and leaf pages are just too different to share a concept like this, I think. That said, I'll be happy to be shown code demonstrating the contrary.

Then it is needed to introduce new p2m_type_t - p2m_table and use it. Would it be better?

I still need some type to have ability to distinguish if p2m is valid or not from p2m management and hardware point of view.
If there is no need for such distinguish why all archs introduce p2m_invalid? Isn't enough just to use P2M PTE valid bit?

Quoted 66 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 10 lines
Quoted 10 lines
xen/arch/riscv/p2m.c +9 -0
+static struct page_info *p2m_alloc_page(struct domain *d)
+{
+    struct page_info *pg;
+
+    /*
+     * For hardware domain, there should be no limit in the number of pages that
+     * can be allocated, so that the kernel may take advantage of the extended
+     * regions. Hence, allocate p2m pages for hardware domains from heap.
+     */
Quoted 8 lines

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 535

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    if ( is_hardware_domain(d) )
+    {
+        pg = alloc_domheap_page(d, MEMF_no_owner);
+        if ( pg == NULL )
+            printk(XENLOG_G_ERR "Failed to allocate P2M pages for hwdom.\n");
+    }

The comment looks to have been taken verbatim from Arm. Whatever "extended regions" are, does the same concept even exist on RISC-V?

Initially, I missed that it’s used only for Arm. Since it was mentioned in |doc/misc/xen-command-line.pandoc|, I assumed it applied to all architectures. But now I see that it’s Arm-specific:: ### ext_regions (Arm)

Quoted 13 lines

Also, special casing Dom0 like this has benefits, but also comes with a
pitfall: If the system's out of memory, allocations will fail. A pre- populated pool would avoid that (until exhausted, of course). If special- casing of Dom0 is needed, I wonder whether ...

xen/arch/riscv/p2m.c static pte_t p2m_entry_from_mfn(struct p2m_domain *p2m, mfn_t mfn, p2m_type_t t, · line 541

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    else
+    {
+        spin_lock(&d->arch.paging.lock);
+        pg = page_list_remove_head(&d->arch.paging.p2m_freelist);
+        spin_unlock(&d->arch.paging.lock);
+    }

... going this path but with a Dom0-only fallback to general allocation wouldn't be the better route.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

IIUC, then it should be something like:

     static struct page_info *p2m_alloc_page(struct domain *d)
     {
         struct page_info *pg;
         
         spin_lock(&d->arch.paging.lock);
         pg = page_list_remove_head(&d->arch.paging.p2m_freelist);

Note this: Here you _remove_ from freelist, because you want to actually use the page. Then clearly ...

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
         spin_unlock(&d->arch.paging.lock);

         if ( !pg && is_hardware_domain(d) )
         {
               /* Need to allocate more memory from domheap */
               pg = alloc_domheap_page(d, MEMF_no_owner);
               if ( pg == NULL )
               {
                   printk(XENLOG_ERR "Failed to allocate pages.\n");
                   return pg;
               }
               ACCESS_ONCE(d->arch.paging.total_pages)++;
               page_list_add_tail(pg, &d->arch.paging.freelist);
         }
      
         return pg;
}

And basically use|d->arch.paging.freelist| for both dom0less and dom0 domains,
with the only difference being that in the case of Dom0,|d->arch.paging.freelist |could be extended.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

Do I understand your idea correctly?

Broadly yes, but not in the details. For example, I don't think such a page allocated from the general heap would want appending to freelist. Commentary and alike also would want tidying.

Could you please explain why it wouldn't want appending to freelist?

... adding to freelist here is wrong: You want to use this separately allocated page, too. Else once it is freed it'll be added to freelist a 2nd time, leading to a corrupt list.

Got it, I understand why it shouldn’t be added to the freelist.

Incrementing total_pages still makes sense, right?

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

And of course going forward, for split hardware and control domains the latter may want similar treatment.

Could you please clarify what is the difference between hardware and control domains?
I thought that it is the same or is it for the case when we have dom0 (control domain) which runs domD (hardware domain) and guest domain?

Quoted 5 lines

That's the common case, yes, but conceptually the two can be separate. And if you've followed recent discussions on the list you would also have noticed that work is being done in that direction. (But this was really a forward-looking comment; I didn't mean to make you cover that case right away. Just wanted you to be aware.)

Thanks.

~ Oleksii

#132 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 34 lines · Oleksii Kurochko
Quoted 33 lines · Jan Beulich
Quoted 32 lines · Oleksii Kurochko
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Oleksii Kurochko
Quoted 19 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
        return p2m_type_radix_get(p2m, pte) != p2m_invalid;
    }
 

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

Quoted 8 lines
Quoted 4 lines

It doesn't matter whether it is an intermediate page table or a leaf PTE pointing to a page — PTE should be valid. Considering that in the current implementation it’s possible for PTE.v = 0 but P2M.v = 1, it is better to check P2M.v instead of PTE.v.

I'm confused by this reply. If you want to name 2nd level page table entries P2M - fine (but unhelpful). But then for any memory access there's only one of the two involved: A PTE (Xen accesses) or a P2M (guest accesses). Hence how can there be "PTE.v = 0 but P2M.v = 1"?

I think I understand your confusion, let me try to rephrase.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

The reason for having both|p2m_is_valid()| and|pte_is_valid()| is that I want to have the ability to use the P2M PTE valid bit to track which pages were accessed by a vCPU, so that cleaning and invalidating RAM associated with the guest vCPU won't be too expensive, for example.

I don't know what you're talking about here.

https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/mmu/p2m.c#L1649

How does that Arm function matter here? Aiui you don't need anything like that in RISC-V, as there caches don't need disabling temporarily.

Quoted 24 lines
Quoted 17 lines
Quoted 17 lines

In this case, the P2M PTE valid bit will be set to 0, but the P2M PTE type bits will be set to something other than|p2m_invalid| (even for a table entries), so when an MMU fault occurs, we can properly resolve it.

So, if the P2M PTE type (what|p2m_is_valid()| checks) is set to|p2m_invalid|, it means that the valid bit (what|pte_is_valid()| checks) should be set to 0, so the P2M PTE is genuinely invalid.

It could also be the case that the P2M PTE type isn't|p2m_invalid (and P2M PTE valid will be intentionally set to 0 to have ability to track which pages were accessed for the reason I wrote above)|, and when MMU fault occurs we could properly handle it and set to 1 P2M PTE valid bit to 1...

Quoted 2 lines

An intermediate page table entry is something Xen controls entirely. Hence it has no (guest induced) type.

... And actually it is a reason why it is needed to set a type even for an intermediate page table entry.

Quoted 6 lines
Quoted 1 line

I hope now it is a lit bit clearer what and why was done.

Sadly not. I still don't see what use the P2M type in of an intermediate page table is going to be. It surely can't reliably describe all of the entries that page table holds. Intermediate page tables and leaf pages are just too different to share a concept like this, I think. That said, I'll be happy to be shown code demonstrating the contrary.

Quoted 7 lines

Then it is needed to introduce new p2m_type_t - p2m_table and use it. Would it be better?

I still need some type to have ability to distinguish if p2m is valid or not from p2m management and hardware point of view.
If there is no need for such distinguish why all archs introduce p2m_invalid? Isn't enough just to use P2M PTE valid bit?

At least on x86 we don't tag intermediate page tables with P2M types. For ordinary leaf entries the situation is different, as there may be varying reasons why a PTE has its valid (on x86: present) bit cleared. Hence the type is relevant there, just to know what to do when a page is accessed through such a not-present PTE.

Jan

#133 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 35 lines · Jan Beulich
Quoted 34 lines · Oleksii Kurochko
Quoted 33 lines · Jan Beulich
Quoted 32 lines · Oleksii Kurochko
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Oleksii Kurochko
Quoted 19 lines · Jan Beulich
Quoted 6 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +0 -0
@@ -387,6 +387,17 @@ static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte)
         return p2m_type_radix_get(p2m, pte) != p2m_invalid;
     }
 

xen/arch/riscv/p2m.c static inline bool p2me_is_valid(struct p2m_domain *p2m, pte_t pte) · line 399

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+/*
+ * pte_is_* helpers are checking the valid bit set in the
+ * PTE but we have to check p2m_type instead (look at the comment above
+ * p2me_is_valid())
+ * Provide our own overlay to check the valid bit.
+ */
+static inline bool p2me_is_mapping(struct p2m_domain *p2m, pte_t pte)
+{
+    return p2me_is_valid(p2m, pte) && (pte.pte & PTE_ACCESS_MASK);
+}

Same question as on the earlier patch - does P2M type apply to intermediate page tables at all? (Conceptually it shouldn't.)

Quoted 8 lines
Quoted 4 lines

It doesn't matter whether it is an intermediate page table or a leaf PTE pointing to a page — PTE should be valid. Considering that in the current implementation it’s possible for PTE.v = 0 but P2M.v = 1, it is better to check P2M.v instead of PTE.v.

I'm confused by this reply. If you want to name 2nd level page table entries P2M - fine (but unhelpful). But then for any memory access there's only one of the two involved: A PTE (Xen accesses) or a P2M (guest accesses). Hence how can there be "PTE.v = 0 but P2M.v = 1"?

I think I understand your confusion, let me try to rephrase.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

The reason for having both|p2m_is_valid()| and|pte_is_valid()| is that I want to have the ability to use the P2M PTE valid bit to track which pages were accessed by a vCPU, so that cleaning and invalidating RAM associated with the guest vCPU won't be too expensive, for example.

I don't know what you're talking about here.

https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/mmu/p2m.c#L1649

How does that Arm function matter here? Aiui you don't need anything like that in RISC-V, as there caches don't need disabling temporarily.

I thought that it could be needed not only in the case when a d-cache is disabled temporarily, but it seems like that I was just wrong and all other cases are handled by cache coherency protocol.

Quoted 31 lines
Quoted 24 lines
Quoted 18 lines
Quoted 18 lines

In this case, the P2M PTE valid bit will be set to 0, but the P2M PTE type bits will be set to something other than|p2m_invalid| (even for a table entries), so when an MMU fault occurs, we can properly resolve it.

So, if the P2M PTE type (what|p2m_is_valid()| checks) is set to|p2m_invalid|, it means that the valid bit (what|pte_is_valid()| checks) should be set to 0, so the P2M PTE is genuinely invalid.

It could also be the case that the P2M PTE type isn't|p2m_invalid (and P2M PTE valid will be intentionally set to 0 to have ability to track which pages were accessed for the reason I wrote above)|, and when MMU fault occurs we could properly handle it and set to 1 P2M PTE valid bit to 1...

Quoted 2 lines

An intermediate page table entry is something Xen controls entirely. Hence it has no (guest induced) type.

... And actually it is a reason why it is needed to set a type even for an intermediate page table entry.

Quoted 6 lines
Quoted 1 line

I hope now it is a lit bit clearer what and why was done.

Sadly not. I still don't see what use the P2M type in of an intermediate page table is going to be. It surely can't reliably describe all of the entries that page table holds. Intermediate page tables and leaf pages are just too different to share a concept like this, I think. That said, I'll be happy to be shown code demonstrating the contrary.

Quoted 7 lines

Then it is needed to introduce new p2m_type_t - p2m_table and use it. Would it be better?

I still need some type to have ability to distinguish if p2m is valid or not from p2m management and hardware point of view.
If there is no need for such distinguish why all archs introduce p2m_invalid? Isn't enough just to use P2M PTE valid bit?

Quoted 5 lines

At least on x86 we don't tag intermediate page tables with P2M types. For ordinary leaf entries the situation is different, as there may be varying reasons why a PTE has its valid (on x86: present) bit cleared. Hence the type is relevant there, just to know what to do when a page is accessed through such a not-present PTE.

I think that I got your idea now.

Does it make sense to have such optimization when we have 2Mb memory range and it was mapped using 4k pages instead of 1 super-page, could it be useful to invalidate just just page table entry of L1 which corresponds to the start of this 2mb memory range, instead of invalidating each entry on L0? If it could useful then intermediate page tables should be tagged too. Arm has such use cases:
https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/arm/mmu/p2m.c#L1286

~ OLeksii

#134 lore raw

Re: [PATCH v2 14/17] xen/riscv: implement p2m_next_level()

Quoted 42 lines · Oleksii Kurochko
Quoted 33 lines · Jan Beulich
Quoted 25 lines · Oleksii Kurochko
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko

In this case, the P2M PTE valid bit will be set to 0, but the P2M PTE type bits will be set to something other than|p2m_invalid| (even for a table entries), so when an MMU fault occurs, we can properly resolve it.

So, if the P2M PTE type (what|p2m_is_valid()| checks) is set to|p2m_invalid|, it means that the valid bit (what|pte_is_valid()| checks) should be set to 0, so the P2M PTE is genuinely invalid.

It could also be the case that the P2M PTE type isn't|p2m_invalid (and P2M PTE valid will be intentionally set to 0 to have ability to track which pages were accessed for the reason I wrote above)|, and when MMU fault occurs we could properly handle it and set to 1 P2M PTE valid bit to 1...

Quoted 2 lines

An intermediate page table entry is something Xen controls entirely. Hence it has no (guest induced) type.

... And actually it is a reason why it is needed to set a type even for an intermediate page table entry.

Quoted 6 lines
Quoted 1 line

I hope now it is a lit bit clearer what and why was done.

Sadly not. I still don't see what use the P2M type in of an intermediate page table is going to be. It surely can't reliably describe all of the entries that page table holds. Intermediate page tables and leaf pages are just too different to share a concept like this, I think. That said, I'll be happy to be shown code demonstrating the contrary.

Quoted 7 lines

Then it is needed to introduce new p2m_type_t - p2m_table and use it. Would it be better?

I still need some type to have ability to distinguish if p2m is valid or not from p2m management and hardware point of view.
If there is no need for such distinguish why all archs introduce p2m_invalid? Isn't enough just to use P2M PTE valid bit?

Quoted 5 lines

At least on x86 we don't tag intermediate page tables with P2M types. For ordinary leaf entries the situation is different, as there may be varying reasons why a PTE has its valid (on x86: present) bit cleared. Hence the type is relevant there, just to know what to do when a page is accessed through such a not-present PTE.

I think that I got your idea now.

Quoted 7 lines

Does it make sense to have such optimization when we have 2Mb memory range and it was mapped using 4k pages instead of 1 super-page, could it be useful to invalidate just just page table entry of L1 which corresponds to the start of this 2mb memory range, instead of invalidating each entry on L0? If it could useful then intermediate page tables should be tagged too. Arm has such use cases:

   https://gitlab.com/xen-project/people/olkur/xen/-/blob/staging/xen/arch/arm/mmu/p2m.c#L1286

I don't currently see how that's related to the topic at hand.

Furthermore range-constrained TLB flushing is never at just an address, i.e. "L1 which corresponds to the start of this 2mb memory range" isn't meaningful here. It's always a range (typically expressed by address and size), and it always needs to be the full range that is invalidated. This can be a solitary low-level flush operation when you know a large page mapping would _not_ be split. When splitting is done in software or when hardware may split behind your back, you always need to invalidate the entire range. Or else, in your example, 4k TLB entries may remain for any but the first page of the 2M super-page. (Whether such a range can still be done in a single invalidation operation is a separate question. But I don't see how maintaining the type at the L1 level would help there.)

Jan

#135 lore raw

PATCH v2 15/17 xen/riscv: Implement superpage splitting for p2m mappings

In reply to Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table. Additionally, the page table walk logic has been adjusted, as ARM uses the opposite walk order compared to RISC-V.

---
Changes in V2:
- New patch. It was a part of a big patch "xen/riscv: implement p2m mapping functionality" which was splitted to smaller.
- Update the commit above the cycle which creates new page table as RISC-V travserse page tables in an opposite to ARM order.
- RISC-V doesn't require BBM so there is no needed for invalidating and TLB flushing before updating PTE.

---
 xen/arch/riscv/p2m.c | 102 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 101 insertions(+), 1 deletion(-)
xen/arch/riscv/p2m.c +101 -1
index 87dd636b80..79c4473f1f 100644
@@ -743,6 +743,77 @@ static void p2m_free_entry(struct p2m_domain *p2m,
     p2m_free_page(p2m->domain, pg);
 }
 
+static bool p2m_split_superpage(struct p2m_domain *p2m, pte_t *entry,
+                                unsigned int level, unsigned int target,
+                                const unsigned int *offsets)
+{
+    struct page_info *page;
+    unsigned int i;
+    pte_t pte, *table;
+    bool rv = true;
+
+    /* Convenience aliases */
+    mfn_t mfn = pte_get_mfn(*entry);
+    unsigned int next_level = level - 1;
+    unsigned int level_order = XEN_PT_LEVEL_ORDER(next_level);
+
+    /*
+     * This should only be called with target != level and the entry is
+     * a superpage.
+     */
+    ASSERT(level > target);
+    ASSERT(p2me_is_superpage(p2m, *entry, level));
+
+    page = p2m_alloc_page(p2m->domain);
+    if ( !page )
+        return false;
+
+    page_list_add(page, &p2m->pages);
+    table = __map_domain_page(page);
+
+    /*
+     * We are either splitting a second level 1G page into 512 first level
+     * 2M pages, or a first level 2M page into 512 zero level 4K pages.
+     */
+    for ( i = 0; i < XEN_PT_ENTRIES; i++ )
+    {
+        pte_t *new_entry = table + i;
+
+        /*
+         * Use the content of the superpage entry and override
+         * the necessary fields. So the correct permission are kept.
+         */
+        pte = *entry;
+        pte_set_mfn(&pte, mfn_add(mfn, i << level_order));
+
+        write_pte(new_entry, pte);
+    }
+
+    /*
+     * Shatter superpage in the page to the level we want to make the
+     * changes.
+     * This is done outside the loop to avoid checking the offset to
+     * know whether the entry should be shattered for every entry.
+     */
+    if ( next_level != target )
+        rv = p2m_split_superpage(p2m, table + offsets[next_level],
+                                 level - 1, target, offsets);
+
+    /* TODO: why it is necessary to have clean here? Not somewhere in the caller */
+    if ( p2m->clean_pte )
+        clean_dcache_va_range(table, PAGE_SIZE);
+
+    unmap_domain_page(table);
+
+    /*
+     * Even if we failed, we should install the newly allocated PTE
+     * entry. The caller will be in charge to free the sub-tree.
+     */
+    p2m_write_pte(entry, page_to_p2m_table(p2m, page), p2m->clean_pte);
+
+    return rv;
+}
+
 /*
  * Insert an entry in the p2m. This should be called with a mapping
  * equal to a page/superpage.
@@ -806,7 +877,36 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
      */
     if ( level > target )
     {
-        panic("Shattering isn't implemented\n");
+        /* We need to split the original page. */
+        pte_t split_pte = *entry;
+
+        ASSERT(p2me_is_superpage(p2m, *entry, level));
+
+        if ( !p2m_split_superpage(p2m, &split_pte, level, target, offsets) )
+        {
+            /* Free the allocated sub-tree */
+            p2m_free_entry(p2m, split_pte, level);
+
+            rc = -ENOMEM;
+            goto out;
+        }
+
+        p2m_write_pte(entry, split_pte, p2m->clean_pte);
+
+        /* Then move to the level we want to make real changes */
+        for ( ; level < target; level++ )
+        {
+            rc = p2m_next_level(p2m, true, level, &table, offsets[level]);
+
+            /*
+             * The entry should be found and either be a table
+             * or a superpage if level 0 is not targeted
+             */
+            ASSERT(rc == GUEST_TABLE_NORMAL ||
+                   (rc == GUEST_TABLE_SUPER_PAGE && target > 0));
+        }
+
+        entry = table + offsets[level];
     }
 
     /*
Signature · 1 line
2.49.0
#136 lore raw

Re: [PATCH v2 15/17] xen/riscv: Implement superpage splitting for p2m mappings

Quoted 11 lines · Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

Quoted 4 lines

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table.

But some flushing is going to be necessary. As long as you only ever do global flushes, the one after the individual PTE modification (within the split table) will do (if BBM isn't required, see below), but once you move to more fine-grained flushing, that's not going to be enough anymore. Not sure it's a good idea to leave such a pitfall.

As to (no need for) BBM: I couldn't find anything to that effect in the privileged spec. Can you provide some pointer? What I found instead is e.g. this sentence: "To ensure that implicit reads observe writes to the same memory locations, an SFENCE.VMA instruction must be executed after the writes to flush the relevant cached translations." And this: "Accessing the same location using different cacheability attributes may cause loss of coherence." (This may not only occur when the same physical address is mapped twice at different VAs, but also after the shattering of a superpage when the new entry differs in cacheability.)

Quoted 2 lines

Additionally, the page table walk logic has been adjusted, as ARM uses the opposite walk order compared to RISC-V.

I think you used some similar wording already in an earlier patch. I find this confusing: Walk order is, aiui, the same. It's merely the numbering of levels that is the opposite way round, isn't it?

Quoted 37 lines

---
Changes in V2:
- New patch. It was a part of a big patch "xen/riscv: implement p2m mapping functionality" which was splitted to smaller.
- Update the commit above the cycle which creates new page table as RISC-V travserse page tables in an opposite to ARM order.
- RISC-V doesn't require BBM so there is no needed for invalidating and TLB flushing before updating PTE.

---
 xen/arch/riscv/p2m.c | 102 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 101 insertions(+), 1 deletion(-)
xen/arch/riscv/p2m.c +16 -0
index 87dd636b80..79c4473f1f 100644
@@ -743,6 +743,77 @@ static void p2m_free_entry(struct p2m_domain *p2m,
     p2m_free_page(p2m->domain, pg);
 }
 
+static bool p2m_split_superpage(struct p2m_domain *p2m, pte_t *entry,
+                                unsigned int level, unsigned int target,
+                                const unsigned int *offsets)
+{
+    struct page_info *page;
+    unsigned int i;
+    pte_t pte, *table;
+    bool rv = true;
+
+    /* Convenience aliases */
+    mfn_t mfn = pte_get_mfn(*entry);
+    unsigned int next_level = level - 1;
+    unsigned int level_order = XEN_PT_LEVEL_ORDER(next_level);
+
+    /*
+     * This should only be called with target != level and the entry is

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 771

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+     * a superpage.
+     */
+    ASSERT(level > target);
+    ASSERT(p2me_is_superpage(p2m, *entry, level));
+
+    page = p2m_alloc_page(p2m->domain);
+    if ( !page )
+        return false;
+
+    page_list_add(page, &p2m->pages);

Is there a reason this list maintenance isn't done in p2m_alloc_page()?

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    table = __map_domain_page(page);
+
+    /*
+     * We are either splitting a second level 1G page into 512 first level
+     * 2M pages, or a first level 2M page into 512 zero level 4K pages.
+     */

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 787

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+    for ( i = 0; i < XEN_PT_ENTRIES; i++ )
+    {
+        pte_t *new_entry = table + i;
+
+        /*
+         * Use the content of the superpage entry and override
+         * the necessary fields. So the correct permission are kept.
+         */
+        pte = *entry;
+        pte_set_mfn(&pte, mfn_add(mfn, i << level_order));

While okay as long as you only permit superpages up to 1G, this is another trap for someone to fall into: Imo i would better be unsigned long right away, considering that RISC-V permits large pages at all levels.

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 800

Quoted 12 lines
xen/arch/riscv/p2m.c +12 -0
+        write_pte(new_entry, pte);
+    }
+
+    /*
+     * Shatter superpage in the page to the level we want to make the
+     * changes.
+     * This is done outside the loop to avoid checking the offset to
+     * know whether the entry should be shattered for every entry.
+     */
+    if ( next_level != target )
+        rv = p2m_split_superpage(p2m, table + offsets[next_level],
+                                 level - 1, target, offsets);

I don't understand the comment: Under what conditions would every entry need (further) shattering? And where's that happening? Or is this merely a word ordering issue in the sentence, and "for every entry" wants moving ahead? (In that case I'm unconvinced this is in need of commenting upon.)

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 806

Quoted 5 lines
xen/arch/riscv/p2m.c +5 -0
+    /* TODO: why it is necessary to have clean here? Not somewhere in the caller */
+    if ( p2m->clean_pte )
+        clean_dcache_va_range(table, PAGE_SIZE);
+
+    unmap_domain_page(table);

Again likely not something that wants taking care of right away, but there again is an inefficiency here: The caller almost certainly wants to map the same page again, to update the one entry that caused the request to shatter the page.

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 812

Quoted 5 lines
xen/arch/riscv/p2m.c +5 -0
+    /*
+     * Even if we failed, we should install the newly allocated PTE
+     * entry. The caller will be in charge to free the sub-tree.
+     */
+    p2m_write_pte(entry, page_to_p2m_table(p2m, page), p2m->clean_pte);

Why would it be wrong to free the page right here, vacating the entry at the same time (or leaving just that to the caller)? (IOW - if this is an implementation decision of yours, I think the word "should" would want dropping.) After all, the caller invoking p2m_free_entry() on the thus split PTE is less efficient (needs to iterate over all entries) than on the original one (where it's just a single superpage).

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 878

Quoted 3 lines
xen/arch/riscv/p2m.c +0 -0
@@ -806,7 +877,36 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
      */
     if ( level > target )

This condition is likely too strong, unless you actually mean to also split a superpage if it really wouldn't need splitting (new entry written still fitting with the superpage mapping, i.e. suitable MFN and same attributes).

Quoted 10 lines
xen/arch/riscv/p2m.c +8 -1
     {
-        panic("Shattering isn't implemented\n");
+        /* We need to split the original page. */
+        pte_t split_pte = *entry;
+
+        ASSERT(p2me_is_superpage(p2m, *entry, level));
+
+        if ( !p2m_split_superpage(p2m, &split_pte, level, target, offsets) )
+        {
+            /* Free the allocated sub-tree */

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 897

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+            p2m_free_entry(p2m, split_pte, level);
+
+            rc = -ENOMEM;
+            goto out;
+        }
+
+        p2m_write_pte(entry, split_pte, p2m->clean_pte);
+
+        /* Then move to the level we want to make real changes */
+        for ( ; level < target; level++ )

Don't you mean to move downwards here? At which point I wonder: Did you test this code?

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 907

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+        {
+            rc = p2m_next_level(p2m, true, level, &table, offsets[level]);
+
+            /*
+             * The entry should be found and either be a table
+             * or a superpage if level 0 is not targeted
+             */
+            ASSERT(rc == GUEST_TABLE_NORMAL ||
+                   (rc == GUEST_TABLE_SUPER_PAGE && target > 0));
+        }

This, too, is inefficient (but likely good enough as a starting point): You walk tables twice - first when splitting, and then again when finding the target level.

Considering the enclosing if(), this also again is a do/while() candidate.

Jan

#137 lore raw

Re: [PATCH v2 15/17] xen/riscv: Implement superpage splitting for p2m mappings

Quoted 16 lines · Jan Beulich
Quoted 11 lines · Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

Quoted 4 lines

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table.

Quoted 5 lines

But some flushing is going to be necessary. As long as you only ever do global flushes, the one after the individual PTE modification (within the split table) will do (if BBM isn't required, see below), but once you move to more fine-grained flushing, that's not going to be enough anymore. Not sure it's a good idea to leave such a pitfall.

I think that I don't fully understand what is an issue.

Quoted 10 lines

As to (no need for) BBM: I couldn't find anything to that effect in the privileged spec. Can you provide some pointer? What I found instead is e.g. this sentence: "To ensure that implicit reads observe writes to the same memory locations, an SFENCE.VMA instruction must be executed after the writes to flush the relevant cached translations." And this: "Accessing the same location using different cacheability attributes may cause loss of coherence." (This may not only occur when the same physical address is mapped twice at different VAs, but also after the shattering of a superpage when the new entry differs in cacheability.)

I also couldn't find that RISC-V spec mandates BBM explicitly as Arm does it.

What I meant that on RISC-V can do:
- Write new PTE
- Flush TLB

While on Arm it is almost always needed to do:
- Write zero to PTE
- Flush TLB
- Write new PTE

For example, the common CoW code path where you copy from a read only page to a new page, then map that new page as writable just works on RISC-V without extra considerations and on Arm it requires BBM.

It seems to me that BBM is mandated for Arm only because that TLB is shared among cores, so there is no any guarantee that no prior entry for same VA remains in TLB. In case of RISC-V's TLB isn't shared and after a flush it is guaranteed that no prior entry for the same VA remains in the TLB.

But in the same time it could be cases, I guess, where BBM will be needed for RISC-V too. Even the case of CoW mentioned above will need some kind of BBM, but nothing that'll the CPU misbehave by doing CoW without BBM on RISC-V.

Quoted 6 lines
Quoted 3 lines

Additionally, the page table walk logic has been adjusted, as ARM uses the opposite walk order compared to RISC-V.

I think you used some similar wording already in an earlier patch. I find this confusing: Walk order is, aiui, the same. It's merely the numbering of levels that is the opposite way round, isn't it?

Yes, the numbering of levels is different and I counted that as a different walk order. If it is too confusing, I will reword it and use numbering of levels.

Quoted 41 lines
Quoted 41 lines

---
Changes in V2:
- New patch. It was a part of a big patch "xen/riscv: implement p2m mapping functionality" which was splitted to smaller.
- Update the commit above the cycle which creates new page table as RISC-V travserse page tables in an opposite to ARM order.
- RISC-V doesn't require BBM so there is no needed for invalidating and TLB flushing before updating PTE.

---
  xen/arch/riscv/p2m.c | 102 ++++++++++++++++++++++++++++++++++++++++++-
  1 file changed, 101 insertions(+), 1 deletion(-)
xen/arch/riscv/p2m.c +19 -0
index 87dd636b80..79c4473f1f 100644
@@ -743,6 +743,77 @@ static void p2m_free_entry(struct p2m_domain *p2m,
      p2m_free_page(p2m->domain, pg);
  }
 
+static bool p2m_split_superpage(struct p2m_domain *p2m, pte_t *entry,
+                                unsigned int level, unsigned int target,
+                                const unsigned int *offsets)
+{
+    struct page_info *page;
+    unsigned int i;
+    pte_t pte, *table;
+    bool rv = true;
+
+    /* Convenience aliases */
+    mfn_t mfn = pte_get_mfn(*entry);
+    unsigned int next_level = level - 1;
+    unsigned int level_order = XEN_PT_LEVEL_ORDER(next_level);
+
+    /*
+     * This should only be called with target != level and the entry is
+     * a superpage.
+     */
+    ASSERT(level > target);
Quoted 8 lines

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 771

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    ASSERT(p2me_is_superpage(p2m, *entry, level));
+
+    page = p2m_alloc_page(p2m->domain);
+    if ( !page )
+        return false;
+
+    page_list_add(page, &p2m->pages);

Is there a reason this list maintenance isn't done in p2m_alloc_page()?

No there is no any reason, I will move that inside p2m_alloc_page().

Quoted 33 lines
Quoted 7 lines
xen/arch/riscv/p2m.c +6 -0
+    table = __map_domain_page(page);
+
+    /*
+     * We are either splitting a second level 1G page into 512 first level
+     * 2M pages, or a first level 2M page into 512 zero level 4K pages.
+     */

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 787

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+    for ( i = 0; i < XEN_PT_ENTRIES; i++ )
+    {
+        pte_t *new_entry = table + i;
+
+        /*
+         * Use the content of the superpage entry and override
+         * the necessary fields. So the correct permission are kept.
+         */
+        pte = *entry;
+        pte_set_mfn(&pte, mfn_add(mfn, i << level_order));

While okay as long as you only permit superpages up to 1G, this is another trap for someone to fall into: Imo i would better be unsigned long right away, considering that RISC-V permits large pages at all levels.

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 800

Quoted 12 lines
xen/arch/riscv/p2m.c +12 -0
+        write_pte(new_entry, pte);
+    }
+
+    /*
+     * Shatter superpage in the page to the level we want to make the
+     * changes.
+     * This is done outside the loop to avoid checking the offset to
+     * know whether the entry should be shattered for every entry.
+     */
+    if ( next_level != target )
+        rv = p2m_split_superpage(p2m, table + offsets[next_level],
+                                 level - 1, target, offsets);
Quoted 5 lines

I don't understand the comment: Under what conditions would every entry need (further) shattering? And where's that happening? Or is this merely a word ordering issue in the sentence, and "for every entry" wants moving ahead? (In that case I'm unconvinced this is in need of commenting upon.)

It is wording question. It should be something like:
+ /*

+     * Shatter superpage in the page to the level we want to make the
+     * changes.
+     * This is done outside the loop to avoid checking the offset for every
+     * entry (of new page table) to know whether the entry should be shattered.
+     */
Quoted 10 lines

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 806

Quoted 6 lines
xen/arch/riscv/p2m.c +5 -0
+    /* TODO: why it is necessary to have clean here? Not somewhere in the caller */
+    if ( p2m->clean_pte )
+        clean_dcache_va_range(table, PAGE_SIZE);
+
+    unmap_domain_page(table);

Again likely not something that wants taking care of right away, but there again is an inefficiency here: The caller almost certainly wants to map the same page again, to update the one entry that caused the request to shatter the page.

I'll add TODO.

Quoted 6 lines

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 812

Quoted 6 lines
xen/arch/riscv/p2m.c +5 -0
+    /*
+     * Even if we failed, we should install the newly allocated PTE
+     * entry. The caller will be in charge to free the sub-tree.
+     */
+    p2m_write_pte(entry, page_to_p2m_table(p2m, page), p2m->clean_pte);
Quoted 6 lines

Why would it be wrong to free the page right here, vacating the entry at the same time (or leaving just that to the caller)? (IOW - if this is an implementation decision of yours, I think the word "should" would want dropping.) After all, the caller invoking p2m_free_entry() on the thus split PTE is less efficient (needs to iterate over all entries) than on the original one (where it's just a single superpage).

I think that I didn't get your idea.

Quoted 8 lines

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 878

Quoted 4 lines
xen/arch/riscv/p2m.c +0 -0
@@ -806,7 +877,36 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
       */
      if ( level > target )

This condition is likely too strong, unless you actually mean to also split a superpage if it really wouldn't need splitting (new entry written still fitting with the superpage mapping, i.e. suitable MFN and same attributes).

I am not really sure that I fully understand.
My understanding is if level != target then the splitting is needed, I am not really get the part "split a superpage if it really wouldn't need splitting".

Quoted 15 lines
Quoted 15 lines
xen/arch/riscv/p2m.c +12 -1
      {
-        panic("Shattering isn't implemented\n");
+        /* We need to split the original page. */
+        pte_t split_pte = *entry;
+
+        ASSERT(p2me_is_superpage(p2m, *entry, level));
+
+        if ( !p2m_split_superpage(p2m, &split_pte, level, target, offsets) )
+        {
+            /* Free the allocated sub-tree */
+            p2m_free_entry(p2m, split_pte, level);
+
+            rc = -ENOMEM;
+            goto out;
Quoted 8 lines

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 897

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+        }
+
+        p2m_write_pte(entry, split_pte, p2m->clean_pte);
+
+        /* Then move to the level we want to make real changes */
+        for ( ; level < target; level++ )

Don't you mean to move downwards here? At which point I wonder: Did you test this code?

No as the test for this case was missed. I will add one.

Quoted 11 lines

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 907

Quoted 11 lines
xen/arch/riscv/p2m.c +10 -0
+        {
+            rc = p2m_next_level(p2m, true, level, &table, offsets[level]);
+
+            /*
+             * The entry should be found and either be a table
+             * or a superpage if level 0 is not targeted
+             */
+            ASSERT(rc == GUEST_TABLE_NORMAL ||
+                   (rc == GUEST_TABLE_SUPER_PAGE && target > 0));
+        }
Quoted 4 lines

This, too, is inefficient (but likely good enough as a starting point): You walk tables twice - first when splitting, and then again when finding the target level.

Considering the enclosing if(), this also again is a do/while() candidate.

I will add TODO to make that part more efficient. And based on your reply regarding statement inside if(), I'll likely to use do/while().

Thanks.

~ Oleksii

#138 lore raw

Re: [PATCH v2 15/17] xen/riscv: Implement superpage splitting for p2m mappings

Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table.

Quoted 7 lines
Quoted 5 lines

But some flushing is going to be necessary. As long as you only ever do global flushes, the one after the individual PTE modification (within the split table) will do (if BBM isn't required, see below), but once you move to more fine-grained flushing, that's not going to be enough anymore. Not sure it's a good idea to leave such a pitfall.

I think that I don't fully understand what is an issue.

Whether a flush is necessary after solely breaking up a superpage is arch- defined. I don't know how much restrictions the spec on possible hardware behavior for RISC-V. However, the eventual change of (at least) one entry of fulfill the original request will surely require a flush. What I was trying to say is that this required flush would better not also cover for the flushes that may or may not be required by the spec. IOW if the spec leaves any room for flushes to possibly be needed, those flushes would better be explicit.

Quoted 16 lines
Quoted 9 lines

As to (no need for) BBM: I couldn't find anything to that effect in the privileged spec. Can you provide some pointer? What I found instead is e.g. this sentence: "To ensure that implicit reads observe writes to the same memory locations, an SFENCE.VMA instruction must be executed after the writes to flush the relevant cached translations." And this: "Accessing the same location using different cacheability attributes may cause loss of coherence." (This may not only occur when the same physical address is mapped twice at different VAs, but also after the shattering of a superpage when the new entry differs in cacheability.)

I also couldn't find that RISC-V spec mandates BBM explicitly as Arm does it.

What I meant that on RISC-V can do:
- Write new PTE
- Flush TLB

Quoted 8 lines

While on Arm it is almost always needed to do:
- Write zero to PTE
- Flush TLB
- Write new PTE

For example, the common CoW code path where you copy from a read only page to a new page, then map that new page as writable just works on RISC-V without extra considerations and on Arm it requires BBM.

CoW is a specific sub-case with increasing privilege.

Quoted 4 lines

It seems to me that BBM is mandated for Arm only because that TLB is shared among cores, so there is no any guarantee that no prior entry for same VA remains in TLB. In case of RISC-V's TLB isn't shared and after a flush it is guaranteed that no prior entry for the same VA remains in the TLB.

Not sure that's the sole reason. But again the question is: Is this written down explicitly anywhere? Generally there can be multiple levels of TLBs, and while some of them may be per-core, others may be shared.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
xen/arch/riscv/p2m.c +5 -0
+    /*
+     * Even if we failed, we should install the newly allocated PTE
+     * entry. The caller will be in charge to free the sub-tree.
+     */
+    p2m_write_pte(entry, page_to_p2m_table(p2m, page), p2m->clean_pte);
Quoted 8 lines
Quoted 6 lines

Why would it be wrong to free the page right here, vacating the entry at the same time (or leaving just that to the caller)? (IOW - if this is an implementation decision of yours, I think the word "should" would want dropping.) After all, the caller invoking p2m_free_entry() on the thus split PTE is less efficient (needs to iterate over all entries) than on the original one (where it's just a single superpage).

I think that I didn't get your idea.

Well, first and foremost this was a question to you, as it's not clear to me whether "should" is the correct word to use here. It would be appropriate if the spec mandated this behavior. It would seem less appropriate if this arrangement was an implementation choice of yours. And it looks to me as if the latter was the case here.

Quoted 11 lines
Quoted 7 lines

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 878

Quoted 3 lines
xen/arch/riscv/p2m.c +0 -0
@@ -806,7 +877,36 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
       */
      if ( level > target )

This condition is likely too strong, unless you actually mean to also split a superpage if it really wouldn't need splitting (new entry written still fitting with the superpage mapping, i.e. suitable MFN and same attributes).

I am not really sure that I fully understand.
My understanding is if level != target then the splitting is needed, I am not really get the part "split a superpage if it really wouldn't need splitting".

Hmm, maybe I was wrong here. The caller determines at what level the actual change needs to occur? In which case what you have may be right.

Jan

#139 lore raw

Re: [PATCH v2 15/17] xen/riscv: Implement superpage splitting for p2m mappings

Quoted 24 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table.

Quoted 6 lines
Quoted 5 lines

But some flushing is going to be necessary. As long as you only ever do global flushes, the one after the individual PTE modification (within the split table) will do (if BBM isn't required, see below), but once you move to more fine-grained flushing, that's not going to be enough anymore. Not sure it's a good idea to leave such a pitfall.

I think that I don't fully understand what is an issue.

Quoted 8 lines

Whether a flush is necessary after solely breaking up a superpage is arch- defined. I don't know how much restrictions the spec on possible hardware behavior for RISC-V. However, the eventual change of (at least) one entry of fulfill the original request will surely require a flush. What I was trying to say is that this required flush would better not also cover for the flushes that may or may not be required by the spec. IOW if the spec leaves any room for flushes to possibly be needed, those flushes would better be explicit.

I think that I still don't understand why what I wrote above will work as long as global flushes is working and will stop to work when more fine-grained flushing is used.

Inside p2m_split_superpage() we don't need any kind of TLB flush operation as it is allocation a new page for a table and works with it, so no one could use this page at the moment and cache it in TLB.

The only question is that if it is needed BBM before staring using splitted entry:

        ....
        if ( !p2m_split_superpage(p2m, &split_pte, level, target, offsets) )
        {
            /* Free the allocated sub-tree */
            p2m_free_subtree(p2m, split_pte, level);

            rc = -ENOMEM;
            goto out;
        }

------> /* Should be BBM used here ? */

        p2m_write_pte(entry, split_pte, p2m->clean_pte);

And I can't find anything in the spec what tells me to use BBM here like Arm does:

        /*
         * Follow the break-before-sequence to update the entry.
         * For more details see (D4.7.1 in ARM DDI 0487A.j).
         */
        p2m_remove_pte(entry, p2m->clean_pte);
        p2m_force_tlb_flush_sync(p2m);

        p2m_write_pte(entry, split_pte, p2m->clean_pte);

I agree that even BBM isn't mandated explicitly sometime it could be useful, but here I am not really sure what is the point to do TLB flush before p2m_write_pte() as nothing serious will happen if and old mapping will be used for a some time as we are keeping the same rights for smaller (splited) mapping.
The reason why Arm does p2m_remove_pte() & p2m_force_tlb_flush() before updating an entry here as it is doesn't guarantee that everything will be okay and they explicitly tells:
This situation can possibly break coherency and ordering guarantees, leading to inconsistent unwanted behavior in our Processing Element (PE).

Quoted 20 lines
Quoted 20 lines
Quoted 9 lines

As to (no need for) BBM: I couldn't find anything to that effect in the privileged spec. Can you provide some pointer? What I found instead is e.g. this sentence: "To ensure that implicit reads observe writes to the same memory locations, an SFENCE.VMA instruction must be executed after the writes to flush the relevant cached translations." And this: "Accessing the same location using different cacheability attributes may cause loss of coherence." (This may not only occur when the same physical address is mapped twice at different VAs, but also after the shattering of a superpage when the new entry differs in cacheability.)

I also couldn't find that RISC-V spec mandates BBM explicitly as Arm does it.

What I meant that on RISC-V can do:
- Write new PTE
- Flush TLB

While on Arm it is almost always needed to do:
- Write zero to PTE
- Flush TLB
- Write new PTE

Quoted 4 lines
Quoted 3 lines

For example, the common CoW code path where you copy from a read only page to a new page, then map that new page as writable just works on RISC-V without extra considerations and on Arm it requires BBM.

CoW is a specific sub-case with increasing privilege.

Could you please explain why it matters increasing of privilege in terms of TLB flushing and PTE clearing before writing a new PTE?

Quoted 8 lines
Quoted 5 lines

It seems to me that BBM is mandated for Arm only because that TLB is shared among cores, so there is no any guarantee that no prior entry for same VA remains in TLB. In case of RISC-V's TLB isn't shared and after a flush it is guaranteed that no prior entry for the same VA remains in the TLB.

Not sure that's the sole reason. But again the question is: Is this written down explicitly anywhere? Generally there can be multiple levels of TLBs, and while some of them may be per-core, others may be shared.

Spec. mentions that:
If a hart employs an address-translation cache, that cache must appear to be private to that hart.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

xen/arch/riscv/p2m.c static void p2m_free_entry(struct p2m_domain *p2m, · line 812

Quoted 6 lines
xen/arch/riscv/p2m.c +5 -0
+    /*
+     * Even if we failed, we should install the newly allocated PTE
+     * entry. The caller will be in charge to free the sub-tree.
+     */
+    p2m_write_pte(entry, page_to_p2m_table(p2m, page), p2m->clean_pte);

Why would it be wrong to free the page right here, vacating the entry at the same time (or leaving just that to the caller)? (IOW - if this is an implementation decision of yours, I think the word "should" would want dropping.) After all, the caller invoking p2m_free_entry() on the thus split PTE is less efficient (needs to iterate over all entries) than on the original one (where it's just a single superpage).

Quoted 6 lines
Quoted 1 line

I think that I didn't get your idea.

Well, first and foremost this was a question to you, as it's not clear to me whether "should" is the correct word to use here. It would be appropriate if the spec mandated this behavior. It would seem less appropriate if this arrangement was an implementation choice of yours. And it looks to me as if the latter was the case here.

No, the spec doesn't mandate such behavior, it is just implementation specific. I can mention that in the comment.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

xen/arch/riscv/p2m.c static int __p2m_set_entry(struct p2m_domain *p2m, · line 878

Quoted 4 lines
xen/arch/riscv/p2m.c +0 -0
@@ -806,7 +877,36 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
        */
       if ( level > target )

This condition is likely too strong, unless you actually mean to also split a superpage if it really wouldn't need splitting (new entry written still fitting with the superpage mapping, i.e. suitable MFN and same attributes).

Quoted 5 lines
Quoted 3 lines

I am not really sure that I fully understand.
My understanding is if level != target then the splitting is needed, I am not really get the part "split a superpage if it really wouldn't need splitting".

Hmm, maybe I was wrong here. The caller determines at what level the actual change needs to occur? In which case what you have may be right.

Yes, the caller determines expected level and then asks __p2m_set_entry() to map on this level.

~ Oleksii

#140 lore raw

Re: [PATCH v2 15/17] xen/riscv: Implement superpage splitting for p2m mappings

Quoted 54 lines · Oleksii Kurochko
Quoted 24 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table.

Quoted 6 lines
Quoted 5 lines

But some flushing is going to be necessary. As long as you only ever do global flushes, the one after the individual PTE modification (within the split table) will do (if BBM isn't required, see below), but once you move to more fine-grained flushing, that's not going to be enough anymore. Not sure it's a good idea to leave such a pitfall.

I think that I don't fully understand what is an issue.

Quoted 8 lines

Whether a flush is necessary after solely breaking up a superpage is arch- defined. I don't know how much restrictions the spec on possible hardware behavior for RISC-V. However, the eventual change of (at least) one entry of fulfill the original request will surely require a flush. What I was trying to say is that this required flush would better not also cover for the flushes that may or may not be required by the spec. IOW if the spec leaves any room for flushes to possibly be needed, those flushes would better be explicit.

I think that I still don't understand why what I wrote above will work as long as global flushes is working and will stop to work when more fine-grained flushing is used.

Inside p2m_split_superpage() we don't need any kind of TLB flush operation as it is allocation a new page for a table and works with it, so no one could use this page at the moment and cache it in TLB.

The only question is that if it is needed BBM before staring using splitted entry:

         ....
         if ( !p2m_split_superpage(p2m, &split_pte, level, target, offsets) )
         {
             /* Free the allocated sub-tree */
             p2m_free_subtree(p2m, split_pte, level);

             rc = -ENOMEM;
             goto out;
         }
Quoted 5 lines

------> /* Should be BBM used here ? */

         p2m_write_pte(entry, split_pte, p2m->clean_pte);

And I can't find anything in the spec what tells me to use BBM here like Arm does:

But what you need is a statement in the spec that you can get away without. Imo at least.

Quoted 40 lines
         /*
          * Follow the break-before-sequence to update the entry.
          * For more details see (D4.7.1 in ARM DDI 0487A.j).
          */
         p2m_remove_pte(entry, p2m->clean_pte);
         p2m_force_tlb_flush_sync(p2m);

         p2m_write_pte(entry, split_pte, p2m->clean_pte);

I agree that even BBM isn't mandated explicitly sometime it could be useful, but here I am not really sure what is the point to do TLB flush before p2m_write_pte() as nothing serious will happen if and old mapping will be used for a some time as we are keeping the same rights for smaller (splited) mapping.
The reason why Arm does p2m_remove_pte() & p2m_force_tlb_flush() before updating an entry here as it is doesn't guarantee that everything will be okay and they explicitly tells:
This situation can possibly break coherency and ordering guarantees, leading to inconsistent unwanted behavior in our Processing Element (PE).

Quoted 20 lines
Quoted 20 lines
Quoted 9 lines

As to (no need for) BBM: I couldn't find anything to that effect in the privileged spec. Can you provide some pointer? What I found instead is e.g. this sentence: "To ensure that implicit reads observe writes to the same memory locations, an SFENCE.VMA instruction must be executed after the writes to flush the relevant cached translations." And this: "Accessing the same location using different cacheability attributes may cause loss of coherence." (This may not only occur when the same physical address is mapped twice at different VAs, but also after the shattering of a superpage when the new entry differs in cacheability.)

I also couldn't find that RISC-V spec mandates BBM explicitly as Arm does it.

What I meant that on RISC-V can do:
- Write new PTE
- Flush TLB

While on Arm it is almost always needed to do:
- Write zero to PTE
- Flush TLB
- Write new PTE

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

For example, the common CoW code path where you copy from a read only page to a new page, then map that new page as writable just works on RISC-V without extra considerations and on Arm it requires BBM.

CoW is a specific sub-case with increasing privilege.

Could you please explain why it matters increasing of privilege in terms of TLB flushing and PTE clearing before writing a new PTE?

Some architectures automatically re-walk page tables when encountering a permission violation based on TLB contents. Hence increasing privilege can be a special case.

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

It seems to me that BBM is mandated for Arm only because that TLB is shared among cores, so there is no any guarantee that no prior entry for same VA remains in TLB. In case of RISC-V's TLB isn't shared and after a flush it is guaranteed that no prior entry for the same VA remains in the TLB.

Not sure that's the sole reason. But again the question is: Is this written down explicitly anywhere? Generally there can be multiple levels of TLBs, and while some of them may be per-core, others may be shared.

Spec. mentions that:
If a hart employs an address-translation cache, that cache must appear to be private to that hart.

Hmm, okay, that indeed pretty much excludes shared TLBs.

Jan

#141 lore raw

Re: [PATCH v2 15/17] xen/riscv: Implement superpage splitting for p2m mappings

Quoted 53 lines · Jan Beulich
Quoted 52 lines · Oleksii Kurochko
Quoted 24 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table.

Quoted 6 lines
Quoted 5 lines

But some flushing is going to be necessary. As long as you only ever do global flushes, the one after the individual PTE modification (within the split table) will do (if BBM isn't required, see below), but once you move to more fine-grained flushing, that's not going to be enough anymore. Not sure it's a good idea to leave such a pitfall.

I think that I don't fully understand what is an issue.

Quoted 8 lines

Whether a flush is necessary after solely breaking up a superpage is arch- defined. I don't know how much restrictions the spec on possible hardware behavior for RISC-V. However, the eventual change of (at least) one entry of fulfill the original request will surely require a flush. What I was trying to say is that this required flush would better not also cover for the flushes that may or may not be required by the spec. IOW if the spec leaves any room for flushes to possibly be needed, those flushes would better be explicit.

I think that I still don't understand why what I wrote above will work as long as global flushes is working and will stop to work when more fine-grained flushing is used.

Inside p2m_split_superpage() we don't need any kind of TLB flush operation as it is allocation a new page for a table and works with it, so no one could use this page at the moment and cache it in TLB.

The only question is that if it is needed BBM before staring using splitted entry:

          ....
          if ( !p2m_split_superpage(p2m, &split_pte, level, target, offsets) )
          {
              /* Free the allocated sub-tree */
              p2m_free_subtree(p2m, split_pte, level);

              rc = -ENOMEM;
              goto out;
          }
Quoted 7 lines
Quoted 5 lines

------> /* Should be BBM used here ? */

          p2m_write_pte(entry, split_pte, p2m->clean_pte);

And I can't find anything in the spec what tells me to use BBM here like Arm does:

But what you need is a statement in the spec that you can get away without. Imo at least.

In the spec. it is mentioned that:
It is permitted for multiple address-translation cache entries to co-exist for the same address. This represents the fact that in a conventional TLB hierarchy, it is possible for multiple entries to match a single address if, for example, a page is upgraded to a superpage without first clearing the original non-leaf PTE’s valid bit and executing an SFENCE.VMA with rs1=x0, or if multiple TLBs exist in parallel at a given level of the hierarchy. In this case, just as if an SFENCE.VMA is not executed between a write to the memory-management tables and subsequent implicit read of the same address: it is unpredictable whether the old non-leaf PTE or the new leaf PTE is used, but the behavior is otherwise well defined.
The phrase*"but the behavior is otherwise well defined"* emphasizes that even if the TLB sees two versions (the old and the new), the architecture guarantees stability, and the behavior remains safe — though unpredictable in terms of which translation will be used. And I think that this unpredictability is okay, at least, in the case if superpage splitting and therefore TLB flushing can be deferred since the old pages (which are used for old mapping) still exist and the permissions of the new entries match those of the original ones. Also, it seems like there clearing PTE before TLB flushing isn't need too.

Does it make sense?

#142 lore raw

Re: [PATCH v2 15/17] xen/riscv: Implement superpage splitting for p2m mappings

Quoted 81 lines · Oleksii Kurochko
Quoted 53 lines · Jan Beulich
Quoted 52 lines · Oleksii Kurochko
Quoted 24 lines · Jan Beulich
Quoted 17 lines · Oleksii Kurochko
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko

Add support for down large memory mappings ("superpages") in the RISC-V p2m mapping so that smaller, more precise mappings ("finer-grained entries") can be inserted into lower levels of the page table hierarchy.

To implement that the following is done:
- Introduce p2m_split_superpage(): Recursively shatters a superpage into smaller page table entries down to the target level, preserving original permissions and attributes.
- __p2m_set_entry() updated to invoke superpage splitting when inserting entries at lower levels within a superpage-mapped region.

This implementation is based on the ARM code, with modifications to the part that follows the BBM (break-before-make) approach. Unlike ARM, RISC-V does not require BBM, so there is no need to invalidate the PTE and flush the TLB before updating it with the newly created, split page table.

Quoted 6 lines
Quoted 5 lines

But some flushing is going to be necessary. As long as you only ever do global flushes, the one after the individual PTE modification (within the split table) will do (if BBM isn't required, see below), but once you move to more fine-grained flushing, that's not going to be enough anymore. Not sure it's a good idea to leave such a pitfall.

I think that I don't fully understand what is an issue.

Quoted 8 lines

Whether a flush is necessary after solely breaking up a superpage is arch- defined. I don't know how much restrictions the spec on possible hardware behavior for RISC-V. However, the eventual change of (at least) one entry of fulfill the original request will surely require a flush. What I was trying to say is that this required flush would better not also cover for the flushes that may or may not be required by the spec. IOW if the spec leaves any room for flushes to possibly be needed, those flushes would better be explicit.

I think that I still don't understand why what I wrote above will work as long as global flushes is working and will stop to work when more fine-grained flushing is used.

Inside p2m_split_superpage() we don't need any kind of TLB flush operation as it is allocation a new page for a table and works with it, so no one could use this page at the moment and cache it in TLB.

The only question is that if it is needed BBM before staring using splitted entry:

          ....
          if ( !p2m_split_superpage(p2m, &split_pte, level, target, offsets) )
          {
              /* Free the allocated sub-tree */
              p2m_free_subtree(p2m, split_pte, level);

              rc = -ENOMEM;
              goto out;
          }
Quoted 7 lines
Quoted 5 lines

------> /* Should be BBM used here ? */

          p2m_write_pte(entry, split_pte, p2m->clean_pte);

And I can't find anything in the spec what tells me to use BBM here like Arm does:

But what you need is a statement in the spec that you can get away without. Imo at least.

In the spec. it is mentioned that:
It is permitted for multiple address-translation cache entries to co-exist for the same address. This represents the fact that in a conventional TLB hierarchy, it is possible for multiple entries to match a single address if, for example, a page is upgraded to a superpage without first clearing the original non-leaf PTE’s valid bit and executing an SFENCE.VMA with rs1=x0, or if multiple TLBs exist in parallel at a given level of the hierarchy. In this case, just as if an SFENCE.VMA is not executed between a write to the memory-management tables and subsequent implicit read of the same address: it is unpredictable whether the old non-leaf PTE or the new leaf PTE is used, but the behavior is otherwise well defined.
The phrase*"but the behavior is otherwise well defined"* emphasizes that even if the TLB sees two versions (the old and the new), the architecture guarantees stability, and the behavior remains safe — though unpredictable in terms of which translation will be used. And I think that this unpredictability is okay, at least, in the case if superpage splitting and therefore TLB flushing can be deferred since the old pages (which are used for old mapping) still exist and the permissions of the new entries match those of the original ones. Also, it seems like there clearing PTE before TLB flushing isn't need too.

Quoted 1 line

Does it make sense?

Yes, I think this indeed is sufficient as a spec requirement.

Jan

#143 lore raw

PATCH v2 16/17 xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

In reply to Oleksii Kurochko

Implement the mfn_valid() macro to verify whether a given MFN is valid by checking that it falls within the range [start_page, max_page). These bounds are initialized based on the start and end addresses of RAM.

As part of this patch, start_page is introduced and initialized with the PFN of the first RAM page.

Also, after providing a non-stub implementation of the mfn_valid() macro,
the following compilation errors started to occur:
riscv64-linux-gnu-ld: prelink.o: in function `__next_node':
/build/xen/./include/xen/nodemask.h:202: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `get_free_buddy':
/build/xen/common/page_alloc.c:881: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `alloc_heap_pages':
/build/xen/common/page_alloc.c:1043: undefined reference to `page_get_owner_and_reference'
riscv64-linux-gnu-ld: /build/xen/common/page_alloc.c:1098: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `ns16550_interrupt':
/build/xen/drivers/char/ns16550.c:205: undefined reference to `get_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `page_get_owner_and_reference' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value
make[2]: *** [arch/riscv/Makefile:35: xen-syms] Error 1
To resolve these errors, the following functions have also been introduced,
based on their Arm counterparts:
- page_get_owner_and_reference() and its variant to safely acquire a
reference to a page and retrieve its owner.
- put_page() and put_page_nr() to release page references and free the page
when the count drops to zero.
For put_page_nr(), code related to static memory configuration is wrapped
with CONFIG_STATIC_MEMORY, as this configuration has not yet been moved to
common code. Therefore, PGC_static and free_domstatic_page() are not
introduced for RISC-V. However, since this configuration could be useful
in the future, the relevant code is retained and conditionally compiled.
- A stub for page_is_ram_type() that currently always returns 0 and asserts unreachable, as RAM type checking is not yet implemented.

---
Changes in V2:
- New patch.

---
 xen/arch/riscv/include/asm/mm.h |  9 ++-
 xen/arch/riscv/mm.c             | 97 +++++++++++++++++++++++++++++++--
 2 files changed, 99 insertions(+), 7 deletions(-)
xen/arch/riscv/include/asm/mm.h +7 -2
index 41bf9002d7..bd8511e5f9 100644
@@ -5,6 +5,7 @@
 
 #include <public/xen.h>
 #include <xen/bug.h>
+#include <xen/compiler.h>
 #include <xen/const.h>
 #include <xen/mm-frame.h>
 #include <xen/pdx.h>
@@ -288,8 +289,12 @@ static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
 #define page_get_owner(p)    (p)->v.inuse.domain
 #define page_set_owner(p, d) ((p)->v.inuse.domain = (d))
 
-/* TODO: implement */
-#define mfn_valid(mfn) ({ (void)(mfn); 0; })
+extern unsigned long start_page;
+
+#define mfn_valid(mfn) ({                                   \
+    unsigned long mfn__ = mfn_x(mfn);                       \
+    likely((mfn__ >= start_page) && (mfn__ < max_page));    \
+})
 
 #define domain_set_alloc_bitsize(d) ((void)(d))
 #define domain_clamp_alloc_bitsize(d, b) ((void)(d), (b))
xen/arch/riscv/mm.c +92 -5

118 lines not on this page · raw message

Signature · 1 line
2.49.0
#144 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 26 lines · Oleksii Kurochko

Implement the mfn_valid() macro to verify whether a given MFN is valid by checking that it falls within the range [start_page, max_page). These bounds are initialized based on the start and end addresses of RAM.

As part of this patch, start_page is introduced and initialized with the PFN of the first RAM page.

Also, after providing a non-stub implementation of the mfn_valid() macro,
the following compilation errors started to occur:
riscv64-linux-gnu-ld: prelink.o: in function `__next_node':
/build/xen/./include/xen/nodemask.h:202: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `get_free_buddy':
/build/xen/common/page_alloc.c:881: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `alloc_heap_pages':
/build/xen/common/page_alloc.c:1043: undefined reference to `page_get_owner_and_reference'
riscv64-linux-gnu-ld: /build/xen/common/page_alloc.c:1098: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `ns16550_interrupt':
/build/xen/drivers/char/ns16550.c:205: undefined reference to `get_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `page_get_owner_and_reference' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value
make[2]: *** [arch/riscv/Makefile:35: xen-syms] Error 1
To resolve these errors, the following functions have also been introduced,
based on their Arm counterparts:
- page_get_owner_and_reference() and its variant to safely acquire a
reference to a page and retrieve its owner.
- put_page() and put_page_nr() to release page references and free the page

Quoted 8 lines

when the count drops to zero.
For put_page_nr(), code related to static memory configuration is wrapped with CONFIG_STATIC_MEMORY, as this configuration has not yet been moved to common code. Therefore, PGC_static and free_domstatic_page() are not introduced for RISC-V. However, since this configuration could be useful in the future, the relevant code is retained and conditionally compiled.
- A stub for page_is_ram_type() that currently always returns 0 and asserts unreachable, as RAM type checking is not yet implemented.

How does this end up working when common code references the function?

xen/arch/riscv/include/asm/mm.h static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr) · line 297

Quoted 12 lines
xen/arch/riscv/include/asm/mm.h +6 -2
@@ -288,8 +289,12 @@ static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
 #define page_get_owner(p)    (p)->v.inuse.domain
 #define page_set_owner(p, d) ((p)->v.inuse.domain = (d))
 
-/* TODO: implement */
-#define mfn_valid(mfn) ({ (void)(mfn); 0; })
+extern unsigned long start_page;
+
+#define mfn_valid(mfn) ({                                   \
+    unsigned long mfn__ = mfn_x(mfn);                       \
+    likely((mfn__ >= start_page) && (mfn__ < max_page));    \
+})

I don't think you should try to be clever and avoid using __mfn_valid() here, at least not without an easily identifiable TODO. Surely you've seen that both Arm and x86 use it.

Also, according to all I know, likely() doesn't work very well when used like this, except for architectures supporting conditionally executed insns (like Arm32 or IA-64, i.e. beyond conditional branches). I.e. if you want to use likely() here, I think you need two of them.

xen/arch/riscv/mm.c static void __init setup_directmap_mappings(unsigned long base_mfn, · line 523

Quoted 5 lines
xen/arch/riscv/mm.c +1 -0
@@ -525,6 +520,8 @@ static void __init setup_directmap_mappings(unsigned long base_mfn,
 #error setup_{directmap,frametable}_mapping() should be implemented for RV_32
 #endif
 
+unsigned long __read_mostly start_page;

Memory hotplug question again: __read_mostly or __ro_after_init?

Quoted 14 lines
xen/arch/riscv/mm.c +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
 {
     return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
 }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 6 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 24 lines
xen/arch/riscv/mm.c +24 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}
+
+struct domain *page_get_owner_and_reference(struct page_info *page)
+{
+    return page_get_owner_and_nr_reference(page, 1);
+}
+
+void put_page_nr(struct page_info *page, unsigned long nr)
+{
+    unsigned long nx, x, y = page->count_info;
+
+    do {
+        ASSERT((y & PGC_count_mask) >= nr);
+        x  = y;
+        nx = x - nr;

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 675

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+    }
+    while ( unlikely((y = cmpxchg(&page->count_info, x, nx)) != x) );
+
+    if ( unlikely((nx & PGC_count_mask) == 0) )
+    {
+#ifdef CONFIG_STATIC_MEMORY
+        if ( unlikely(nx & PGC_static) )
+            free_domstatic_page(page);
+        else
+#endif

Such #ifdef-ed-out code is liable to go stale. Minimally use IS_ENABLED(). Even better would imo be if you introduced a "stub" PGC_static, resolving to 0 (i.e. for now unconditionally).

Jan

#145 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 43 lines · Jan Beulich
Quoted 14 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
 {
     return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
 }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 6 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 10 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 651

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}

There also looks to be a dead code concern here (towards the "nr" parameters here and elsewhere, when STATIC_SHM=n). Just that apparently we decided to leave out Misra rule 2.2 entirely.

Jan

#146 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 42 lines · Jan Beulich
Quoted 41 lines · Jan Beulich
Quoted 14 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
  {
      return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
  }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 6 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 13 lines
Quoted 10 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 651

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}

There also looks to be a dead code concern here (towards the "nr" parameters here and elsewhere, when STATIC_SHM=n). Just that apparently we decided to leave out Misra rule 2.2 entirely.

I think that I didn't get what is an issue when STATIC_SHM=n, functions is still going to be called through page_get_owner_and_reference(), at least, in page_alloc.c .

~ Oleksii

#147 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 58 lines · Oleksii Kurochko
Quoted 42 lines · Jan Beulich
Quoted 41 lines · Jan Beulich
Quoted 14 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
  {
      return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
  }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 6 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 13 lines
Quoted 10 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 651

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}

There also looks to be a dead code concern here (towards the "nr" parameters here and elsewhere, when STATIC_SHM=n). Just that apparently we decided to leave out Misra rule 2.2 entirely.

Quoted 2 lines

I think that I didn't get what is an issue when STATIC_SHM=n, functions is still going to be called through page_get_owner_and_reference(), at least, in page_alloc.c .

Yes, but will "nr" ever be anything other than 1 then? IOW omitting the parameter would be fine. And that's what "dead code" is about.

Jan

#148 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 57 lines · Jan Beulich
Quoted 56 lines · Oleksii Kurochko
Quoted 55 lines · Jan Beulich
Quoted 41 lines · Jan Beulich
Quoted 14 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
   {
       return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
   }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 6 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 10 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 651

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}

There also looks to be a dead code concern here (towards the "nr" parameters here and elsewhere, when STATIC_SHM=n). Just that apparently we decided to leave out Misra rule 2.2 entirely.

Quoted 4 lines
Quoted 2 lines

I think that I didn't get what is an issue when STATIC_SHM=n, functions is still going to be called through page_get_owner_and_reference(), at least, in page_alloc.c .

Yes, but will "nr" ever be anything other than 1 then? IOW omitting the parameter would be fine. And that's what "dead code" is about.

Got it.

So we don't have any SAF-x tag to mark this function as safe. What is the best one solution for now if nr argument will be needed in the future for STATIC_SHM=y?

~ Oleksii

#149 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 60 lines · Oleksii Kurochko
Quoted 59 lines · Jan Beulich
Quoted 58 lines · Oleksii Kurochko
Quoted 42 lines · Jan Beulich
Quoted 41 lines · Jan Beulich
Quoted 14 lines · Oleksii Kurochko
xen/arch/riscv/mm.c +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
   {
       return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
   }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 6 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 13 lines
Quoted 10 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 651

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}

There also looks to be a dead code concern here (towards the "nr" parameters here and elsewhere, when STATIC_SHM=n). Just that apparently we decided to leave out Misra rule 2.2 entirely.

I think that I didn't get what is an issue when STATIC_SHM=n, functions is still going to be called through page_get_owner_and_reference(), at least, in page_alloc.c .

Quoted 7 lines
Quoted 2 lines

Yes, but will "nr" ever be anything other than 1 then? IOW omitting the parameter would be fine. And that's what "dead code" is about.

Got it.

So we don't have any SAF-x tag to mark this function as safe. What is the best one solution for now if nr argument will be needed in the future for STATIC_SHM=y?

Add the parameter at that point. Just like was done for Arm.

Jan

#150 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 61 lines · Jan Beulich
Quoted 60 lines · Oleksii Kurochko
Quoted 59 lines · Jan Beulich
Quoted 58 lines · Oleksii Kurochko
Quoted 42 lines · Jan Beulich
Quoted 41 lines · Jan Beulich
Quoted 14 lines · Oleksii Kurochko
(quoted diff) +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
    {
        return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
    }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

Review void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
(quoted diff) +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

Review void __iomem *ioremap(paddr_t pa, size_t len) · line 640

Quoted 6 lines
(quoted diff) +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 13 lines
Quoted 10 lines

Review void __iomem *ioremap(paddr_t pa, size_t len) · line 650

Quoted 10 lines
(quoted diff) +10 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}

There also looks to be a dead code concern here (towards the "nr" parameters here and elsewhere, when STATIC_SHM=n). Just that apparently we decided to leave out Misra rule 2.2 entirely.

I think that I didn't get what is an issue when STATIC_SHM=n, functions is still going to be called through page_get_owner_and_reference(), at least, in page_alloc.c .

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

Yes, but will "nr" ever be anything other than 1 then? IOW omitting the parameter would be fine. And that's what "dead code" is about.

Got it.

So we don't have any SAF-x tag to mark this function as safe. What is the best one solution for now if nr argument will be needed in the future for STATIC_SHM=y?

Add the parameter at that point. Just like was done for Arm.

Hmm, it seems like I am confusing something... Arm has the same defintion and declaration of page_get_owner_and_nr_reference().

~ Oleksii

#151 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 65 lines · Oleksii Kurochko
Quoted 63 lines · Jan Beulich
Quoted 62 lines · Oleksii Kurochko
Quoted 61 lines · Jan Beulich
Quoted 56 lines · Oleksii Kurochko
Quoted 42 lines · Jan Beulich
Quoted 41 lines · Jan Beulich
Quoted 14 lines · Oleksii Kurochko
(quoted diff) +10 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
    {
        return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
    }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)

Review void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 10 lines
(quoted diff) +10 -0
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

Review void __iomem *ioremap(paddr_t pa, size_t len) · line 640

Quoted 6 lines
(quoted diff) +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 13 lines
Quoted 10 lines

Review void __iomem *ioremap(paddr_t pa, size_t len) · line 650

Quoted 10 lines
(quoted diff) +10 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}

There also looks to be a dead code concern here (towards the "nr" parameters here and elsewhere, when STATIC_SHM=n). Just that apparently we decided to leave out Misra rule 2.2 entirely.

Quoted 2 lines

I think that I didn't get what is an issue when STATIC_SHM=n, functions is still going to be called through page_get_owner_and_reference(), at least, in page_alloc.c .

Yes, but will "nr" ever be anything other than 1 then? IOW omitting the parameter would be fine. And that's what "dead code" is about.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

Got it.

So we don't have any SAF-x tag to mark this function as safe. What is the best one solution for now if nr argument will be needed in the future for STATIC_SHM=y?

Add the parameter at that point. Just like was done for Arm.

Hmm, it seems like I am confusing something... Arm has the same defintion and declaration of page_get_owner_and_nr_reference().

But it didn't always have it. And there is at least one pending issue there. Hence my request to use the simpler variant until someone actually makes STATIC_SHM work on RISC-V. And hopefully by then the issue in Arm code is sorted, and you can clone the code without raising questions.

Jan

#152 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

In reply to Jan Beulich
Quoted 90 lines · Jan Beulich
Quoted 26 lines · Oleksii Kurochko

Implement the mfn_valid() macro to verify whether a given MFN is valid by checking that it falls within the range [start_page, max_page). These bounds are initialized based on the start and end addresses of RAM.

As part of this patch, start_page is introduced and initialized with the PFN of the first RAM page.

Also, after providing a non-stub implementation of the mfn_valid() macro,
the following compilation errors started to occur:
riscv64-linux-gnu-ld: prelink.o: in function `__next_node':
/build/xen/./include/xen/nodemask.h:202: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `get_free_buddy':
/build/xen/common/page_alloc.c:881: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `alloc_heap_pages':
/build/xen/common/page_alloc.c:1043: undefined reference to `page_get_owner_and_reference'
riscv64-linux-gnu-ld: /build/xen/common/page_alloc.c:1098: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `ns16550_interrupt':
/build/xen/drivers/char/ns16550.c:205: undefined reference to `get_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `page_get_owner_and_reference' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value
make[2]: *** [arch/riscv/Makefile:35: xen-syms] Error 1
To resolve these errors, the following functions have also been introduced,
based on their Arm counterparts:
- page_get_owner_and_reference() and its variant to safely acquire a
reference to a page and retrieve its owner.
- put_page() and put_page_nr() to release page references and free the page

Quoted 8 lines

when the count drops to zero.
For put_page_nr(), code related to static memory configuration is wrapped with CONFIG_STATIC_MEMORY, as this configuration has not yet been moved to common code. Therefore, PGC_static and free_domstatic_page() are not introduced for RISC-V. However, since this configuration could be useful in the future, the relevant code is retained and conditionally compiled.
- A stub for page_is_ram_type() that currently always returns 0 and asserts unreachable, as RAM type checking is not yet implemented.

How does this end up working when common code references the function?

xen/arch/riscv/include/asm/mm.h static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr) · line 297

Quoted 12 lines
xen/arch/riscv/include/asm/mm.h +6 -2
@@ -288,8 +289,12 @@ static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
 #define page_get_owner(p)    (p)->v.inuse.domain
 #define page_set_owner(p, d) ((p)->v.inuse.domain = (d))
 
-/* TODO: implement */
-#define mfn_valid(mfn) ({ (void)(mfn); 0; })
+extern unsigned long start_page;
+
+#define mfn_valid(mfn) ({                                   \
+    unsigned long mfn__ = mfn_x(mfn);                       \
+    likely((mfn__ >= start_page) && (mfn__ < max_page));    \
+})

I don't think you should try to be clever and avoid using __mfn_valid() here, at least not without an easily identifiable TODO. Surely you've seen that both Arm and x86 use it.

Also, according to all I know, likely() doesn't work very well when used like this, except for architectures supporting conditionally executed insns (like Arm32 or IA-64, i.e. beyond conditional branches). I.e. if you want to use likely() here, I think you need two of them.

xen/arch/riscv/mm.c static void __init setup_directmap_mappings(unsigned long base_mfn, · line 523

Quoted 5 lines
xen/arch/riscv/mm.c +1 -0
@@ -525,6 +520,8 @@ static void __init setup_directmap_mappings(unsigned long base_mfn,
 #error setup_{directmap,frametable}_mapping() should be implemented for RV_32
 #endif
 
+unsigned long __read_mostly start_page;

Memory hotplug question again: __read_mostly or __ro_after_init?

Quoted 22 lines
xen/arch/riscv/mm.c +18 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
 {
     return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
 }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
+        ASSERT_UNREACHABLE();
Quoted 8 lines

Review void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 2 lines
(quoted diff) +2 -0
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

Honestly I don't know why this assert was placed here. I checked the code and we don't limit nr_shm_borrowers in any place, so in theory it's possible to end up here.

~Michal

Quoted 52 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 7 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 24 lines
xen/arch/riscv/mm.c +24 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}
+
+struct domain *page_get_owner_and_reference(struct page_info *page)
+{
+    return page_get_owner_and_nr_reference(page, 1);
+}
+
+void put_page_nr(struct page_info *page, unsigned long nr)
+{
+    unsigned long nx, x, y = page->count_info;
+
+    do {
+        ASSERT((y & PGC_count_mask) >= nr);
+        x  = y;
+        nx = x - nr;

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 675

Quoted 10 lines
xen/arch/riscv/mm.c +10 -0
+    }
+    while ( unlikely((y = cmpxchg(&page->count_info, x, nx)) != x) );
+
+    if ( unlikely((nx & PGC_count_mask) == 0) )
+    {
+#ifdef CONFIG_STATIC_MEMORY
+        if ( unlikely(nx & PGC_static) )
+            free_domstatic_page(page);
+        else
+#endif

Such #ifdef-ed-out code is liable to go stale. Minimally use IS_ENABLED(). Even better would imo be if you introduced a "stub" PGC_static, resolving to 0 (i.e. for now unconditionally).

Jan

#153 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

In reply to Jan Beulich
Quoted 35 lines · Jan Beulich
Quoted 26 lines · Oleksii Kurochko

Implement the mfn_valid() macro to verify whether a given MFN is valid by checking that it falls within the range [start_page, max_page). These bounds are initialized based on the start and end addresses of RAM.

As part of this patch, start_page is introduced and initialized with the PFN of the first RAM page.

Also, after providing a non-stub implementation of the mfn_valid() macro,
the following compilation errors started to occur:
riscv64-linux-gnu-ld: prelink.o: in function `__next_node':
/build/xen/./include/xen/nodemask.h:202: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `get_free_buddy':
/build/xen/common/page_alloc.c:881: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `alloc_heap_pages':
/build/xen/common/page_alloc.c:1043: undefined reference to `page_get_owner_and_reference'
riscv64-linux-gnu-ld: /build/xen/common/page_alloc.c:1098: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `ns16550_interrupt':
/build/xen/drivers/char/ns16550.c:205: undefined reference to `get_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `page_get_owner_and_reference' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value
make[2]: *** [arch/riscv/Makefile:35: xen-syms] Error 1
To resolve these errors, the following functions have also been introduced,
based on their Arm counterparts:
- page_get_owner_and_reference() and its variant to safely acquire a
reference to a page and retrieve its owner.
- put_page() and put_page_nr() to release page references and free the page

Quoted 8 lines

when the count drops to zero.
For put_page_nr(), code related to static memory configuration is wrapped with CONFIG_STATIC_MEMORY, as this configuration has not yet been moved to common code. Therefore, PGC_static and free_domstatic_page() are not introduced for RISC-V. However, since this configuration could be useful in the future, the relevant code is retained and conditionally compiled.
- A stub for page_is_ram_type() that currently always returns 0 and asserts unreachable, as RAM type checking is not yet implemented.

Quoted 1 line

How does this end up working when common code references the function?

Based on the following commit message:
Callers are VT-d (so x86 specific) and various bits of page offlining support, which although it looks generic (and is in xen/common) does things like diving into page_info->count_info which is not generic.

In any case on this is only reachable via XEN_SYSCTL_page_offline_op, which clearly shouldn't be called on ARM just yet.

There is no callers for page_is_ram_type() for Arm now, and I expect something similar for RISC-V. As we don't even introduce hypercalls for RISC-V, we can just live without it.

Quoted 5 lines
Quoted 5 lines
xen/arch/riscv/include/asm/mm.h +0 -0
@@ -288,8 +289,12 @@ static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
  #define page_get_owner(p)    (p)->v.inuse.domain
  #define page_set_owner(p, d) ((p)->v.inuse.domain = (d))
 
Quoted 11 lines

xen/arch/riscv/include/asm/mm.h static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr) · line 297

Quoted 8 lines
xen/arch/riscv/include/asm/mm.h +6 -2
-/* TODO: implement */
-#define mfn_valid(mfn) ({ (void)(mfn); 0; })
+extern unsigned long start_page;
+
+#define mfn_valid(mfn) ({                                   \
+    unsigned long mfn__ = mfn_x(mfn);                       \
+    likely((mfn__ >= start_page) && (mfn__ < max_page));    \
+})

I don't think you should try to be clever and avoid using __mfn_valid() here, at least not without an easily identifiable TODO. Surely you've seen that both Arm and x86 use it.

Overlooked that pdx.c is compiled unconditionally, so I thought that __mfn_valid() common implementation isn't avaiable as, at the moment, RISC-V doesn't support PDX_COMPRESSION...

Quoted 4 lines

Also, according to all I know, likely() doesn't work very well when used like this, except for architectures supporting conditionally executed insns (like Arm32 or IA-64, i.e. beyond conditional branches). I.e. if you want to use likely() here, I think you need two of them.

... I will update mfn_valid() definition according to your recommendations.

Quoted 29 lines

xen/arch/riscv/mm.c static void __init setup_directmap_mappings(unsigned long base_mfn, · line 523

Quoted 6 lines
xen/arch/riscv/mm.c +1 -0
@@ -525,6 +520,8 @@ static void __init setup_directmap_mappings(unsigned long base_mfn,
  #error setup_{directmap,frametable}_mapping() should be implemented for RV_32
  #endif
 
+unsigned long __read_mostly start_page;

Memory hotplug question again: __read_mostly or __ro_after_init?

Quoted 21 lines
xen/arch/riscv/mm.c +17 -0
@@ -613,3 +612,91 @@ void __iomem *ioremap(paddr_t pa, size_t len)
  {
      return ioremap_attr(pa, len, PAGE_HYPERVISOR_NOCACHE);
  }
+
+int page_is_ram_type(unsigned long mfn, unsigned long mem_type)
+{
+    ASSERT_UNREACHABLE();
+
+    return 0;
+}
+
+static struct domain *page_get_owner_and_nr_reference(struct page_info *page,
+                                                      unsigned long nr)
+{
+    unsigned long x, y = page->count_info;
+    struct domain *owner;
+
+    /* Restrict nr to avoid "double" overflow */
+    if ( nr >= PGC_count_mask )
+    {
Quoted 8 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 634

Quoted 3 lines
xen/arch/riscv/mm.c +3 -0
+        ASSERT_UNREACHABLE();
+        return NULL;
+    }

I question the validity of this, already in the Arm original: I can't spot how the caller guarantees to stay below that limit. Without such an (attempted) guarantee, ASSERT_UNREACHABLE() is wrong to use. All I can see is process_shm_node() incrementing shmem_extra[].nr_shm_borrowers, without any limit check.

Agree, it could be really dropped.

Quoted 40 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 641

Quoted 7 lines
xen/arch/riscv/mm.c +6 -0
+    do {
+        x = y;
+        /*
+         * Count ==  0: Page is not allocated, so we cannot take a reference.
+         * Count == -1: Reference count would wrap, which is invalid.
+         */

May I once again ask that you look carefully at comments (as much as at code) you copy. Clearly this comment wasn't properly updated when the bumping by 1 was changed to bumping by nr.

Quoted 29 lines
xen/arch/riscv/mm.c +29 -0
+        if ( unlikely(((x + nr) & PGC_count_mask) <= nr) )
+            return NULL;
+    }
+    while ( (y = cmpxchg(&page->count_info, x, x + nr)) != x );
+
+    owner = page_get_owner(page);
+    ASSERT(owner);
+
+    return owner;
+}
+
+struct domain *page_get_owner_and_reference(struct page_info *page)
+{
+    return page_get_owner_and_nr_reference(page, 1);
+}
+
+void put_page_nr(struct page_info *page, unsigned long nr)
+{
+    unsigned long nx, x, y = page->count_info;
+
+    do {
+        ASSERT((y & PGC_count_mask) >= nr);
+        x  = y;
+        nx = x - nr;
+    }
+    while ( unlikely((y = cmpxchg(&page->count_info, x, nx)) != x) );
+
+    if ( unlikely((nx & PGC_count_mask) == 0) )
+    {
Quoted 8 lines

xen/arch/riscv/mm.c void __iomem *ioremap(paddr_t pa, size_t len) · line 675

Quoted 5 lines
xen/arch/riscv/mm.c +5 -0
+#ifdef CONFIG_STATIC_MEMORY
+        if ( unlikely(nx & PGC_static) )
+            free_domstatic_page(page);
+        else
+#endif

Such #ifdef-ed-out code is liable to go stale. Minimally use IS_ENABLED(). Even better would imo be if you introduced a "stub" PGC_static, resolving to 0 (i.e. for now unconditionally).

An introduction of PGC_static would be better.

Thanks.

~ Oleksii

#154 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 38 lines · Oleksii Kurochko
Quoted 35 lines · Jan Beulich
Quoted 26 lines · Oleksii Kurochko

Implement the mfn_valid() macro to verify whether a given MFN is valid by checking that it falls within the range [start_page, max_page). These bounds are initialized based on the start and end addresses of RAM.

As part of this patch, start_page is introduced and initialized with the PFN of the first RAM page.

Also, after providing a non-stub implementation of the mfn_valid() macro,
the following compilation errors started to occur:
riscv64-linux-gnu-ld: prelink.o: in function `__next_node':
/build/xen/./include/xen/nodemask.h:202: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `get_free_buddy':
/build/xen/common/page_alloc.c:881: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `alloc_heap_pages':
/build/xen/common/page_alloc.c:1043: undefined reference to `page_get_owner_and_reference'
riscv64-linux-gnu-ld: /build/xen/common/page_alloc.c:1098: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `ns16550_interrupt':
/build/xen/drivers/char/ns16550.c:205: undefined reference to `get_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `page_get_owner_and_reference' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value
make[2]: *** [arch/riscv/Makefile:35: xen-syms] Error 1
To resolve these errors, the following functions have also been introduced,
based on their Arm counterparts:
- page_get_owner_and_reference() and its variant to safely acquire a
reference to a page and retrieve its owner.
- put_page() and put_page_nr() to release page references and free the page

Quoted 8 lines

when the count drops to zero.
For put_page_nr(), code related to static memory configuration is wrapped with CONFIG_STATIC_MEMORY, as this configuration has not yet been moved to common code. Therefore, PGC_static and free_domstatic_page() are not introduced for RISC-V. However, since this configuration could be useful in the future, the relevant code is retained and conditionally compiled.
- A stub for page_is_ram_type() that currently always returns 0 and asserts unreachable, as RAM type checking is not yet implemented.

Quoted 1 line

How does this end up working when common code references the function?

Quoted 7 lines

Based on the following commit message:
Callers are VT-d (so x86 specific) and various bits of page offlining support, which although it looks generic (and is in xen/common) does things like diving into page_info->count_info which is not generic.

In any case on this is only reachable via XEN_SYSCTL_page_offline_op, which clearly shouldn't be called on ARM just yet.

What commit message are you talking about? Nothing like the above is anywhere in this patch.

Quoted 3 lines

There is no callers for page_is_ram_type() for Arm now, and I expect something similar for RISC-V. As we don't even introduce hypercalls for RISC-V, we can just live without it.

If there's no caller, why the stub?

Jan

#155 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

Quoted 43 lines · Jan Beulich
Quoted 42 lines · Oleksii Kurochko
Quoted 35 lines · Jan Beulich
Quoted 26 lines · Oleksii Kurochko

Implement the mfn_valid() macro to verify whether a given MFN is valid by checking that it falls within the range [start_page, max_page). These bounds are initialized based on the start and end addresses of RAM.

As part of this patch, start_page is introduced and initialized with the PFN of the first RAM page.

Also, after providing a non-stub implementation of the mfn_valid() macro,
the following compilation errors started to occur:
riscv64-linux-gnu-ld: prelink.o: in function `__next_node':
/build/xen/./include/xen/nodemask.h:202: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `get_free_buddy':
/build/xen/common/page_alloc.c:881: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `alloc_heap_pages':
/build/xen/common/page_alloc.c:1043: undefined reference to `page_get_owner_and_reference'
riscv64-linux-gnu-ld: /build/xen/common/page_alloc.c:1098: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `ns16550_interrupt':
/build/xen/drivers/char/ns16550.c:205: undefined reference to `get_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `page_get_owner_and_reference' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value
make[2]: *** [arch/riscv/Makefile:35: xen-syms] Error 1
To resolve these errors, the following functions have also been introduced,
based on their Arm counterparts:
- page_get_owner_and_reference() and its variant to safely acquire a
reference to a page and retrieve its owner.
- put_page() and put_page_nr() to release page references and free the page

Quoted 8 lines

when the count drops to zero.
For put_page_nr(), code related to static memory configuration is wrapped with CONFIG_STATIC_MEMORY, as this configuration has not yet been moved to common code. Therefore, PGC_static and free_domstatic_page() are not introduced for RISC-V. However, since this configuration could be useful in the future, the relevant code is retained and conditionally compiled.
- A stub for page_is_ram_type() that currently always returns 0 and asserts unreachable, as RAM type checking is not yet implemented.

Quoted 1 line

How does this end up working when common code references the function?

Based on the following commit message:
Callers are VT-d (so x86 specific) and various bits of page offlining support, which although it looks generic (and is in xen/common) does things like diving into page_info->count_info which is not generic.

Quoted 4 lines
Quoted 2 lines

In any case on this is only reachable via XEN_SYSCTL_page_offline_op, which clearly shouldn't be called on ARM just yet.

What commit message are you talking about? Nothing like the above is anywhere in this patch.

It's pretty old commit:
commit 214c4cd94a80bcaf042f25158eaa7d0e5bbc3b5b
Author: Ian Campbell<[email protected]>
Date: Wed Dec 19 14:16:23 2012 +0000

But since that time page_is_ram_type() hasn't been changed for Arm.

Quoted 5 lines
Quoted 4 lines

There is no callers for page_is_ram_type() for Arm now, and I expect something similar for RISC-V. As we don't even introduce hypercalls for RISC-V, we can just live without it.

If there's no caller, why the stub?

Because that parts of common code which are using it aren't under ifdef, for example, this one function:
int offline_page(mfn_t mfn, int broken, uint32_t *status)

And specifically this function is called when XEN_SYSCTL_page_offline_op is handled.

But I agree that it seems like nothing prevents to call XEN_SYSCTL_page_offline_op.

~ Oleksii

#156 lore raw

Re: [PATCH v2 16/17] xen/riscv: implement mfn_valid() and page reference, ownership handling helpers

In reply to Oleksii Kurochko
Quoted 38 lines · Oleksii Kurochko
Quoted 35 lines · Jan Beulich
Quoted 26 lines · Oleksii Kurochko

Implement the mfn_valid() macro to verify whether a given MFN is valid by checking that it falls within the range [start_page, max_page). These bounds are initialized based on the start and end addresses of RAM.

As part of this patch, start_page is introduced and initialized with the PFN of the first RAM page.

Also, after providing a non-stub implementation of the mfn_valid() macro,
the following compilation errors started to occur:
riscv64-linux-gnu-ld: prelink.o: in function `__next_node':
/build/xen/./include/xen/nodemask.h:202: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `get_free_buddy':
/build/xen/common/page_alloc.c:881: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `alloc_heap_pages':
/build/xen/common/page_alloc.c:1043: undefined reference to `page_get_owner_and_reference'
riscv64-linux-gnu-ld: /build/xen/common/page_alloc.c:1098: undefined reference to `page_is_ram_type'
riscv64-linux-gnu-ld: prelink.o: in function `ns16550_interrupt':
/build/xen/drivers/char/ns16550.c:205: undefined reference to `get_page'
riscv64-linux-gnu-ld: ./.xen-syms.0: hidden symbol `page_get_owner_and_reference' isn't defined
riscv64-linux-gnu-ld: final link failed: bad value
make[2]: *** [arch/riscv/Makefile:35: xen-syms] Error 1
To resolve these errors, the following functions have also been introduced,
based on their Arm counterparts:
- page_get_owner_and_reference() and its variant to safely acquire a
reference to a page and retrieve its owner.
- put_page() and put_page_nr() to release page references and free the page

Quoted 8 lines

when the count drops to zero.
For put_page_nr(), code related to static memory configuration is wrapped with CONFIG_STATIC_MEMORY, as this configuration has not yet been moved to common code. Therefore, PGC_static and free_domstatic_page() are not introduced for RISC-V. However, since this configuration could be useful in the future, the relevant code is retained and conditionally compiled.
- A stub for page_is_ram_type() that currently always returns 0 and asserts unreachable, as RAM type checking is not yet implemented.

Quoted 1 line

How does this end up working when common code references the function?

Quoted 7 lines

Based on the following commit message:
Callers are VT-d (so x86 specific) and various bits of page offlining support, which although it looks generic (and is in xen/common) does things like diving into page_info->count_info which is not generic.

In any case on this is only reachable via XEN_SYSCTL_page_offline_op, which clearly shouldn't be called on ARM just yet.

Assuming this is from an old commit, then I have to question this justification. I see nothing preventing XEN_SYSCTL_page_offline_op to be invoked on an Arm system. Hence (unless I'm overlooking somthing) ASSERT_UNREACHABLE() is simply inappropriate (and wants fixing). Luckily it being sysctl-s only, there's no need for an XSA. In no case should known flawed code be copied into another port.

Jan

#157 lore raw

PATCH v2 17/17 xen/riscv: add support of page lookup by GFN

In reply to Oleksii Kurochko

Introduce helper functions for safely querying the P2M (physical-to-machine) mapping:
- add p2m_read_lock(), p2m_read_unlock(), and p2m_is_locked() for managing P2M lock state.
- Implement p2m_get_entry() to retrieve mapping details for a given GFN, including MFN, page order, and validity.
- Add p2m_lookup() to encapsulate read-locked MFN retrieval.
- Introduce p2m_get_page_from_gfn() to convert a GFN into a page_info pointer, acquiring a reference to the page if valid.

Implementations are based on Arm's functions with some minor modifications:
- p2m_get_entry():
- Reverse traversal of page tables, as RISC-V uses the opposite order compared to Arm.
- Removed the return of p2m_access_t from p2m_get_entry() since mem_access_settings is not introduced for RISC-V.
- Updated BUILD_BUG_ON() to check using the level 0 mask, which corresponds to Arm's THIRD_MASK.
- Replaced open-coded bit shifts with the BIT() macro.
- Other minor changes, such as using RISC-V-specific functions to validate P2M PTEs, and replacing Arm-specific GUEST_* macros with their RISC-V equivalents.
- p2m_get_page_from_gfn():
- Removed p2m_is_foreign() and related logic, as this functionality is not implemented for RISC-V.

---
Changes in V2:
- New patch.

---
 xen/arch/riscv/include/asm/p2m.h |  18 +++++
 xen/arch/riscv/p2m.c             | 131 +++++++++++++++++++++++++++++++
 2 files changed, 149 insertions(+)
xen/arch/riscv/include/asm/p2m.h +18 -0
index fdebd18356..96e0790dbc 100644
@@ -184,6 +184,24 @@ static inline int p2m_is_write_locked(struct p2m_domain *p2m)
     return rw_is_write_locked(&p2m->lock);
 }
 
+static inline void p2m_read_lock(struct p2m_domain *p2m)
+{
+    read_lock(&p2m->lock);
+}
+
+static inline void p2m_read_unlock(struct p2m_domain *p2m)
+{
+    read_unlock(&p2m->lock);
+}
+
+static inline int p2m_is_locked(struct p2m_domain *p2m)
+{
+    return rw_is_locked(&p2m->lock);
+}
+
+struct page_info *p2m_get_page_from_gfn(struct domain *d, gfn_t gfn,
+                                        p2m_type_t *t);
+
 #endif /* ASM__RISCV__P2M_H */
 
 /*
xen/arch/riscv/p2m.c +131 -0

134 lines not on this page · raw message

Signature · 1 line
2.49.0
#158 lore raw

Re: [PATCH v2 17/17] xen/riscv: add support of page lookup by GFN

Quoted 10 lines · Oleksii Kurochko

Introduce helper functions for safely querying the P2M (physical-to-machine) mapping:
- add p2m_read_lock(), p2m_read_unlock(), and p2m_is_locked() for managing P2M lock state.
- Implement p2m_get_entry() to retrieve mapping details for a given GFN, including MFN, page order, and validity.
- Add p2m_lookup() to encapsulate read-locked MFN retrieval.
- Introduce p2m_get_page_from_gfn() to convert a GFN into a page_info pointer, acquiring a reference to the page if valid.

Quoted 6 lines

Implementations are based on Arm's functions with some minor modifications:
- p2m_get_entry():
- Reverse traversal of page tables, as RISC-V uses the opposite order compared to Arm.
- Removed the return of p2m_access_t from p2m_get_entry() since mem_access_settings is not introduced for RISC-V.

Didn't I see uses of p2m_access in earlier patches? If you don't mean to have that, then please consistently {every,no}where.

Quoted 9 lines

- Updated BUILD_BUG_ON() to check using the level 0 mask, which corresponds to Arm's THIRD_MASK.
- Replaced open-coded bit shifts with the BIT() macro.
- Other minor changes, such as using RISC-V-specific functions to validate P2M PTEs, and replacing Arm-specific GUEST_* macros with their RISC-V equivalents.
- p2m_get_page_from_gfn():
- Removed p2m_is_foreign() and related logic, as this functionality is not implemented for RISC-V.

Yet I expect you'll need this, sooner or later.

Quoted 13 lines
xen/arch/riscv/include/asm/p2m.h +7 -0
@@ -184,6 +184,24 @@ static inline int p2m_is_write_locked(struct p2m_domain *p2m)
     return rw_is_write_locked(&p2m->lock);
 }
 
+static inline void p2m_read_lock(struct p2m_domain *p2m)
+{
+    read_lock(&p2m->lock);
+}
+
+static inline void p2m_read_unlock(struct p2m_domain *p2m)
+{

xen/arch/riscv/include/asm/p2m.h static inline int p2m_is_write_locked(struct p2m_domain *p2m) · line 203

Quoted 10 lines
xen/arch/riscv/include/asm/p2m.h +10 -0
+    read_unlock(&p2m->lock);
+}
+
+static inline int p2m_is_locked(struct p2m_domain *p2m)
+{
+    return rw_is_locked(&p2m->lock);
+}
+
+struct page_info *p2m_get_page_from_gfn(struct domain *d, gfn_t gfn,
+                                        p2m_type_t *t);

Once again I don't think you can pass struct domain * here, when in the long run a domain can have multiple P2Ms.

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1063

Quoted 12 lines
xen/arch/riscv/p2m.c +6 -0
@@ -1055,3 +1055,134 @@ int guest_physmap_add_entry(struct domain *d,
 {
     return p2m_insert_mapping(d, gfn, (1 << page_order), mfn, t);
 }
+
+/*
+ * Get the details of a given gfn.
+ *
+ * If the entry is present, the associated MFN will be returned and the
+ * access and type filled up. The page_order will correspond to the

You removed p2m_access_t * from the parameters; you need to also update the comment then accordingly.

Quoted 9 lines
xen/arch/riscv/p2m.c +9 -0
+ * order of the mapping in the page table (i.e it could be a superpage).
+ *
+ * If the entry is not present, INVALID_MFN will be returned and the
+ * page_order will be set according to the order of the invalid range.
+ *
+ * valid will contain the value of bit[0] (e.g valid bit) of the
+ * entry.
+ */
+static mfn_t p2m_get_entry(struct p2m_domain *p2m, gfn_t gfn,

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1082

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+                           p2m_type_t *t,
+                           unsigned int *page_order,
+                           bool *valid)
+{
+    paddr_t addr = gfn_to_gaddr(gfn);
+    unsigned int level = 0;
+    pte_t entry, *table;
+    int rc;
+    mfn_t mfn = INVALID_MFN;
+    p2m_type_t _t;

Please no local variables with leading underscores. In x86 we commonly name such variables p2mt.

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1083

Quoted 1 line
xen/arch/riscv/p2m.c +1 -0
+    DECLARE_OFFSETS(offsets, addr);

This is the sole use of "addr". Is such a local variable really worth having?

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    ASSERT(p2m_is_locked(p2m));
+    BUILD_BUG_ON(XEN_PT_LEVEL_MAP_MASK(0) != PAGE_MASK);
+
+    /* Allow t to be NULL */
+    t = t ?: &_t;
+
+    *t = p2m_invalid;

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1101

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+
+    if ( valid )
+        *valid = false;
+
+    /* XXX: Check if the mapping is lower than the mapped gfn */
+
+    /* This gfn is higher than the highest the p2m map currently holds */
+    if ( gfn_x(gfn) > gfn_x(p2m->max_mapped_gfn) )
+    {
+        for ( level = P2M_ROOT_LEVEL; level ; level-- )

Nit: Stray blank before the 2nd semicolon. (Again at least once below.)

Quoted 14 lines
xen/arch/riscv/p2m.c +14 -0
+            if ( (gfn_x(gfn) & (XEN_PT_LEVEL_MASK(level) >> PAGE_SHIFT)) >
+                 gfn_x(p2m->max_mapped_gfn) )
+                break;
+
+        goto out;
+    }
+
+    table = p2m_get_root_pointer(p2m, gfn);
+
+    /*
+     * the table should always be non-NULL because the gfn is below
+     * p2m->max_mapped_gfn and the root table pages are always present.
+     */
+    if ( !table )

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1125

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+    {
+        ASSERT_UNREACHABLE();
+        level = P2M_ROOT_LEVEL;
+        goto out;
+    }
+
+    for ( level = P2M_ROOT_LEVEL; level ; level-- )
+    {
+        rc = p2m_next_level(p2m, true, level, &table, offsets[level]);
+        if ( (rc == GUEST_TABLE_MAP_NONE) && (rc != GUEST_TABLE_MAP_NOMEM) )

This condition looks odd. As written the rhs of the && is redundant.

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1127

Quoted 2 lines
xen/arch/riscv/p2m.c +2 -0
+            goto out_unmap;
+        else if ( rc != GUEST_TABLE_NORMAL )

As before, no real need for "else" in such cases.

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1135

Quoted 8 lines
xen/arch/riscv/p2m.c +8 -0
+            break;
+    }
+
+    entry = table[offsets[level]];
+
+    if ( p2me_is_valid(p2m, entry) )
+    {
+        *t = p2m_type_radix_get(p2m, entry);

If the incoming argument is NULL, the somewhat expensive radix tree lookup is unnecessary here.

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1146

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+        mfn = pte_get_mfn(entry);
+        /*
+         * The entry may point to a superpage. Find the MFN associated
+         * to the GFN.
+         */
+        mfn = mfn_add(mfn,
+                      gfn_x(gfn) & (BIT(XEN_PT_LEVEL_ORDER(level), UL) - 1));
+
+        if ( valid )
+            *valid = pte_is_valid(entry);

Interesting. Why not the P2M counterpart of the function? Yes, the comment ahead of the function says so, but I don't see why the valid bit suddenly is relevant here (besides the P2M type).

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1152

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    }
+
+out_unmap:
+    unmap_domain_page(table);
+
+out:

Nit: Style (bot labels).

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1159

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    if ( page_order )
+        *page_order = XEN_PT_LEVEL_ORDER(level);
+
+    return mfn;
+}
+
+static mfn_t p2m_lookup(struct domain *d, gfn_t gfn, p2m_type_t *t)

pointer-to-const for the 1st arg? But again more likely struct p2m_domain * anyway?

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1171

Quoted 12 lines
xen/arch/riscv/p2m.c +12 -0
+{
+    mfn_t mfn;
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
+
+    p2m_read_lock(p2m);
+    mfn = p2m_get_entry(p2m, gfn, t, NULL, NULL);
+    p2m_read_unlock(p2m);
+
+    return mfn;
+}
+
+struct page_info *p2m_get_page_from_gfn(struct domain *d, gfn_t gfn,

Same here - likely you mean struct p2m_domain * instead.

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1174

Quoted 3 lines
xen/arch/riscv/p2m.c +3 -0
+                                        p2m_type_t *t)
+{
+    p2m_type_t p2mt = {0};

Why a compound initializer for something that isn't a compound object? And why plain 0 for something that is an enumerated type?

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1180

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    struct page_info *page;
+
+    mfn_t mfn = p2m_lookup(d, gfn, &p2mt);
+
+    if ( t )
+        *t = p2mt;

What's wrong with passing t directly to p2m_lookup()?

Jan

#159 lore raw

Re: [PATCH v2 17/17] xen/riscv: add support of page lookup by GFN

Quoted 11 lines · Jan Beulich
Quoted 10 lines · Oleksii Kurochko

Introduce helper functions for safely querying the P2M (physical-to-machine) mapping:
- add p2m_read_lock(), p2m_read_unlock(), and p2m_is_locked() for managing P2M lock state.
- Implement p2m_get_entry() to retrieve mapping details for a given GFN, including MFN, page order, and validity.
- Add p2m_lookup() to encapsulate read-locked MFN retrieval.
- Introduce p2m_get_page_from_gfn() to convert a GFN into a page_info pointer, acquiring a reference to the page if valid.

Quoted 8 lines
Quoted 6 lines

Implementations are based on Arm's functions with some minor modifications:
- p2m_get_entry():
- Reverse traversal of page tables, as RISC-V uses the opposite order compared to Arm.
- Removed the return of p2m_access_t from p2m_get_entry() since mem_access_settings is not introduced for RISC-V.

Didn't I see uses of p2m_access in earlier patches? If you don't mean to have that, then please consistently {every,no}where.

Yes, it was used. I think it would be better just usage of p2m_access from earlier patches.

Quoted 10 lines
Quoted 9 lines

- Updated BUILD_BUG_ON() to check using the level 0 mask, which corresponds to Arm's THIRD_MASK.
- Replaced open-coded bit shifts with the BIT() macro.
- Other minor changes, such as using RISC-V-specific functions to validate P2M PTEs, and replacing Arm-specific GUEST_* macros with their RISC-V equivalents.
- p2m_get_page_from_gfn():
- Removed p2m_is_foreign() and related logic, as this functionality is not implemented for RISC-V.

Yet I expect you'll need this, sooner or later.

Then I'll add correspondent code in this patch.

Quoted 37 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1063

Quoted 12 lines
xen/arch/riscv/p2m.c +6 -0
@@ -1055,3 +1055,134 @@ int guest_physmap_add_entry(struct domain *d,
  {
      return p2m_insert_mapping(d, gfn, (1 << page_order), mfn, t);
  }
+
+/*
+ * Get the details of a given gfn.
+ *
+ * If the entry is present, the associated MFN will be returned and the
+ * access and type filled up. The page_order will correspond to the

You removed p2m_access_t * from the parameters; you need to also update the comment then accordingly.

Quoted 9 lines
xen/arch/riscv/p2m.c +9 -0
+ * order of the mapping in the page table (i.e it could be a superpage).
+ *
+ * If the entry is not present, INVALID_MFN will be returned and the
+ * page_order will be set according to the order of the invalid range.
+ *
+ * valid will contain the value of bit[0] (e.g valid bit) of the
+ * entry.
+ */
+static mfn_t p2m_get_entry(struct p2m_domain *p2m, gfn_t gfn,

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1082

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+                           p2m_type_t *t,
+                           unsigned int *page_order,
+                           bool *valid)
+{
+    paddr_t addr = gfn_to_gaddr(gfn);
+    unsigned int level = 0;
+    pte_t entry, *table;
+    int rc;
+    mfn_t mfn = INVALID_MFN;
+    p2m_type_t _t;

Please no local variables with leading underscores. In x86 we commonly name such variables p2mt.

Quoted 2 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1083

Quoted 1 line
xen/arch/riscv/p2m.c +1 -0
+    DECLARE_OFFSETS(offsets, addr);

This is the sole use of "addr". Is such a local variable really worth having?

Not really, I'll drop it.

Quoted 36 lines
Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    ASSERT(p2m_is_locked(p2m));
+    BUILD_BUG_ON(XEN_PT_LEVEL_MAP_MASK(0) != PAGE_MASK);
+
+    /* Allow t to be NULL */
+    t = t ?: &_t;
+
+    *t = p2m_invalid;

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1101

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+
+    if ( valid )
+        *valid = false;
+
+    /* XXX: Check if the mapping is lower than the mapped gfn */
+
+    /* This gfn is higher than the highest the p2m map currently holds */
+    if ( gfn_x(gfn) > gfn_x(p2m->max_mapped_gfn) )
+    {
+        for ( level = P2M_ROOT_LEVEL; level ; level-- )

Nit: Stray blank before the 2nd semicolon. (Again at least once below.)

Quoted 17 lines
xen/arch/riscv/p2m.c +17 -0
+            if ( (gfn_x(gfn) & (XEN_PT_LEVEL_MASK(level) >> PAGE_SHIFT)) >
+                 gfn_x(p2m->max_mapped_gfn) )
+                break;
+
+        goto out;
+    }
+
+    table = p2m_get_root_pointer(p2m, gfn);
+
+    /*
+     * the table should always be non-NULL because the gfn is below
+     * p2m->max_mapped_gfn and the root table pages are always present.
+     */
+    if ( !table )
+    {
+        ASSERT_UNREACHABLE();
+        level = P2M_ROOT_LEVEL;
Quoted 8 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1125

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+        goto out;
+    }
+
+    for ( level = P2M_ROOT_LEVEL; level ; level-- )
+    {
+        rc = p2m_next_level(p2m, true, level, &table, offsets[level]);
+        if ( (rc == GUEST_TABLE_MAP_NONE) && (rc != GUEST_TABLE_MAP_NOMEM) )

This condition looks odd. As written the rhs of the && is redundant.

And it is wrong. It should:
if ( (rc == P2M_TABLE_MAP_NONE) || (rc == P2M_TABLE_MAP_NOMEM) )

Quoted 4 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1127

Quoted 2 lines
xen/arch/riscv/p2m.c +2 -0
+            goto out_unmap;
+        else if ( rc != GUEST_TABLE_NORMAL )

As before, no real need for "else" in such cases.

Quoted 10 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1135

Quoted 8 lines
xen/arch/riscv/p2m.c +8 -0
+            break;
+    }
+
+    entry = table[offsets[level]];
+
+    if ( p2me_is_valid(p2m, entry) )
+    {
+        *t = p2m_type_radix_get(p2m, entry);

If the incoming argument is NULL, the somewhat expensive radix tree lookup is unnecessary here.

Good point.

Quoted 10 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1146

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+        mfn = pte_get_mfn(entry);
+        /*
+         * The entry may point to a superpage. Find the MFN associated
+         * to the GFN.
+         */
+        mfn = mfn_add(mfn,
+                      gfn_x(gfn) & (BIT(XEN_PT_LEVEL_ORDER(level), UL) - 1));
+
+        if ( valid )
+            *valid = pte_is_valid(entry);
Quoted 3 lines

Interesting. Why not the P2M counterpart of the function? Yes, the comment ahead of the function says so, but I don't see why the valid bit suddenly is relevant here (besides the P2M type).

This valid variable is expected to be used in the caller (something what Arm does here
https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/p2m.c#L375) to check if it is needed to do flush_page_to_ram(), so if the the valid bit in PTE was set to 0 then it means nothing should be flushed as entry wasn't used as it marked invalid.

Quoted 8 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1152

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    }
+
+out_unmap:
+    unmap_domain_page(table);
+
+out:

Nit: Style (bot labels).

Quoted 9 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1159

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    if ( page_order )
+        *page_order = XEN_PT_LEVEL_ORDER(level);
+
+    return mfn;
+}
+
+static mfn_t p2m_lookup(struct domain *d, gfn_t gfn, p2m_type_t *t)

pointer-to-const for the 1st arg? But again more likely struct p2m_domain * anyway?

|struct p2_domain| would be better, but I’m not really sure that a pointer-to-const can be used here. I expect that, in that case,

|p2m_read_lock()| would also need to accept a pointer-to-const, and since it
calls|read_lock()| internally, that could be a problem because|read_lock() |accepts a|rwlock_t *l|.
Quoted 14 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1171

Quoted 12 lines
xen/arch/riscv/p2m.c +12 -0
+{
+    mfn_t mfn;
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
+
+    p2m_read_lock(p2m);
+    mfn = p2m_get_entry(p2m, gfn, t, NULL, NULL);
+    p2m_read_unlock(p2m);
+
+    return mfn;
+}
+
+struct page_info *p2m_get_page_from_gfn(struct domain *d, gfn_t gfn,

Same here - likely you mean struct p2m_domain * instead.

Quoted 5 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1174

Quoted 3 lines
xen/arch/riscv/p2m.c +3 -0
+                                        p2m_type_t *t)
+{
+    p2m_type_t p2mt = {0};

Why a compound initializer for something that isn't a compound object? And why plain 0 for something that is an enumerated type?

Agree, it should be a compound initializer. I'll drop it.

Quoted 8 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1180

Quoted 7 lines
xen/arch/riscv/p2m.c +6 -0
+    struct page_info *page;
+
+    mfn_t mfn = p2m_lookup(d, gfn, &p2mt);
+
+    if ( t )
+        *t = p2mt;

What's wrong with passing t directly to p2m_lookup()?

It was needed before when the code of p2m_get_page_from_gfn() looked like:
struct page_info *p2m_get_page_from_gfn(struct domain *d, gfn_t gfn,

                                          p2m_type_t *t)
  {
      struct page_info *page;
      p2m_type_t p2mt;
      mfn_t mfn = p2m_lookup(d, gfn, &p2mt);
  
      if ( t )
          *t = p2mt;
  
      if ( !p2m_is_any_ram(p2mt) )
          return NULL;

So it was needed to make sure that p2m_is_any_ram(*t) doesn't try to dereference a NULL pointer.

Even with the current one implementation the similar issue could be with if use *t instead of p2mt:
struct page_info *p2m_get_page_from_gfn(struct p2m_domain *p2m, gfn_t gfn,

                                          p2m_type_t *t)
  {
      ...
      if ( p2m_is_foreign(p2mt) )
      {
          struct domain *fdom = page_get_owner_and_reference(page);

And the second reason it is because of p2m_get_entry() (which is used inside p2m_lookup()) could return for t a pointer to local variable:
static mfn_t p2m_get_entry(struct p2m_domain *p2m, gfn_t gfn,
p2m_type_t *t,
unsigned int *page_order,
bool *valid)

  {
      ...
      p2m_type_t p2mt;
      ...
      /* Allow t to be NULL */
      t = t ?: &p2mt;
      ...

What looks wrong. I will remove this part of the code and pass t directly to p2m_lookup().

And after p2m_lookup() call will just check if t argument is NULL then init it with p2mt:
struct page_info *p2m_get_page_from_gfn(struct p2m_domain *p2m, gfn_t gfn,

                                           p2m_type_t *t)
   {
       struct page_info *page;
       p2m_type_t p2mt = p2m_invalid;
       mfn_t mfn = p2m_lookup(p2m, gfn, t);
   
       if ( !mfn_valid(mfn) )
           return NULL;
   
       if ( !t )
           p2mt = *t;
   
       ...

}

Thanks.

~ Oleksii

#160 lore raw

Re: [PATCH v2 17/17] xen/riscv: add support of page lookup by GFN

Quoted 119 lines · Oleksii Kurochko
Quoted 38 lines · Jan Beulich

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1063

Quoted 12 lines · Oleksii Kurochko
xen/arch/riscv/p2m.c +6 -0
@@ -1055,3 +1055,134 @@ int guest_physmap_add_entry(struct domain *d,
  {
      return p2m_insert_mapping(d, gfn, (1 << page_order), mfn, t);
  }
+
+/*
+ * Get the details of a given gfn.
+ *
+ * If the entry is present, the associated MFN will be returned and the
+ * access and type filled up. The page_order will correspond to the

You removed p2m_access_t * from the parameters; you need to also update the comment then accordingly.

Quoted 9 lines
xen/arch/riscv/p2m.c +9 -0
+ * order of the mapping in the page table (i.e it could be a superpage).
+ *
+ * If the entry is not present, INVALID_MFN will be returned and the
+ * page_order will be set according to the order of the invalid range.
+ *
+ * valid will contain the value of bit[0] (e.g valid bit) of the
+ * entry.
+ */
+static mfn_t p2m_get_entry(struct p2m_domain *p2m, gfn_t gfn,

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1082

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+                           p2m_type_t *t,
+                           unsigned int *page_order,
+                           bool *valid)
+{
+    paddr_t addr = gfn_to_gaddr(gfn);
+    unsigned int level = 0;
+    pte_t entry, *table;
+    int rc;
+    mfn_t mfn = INVALID_MFN;
+    p2m_type_t _t;

Please no local variables with leading underscores. In x86 we commonly name such variables p2mt.

Quoted 2 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1083

Quoted 1 line
xen/arch/riscv/p2m.c +1 -0
+    DECLARE_OFFSETS(offsets, addr);

This is the sole use of "addr". Is such a local variable really worth having?

Not really, I'll drop it.

Quoted 36 lines
Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    ASSERT(p2m_is_locked(p2m));
+    BUILD_BUG_ON(XEN_PT_LEVEL_MAP_MASK(0) != PAGE_MASK);
+
+    /* Allow t to be NULL */
+    t = t ?: &_t;
+
+    *t = p2m_invalid;

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1101

Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+
+    if ( valid )
+        *valid = false;
+
+    /* XXX: Check if the mapping is lower than the mapped gfn */
+
+    /* This gfn is higher than the highest the p2m map currently holds */
+    if ( gfn_x(gfn) > gfn_x(p2m->max_mapped_gfn) )
+    {
+        for ( level = P2M_ROOT_LEVEL; level ; level-- )

Nit: Stray blank before the 2nd semicolon. (Again at least once below.)

Quoted 17 lines
xen/arch/riscv/p2m.c +17 -0
+            if ( (gfn_x(gfn) & (XEN_PT_LEVEL_MASK(level) >> PAGE_SHIFT)) >
+                 gfn_x(p2m->max_mapped_gfn) )
+                break;
+
+        goto out;
+    }
+
+    table = p2m_get_root_pointer(p2m, gfn);
+
+    /*
+     * the table should always be non-NULL because the gfn is below
+     * p2m->max_mapped_gfn and the root table pages are always present.
+     */
+    if ( !table )
+    {
+        ASSERT_UNREACHABLE();
+        level = P2M_ROOT_LEVEL;
Quoted 8 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1125

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+        goto out;
+    }
+
+    for ( level = P2M_ROOT_LEVEL; level ; level-- )
+    {
+        rc = p2m_next_level(p2m, true, level, &table, offsets[level]);
+        if ( (rc == GUEST_TABLE_MAP_NONE) && (rc != GUEST_TABLE_MAP_NOMEM) )

This condition looks odd. As written the rhs of the && is redundant.

And it is wrong. It should:
if ( (rc == P2M_TABLE_MAP_NONE) || (rc == P2M_TABLE_MAP_NOMEM) )

Quoted 4 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1127

Quoted 2 lines
xen/arch/riscv/p2m.c +2 -0
+            goto out_unmap;
+        else if ( rc != GUEST_TABLE_NORMAL )

As before, no real need for "else" in such cases.

Quoted 10 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1135

Quoted 8 lines
xen/arch/riscv/p2m.c +8 -0
+            break;
+    }
+
+    entry = table[offsets[level]];
+
+    if ( p2me_is_valid(p2m, entry) )
+    {
+        *t = p2m_type_radix_get(p2m, entry);

If the incoming argument is NULL, the somewhat expensive radix tree lookup is unnecessary here.

Good point.

Quoted 10 lines
Quoted 10 lines
xen/arch/riscv/p2m.c +10 -0
+        mfn = pte_get_mfn(entry);
+        /*
+         * The entry may point to a superpage. Find the MFN associated
+         * to the GFN.
+         */
+        mfn = mfn_add(mfn,
+                      gfn_x(gfn) & (BIT(XEN_PT_LEVEL_ORDER(level), UL) - 1));
+
+        if ( valid )
+            *valid = pte_is_valid(entry);
Quoted 8 lines
Quoted 3 lines

Interesting. Why not the P2M counterpart of the function? Yes, the comment ahead of the function says so, but I don't see why the valid bit suddenly is relevant here (besides the P2M type).

This valid variable is expected to be used in the caller (something what Arm does here
https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/p2m.c#L375) to check if it is needed to do flush_page_to_ram(), so if the the valid bit in PTE was set to 0 then it means nothing should be flushed as entry wasn't used as it marked invalid.

I hope you see what a mess you get if you have two flavors of "valid" for a PTE.

Quoted 18 lines
Quoted 8 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1152

Quoted 6 lines
xen/arch/riscv/p2m.c +6 -0
+    }
+
+out_unmap:
+    unmap_domain_page(table);
+
+out:

Nit: Style (bot labels).

Quoted 9 lines

xen/arch/riscv/p2m.c int guest_physmap_add_entry(struct domain *d, · line 1159

Quoted 7 lines
xen/arch/riscv/p2m.c +7 -0
+    if ( page_order )
+        *page_order = XEN_PT_LEVEL_ORDER(level);
+
+    return mfn;
+}
+
+static mfn_t p2m_lookup(struct domain *d, gfn_t gfn, p2m_type_t *t)

pointer-to-const for the 1st arg? But again more likely struct p2m_domain * anyway?

Quoted 2 lines

|struct p2_domain| would be better, but I’m not really sure that a pointer-to-const can be used here.

Note how I also didn't suggest const there.

Quoted 3 lines

I expect that, in that case,

|p2m_read_lock()| would also need to accept a pointer-to-const, and since it
calls|read_lock()| internally, that could be a problem because|read_lock() |accepts a|rwlock_t *l|.

Correct.

Jan