page_alloc: use first half of higher order chunks when halving

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This change more gracefully handles the dom0 use case and returns contiguous memory for subsequent allocations.

Cc [email protected]

Cc Keir Fraser <[email protected]>

Cc Jan Beulich <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc Tim Deegan <[email protected]>

Cc Andrew Cooper <[email protected]>

Signed-off-by Matt Rushton <[email protected]>

Signed-off-by Matt Wilson <[email protected]>

---
 xen/common/page_alloc.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

xen/common/page_alloc.c +0 -0

index 601319c..27e7f18 100644

xen/common/page_alloc.c +2 -1

+        pg2 = pg + (1 << --j);
+        PFN_ORDER(pg) = j;
         page_list_add_tail(pg, &heap(node, zone, j));
-        pg += 1 << j;
     }
 
     ASSERT(avail[node][zone] >= request);

Quoted 14 lines · Matt Wilson

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This change more gracefully handles the dom0 use case and returns contiguous memory for subsequent allocations.

Quoted 8 lines

Cc [email protected]

Cc Keir Fraser <[email protected]>

Cc Jan Beulich <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc Tim Deegan <[email protected]>

Cc Andrew Cooper <[email protected]>

Signed-off-by Matt Rushton <[email protected]>

Signed-off-by Matt Wilson <[email protected]>

How does dom0 work out that it is safe to join multiple pfns into a dma buffer without the swiotlb?

Quoted 4 lines

---
 xen/common/page_alloc.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

xen/common/page_alloc.c static struct page_info *alloc_heap_pages( · line 680

Quoted 10 lines

xen/common/page_alloc.c +1 -1

index 601319c..27e7f18 100644
@@ -677,9 +677,10 @@ static struct page_info *alloc_heap_pages(
     /* We may have to halve the chunk a number of times. */
     while ( j != order )
     {
-        PFN_ORDER(pg) = --j;
+        struct page_info *pg2;

At the very least, Xen style mandates a blank line after this variable declaration.

~Andrew

Quoted 7 lines

xen/common/page_alloc.c +2 -1

+        pg2 = pg + (1 << --j);
+        PFN_ORDER(pg) = j;
         page_list_add_tail(pg, &heap(node, zone, j));
-        pg += 1 << j;
     }
 
     ASSERT(avail[node][zone] >= request);

Quoted 23 lines · Andrew Cooper

Quoted 14 lines · Matt Wilson

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This change more gracefully handles the dom0 use case and returns contiguous memory for subsequent allocations.

Quoted 8 lines

Cc [email protected]

Cc Keir Fraser <[email protected]>

Cc Jan Beulich <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc Tim Deegan <[email protected]>

Cc Andrew Cooper <[email protected]>

Signed-off-by Matt Rushton <[email protected]>

Signed-off-by Matt Wilson <[email protected]>

Quoted 2 lines

How does dom0 work out that it is safe to join multiple pfns into a dma buffer without the swiotlb?

I'm not familiar enough with how this works to say. Perhaps Matt R. can chime in during his day. My guess is that xen_swiotlb_alloc_coherent() avoids allocating a contiguous region if the pages allocated already happen to be physically contiguous.

Konrad, can you enlighten us? The setup code in question that does the remapping one page at a time is in arch/x86/xen/setup.c.

Quoted 4 lines

Quoted 4 lines

---
  xen/common/page_alloc.c |    5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)

Quoted 12 lines

xen/common/page_alloc.c static struct page_info *alloc_heap_pages( · line 680

Quoted 10 lines

xen/common/page_alloc.c +1 -1

index 601319c..27e7f18 100644
@@ -677,9 +677,10 @@ static struct page_info *alloc_heap_pages(
      /* We may have to halve the chunk a number of times. */
      while ( j != order )
      {
-        PFN_ORDER(pg) = --j;
+        struct page_info *pg2;

At the very least, Xen style mandates a blank line after this variable declaration.

Ack.

Quoted 9 lines

~Andrew

Quoted 7 lines

xen/common/page_alloc.c +2 -1

+        pg2 = pg + (1 << --j);
+        PFN_ORDER(pg) = j;
          page_list_add_tail(pg, &heap(node, zone, j));
-        pg += 1 << j;
      }
 
      ASSERT(avail[node][zone] >= request);

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This change more gracefully handles the dom0 use case and returns contiguous memory for subsequent allocations.

Cc [email protected]

Cc Keir Fraser <[email protected]>

Cc Jan Beulich <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc Tim Deegan <[email protected]>

Cc Andrew Cooper <[email protected]>

Signed-off-by Matt Rushton <[email protected]>

Signed-off-by Matt Wilson <[email protected]>

---
 xen/common/page_alloc.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

xen/common/page_alloc.c +0 -0

index 601319c..27e7f18 100644

Quoted 10 lines · Matt Wilson

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This seems like something better fixed on the dom0 side, by asking explicitly for contiguous memory in cases where it makes a difference. On the Xen side, this change seems harmless, but we might like to keep the explicitly reversed allocation on debug builds, to flush out guests that rely on their memory being contiguous.

Quoted 24 lines

This change more gracefully handles the dom0 use case and returns contiguous memory for subsequent allocations.

Cc [email protected]

Cc Keir Fraser <[email protected]>

Cc Jan Beulich <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc Tim Deegan <[email protected]>

Cc Andrew Cooper <[email protected]>

Signed-off-by Matt Rushton <[email protected]>

Signed-off-by Matt Wilson <[email protected]>

---
 xen/common/page_alloc.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

xen/common/page_alloc.c +0 -1

index 601319c..27e7f18 100644
@@ -677,9 +677,10 @@ static struct page_info *alloc_heap_pages(
     /* We may have to halve the chunk a number of times. */
     while ( j != order )
     {
-        PFN_ORDER(pg) = --j;

xen/common/page_alloc.c static struct page_info *alloc_heap_pages( · line 682

Quoted 5 lines

xen/common/page_alloc.c +3 -1

+        struct page_info *pg2;
+        pg2 = pg + (1 << --j);
+        PFN_ORDER(pg) = j;
         page_list_add_tail(pg, &heap(node, zone, j));
-        pg += 1 << j;

AFAICT this uses the low half (pg) for the allocation _and_ puts it on the freelist, and just leaks the high half (pg2). Am I missing something?

Quoted 11 lines · Tim Deegan

Quoted 10 lines · Matt Wilson

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

Quoted 5 lines

This seems like something better fixed on the dom0 side, by asking explicitly for contiguous memory in cases where it makes a difference. On the Xen side, this change seems harmless, but we might like to keep the explicitly reversed allocation on debug builds, to flush out guests that rely on their memory being contiguous.

Yes, I think that retaining the reverse allocation on debug builds is fine. I'd like Konrad's take on if it's better or possible to fix this on the Linux side.

Quoted 15 lines

Quoted 15 lines

This change more gracefully handles the dom0 use case and returns contiguous memory for subsequent allocations.

Cc [email protected]

Cc Keir Fraser <[email protected]>

Cc Jan Beulich <[email protected]>

Cc Konrad Rzeszutek Wilk <[email protected]>

Cc Tim Deegan <[email protected]>

Cc Andrew Cooper <[email protected]>

Signed-off-by Matt Rushton <[email protected]>

Signed-off-by Matt Wilson <[email protected]>

---
  xen/common/page_alloc.c |    5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)

Quoted 16 lines

Quoted 5 lines

xen/common/page_alloc.c +0 -0

index 601319c..27e7f18 100644

xen/common/page_alloc.c static struct page_info *alloc_heap_pages( · line 682

Quoted 9 lines

xen/common/page_alloc.c +3 -2

@@ -677,9 +677,10 @@ static struct page_info *alloc_heap_pages(
      /* We may have to halve the chunk a number of times. */
      while ( j != order )
      {
-        PFN_ORDER(pg) = --j;
+        struct page_info *pg2;
+        pg2 = pg + (1 << --j);
+        PFN_ORDER(pg) = j;
          page_list_add_tail(pg, &heap(node, zone, j));
-        pg += 1 << j;

AFAICT this uses the low half (pg) for the allocation _and_ puts it on the freelist, and just leaks the high half (pg2). Am I missing something?

Argh, oops. this is totally my fault (not Matt R.'s). I ported the patch out of our development tree incorrectly. The code should have read:

      while ( j != order )
      {
          struct page_info *pg2;

          pg2 = pg + (1 << --j);
          PFN_ORDER(pg2) = j;
          page_list_add_tail(pg2, &heap(node, zone, j));
      }

Apologies to Matt for my mangling of his patch (which also already had the correct blank line per Andy's comment).

--msw

Quoted 17 lines · Matt Wilson

Quoted 16 lines · Tim Deegan

Quoted 10 lines · Matt Wilson

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This seems like something better fixed on the dom0 side, by asking explicitly for contiguous memory in cases where it makes a difference. On the Xen side, this change seems harmless, but we might like to keep the explicitly reversed allocation on debug builds, to flush out guests that rely on their memory being contiguous.

Quoted 3 lines

Yes, I think that retaining the reverse allocation on debug builds is fine. I'd like Konrad's take on if it's better or possible to fix this on the Linux side.

I considered fixing it in Linux but this was a more straight forward change with no downside as far as I can tell. I see no reason in not fixing it in both places but this at least behaves more reasonably for one potential use case. I'm also interested in other opinions.

Hi,

Quoted 18 lines · Matthew Rushton

Quoted 17 lines · Matt Wilson

Quoted 16 lines · Tim Deegan

Quoted 10 lines · Matt Wilson

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This seems like something better fixed on the dom0 side, by asking explicitly for contiguous memory in cases where it makes a difference. On the Xen side, this change seems harmless, but we might like to keep the explicitly reversed allocation on debug builds, to flush out guests that rely on their memory being contiguous.

Quoted 8 lines

Quoted 3 lines

Yes, I think that retaining the reverse allocation on debug builds is fine. I'd like Konrad's take on if it's better or possible to fix this on the Linux side.

I considered fixing it in Linux but this was a more straight forward change with no downside as far as I can tell. I see no reason in not fixing it in both places but this at least behaves more reasonably for one potential use case. I'm also interested in other opinions.

Well, I'm happy enough with changing Xen (though it's common code so you'll need Keir's ack anyway rather than mine), since as you say it happens to make one use case a bit better and is otherwise harmless. But that comes with a stinking great warning:

- This is not 'fixing' anything in Xen because Xen is doing exactly what dom0 asks for in the current code; and conversely

- dom0 (and other guests) _must_not_ rely on it, whether for performance or correctness. Xen might change its page allocator at some point in the future, for any reason, and if linux perf starts sucking when that happens, that's (still) a linux bug.

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

Quoted 6 lines · Matt Wilson

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

We plan to fix problems caused by non-machine-contiguous memory by setting up the IOMMU to have 1:1 bus to pseudo-physical mappings. This would avoid using the swiotlb always[1], regardless of the machine layout of dom0 or the driver domain.

I think I would prefer this approach rather than making xen/setup.c even more horribly complicated.

Malcolm has a working prototype of this already.

David

[1] Unless you have a non-64 bit DMA capable device, but we don't care about performance with these.

Quoted 13 lines · David Vrabel

Quoted 6 lines · Matt Wilson

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

We plan to fix problems caused by non-machine-contiguous memory by setting up the IOMMU to have 1:1 bus to pseudo-physical mappings. This would avoid using the swiotlb always[1], regardless of the machine layout of dom0 or the driver domain.

Quoted 2 lines

I think I would prefer this approach rather than making xen/setup.c even more horribly complicated.

I imagine that some users will not want to run dom0 under an IOMMU.

Quoted 14 lines · Matt Wilson

Quoted 13 lines · David Vrabel

Quoted 6 lines · Matt Wilson

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

We plan to fix problems caused by non-machine-contiguous memory by setting up the IOMMU to have 1:1 bus to pseudo-physical mappings. This would avoid using the swiotlb always[1], regardless of the machine layout of dom0 or the driver domain.

Quoted 4 lines

Quoted 2 lines

I think I would prefer this approach rather than making xen/setup.c even more horribly complicated.

I imagine that some users will not want to run dom0 under an IOMMU.

Then they have chosen that (for whatever reason) over performance, right?

Quoted 15 lines · Ian Campbell

Quoted 14 lines · Matt Wilson

Quoted 13 lines · David Vrabel

Quoted 5 lines · Matt Wilson

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

We plan to fix problems caused by non-machine-contiguous memory by setting up the IOMMU to have 1:1 bus to pseudo-physical mappings. This would avoid using the swiotlb always[1], regardless of the machine layout of dom0 or the driver domain.

I think I would prefer this approach rather than making xen/setup.c even more horribly complicated.

Quoted 3 lines

Quoted 1 line

I imagine that some users will not want to run dom0 under an IOMMU.

Then they have chosen that (for whatever reason) over performance, right?

IOMMU is not free, so I imagine that some users would actually be choosing to avoid it specifically for performance reasons.

--msw

Quoted 10 lines · Matt Wilson

Quoted 5 lines · Ian Campbell

Quoted 1 line · Matt Wilson

I imagine that some users will not want to run dom0 under an IOMMU.

Then they have chosen that (for whatever reason) over performance, right?

IOMMU is not free, so I imagine that some users would actually be choosing to avoid it specifically for performance reasons.

Quoted 1 line

--msw

True, but if people are actually looking for performance, they will be turning on features like GRO and more generically scatter/gather which results in drivers using single page DMA mappings at a time, which completely bypass the bouncing in the swiotlb.

Quoted 13 lines · David Vrabel

Quoted 6 lines · Matt Wilson

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

We plan to fix problems caused by non-machine-contiguous memory by setting up the IOMMU to have 1:1 bus to pseudo-physical mappings. This would avoid using the swiotlb always[1], regardless of the machine layout of dom0 or the driver domain.

Quoted 2 lines

I think I would prefer this approach rather than making xen/setup.c even more horribly complicated.

I imagine that some users will not want to run dom0 under an IOMMU. If changing Linux Xen PV setup is (rightly) objectionable due to complexity, perhaps this small change to the hypervisor is a better short-term fix.

Quoted 1 line

Malcolm has a working prototype of this already.

Is it ready for a RFC posting?

Quoted 17 lines · Matt Wilson

Quoted 15 lines · David Vrabel

Quoted 6 lines · Matt Wilson

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

We plan to fix problems caused by non-machine-contiguous memory by setting up the IOMMU to have 1:1 bus to pseudo-physical mappings. This would avoid using the swiotlb always[1], regardless of the machine layout of dom0 or the driver domain.

I think I would prefer this approach rather than making xen/setup.c even more horribly complicated.

Quoted 4 lines

I imagine that some users will not want to run dom0 under an IOMMU. If changing Linux Xen PV setup is (rightly) objectionable due to complexity, perhaps this small change to the hypervisor is a better short-term fix.

Users who are not using the IOMMU for performance reasons but are complaining about swiotlb costs? I'm not sure that's an interesting set of users...

Quoted 30 lines · Tim Deegan

Hi,

Quoted 18 lines · Matthew Rushton

Quoted 17 lines · Matt Wilson

Quoted 16 lines · Tim Deegan

Quoted 10 lines · Matt Wilson

From: Matt Rushton <[email protected]>

This patch makes the Xen heap allocator use the first half of higher order chunks instead of the second half when breaking them down for smaller order allocations.

Linux currently remaps the memory overlapping PCI space one page at a time. Before this change this resulted in the mfns being allocated in reverse order and led to discontiguous dom0 memory. This forced dom0 to use bounce buffers for doing DMA and resulted in poor performance.

This seems like something better fixed on the dom0 side, by asking explicitly for contiguous memory in cases where it makes a difference. On the Xen side, this change seems harmless, but we might like to keep the explicitly reversed allocation on debug builds, to flush out guests that rely on their memory being contiguous.

Quoted 8 lines

Quoted 3 lines

Yes, I think that retaining the reverse allocation on debug builds is fine. I'd like Konrad's take on if it's better or possible to fix this on the Linux side.

I considered fixing it in Linux but this was a more straight forward change with no downside as far as I can tell. I see no reason in not fixing it in both places but this at least behaves more reasonably for one potential use case. I'm also interested in other opinions.

Quoted 4 lines

Well, I'm happy enough with changing Xen (though it's common code so you'll need Keir's ack anyway rather than mine), since as you say it happens to make one use case a bit better and is otherwise harmless. But that comes with a stinking great warning:

Anyone can Ack or Nack, but I wouldn't want to move forward on a change like this without Keir's Ack. :-)

Quoted 7 lines

- This is not 'fixing' anything in Xen because Xen is doing exactly what dom0 asks for in the current code; and conversely

- dom0 (and other guests) _must_not_ rely on it, whether for performance or correctness. Xen might change its page allocator at some point in the future, for any reason, and if linux perf starts sucking when that happens, that's (still) a linux bug.

I agree with both of these. This was just the "least change" patch to a particular problem we observed.

Konrad, what's the possibility of fixing this in Linux Xen PV setup code? I think it'd be a matter batching up pages and doing larger order allocations in linux/arch/x86/xen/setup.c:xen_do_chunk(), falling back to smaller pages if allocations fail due to fragmentation, etc.

--msw

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

--msw

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Quoted 2 lines

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

Quoted 6 lines

Quoted 2 lines

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

I was looking at network IO performance so the buffers would not have been that large. I think large in this context is relative to the 4k page size and the odds of the buffer spanning a page boundary. For context I saw ~5-10% performance increase with guest network throughput by avoiding bounce buffers and also saw dom0 tcp streaming performance go from ~6Gb/s to over 9Gb/s on my test setup with a 10Gb NIC.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 16 lines · Matthew Rushton

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

Quoted 7 lines

I was looking at network IO performance so the buffers would not have been that large. I think large in this context is relative to the 4k page size and the odds of the buffer spanning a page boundary. For context I saw ~5-10% performance increase with guest network throughput by avoiding bounce buffers and also saw dom0 tcp streaming performance go from ~6Gb/s to over 9Gb/s on my test setup with a 10Gb NIC.

OK, but that would not be the dma_alloc_coherent ones then? That sounds more like the generic TCP mechanism allocated 64KB pages instead of 4KB and used those.

Did you try looking at this hack that Ian proposed a long time ago to verify that it is said problem?

https://lkml.org/lkml/2013/9/4/540

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 24 lines · Konrad Rzeszutek Wilk

Quoted 16 lines · Matthew Rushton

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

Quoted 7 lines

I was looking at network IO performance so the buffers would not have been that large. I think large in this context is relative to the 4k page size and the odds of the buffer spanning a page boundary. For context I saw ~5-10% performance increase with guest network throughput by avoiding bounce buffers and also saw dom0 tcp streaming performance go from ~6Gb/s to over 9Gb/s on my test setup with a 10Gb NIC.

Quoted 9 lines

OK, but that would not be the dma_alloc_coherent ones then? That sounds more like the generic TCP mechanism allocated 64KB pages instead of 4KB and used those.

Did you try looking at this hack that Ian proposed a long time ago to verify that it is said problem?

https://lkml.org/lkml/2013/9/4/540

Yes I had seen that and intially had the same reaction but the change was relatively recent and not relevant. I *think* all the coherent allocations are ok since the swiotlb makes them contiguous. The problem comes with the use of the streaming api. As one example with jumbo frames enabled a driver might use larger rx buffers which triggers the problem.

I think the right thing to do is to make the dma streaming api work better with larger buffers on dom0. That way it works across all

drivers and device types regardless of how they were designed.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 2 lines

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 41 lines · Matthew Rushton

Quoted 24 lines · Konrad Rzeszutek Wilk

Quoted 16 lines · Matthew Rushton

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

Quoted 7 lines

I was looking at network IO performance so the buffers would not have been that large. I think large in this context is relative to the 4k page size and the odds of the buffer spanning a page boundary. For context I saw ~5-10% performance increase with guest network throughput by avoiding bounce buffers and also saw dom0 tcp streaming performance go from ~6Gb/s to over 9Gb/s on my test setup with a 10Gb NIC.

Quoted 9 lines

OK, but that would not be the dma_alloc_coherent ones then? That sounds more like the generic TCP mechanism allocated 64KB pages instead of 4KB and used those.

Did you try looking at this hack that Ian proposed a long time ago to verify that it is said problem?

https://lkml.org/lkml/2013/9/4/540

Yes I had seen that and intially had the same reaction but the change was relatively recent and not relevant. I *think* all the coherent allocations are ok since the swiotlb makes them contiguous. The problem comes with the use of the streaming api. As one example with jumbo frames enabled a driver might use larger rx buffers which triggers the problem.

Quoted 2 lines

I think the right thing to do is to make the dma streaming api work better with larger buffers on dom0. That way it works across all

OK.

Quoted 1 line

drivers and device types regardless of how they were designed.

Can you point me to an example of the DMA streaming API?

I am not sure if you mean 'streaming API' as scatter gather operations using DMA API?

Is there a particular easy way for me to reproduce this. I have to say I hadn't enabled Jumbo frame on my box since I am not even sure if the switch I have can do it. Is there a idiots-punch-list of how to reproduce this?

Thanks!

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 51 lines · Konrad Rzeszutek Wilk

Quoted 41 lines · Matthew Rushton

Quoted 24 lines · Konrad Rzeszutek Wilk

Quoted 16 lines · Matthew Rushton

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

Quoted 7 lines

I was looking at network IO performance so the buffers would not have been that large. I think large in this context is relative to the 4k page size and the odds of the buffer spanning a page boundary. For context I saw ~5-10% performance increase with guest network throughput by avoiding bounce buffers and also saw dom0 tcp streaming performance go from ~6Gb/s to over 9Gb/s on my test setup with a 10Gb NIC.

Quoted 9 lines

OK, but that would not be the dma_alloc_coherent ones then? That sounds more like the generic TCP mechanism allocated 64KB pages instead of 4KB and used those.

Did you try looking at this hack that Ian proposed a long time ago to verify that it is said problem?

https://lkml.org/lkml/2013/9/4/540

Yes I had seen that and intially had the same reaction but the change was relatively recent and not relevant. I *think* all the coherent allocations are ok since the swiotlb makes them contiguous. The problem comes with the use of the streaming api. As one example with jumbo frames enabled a driver might use larger rx buffers which triggers the problem.

Quoted 2 lines

I think the right thing to do is to make the dma streaming api work better with larger buffers on dom0. That way it works across all

OK.

Quoted 1 line

drivers and device types regardless of how they were designed.

Can you point me to an example of the DMA streaming API?

I am not sure if you mean 'streaming API' as scatter gather operations using DMA API?

Quoted 6 lines

Is there a particular easy way for me to reproduce this. I have to say I hadn't enabled Jumbo frame on my box since I am not even sure if the switch I have can do it. Is there a idiots-punch-list of how to reproduce this?

Thanks!

By streaming API I'm just referring to drivers that use dma_map_single/dma_unmap_single on every buffer instead of using coherent allocations. So not related to sg in my case. If you want an example of this you can look at the bnx2x Broadcom driver. To reproduce this at a minimum you'll need to have:

1) Enough dom0 memory so it overlaps with PCI space and gets remapped by Linux at boot

2) A driver that uses dma_map_single/dma_unmap_single

3) Large enough buffers so that they span page boundaries

Things that may help with 3 are enabling jumbos and various offload settings in either guests or dom0.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 2 lines

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 58 lines · Matthew Rushton

Quoted 51 lines · Konrad Rzeszutek Wilk

Quoted 41 lines · Matthew Rushton

Quoted 24 lines · Konrad Rzeszutek Wilk

Quoted 16 lines · Matthew Rushton

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

Quoted 7 lines

I was looking at network IO performance so the buffers would not have been that large. I think large in this context is relative to the 4k page size and the odds of the buffer spanning a page boundary. For context I saw ~5-10% performance increase with guest network throughput by avoiding bounce buffers and also saw dom0 tcp streaming performance go from ~6Gb/s to over 9Gb/s on my test setup with a 10Gb NIC.

Quoted 9 lines

OK, but that would not be the dma_alloc_coherent ones then? That sounds more like the generic TCP mechanism allocated 64KB pages instead of 4KB and used those.

Did you try looking at this hack that Ian proposed a long time ago to verify that it is said problem?

https://lkml.org/lkml/2013/9/4/540

Yes I had seen that and intially had the same reaction but the change was relatively recent and not relevant. I *think* all the coherent allocations are ok since the swiotlb makes them contiguous. The problem comes with the use of the streaming api. As one example with jumbo frames enabled a driver might use larger rx buffers which triggers the problem.

Quoted 2 lines

I think the right thing to do is to make the dma streaming api work better with larger buffers on dom0. That way it works across all

OK.

Quoted 1 line

drivers and device types regardless of how they were designed.

Can you point me to an example of the DMA streaming API?

I am not sure if you mean 'streaming API' as scatter gather operations using DMA API?

Quoted 6 lines

Is there a particular easy way for me to reproduce this. I have to say I hadn't enabled Jumbo frame on my box since I am not even sure if the switch I have can do it. Is there a idiots-punch-list of how to reproduce this?

Thanks!

Quoted 8 lines

By streaming API I'm just referring to drivers that use dma_map_single/dma_unmap_single on every buffer instead of using coherent allocations. So not related to sg in my case. If you want an example of this you can look at the bnx2x Broadcom driver. To reproduce this at a minimum you'll need to have:

1) Enough dom0 memory so it overlaps with PCI space and gets remapped by Linux at boot

Hm? Could you give a bit details? As in is the:

[    0.000000] Allocating PCI resources starting at 7f800000 (gap: 7f800000:7c800000)

value?

As in that value should be in the PCI space and I am not sure how your dom0 memory overlaps? If you do say dom0_mem=max:3G the kernel will balloon out of the MMIO regions and the gaps (so PCI space) and put that memory past the 4GB. So the MMIO regions end up being MMIO regions.

Quoted 1 line

2) A driver that uses dma_map_single/dma_unmap_single

OK,

Quoted 1 line

3) Large enough buffers so that they span page boundaries

Um, right, so I think the get_order hack that was posted would help in that so you would not span page boundaries?

Quoted 2 lines

Things that may help with 3 are enabling jumbos and various offload settings in either guests or dom0.

If you booted baremetal with 'iommu=soft swiotlb=force' the same problem should show up - at least based on the 2) and 3) issue.

Well, except that there are no guests but one should be able to trigger this.

What do you use for driving traffic? iperf with certain parameters?

Thanks!

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 71 lines · Konrad Rzeszutek Wilk

Quoted 58 lines · Matthew Rushton

Quoted 51 lines · Konrad Rzeszutek Wilk

Quoted 41 lines · Matthew Rushton

Quoted 24 lines · Konrad Rzeszutek Wilk

Quoted 16 lines · Matthew Rushton

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Matthew Rushton

Quoted 6 lines · Matt Wilson

Quoted 5 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 7 lines

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

Large size like larger than 512kB? That would also cause problems on baremetal then when swiotlb is activated I believe.

Quoted 7 lines

I was looking at network IO performance so the buffers would not have been that large. I think large in this context is relative to the 4k page size and the odds of the buffer spanning a page boundary. For context I saw ~5-10% performance increase with guest network throughput by avoiding bounce buffers and also saw dom0 tcp streaming performance go from ~6Gb/s to over 9Gb/s on my test setup with a 10Gb NIC.

Quoted 9 lines

OK, but that would not be the dma_alloc_coherent ones then? That sounds more like the generic TCP mechanism allocated 64KB pages instead of 4KB and used those.

Did you try looking at this hack that Ian proposed a long time ago to verify that it is said problem?

https://lkml.org/lkml/2013/9/4/540

Yes I had seen that and intially had the same reaction but the change was relatively recent and not relevant. I *think* all the coherent allocations are ok since the swiotlb makes them contiguous. The problem comes with the use of the streaming api. As one example with jumbo frames enabled a driver might use larger rx buffers which triggers the problem.

Quoted 2 lines

I think the right thing to do is to make the dma streaming api work better with larger buffers on dom0. That way it works across all

OK.

Quoted 1 line

drivers and device types regardless of how they were designed.

Can you point me to an example of the DMA streaming API?

I am not sure if you mean 'streaming API' as scatter gather operations using DMA API?

Quoted 6 lines

Is there a particular easy way for me to reproduce this. I have to say I hadn't enabled Jumbo frame on my box since I am not even sure if the switch I have can do it. Is there a idiots-punch-list of how to reproduce this?

Thanks!

Quoted 8 lines

By streaming API I'm just referring to drivers that use dma_map_single/dma_unmap_single on every buffer instead of using coherent allocations. So not related to sg in my case. If you want an example of this you can look at the bnx2x Broadcom driver. To reproduce this at a minimum you'll need to have:

1) Enough dom0 memory so it overlaps with PCI space and gets remapped by Linux at boot

Hm? Could you give a bit details? As in is the:

[    0.000000] Allocating PCI resources starting at 7f800000 (gap: 7f800000:7c800000)

Quoted 7 lines

value?

As in that value should be in the PCI space and I am not sure how your dom0 memory overlaps? If you do say dom0_mem=max:3G the kernel will balloon out of the MMIO regions and the gaps (so PCI space) and put that memory past the 4GB. So the MMIO regions end up being MMIO regions.

You should see the message from xen_do_chunk() about adding pages back. Something along the lines of:

Populating 380000-401fb6 pfn range: 542250 pages added

These pages get added in reverse order (mfns reversed) without my proposed Xen change.

Quoted 5 lines

Quoted 1 line

2) A driver that uses dma_map_single/dma_unmap_single

OK,

Quoted 1 line

3) Large enough buffers so that they span page boundaries

Um, right, so I think the get_order hack that was posted would help in that so you would not span page boundaries?

That patch doesn't apply in my case but in principal you're right, any change that would decrease buffers spanning page boundaries would limit bounce buffer usage.

Quoted 7 lines

Quoted 2 lines

Things that may help with 3 are enabling jumbos and various offload settings in either guests or dom0.

If you booted baremetal with 'iommu=soft swiotlb=force' the same problem should show up - at least based on the 2) and 3) issue.

Well, except that there are no guests but one should be able to trigger this.

If that forces the use of bounce buffers than it would be a similar net result if you wanted to see the performance overhead of doing the copies.

Quoted 1 line

What do you use for driving traffic? iperf with certain parameters?

I was using netperf. There weren't any magic params to trigger this. I believe with the default tcp stream test I ran into the issue.

Quoted 1 line

Quoted 1 line

Thanks!

Are there any concerns about the proposed Xen change as a reasonable work around for the current implementation? Thank you!

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 2 lines

--msw

It's the dma streaming api I've noticed the problem with, so dma_map_single(). Applicable swiotlb code would be xen_swiotlb_map_page() and range_straddles_page_boundary(). So yes for larger buffers it can cause bouncing.

Quoted 2 lines · Matthew Rushton

Are there any concerns about the proposed Xen change as a reasonable work around for the current implementation? Thank you!

So I finally understood what the concern was about it - the balloon mechanics get the pages in worst possible order. I am wondeirng if there is something on the Linux side we can do to tell Xen to give them to use in the proper order?

Could we swap the order of xen_do_chunk so it starts from the end and goes to start? Would that help?

Quoted 7 lines · Konrad Rzeszutek Wilk

Quoted 2 lines · Matthew Rushton

Are there any concerns about the proposed Xen change as a reasonable work around for the current implementation? Thank you!

So I finally understood what the concern was about it - the balloon mechanics get the pages in worst possible order. I am wondeirng if there is something on the Linux side we can do to tell Xen to give them to use in the proper order?

The best way, at least from Xen's point of view, is to explicitly allocate contiguous pages in the cases where it'll make a difference AIUI linux already does this for some classes of dma-able memory.

Quoted 2 lines

Could we swap the order of xen_do_chunk so it starts from the end and goes to start? Would that help?

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Cheers,

Tim.

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

Quoted 8 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Quoted 4 lines

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

Quoted 1 line

Quoted 1 line · On 02.04.14 at 12:06, <[email protected]> wrote:

Quoted 13 lines · Jan Beulich

Quoted 12 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

Quoted 2 lines

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

Does it even actually solve the problem? It seems like it is just deferring it until sufficient fragmentation has occurred in the system. All its really done is make the eventual issue much harder to debug.

Quoted 18 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 12:06, <[email protected]> wrote:

Quoted 15 lines · Jan Beulich

Quoted 8 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Quoted 4 lines

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

Quoted 3 lines

Does it even actually solve the problem? It seems like it is just deferring it until sufficient fragmentation has occurred in the system. All its really done is make the eventual issue much harder to debug.

Wasn't this largely for Dom0 (in which case fragmentation shouldn't matter yet)?

Quoted 9 lines · Jan Beulich

Quoted 9 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 12:06, <[email protected]> wrote:

Quoted 7 lines · Jan Beulich

Quoted 7 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

Quoted 5 lines

Quoted 3 lines

Does it even actually solve the problem? It seems like it is just deferring it until sufficient fragmentation has occurred in the system. All its really done is make the eventual issue much harder to debug.

Wasn't this largely for Dom0 (in which case fragmentation shouldn't matter yet)?

Dom0 ballooning breaks any assumptions you might make about relying on early allocations.

Ian.

Quoted 1 line

Quoted 1 line · On 10.04.14 at 00:21, <[email protected]> wrote:

Quoted 2 lines · Ian Campbell

Dom0 ballooning breaks any assumptions you might make about relying on early allocations.

I think you're missing the point. I'm not arguing that this change is a general purpose solution to guarantee that dom0 is contiguous. Fragmentation can exist even if dom0 asks for larger allocations like it should (which the balloon driver does I believe). What the change does do is solve a real problem in the current Linux PCI remapping implementation which happens during dom0 intialization. If the allocation strategy is arbitrary why not make the proposed hypervisor change to make existing Linux implementations behave better and in addition fix the problem in Linux so moving forward things are safe?

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 10.04.14 at 00:21, <[email protected]> wrote:

Quoted 2 lines · Ian Campbell

Dom0 ballooning breaks any assumptions you might make about relying on early allocations.

Quoted 9 lines

I think you're missing the point. I'm not arguing that this change is a general purpose solution to guarantee that dom0 is contiguous. Fragmentation can exist even if dom0 asks for larger allocations like it should (which the balloon driver does I believe). What the change does do is solve a real problem in the current Linux PCI remapping implementation which happens during dom0 intialization. If the allocation strategy is arbitrary why not make the proposed hypervisor change to make existing Linux implementations behave better and in addition fix the problem in Linux so moving forward things are safe?

Apart from all other arguments speaking against this, did you consider that altering the hypervisor behavior may adversely affect some other Dom0-capable OS?

Problems in Linux should, as said before, get fixed in Linux. If older versions are affected, stable backports should subsequently be requested/done.

Jan

Quoted 10 lines · Ian Campbell

Quoted 9 lines · Jan Beulich

Quoted 9 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 12:06, <[email protected]> wrote:

Quoted 7 lines · Jan Beulich

Quoted 7 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

OK understood see below.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Does it even actually solve the problem? It seems like it is just deferring it until sufficient fragmentation has occurred in the system. All its really done is make the eventual issue much harder to debug.

Wasn't this largely for Dom0 (in which case fragmentation shouldn't matter yet)?

Dom0 ballooning breaks any assumptions you might make about relying on early allocations.

I think you're missing the point. I'm not arguing that this change is a general purpose solution to guarantee that dom0 is contiguous. Fragmentation can exist even if dom0 asks for larger allocations like it should (which the balloon driver does I believe). What the change does do is solve a real problem in the current Linux PCI remapping implementation which happens during dom0 intialization. If the allocation strategy is arbitrary why not make the proposed hypervisor change to make existing Linux implementations behave better and in addition fix the problem in Linux so moving forward things are safe?

Quoted 3 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Quoted 26 lines · Matthew Rushton

Quoted 10 lines · Ian Campbell

Quoted 9 lines · Jan Beulich

Quoted 9 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 12:06, <[email protected]> wrote:

Quoted 7 lines · Jan Beulich

Quoted 7 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

OK understood see below.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Does it even actually solve the problem? It seems like it is just deferring it until sufficient fragmentation has occurred in the system. All its really done is make the eventual issue much harder to debug.

Wasn't this largely for Dom0 (in which case fragmentation shouldn't matter yet)?

Dom0 ballooning breaks any assumptions you might make about relying on early allocations.

Quoted 10 lines

I think you're missing the point. I'm not arguing that this change is a general purpose solution to guarantee that dom0 is contiguous. Fragmentation can exist even if dom0 asks for larger allocations like it should (which the balloon driver does I believe). What the change does do is solve a real problem in the current Linux PCI remapping implementation which happens during dom0 intialization. If the allocation strategy is arbitrary why not make the proposed hypervisor change to make existing Linux implementations behave better and in addition fix the problem in Linux so moving forward things are safe?

I think Tim was OK with that - as long as it was based on a flag - meaning when we do the increase_reservation call we use an extra flag to ask for contingous PFNs.

Quoted 7 lines

Quoted 2 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Quoted 37 lines · Konrad Rzeszutek Wilk

Quoted 36 lines · Matthew Rushton

Quoted 10 lines · Ian Campbell

Quoted 9 lines · Jan Beulich

Quoted 9 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 12:06, <[email protected]> wrote:

Quoted 7 lines · Jan Beulich

Quoted 7 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

OK understood see below.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Does it even actually solve the problem? It seems like it is just deferring it until sufficient fragmentation has occurred in the system. All its really done is make the eventual issue much harder to debug.

Wasn't this largely for Dom0 (in which case fragmentation shouldn't matter yet)?

Dom0 ballooning breaks any assumptions you might make about relying on early allocations.

I think you're missing the point. I'm not arguing that this change is a general purpose solution to guarantee that dom0 is contiguous. Fragmentation can exist even if dom0 asks for larger allocations like it should (which the balloon driver does I believe). What the change does do is solve a real problem in the current Linux PCI remapping implementation which happens during dom0 intialization. If the allocation strategy is arbitrary why not make the proposed hypervisor change to make existing Linux implementations behave better and in addition fix the problem in Linux so moving forward things are safe?

Quoted 3 lines

I think Tim was OK with that - as long as it was based on a flag - meaning when we do the increase_reservation call we use an extra flag to ask for contingous PFNs.

OK the extra flag feels a little dirty to me but it would solve the problem. What are your thoughts on changing Linux to make higher order allocations or more minimally adding a boot parameter to not remap the memory at all for those that care about performance? I

know the Linux code is already fairly complex and your preference was not to make it worse.

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Quoted 28 lines · Matthew Rushton

Quoted 10 lines · Ian Campbell

Quoted 9 lines · Jan Beulich

Quoted 9 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 12:06, <[email protected]> wrote:

Quoted 7 lines · Jan Beulich

Quoted 7 lines

Quoted 1 line

Quoted 1 line · On 02.04.14 at 02:17, <[email protected]> wrote:

Quoted 5 lines · Tim Deegan

As long as we don't also change the default allocation order in Xen. :) In general, linux shouldn't rely on the order that Xen allocates memory, as that might change later. If the current API can't do what's needed, maybe we can add another allocator hypercall or flag?

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

Agree on not relying on the order in the long run. A new hypercall or flag seems like overkill right now. The question for me comes down to my proposed change which is more simple and solves the short term problem or investing time in reworking the Linux code to make large allocations.

I think it has become pretty clear by now that we'd rather not alter the hypervisor allocator for a purpose like this.

OK understood see below.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Does it even actually solve the problem? It seems like it is just deferring it until sufficient fragmentation has occurred in the system. All its really done is make the eventual issue much harder to debug.

Wasn't this largely for Dom0 (in which case fragmentation shouldn't matter yet)?

Dom0 ballooning breaks any assumptions you might make about relying on early allocations.

Quoted 10 lines

I think you're missing the point. I'm not arguing that this change is a general purpose solution to guarantee that dom0 is contiguous. Fragmentation can exist even if dom0 asks for larger allocations like it should (which the balloon driver does I believe). What the change does do is solve a real problem in the current Linux PCI remapping implementation which happens during dom0 intialization. If the allocation strategy is arbitrary why not make the proposed hypervisor change to make existing Linux implementations behave better and in addition fix the problem in Linux so moving forward things are safe?

I think Tim was OK with that - as long as it was based on a flag - meaning when we do the increase_reservation call we use an extra flag to ask for contingous PFNs.

Quoted 1 line

Quoted 1 line · On 13.04.14 at 23:32, <[email protected]> wrote:

(c) If for some reason the current hypercall API is not sufficient for dom0 to get what it wants, we should consider adding some new operation/flag/mode somewhere. But since AFAIK there's already another path in linux that allocates contiguous DMA buffers for device drivers, presumably this isn't the case.

Quoted 6 lines

Quoted 1 line

Quoted 1 line · On 13.04.14 at 23:32, <[email protected]> wrote:

(c) If for some reason the current hypercall API is not sufficient for dom0 to get what it wants, we should consider adding some new operation/flag/mode somewhere. But since AFAIK there's already another path in linux that allocates contiguous DMA buffers for device drivers, presumably this isn't the case.

And it should be kept in mind that requesting contiguous memory shouldn't be done at will, as it may end up exhausting the portion of memory intended for DMA-style allocations (SWIOTLB / DMA- coherent allocations in Linux). I.e. neither Dom0 nor DomU should be trying to populate large parts of their memory with contiguous allocation requests to the hypervisor. They may, if they so desire, go and re-arrange their P2M mapping (solely based on what they got handed by doing order-0 allocations).

Jan

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

Jan

Quoted 7 lines · Jan Beulich

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 7 lines

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

OK.

Matt & Matthew - my plate is quite filled and I fear that in the next three weeks there is not going to be much time to code up a prototype.

Would either one of you be willing to take a crack at this? It would be neat as we could remove a lot of the balloon increase/decrease code in arch/x86/xen/setup.c.

Thanks!

Quoted 2 lines

Jan

Quoted 7 lines · Jan Beulich

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 7 lines

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

OK.

Matt & Matthew - my plate is quite filled and I fear that in the next three weeks there is not going to be much time to code up a prototype.

Would either one of you be willing to take a crack at this? It would be neat as we could remove a lot of the balloon increase/decrease code in arch/x86/xen/setup.c.

Thanks!

Quoted 2 lines

Jan

Quoted 20 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Jan Beulich

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 7 lines

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

OK.

Matt & Matthew - my plate is quite filled and I fear that in the next three weeks there is not going to be much time to code up a prototype.

Quoted 5 lines

Would either one of you be willing to take a crack at this? It would be neat as we could remove a lot of the balloon increase/decrease code in arch/x86/xen/setup.c.

Thanks!

I have a first pass at this. Just need to test it and should have something ready sometime next week or so.

Quoted 3 lines

Quoted 3 lines

Jan

Quoted 20 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Jan Beulich

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 7 lines

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

OK.

Matt & Matthew - my plate is quite filled and I fear that in the next three weeks there is not going to be much time to code up a prototype.

Quoted 5 lines

Would either one of you be willing to take a crack at this? It would be neat as we could remove a lot of the balloon increase/decrease code in arch/x86/xen/setup.c.

Thanks!

I have a first pass at this. Just need to test it and should have something ready sometime next week or so.

Quoted 3 lines

Quoted 3 lines

Jan

Quoted 21 lines · Matthew Rushton

Quoted 20 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Jan Beulich

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 7 lines

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

OK.

Matt & Matthew - my plate is quite filled and I fear that in the next three weeks there is not going to be much time to code up a prototype.

Quoted 7 lines

Quoted 5 lines

Would either one of you be willing to take a crack at this? It would be neat as we could remove a lot of the balloon increase/decrease code in arch/x86/xen/setup.c.

Thanks!

I have a first pass at this. Just need to test it and should have something ready sometime next week or so.

Daniel pointed me to this commit:
ommit 2e2fb75475c2fc74c98100f1468c8195fee49f3b

Author: Konrad Rzeszutek Wilk <[email protected]>
Date:   Fri Apr 6 10:07:11 2012 -0400

xen/setup: Populate freed MFNs from non-RAM E820 entries and gaps to E820 RAM
..
The other solution (that did not work) was to transplant the MFN in the P2M tree - the ones that were going to be freed were put in the E820_RAM regions past the nr_pages. But the modifications to the M2P array (the other side of creating PTEs) were not carried away. As the hypervisor is the only one capable of modifying that and the only two hypercalls that would do this are: the update_va_mapping (which won't work, as during initial bootup only PFNs up to nr_pages are mapped in the guest) or via the populate hypercall.

Where I talk about the 'update_va_mapping' - and I seem to think that it would not work (due to the nr_pages limit). I don't actually remember the details - so I might have been incorrect (hopefully!?).

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Jan

Quoted 45 lines · Konrad Rzeszutek Wilk

Quoted 21 lines · Matthew Rushton

Quoted 20 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Jan Beulich

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 7 lines

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

OK.

Matt & Matthew - my plate is quite filled and I fear that in the next three weeks there is not going to be much time to code up a prototype.

Quoted 7 lines

Quoted 5 lines

Would either one of you be willing to take a crack at this? It would be neat as we could remove a lot of the balloon increase/decrease code in arch/x86/xen/setup.c.

Thanks!

I have a first pass at this. Just need to test it and should have something ready sometime next week or so.

Daniel pointed me to this commit:
ommit 2e2fb75475c2fc74c98100f1468c8195fee49f3b

Author: Konrad Rzeszutek Wilk <[email protected]>
Date:   Fri Apr 6 10:07:11 2012 -0400

xen/setup: Populate freed MFNs from non-RAM E820 entries and gaps to E820 RAM
..
The other solution (that did not work) was to transplant the MFN in the P2M tree - the ones that were going to be freed were put in the E820_RAM regions past the nr_pages. But the modifications to the M2P array (the other side of creating PTEs) were not carried away. As the hypervisor is the only one capable of modifying that and the only two hypercalls that would do this are: the update_va_mapping (which won't work, as during initial bootup only PFNs up to nr_pages are mapped in the guest) or via the populate hypercall.

Quoted 4 lines

Where I talk about the 'update_va_mapping' - and I seem to think that it would not work (due to the nr_pages limit). I don't actually remember the details - so I might have been incorrect (hopefully!?).

Ok I finally have something I'm happy with using the mmu_update hypercall and placing things in the existing E820 map. I don't think the update_va_mapping hypercall is necessary. It ended up being a little more complicated than I originally thought to handle not allocating additional p2m leaf nodes. I'm going on vacation here shortly and can post it when I get back.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Jan

Quoted 50 lines · Matthew Rushton

Quoted 45 lines · Konrad Rzeszutek Wilk

Quoted 21 lines · Matthew Rushton

Quoted 20 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Jan Beulich

Quoted 4 lines

Quoted 1 line

Quoted 1 line · On 14.04.14 at 16:40, <[email protected]> wrote:

That was OK, but the M2P lookup table was not too thrilled with this. Perhaps I should have used another hypercall to re-arrange the M2P? I think I did try 'XENMEM_exchange' but that is not the right call either.

Yeah, that's allocating new pages in exchange for your old ones. Not really what you want.

Quoted 7 lines

Quoted 2 lines

Perhaps I should use XENMEM_remove_from_physmap/XENMEM_add_to_physmap combo ?

A pair of MMU_MACHPHYS_UPDATE operations would seem to be the right way of doing this (along with respective kernel internal accounting like set_phys_to_machine(), and perhaps a pair of update_va_mapping operations if the 1:1 map is already in place at that time, and you care about which page contents appears at which virtual address).

OK.

Matt & Matthew - my plate is quite filled and I fear that in the next three weeks there is not going to be much time to code up a prototype.

Quoted 7 lines

Quoted 5 lines

Would either one of you be willing to take a crack at this? It would be neat as we could remove a lot of the balloon increase/decrease code in arch/x86/xen/setup.c.

Thanks!

I have a first pass at this. Just need to test it and should have something ready sometime next week or so.

Daniel pointed me to this commit:
ommit 2e2fb75475c2fc74c98100f1468c8195fee49f3b

Author: Konrad Rzeszutek Wilk <[email protected]>
Date:   Fri Apr 6 10:07:11 2012 -0400

xen/setup: Populate freed MFNs from non-RAM E820 entries and gaps to E820 RAM
..
The other solution (that did not work) was to transplant the MFN in the P2M tree - the ones that were going to be freed were put in the E820_RAM regions past the nr_pages. But the modifications to the M2P array (the other side of creating PTEs) were not carried away. As the hypervisor is the only one capable of modifying that and the only two hypercalls that would do this are: the update_va_mapping (which won't work, as during initial bootup only PFNs up to nr_pages are mapped in the guest) or via the populate hypercall.

Quoted 4 lines

Where I talk about the 'update_va_mapping' - and I seem to think that it would not work (due to the nr_pages limit). I don't actually remember the details - so I might have been incorrect (hopefully!?).

Quoted 6 lines

Ok I finally have something I'm happy with using the mmu_update hypercall and placing things in the existing E820 map. I don't think the update_va_mapping hypercall is necessary. It ended up being a little more complicated than I originally thought to handle not allocating additional p2m leaf nodes. I'm going on vacation here shortly and can post it when I get back.

Enjoy the vacation and I am looking forward to seeing the patches when you come back!

Thank you!

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Jan

Sent patch to lkml late last week ([PATCH] xen/setup: Remap Xen Identity Mapped RAM). Will cc xen-devel on further correspondance.

Quoted 6 lines · Konrad Rzeszutek Wilk

Could you elaborate a bit more on the use-case please?
My understanding is that most drivers use a scatter gather list - in which case it does not matter if the underlaying MFNs in the PFNs spare are not contingous.

Quoted 3 lines

But I presume the issue you are hitting is with drivers doing dma_map_page and the page is not 4KB but rather large (compound page). Is that the problem you have observed?

Drivers are using very large size arguments to dma_alloc_coherent() for things like RX and TX descriptor rings.

--msw