Balloon down memory to achive enough DMA32 memory for PV guests with PCI pass-through to succesfully launch.
PATCH 1/1 Balloon down memory to achive enough DMA32 memory for PV guests with PCI pass-through to succesfully launch.
# HG changeset patch
# User [email protected]
# Date 1258150318 18000
# Node ID 82762bc10aa5a193173d8a83a5dbada1003bdcd2
# Parent 88adf22e0fe3a77d0be95530b74c3781ffc918f1
Balloon down memory to achive enough DMA32 memory for PV guests with PCI pass-through to succesfully launch.
If the user hasn't used dom0_mem= bootup parameter, the privileged domain usurps all of the memory. During launch of PV guests with PCI pass-through we ratchet down the memory for the privileged domain to the required memory for the PV guest. However, for PV guests with PCI pass-through we do not take into account that the PV guest is going to swap its SWIOTLB memory for DMA32 memory - in fact, swap 64MB of it. This patch balloon's down the privileged domain so that there are 64MB of DMA32 memory available.
Note: If 'dom0_mem' is used, the user will probably never encounter this failure.
P.S.
If you see:
about to get started...
And nothing after that, and xenctx shows
Call Trace: [<ffffffff8132cfe3>] __const_udelay+0x1e <-- [<ffffffff816b9043>] panic+0x1c0 [<ffffffff81013335>] xen_swiotlb_fixup+0x123 [<ffffffff81a05e17>] xen_swiotlb_init_with_default_size+0x9c [<ffffffff81a05f91>] xen_swiotlb_init+0x4b [<ffffffff81a0ab72>] pci_iommu_alloc+0x86 [<ffffffff81a22972>] mem_init+0x28 [<ffffffff813201a9>] sort_extable+0x39 [<ffffffff819feb90>] start_kernel+0x301 [<ffffffff819fdf76>] x86_64_start_reservations+0x101 [<ffffffff81a03cdf>] xen_start_kernel+0x715
Then this is the patch for this.
tools/python/xen/lowlevel/xc/xc.c +18 -0
@@ -1059,6 +1059,7 @@int i, j, max_cpu_id;uint64_t free_heap;PyObject *ret_obj, *node_to_cpu_obj, *node_to_memory_obj; +PyObject *node_to_dma32_mem_obj;xc_cpu_to_node_t map[MAX_CPU_ID +1 ];const char *virtcap_names[] = {"hvm" ,"hvm_directio" };
@@ -1128,10 +1129,27 @@Py_DECREF (pyint); } +xc_dom_loginit (); +/* DMA memory. */ + node_to_dma32_mem_obj =PyList_New (0 ); + +for ( i =0 ; i < info.nr_nodes; i++ ) + { +PyObject *pyint; + +xc_availheap (self->xc_handle,0 ,32 , i, &free_heap); +xc_dom_printf ("Node:%d: DMA32:%ld\n" , i, free_heap); + pyint =PyInt_FromLong (free_heap /1024 ); +PyList_Append (node_to_dma32_mem_obj, pyint); +Py_DECREF (pyint); + } +PyDict_SetItemString (ret_obj,"node_to_cpu" , node_to_cpu_obj);Py_DECREF (node_to_cpu_obj);PyDict_SetItemString (ret_obj,"node_to_memory" , node_to_memory_obj);Py_DECREF (node_to_memory_obj); +PyDict_SetItemString (ret_obj,"node_to_dma32_mem" , node_to_dma32_mem_obj); +Py_DECREF (node_to_dma32_mem_obj);return ret_obj;#undef MAX_CPU_ID
tools/python/xen/xend/XendConfig.py +7 -0
@@ -2111,6 +2111,13 @@ def is_hap(self): return self['platform'].get('hap', 0) + def is_pv_and_has_pci(self): + for dev_type, dev_info in self.all_devices_sxpr(): + if dev_type != 'pci': + continue + return not self.is_hvm() + return False + def update_platform_pci(self): pci = [] for dev_type, dev_info in self.all_devices_sxpr():
tools/python/xen/xend/XendDomainInfo.py +54 -2
86 lines not on this page · raw message
tools/python/xen/xend/XendNode.py +8 -4
@@ -872,11 +872,11 @@ except: str='none\n' return str[:-1]; - def format_node_to_memory(self, pinfo): + def format_node_to_memory(self, pinfo, key): str='' whitespace='' try: - node_to_memory=pinfo['node_to_memory'] + node_to_memory=pinfo[key] for i in range(0, pinfo['nr_nodes']): str+='%snode%d:%d\n' % (whitespace, i,
@@ -896,7 +896,10 @@ info['total_memory'] = info['total_memory'] / 1024 info['free_memory'] = info['free_memory'] / 1024 info['node_to_cpu'] = self.format_node_to_cpu(info) - info['node_to_memory'] = self.format_node_to_memory(info) + info['node_to_memory'] = self.format_node_to_memory(info, + 'node_to_memory') + info['node_to_dma32_mem'] = self.format_node_to_memory(info, + 'node_to_dma32_mem') ITEM_ORDER = ['nr_cpus', 'nr_nodes',
@@ -908,7 +911,8 @@ 'total_memory', 'free_memory', 'node_to_cpu', - 'node_to_memory' + 'node_to_memory', + 'node_to_dma32_mem' ] return [[k, info[k]] for k in ITEM_ORDER]
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Konrad Rzeszutek Wilk <[email protected]> 13.11.09 23:16 >>>
Balloon down memory to achive enough DMA32 memory for PV guests with PCI pass-through to succesfully launch.
If the user hasn't used dom0_mem= bootup parameter, the privileged domain usurps all of the memory. During launch of PV guests with PCI pass-through we ratchet down the memory for the privileged domain to the required memory for the PV guest. However, for PV guests with PCI pass-through we do not take into account that the PV guest is going to swap its SWIOTLB memory for DMA32 memory - in fact, swap 64MB of it. This patch balloon's down the privileged domain so that there are 64MB of DMA32 memory available.
While I realize that the patch was already committed, I still wanted to raise my concerns with it:
For one, it is logically (just for a much smaller total amount and for a more narrow memory range) identical to what would be needed for 32-bit-pv DomU-s on 64-bit hv, so *if* this patch is considered conceptually valid, then it should be abstracted and used for both purposes.
I think, however, that it is conceptually wrong, because it may mean that all of the memory possibly removable from Dom0 can get ballooned out without in fact yielding the memory needed by the to be started DomU.
Besides that, hard-coding the value to 64Mb doesn't seem very nice either (while I realize that both 2.6.18 and pv-ops default to 64Mb, I do not think this is really appropriate, especially given that in the 2.6.18 tree Dom0 can get run with as little as 2Mb, and I highly doubt that the demand of a DomU can by default be assumed to exceed that of Dom0), and in particular doesn't help with the case where one really has to use a larger than the default size swiotlb.
Jan
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Konrad Rzeszutek Wilk <[email protected]> 13.11.09 23:16 >>>
Balloon down memory to achive enough DMA32 memory for PV guests with PCI pass-through to succesfully launch.
If the user hasn't used dom0_mem= bootup parameter, the privileged domain usurps all of the memory. During launch of PV guests with PCI pass-through we ratchet down the memory for the privileged domain to the required memory for the PV guest. However, for PV guests with PCI pass-through we do not take into account that the PV guest is going to swap its SWIOTLB memory for DMA32 memory - in fact, swap 64MB of it. This patch balloon's down the privileged domain so that there are 64MB of DMA32 memory available.
Quoted 2 lines
While I realize that the patch was already committed, I still wanted to raise my concerns with it:
Thank you for raising them.
Quoted 4 lines
For one, it is logically (just for a much smaller total amount and for a more narrow memory range) identical to what would be needed for 32-bit-pv DomU-s on 64-bit hv, so *if* this patch is considered conceptually valid,
Meaning if you want to run a 8GB 32-bit PV, do the same. But if the PV is say, using 512MB, there is no need to allocate 64MB?
Quoted 5 lines
then it should be abstracted and used for both purposes.
I think, however, that it is conceptually wrong, because it may mean that all of the memory possibly removable from Dom0 can get ballooned out without in fact yielding the memory needed by the to be started DomU.
There is a limit at which it stops. Perhaps I should add a failsafe wherein if we don't get enough of the memory, we give it back to Dom0?
Quoted 8 lines
Besides that, hard-coding the value to 64Mb doesn't seem very nice either (while I realize that both 2.6.18 and pv-ops default to 64Mb, I do not think this is really appropriate, especially given that in the 2.6.18 tree Dom0 can get run with as little as 2Mb, and I highly doubt that the demand of a DomU can by default be assumed to exceed that of Dom0), and in particular doesn't help with the case where one really has to use a larger than the default size swiotlb.
Sure. But the user will get a notice in the log pointing them to the fact that we could not get enough memory. Maybe I should expand it some more and say something along these lines: "You best bet is to use dom0_mem=2GB. We've tried to deflate the amount of memory the privileged domain is using, but we fear to go any lower. Your guest might not start."
Since the SWIOTLB size is determined by the 'swiotlb' argument passed to the guest, what if we scanned for that and if it has a number, calculate how much memory that amounts to and use that value? The default still being at 64MB.
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Konrad Rzeszutek Wilk <[email protected]> 16.11.09 16:04 >>>
Quoted 4 lines · Jan Beulich
For one, it is logically (just for a much smaller total amount and for a more narrow memory range) identical to what would be needed for 32-bit-pv DomU-s on 64-bit hv, so *if* this patch is considered conceptually valid,
Meaning if you want to run a 8GB 32-bit PV, do the same. But if the PV is say, using 512MB, there is no need to allocate 64MB?
No, you probably misunderstood (and I probably implied to much in my response): On a system with more than 168G, just ballooning out arbitrary memory from Dom0 in order to start a 32-bit pv DomU won't guarantee that the domain can actually start, as memory beyond the 128G boundary is unusable there for such guests.
Conceptually, ballooning out arbitrary amounts of memory to find a certain amount below 4G is identical to ballooning out more than the amount a guest needs in order to find as much as it needs below 128G.
Quoted 8 lines
Quoted 5 lines
then it should be abstracted and used for both purposes.
I think, however, that it is conceptually wrong, because it may mean that all of the memory possibly removable from Dom0 can get ballooned out without in fact yielding the memory needed by the to be started DomU.
There is a limit at which it stops. Perhaps I should add a failsafe wherein if we don't get enough of the memory, we give it back to Dom0?
That would reduce the risk for Dom0, yes, but it doesn't eliminate it (and I am of the opinion that especially as long as Dom0 is not restartable we have to avoid putting any sort of extra risk on it).
Quoted 14 lines
Quoted 7 lines
Besides that, hard-coding the value to 64Mb doesn't seem very nice either (while I realize that both 2.6.18 and pv-ops default to 64Mb, I do not think this is really appropriate, especially given that in the 2.6.18 tree Dom0 can get run with as little as 2Mb, and I highly doubt that the demand of a DomU can by default be assumed to exceed that of Dom0), and in particular doesn't help with the case where one really has to use a larger than the default size swiotlb.
Sure. But the user will get a notice in the log pointing them to the fact that we could not get enough memory. Maybe I should expand it some more and say something along these lines: "You best bet is to use dom0_mem=2GB. We've tried to deflate the amount of memory the privileged domain is using, but we fear to go any lower. Your guest might not start."
Quoted 3 lines
Since the SWIOTLB size is determined by the 'swiotlb' argument passed to the guest, what if we scanned for that and if it has a number, calculate how much memory that amounts to and use that value? The default still being at 64MB.
That might be an option, but is very Linux-centric. I think the amount should be configurable per guest if something like this is being done at all.
Jan
Quoted 5 lines
No, you probably misunderstood (and I probably implied to much in my response): On a system with more than 168G, just ballooning out arbitrary memory from Dom0 in order to start a 32-bit pv DomU won't guarantee that the domain can actually start, as memory beyond the 128G boundary is unusable there for such guests.
I see what you mean.
Quoted 4 lines
Conceptually, ballooning out arbitrary amounts of memory to find a certain amount below 4G is identical to ballooning out more than the amount a guest needs in order to find as much as it needs below 128G.
Perhaps an extra mechanism in the balloon driver in the Dom0 kernel to only give back to Xen pages below the 4GB mark? And if it fails return a failure back to Xend?
That would solve the problem of ballooning out more than needed and would eliminate the "oh-lets balloon out 2 more MBs and see if that did it".
.. snip ..
Quoted 6 lines
Quoted 3 lines
Since the SWIOTLB size is determined by the 'swiotlb' argument passed to the guest, what if we scanned for that and if it has a number, calculate how much memory that amounts to and use that value? The default still being at 64MB.
That might be an option, but is very Linux-centric. I think the amount should be configurable per guest if something like this is being done at all.
How about both? Have a 'pci_mem' argument to set the default and also check the swiotlb argument in 'extra.' If there is a 'swiotlb' argument in 'extra' use that value (computed for kilobytes of course).
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Konrad Rzeszutek Wilk <[email protected]> 16.11.09 17:38 >>>
Perhaps an extra mechanism in the balloon driver in the Dom0 kernel to only give back to Xen pages below the 4GB mark? And if it fails return a failure back to Xend?
That would solve the problem of ballooning out more than needed and would eliminate the "oh-lets balloon out 2 more MBs and see if that did it".
Unfortunately that's something you won't easily get to work, as you can't tell Linux' page allocator that you want it to try to allocate a specific (preselected by you, i.e. the balloon driver) page. But that would be necessary here, as you would have to scan the p2m table to find suitable pages...
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Since the SWIOTLB size is determined by the 'swiotlb' argument passed to the guest, what if we scanned for that and if it has a number, calculate how much memory that amounts to and use that value? The default still being at 64MB.
That might be an option, but is very Linux-centric. I think the amount should be configurable per guest if something like this is being done at all.
How about both? Have a '
pci_mem' argument to set the default and also check the swiotlb argument in 'extra.' If there is a 'swiotlb' argument in 'extra' use that value (computed for kilobytes of course).
Oh, yes, I was thinking of one augmenting the other, of course.
Jan