swiotlb detection should be memory hotplug aware ?

DISCUSSION Alok Kataria 5 participants 20 messages Mar 13, 2010 – Jul 28, 2010 ACPI

Share & tools

Alok Kataria

author

2010-03-13 02:07 UTC #0 lore raw

Hi,

Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.

<3>[ 815.921504] nommu_map_sg: overflow 32ffd6000+4096 of device mask ffffffff
<3>[ 815.944860] nommu_map_sg: overflow 32ffd6000+4096 of device mask ffffffff
<3>[ 815.968808] nommu_map_sg: overflow 32ffd6000+4096 of device mask ffffffff
<3>[ 815.992821] nommu_map_sg: overflow 32ffd6000+4096 of device mask ffffffff
<3>[ 816.016796] nommu_map_sg: overflow 32ffd6000+4096 of device mask ffffffff

For systems which have no HW-IOMMU but are capable of memory hotadd this can be a potential problem. IMO, there can be few possible solutions to this.

1. Enable swiotlb for all 64bit kernels which have memory hot-add support.
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?

Let me know if you have any other ideas as well.

Thanks in advance,
Alok

Andi Kleen

2010-03-13 03:09 UTC #1 lore raw

, Alok Kataria wrote:

Hi Alok,

Quoted 10 lines

Hi,
Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.

Yes seems like a real problem.

Quoted 3 lines

1. Enable swiotlb for all 64bit kernels which have memory hot-add support.

I don't think that's a good idea. It would enable it everywhere on distributions which compile with hotadd. Need (2)

Quoted 8 lines

2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?

I have a patchkit to revamp the SRAT parsing to store the hotadd information more efficiently (the current way is pretty dumb) I need to repost that.

With that it would be relatively easy to do I think.

-Andi

Alok Kataria

author

2010-03-15 17:22 UTC #2 lore raw

Hi Andi,

Quoted 33 lines · Andi Kleen

, Alok Kataria wrote:
Hi Alok,
Quoted 10 lines
Hi,
Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.
Yes seems like a real problem.
Quoted 3 lines
1. Enable swiotlb for all 64bit kernels which have memory hot-add support.
I don't think that's a good idea. It would enable it everywhere on distributions which compile with hotadd. Need (2)
Quoted 8 lines
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?

Quoted 2 lines

I have a patchkit to revamp the SRAT parsing to store the hotadd information more efficiently (the current way is pretty dumb) I need to repost that.

Can you please send me any pointers to that patch series. Also am just curious to know about any merge plans for that patchkit.

Thanks,
Alok

Quoted 4 lines

With that it would be relatively easy to do I think.
-Andi

Konrad Rzeszutek Wilk

2010-03-16 01:18 UTC #3 lore raw

In reply to Andi Kleen

Quoted 33 lines · Andi Kleen

, Alok Kataria wrote:
Hi Alok,
Quoted 10 lines
Hi,
Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.
Yes seems like a real problem.
Quoted 3 lines
1. Enable swiotlb for all 64bit kernels which have memory hot-add support.
I don't think that's a good idea. It would enable it everywhere on distributions which compile with hotadd. Need (2)
Quoted 8 lines
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?

Quoted 1 line

I have a patchkit to revamp the SRAT parsing to store the hotadd information

There is a late mechanism to do kickoff the SWIOTLB. Perhaps the hot-add could use swiotlb_init_late and start up the SWIOTLB?

Quoted 10 lines

more efficiently (the current way is pretty dumb) I need to repost that.
With that it would be relatively easy to do I think.
-Andi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

FUJITA Tomonori

2010-03-16 01:37 UTC #4 lore raw

Quoted 34 lines · On Mon, 15 Mar 2010 20:51:40 -0400 Konrad Rzeszutek Wilk…

Quoted 33 lines · Andi Kleen
, Alok Kataria wrote:
Hi Alok,
Quoted 10 lines
Hi,
Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.
Yes seems like a real problem.
Quoted 3 lines
1. Enable swiotlb for all 64bit kernels which have memory hot-add support.
I don't think that's a good idea. It would enable it everywhere on distributions which compile with hotadd. Need (2)
Quoted 8 lines
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?

Quoted 4 lines

Quoted 1 line
I have a patchkit to revamp the SRAT parsing to store the hotadd information
There is a late mechanism to do kickoff the SWIOTLB. Perhaps the hot-add could use swiotlb_init_late and start up the SWIOTLB?

I guess that you are talking about swiotlb_late_init_with_default_size(), which IA64 uses. However, you can use swiotlb_late_init_with_default_size() only before we initialize devices. Making it work after initializing devices is not so easy, I think (that is, we need to change dma_ops).

Konrad Rzeszutek Wilk

2010-03-16 15:40 UTC #5 lore raw

Quoted 39 lines · FUJITA Tomonori

Quoted 36 lines · On Mon, 15 Mar 2010 20:51:40 -0400 Konrad Rzeszutek Wilk…
Quoted 34 lines · Andi Kleen
, Alok Kataria wrote:
Hi Alok,
Quoted 10 lines
Hi,
Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.
Yes seems like a real problem.
Quoted 3 lines
1. Enable swiotlb for all 64bit kernels which have memory hot-add support.
I don't think that's a good idea. It would enable it everywhere on distributions which compile with hotadd. Need (2)
Quoted 8 lines
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?
I have a patchkit to revamp the SRAT parsing to store the hotadd information

Quoted 8 lines

Quoted 2 lines
There is a late mechanism to do kickoff the SWIOTLB. Perhaps the hot-add could use swiotlb_init_late and start up the SWIOTLB?
I guess that you are talking about swiotlb_late_init_with_default_size(), which IA64 uses. However, you can use swiotlb_late_init_with_default_size() only before we initialize devices. Making it work after initializing devices is not so easy, I think (that is, we need to change dma_ops).

That is a good point. Especially if we have some outstanding DMA pages allocated via dma_alloc_coherent.

I thought that the machines that have hot-add memory they have their own fancy IOMMU. For example the IBM x3955 (and its family) utilize the Calgary IOMMU. The HP boxes utilize the Intel VT-D (or the AMD equivalant).

So is this mostly specialized in the areas of virtualized guests? (Xen PV guests with PCI passthrough suffer the same problem, btw).

Alok Kataria

author

2010-03-17 22:48 UTC #6 lore raw

Quoted 38 lines · Konrad Rzeszutek Wilk

Quoted 37 lines · FUJITA Tomonori
Quoted 34 lines · On Mon, 15 Mar 2010 20:51:40 -0400 Konrad Rzeszutek Wilk…
Quoted 33 lines · Andi Kleen
, Alok Kataria wrote:
Hi Alok,
Quoted 10 lines
Hi,
Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.
Yes seems like a real problem.
Quoted 3 lines
1. Enable swiotlb for all 64bit kernels which have memory hot-add support.
I don't think that's a good idea. It would enable it everywhere on distributions which compile with hotadd. Need (2)
Quoted 8 lines
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
I have a patchkit to revamp the SRAT parsing to store the hotadd information

Andi...ping any pointers to the patchkit.

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
There is a late mechanism to do kickoff the SWIOTLB. Perhaps the hot-add could use swiotlb_init_late and start up the SWIOTLB?

I don't see why we need to do this via late_init, swiotlb detection that happens through pci_swiotlb_detect, is already late enough that SRAT is already parsed. Or am I missing something ?

Quoted 10 lines

Quoted 6 lines
I guess that you are talking about swiotlb_late_init_with_default_size(), which IA64 uses. However, you can use swiotlb_late_init_with_default_size() only before we initialize devices. Making it work after initializing devices is not so easy, I think (that is, we need to change dma_ops).
That is a good point. Especially if we have some outstanding DMA pages allocated via dma_alloc_coherent.

Quoted 6 lines

I thought that the machines that have hot-add memory they have their own fancy IOMMU. For example the IBM x3955 (and its family) utilize the Calgary IOMMU. The HP boxes utilize the Intel VT-D (or the AMD equivalant).
So is this mostly specialized in the areas of virtualized guests? (Xen PV guests with PCI passthrough suffer the same problem, btw).

I am assuming that there were Intel based servers which supported memory hot-add before VT-d too. So, IMO this is not specialized to virtualization, though might be hard to prove if there are actual physical machines out there which have similar constraints (no HWIOMMU + MEMHOT add support)

Thanks,
Alok

Alok Kataria

author

2010-07-20 22:14 UTC #7 lore raw

Hi,

Reviving a 4 month old thread.
I am still waiting for any clues on this question below.

Quoted 8 lines

Quoted 8 lines
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?

Thanks,
Alok

Quoted 76 lines · Alok Kataria

Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 37 lines · FUJITA Tomonori
Quoted 34 lines · On Mon, 15 Mar 2010 20:51:40 -0400 Konrad Rzeszutek Wilk…
Quoted 33 lines · Andi Kleen
, Alok Kataria wrote:
Hi Alok,
Quoted 10 lines
Hi,
Looking at the current code swiotlb is initialized for 64bit kernels only when the max_pfn value is greater than 4G (MAX_DMA32_PFN value). So in cases when the initial memory is less than 4GB the kernel boots without enabling swiotlb, when we hotadd memory to such a kernel and go beyond the 4G limit, swiotlb is still disabled. As a result when any 32bit devices start using this newly added memory beyond 4G, the kernel starts spitting error messages like below or in some cases it causes kernel panics.
Yes seems like a real problem.
Quoted 3 lines
1. Enable swiotlb for all 64bit kernels which have memory hot-add support.
I don't think that's a good idea. It would enable it everywhere on distributions which compile with hotadd. Need (2)
Quoted 8 lines
2. Instead of checking the max_pfn value in pci_swiotlb_detect, check for max_hotpluggable_pfn (or some such) value. Though I don't see such a value readily available. I could parse the SRAT and get hotplug memory information but that will make swiotlb detection logic a little too complex. A quick look around srat_xx.c files and the acpi_memhotplug module didn't find any useful API that could be used directly either. So was wondering if any of you are aware of an easy way to get such information ?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
I have a patchkit to revamp the SRAT parsing to store the hotadd information
Andi...ping any pointers to the patchkit.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
There is a late mechanism to do kickoff the SWIOTLB. Perhaps the hot-add could use swiotlb_init_late and start up the SWIOTLB?
I don't see why we need to do this via late_init, swiotlb detection that happens through pci_swiotlb_detect, is already late enough that SRAT is already parsed. Or am I missing something ?
Quoted 10 lines
Quoted 6 lines
I guess that you are talking about swiotlb_late_init_with_default_size(), which IA64 uses. However, you can use swiotlb_late_init_with_default_size() only before we initialize devices. Making it work after initializing devices is not so easy, I think (that is, we need to change dma_ops).
That is a good point. Especially if we have some outstanding DMA pages allocated via dma_alloc_coherent.
Quoted 6 lines
I thought that the machines that have hot-add memory they have their own fancy IOMMU. For example the IBM x3955 (and its family) utilize the Calgary IOMMU. The HP boxes utilize the Intel VT-D (or the AMD equivalant).
So is this mostly specialized in the areas of virtualized guests? (Xen PV guests with PCI passthrough suffer the same problem, btw).
I am assuming that there were Intel based servers which supported memory hot-add before VT-d too. So, IMO this is not specialized to virtualization, though might be hard to prove if there are actual physical machines out there which have similar constraints (no HWIOMMU + MEMHOT add support)
Thanks,
Alok

FUJITA Tomonori

2010-07-21 05:03 UTC #8 lore raw

Quoted 2 lines · On Tue, 20 Jul 2010 15:14:57 -0700 Alok Kataria <[email protected]> wrote:

Reviving a 4 month old thread.
I am still waiting for any clues on this question below.

Basically, you want to add hot-plug memory and enable swiotlb, right?

We can't start swiotlb reliably after a system starts.

See dma32_reserve_boatmen() and dma32_free_bootmem(). What we do is reserving huge memory in DMA32 zone for swiotlb and releasing it if we find that we don't need swiotlb. We can't find enough memory for swiotlb in dma32 after a system starts.

Alok Kataria

author

2010-07-21 17:13 UTC #9 lore raw

Quoted 7 lines · FUJITA Tomonori

Quoted 2 lines · On Tue, 20 Jul 2010 15:14:57 -0700 Alok Kataria <[email protected]> wrote:
Reviving a 4 month old thread.
I am still waiting for any clues on this question below.
Basically, you want to add hot-plug memory and enable swiotlb, right?

Not really, I am planning to do something like this,

@@ -52,7 +52,7 @@ int __init pci_swiotlb_detect(void)

/* don't initialize swiotlb if iommu=off (no_iommu=1) */
#ifdef CONFIG_X86_64
- if (!no_iommu && max_pfn > MAX_DMA32_PFN)
+ if (!no_iommu && (max_pfn > MAX_DMA32_PFN || hotplug_possible()))
swiotlb = 1;
#endif
if (swiotlb_force)

BUT, I don't know how that hotplug_possible function will look like or if such an interface already exists in the kernel (my search didn't turn up any) ?

IMO, it should be possible to go read the SRAT to see if this system has support for hotplug memory and then enable swiotlb if it does.

Sounds right ?

Thanks,
Alok

Quoted 7 lines

We can't start swiotlb reliably after a system starts.
See dma32_reserve_boatmen() and dma32_free_bootmem(). What we do is reserving huge memory in DMA32 zone for swiotlb and releasing it if we find that we don't need swiotlb. We can't find enough memory for swiotlb in dma32 after a system starts.

FUJITA Tomonori

2010-07-21 23:44 UTC #10 lore raw

Review int __init pci_swiotlb_detect(void) · line 56

Quoted 11 lines · On Wed, 21 Jul 2010 10:13:34 -0700 Alok Kataria <[email protected]> wrote:

Quoted 1 line
Basically, you want to add hot-plug memory and enable swiotlb, right?
Not really, I am planning to do something like this,
(quoted diff) +1 -1
@@ -52,7 +52,7 @@ int __init pci_swiotlb_detect(void)
 
 	/* don't initialize swiotlb if iommu=off (no_iommu=1) */
 #ifdef CONFIG_X86_64
-	if (!no_iommu && max_pfn > MAX_DMA32_PFN)
+	if (!no_iommu && (max_pfn > MAX_DMA32_PFN || hotplug_possible()))
 		swiotlb = 1;

Always enable swiotlb with memory hotplug enabled? Wasting 64MB on a x86_64 system with 128MB doesn't look to be a good idea. I don't think that there is an easy solution for this issue though.

FUJITA Tomonori

2010-07-22 00:04 UTC #11 lore raw

Quoted 9 lines · On Thu, 22 Jul 2010 08:44:42 +0900 FUJITA Tomonori…

Quoted 6 lines · On Wed, 21 Jul 2010 10:13:34 -0700 Alok Kataria <[email protected]> wrote:
Quoted 1 line
Basically, you want to add hot-plug memory and enable swiotlb, right?
Not really, I am planning to do something like this,
(quoted diff) +0 -0
@@ -52,7 +52,7 @@ int __init pci_swiotlb_detect(void)
 

Quoted 9 lines

Review int __init pci_swiotlb_detect(void) · line 56
Quoted 5 lines
(quoted diff) +1 -1
 	/* don't initialize swiotlb if iommu=off (no_iommu=1) */
 #ifdef CONFIG_X86_64
-	if (!no_iommu && max_pfn > MAX_DMA32_PFN)
+	if (!no_iommu && (max_pfn > MAX_DMA32_PFN || hotplug_possible()))
 		swiotlb = 1;
Always enable swiotlb with memory hotplug enabled? Wasting 64MB on a x86_64 system with 128MB doesn't look to be a good idea. I don't think that there is an easy solution for this issue though.

btw, you need more work to enable switch on the fly.

You need to change the dma_ops pointer (see get_dma_ops()). It means that you need to track outstanding dma operations per device, locking, etc.

Alok Kataria

author

2010-07-22 18:34 UTC #12 lore raw

Hi,

Quoted 12 lines · FUJITA Tomonori

Quoted 9 lines · On Thu, 22 Jul 2010 08:44:42 +0900 FUJITA Tomonori…
Quoted 6 lines · On Wed, 21 Jul 2010 10:13:34 -0700 Alok Kataria <[email protected]> wrote:
Quoted 1 line
Basically, you want to add hot-plug memory and enable swiotlb, right?
Not really, I am planning to do something like this,
(quoted diff) +0 -0
@@ -52,7 +52,7 @@ int __init pci_swiotlb_detect(void)
 

Quoted 7 lines

Quoted 7 lines
Review int __init pci_swiotlb_detect(void) · line 56
Quoted 5 lines
(quoted diff) +1 -1
 	/* don't initialize swiotlb if iommu=off (no_iommu=1) */
 #ifdef CONFIG_X86_64
-	if (!no_iommu && max_pfn > MAX_DMA32_PFN)
+	if (!no_iommu && (max_pfn > MAX_DMA32_PFN || hotplug_possible()))
 		swiotlb = 1;
Always enable swiotlb with memory hotplug enabled?

yep though only on systems which have hotpluggable memory support.

Quoted 3 lines

Wasting 64MB on a
Quoted 2 lines
x86_64 system with 128MB doesn't look to be a good idea. I don't think that there is an easy solution for this issue though.

Good now that you agree that, that's the only feasible solution, do you have any suggestions for any interfaces that are available from SRAT for implementing hotplug_possible ?

Quoted 6 lines

btw, you need more work to enable switch on the fly.
You need to change the dma_ops pointer (see get_dma_ops()). It means that you need to track outstanding dma operations per device, locking, etc.

Yeah though if we are doing this during swiotlb_init time i.e. at bootup as suggested in the pseudo patch, we don't need to worry about all this, right ?

Thanks,
Alok

Konrad Rzeszutek Wilk

2010-07-23 14:24 UTC #13 lore raw

Quoted 23 lines · Alok Kataria

Hi,
Quoted 12 lines · FUJITA Tomonori
Quoted 9 lines · On Thu, 22 Jul 2010 08:44:42 +0900 FUJITA Tomonori…
Quoted 6 lines · On Wed, 21 Jul 2010 10:13:34 -0700 Alok Kataria <[email protected]> wrote:
Quoted 1 line
Basically, you want to add hot-plug memory and enable swiotlb, right?
Not really, I am planning to do something like this,
(quoted diff) +0 -0
@@ -52,7 +52,7 @@ int __init pci_swiotlb_detect(void)
 
Quoted 7 lines
Quoted 7 lines
Review int __init pci_swiotlb_detect(void) · line 56
Quoted 5 lines
(quoted diff) +1 -1
 	/* don't initialize swiotlb if iommu=off (no_iommu=1) */
 #ifdef CONFIG_X86_64
-	if (!no_iommu && max_pfn > MAX_DMA32_PFN)
+	if (!no_iommu && (max_pfn > MAX_DMA32_PFN || hotplug_possible()))
 		swiotlb = 1;
Always enable swiotlb with memory hotplug enabled?

Quoted 1 line

yep though only on systems which have hotpluggable memory support.

What machines are there that have hotplug support and no hardware IOMMU? I know of the IBM ones - but they use the Calgary IOMMU.

Quoted 8 lines

Quoted 4 lines
Wasting 64MB on a
Quoted 2 lines
x86_64 system with 128MB doesn't look to be a good idea. I don't think that there is an easy solution for this issue though.
Good now that you agree that, that's the only feasible solution, do you have any suggestions for any interfaces that are available from SRAT for implementing hotplug_possible ?

I thought SRAT has NUMA affinity information - so for example my AMD desktop box has that, but it does not support hotplug capability.

I think first your 'hotplug_possible' code needs to be more specific - not just check if SRAT exists, but also if there are swaths of memory that are non-populated. It would also help if there was some indication of whether the box truly does a hardware hotplug - is there a way to do this?

Quoted 11 lines

Quoted 7 lines
btw, you need more work to enable switch on the fly.
You need to change the dma_ops pointer (see get_dma_ops()). It means that you need to track outstanding dma operations per device, locking, etc.
Yeah though if we are doing this during swiotlb_init time i.e. at bootup as suggested in the pseudo patch, we don't need to worry about all this, right ?

Right..

Quoted 3 lines

Thanks,
Alok

Andi Kleen

2010-07-23 14:33 UTC #14 lore raw

Quoted 9 lines

I thought SRAT has NUMA affinity information - so for example my AMD desktop box has that, but it does not support hotplug capability.
I think first your 'hotplug_possible' code needs to be more specific - not just check if SRAT exists, but also if there are swaths of memory that are non-populated. It would also help if there was some indication of whether the box truly does a hardware hotplug - is there a way to do this?

The SRAT declares hotplug memory ranges in advance. And Linux already uses this information in the SRAT parser (just the code for doing this is a bit dumb, I have a rewrite somewhere)

The only drawback is that some older systems claimed to have large hotplug memory ranges when they didn't actually support it. So it's better to not do anything with a lot of overhead.

So yes it would be reasonable to let swiotlb (and possibly other code sizing itself based on memory) call into the SRAT parser and check the hotplug ranges too.

BTW longer term swiotlb should be really more dynamic anyways and grow and shrink on demand. I attempted this some time ago with my DMA allocator patchkit, unfortunately that didn't go forward.

-Andi

Konrad Rzeszutek Wilk

2010-07-23 15:00 UTC #15 lore raw

Quoted 22 lines · Andi Kleen

Quoted 9 lines
I thought SRAT has NUMA affinity information - so for example my AMD desktop box has that, but it does not support hotplug capability.
I think first your 'hotplug_possible' code needs to be more specific - not just check if SRAT exists, but also if there are swaths of memory that are non-populated. It would also help if there was some indication of whether the box truly does a hardware hotplug - is there a way to do this?
The SRAT declares hotplug memory ranges in advance. And Linux already uses this
information in the SRAT parser (just the code for doing this is a bit dumb, I have a rewrite
somewhere)
The only drawback is that some older systems claimed to have large hotplug memory ranges
when they didn't actually support it. So it's better to not do anything with a lot
of overhead.

Quoted 6 lines

So yes it would be reasonable to let swiotlb (and possibly other code sizing itself
based on memory) call into the SRAT parser and check the hotplug ranges too.
BTW longer term swiotlb should be really more dynamic anyways and grow and shrink on demand. I attempted this some time ago with my DMA

I was thinking about this at some point. I think the first step is to make SWIOTLB use the debugfs to actually print out how much of its buffers are used - and see if the 64MB is a good fit.

The shrinking part scares me - I think it might be more prudent to first explore on how to grow it. The big problem looks to allocate a physical contiguity set of pages. And I guess SWIOTLB would need to change from using one big region to something of a pool system?

Quoted 2 lines

allocator patchkit,
unfortunately that didn't go forward.

I wasn't present at that time so I don't know what the issues were - you wouldn't have a link to LKML for this?

Andi Kleen

2010-07-23 15:23 UTC #16 lore raw

Quoted 4 lines

I was thinking about this at some point. I think the first step is to make SWIOTLB use the debugfs to actually print out how much of its buffers are used - and see if the 64MB is a good fit.

swiotlb is near always wrongly sized. For most system it's far too much, but for some not enough. I have some systemtap scripts around to instrument it.

Also it depends on the IO load, so if you size it reasonable you risk overflow on large IO (however these days this very rarely happens because all "serious" IO devices don't need swiotlb anymore)

The other problem is that using only two bits for the needed address space is also extremly inefficient (4GB and 16MB on x86). Really want masks everywhere and optimize for the actual requirements.

Quoted 5 lines

The shrinking part scares me - I think it might be more prudent to first explore on how to grow it. The big problem looks to allocate a physical contiguity set of pages. And I guess SWIOTLB would need to change from using one big region to something of a pool system?

Shrinking: you define a movable zone, so with some delay it can be always freed.

The problem with swiotlb is however it still cannot block, but it can adapt to load.

The real fix would be blockable swiotlb, but the way drivers are set up this is difficult (at least in kernels using spinlocks)

Quoted 4 lines

Quoted 2 lines
allocator patchkit,
unfortunately that didn't go forward.
I wasn't present at that time so I don't know what the issues were - you wouldn't have a link to LKML for this?

There wasn't all that much opposition, but I ran out of time because fixing the infrastructure (e.g. getting rid of all of GFP_DMA) is a lot of work. In a sense it's a big tree sweep project like getting rid of BKL.

The old patch kit is at ftp://firstfloor.org/pub/ak/dma/ "intro" has the rationale.

I have a slightly newer version of the SCSI & misc drivers patchkit somewhere.

-Andi

FUJITA Tomonori

2010-07-28 10:10 UTC #17 lore raw

Quoted 8 lines · On Fri, 23 Jul 2010 17:23:33 +0200 Andi Kleen <[email protected]> wrote:

Quoted 4 lines
I was thinking about this at some point. I think the first step is to make SWIOTLB use the debugfs to actually print out how much of its buffers are used - and see if the 64MB is a good fit.
swiotlb is near always wrongly sized. For most system it's far too much, but for some
not enough. I have some systemtap scripts around to instrument it.

True, it's impossible to preallocate the best iotlb size statically.

Quoted 4 lines

Also it depends on the IO load, so if you size it reasonable you risk overflow on large IO (however these days this very rarely happens because
all "serious" IO devices don't need swiotlb anymore)

Yeah, nowadays it's pointless to try to get the good performance with swiotlb.

Quoted 5 lines

The other problem is that using only two bits for the needed address space is also extremly
inefficient (4GB and 16MB on x86). Really want masks everywhere and optimize for the
actual requirements.

swiotlb doesn't allocate GFP_DMA memory. It handles only GFP_DMA32.

swiotlb doesn't work for drivers with some odd dma mask (non 32bit) but we have been lived with it so I don't think that it's a big issue.

I think, supporting expanding swiotlb dynamically is enough. The default swiotlb size, 64MB is too large for majority.

I have a half-baked patch for it. I'll send it later.

Andi Kleen

2010-07-28 11:09 UTC #18 lore raw

Quoted 8 lines · FUJITA Tomonori

Quoted 6 lines
The other problem is that using only two bits for the needed address space is also extremly
inefficient (4GB and 16MB on x86). Really want masks everywhere and optimize for the
actual requirements.
swiotlb doesn't allocate GFP_DMA memory. It handles only GFP_DMA32.

I was lumping GFP_DMA and swiotlb together here. The pci_alloc_consistent() function uses both interchangedly. They really effectively are the same thing these days and just separated by historical accident.

Quoted 1 line

I have a half-baked patch for it. I'll send it later.

The problem are still the *_map users which usually cannot sleep, and then it's difficult to grow.
For *_alloc it's relatively easy and to some extend already implemented.

-Andi

Signature · 1 line

[email protected] -- Speaking for myself only.

FUJITA Tomonori

2010-07-28 14:21 UTC #19 lore raw

Quoted 8 lines · On Wed, 28 Jul 2010 13:09:57 +0200 Andi Kleen <[email protected]> wrote:

Quoted 7 lines · FUJITA Tomonori
Quoted 6 lines
The other problem is that using only two bits for the needed address space is also extremly
inefficient (4GB and 16MB on x86). Really want masks everywhere and optimize for the
actual requirements.

Quoted 6 lines

Quoted 1 line
swiotlb doesn't allocate GFP_DMA memory. It handles only GFP_DMA32.
I was lumping GFP_DMA and swiotlb together here. The pci_alloc_consistent() function uses both interchangedly. They really effectively are the same thing these days and just separated by historical accident.

Sorry, I meant to ZONE_DMA.

You are talking about your dma mask allocation patchset, right?

I meant that swiotlb doesn't need to handle ZONE_DMA. It handles only devices that can handle ZONE_DMA32.

Quoted 4 lines

Quoted 1 line
I have a half-baked patch for it. I'll send it later.
The problem are still the *_map users which usually cannot sleep, and then it's difficult to grow.

Why we can't use GFP_NOWAIT?

My approach is starting with small (like 4MB) and increasing io_tbl by chunk such as 4MB.