Skip to contentSkip to conversation

x86, calgary: use 8M TCE table size by default

PATCH Superseded WANG Chao 3 participants 7 messages Mar 7, 2014 – Mar 10, 2014

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

WANG Chao

author
#0 lore raw

PATCH 1/1 x86, calgary: use 8M TCE table size by default

kexec-tools wants to pass kdump kernel needed memmap via E820 directly, instead of memmap=exactmap. That'll make saved_max_pfn totally useless.

Muli suggest to hard code TCE table size to max (8M) so that kdump kernel could use this default size and kexec-tools doesn't need to pass saved_max_pfn to kdump kernel in any way.

---
 arch/x86/kernel/pci-calgary_64.c | 25 ++++++++-----------------
 1 file changed, 8 insertions(+), 17 deletions(-)
arch/x86/kernel/pci-calgary_64.c +8 -17
index 299d493..b26ab94 100644
@@ -1207,25 +1207,17 @@ error:
 	return ret;
 }
 
-static inline int __init determine_tce_table_size(u64 ram)
+static inline int __init determine_tce_table_size(void)
 {
-	int ret;
-
 	if (specified_table_size != TCE_TABLE_SIZE_UNSPECIFIED)
 		return specified_table_size;
+	else
+		/*
+		 * Use 8M by default to get rid of saved_max_pfn,
+		 * suggested by Muli
+		 */
+		return TCE_TABLE_SIZE_8M;
 
-	/*
-	 * Table sizes are from 0 to 7 (TCE_TABLE_SIZE_64K to
-	 * TCE_TABLE_SIZE_8M). Table size 0 has 8K entries and each
-	 * larger table size has twice as many entries, so shift the
-	 * max ram address by 13 to divide by 8K and then look at the
-	 * order of the result to choose between 0-7.
-	 */
-	ret = get_order(ram >> 13);
-	if (ret > TCE_TABLE_SIZE_8M)
-		ret = TCE_TABLE_SIZE_8M;
-
-	return ret;
 }
 
 static int __init build_detail_arrays(void)
@@ -1418,8 +1410,7 @@ int __init detect_calgary(void)
 		return -ENOMEM;
 	}
 
-	specified_table_size = determine_tce_table_size((is_kdump_kernel() ?
-					saved_max_pfn : max_pfn) * PAGE_SIZE);
+	specified_table_size = determine_tce_table_size();
 
 	for (bus = 0; bus < MAX_PHB_BUS_NUM; bus++) {
 		struct calgary_bus_info *info = &bus_info[bus];
Signature · 7 lines
1.8.5.3


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

On Fri, Mar 07, 2014 at 04:10:16PM +0800, WANG Chao wrote:

[..]

arch/x86/kernel/pci-calgary_64.c int __init detect_calgary(void) · line 1413

Quoted 5 lines
arch/x86/kernel/pci-calgary_64.c +1 -2
 	}
 
-	specified_table_size = determine_tce_table_size((is_kdump_kernel() ?
-					saved_max_pfn : max_pfn) * PAGE_SIZE);
+	specified_table_size = determine_tce_table_size();

I don't think you can get rid of saved_max_pfn right away. What if somebody is using old first kernel and new second kernel. Then old kernel will still be using a table size which is smaller than 8M.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

author
#2 lore raw

Hi, Vivek

Quoted 5 lines · Vivek Goyal

On Fri, Mar 07, 2014 at 04:10:16PM +0800, WANG Chao wrote:

[..]

Quoted 2 lines

}

Quoted 7 lines

arch/x86/kernel/pci-calgary_64.c int __init detect_calgary(void) · line 1413

Quoted 3 lines
arch/x86/kernel/pci-calgary_64.c +1 -2
-	specified_table_size = determine_tce_table_size((is_kdump_kernel() ?
-					saved_max_pfn : max_pfn) * PAGE_SIZE);
+	specified_table_size = determine_tce_table_size();

I don't think you can get rid of saved_max_pfn right away. What if somebody is using old first kernel and new second kernel. Then old kernel will still be using a table size which is smaller than 8M.

If TCE table size is hard coded to 8M, the new 1st kernel can never be compatible with the old 2nd kernel. Because old kernel's TCE table size is depending on saved_max_pfn, which could be from 64K to 8M, not necessarily the hard coded value 8M.

If we want to drop saved_max_pfn once and for all, that's a trade-off which is worth doing. Because as Muli pointed out last time, there are very few people likely running upstream kernel on calgary machine. And most of them would use the same kernel as 1st and 2nd kernel. For the very very few people who suffers from it, we can always encourage them to use the same kernel for 1st and 2nd kernel or other solutions we have.

Thanks
WANG Chao

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
Quoted 16 lines · WANG Chao

Hi, Vivek

Quoted 5 lines · Vivek Goyal

On Fri, Mar 07, 2014 at 04:10:16PM +0800, WANG Chao wrote:

[..]

Quoted 2 lines

}

Quoted 7 lines

arch/x86/kernel/pci-calgary_64.c int __init detect_calgary(void) · line 1413

Quoted 3 lines
arch/x86/kernel/pci-calgary_64.c +1 -2
-	specified_table_size = determine_tce_table_size((is_kdump_kernel() ?
-					saved_max_pfn : max_pfn) * PAGE_SIZE);
+	specified_table_size = determine_tce_table_size();

I don't think you can get rid of saved_max_pfn right away. What if somebody is using old first kernel and new second kernel. Then old kernel will still be using a table size which is smaller than 8M.

Quoted 2 lines

If TCE table size is hard coded to 8M, the new 1st kernel can never be compatible with the old 2nd kernel.

I gave example of old first kernel and new second kernel, and not vice-versa.

So we have two scenarios.

- Old first kernel and new second kernel.
- New first kernel and old second kernel.

If we want to make these two scenarios work with calgary, we need to use kexec-tools with --pass-memmap option. And also in new kernel we need to retain saved_max_pfn. Only difference will be, that we will set saved_max_pfn only if it is kdump kernel and if user passed a memory map on kernel command line.

So it is doable if it want to do it.

Now the real question is, is it worth introducing all this complication and confusion given the fact most of the people use same first and second kernel and given the fact that calgary is old hardware. What are the chances somebody will run into it.

I would say it is not very complicated to maintain backward compatibility in this case. So let us keep saved_max_pfn for some time and make kexec-tools changes. Some time down the line, one can get rid of saved_max_pfn completely.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw
Quoted 4 lines · Vivek Goyal

I would say it is not very complicated to maintain backward compatibility in this case. So let us keep saved_max_pfn for some time and make kexec-tools changes. Some time down the line, one can get rid of saved_max_pfn completely.

If this is not too painful then this would be my preference too.

Cheers,
Muli

author
#5 lore raw
In reply to Vivek Goyal
Quoted 25 lines · Vivek Goyal
Quoted 16 lines · WANG Chao

Hi, Vivek

Quoted 5 lines · Vivek Goyal

On Fri, Mar 07, 2014 at 04:10:16PM +0800, WANG Chao wrote:

[..]

Quoted 2 lines

}

Quoted 7 lines

arch/x86/kernel/pci-calgary_64.c int __init detect_calgary(void) · line 1413

Quoted 3 lines
arch/x86/kernel/pci-calgary_64.c +1 -2
-	specified_table_size = determine_tce_table_size((is_kdump_kernel() ?
-					saved_max_pfn : max_pfn) * PAGE_SIZE);
+	specified_table_size = determine_tce_table_size();

I don't think you can get rid of saved_max_pfn right away. What if somebody is using old first kernel and new second kernel. Then old kernel will still be using a table size which is smaller than 8M.

Quoted 2 lines

If TCE table size is hard coded to 8M, the new 1st kernel can never be compatible with the old 2nd kernel.

I gave example of old first kernel and new second kernel, and not vice-versa.

So we have two scenarios.

Quoted 8 lines

- Old first kernel and new second kernel.
- New first kernel and old second kernel.

If we want to make these two scenarios work with calgary, we need to use kexec-tools with --pass-memmap option. And also in new kernel we need to retain saved_max_pfn. Only difference will be, that we will set saved_max_pfn only if it is kdump kernel and if user passed a memory map on kernel command line.

Old first kernel and new second kernel is easy to handle. Like you said, we can use exactmap and set saved_max_pfn in kdump kernel, just as the old way.

But it's still not clear to me how we can handle the case of new first kernel and old second kernel.

Let's say, a calgary system has 2G memory. The new first kernel TCE table size are hard coded to 8M. When the old kdump kernel is booting, memmap=exactmap is parsed and saved_max_pfn is set to 2G/PAGE_SIZE. TCE table size is determined by 2G ram size. So the size is 4M. We run into the situation that TCE table is 8M in first kernel and 4M in second kernel. This inconsistency of TCE table size would cause kdump kernel fails somehow, that's why we brought in saved_max_pfn in the first place as you know.

The problem is how to keep TCE table size being consistent between new first kernel and old second kernel.

I can only think of exporting this value to user space but once we does that, this knob file could introduce another layer of backward compatible issue in the future.

Quoted 7 lines

So it is doable if it want to do it.

Now the real question is, is it worth introducing all this complication and confusion given the fact most of the people use same first and second kernel and given the fact that calgary is old hardware. What are the chances somebody will run into it.

I understand. I'd definitely try to keep things compatible between the new and old if that's doable. But chances that people would run into an issue may be lower than both of us think.

Quoted 5 lines

I would say it is not very complicated to maintain backward compatibility in this case. So let us keep saved_max_pfn for some time and make kexec-tools changes. Some time down the line, one can get rid of saved_max_pfn completely.

I'm just not convinced that we can properly handle the case of new first kernel and old second kernel.

Could you be more specific about how that could be solved please?

Thanks
WANG Chao

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw

On Mon, Mar 10, 2014 at 04:38:34PM +0800, WANG Chao wrote:

[..]

Quoted 27 lines
Quoted 10 lines

So we have two scenarios.

- Old first kernel and new second kernel.
- New first kernel and old second kernel.

If we want to make these two scenarios work with calgary, we need to use kexec-tools with --pass-memmap option. And also in new kernel we need to retain saved_max_pfn. Only difference will be, that we will set saved_max_pfn only if it is kdump kernel and if user passed a memory map on kernel command line.

Old first kernel and new second kernel is easy to handle. Like you said, we can use exactmap and set saved_max_pfn in kdump kernel, just as the old way.

But it's still not clear to me how we can handle the case of new first kernel and old second kernel.

Let's say, a calgary system has 2G memory. The new first kernel TCE table size are hard coded to 8M. When the old kdump kernel is booting, memmap=exactmap is parsed and saved_max_pfn is set to 2G/PAGE_SIZE. TCE table size is determined by 2G ram size. So the size is 4M. We run into the situation that TCE table is 8M in first kernel and 4M in second kernel. This inconsistency of TCE table size would cause kdump kernel fails somehow, that's why we brought in saved_max_pfn in the first place as you know.

Quoted 2 lines

The problem is how to keep TCE table size being consistent between new first kernel and old second kernel.

You are right. I did not think through it. So we can't handle the case of new first kernel and second old kernel without exporting size of tables to user space. Given the fact that calgary is old hardware, exporting table size to user space is not very prudent.

I would say let us handle the case of old first kernel and second new kernel to maintain backward compatibility.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec