powerpc: update ibm,client-architecture
PATCH 1/1 powerpc: update ibm,client-architecture
In order to boot with more than 64 cores on machines that support the ibm,client-architecture RTAS call a new field has been added to the structure. This patch updates that field and adds a few others in the process. It would be good if this could go in as a bugfix.
arch/powerpc/kernel/prom_init.c +19 -2
@@ -654,6 +654,9 @@ static void __init early_cmdline_parse(v#define OV5_CMO 0x00#endif +/* Option Vector 6: IBM PAPR hints */ +#define OV6_LINUX 0x02/* Linux is our OS */ +/* * The architecture vector has an array of PVR mask/value pairs, * followed by # option vectors - 1, followed by the option vectors.
@@ -665,7 +668,7 @@ static unsigned char ibm_architecture_veW (0xffffffff ),W (0x0f000003 ),/* all 2.06-compliant */ W (0xffffffff ),W (0x0f000002 ),/* all 2.05-compliant */ W (0xfffffffe ),W (0x0f000001 ),/* all 2.04-compliant and earlier */ -5 -1 ,/* 5 option vectors */ +6 -1 ,/* 6 option vectors */ /* option vector 1: processor architectures supported */ 3 - 2, /* length */
@@ -697,12 +700,26 @@ static unsigned char ibm_architecture_ve0 ,/* don't halt */ /* option vector 5: PAPR/OF options */ -5 -2 ,/* length */ +13 -2 ,/* length */ 0 ,/* don't ignore, don't halt */ OV5_LPAR |OV5_SPLPAR |OV5_LARGE_PAGES |OV5_DRCONF_MEMORY |OV5_DONATE_DEDICATE_CPU |OV5_MSI ,0 ,OV5_CMO , +0 ,/* reserved */ +0 ,/* reserved */ +0 ,/* reserved */ +0 ,/* reserved */ +W (NR_CPUS /4 ),/* max cores supported */ + +/* option vector 6: IBM PAPR hints */ +4 -2 , +0 ,/* reserved */ +0 ,/* 1 == no secondary pteg */ + OV6_LINUX, + + /* option vector 7: IBM PAPR OS identification */ + /* a human readable ascii string will go here */ };/* Old method - ELF header with PT_NOTE sections */
Quoted 1 line
Subject: Re: [PATCH] powerpc: update ibm,client-architecture
Please give this a more appropriate name.
Quoted 4 lines
In order to boot with more than 64 cores on machines that support the ibm,client-architecture RTAS call a new field has been added to the structure. This patch updates that field and adds a few others in the process.
Please detail what these are here.
Quoted 33 lines
It would be good if this could go in as a bugfix.
Signed-off-by Joel Schopp <[email protected]>Index:
linux-2.6.git/arch/powerpc/kernel/prom_init.c
===================================================================arch/powerpc/kernel/prom_init.c +5
-2@@ -654,6 +654,9 @@ static void __init early_cmdline_parse(v#define OV5_CMO 0x00#endif +/* Option Vector 6: IBM PAPR hints */ +#define OV6_LINUX 0x02/* Linux is our OS */ +/* * The architecture vector has an array of PVR mask/value pairs, * followed by # option vectors - 1, followed by the option vectors. @@ -665,7 +668,7 @@ static unsigned char ibm_architecture_veW (0xffffffff ),W (0x0f000003 ),/* all 2.06-compliant */ W (0xffffffff ),W (0x0f000002 ),/* all 2.05-compliant */ W (0xfffffffe ),W (0x0f000001 ),/* all 2.04-compliant and earlier */ -5 -1 ,/* 5 option vectors */ +6 -1 ,/* 6 option vectors */ /* option vector 1: processor architectures supported */ 3 - 2, /* length */@@ -697,12 +700,26 @@ static unsigned char ibm_architecture_ve0 ,/* don't halt */ /* option vector 5: PAPR/OF options */ - 5 - 2, /* length */ + 13 - 2, /* length */
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 713
Quoted 10 lines
arch/powerpc/kernel/prom_init.c +5
-00 ,/* don't ignore, don't halt */ OV5_LPAR |OV5_SPLPAR |OV5_LARGE_PAGES |OV5_DRCONF_MEMORY |OV5_DONATE_DEDICATE_CPU |OV5_MSI ,0 ,OV5_CMO , +0 ,/* reserved */ +0 ,/* reserved */ +0 ,/* reserved */ +0 ,/* reserved */ + W(NR_CPUS/4), /* max cores supported */
4?
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 716
Quoted 3 lines
arch/powerpc/kernel/prom_init.c +3
-0+ +/* option vector 6: IBM PAPR hints */ + 4 - 2,
Please put a comment next to these to say what it is.
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 722
Quoted 6 lines
arch/powerpc/kernel/prom_init.c +6
-0+0 ,/* reserved */ +0 ,/* 1 == no secondary pteg */ + OV6_LINUX, + +/* option vector 7: IBM PAPR OS identification */ +/* a human readable ascii string will go here */
Can we do this now or remove the comment. Maybe UTS_RELEASE or something like that.
Mikey
Quoted 10 lines
arch/powerpc/kernel/prom_init.c +0
-0};/* Old method - ELF header with PT_NOTE sections */ _______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev
Quoted 6 lines
Quoted 3 lines
Subject: Re: [PATCH] powerpc: update ibm,client-architecture
Please give this a more appropriate name.
Any suggestions?
Quoted 9 lines
Quoted 6 lines
In order to boot with more than 64 cores on machines that support the ibm,client-architecture RTAS call a new field has been added to the structure. This patch updates that field and adds a few others in the process.
Please detail what these are here.
OK.
Quoted 6 lines
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 713
Quoted 2 lines
arch/powerpc/kernel/prom_init.c +1
-0+ W(NR_CPUS/4), /* max cores supported */4?
4 is the new 2. Since you don't know the actual threads per core at this point in boot you have to be conservative and go with the maximum number of any processor. See page 4 of these charts:
http://www.power.org/events/powercon09/taiwan09/IBM_Overview_POWER7.pdf
Quoted 5 lines
Can we do this now or remove the comment. Maybe
UTS_RELEASEor something like that.
I'll just remove the comment for now.
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Subject: Re: [PATCH] powerpc: update ibm,client-architecture
Please give this a more appropriate name.
Any suggestions?
Something that's more descriptive of what the patch does. Say "Add max CPU nodes field to ibm,client-architecture call"
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines
In order to boot with more than 64 cores on machines that support the ibm,client-architecture RTAS call a new field has been added to the structure. This patch updates that field and adds a few others in the process.
Please detail what these are here.
OK.
Quoted 1 line
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 713
Quoted 1 line
arch/powerpc/kernel/prom_init.c +1
-0+ W(NR_CPUS/4), /* max cores supported */
FYI reading the PAPR, this comment should technically be "max 'cpu' nodes presented".
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
4?
4 is the new 2.
I'd still be asking what 2 is. It's needs a #define to make clearer what you are doing.
Quoted 4 lines
Since you don't know the actual threads per core at
this point in boot you have to be conservative and go with the maximum number of any processor. See page 4 of these charts:
http://www.power.org/events/powercon09/taiwan09/IBM_Overview_POWER7.pdf
I don't think hard wiring 4 in here is right. If we are booting a machine with SMT2, we will put only half the number of cores that we can handle in this field. This is going to break a lot of machines where people have compiled with NR_CPUS = thread number.
I think you just want to put NR_CPUS here.
Mikey
Quoted 11 lines
Quoted 5 lines
Can we do this now or remove the comment. Maybe
UTS_RELEASEor something like that.I'll just remove the comment for now.
_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev
Quoted 9 lines
Quoted 5 lines
OK.
Quoted 2 lines
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 713
Quoted 2 lines
arch/powerpc/kernel/prom_init.c +1
-0+ W(NR_CPUS/4), /* max cores supported */FYI reading the PAPR, this comment should technically be "max 'cpu' nodes presented".
I applied a disambiguation filter to the comment since cpus can mean a lot of things these days ( ie hardware threads, cores, chips) , but a core is a core.
Quoted 7 lines
Quoted 2 lines
4 is the new 2.
I'd still be asking what 2 is. It's needs a #define to make clearer what you are doing.
I'll add a #define
Quoted 4 lines
Quoted 3 lines
Since you don't know the actual threads per core at this point in boot you have to be conservative and go with the maximum number of any processor. See page 4 of these charts:
http://www.power.org/events/powercon09/taiwan09/IBM_Overview_POWER7.pdf
Quoted 8 lines
I don't think hard wiring 4 in here is right. If we are booting a machine with SMT2, we will put only half the number of cores that we can handle in this field. This is going to break a lot of machines where people have compiled with
NR_CPUS= thread number.I think you just want to put
NR_CPUShere.
It's a bad interface. No matter what you choose there will be a downside. 1) If you choose NR_CPUS, the best case of how many you could boot without SMT, then when you boot with SMT2 or SMT4 you can get assigned more cpus than you can boot. 2) If you choose NR_CPUS/4, the worst case of how many you could boot, and you get a large machine with SMT2 or SMT1 you might have said you support less cpus than you actually do and thus not boot all the cpus. So no matter what you choose you could be not booting cpus in some theoretical scenario.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
OK.
Quoted 2 lines
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 713
Quoted 2 lines
arch/powerpc/kernel/prom_init.c +1
-0+ W(NR_CPUS/4), /* max cores supported */FYI reading the PAPR, this comment should technically be "max 'cpu' nodes presented".
I applied a disambiguation filter to the comment since cpus can mean a lot of things these days ( ie hardware threads, cores, chips) , but a core is a core.
I think you applied ambiguation filter actually :-)
The PAPR is clear and perfectly understandable in this context. It's "OF device tree nodes of type 'cpu'".
Quoted 22 lines
Quoted 7 lines
Quoted 2 lines
4 is the new 2.
I'd still be asking what 2 is. It's needs a #define to make clearer what you are doing.
I'll add a #define
Quoted 6 lines
Quoted 5 lines
Since you don't know the actual threads per core at
this point in boot you have to be conservative and go with the maximum number of any processor. See page 4 of these charts:
http://www.power.org/events/powercon09/taiwan09/IBM_Overview_POWER7.pdfQuoted 8 lines
I don't think hard wiring 4 in here is right. If we are booting a machine with SMT2, we will put only half the number of cores that we can handle in this field. This is going to break a lot of machines where people have compiled with
NR_CPUS= thread number.I think you just want to put
NR_CPUShere.
Quoted 8 lines
It's a bad interface. No matter what you choose there will be a downside. 1) If you choose
NR_CPUS, the best case of how many you could boot without SMT, then when you boot with SMT2 or SMT4 you can get assigned more cpus than you can boot. 2) If you choose NR_CPUS/4, the worst case of how many you could boot, and you get a large machine with SMT2 or SMT1 you might have said you support less cpus than you actually do and thus not boot all the cpus. So no matter what you choose you could be not booting cpus in some theoretical scenario.
Sure but 1) is how Linux behaves now (we can always get more cpus than we can handle) and 2) is what you are changing it to. 2) is going to break existing users.
Anyway, if you are intending to change functionality like this, please make it clear in the check in comment.
Mikey
Quoted 9 lines · Joel Schopp
It's a bad interface. No matter what you choose there will be a downside. 1) If you choose
NR_CPUS, the best case of how many you could boot without SMT, then when you boot with SMT2 or SMT4 you can get assigned more cpus than you can boot. 2) If you choose NR_CPUS/4, the worst case of how many you could boot, and you get a large machine with SMT2 or SMT1 you might have said you support less cpus than you actually do and thus not boot all the cpus. So no matter what you choose you could be not booting cpus in some theoretical scenario.
We're far enough through boot to determine the threading model, so you /could/ work out what SMT we're in and divide NR_CPUS by that and give that to firmware.
Yours Tony
Quoted 4 lines · Joel Schopp
In order to boot with more than 64 cores on machines that support the ibm,client-architecture RTAS call a new field has been added to the structure. This patch updates that field and adds a few others in the process. It would be good if this could go in as a bugfix.
But it's not really is it? What does it fix?
Quoted 30 lines
Signed-off-by Joel Schopp <[email protected]>Index:
linux-2.6.git/arch/powerpc/kernel/prom_init.c
===================================================================arch/powerpc/kernel/prom_init.c +4
-2@@ -654,6 +654,9 @@ static void __init early_cmdline_parse(v#define OV5_CMO 0x00#endif +/* Option Vector 6: IBM PAPR hints */ +#define OV6_LINUX 0x02/* Linux is our OS */ +/* * The architecture vector has an array of PVR mask/value pairs, * followed by # option vectors - 1, followed by the option vectors. @@ -665,7 +668,7 @@ static unsigned char ibm_architecture_veW (0xffffffff ),W (0x0f000003 ),/* all 2.06-compliant */ W (0xffffffff ),W (0x0f000002 ),/* all 2.05-compliant */ W (0xfffffffe ),W (0x0f000001 ),/* all 2.04-compliant and earlier */ -5 -1 ,/* 5 option vectors */ +6 -1 ,/* 6 option vectors */ /* option vector 1: processor architectures supported */ 3 - 2, /* length */@@ -697,12 +700,26 @@ static unsigned char ibm_architecture_ve 0, /* don't halt *//* option vector 5: PAPR/OF options */ - 5 - 2, /* length */
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 712
Quoted 10 lines
arch/powerpc/kernel/prom_init.c +5
-0+13 -2 ,/* length */ 0 ,/* don't ignore, don't halt */ OV5_LPAR |OV5_SPLPAR |OV5_LARGE_PAGES |OV5_DRCONF_MEMORY |OV5_DONATE_DEDICATE_CPU |OV5_MSI ,0 ,OV5_CMO , +0 ,/* reserved */ +0 ,/* reserved */ +0 ,/* reserved */ + 0, /* reserved */
Not so much reserved, as unused by us.
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 713
Quoted 1 line
arch/powerpc/kernel/prom_init.c +1
-0+ W(NR_CPUS/4), /* max cores supported */
4? I can see a case for 2 or just plain NR_CPUS, but 4 is wrong.
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 716
Quoted 2 lines
arch/powerpc/kernel/prom_init.c +2
-0+/* option vector 6: IBM PAPR hints */ + 4 - 2,
We all know this is a length, but please follow the style in this structure.
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 717
Quoted 2 lines
arch/powerpc/kernel/prom_init.c +2
-0+0 ,/* reserved */ + 0, /* 1 == no secondary pteg */
I think the "1 == " confuses things.
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 719
Quoted 1 line
arch/powerpc/kernel/prom_init.c +1
-0+ OV6_LINUX,
Does filling in the vector actually let us boot on systems where we do not already?
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 722
Quoted 3 lines
arch/powerpc/kernel/prom_init.c +3
-0+ +/* option vector 7: IBM PAPR OS identification */ +/* a human readable ascii string will go here */
Either fill it in or leave the comments out.
Yours Tony
Quoted 10 lines · Tony Breeds
Quoted 5 lines · Joel Schopp
In order to boot with more than 64 cores on machines that support the ibm,client-architecture RTAS call a new field has been added to the structure. This patch updates that field and adds a few others in the process. It would be good if this could go in as a bugfix.
But it's not really is it? What does it fix?
It fixes kernels compiled with NR_CPUS where NR_CPUS > 256.
Quoted 17 lines
Quoted 4 lines
arch/powerpc/kernel/prom_init.c +0
-1/* option vector 5: PAPR/OF options */ - 5 - 2, /* length */arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 712
Quoted 10 lines
arch/powerpc/kernel/prom_init.c +5
-0+13 -2 ,/* length */ 0 ,/* don't ignore, don't halt */ OV5_LPAR |OV5_SPLPAR |OV5_LARGE_PAGES |OV5_DRCONF_MEMORY |OV5_DONATE_DEDICATE_CPU |OV5_MSI ,0 ,OV5_CMO , +0 ,/* reserved */ +0 ,/* reserved */ +0 ,/* reserved */ + 0, /* reserved */Not so much reserved, as unused by us.
They were all reserved until recently, and the last 2 of the 4 are still reserved, while we just don't use the first two yet. I'll remove the comments from the first 2.
Quoted 6 lines
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 713
Quoted 3 lines
arch/powerpc/kernel/prom_init.c +1
-0+ W(NR_CPUS/4), /* max cores supported */4? I can see a case for 2 or just plain
NR_CPUS, but 4 is wrong.
4 is the new 2. Since the actual threads per core is unknown at this point in boot you have to be conservative and go with the maximum number of any processor. See page 4 of these charts:
http://www.power.org/events/powercon09/taiwan09/IBM_Overview_POWER7.pdf
Quoted 7 lines
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 716
Quoted 4 lines
arch/powerpc/kernel/prom_init.c +2
-0+/* option vector 6: IBM PAPR hints */ + 4 - 2,We all know this is a length, but please follow the style in this structure.
OK.
Quoted 7 lines
Review
Quoted 4 lines
(quoted diff) +2
-0+ 0, /* reserved */ + 0, /* 1 == no secondary pteg */I think the "1 == " confuses things.
I'll remove it then.
Quoted 6 lines
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 719
Quoted 3 lines
arch/powerpc/kernel/prom_init.c +1
-0+ OV6_LINUX,Does filling in the vector actually let us boot on systems where we do not already?
Will add description to the email body on this one.
Quoted 8 lines
arch/powerpc/kernel/prom_init.c static unsigned char ibm_architecture_ve · line 722
Quoted 5 lines
arch/powerpc/kernel/prom_init.c +3
-0+ +/* option vector 7: IBM PAPR OS identification */ +/* a human readable ascii string will go here */ Either fill it in or leave the comments out.
I'll leave the comment out.
Quoted 4 lines · Joel Schopp
4 is the new 2. Since the actual threads per core is unknown at this point in boot you have to be conservative and go with the maximum number of any processor. See page 4 of these charts:
http://www.power.org/events/powercon09/taiwan09/IBM_Overview_POWER7.pdf
Sure P7 /can/ be 4-way SMT, but consider the case where you know you have 128 threads so you set NR_CPUS to 128, then you boot with "max cores" of 32, so you lose half of your threads. I guess that's only a problem when you build your own kernels, distros are probably setting NR_CPUS high enough to cover all SMT2 systems anyway.
I guess if the patch only set the "max cores" and described why, I'd (FWIW :D) be more comfortable.
Yours Tony