badness in xics_set_cpu_giq on JS20 blade
With 2.6.28-rc5 the WARN_ON in xics_set_cpu_giq is triggering on a JS20. I changed it to a WARN to get the actual status returned:
[boot]0020 XICS Init
set-indicator returned -22
------------[ cut here ]------------ Badness at arch/powerpc/platforms/pseries/xics.c:733 NIP: c000000000047454 LR: c000000000047450 CTR: 0000000000033070 REGS: c0000000006b3a20 TRAP: 0700 Not tainted (2.6.28-rc5-autokern1) MSR: 8000000000021032 <ME,IR,DR> CR: 22000022 XER: 00000000
TASK = c0000000005f0fe0[0] 'swapper' THREAD: c0000000006b0000 CPU: 0
GPR00: c000000000047450 c0000000006b3ca0 c0000000006af710 000000000000001e
GPR04: 0000000000000000 ffffffffffffffff 0a000000006b39d0 000000000920224a
GPR08: 0000000000000001 c00000000060f5b0 c00000000070b49c c00000000070b498
GPR12: 8000000000001032 c0000000006ef300 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000001c00000 c0000000004e0ed8
GPR20: c0000000005adb70 00000000021adde0 c0000000005adde0 0000000000000000
GPR24: 0000000000250800 0000000001a1f900 c000000000700950 c0000000005aaed8
GPR28: c0000000006ec030 c00000001dffc3f8 c000000000655940 0000000000000001 NIP [c000000000047454] .xics_set_cpu_giq+0x54/0x68
LR [c000000000047450] .xics_set_cpu_giq+0x50/0x68
Call Trace: [c0000000006b3ca0] [c000000000047450] .xics_set_cpu_giq+0x50/0x68 (unreliable) [c0000000006b3d10] [c0000000005927b8] .xics_init_IRQ+0x2f4/0x338 [c0000000006b3de0] [c000000000591bcc] .pseries_xics_init_IRQ+0x14/0x2c [c0000000006b3e60] [c000000000580488] .init_IRQ+0x40/0x5c [c0000000006b3ee0] [c0000000005787d8] .start_kernel+0x250/0x478 [c0000000006b3f90] [c0000000000083b8] .start_here_common+0x1c/0x64
Instruction dump:7c832050 3860232d 3884ffff 7c8407b4 4bfd7521 60000000 2f830000 7c641b78 40bc0014 e8629f60 4801e32d 60000000 <0fe00000> 38210070 e8010010 7c0803a6 [boot]0021 XICS Done
-22 is -EINVAL, which maps to a -3 return code from RTAS (see rtas_error_rc).
The system appears to boot and function normally after this, though. FWIW, it looks like its firmware is up to date (FW08401160 from March 2008).
Quoted 7 lines · Nathan Lynch
With 2.6.28-rc5 the
WARN_ONinxics_set_cpu_giqis triggering on a JS20. I changed it to a WARN to get the actual status returned:[boot]0020 XICS Init
set-indicator returned -22------------[ cut here ]------------ Badness at arch/powerpc/platforms/pseries/xics.c:733
...
Quoted 7 lines
Call Trace: [c0000000006b3ca0] [c000000000047450] .xics_set_cpu_giq+0x50/0x68 (unreliable) [c0000000006b3d10] [c0000000005927b8] .xics_init_IRQ+0x2f4/0x338 [c0000000006b3de0] [c000000000591bcc] .pseries_xics_init_IRQ+0x14/0x2c [c0000000006b3e60] [c000000000580488] .init_IRQ+0x40/0x5c [c0000000006b3ee0] [c0000000005787d8] .start_kernel+0x250/0x478 [c0000000006b3f90] [c0000000000083b8] .start_here_common+0x1c/0x64
...
Quoted 6 lines
-22 is
-EINVAL, which maps to a -3 return code from RTAS (seertas_error_rc).The system appears to boot and function normally after this, though. FWIW, it looks like its firmware is up to date (FW08401160 from March 2008).
b4963255ad5a426f04a0bb15c4315fa4bb40cde9 "Factor out cpu joining/unjoining the GIQ" consolidated the join and remove call sites.
Looking closer it also added warn if rtas-indicator returned an error
on join in addition to leave.
I don't have my PAPR here, but from rtas_error_rc (rtas.c) -3 is /* Bad indicator/domain/etc */. This indicator was added to support cpu add and remove. I'm guessing js20 doesn't support that from rtas (ie doesn't support cpu hotadd), and the indicator is not available. (I know js21 has it because I had a bit of time to see its broken emulation of this call once).
When we get control of a cpu from OF it is already in the joined state.
We join on all threads because we need to do so on secondary threads
and because we did a remove on (previously all, but now secondary) threads during kexec.
If memory serves, there is a property in the rtas node to indicate each sensor that is present. If so, we should search for that property before calling set-indicator.
milton
Quoted 21 lines · Milton Miller
Quoted 7 lines · Nathan Lynch
With 2.6.28-rc5 the
WARN_ONinxics_set_cpu_giqis triggering on a JS20. I changed it to a WARN to get the actual status returned:[boot]0020 XICS Init
set-indicator returned -22------------[ cut here ]------------ Badness at arch/powerpc/platforms/pseries/xics.c:733...
Quoted 8 lines
Call Trace: [c0000000006b3ca0] [c000000000047450] .xics_set_cpu_giq+0x50/0x68(unreliable)
[c0000000006b3d10] [c0000000005927b8] .xics_init_IRQ+0x2f4/0x338 [c0000000006b3de0] [c000000000591bcc] .pseries_xics_init_IRQ+0x14/0x2c [c0000000006b3e60] [c000000000580488] .init_IRQ+0x40/0x5c [c0000000006b3ee0] [c0000000005787d8] .start_kernel+0x250/0x478 [c0000000006b3f90] [c0000000000083b8] .start_here_common+0x1c/0x64...
Quoted 3 lines
-22 is
-EINVAL, which maps to a -3 return code from RTAS (seertas_error_rc).
Quoted 8 lines
Quoted 3 lines
The system appears to boot and function normally after this, though. FWIW, it looks like its firmware is up to date (FW08401160 from March 2008).
b4963255ad5a426f04a0bb15c4315fa4bb40cde9"Factor out cpu joining/unjoining the GIQ" consolidated the join and remove call sites. Looking closer it also added warn if rtas-indicator returned an error on join in addition to leave.
Thanks, that makes it clear why we didn't see the warning before.
Quoted 6 lines
I don't have my PAPR here, but from
rtas_error_rc(rtas.c) -3 is /* Bad indicator/domain/etc */. This indicator was added to support cpu add and remove. I'm guessing js20 doesn't support that from rtas (ie doesn't support cpu hotadd), and the indicator is not available. (I know js21 has it because I had a bit of time to see its broken emulation of this call once).
Right, JS20 doesn't support cpu offline so it makes sense that the indicator isn't valid.
Quoted 8 lines
When we get control of a cpu from OF it is already in the joined state. We join on all threads because we need to do so on secondary threads and because we did a remove on (previously all, but now secondary) threads during kexec.
If memory serves, there is a property in the rtas node to indicate each sensor that is present. If so, we should search for that property before calling set-indicator.
I checked PAPR; it looks like we should be looking for an rtas-indicators property, which doesn't exist on this system. It does exist on Power5 and Power6 systems I've checked, and it has the expected entry for GIQ (9005).
Something like this? I've booted it on the problem JS20 as well as Power5 and 6.
arch/powerpc/include/asm/rtas.h | 1 + arch/powerpc/kernel/rtas.c | 22 ++++++++++++++++++++++ arch/powerpc/platforms/pseries/xics.c | 11 ++++++++--- 3 files changed, 31 insertions(+), 3 deletions(-)
arch/powerpc/include/asm/rtas.h +1 -0
@@ -168,6 +168,7 @@ extern void rtas_os_term(char *str);extern int rtas_get_sensor (int sensor,int index,int *state);extern int rtas_get_power_level (int powerdomain,int *level);extern int rtas_set_power_level (int powerdomain,int level,int *setlevel); +extern bool rtas_indicator_present (int indicator);extern int rtas_set_indicator (int indicator,int index,int new_value);extern int rtas_set_indicator_fast (int indicator,int index,int new_value);extern void rtas_progress (char *s,unsigned short hex);
arch/powerpc/kernel/rtas.c +22 -0
@@ -566,6 +566,28 @@ int rtas_get_sensor(int sensor, int index, int *state) }EXPORT_SYMBOL (rtas_get_sensor); +bool rtas_indicator_present (int indicator) +{ +int proplen, count, i; +const struct indicator_elem { +u32 token; +u32 maxindex; + } *indicators; + + indicators =of_get_property (rtas.dev,"rtas-indicators" , &proplen); +if (!indicators) +return false; + + count = proplen /sizeof (struct indicator_elem ); + +for (i =0 ; i < count; i++) +if (indicators[i].token == indicator) +return true; + +return false; +} +EXPORT_SYMBOL (rtas_indicator_present); +int rtas_set_indicator (int indicator,int index,int new_value) {int token =rtas_token ("set-indicator" );
arch/powerpc/platforms/pseries/xics.c +8 -3
@@ -728,9 +728,14 @@ static void xics_set_cpu_priority(unsigned char cppr)/* Have the calling processor join or leave the specified global queue */ static void xics_set_cpu_giq (unsigned int gserver,unsigned int join) { -int status =rtas_set_indicator_fast (GLOBAL_INTERRUPT_QUEUE , - (1UL << interrupt_server_size) -1 - gserver, join); -WARN_ON (status <0 ); +int status; + +if (!rtas_indicator_present (GLOBAL_INTERRUPT_QUEUE )) +return ; + + status =rtas_set_indicator_fast (GLOBAL_INTERRUPT_QUEUE , + (1UL << interrupt_server_size) -1 - gserver, join); +WARN (status <0 ,"set-indicator returned %d\n" , status); }void xics_setup_cpu (void )
Quoted 21 lines · Nathan Lynch
Quoted 8 lines · Milton Miller
Quoted 2 lines · Nathan Lynch
With 2.6.28-rc5 the
WARN_ONinxics_set_cpu_giqis triggering on a JS20. I changed it to a WARN to get the actual status returned:
b4963255ad5a426f04a0bb15c4315fa4bb40cde9"Factor out cpu joining/unjoining the GIQ" consolidated the join and remove call sites. Looking closer it also added warn if rtas-indicator returned an error on join in addition to leave.Thanks, that makes it clear why we didn't see the warning before.
Quoted 8 lines
When we get control of a cpu from OF it is already in the joined state. We join on all threads because we need to do so on secondary threads and because we did a remove on (previously all, but now secondary) threads during kexec.
If memory serves, there is a property in the rtas node to indicate each sensor that is present. If so, we should search for that property before calling set-indicator.
Quoted 7 lines
I checked PAPR; it looks like we should be looking for an rtas-indicators property, which doesn't exist on this system. It does exist on Power5 and Power6 systems I've checked, and it has the expected entry for GIQ (9005).
Something like this? I've booted it on the problem JS20 as well as Power5 and 6.
Pretty close. We need to turn the above text into a changelog and a few comments below.
We should also check JS21 and cell blade.
Quoted 41 lines
arch/powerpc/include/asm/rtas.h | 1 + arch/powerpc/kernel/rtas.c | 22 ++++++++++++++++++++++ arch/powerpc/platforms/pseries/xics.c | 11 ++++++++--- 3 files changed, 31 insertions(+), 3 deletions(-)arch/powerpc/include/asm/rtas.h +1
-0@@ -168,6 +168,7 @@ extern void rtas_os_term(char *str);extern int rtas_get_sensor (int sensor,int index,int *state);extern int rtas_get_power_level (int powerdomain,int *level);extern int rtas_set_power_level (int powerdomain,int level,int *setlevel); +extern bool rtas_indicator_present (int indicator);extern int rtas_set_indicator (int indicator,int index,int new_value);extern int rtas_set_indicator_fast(int indicator, int index, int
new_value);
extern voidrtas_progress(char *s, unsigned short hex);arch/powerpc/kernel/rtas.c +11
-0@@ -566,6 +566,28 @@ int rtas_get_sensor(int sensor, int index, int *state) }EXPORT_SYMBOL (rtas_get_sensor); +bool rtas_indicator_present (int indicator) +{ +int proplen, count, i; +const struct indicator_elem { +u32 token; +u32 maxindex; + } *indicators; + + indicators =of_get_property (rtas.dev,"rtas-indicators" , &proplen); +if (!indicators) +return false;
arch/powerpc/kernel/rtas.c int rtas_get_sensor(int sensor, int index, int · line 590
Quoted 10 lines
arch/powerpc/kernel/rtas.c +10
-0+ + count = proplen /sizeof (struct indicator_elem ); + +for (i =0 ; i < count; i++) +if (indicators[i].token == indicator) +return true; + +return false; +} +EXPORT_SYMBOL (rtas_indicator_present);
I didn't look at the stuff we do to create the indicator directory in /proc/rtas. Should this take an optional pointer arg to return the max count? Just noticing that we are ignoring the count parameter.
Quoted 14 lines
arch/powerpc/kernel/rtas.c +1
-0+int rtas_set_indicator (int indicator,int index,int new_value) {arch/powerpc/kernel/rtas.c +0
-0int token =rtas_token ("set-indicator" );arch/powerpc/platforms/pseries/xics.c +0
-0@@ -728,9 +728,14 @@ static void xics_set_cpu_priority(unsigned char cppr)/* Have the calling processor join or leave the specified global queue */
static voidxics_set_cpu_giq(unsigned int gserver, unsigned int join){
arch/powerpc/platforms/pseries/xics.c static void xics_set_cpu_priority(unsigned char cppr) · line 734
Quoted 7 lines
arch/powerpc/platforms/pseries/xics.c +4
-3-int status =rtas_set_indicator_fast (GLOBAL_INTERRUPT_QUEUE , - (1UL << interrupt_server_size) -1 - gserver, join); -WARN_ON (status <0 ); +int status; + +if (!rtas_indicator_present (GLOBAL_INTERRUPT_QUEUE )) +return ;
I was thinking we would cache this result, but we don't save the token for rtas_set_indicator_fast and we only call this twice per cpu on each kernel (startup and kexec) so I'm ok with it. (Athought it points out we are parsing the device tree in the kdump path).
arch/powerpc/platforms/pseries/xics.c static void xics_set_cpu_priority(unsigned char cppr) · line 738
Quoted 4 lines
arch/powerpc/platforms/pseries/xics.c +4
-0+ + status =rtas_set_indicator_fast (GLOBAL_INTERRUPT_QUEUE , + (1UL << interrupt_server_size) -1 - gserver, join); +WARN (status <0 ,"set-indicator returned %d\n" , status);
This WARN should at least indicate which indicator we are trying to set (GLOBAL_INTERRUPT_QUEUE), if not the number (which is likely to always be 0 and not checked by current firmware). That way we won't have to remember that the set-indicator in xics.c is to adjust cpu irq distribution when looking at the warning in the future.
Quoted 4 lines
}
void
xics_setup_cpu(void)
thanks,
milton
PATCH 1/1 check for GIQ indicator before calling set-indicator
Since "Factor out cpu joining/unjoining the GIQ" (b4963255ad5a426f04a0bb15c4315fa4bb40cde9) the WARN_ON in xics_set_cpu_giq() is being triggered during boot on JS20 because the GIQ indicator is not available on that platform. While the warning is harmless and the system runs normally, it's nicer to check for the existence of the indicator before trying to manipulate it.
Implement rtas_indicator_present(), which searches the /rtas/rtas-indicators property for the given indicator token, and use this function in xics_set_cpu_giq().
Also use a WARN statement in xics_set_cpu_giq to get better information on failure.
--- arch/powerpc/include/asm/rtas.h | 1 + arch/powerpc/kernel/rtas.c | 26 ++++++++++++++++++++++++++ arch/powerpc/platforms/pseries/xics.c | 15 ++++++++++++--- 3 files changed, 39 insertions(+), 3 deletions(-)
arch/powerpc/include/asm/rtas.h +1 -0
@@ -168,6 +168,7 @@ extern void rtas_os_term(char *str);extern int rtas_get_sensor (int sensor,int index,int *state);extern int rtas_get_power_level (int powerdomain,int *level);extern int rtas_set_power_level (int powerdomain,int level,int *setlevel); +extern bool rtas_indicator_present (int token,int *maxindex);extern int rtas_set_indicator (int indicator,int index,int new_value);extern int rtas_set_indicator_fast (int indicator,int index,int new_value);extern void rtas_progress (char *s,unsigned short hex);
arch/powerpc/kernel/rtas.c +26 -0
@@ -566,6 +566,32 @@ int rtas_get_sensor(int sensor, int index, int *state) }EXPORT_SYMBOL (rtas_get_sensor); +bool rtas_indicator_present (int token,int *maxindex) +{ +int proplen, count, i; +const struct indicator_elem { +u32 token; +u32 maxindex; + } *indicators; + + indicators =of_get_property (rtas.dev,"rtas-indicators" , &proplen); +if (!indicators) +return false; + + count = proplen /sizeof (struct indicator_elem ); + +for (i =0 ; i < count; i++) { +if (indicators[i].token != token) +continue ; +if (maxindex) + *maxindex = indicators[i].maxindex; +return true; + } + +return false; +} +EXPORT_SYMBOL (rtas_indicator_present); +int rtas_set_indicator (int indicator,int index,int new_value) {int token =rtas_token ("set-indicator" );
arch/powerpc/platforms/pseries/xics.c +12 -3
@@ -728,9 +728,18 @@ static void xics_set_cpu_priority(unsigned char cppr)/* Have the calling processor join or leave the specified global queue */ static void xics_set_cpu_giq (unsigned int gserver,unsigned int join) { -int status =rtas_set_indicator_fast (GLOBAL_INTERRUPT_QUEUE , - (1UL << interrupt_server_size) -1 - gserver, join); -WARN_ON (status <0 ); +int index; +int status; + +if (!rtas_indicator_present (GLOBAL_INTERRUPT_QUEUE ,NULL )) +return ; + + index = (1UL << interrupt_server_size) -1 - gserver; + + status =rtas_set_indicator_fast (GLOBAL_INTERRUPT_QUEUE , index, join); + +WARN (status <0 ,"set-indicator(%d, %d, %u) returned %d\n" , +GLOBAL_INTERRUPT_QUEUE , index, join, status); }void xics_setup_cpu (void )
Signature · 1 line
1.5.5
Re: [PATCH] check for GIQ indicator before calling set-indicator
Quoted 7 lines · Nathan Lynch
Since "Factor out cpu joining/unjoining the GIQ" (
b4963255ad5a426f04a0bb15c4315fa4bb40cde9) theWARN_ONinxics_set_cpu_giq()is being triggered during boot on JS20 because the GIQ indicator is not available on that platform. While the warning is harmless and the system runs normally, it's nicer to check for the existence of the indicator before trying to manipulate it.
Quoted 8 lines
Implement
rtas_indicator_present(), which searches the /rtas/rtas-indicators property for the given indicator token, and use this function inxics_set_cpu_giq().Also use a WARN statement in
xics_set_cpu_giqto get better information on failure.Signed-off-by Nathan Lynch <[email protected]>
...
Quoted 32 lines
arch/powerpc/platforms/pseries/xics.c +0
-0@@ -728,9 +728,18 @@ static void xics_set_cpu_priority(unsigned char cppr)/* Have the calling processor join or leave the specified global queue */
static voidxics_set_cpu_giq(unsigned int gserver, unsigned int join){arch/powerpc/platforms/pseries/xics.c +9
-3-int status =rtas_set_indicator_fast (GLOBAL_INTERRUPT_QUEUE , - (1UL << interrupt_server_size) -1 - gserver, join); -WARN_ON (status <0 ); +int index; +int status; + +if (!rtas_indicator_present (GLOBAL_INTERRUPT_QUEUE ,NULL )) +return ; + + index = (1UL << interrupt_server_size) -1 - gserver; + + status = rtas_set_indicator_fast(GLOBAL_INTERRUPT_QUEUE, index,join);
arch/powerpc/platforms/pseries/xics.c +3
-0+ +WARN (status <0 ,"set-indicator(%d, %d, %u) returned %d\n" , +GLOBAL_INTERRUPT_QUEUE , index, join, status); }void xics_setup_cpu (void )Signature · 1 line
1.5.5