Abnormal behaviour towards "INIT" interrupt management
Hi, I've already sent this mail, but receiving no feedback, I try again
(sorry for people who don't feel concerned by that stuff). I'd like to report what I estimate a potential bug in the INIT management (at the present time I don't know exactly what the origin of this problem is).
Testing the "dump" button on a 4-way Itanium-2 machine equipped with a SAL 3.0, I've noticed a not-inline with SAL specifications behaviour. Indeed, in the "ia64_mca_init" function (in arch/ia64/kernel/mca.c file) we register the functions "ia64_monarch_init_handler" and "ia64_slave_init_handler".
Then I push on the "dump" button that generates an INIT interruption to all the processors. This signal is then caught by PAL, and SAL which calls (with a reason [register GR11] equal to 2) "ia64_monarch_init_handler" on the monarch processor and "ia64_slave_init_handler" on the slave ones (to this point, I hope to be right, isn't it ?).
What I've noticed using traces (and further an ITP tool) is that for each processor the "ia64_monarch_init_handler" is ever called. :-( Could someone tell me if he already has encountered this problem or if it is an expected behavior ?
For information, I've done the same test with a 16-way Itanium-2 machine with a SAL 3.1 and the result is exactly the same.
Thanks for your help.
Best regards,
Francois WELLENREITER
Quoted 6 lines · Francois Wellenreiter
Then I push on the "dump" button that generates an INIT interruption to all the processors. This signal is then caught by PAL, and SAL which calls (with a reason [register GR11] equal to 2) "
ia64_monarch_init_handler" on the monarch processor and "ia64_slave_init_handler" on the slave ones (to this point, I hope to be right, isn't it ?).
Yes. If I understand correctly, you observe that one processor calls ia64_monarch_init_handler(), and all the others call ia64_slave_init_handler(). So far, that is correct behavior.
Quoted 2 lines
What I've noticed using traces (and further an ITP tool) is that for each processor the "
ia64_monarch_init_handler" is ever called. :-(
Are you saying that more than one processor calls ia64_monarch_init_handler()? If so, I think something is broken. But I haven't seen that behavior. Here is ia64_slave_init_handler():
GLOBAL_ENTRY(ia64_slave_init_handler)
1: br.sptk 1b
END(ia64_slave_init_handler)
So I'd be surprised to see the slave processors do anything interesting.
I do know that we are missing some useful behavior in this area -- namely, we don't extract the min-state area for the slave processors, so we don't get backtraces for the currently-running tasks on them.
Bjorn
Quoted 6 lines
Quoted 2 lines
What I've noticed using traces (and further an ITP tool) is that for each processor the "
ia64_monarch_init_handler" is ever called. :-(Are you saying that more than one processor calls
ia64_monarch_init_handler()? If so, I think something is broken.
Or are you saying that NO processor gets to ia64_monarch_init_handler?
Just a sanity check ... when you set the ITP breakpoint to catch the entry ... you did set it at the *physical* address of this function?
-Tony
Quoted 9 lines
Quoted 7 lines
Then I push on the "dump" button that generates an INIT interruption to all the processors. This signal is then caught by PAL, and SAL which calls (with a reason [register GR11] equal to 2) "
ia64_monarch_init_handler" on the monarch processor and "ia64_slave_init_handler" on the slave ones (to this point, I hope to be right, isn't it ?).
Quoted 3 lines
Yes. If I understand correctly, you observe that one processor calls
ia64_monarch_init_handler(), and all the others callia64_slave_init_handler(). So far, that is correct behavior.
Hum, in fact, that's the behaviour I was expecting to observe... reading the kernel code.
Quoted 4 lines
Quoted 2 lines
What I've noticed using traces (and further an ITP tool) is that for each processor the "
ia64_monarch_init_handler" is ever called. :-(
Quoted 9 lines
Are you saying that more than one processor calls
ia64_monarch_init_handler()? If so, I think something is broken. But I haven't seen that behavior. Here isia64_slave_init_handler():
GLOBAL_ENTRY(ia64_slave_init_handler)
1: br.sptk 1b
END(ia64_slave_init_handler)
Right, I was just surprised to see 4 times the trace "Entered OS INIT handler" appearing on my console screen. Then, I ran a JTAG debugger to set breakpoints in the kernel code and noticed that all CPUs called the same function, e.g. "ia64_monarch_init_handler" (without using any Rendez-Vous mechanism ?!).
Quoted 6 lines
So I'd be surprised to see the slave processors do anything interesting.
I do know that we are missing some useful behavior in this area -- namely, we don't extract the min-state area for the slave processors, so we don't get backtraces for the currently-running tasks on them.
On that precise point, I thougt that PAL was recording register states in a processor dedicated area called "min-state area" for each processor receiving INIT signal (and in the present case all the processors catch it). Can you confirm this ?
In fact, I want to be able pushing the "dump" button to get the information for all the running processors (and not just one) and then determine for example which one is "deadlocking".
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
What I've noticed using traces (and further an ITP tool) is that for each processor the "
ia64_monarch_init_handler" is ever called. :-(Are you saying that more than one processor calls
ia64_monarch_init_handler()? If so, I think something is broken.Or are you saying that NO processor gets to
ia64_monarch_init_handler?
All the processors call the "ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).
Quoted 2 lines
Just a sanity check ... when you set the ITP breakpoint to catch the entry ... you did set it at the *physical* address of this function?
Yes, using Hardware breakpoints. When managing INIT interrupt, SAL calls the physical address of my function.
Quoted 4 lines · Francois Wellenreiter
All the processors call the "
ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).
What kind of machine is this? Linux registers different OS_INIT procedures for the monarch and non-monarch processors, so it sounds like a SAL bug if more than one CPU calls ia64_monarch_init_handler().
Quoted 10 lines · Bjorn Helgaas
Quoted 4 lines · Francois Wellenreiter
All the processors call the "
ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).What kind of machine is this? Linux registers different
OS_INITprocedures for the monarch and non-monarch processors, so it sounds like a SAL bug if more than one CPU callsia64_monarch_init_handler().
We see this also on Intel Tiger4's.
Jim
Quoted 9 lines
Quoted 6 lines
All the processors call the "
ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).What kind of machine is this?
I've done my tests first on Tiger4 (with a SAL version 3.00) and after on a Bull NOvaScale 5160.
These tests went to the same results...
Quoted 4 lines
Linux registers different
OS_INIT
procedures for the monarch and non-monarch processors, so it sounds like a SAL bug if more than one CPU callsia64_monarch_init_handler().
That's what I fear.
Francois WELLENREITER