Skip to contentSkip to conversation

Abnormal behaviour towards "INIT" interrupt management

DISCUSSION Francois Wellenreiter 4 participants 8 messages Mar 30, 2004 – Apr 1, 2004

lore
Share & tools
lore
#0 lore raw
			Hi,

	I've already sent this mail, but receiving no feedback, I try again 

(sorry for people who don't feel concerned by that stuff). I'd like to report what I estimate a potential bug in the INIT management (at the present time I don't know exactly what the origin of this problem is).

Testing the "dump" button on a 4-way Itanium-2 machine equipped with a SAL 3.0, I've noticed a not-inline with SAL specifications behaviour. Indeed, in the "ia64_mca_init" function (in arch/ia64/kernel/mca.c file) we register the functions "ia64_monarch_init_handler" and "ia64_slave_init_handler".

Then I push on the "dump" button that generates an INIT interruption to all the processors. This signal is then caught by PAL, and SAL which calls (with a reason [register GR11] equal to 2) "ia64_monarch_init_handler" on the monarch processor and "ia64_slave_init_handler" on the slave ones (to this point, I hope to be right, isn't it ?).
What I've noticed using traces (and further an ITP tool) is that for each processor the "ia64_monarch_init_handler" is ever called. :-( Could someone tell me if he already has encountered this problem or if it is an expected behavior ?

For information, I've done the same test with a 16-way Itanium-2 machine with a SAL 3.1 and the result is exactly the same.

Thanks for your help.

Best regards,

Francois WELLENREITER

#1 lore raw
Quoted 6 lines · Francois Wellenreiter

Then I push on the "dump" button that generates an INIT interruption to all the processors. This signal is then caught by PAL, and SAL which calls (with a reason [register GR11] equal to 2) "ia64_monarch_init_handler" on the monarch processor and "ia64_slave_init_handler" on the slave ones (to this point, I hope to be right, isn't it ?).

Yes. If I understand correctly, you observe that one processor calls ia64_monarch_init_handler(), and all the others call ia64_slave_init_handler(). So far, that is correct behavior.

Quoted 2 lines

What I've noticed using traces (and further an ITP tool) is that for each processor the "ia64_monarch_init_handler" is ever called. :-(

Are you saying that more than one processor calls ia64_monarch_init_handler()? If so, I think something is broken. But I haven't seen that behavior. Here is ia64_slave_init_handler():

GLOBAL_ENTRY(ia64_slave_init_handler)
1: br.sptk 1b
END(ia64_slave_init_handler)

So I'd be surprised to see the slave processors do anything interesting.

I do know that we are missing some useful behavior in this area -- namely, we don't extract the min-state area for the slave processors, so we don't get backtraces for the currently-running tasks on them.

Bjorn

#2 lore raw
In reply to Francois Wellenreiter
Quoted 6 lines
Quoted 2 lines

What I've noticed using traces (and further an ITP tool) is that for each processor the "ia64_monarch_init_handler" is ever called. :-(

Are you saying that more than one processor calls ia64_monarch_init_handler()? If so, I think something is broken.

Or are you saying that NO processor gets to ia64_monarch_init_handler?

Just a sanity check ... when you set the ITP breakpoint to catch the entry ... you did set it at the *physical* address of this function?

-Tony

#3 lore raw
In reply to Francois Wellenreiter
Quoted 9 lines
Quoted 7 lines

Then I push on the "dump" button that generates an INIT interruption to all the processors. This signal is then caught by PAL, and SAL which calls (with a reason [register GR11] equal to 2) "ia64_monarch_init_handler" on the monarch processor and "ia64_slave_init_handler" on the slave ones (to this point, I hope to be right, isn't it ?).

Quoted 3 lines

Yes. If I understand correctly, you observe that one processor calls ia64_monarch_init_handler(), and all the others call ia64_slave_init_handler(). So far, that is correct behavior.

Hum, in fact, that's the behaviour I was expecting to observe... reading the kernel code.

Quoted 4 lines
Quoted 2 lines

What I've noticed using traces (and further an ITP tool) is that for each processor the "ia64_monarch_init_handler" is ever called. :-(

Quoted 9 lines

Are you saying that more than one processor calls ia64_monarch_init_handler()? If so, I think something is broken. But I haven't seen that behavior. Here is ia64_slave_init_handler():

GLOBAL_ENTRY(ia64_slave_init_handler)
1: br.sptk 1b
END(ia64_slave_init_handler)

Right, I was just surprised to see 4 times the trace "Entered OS INIT handler" appearing on my console screen. Then, I ran a JTAG debugger to set breakpoints in the kernel code and noticed that all CPUs called the same function, e.g. "ia64_monarch_init_handler" (without using any Rendez-Vous mechanism ?!).

Quoted 6 lines

So I'd be surprised to see the slave processors do anything interesting.

I do know that we are missing some useful behavior in this area -- namely, we don't extract the min-state area for the slave processors, so we don't get backtraces for the currently-running tasks on them.

On that precise point, I thougt that PAL was recording register states in a processor dedicated area called "min-state area" for each processor receiving INIT signal (and in the present case all the processors catch it). Can you confirm this ?

In fact, I want to be able pushing the "dump" button to get the information for all the running processors (and not just one) and then determine for example which one is "deadlocking".

#4 lore raw
In reply to Francois Wellenreiter
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

What I've noticed using traces (and further an ITP tool) is that for each processor the "ia64_monarch_init_handler" is ever called. :-(

Are you saying that more than one processor calls ia64_monarch_init_handler()? If so, I think something is broken.

Or are you saying that NO processor gets to ia64_monarch_init_handler?

All the processors call the "ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).

Quoted 2 lines

Just a sanity check ... when you set the ITP breakpoint to catch the entry ... you did set it at the *physical* address of this function?

Yes, using Hardware breakpoints. When managing INIT interrupt, SAL calls the physical address of my function.

#5 lore raw
In reply to Francois Wellenreiter
Quoted 4 lines · Francois Wellenreiter

All the processors call the "ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).

What kind of machine is this? Linux registers different OS_INIT procedures for the monarch and non-monarch processors, so it sounds like a SAL bug if more than one CPU calls ia64_monarch_init_handler().

#6 lore raw
In reply to Francois Wellenreiter
Quoted 10 lines · Bjorn Helgaas
Quoted 4 lines · Francois Wellenreiter

All the processors call the "ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).

What kind of machine is this? Linux registers different OS_INIT procedures for the monarch and non-monarch processors, so it sounds like a SAL bug if more than one CPU calls ia64_monarch_init_handler().

We see this also on Intel Tiger4's.

Jim

#7 lore raw
In reply to Francois Wellenreiter
Quoted 9 lines
Quoted 6 lines

All the processors call the "ia64_monarch_init_handler",
the result is that the same trace appears many times (1 per CPU in fact) and a kernel oops occurs (I think it is due to a concurrent access to the same stack without any lock mechanism).

What kind of machine is this?

I've done my tests first on Tiger4 (with a SAL version 3.00) and after on a Bull NOvaScale 5160.
These tests went to the same results...

Quoted 4 lines

Linux registers different OS_INIT
procedures for the monarch and non-monarch processors, so it sounds like a SAL bug if more than one CPU calls ia64_monarch_init_handler().

That's what I fear.

Francois WELLENREITER