Skip to contentSkip to conversation

Question re HYP mode and IRQ/FIQ stack setting

U-Boot Albert ARIBAUD 3 participants 8 messages Nov 12, 2013 – Nov 13, 2013

lore
Share & tools
lore
#0 lore raw

(Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)

Hello,

I am working on changing the way IRQ/FIQ stacks are set up, from "on-the-fly in a hurry while in the handler" to "during init, so that when entering the handler, the stack is already correct".

Setting the stack then requires switching from the current mode (in most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to set the right banked SP, then back into the original mode.

However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the possibility of U-Boot being started in HYP mode (0x1A) is considered and, if in HYP mode, no switch to SVC32 is performed.

I understand that the problem here is, if we drop from HYP to SVC32, then we cannot go back to HYP, and we want to be able to remain in HYP.

Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if we do such a drop, are we prevented from rising back from IRQ or FIQ mode to HYP? I seem to remember such an issue, but I am no specialist in HYP, so any help is welcome.

Amicalement,

Signature · 1 line
Albert.
#1 lore raw
Quoted 12 lines · Albert ARIBAUD

(Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)

Hello,

I am working on changing the way IRQ/FIQ stacks are set up, from "on-the-fly in a hurry while in the handler" to "during init, so that when entering the handler, the stack is already correct".

Setting the stack then requires switching from the current mode (in most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to set the right banked SP, then back into the original mode.

Quoted 6 lines

However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the possibility of U-Boot being started in HYP mode (0x1A) is considered and, if in HYP mode, no switch to SVC32 is performed.

I understand that the problem here is, if we drop from HYP to SVC32, then we cannot go back to HYP, and we want to be able to remain in HYP.

correct (not without setting up a trap handler in Hyp mode and trapping to Hyp mode)

Quoted 5 lines

Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if we do such a drop, are we prevented from rising back from IRQ or FIQ mode to HYP? I seem to remember such an issue, but I am no specialist in HYP, so any help is welcome.

Yes, it also applies. Hyp is strictly more privileged (PL2) than all the PL1 modes (SVC, SYS, IRQ, FIQ, ABT, UND) and therefore requires a trap to go from PL1 to PL2 (basically this is how hardware protection works - just like with syscalls from user mode to PL1).

You can use MSR and MRS instructions to access the IRQ and FIQ registers directly from Hyp mode though.

-Christoffer

#2 lore raw
Quoted 13 lines · Christoffer Dall
Quoted 12 lines · Albert ARIBAUD

(Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)

Hello,

I am working on changing the way IRQ/FIQ stacks are set up, from "on-the-fly in a hurry while in the handler" to "during init, so that when entering the handler, the stack is already correct".

Setting the stack then requires switching from the current mode (in most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to set the right banked SP, then back into the original mode.

Quoted 6 lines
Quoted 6 lines

However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the possibility of U-Boot being started in HYP mode (0x1A) is considered and, if in HYP mode, no switch to SVC32 is performed.

I understand that the problem here is, if we drop from HYP to SVC32, then we cannot go back to HYP, and we want to be able to remain in HYP.

Right, that is to keep the HYP mode in case the firmware already enabled it. This is for instance the case on the new Calxeda Midway. Actually this approach will become more widespread, since it is required to provide proper PSCI support (which needs to run in secure state, so requires an even higher privilege level than HYP: EL3 in the new ARM speak).

Quoted 9 lines

correct (not without setting up a trap handler in Hyp mode and trapping to Hyp mode)

Quoted 5 lines

Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if we do such a drop, are we prevented from rising back from IRQ or FIQ mode to HYP? I seem to remember such an issue, but I am no specialist in HYP, so any help is welcome.

Quoted 4 lines

Yes, it also applies. Hyp is strictly more privileged (PL2) than all the PL1 modes (SVC, SYS, IRQ, FIQ, ABT, UND) and therefore requires a trap to go from PL1 to PL2 (basically this is how hardware protection works - just like with syscalls from user mode to PL1).

Thanks Christoffer for clarifying this, I wasn't sure about FIQ, but of course your explanation (EL1 vs. EL2) makes totally sense.

But I wonder what happens when we enter FIQ or IRQ due to an actual interrupt. Will the CPU return into HYP mode when the handler returns? That is subject to some HYP mode register configuration, right?

Quoted 2 lines

You can use MSR and MRS instructions to access the IRQ and FIQ registers directly from Hyp mode though.

Albert,
so does "msr sp_{fiq,irq}, r<n>" fix your problem? Or do you still need to actually go into one of these modes for further setup?

Regards,
Andre.

#3 lore raw
Quoted 25 lines · Andre Przywara
Quoted 16 lines · Christoffer Dall
Quoted 13 lines · Albert ARIBAUD

(Cc:ing Andre and Christoffer as they have discussed HYP on the ML.)

Hello,

I am working on changing the way IRQ/FIQ stacks are set up, from "on-the-fly in a hurry while in the handler" to "during init, so that when entering the handler, the stack is already correct".

Setting the stack then requires switching from the current mode (in most cases, SVC32, 0x13) to IRQ (0x11) or FIQ (0x12) mode, in order to set the right banked SP, then back into the original mode.

Quoted 6 lines
Quoted 6 lines

However, in the first lines of reset in arch/arm/cpu/armv7/start.S, the possibility of U-Boot being started in HYP mode (0x1A) is considered and, if in HYP mode, no switch to SVC32 is performed.

I understand that the problem here is, if we drop from HYP to SVC32, then we cannot go back to HYP, and we want to be able to remain in HYP.

Quoted 6 lines

Right, that is to keep the HYP mode in case the firmware already enabled it. This is for instance the case on the new Calxeda Midway. Actually this approach will become more widespread, since it is required to provide proper PSCI support (which needs to run in secure state, so requires an even higher privilege level than HYP: EL3 in the new ARM speak).

What Andre is referring to here is that the PSCI specs mandates the CPUs that a are turned on using PSCI go through the highest privilege level implemented in the non-secure state. I couldn't actually find the place in the PSCI specs where this is also true for the boot CPU, but granted, anything else will be quite strange.

So to stay in 32-bit ARMv7 terminology here, a likely case for systems with PSCI support in the firmware is that all cores will enter U-Boot in non-secure PL2 (hyp) mode and power management is done by calling SMC calls to the firmware that sits in the secure monitor.

Quoted 17 lines
Quoted 11 lines

correct (not without setting up a trap handler in Hyp mode and trapping to Hyp mode)

Quoted 5 lines

Does this also apply to dropping from HYP to IRQ or FIQ mode, i.e., if we do such a drop, are we prevented from rising back from IRQ or FIQ mode to HYP? I seem to remember such an issue, but I am no specialist in HYP, so any help is welcome.

Quoted 4 lines

Yes, it also applies. Hyp is strictly more privileged (PL2) than all the PL1 modes (SVC, SYS, IRQ, FIQ, ABT, UND) and therefore requires a trap to go from PL1 to PL2 (basically this is how hardware protection works - just like with syscalls from user mode to PL1).

Quoted 7 lines

Thanks Christoffer for clarifying this, I wasn't sure about FIQ, but of course your explanation (EL1 vs. EL2) makes totally sense.

But I wonder what happens when we enter FIQ or IRQ due to an actual interrupt. Will the CPU return into HYP mode when the handler returns? That is subject to some HYP mode register configuration, right?

Not quite, interrupts in Hyp mode will always be taken to Hyp mode (unless it's secure interrupts, which are taken to monitor mode). In fact, when you have the virtualization extensions, you also have the security extensions, and I think all FIQs go to the secure mode here, so you can't ever see an FIQ in Hyp mode. I may be mistaken here, someone should look it up and verify.

Now, you can configure Hyp mode to trap interrupts raised while in any PL1 or PL0 mode to Hyp mode.

I suspect that if you are in Hyp mode, you should not worry about FIQ/IRQ mode, but just make sure to configure Hyp mode properly to handle interrupts. (it's a separate entry in the exception vector and you probably need to look at the HSR register whn you've taken an interrupt). So, as Andre suggests below, it depends on your use case.

Quoted 12 lines
Quoted 3 lines

You can use MSR and MRS instructions to access the IRQ and FIQ registers directly from Hyp mode though.

Albert,
so does "msr sp_{fiq,irq}, r<n>" fix your problem? Or do you still need to actually go into one of these modes for further setup?

Regards,
Andre.

#4 lore raw

Hi Christoffer,

Quoted 5 lines · Christoffer Dall

I suspect that if you are in Hyp mode, you should not worry about FIQ/IRQ mode, but just make sure to configure Hyp mode properly to handle interrupts. (it's a separate entry in the exception vector and you probably need to look at the HSR register whn you've taken an interrupt). So, as Andre suggests below, it depends on your use case.

Ok, so les me try to sum that up from my perspective (IRQ and FIQ stacks in U-Boot):

Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.

For targets which run in SVC32 mode: aborts execute in abort mode, IRQs in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.

For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP mode, using always the same stack.

In both types of targers: prefetch and data aborts, FIQs and IRQs execute through the usual vectors; in SVC32 mode, because there's no other way; in HYP mode, because exceptions occurring while in HYP mode use the normal vector, not the HYP vector.

Correct?

If so, then U-Boot stack requirements for SVC32 and HYP mode are as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main (SVC32) stack does not have to accommodate for interrupt handler context storage. In HYP mode, the only stack is the HYP one and exception handlers will use it too, so it has to accommodate for their context storage.

Still correct?

Amicalement,

Signature · 1 line
Albert.
#5 lore raw
Quoted 11 lines · Albert ARIBAUD

Hi Christoffer,

Quoted 5 lines · Christoffer Dall

I suspect that if you are in Hyp mode, you should not worry about FIQ/IRQ mode, but just make sure to configure Hyp mode properly to handle interrupts. (it's a separate entry in the exception vector and you probably need to look at the HSR register whn you've taken an interrupt). So, as Andre suggests below, it depends on your use case.

Quoted 7 lines

Ok, so les me try to sum that up from my perspective (IRQ and FIQ stacks in U-Boot):

Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.

For targets which run in SVC32 mode: aborts execute in abort mode, IRQs in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.

correct

Quoted 3 lines

For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP mode, using always the same stack.

correct.

Quoted 8 lines

In both types of targers: prefetch and data aborts, FIQs and IRQs execute through the usual vectors; in SVC32 mode, because there's no other way; in HYP mode, because exceptions occurring while in HYP mode use the normal vector, not the HYP vector.

Correct?

no, this sounds fishy. Look at the ARM ARMv7 (DDI 0406C.b) page B1-1166. Table B1-3 will tell you what you need to know.

Hyp mode has its own vectors, pointed to by the HVBAR control register.

I strongly suggest you familiarize yourself with these parts of the ARM ARM before writing code to that effect.

Quoted 7 lines

If so, then U-Boot stack requirements for SVC32 and HYP mode are as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main (SVC32) stack does not have to accommodate for interrupt handler context storage. In HYP mode, the only stack is the HYP one and exception handlers will use it too, so it has to accommodate for their context storage.

That is correct, your Hyp mode stack should always be valid (have a page or so, like the kernel) and then you can always push things on there, even when taking an exception.

-Christoffer

#6 lore raw

Hi Christoffer,

Quoted 37 lines · Christoffer Dall
Quoted 11 lines · Albert ARIBAUD

Hi Christoffer,

Quoted 5 lines · Christoffer Dall

I suspect that if you are in Hyp mode, you should not worry about FIQ/IRQ mode, but just make sure to configure Hyp mode properly to handle interrupts. (it's a separate entry in the exception vector and you probably need to look at the HSR register whn you've taken an interrupt). So, as Andre suggests below, it depends on your use case.

Quoted 7 lines

Ok, so les me try to sum that up from my perspective (IRQ and FIQ stacks in U-Boot):

Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.

For targets which run in SVC32 mode: aborts execute in abort mode, IRQs in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.

correct

Quoted 3 lines

For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP mode, using always the same stack.

correct.

Quoted 8 lines

In both types of targers: prefetch and data aborts, FIQs and IRQs execute through the usual vectors; in SVC32 mode, because there's no other way; in HYP mode, because exceptions occurring while in HYP mode use the normal vector, not the HYP vector.

Correct?

Quoted 7 lines

no, this sounds fishy. Look at the ARM ARMv7 (DDI 0406C.b) page B1-1166. Table B1-3 will tell you what you need to know.

Hyp mode has its own vectors, pointed to by the HVBAR control register.

I strongly suggest you familiarize yourself with these parts of the ARM ARM before writing code to that effect.

To clarify my meaning: I wasn't speaking about the exceptions base address, but about the exception vector offset. IOW, I did not mean to say that in HYP mode, aborts, FIQs, IRQs etc use the SVC32 table of vectors rather than the HYP table. I meant to say that e.g. an IRQ will fire at offset 0x18, whether it fires while from SVC32 into IRQ mode, or from HYP mode into HYP mode. This seemed consistent with figure B1-8 on page B1-1179 of DDI 0406C.b, respectively first and sixth "exit box" on the right.

Is this clearer and more correct?

Quoted 10 lines
Quoted 7 lines

If so, then U-Boot stack requirements for SVC32 and HYP mode are as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main (SVC32) stack does not have to accommodate for interrupt handler context storage. In HYP mode, the only stack is the HYP one and exception handlers will use it too, so it has to accommodate for their context storage.

That is correct, your Hyp mode stack should always be valid (have a page or so, like the kernel) and then you can always push things on there, even when taking an exception.

Thanks Christoffer! This should allow me to refactor interrupt stack settings as I intended.

Quoted 1 line

-Christoffer

Amicalement,

Signature · 1 line
Albert.
#7 lore raw
Quoted 59 lines · Albert ARIBAUD

Hi Christoffer,

Quoted 37 lines · Christoffer Dall
Quoted 11 lines · Albert ARIBAUD

Hi Christoffer,

Quoted 5 lines · Christoffer Dall

I suspect that if you are in Hyp mode, you should not worry about FIQ/IRQ mode, but just make sure to configure Hyp mode properly to handle interrupts. (it's a separate entry in the exception vector and you probably need to look at the HSR register whn you've taken an interrupt). So, as Andre suggests below, it depends on your use case.

Quoted 7 lines

Ok, so les me try to sum that up from my perspective (IRQ and FIQ stacks in U-Boot):

Some ARM U-Boot targets run in SVC32 mode, and some in HYP mode.

For targets which run in SVC32 mode: aborts execute in abort mode, IRQs in IRQ mode, FIQs in FIQ mode, etc., each mode having its own stack.

correct

Quoted 3 lines

For targets which run in HYP mode: aborts, FIQ and IRQs all run in HYP mode, using always the same stack.

correct.

Quoted 8 lines

In both types of targers: prefetch and data aborts, FIQs and IRQs execute through the usual vectors; in SVC32 mode, because there's no other way; in HYP mode, because exceptions occurring while in HYP mode use the normal vector, not the HYP vector.

Correct?

Quoted 7 lines

no, this sounds fishy. Look at the ARM ARMv7 (DDI 0406C.b) page B1-1166. Table B1-3 will tell you what you need to know.

Hyp mode has its own vectors, pointed to by the HVBAR control register.

I strongly suggest you familiarize yourself with these parts of the ARM ARM before writing code to that effect.

To clarify my meaning: I wasn't speaking about the exceptions base address, but about the exception vector offset. IOW, I did not mean to say that in HYP mode, aborts, FIQs, IRQs etc use the SVC32 table of vectors rather than the HYP table. I meant to say that e.g. an IRQ will fire at offset 0x18, whether it fires while from SVC32 into IRQ mode, or from HYP mode into HYP mode. This seemed consistent with figure B1-8 on page B1-1179 of DDI 0406C.b, respectively first and sixth "exit box" on the right.

Quoted 2 lines

Is this clearer and more correct?

yes, now I understand. It's correct.

Quoted 19 lines
Quoted 10 lines
Quoted 7 lines

If so, then U-Boot stack requirements for SVC32 and HYP mode are as follows: in SVC32, we need separate IRQ and FIQ stacks, and the main (SVC32) stack does not have to accommodate for interrupt handler context storage. In HYP mode, the only stack is the HYP one and exception handlers will use it too, so it has to accommodate for their context storage.

That is correct, your Hyp mode stack should always be valid (have a page or so, like the kernel) and then you can always push things on there, even when taking an exception.

Thanks Christoffer! This should allow me to refactor interrupt stack settings as I intended.

Quoted 1 line

-Christoffer

Amicalement,
--
Albert.