armv7 + lpae broken in 4.1.12+ ?

I was recently building a 4.1.12 kernel for a generic V7 platform with
LPAE. I could not get it to boot and then discovered two things:

- the early printk code in tty/serial call ioremap very early and it fails because there's no heap
- the code in proc-v7-3level.S/cpu_v7_set_pte_ext doesn't work

The printk issue is no big deal. The pte issue is more important.

I found that the pte entries were garbage, which really confused me. I finally discovered that the code in proc-v7-3level.S expects data in r0, r2 & r3, but in fact it comes in on r0, r1 & r2.

My guess is that this is some vestige of the change made to cpu_v7_switch_mm() some time ago to make the PA a 64 bit value (hence using r0,r1). But this change was not made to cpu_v7_set_pte_ext, so it writes garbage.

lol. I guess I'm the only one who turns on LPAE for v7 cpu's...

The simple fix is to add "mov r3, r2; mov r2,r1" at the begining of cpu_v7_set_pte_ext(), but that's a hack (but it does solve the problem and the kernel boots).

I suspect the "right thing" is to make the prototype for cpu_v7_set_pte_ext() in proc-fns.h use a phys_addr_t but I have not tried that yet.

Can someone confirm that what I am seeing is correct?

-brad

This seems wrong. If you're building LPAE, then you should have AEABI enabled, which means you're using EABI. (ARM_LPAE should depend on AEABI, but it doesn't.)

EABI places 64-bit arguments in an even/odd register pair. So:

void cpu_set_pte_ext(pte_t *ptep, pte_t pte, unsigned int ext);

ptep is in r0, and if pte_t is 64-bit, then pte is passed in r2 and r3. Not r1 and r2. ext is passed on the stack.

FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

FWIW, anybody using virtualisation with a v7 CPU has LPAE enabled. Whilst that's admittedly not the majority of users, you're certainly not alone.

You are, however, the only person who seems to be hitting this problem.

As Russell says, AEABI requires the 64-bit pte value to be passed in r2 and r3 for cpu_v7_set_pte_ext:

C.3 If the argument requires double-word alignment (8-byte), the NCRN
is rounded up to the next even register number.

... so this would actually cause breakage for other people.

Which toolchain are you using?

Will

On 11/27/15 4:36 AM, Will Deacon wrote:> As Russell says, AEABI requires the 64-bit pte value to be passed in > r2 and r3 for cpu_v7_set_pte_ext:

NCRN is rounded up to the next even register number. > >> The simple fix is to add "mov r3, r2; mov r2,r1" at the begining >> of cpu_v7_set_pte_ext(), but that's a hack (but it does solve the >> problem and the kernel boots). > > ... so this would actually cause breakage for other people. > > Which toolchain are you using? [sorry for the re-send; I keep getting bounced by the email.html filter; not sure why]

That all makes sense. I'm using a linaro toolchain (which I'll admit I grabbed because I was lazy)

gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux/bin/arm-linux-gnueabihf-

It seems really odd. I would have thought "gnueabihf" would be EABI, so a 64 bit arg should be in r2,r3; but it's not. It's in r1, r2.

could something in my .config cause this? (I'd be surprised if that were the case, but...).

I'm happy to try a better toolchain if you can point me to one (binary).

-brad

It doesn't matter what toolchain you have. We don't rely on how the toolchain happened to be configured to build the kernel, we explicitly state to the compiler what we want it to do.

Many people tend to be under the incorrect impression that they need specific compilers to build the kernel in certain ways. This is untrue, and has been untrue for the last 20 years.

If you have CONFIG_AEABI disabled, you will get an OABI kernel, or an error from the toolchain if the toolchain doesn't support it. If you have CONFIG_AEABI enabled, you will get an EABI kernel, or an error from the toolchain if the toolchain doesn't support it.

As I said in my previous reply, LPAE really ought to depend on AEABI being enabled.

Please check whether you have CONFIG_AEABI enabled.

FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

Yes. That was it - sorry. I didn't realize that was a config item (but with your explanation and a look at the makefile I see what it's doing). I guess I was blissfully unaware that that option even existed. But not any more :-)

thanks!

That option seem on by default for thumb2 kernels, but not for aarch32.

-brad

I'm surprised, because if you try booting an OABI only kernel with an EABI userspace, it'll hang after getting to userspace. Ditto the other way around, unless you have OABI_COMPAT enabled. However, I guess you're not getting that far yet...

FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.