AMR: sun7i: CPU hotplug support?

DISCUSSION Jan Kiszka 3 participants 9 messages Nov 9, 2014 – Nov 10, 2014

Share & tools

Jan Kiszka

author

2014-11-09 19:35 UTC #0 lore raw

Hi,

did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.

Thanks,
Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20141109/8916a565/attachment.sig>

Maxime Ripard

2014-11-09 23:17 UTC #1 lore raw

Hi Jan,

Quoted 5 lines · Jan Kiszka

did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.

Having hotplug support would indeed be very welcome.

However, it should be done in u-boot, through PSCI, and not in the kernel itself.

As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144

Maxime

Signature · 10 lines

Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20141110/7210680c/attachment.sig>

Jan Kiszka

author

2014-11-10 06:03 UTC #2 lore raw

Quoted 9 lines · Maxime Ripard

Hi Jan,
Quoted 5 lines · Jan Kiszka
did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.

Quoted 8 lines

Having hotplug support would indeed be very welcome.
However, it should be done in u-boot, through PSCI, and not in the kernel itself.
As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144

OK - I guess I will need a little guidance in then: Is there a good reference board to study and to derive from? And maybe also: What is missing or not working in that u-boot branch? If I get this interface right, I just takes some device tree bits to enable this for the kernel afterward, correct?

Thanks,
Jan

PS: That the GMAC on B-Pi has some TX reliability issues, both with kernel and u-boot, is known, right? If not, I'll open a separate thread.

Jan Kiszka

author

2014-11-10 08:25 UTC #3 lore raw

Quoted 19 lines · Jan Kiszka

Quoted 9 lines · Maxime Ripard
Hi Jan,
Quoted 5 lines · Jan Kiszka
did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.
Quoted 8 lines
Having hotplug support would indeed be very welcome.
However, it should be done in u-boot, through PSCI, and not in the kernel itself.
As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144

Quoted 5 lines

OK - I guess I will need a little guidance in then: Is there a good reference board to study and to derive from? And maybe also: What is missing or not working in that u-boot branch? If I get this interface right, I just takes some device tree bits to enable this for the kernel afterward, correct?

Started to play with that patch in naive ways: CPU0 locks up when offlining CPU1 - unless I disable the FIQ signal from CPU1. Then it "works", both offlining and onlining again. However, I suspect that this only parks CPU1 in wfi and does not do anything interesting to it.

Jan

Marc Zyngier

2014-11-10 09:17 UTC #4 lore raw

Quoted 17 lines · Jan Kiszka

Quoted 16 lines · Jan Kiszka
Quoted 15 lines · Maxime Ripard
Hi Jan,
Quoted 6 lines · Jan Kiszka
did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.
Having hotplug support would indeed be very welcome.
However, it should be done in u-boot, through PSCI, and not in the kernel itself.

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144

OK - I guess I will need a little guidance in then: Is there a good

Quoted 10 lines

Quoted 4 lines
reference board to study and to derive from? And maybe also: What is missing or not working in that u-boot branch? If I get this interface right, I just takes some device tree bits to enable this for the kernel afterward, correct?
Started to play with that patch in naive ways: CPU0 locks up when offlining CPU1 - unless I disable the FIQ signal from CPU1. Then it "works", both offlining and onlining again. However, I suspect that this only parks CPU1 in wfi and does not do anything interesting to it.

Here's how this is supposed to work:
- CPU1 sends a FIQ to CPU0, bringing it into secure mode.
- CPU0 then kills CPU1 by doing the magic incantations on the power controller

What is missing here is all the cache cleaning before signalling CPU0. If you add that, things should look a lot better (patches welcome).

Thanks,

M.

Signature · 1 line

Jazz is not dead. It just smells funny...

Jan Kiszka

author

2014-11-10 09:36 UTC #5 lore raw

Quoted 36 lines · Marc Zyngier

Quoted 17 lines · Jan Kiszka
Quoted 16 lines · Jan Kiszka
Quoted 15 lines · Maxime Ripard
Hi Jan,
Quoted 6 lines · Jan Kiszka
did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.
Having hotplug support would indeed be very welcome.
However, it should be done in u-boot, through PSCI, and not in the kernel itself.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144
OK - I guess I will need a little guidance in then: Is there a good
Quoted 10 lines
Quoted 4 lines
reference board to study and to derive from? And maybe also: What is missing or not working in that u-boot branch? If I get this interface right, I just takes some device tree bits to enable this for the kernel afterward, correct?
Started to play with that patch in naive ways: CPU0 locks up when offlining CPU1 - unless I disable the FIQ signal from CPU1. Then it "works", both offlining and onlining again. However, I suspect that this only parks CPU1 in wfi and does not do anything interesting to it.

Quoted 7 lines

Here's how this is supposed to work:
- CPU1 sends a FIQ to CPU0, bringing it into secure mode.
- CPU0 then kills CPU1 by doing the magic incantations on the power controller
What is missing here is all the cache cleaning before signalling CPU0. If you add that, things should look a lot better (patches welcome).

Unsure about this, or maybe this was too simplistic: I added calls to u-boot's flush_dcache_all and invalidate_icache_all (right after disabling the cache, just like the vendor kernel does), but CPU0 still locks up. I suspect there is still a bug in the FIQ handling. There is also a suspicious single "@" printed on the console. I'll play with the FIQ handler a bit.

Jan

Marc Zyngier

2014-11-10 09:52 UTC #6 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 21 lines · Jan Kiszka

Quoted 20 lines · Marc Zyngier
Quoted 19 lines · Jan Kiszka
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Maxime Ripard
Hi Jan,
Quoted 7 lines · Jan Kiszka
did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.
Having hotplug support would indeed be very welcome.
However, it should be done in u-boot, through PSCI, and not in the kernel itself.

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144

OK - I guess I will need a little guidance in then: Is there a good

Quoted 19 lines

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
reference board to study and to derive from? And maybe also:
What is missing or not working in that u-boot branch? If I get this interface right, I just takes some device tree bits to enable this for the kernel afterward, correct?
Started to play with that patch in naive ways: CPU0 locks up when offlining CPU1 - unless I disable the FIQ signal from CPU1. Then it "works", both offlining and onlining again. However, I suspect that this only parks CPU1 in wfi and does not do anything interesting to it.
Quoted 7 lines
Here's how this is supposed to work: - CPU1 sends a FIQ to CPU0, bringing it into secure mode. - CPU0 then kills CPU1 by doing the magic incantations on the power controller
What is missing here is all the cache cleaning before signalling CPU0. If you add that, things should look a lot better (patches welcome).

Quoted 6 lines

Unsure about this, or maybe this was too simplistic: I added calls to u-boot's flush_dcache_all and invalidate_icache_all (right after disabling the cache, just like the vendor kernel does), but CPU0 still locks up. I suspect there is still a bug in the FIQ handling. There is also a suspicious single "@" printed on the console. I'll play with the FIQ handler a bit.

The '@' is just my own debug stuff, and might be causing issues too.

Now, you have to realise that by the time you call into this code, u-boot itself is long gone. Only the tiny bit of code dealing with PSCI still lives in a bank of static, secure memory. So calling into u-boot for anything is doomed. You need to actually put the code inside the PSCI backend.

Thanks,

M.

- --
Jazz is not dead. It just smells funny...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBAgAGBQJUYIrRAAoJECPQ0LrRPXpDyjQQAMpDGLUwVODYi0lHFb20axqZ Jjptkh6L2UJ2D2a2Ow17umrnIJqFjvVDasKtoxrBj79ya7LvHdfti4cFAjhZbh53 dOYmGqWHLP5N2Ue5KGs6zEpmW3bkUQRM2n03pTsqs907zN3XrjhfdtEqun5qAKTX Kbk2X/CieQnGx9rYeZUAIiJm0qO4e9ijKFQCvxRDItr0Eb4voRwRCtWCiD530ZTc 34G7fpqTR4HKNRsaLeBRCUz3dJXP7KgW3VymIu35fAAyUlt0oMENrcX9qKb6AHR7 KM0JXkts+bMmxeddRPzEEikHvRfSgbFxTY0+6vsqi4DrsuihI5OG+6VqckZRrfpV S74IkyEyBscSUhkRcXUEWsly9AcK7gvxlK0DqyR+5v+OOQd6RGfAfA8xYT6GbtpN EZ8suyFWsl55QdolWx5Pqb8VfL0Z70AteWcMhJia6eg69hGo1YJVM6DJ93rvSwzx x1UniBVBkp9/jZYJPgrCw0ywGXbWex7OcnTCscGAChkFMgFT/LiBCF16PKoyLhYu oGoClNbwlLgo4xAoeB/pDLyO0eaOLQS7/w2LdLZGbPXMVqcXkYVS+vkdkfe099Hb dYXOCkI8mcxDDisodtSg7W797bJMfjnCmfrXJuyfTbJW2nSrYgqw5yZNhhEhlWB3 n24nemun99pZ8di2HBSl
=C/8W
-----END PGP SIGNATURE-----

Jan Kiszka

author

2014-11-10 10:33 UTC #7 lore raw

Quoted 57 lines · Marc Zyngier

Quoted 21 lines · Jan Kiszka
Quoted 20 lines · Marc Zyngier
Quoted 19 lines · Jan Kiszka
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Maxime Ripard
Hi Jan,
Quoted 7 lines · Jan Kiszka
did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.
Having hotplug support would indeed be very welcome.
However, it should be done in u-boot, through PSCI, and not in the kernel itself.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144
OK - I guess I will need a little guidance in then: Is there a good
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
reference board to study and to derive from? And maybe also:
What is missing or not working in that u-boot branch? If I get this interface right, I just takes some device tree bits to enable this for the kernel afterward, correct?
Started to play with that patch in naive ways: CPU0 locks up when offlining CPU1 - unless I disable the FIQ signal from CPU1. Then it "works", both offlining and onlining again. However, I suspect that this only parks CPU1 in wfi and does not do anything interesting to it.
Quoted 7 lines
Here's how this is supposed to work: - CPU1 sends a FIQ to CPU0, bringing it into secure mode. - CPU0 then kills CPU1 by doing the magic incantations on the power controller
What is missing here is all the cache cleaning before signalling CPU0. If you add that, things should look a lot better (patches welcome).
Quoted 6 lines
Unsure about this, or maybe this was too simplistic: I added calls to u-boot's flush_dcache_all and invalidate_icache_all (right after disabling the cache, just like the vendor kernel does), but CPU0 still locks up. I suspect there is still a bug in the FIQ handling. There is also a suspicious single "@" printed on the console. I'll play with the FIQ handler a bit.

Quoted 7 lines

The '@' is just my own debug stuff, and might be causing issues too.
Now, you have to realise that by the time you call into this code, u-boot itself is long gone. Only the tiny bit of code dealing with PSCI still lives in a bank of static, secure memory. So calling into u-boot for anything is doomed. You need to actually put the code inside the PSCI backend.

OK - seems like quite a bit of code needs to be pulled over...

Meanwhile I'm still starring at psci_fiq_enter:

	...
	movw	r8, #(GICC_BASE & 0xffff)
	movt	r8, #(GICC_BASE >> 16)
	ldr	r9, [r8, #GICC_IAR]
	movw	r10, #0x3ff
	movt	r10, #0
	cmp	r9, r10
	bne	out1
	movw	r10, #0x3fe
	cmp	r9, r10
	bne	out1
	...

How can GICC_IAR be 0x3ff and 0x3fe at the same time? These tests seems bogus. What was the actual intention here?

Jan

Marc Zyngier

2014-11-10 11:07 UTC #8 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 23 lines · Jan Kiszka

Quoted 22 lines · Marc Zyngier
Quoted 21 lines · Jan Kiszka
Quoted 20 lines · Marc Zyngier
Quoted 19 lines · Jan Kiszka
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Maxime Ripard
Hi Jan,
Quoted 7 lines · Jan Kiszka
did anyone already happen to look into enabling CPU hotplug for the Allwinner A20 in upstream? I'm currently running the sunxi-next branch on Banana Pi, and echo 0 > .../cpu1/online just hangs the system. The old 3.4 LeMaker kernel works fine in this regard. I can try to look into details and port things over, just want to avoid duplicate efforts.
Having hotplug support would indeed be very welcome.
However, it should be done in u-boot, through PSCI, and not in the kernel itself.

Quoted 10 lines

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
As far as I'm aware, no one worked actively on it, beside some WIP commit from Marc a while ago:
https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/commit/?h=wip/psci&id=45379c0f9cf812f0f62722f4015ec907fa5dc144

OK - I guess I will need a little guidance in then: Is there a good

Quoted 38 lines

Quoted 27 lines
Quoted 20 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
reference board to study and to derive from? And maybe
also: What is missing or not working in that u-boot branch? If I get this interface right, I just takes some device tree bits to enable this for the kernel afterward, correct?
Started to play with that patch in naive ways: CPU0 locks up when offlining CPU1 - unless I disable the FIQ signal from CPU1. Then it "works", both offlining and onlining again. However, I suspect that this only parks CPU1 in wfi and does not do anything interesting to it.
Quoted 7 lines
Here's how this is supposed to work: - CPU1 sends a FIQ to CPU0, bringing it into secure mode. - CPU0 then kills CPU1 by doing the magic incantations on the power controller
What is missing here is all the cache cleaning before signalling CPU0. If you add that, things should look a lot better (patches welcome).
Quoted 6 lines
Unsure about this, or maybe this was too simplistic: I added calls to u-boot's flush_dcache_all and invalidate_icache_all (right after disabling the cache, just like the vendor kernel does), but CPU0 still locks up. I suspect there is still a bug in the FIQ handling. There is also a suspicious single "@" printed on the console. I'll play with the FIQ handler a bit.
Quoted 8 lines
The '@' is just my own debug stuff, and might be causing issues too.
Now, you have to realise that by the time you call into this code, u-boot itself is long gone. Only the tiny bit of code dealing with PSCI still lives in a bank of static, secure memory. So calling into u-boot for anything is doomed. You need to actually put the code inside the PSCI backend.
OK - seems like quite a bit of code needs to be pulled over...

Quoted 7 lines

Meanwhile I'm still starring at psci_fiq_enter: ... movw r8,
#(GICC_BASE & 0xffff) movt r8, #(GICC_BASE >> 16) ldr r9, [r8,
#GICC_IAR] movw r10, #0x3ff movt r10, #0 cmp r9, r10 bne out1 movw
r10, #0x3fe cmp r9, r10 bne out1 ...
How can GICC_IAR be 0x3ff and 0x3fe at the same time? These tests seems bogus. What was the actual intention here?

Yup, that looks like a massive brain fart. These two 'bne' should really be 'beq's... You want to get out if you either read that there is no interrupt (0x3ff) or an interrupt for non-secure (0x3fe).

M.

- --
Jazz is not dead. It just smells funny...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBAgAGBQJUYJx9AAoJECPQ0LrRPXpD/0kP/jwdNF2Ahtj05biOkCOTmp3V PUPNbffo8366UZeOLMjt7Wplde+//swOiLOq0uERS5IYj+QCMOA4F9hWOSO527zn v1j6374hAW1F8p8waGRxOGEffUgZGhXPmOxQAIsr7iqsIrvUVbE6KcNS5ofs+29R lWEaNW08L09CxcDA7szZzhVcMfu3oUxjzVFnnh6uhHG5DMxhCWt3IgEj2cLUR+VC ER/VAS9r0/4Bl5sZ7VO+GeToolgswIlIGgjkqifwrKFYH8XSyIxN8LDrNLAR1F4Q RQYfo3I7UGZcOPzz+BJcOsROohx2wtDeW6VxnRsE5WkE/udx++C7Ty9IquFrvPxA 1u8pzTrra27ZdxbrV2vS8YVo8bQktTdz8IjITuvWVX/cutXBdqIeP75EXYQQXeCg TuO26Huce4McEtn7bOEQac6SxLpTitGM+EAdrlhV9Dg7bCp1mKfzlIM4t6KuowEU 53mniGtDoLlnyUy+N4X6KyUmvqqyRNyC+c3ZWmP5xoB3VcQ1IJKuSEHSS9bJxIKb J2FNQN7v+C9P0d/1bItVqlpz6PiLcdWFFrfvDCxOe0uqkacwzhdlMMzrYYuMgAIi e0u8fqYunECZliT5OMaRhqBwSsniZmw2bBf6ZaUqlKgXc5MsxjncNakOKMQkQjip FwtcX/9StOztp43Laj7o
=LZJR
-----END PGP SIGNATURE-----