Skip to contentSkip to conversation

nios2 crash/hang in mainline due to 'lib: update LZ4 compressor module'

DISCUSSION Guenter Roeck 4 participants 18 messages Feb 26, 2017 – Mar 9, 2017 NIOS2 architecture

lore
Share & tools
lore
#0 lore raw

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

I tried to track down the problem, with no success. Just the presence of the LZ4 code seems to be sufficient to cause the problem; I have no idea why that would be the case.

Please let me know if there is anything I can do to help tracking down the problem.

Thanks,
Guenter

---
# bad: [c4f3f22eddc982d247ffe2a6690c3e4a5c46dd09] Merge tag 'linux-kselftest-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
# good: [9e314890292c0dd357eadef6a043704fa0b4c157] Merge tag 'openrisc-for-linus' of git://github.com/openrisc/linux
git bisect start 'HEAD' '9e31489'
# bad: [7067739df23ffd641ca99c967830e0ed2ba39eab] Merge branch 'i2c/for-4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect bad 7067739df23ffd641ca99c967830e0ed2ba39eab
# good: [c5adae9583ef6985875532904160c6bf9f07b453] lib: add CONFIG_TEST_SORT to enable self-test of sort()
git bisect good c5adae9583ef6985875532904160c6bf9f07b453
# bad: [edccb59429657b09806146339e2b27594c1d1da0] Merge tag 'fbdev-v4.11' of git://github.com/bzolnier/linux
git bisect bad edccb59429657b09806146339e2b27594c1d1da0
# good: [72db33355c1431fefcabb06c9c25705e8226eed5] fbdev: ssd1307fb: Start to use gpiod API for reset gpio
git bisect good 72db33355c1431fefcabb06c9c25705e8226eed5
# bad: [95330473636e5e4546f94874c957c3be66bb2140] checkpatch: remove false unbalanced braces warning
git bisect bad 95330473636e5e4546f94874c957c3be66bb2140
# bad: [69c78423b8f439b077929410bdf8f88e7031b891] lib/lz4: remove back-compat wrappers
git bisect bad 69c78423b8f439b077929410bdf8f88e7031b891
# bad: [e23d54e48346e775be53b3cc25a95d65da960393] lib/decompress_unlz4: change module to work with new LZ4 module version
git bisect bad e23d54e48346e775be53b3cc25a95d65da960393
# bad: [4e1a33b105ddf201f66dcc44490c6086a25eca0b] lib: update LZ4 compressor module
git bisect bad 4e1a33b105ddf201f66dcc44490c6086a25eca0b
# good: [8893f519330bb073a49c5b4676fce4be6f1be15d] lib/test_sort.c: make it explicitly non-modular
git bisect good 8893f519330bb073a49c5b4676fce4be6f1be15d

# first bad commit: [4e1a33b105ddf201f66dcc44490c6086a25eca0b] lib: update LZ4 compressor module
#1 lore raw

Hi Guenter,

thanks for your testing!

I must admit, I'm fairly new to kernel development and a little overwhelmed by all that tools used.
So I do not really know how to reproduce your test using your script. I installed qemu from the master branch and buildroot. Unfortunately, that's the point I'm stuck. I would be grateful if you provide me some lead how to continue. Would I make the kernel using ARCH=nios2 and a defconf and pass it to qemu? What arguments do I provide to that script (especially, the machine param)?

Quoted 10 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

So, seems like it's the decompressor? Which decompression code do you mean exactly? LZ4_decompress_fast/_safe/_generic? Since the decompression functions worked fine in all previous tests, and this is a problem during boot, my first guess would be the lib/decompress_unlz4.c, providing the functions for decompressing a lz4-compressed kernel image. But then it should only result in problems when the kernel image is compressed, wouldn't it?

Quoted 9 lines

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

I tried to track down the problem, with no success. Just the presence of the LZ4 code seems to be sufficient to cause the problem; I have no idea why that would be the case.

Maybe there's someone who has an idea and/or is experiencing similar issues. Hopefully, we can track this down.

Quoted 28 lines

Please let me know if there is anything I can do to help tracking down the problem.

Thanks,
Guenter

---
# bad: [c4f3f22eddc982d247ffe2a6690c3e4a5c46dd09] Merge tag 'linux-kselftest-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
# good: [9e314890292c0dd357eadef6a043704fa0b4c157] Merge tag 'openrisc-for-linus' of git://github.com/openrisc/linux
git bisect start 'HEAD' '9e31489'
# bad: [7067739df23ffd641ca99c967830e0ed2ba39eab] Merge branch 'i2c/for-4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect bad 7067739df23ffd641ca99c967830e0ed2ba39eab
# good: [c5adae9583ef6985875532904160c6bf9f07b453] lib: add CONFIG_TEST_SORT to enable self-test of sort()
git bisect good c5adae9583ef6985875532904160c6bf9f07b453
# bad: [edccb59429657b09806146339e2b27594c1d1da0] Merge tag 'fbdev-v4.11' of git://github.com/bzolnier/linux
git bisect bad edccb59429657b09806146339e2b27594c1d1da0
# good: [72db33355c1431fefcabb06c9c25705e8226eed5] fbdev: ssd1307fb: Start to use gpiod API for reset gpio
git bisect good 72db33355c1431fefcabb06c9c25705e8226eed5
# bad: [95330473636e5e4546f94874c957c3be66bb2140] checkpatch: remove false unbalanced braces warning
git bisect bad 95330473636e5e4546f94874c957c3be66bb2140
# bad: [69c78423b8f439b077929410bdf8f88e7031b891] lib/lz4: remove back-compat wrappers
git bisect bad 69c78423b8f439b077929410bdf8f88e7031b891
# bad: [e23d54e48346e775be53b3cc25a95d65da960393] lib/decompress_unlz4: change module to work with new LZ4 module version
git bisect bad e23d54e48346e775be53b3cc25a95d65da960393
# bad: [4e1a33b105ddf201f66dcc44490c6086a25eca0b] lib: update LZ4 compressor module
git bisect bad 4e1a33b105ddf201f66dcc44490c6086a25eca0b
# good: [8893f519330bb073a49c5b4676fce4be6f1be15d] lib/test_sort.c: make it explicitly non-modular
git bisect good 8893f519330bb073a49c5b4676fce4be6f1be15d

Quoted 1 line
# first bad commit: [4e1a33b105ddf201f66dcc44490c6086a25eca0b] lib: update LZ4 compressor module

Thank you,

Sven

#2 lore raw
Quoted 10 lines · Sven Schmidt

Hi Guenter,

thanks for your testing!

I must admit, I'm fairly new to kernel development and a little overwhelmed by all that tools used.
So I do not really know how to reproduce your test using your script. I installed qemu from the master branch and buildroot. Unfortunately, that's the point I'm stuck. I would be grateful if you provide me some lead how to continue. Would I make the kernel using ARCH=nios2 and a defconf and pass it to qemu? What arguments do I provide to that script (especially, the machine param)?

run-qemu-nios2.sh doesn't need any parameters, though you would have to update PATH_NIOS2 to match your toolchain and QEMU to match the qemu binary location. Otherwise just run the script from your linux repository.

You can also build a nios2 image using 10m50_defconfig and run qemu directly. Just remember to enable CONFIG_NIOS2_PASS_CMDLINE=y and CONFIG_BLK_DEV_INITRD=y. CONFIG_BLK_DEV_INITRD=y enables CONFIG_RD_LZ4 which triggers the problem.

path-to-qemu/qemu-system-nios2 -M 10m50-ghrd -kernel vmlinux -no-reboot \
	-dtb arch/nios2/boot/dts/10m50_devboard.dtb \
	--append "rdinit=/sbin/init" \
	-initrd busybox-nios2.cpio \
	-nographic -monitor none

should do it (assuming you copied the root file system from
https://github.com/groeck/linux-build-test/blob/master/rootfs/nios2/busybox-nios2.cpio).

Quoted 12 lines · Guenter Roeck
Quoted 10 lines

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

Quoted 5 lines

So, seems like it's the decompressor? Which decompression code do you mean exactly? LZ4_decompress_fast/_safe/_generic? Since the decompression functions worked fine in all previous tests, and this is a problem during boot, my first guess would be the lib/decompress_unlz4.c, providing the functions for decompressing a lz4-compressed kernel image. But then it should only result in problems when the kernel image is compressed, wouldn't it?

I am booting the uncompressed kernel. Also, if I disable CONFIG_RD_LZ4 in the configuration, everything works just fine. Just the _presence_ of the decompression code seems to trigger the problem. No idea if enabling CONFIG_RD_LZ4 results in some LZ4 compressed code to be generated (I do see usr/initramfs_data.cpio.lz4).

I added "return -EINVAL;" to the top of LZ4_decompress_generic(), which also helped. Adding it to the individual decompression functions seemed to be an on/off thing; sometimes it helped, sometimes not.

Quoted 5 lines
Quoted 5 lines

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 7 lines
Quoted 4 lines

I tried to track down the problem, with no success. Just the presence of the LZ4 code seems to be sufficient to cause the problem; I have no idea why that would be the case.

Maybe there's someone who has an idea and/or is experiencing similar issues. Hopefully, we can track this down.

Agreed. For my part I am pretty much out of ideas. I could explicitly disable CONFIG_RD_LZ4 in my tests, but that would really just defeat the purpose.

Guenter

Quoted 33 lines
Quoted 28 lines

Please let me know if there is anything I can do to help tracking down the problem.

Thanks,
Guenter

---
# bad: [c4f3f22eddc982d247ffe2a6690c3e4a5c46dd09] Merge tag 'linux-kselftest-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest
# good: [9e314890292c0dd357eadef6a043704fa0b4c157] Merge tag 'openrisc-for-linus' of git://github.com/openrisc/linux
git bisect start 'HEAD' '9e31489'
# bad: [7067739df23ffd641ca99c967830e0ed2ba39eab] Merge branch 'i2c/for-4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect bad 7067739df23ffd641ca99c967830e0ed2ba39eab
# good: [c5adae9583ef6985875532904160c6bf9f07b453] lib: add CONFIG_TEST_SORT to enable self-test of sort()
git bisect good c5adae9583ef6985875532904160c6bf9f07b453
# bad: [edccb59429657b09806146339e2b27594c1d1da0] Merge tag 'fbdev-v4.11' of git://github.com/bzolnier/linux
git bisect bad edccb59429657b09806146339e2b27594c1d1da0
# good: [72db33355c1431fefcabb06c9c25705e8226eed5] fbdev: ssd1307fb: Start to use gpiod API for reset gpio
git bisect good 72db33355c1431fefcabb06c9c25705e8226eed5
# bad: [95330473636e5e4546f94874c957c3be66bb2140] checkpatch: remove false unbalanced braces warning
git bisect bad 95330473636e5e4546f94874c957c3be66bb2140
# bad: [69c78423b8f439b077929410bdf8f88e7031b891] lib/lz4: remove back-compat wrappers
git bisect bad 69c78423b8f439b077929410bdf8f88e7031b891
# bad: [e23d54e48346e775be53b3cc25a95d65da960393] lib/decompress_unlz4: change module to work with new LZ4 module version
git bisect bad e23d54e48346e775be53b3cc25a95d65da960393
# bad: [4e1a33b105ddf201f66dcc44490c6086a25eca0b] lib: update LZ4 compressor module
git bisect bad 4e1a33b105ddf201f66dcc44490c6086a25eca0b
# good: [8893f519330bb073a49c5b4676fce4be6f1be15d] lib/test_sort.c: make it explicitly non-modular
git bisect good 8893f519330bb073a49c5b4676fce4be6f1be15d

Quoted 1 line
# first bad commit: [4e1a33b105ddf201f66dcc44490c6086a25eca0b] lib: update LZ4 compressor module

Thank you,

Sven

#3 lore raw
In reply to Guenter Roeck

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 10 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

Quoted 4 lines

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

Thanks
Tobias

#4 lore raw
Quoted 24 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 10 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

Quoted 4 lines

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

Quoted 8 lines

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

I tried the following combinations of gcc and binutils:

	gcc 6.1.0; binutils 2.26.1
	gcc 6.3.0; binutils 2.25.1
	gcc 6.3.0; binutils 2.26.1
	gcc 6.3.0; binutils 2.27

I used buildroot 2017.02-rc3 to build the toolchains. All have the problem.

Another data point, confirming what you say above: I am relatively (99%) sure that the code in question is not actually called. I added reference counters to make sure that this is the case. Just the existence of the LZ4 decompression code appears to be sufficient to cause the boot failure. Weird.

I just hope this only affects nios2.

Guenter

#5 lore raw
In reply to Tobias Klauser
Quoted 24 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 10 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

Quoted 4 lines

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

Quoted 7 lines

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

This message doesn't give me enough context to know what is going on, especially without seeing the rest of the thread. Generally speaking, Mentor recommends you use one of our stable releases instead of trying to roll your own from mainline sources. As an upstream binutils and gcc maintainer I do try my best to look at bug reports for those components, but I need a reproducible standalone testcase and specific versions of the different components involved.

-Sandra

#6 lore raw
Quoted 33 lines · Sandra Loosemore
Quoted 24 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 10 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

Quoted 4 lines

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

Quoted 7 lines

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

Quoted 8 lines

This message doesn't give me enough context to know what is going on, especially without seeing the rest of the thread. Generally speaking, Mentor recommends you use one of our stable releases instead of trying to roll your own from mainline sources. As an upstream binutils and gcc maintainer I do try my best to look at bug reports for those components, but I need a reproducible standalone testcase and specific versions of the different components involved.

The problem is also seen with Sourcery CodeBench Lite 2016.11-32 (gcc 6.2.0, binutils 2.26.51). I can provide additional details if needed, but we don't have a well enough understanding of the problem to be able to provide a reduced size test case. The test used to reproduce the problem is available at https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2,
run on the ToT linux kernel.

Guenter

#7 lore raw

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

It seems a bit strange that code which is not actually called causes problems like that.

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

Quoted 16 lines
Quoted 8 lines
Quoted 7 lines

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

Quoted 8 lines

This message doesn't give me enough context to know what is going on, especially without seeing the rest of the thread. Generally speaking, Mentor recommends you use one of our stable releases instead of trying to roll your own from mainline sources. As an upstream binutils and gcc maintainer I do try my best to look at bug reports for those components, but I need a reproducible standalone testcase and specific versions of the different components involved.

Quoted 8 lines

The problem is also seen with Sourcery CodeBench Lite 2016.11-32 (gcc 6.2.0, binutils 2.26.51). I can provide additional details if needed, but we don't have a well enough understanding of the problem to be able to provide a reduced size test case. The test used to reproduce the problem is available at https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2,
run on the ToT linux kernel.

Guenter

Regards,

Sven

#8 lore raw
Quoted 26 lines · Sven Schmidt

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

It seems a bit strange that code which is not actually called causes problems like that.

Yes, it is, though it is always possible. The code isn't exactly easy to understand; there may be some hidden caveats such as global variables. It may also be that some jump target exceeds its range (though why that would only be seen with the LZ4 code is another question), or that the compiler gets confused by the forced inlines (disabling that didn't make a difference, though, nor did disabling -O3).

Quoted 3 lines

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

For my part I am all but clueless. Unless someone has an idea, we may to disable LZ4 support for nios2 for the time being. Does anyone have thoughts on that ? Of course, that would not help if the problem also affects recent gcc/binutil versions on other architectures.

Thanks,
Guenter

Quoted 28 lines
Quoted 16 lines
Quoted 8 lines
Quoted 7 lines

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

Quoted 8 lines

This message doesn't give me enough context to know what is going on, especially without seeing the rest of the thread. Generally speaking, Mentor recommends you use one of our stable releases instead of trying to roll your own from mainline sources. As an upstream binutils and gcc maintainer I do try my best to look at bug reports for those components, but I need a reproducible standalone testcase and specific versions of the different components involved.

Quoted 8 lines

The problem is also seen with Sourcery CodeBench Lite 2016.11-32 (gcc 6.2.0, binutils 2.26.51). I can provide additional details if needed, but we don't have a well enough understanding of the problem to be able to provide a reduced size test case. The test used to reproduce the problem is available at https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2,
run on the ToT linux kernel.

Guenter

Regards,

Sven

#9 lore raw
Quoted 42 lines · Guenter Roeck
Quoted 26 lines · Sven Schmidt

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

It seems a bit strange that code which is not actually called causes problems like that.

Yes, it is, though it is always possible. The code isn't exactly easy to understand; there may be some hidden caveats such as global variables. It may also be that some jump target exceeds its range (though why that would only be seen with the LZ4 code is another question), or that the compiler gets confused by the forced inlines (disabling that didn't make a difference, though, nor did disabling -O3).

Quoted 8 lines
Quoted 3 lines

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

For my part I am all but clueless. Unless someone has an idea, we may to disable LZ4 support for nios2 for the time being. Does anyone have thoughts on that ? Of course, that would not help if the problem also affects recent gcc/binutil versions on other architectures.

After some further investigations, I'd say this isn't "caused" by LZ4 specifically but by a more general problem with one of the nios2 arch specific tools involved.

I manually enabled random additional CONFIG_* options and in some cases I got the kernel to boot (with CONFIG_RD_LZ4 enabled and no return -EINVAL in place) while in others I didn't. So I'd rather suspect this problem to be connected to the size or structure of the generated vmlinux image.

Or could this even be a problem with qemu? Did anyone already verify this on the 10m50 devboard? (Unfortunately I don't have any nios2 devboard available right now, otherwise I would have done this...)

Other than that I'm also becoming all but clueless... One option I thought of was using the QEMU monitor to dump the CPU state after the hang but so far I didn't manage to get it to work (hints appreciated ;)

Thanks
Tobias

#10 lore raw
Quoted 62 lines · Tobias Klauser
Quoted 42 lines · Guenter Roeck
Quoted 26 lines · Sven Schmidt

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

It seems a bit strange that code which is not actually called causes problems like that.

Yes, it is, though it is always possible. The code isn't exactly easy to understand; there may be some hidden caveats such as global variables. It may also be that some jump target exceeds its range (though why that would only be seen with the LZ4 code is another question), or that the compiler gets confused by the forced inlines (disabling that didn't make a difference, though, nor did disabling -O3).

Quoted 8 lines
Quoted 3 lines

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

For my part I am all but clueless. Unless someone has an idea, we may to disable LZ4 support for nios2 for the time being. Does anyone have thoughts on that ? Of course, that would not help if the problem also affects recent gcc/binutil versions on other architectures.

After some further investigations, I'd say this isn't "caused" by LZ4 specifically but by a more general problem with one of the nios2 arch specific tools involved.

I manually enabled random additional CONFIG_* options and in some cases I got the kernel to boot (with CONFIG_RD_LZ4 enabled and no return -EINVAL in place) while in others I didn't. So I'd rather suspect this problem to be connected to the size or structure of the generated vmlinux image.

Quoted 4 lines

Or could this even be a problem with qemu? Did anyone already verify this on the 10m50 devboard? (Unfortunately I don't have any nios2 devboard available right now, otherwise I would have done this...)

That is of course always possible.

Quoted 4 lines

Other than that I'm also becoming all but clueless... One option I thought of was using the QEMU monitor to dump the CPU state after the hang but so far I didn't manage to get it to work (hints appreciated ;)

Something like

qemu-system-nios2 -M 10m50-ghrd -kernel vmlinux -no-reboot \
	-dtb arch/nios2/boot/dts/10m50_devboard.dtb \
	--append "rdinit=/sbin/init" -initrd busybox-nios2.cpio

gives you a qemu monitor window. Use "info registers" to see registers. Looks like it is stuck in init_bootmem_core, or at least that is what it shows for me.

Guenter

#11 lore raw
Quoted 77 lines · Guenter Roeck
Quoted 62 lines · Tobias Klauser
Quoted 42 lines · Guenter Roeck
Quoted 26 lines · Sven Schmidt

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

It seems a bit strange that code which is not actually called causes problems like that.

Yes, it is, though it is always possible. The code isn't exactly easy to understand; there may be some hidden caveats such as global variables. It may also be that some jump target exceeds its range (though why that would only be seen with the LZ4 code is another question), or that the compiler gets confused by the forced inlines (disabling that didn't make a difference, though, nor did disabling -O3).

Quoted 8 lines
Quoted 3 lines

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

For my part I am all but clueless. Unless someone has an idea, we may to disable LZ4 support for nios2 for the time being. Does anyone have thoughts on that ? Of course, that would not help if the problem also affects recent gcc/binutil versions on other architectures.

After some further investigations, I'd say this isn't "caused" by LZ4 specifically but by a more general problem with one of the nios2 arch specific tools involved.

I manually enabled random additional CONFIG_* options and in some cases I got the kernel to boot (with CONFIG_RD_LZ4 enabled and no return -EINVAL in place) while in others I didn't. So I'd rather suspect this problem to be connected to the size or structure of the generated vmlinux image.

Quoted 4 lines

Or could this even be a problem with qemu? Did anyone already verify this on the 10m50 devboard? (Unfortunately I don't have any nios2 devboard available right now, otherwise I would have done this...)

That is of course always possible.

Quoted 4 lines

Other than that I'm also becoming all but clueless... One option I thought of was using the QEMU monitor to dump the CPU state after the hang but so far I didn't manage to get it to work (hints appreciated ;)

Something like

Quoted 7 lines
qemu-system-nios2 -M 10m50-ghrd -kernel vmlinux -no-reboot \
	-dtb arch/nios2/boot/dts/10m50_devboard.dtb \
	--append "rdinit=/sbin/init" -initrd busybox-nios2.cpio

gives you a qemu monitor window. Use "info registers" to see registers. Looks like it is stuck in init_bootmem_core, or at least that is what it shows for me.

Thanks a lot for the hint, this worked perfectly. I'm not all that familiar with qemu :-/

Using the qemu gdbserver I can indeed confirm that it seems to be stuck in init_bootmem_core:

(gdb) file vmlinux
Reading symbols from vmlinux...done.
(gdb) target remote localhost:1234
Remote debugging using localhost:1234 link_bootmem (bdata=<optimized out>) at mm/bootmem.c:80
80 if (bdata->node_min_pfn < ent->node_min_pfn) {

This looks like a very weird place for it to get stuck...

So I followed a different path and implemented early printk support for the 8250/16650 serial console on nios2, so I could get debug outputs earlier on (patch below, I'll also officially submit this later one).

Now I get the following output on boot:

Linux version 4.11.0-rc1-dirty (tobiask@ziws08) (gcc version 7.0.1 20170226 (experimental) (GCC) ) #46 Tue Mar 7 13:40:53 CET 2017 bootconsole [early0] enabled
Early console on uart16650 initialized at 0xf8001600
OF: fdt: Error -11 processing FDT

Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

---[ end Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

Looks like the in-memory device tree somehow gets corrupted. Not sure yet why and how this is linked to the Kconfig options selected but at least we now have a possibility to use debug messages earlier on.

---%<---%<---

Patch for 8250/16650 early printk support on nios2 (make sure to select CONFIG_EARLY_PRINTK):

arch/nios2/Kconfig.debug +1 -1
index 2fd08cbfdddb..35b5dd67b15a 100644
@@ -18,7 +18,7 @@ config EARLY_PRINTK
 	bool "Activate early kernel debugging"
 	default y
 	select SERIAL_CORE_CONSOLE
-	depends on SERIAL_ALTERA_JTAGUART_CONSOLE || SERIAL_ALTERA_UART_CONSOLE
+	depends on SERIAL_ALTERA_JTAGUART_CONSOLE || SERIAL_ALTERA_UART_CONSOLE || SERIAL_8250_CONSOLE
 	help
 	  Enable early printk on console
 	  This is useful for kernel debugging when your machine crashes very
arch/nios2/kernel/early_printk.c +40 -4

73 lines not on this page · raw message

#12 lore raw
Quoted 99 lines · Tobias Klauser
Quoted 77 lines · Guenter Roeck
Quoted 62 lines · Tobias Klauser
Quoted 42 lines · Guenter Roeck
Quoted 26 lines · Sven Schmidt

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

It seems a bit strange that code which is not actually called causes problems like that.

Yes, it is, though it is always possible. The code isn't exactly easy to understand; there may be some hidden caveats such as global variables. It may also be that some jump target exceeds its range (though why that would only be seen with the LZ4 code is another question), or that the compiler gets confused by the forced inlines (disabling that didn't make a difference, though, nor did disabling -O3).

Quoted 8 lines
Quoted 3 lines

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

For my part I am all but clueless. Unless someone has an idea, we may to disable LZ4 support for nios2 for the time being. Does anyone have thoughts on that ? Of course, that would not help if the problem also affects recent gcc/binutil versions on other architectures.

After some further investigations, I'd say this isn't "caused" by LZ4 specifically but by a more general problem with one of the nios2 arch specific tools involved.

I manually enabled random additional CONFIG_* options and in some cases I got the kernel to boot (with CONFIG_RD_LZ4 enabled and no return -EINVAL in place) while in others I didn't. So I'd rather suspect this problem to be connected to the size or structure of the generated vmlinux image.

Quoted 4 lines

Or could this even be a problem with qemu? Did anyone already verify this on the 10m50 devboard? (Unfortunately I don't have any nios2 devboard available right now, otherwise I would have done this...)

That is of course always possible.

Quoted 4 lines

Other than that I'm also becoming all but clueless... One option I thought of was using the QEMU monitor to dump the CPU state after the hang but so far I didn't manage to get it to work (hints appreciated ;)

Something like

Quoted 7 lines
qemu-system-nios2 -M 10m50-ghrd -kernel vmlinux -no-reboot \
	-dtb arch/nios2/boot/dts/10m50_devboard.dtb \
	--append "rdinit=/sbin/init" -initrd busybox-nios2.cpio

gives you a qemu monitor window. Use "info registers" to see registers. Looks like it is stuck in init_bootmem_core, or at least that is what it shows for me.

Thanks a lot for the hint, this worked perfectly. I'm not all that familiar with qemu :-/

Using the qemu gdbserver I can indeed confirm that it seems to be stuck in init_bootmem_core:

(gdb) file vmlinux
Reading symbols from vmlinux...done.
(gdb) target remote localhost:1234
Remote debugging using localhost:1234 link_bootmem (bdata=<optimized out>) at mm/bootmem.c:80
80 if (bdata->node_min_pfn < ent->node_min_pfn) {

Quoted 6 lines

This looks like a very weird place for it to get stuck...

So I followed a different path and implemented early printk support for the 8250/16650 serial console on nios2, so I could get debug outputs earlier on (patch below, I'll also officially submit this later one).

That is great; I'll add that to my own tests to get some output.

Quoted 6 lines

Now I get the following output on boot:

Linux version 4.11.0-rc1-dirty (tobiask@ziws08) (gcc version 7.0.1 20170226 (experimental) (GCC) ) #46 Tue Mar 7 13:40:53 CET 2017 bootconsole [early0] enabled
Early console on uart16650 initialized at 0xf8001600
OF: fdt: Error -11 processing FDT

Quoted 8 lines
Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

---[ end Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

Looks like the in-memory device tree somehow gets corrupted. Not sure yet why and how this is linked to the Kconfig options selected but at least we now have a possibility to use debug messages earlier on.

Interesting. I was able to confirm that the lz4 patch is not the root cause. I was not able to reproduce the problem in v4.10, but after adding more and more configuration options I get it to fail starting with commit ac1820fb286b552 ("Merge tag 'for-next-dma_ops' of
git://git.kernel.org/pub/ scm/linux/kernel/git/dledford/rdma"). No idea if that is the root cause either. Kernel configuration for that case is attached.

Of course ac1820fb286b552 doesn't crash anymore with that after applying your patch below, and v4.11-rc1 crashes without any output :-(.

I think I'll add some logging into qemu to see where it puts the dtb.

Guenter

Quoted 95 lines

---%<---%<---

Patch for 8250/16650 early printk support on nios2 (make sure to select CONFIG_EARLY_PRINTK):

arch/nios2/Kconfig.debug +1 -1
index 2fd08cbfdddb..35b5dd67b15a 100644
@@ -18,7 +18,7 @@ config EARLY_PRINTK
 	bool "Activate early kernel debugging"
 	default y
 	select SERIAL_CORE_CONSOLE
-	depends on SERIAL_ALTERA_JTAGUART_CONSOLE || SERIAL_ALTERA_UART_CONSOLE
+	depends on SERIAL_ALTERA_JTAGUART_CONSOLE || SERIAL_ALTERA_UART_CONSOLE || SERIAL_8250_CONSOLE
 	help
 	  Enable early printk on console
 	  This is useful for kernel debugging when your machine crashes very
arch/nios2/kernel/early_printk.c +39 -3
index c08e4c1486fc..24b4506f4969 100644
@@ -22,6 +22,8 @@ static unsigned long base_addr;
 
 #if defined(CONFIG_SERIAL_ALTERA_JTAGUART_CONSOLE)
 
+#define UART_NAME "altera_jtaguart"
+
 #define ALTERA_JTAGUART_DATA_REG		0
 #define ALTERA_JTAGUART_CONTROL_REG		4
 #define ALTERA_JTAGUART_CONTROL_WSPACE_MSK	0xFFFF0000
@@ -53,6 +55,8 @@ static void early_console_write(struct console *con, const char *s, unsigned n)
 
 #elif defined(CONFIG_SERIAL_ALTERA_UART_CONSOLE)
 
+#define UART_NAME "altera_uart"
+
 #define ALTERA_UART_TXDATA_REG		4
 #define ALTERA_UART_STATUS_REG		8
 #define ALTERA_UART_STATUS_TRDY		0x0040
@@ -80,9 +84,40 @@ static void early_console_write(struct console *con, const char *s, unsigned n)
 	}
 }
 
+#elif defined(CONFIG_SERIAL_8250_CONSOLE)
+
+#define UART_NAME "uart16650"
+
+#define UART_LSR_TEMT	0x40 /* Transmitter empty */
+#define UART_LSR_THRE	0x20 /* Transmit-hold-register empty */
+#define BOTH_EMPTY (UART_LSR_TEMT | UART_LSR_THRE)
+
+#define UART_GET_SR() \
+	__builtin_ldwio((void *)(base_addr + 0x14))
+#define UART_SET_TX(v) \
+	__builtin_stwio((void *)(base_addr), v)
+
+static void early_console_putc(char c)
+{
+	while (!((UART_GET_SR() & BOTH_EMPTY) == BOTH_EMPTY))
+		;
+
+	UART_SET_TX(c & 0xff);
+}
+
+static void early_console_write(struct console *con, const char *s, unsigned n)
+{
+	while (n-- && *s) {
+		early_console_putc(*s);
+		if (*s == '\n')
+			early_console_putc('\r');
+		s++;
+	}
+}
+
 #else
-# error Neither SERIAL_ALTERA_JTAGUART_CONSOLE nor SERIAL_ALTERA_UART_CONSOLE \
-selected
+# error Neither SERIAL_ALTERA_JTAGUART_CONSOLE, SERIAL_ALTERA_UART_CONSOLE, \
+        nor SERIAL_8250_CONSOLE selected
 #endif
 
 static struct console early_console_prom = {
@@ -95,7 +130,8 @@ static struct console early_console_prom = {
 void __init setup_early_printk(void)
 {
 #if defined(CONFIG_SERIAL_ALTERA_JTAGUART_CONSOLE) ||	\
-	defined(CONFIG_SERIAL_ALTERA_UART_CONSOLE)
+	defined(CONFIG_SERIAL_ALTERA_UART_CONSOLE) ||	\
+	defined(CONFIG_SERIAL_8250_CONSOLE)
 	base_addr = of_early_console();
 #else
 	base_addr = 0;

arch/nios2/kernel/early_printk.c void __init setup_early_printk(void) · line 154

Quoted 6 lines
arch/nios2/kernel/early_printk.c +1 -1
@@ -114,5 +150,5 @@ void __init setup_early_printk(void)
 
 	early_console = &early_console_prom;
 	register_console(early_console);
-	pr_info("early_console initialized at 0x%08lx\n", base_addr);
+	pr_info("Early console on %s initialized at 0x%08lx\n", UART_NAME, base_addr);
 }

CONFIG_SYSVIPC=y
CONFIG_NO_HZ_IDLE=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_LOG_BUF_SHIFT=14 CONFIG_BLK_DEV_INITRD=y CONFIG_SYSCTL_SYSCALL=y
# CONFIG_ELF_CORE is not set
# CONFIG_EPOLL is not set
# CONFIG_SIGNALFD is not set
# CONFIG_TIMERFD is not set
# CONFIG_EVENTFD is not set
# CONFIG_SHMEM is not set
# CONFIG_AIO is not set
CONFIG_EMBEDDED=y
CONFIG_SLAB=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y CONFIG_NIOS2_MEM_BASE=0x8000000 CONFIG_NIOS2_HW_MUL_SUPPORT=y CONFIG_NIOS2_HW_DIV_SUPPORT=y CONFIG_CUSTOM_CACHE_SETTINGS=y CONFIG_NIOS2_DCACHE_SIZE=0x8000 CONFIG_NIOS2_ICACHE_SIZE=0x8000
# CONFIG_NIOS2_CMDLINE_IGNORE_DTB is not set CONFIG_NIOS2_PASS_CMDLINE=y CONFIG_BINFMT_MISC=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
CONFIG_NET_IPIP=y
CONFIG_NET_IPGRE_DEMUX=y CONFIG_NET_IPGRE=y CONFIG_NET_IPGRE_BROADCAST=y CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y CONFIG_NET_FOU_IP_TUNNELS=y CONFIG_INET_AH=y
CONFIG_INET_ESP=y CONFIG_INET_ESP_OFFLOAD=y CONFIG_INET_IPCOMP=y CONFIG_INET_UDP_DIAG=y CONFIG_INET_RAW_DIAG=y CONFIG_INET_DIAG_DESTROY=y CONFIG_TCP_CONG_ADVANCED=y CONFIG_TCP_CONG_BIC=y CONFIG_TCP_CONG_WESTWOOD=y CONFIG_TCP_CONG_HTCP=y CONFIG_TCP_CONG_HSTCP=y CONFIG_TCP_CONG_HYBLA=y CONFIG_TCP_CONG_NV=y CONFIG_TCP_CONG_SCALABLE=y CONFIG_TCP_CONG_LP=y CONFIG_TCP_CONG_VENO=y CONFIG_TCP_CONG_YEAH=y CONFIG_TCP_CONG_ILLINOIS=y CONFIG_TCP_CONG_DCTCP=y CONFIG_TCP_CONG_CDG=y CONFIG_TCP_CONG_BBR=y CONFIG_NETWORK_SECMARK=y
CONFIG_BRIDGE=y
CONFIG_VLAN_8021Q=y
CONFIG_BT=y
CONFIG_CFG80211=y CONFIG_NL80211_TESTMODE=y CONFIG_CFG80211_DEVELOPER_WARNINGS=y CONFIG_CFG80211_CERTIFICATION_ONUS=y CONFIG_CFG80211_REG_CELLULAR_HINTS=y CONFIG_CFG80211_REG_RELAX_NO_IR=y CONFIG_CFG80211_DEBUGFS=y CONFIG_CFG80211_INTERNAL_REGDB=y CONFIG_CFG80211_WEXT=y
CONFIG_MAC80211=y
CONFIG_MAC80211_MESH=y CONFIG_MAC80211_LEDS=y CONFIG_MAC80211_DEBUGFS=y CONFIG_MAC80211_MESSAGE_TRACING=y CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# CONFIG_FW_LOADER is not set
CONFIG_MTD=y
CONFIG_MTD_CMDLINE_PARTS=y CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
CONFIG_MTD_CFI_INTELEXT=y CONFIG_MTD_CFI_AMDSTD=y CONFIG_BLK_DEV_LOOP=y CONFIG_BLK_DEV_NVME_SCSI=y CONFIG_NVME_FC=y
CONFIG_NVME_TARGET=y CONFIG_NVME_TARGET_LOOP=y CONFIG_NVME_TARGET_FC=y CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
CONFIG_CHR_DEV_OSST=y
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_ATA=y
CONFIG_MD=y
CONFIG_NETDEVICES=y
CONFIG_ALTERA_TSE=y
CONFIG_MARVELL_PHY=y
# CONFIG_WLAN is not set
CONFIG_NVM=y
# CONFIG_INPUT_MOUSE is not set
# CONFIG_VT is not set CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_OF_PLATFORM=y CONFIG_SERIAL_ALTERA_JTAGUART=y
# CONFIG_HW_RANDOM is not set CONFIG_I2C_CHARDEV=y CONFIG_I2C_ARB_GPIO_CHALLENGE=y CONFIG_I2C_MUX_GPIO=y CONFIG_I2C_MUX_PCA9541=y CONFIG_I2C_MUX_PCA954x=y CONFIG_I2C_MUX_REG=y CONFIG_I2C_MUX_MLXCPLD=y CONFIG_I2C_DESIGNWARE_PLATFORM=y CONFIG_I2C_GPIO=y
CONFIG_I2C_OCORES=y CONFIG_I2C_PCA_PLATFORM=y CONFIG_I2C_SIMTEC=y
CONFIG_I2C_XILINX=y CONFIG_I2C_DIOLAN_U2C=y CONFIG_I2C_PARPORT_LIGHT=y CONFIG_I2C_ROBOTFUZZ_OSIF=y CONFIG_I2C_TAOS_EVM=y CONFIG_I2C_TINY_USB=y
CONFIG_I2C_SLAVE=y CONFIG_I2C_SLAVE_EEPROM=y CONFIG_I2C_DEBUG_CORE=y CONFIG_I2C_DEBUG_ALGO=y CONFIG_I2C_DEBUG_BUS=y
CONFIG_SPI=y
CONFIG_SPI_ALTERA=y
CONFIG_SPI_CADENCE=y CONFIG_SPI_DESIGNWARE=y
CONFIG_SPI_GPIO=y
CONFIG_SPI_OC_TINY=y
CONFIG_SPI_ROCKCHIP=y CONFIG_SPI_SC18IS602=y
CONFIG_SPI_XCOMM=y
CONFIG_SPI_XILINX=y CONFIG_SPI_ZYNQMP_GQSPI=y CONFIG_SPI_SPIDEV=y
CONFIG_SPI_TLE62X0=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_ALTERA=y
CONFIG_POWER_RESET=y
CONFIG_POWER_SUPPLY=y CONFIG_SENSORS_AD7314=y CONFIG_SENSORS_AD7414=y CONFIG_SENSORS_AD7418=y CONFIG_SENSORS_ADM1021=y CONFIG_SENSORS_ADM1025=y CONFIG_SENSORS_ADM1026=y CONFIG_SENSORS_ADM1029=y CONFIG_SENSORS_ADT7411=y CONFIG_SENSORS_ADT7462=y CONFIG_SENSORS_ADT7470=y CONFIG_SENSORS_ADT7475=y CONFIG_SENSORS_DS620=y CONFIG_SENSORS_DS1621=y CONFIG_SENSORS_G760A=y CONFIG_SENSORS_G762=y CONFIG_SENSORS_GPIO_FAN=y CONFIG_SENSORS_LTC2945=y CONFIG_SENSORS_LTC2990=y CONFIG_SENSORS_LTC4151=y CONFIG_SENSORS_LTC4215=y CONFIG_SENSORS_LTC4222=y CONFIG_SENSORS_LTC4245=y CONFIG_SENSORS_LTC4260=y CONFIG_SENSORS_LTC4261=y CONFIG_SENSORS_MAX1111=y CONFIG_SENSORS_MAX16065=y CONFIG_SENSORS_MAX1619=y CONFIG_SENSORS_MAX1668=y CONFIG_SENSORS_MAX197=y CONFIG_SENSORS_MAX31722=y CONFIG_SENSORS_MAX6639=y CONFIG_SENSORS_MAX6642=y CONFIG_SENSORS_MAX6650=y CONFIG_SENSORS_MAX6697=y CONFIG_SENSORS_MAX31790=y CONFIG_SENSORS_MCP3021=y CONFIG_SENSORS_TC654=y CONFIG_SENSORS_ADCXX=y CONFIG_SENSORS_LM63=y CONFIG_SENSORS_LM70=y CONFIG_SENSORS_LM73=y CONFIG_SENSORS_LM75=y CONFIG_SENSORS_LM77=y CONFIG_SENSORS_LM78=y CONFIG_SENSORS_LM80=y CONFIG_SENSORS_LM83=y CONFIG_SENSORS_LM85=y CONFIG_SENSORS_LM87=y CONFIG_SENSORS_LM90=y CONFIG_SENSORS_LM92=y CONFIG_SENSORS_LM93=y CONFIG_SENSORS_LM95234=y CONFIG_SENSORS_LM95241=y CONFIG_SENSORS_LM95245=y CONFIG_SENSORS_NTC_THERMISTOR=y CONFIG_SENSORS_NCT7802=y CONFIG_SENSORS_NCT7904=y
CONFIG_PMBUS=y
CONFIG_SENSORS_ADM1275=y CONFIG_SENSORS_LM25066=y CONFIG_SENSORS_LTC2978=y CONFIG_SENSORS_LTC3815=y CONFIG_SENSORS_MAX16064=y CONFIG_SENSORS_MAX20751=y CONFIG_SENSORS_MAX34440=y CONFIG_SENSORS_MAX8688=y CONFIG_SENSORS_TPS40422=y CONFIG_SENSORS_UCD9000=y CONFIG_SENSORS_UCD9200=y CONFIG_SENSORS_ZL6100=y CONFIG_SENSORS_INA209=y
CONFIG_THERMAL=y
CONFIG_WATCHDOG=y
CONFIG_MFD_ACT8945A=y
CONFIG_MFD_AS3711=y
CONFIG_PMIC_ADP5520=y CONFIG_MFD_AAT2870_CORE=y CONFIG_PMIC_DA903X=y CONFIG_MFD_DA9052_SPI=y CONFIG_MFD_DA9052_I2C=y CONFIG_MFD_DA9055=y
CONFIG_MFD_DA9062=y
CONFIG_MFD_DA9063=y
CONFIG_MFD_DA9150=y
CONFIG_MFD_DLN2=y
CONFIG_MFD_MC13XXX_SPI=y CONFIG_MFD_MC13XXX_I2C=y CONFIG_MFD_HI6421_PMIC=y CONFIG_HTC_PASIC3=y
CONFIG_HTC_I2CPLD=y CONFIG_INTEL_SOC_PMIC=y CONFIG_MFD_88PM800=y
CONFIG_MFD_88PM805=y
CONFIG_MFD_MAX14577=y CONFIG_MFD_MAX77620=y CONFIG_MFD_MAX77686=y CONFIG_MFD_MAX77693=y CONFIG_MFD_MAX77843=y
CONFIG_MFD_MAX8907=y
CONFIG_MFD_MAX8925=y
CONFIG_MFD_MAX8997=y
CONFIG_MFD_MAX8998=y
CONFIG_MFD_MT6397=y
CONFIG_REGULATOR=y CONFIG_REGULATOR_FIXED_VOLTAGE=y CONFIG_REGULATOR_VIRTUAL_CONSUMER=y CONFIG_REGULATOR_USERSPACE_CONSUMER=y CONFIG_REGULATOR_ACT8865=y CONFIG_REGULATOR_AD5398=y CONFIG_REGULATOR_DA903X=y CONFIG_REGULATOR_DA9052=y CONFIG_REGULATOR_DA9055=y CONFIG_REGULATOR_DA9062=y CONFIG_REGULATOR_DA9063=y CONFIG_REGULATOR_DA9210=y CONFIG_REGULATOR_DA9211=y CONFIG_REGULATOR_FAN53555=y CONFIG_REGULATOR_GPIO=y CONFIG_REGULATOR_HI6421=y CONFIG_REGULATOR_ISL9305=y CONFIG_REGULATOR_ISL6271A=y CONFIG_REGULATOR_LP3971=y CONFIG_REGULATOR_LP3972=y CONFIG_REGULATOR_LP872X=y CONFIG_REGULATOR_LP8755=y CONFIG_REGULATOR_LTC3589=y CONFIG_REGULATOR_LTC3676=y CONFIG_REGULATOR_MAX14577=y CONFIG_REGULATOR_MAX1586=y CONFIG_REGULATOR_MAX77620=y CONFIG_REGULATOR_MAX8649=y CONFIG_REGULATOR_MAX8660=y CONFIG_REGULATOR_MAX8907=y CONFIG_REGULATOR_MAX8925=y CONFIG_REGULATOR_MAX8952=y CONFIG_REGULATOR_MAX8973=y CONFIG_REGULATOR_MAX8997=y CONFIG_REGULATOR_MAX8998=y CONFIG_REGULATOR_MAX77686=y CONFIG_REGULATOR_MAX77693=y CONFIG_REGULATOR_MAX77802=y CONFIG_REGULATOR_MC13783=y CONFIG_REGULATOR_MC13892=y CONFIG_REGULATOR_MT6311=y CONFIG_REGULATOR_MT6323=y CONFIG_REGULATOR_MT6397=y CONFIG_REGULATOR_PFUZE100=y CONFIG_REGULATOR_PV88060=y CONFIG_REGULATOR_PV88080=y CONFIG_REGULATOR_PV88090=y CONFIG_REGULATOR_TPS51632=y CONFIG_REGULATOR_TPS62360=y CONFIG_REGULATOR_TPS65023=y CONFIG_REGULATOR_TPS6507X=y CONFIG_REGULATOR_TPS6524X=y CONFIG_MEDIA_SUPPORT=y CONFIG_MEDIA_CAMERA_SUPPORT=y CONFIG_MEDIA_ANALOG_TV_SUPPORT=y CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y CONFIG_MEDIA_RADIO_SUPPORT=y CONFIG_MEDIA_SDR_SUPPORT=y CONFIG_MEDIA_RC_SUPPORT=y CONFIG_MEDIA_CEC_SUPPORT=y CONFIG_MEDIA_CONTROLLER=y CONFIG_VIDEO_V4L2_SUBDEV_API=y CONFIG_VIDEO_ADV_DEBUG=y CONFIG_VIDEO_FIXED_MINOR_RANGES=y CONFIG_DVB_DYNAMIC_MINORS=y CONFIG_DVB_DEMUX_SECTION_LOSS_LOG=y CONFIG_USB=y
CONFIG_USB_DYNAMIC_MINORS=y CONFIG_USB_XHCI_HCD=y CONFIG_USB_EHCI_HCD=y CONFIG_USB_EHCI_ROOT_HUB_TT=y CONFIG_USB_OXU210HP_HCD=y CONFIG_USB_ISP116X_HCD=y CONFIG_USB_ISP1362_HCD=y CONFIG_USB_FOTG210_HCD=y CONFIG_USB_MAX3421_HCD=y CONFIG_USB_OHCI_HCD=y CONFIG_USB_SL811_HCD=y CONFIG_USB_R8A66597_HCD=y
CONFIG_USB_ACM=y
CONFIG_USB_STORAGE=y CONFIG_USB_STORAGE_SDDR09=y CONFIG_USB_STORAGE_SDDR55=y CONFIG_USB_STORAGE_JUMPSHOT=y CONFIG_USB_STORAGE_ALAUDA=y CONFIG_USB_STORAGE_ONETOUCH=y CONFIG_USB_STORAGE_KARMA=y CONFIG_USB_STORAGE_CYPRESS_ATACB=y CONFIG_USB_UAS=y
CONFIG_USBIP_CORE=y
CONFIG_USB_DWC3=y
CONFIG_USB_DWC2=y
CONFIG_USB_ISP1760=y
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
CONFIG_LEDS_GPIO=y CONFIG_LEDS_TRIGGER_HEARTBEAT=y CONFIG_IIO=y
CONFIG_IIO_BUFFER_CB=y CONFIG_IIO_SW_DEVICE=y CONFIG_IIO_SW_TRIGGER=y
CONFIG_BMA180=y
CONFIG_BMA220=y
CONFIG_BMC150_ACCEL=y
CONFIG_DA280=y
CONFIG_DA311=y
CONFIG_DMARD06=y
CONFIG_DMARD09=y
CONFIG_DMARD10=y CONFIG_IIO_ST_ACCEL_3AXIS=y
CONFIG_KXSD9=y
CONFIG_KXCJK1013=y
CONFIG_MC3230=y
CONFIG_MMA7455_I2C=y
CONFIG_MMA7455_SPI=y
CONFIG_MMA7660=y
CONFIG_MMA8452=y
CONFIG_MMA9551=y
CONFIG_MMA9553=y
CONFIG_MXC4005=y
CONFIG_MXC6255=y
CONFIG_SCA3000=y
CONFIG_STK8312=y
CONFIG_STK8BA50=y
CONFIG_AM2315=y
CONFIG_DHT11=y
CONFIG_HDC100X=y
CONFIG_HTS221=y
CONFIG_HTU21=y
CONFIG_SI7005=y
CONFIG_SI7020=y
CONFIG_ADJD_S311=y
CONFIG_AL3320A=y
CONFIG_APDS9300=y
CONFIG_APDS9960=y
CONFIG_BH1750=y
CONFIG_BH1780=y
CONFIG_CM32181=y
CONFIG_CM3232=y
CONFIG_CM3323=y
CONFIG_CM3605=y
CONFIG_CM36651=y
CONFIG_GP2AP020A00F=y CONFIG_SENSORS_ISL29018=y CONFIG_ISL29125=y
CONFIG_JSA1212=y
CONFIG_RPR0521=y
CONFIG_LTR501=y
CONFIG_MAX44000=y
CONFIG_OPT3001=y
CONFIG_PA12203001=y
CONFIG_SI1145=y
CONFIG_STK3310=y
CONFIG_TCS3414=y
CONFIG_TCS3472=y
CONFIG_SENSORS_TSL2563=y
CONFIG_TSL2583=y
CONFIG_TSL4531=y
CONFIG_US5182D=y
CONFIG_VCNL4000=y
CONFIG_VEML6070=y
CONFIG_EXT3_FS=y
CONFIG_BTRFS_FS=y CONFIG_BTRFS_FS_POSIX_ACL=y CONFIG_EXPORTFS_BLOCK_OPS=y
# CONFIG_DNOTIFY is not set
# CONFIG_INOTIFY_USER is not set CONFIG_OVERLAY_FS=y
CONFIG_ISO9660_FS=y
CONFIG_UDF_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_NTFS_FS=y
CONFIG_NTFS_DEBUG=y
CONFIG_NTFS_RW=y
CONFIG_JFFS2_FS=y
CONFIG_SQUASHFS=y
CONFIG_SQUASHFS_LZ4=y CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3_ACL=y
CONFIG_ROOT_NFS=y
CONFIG_SUNRPC_DEBUG=y
CONFIG_DEBUG_INFO=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set CONFIG_PANIC_ON_OOPS=y CONFIG_SCHED_STACK_END_CHECK=y

#13 lore raw
In reply to Tobias Klauser

On 03/07/2017 04:46 AM, Tobias Klauser wrote:
[ ... ]

Quoted 5 lines

Linux version 4.11.0-rc1-dirty (tobiask@ziws08) (gcc version 7.0.1 20170226 (experimental) (GCC) ) #46 Tue Mar 7 13:40:53 CET 2017 bootconsole [early0] enabled
Early console on uart16650 initialized at 0xf8001600
OF: fdt: Error -11 processing FDT

Quoted 8 lines
Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

---[ end Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

Looks like the in-memory device tree somehow gets corrupted. Not sure yet why and how this is linked to the Kconfig options selected but at least we now have a possibility to use debug messages earlier on.

I think I found the problem. In unflatten_and_copy_device_tree(), with added debug information:

OF: fdt: initial_boot_params=c861e400, dt=c861f000 size=28874 (0x70ca)

... and then initial_boot_params is copied to dt, which results in corrupted fdt since the memory overlaps. Looks like the initial_boot_params memory is not reserved and (re-)allocated by early_init_dt_alloc_memory_arch().

Guenter

#14 lore raw
Quoted 17 lines · Guenter Roeck

On 03/07/2017 04:46 AM, Tobias Klauser wrote:
[ ... ]

Quoted 5 lines

Linux version 4.11.0-rc1-dirty (tobiask@ziws08) (gcc version 7.0.1 20170226 (experimental) (GCC) ) #46 Tue Mar 7 13:40:53 CET 2017 bootconsole [early0] enabled
Early console on uart16650 initialized at 0xf8001600
OF: fdt: Error -11 processing FDT

Quoted 8 lines
Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

---[ end Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

Looks like the in-memory device tree somehow gets corrupted. Not sure yet why and how this is linked to the Kconfig options selected but at least we now have a possibility to use debug messages earlier on.

Quoted 8 lines

I think I found the problem. In unflatten_and_copy_device_tree(), with added debug information:

OF: fdt: initial_boot_params=c861e400, dt=c861f000 size=28874 (0x70ca)

... and then initial_boot_params is copied to dt, which results in corrupted fdt since the memory overlaps. Looks like the initial_boot_params memory is not reserved and (re-)allocated by early_init_dt_alloc_memory_arch().

Thanks for the analysis. That certainly explains the issue. The following patch solves the issue for me. Though I'm not entirely sure if it is correct and that is all that is needed. Do we need to retain the memory for initial_boot_params after bootmem is freed?

arch/nios2/kernel/prom.c +7 -0
index 099f5ce1f3cb..6869fe03f3ff 100644
@@ -48,6 +48,13 @@ void * __init early_init_dt_alloc_memory_arch(u64 size, u64 align)
 	return alloc_bootmem_align(size, align);
 }
 
+int __init early_init_dt_reserve_memory_arch(phys_addr_t base,
+					phys_addr_t size, bool nomap)
+{
+	reserve_bootmem(base, size, BOOTMEM_DEFAULT);
+	return 0;
+}
+
 void __init early_init_devtree(void *params)
 {
 	__be32 *dtb = (u32 *)__dtb_start;
arch/nios2/kernel/setup.c +3 -0
index 6e57ffa5db27..6044d9be28b4 100644
@@ -201,6 +201,9 @@ void __init setup_arch(char **cmdline_p)
 	}
 #endif /* CONFIG_BLK_DEV_INITRD */
 
+	early_init_fdt_reserve_self();
+	early_init_fdt_scan_reserved_mem();
+
 	unflatten_and_copy_device_tree();
 
 	setup_cpuinfo();
author Tested-by
#15 lore raw
Quoted 23 lines · Tobias Klauser
Quoted 22 lines · Guenter Roeck

On 03/07/2017 04:46 AM, Tobias Klauser wrote:
[ ... ]

Quoted 5 lines

Linux version 4.11.0-rc1-dirty (tobiask@ziws08) (gcc version 7.0.1 20170226 (experimental) (GCC) ) #46 Tue Mar 7 13:40:53 CET 2017 bootconsole [early0] enabled
Early console on uart16650 initialized at 0xf8001600
OF: fdt: Error -11 processing FDT

Quoted 8 lines
Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

---[ end Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

Looks like the in-memory device tree somehow gets corrupted. Not sure yet why and how this is linked to the Kconfig options selected but at least we now have a possibility to use debug messages earlier on.

I think I found the problem. In unflatten_and_copy_device_tree(), with added debug information:

OF: fdt: initial_boot_params=c861e400, dt=c861f000 size=28874 (0x70ca)

Quoted 9 lines
Quoted 3 lines

... and then initial_boot_params is copied to dt, which results in corrupted fdt since the memory overlaps. Looks like the initial_boot_params memory is not reserved and (re-)allocated by early_init_dt_alloc_memory_arch().

Thanks for the analysis. That certainly explains the issue. The following patch solves the issue for me. Though I'm not entirely sure if it is correct and that is all that is needed. Do we need to retain the memory for initial_boot_params after bootmem is freed?

I don't know if it is correct either, but it matches what I came up with, and it does work for me as well. Feel free to add

when you submit the patch for real.

Thanks,
Guenter

Quoted 32 lines
arch/nios2/kernel/prom.c +7 -0
index 099f5ce1f3cb..6869fe03f3ff 100644
@@ -48,6 +48,13 @@ void * __init early_init_dt_alloc_memory_arch(u64 size, u64 align)
 	return alloc_bootmem_align(size, align);
 }
 
+int __init early_init_dt_reserve_memory_arch(phys_addr_t base,
+					phys_addr_t size, bool nomap)
+{
+	reserve_bootmem(base, size, BOOTMEM_DEFAULT);
+	return 0;
+}
+
 void __init early_init_devtree(void *params)
 {
 	__be32 *dtb = (u32 *)__dtb_start;
arch/nios2/kernel/setup.c +3 -0
index 6e57ffa5db27..6044d9be28b4 100644
@@ -201,6 +201,9 @@ void __init setup_arch(char **cmdline_p)
 	}
 #endif /* CONFIG_BLK_DEV_INITRD */
 
+	early_init_fdt_reserve_self();
+	early_init_fdt_scan_reserved_mem();
+
 	unflatten_and_copy_device_tree();
 
 	setup_cpuinfo();
#16 lore raw
In reply to Sven Schmidt
Quoted 32 lines · Sven Schmidt

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

Quoted 27 lines

It seems a bit strange that code which is not actually called causes problems like that.

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

Quoted 22 lines
Quoted 8 lines
Quoted 7 lines

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

Quoted 8 lines

This message doesn't give me enough context to know what is going on, especially without seeing the rest of the thread. Generally speaking, Mentor recommends you use one of our stable releases instead of trying to roll your own from mainline sources. As an upstream binutils and gcc maintainer I do try my best to look at bug reports for those components, but I need a reproducible standalone testcase and specific versions of the different components involved.

The problem is also seen with Sourcery CodeBench Lite 2016.11-32 (gcc 6.2.0, binutils 2.26.51). I can provide additional details if needed, but we don't have a well enough understanding of the problem to be able to provide a reduced size test case. The test used to reproduce the problem is available at https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2,
run on the ToT linux kernel.

Just a suggestion: can you try binutils trunk, too? Alan Modra and I just tracked down and fixed a bug with the linker creating bad executables that the kernel's ELF loader couldn't properly map into memory. IIUC it only affected programs that use dynamic libraries, but maybe there was more to it than that. In any case it would be good to know if the problem has already been fixed before investigating further.

-Sandra

#17 lore raw
Quoted 61 lines · Sandra Loosemore
Quoted 32 lines · Sven Schmidt

Hi Guenter, Tobias and Sandra,

thanks for your effort here.

Quoted 21 lines · Guenter Roeck
Quoted 20 lines · Sandra Loosemore
Quoted 19 lines · Tobias Klauser

(adding Sandra Loosemore to Cc due to possible relation to gcc/binutils for nios2)

Quoted 14 lines · Guenter Roeck

Hi Sven,

my qemu test for nios2 started failing with commit 4e1a33b105dd ("lib:
update LZ4 compressor module"). The test hangs early during boot before any console output is seen. Reverting the offending patch as well as the subsequent lz4 related patches fixes the problem. Disabling CONFIG_RD_LZ4 and with it other LZ4 options also fixes it (as does adding "return -EINVAL;" at the top of the LZ4 decompression code). For reference, bisect log is attached.

I tried with buildroot toolchains using gcc 6.1.0 as well as 6.3.0 and binutils 2.26.1. Scripts used to run the tests are available at
https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2.
Qemu is from qemu mainline or qemu v2.8 with nios2 patches applied.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Looks like this is somehow related to gcc/binutils. Using GCC 4.8.3 and binutils 2.24.51 (both from from Sourcery CodeBench Lite 2014.05) I can get a kernel booting on latest master branch. AFAICT, none of the LZ4_decompress_* functions are called during boot.

Quoted 27 lines

It seems a bit strange that code which is not actually called causes problems like that.

Please let me know if and how I may help you figure out what's happening, especially regarding the differences between the previous LZ4 and the current implementation.

Quoted 22 lines
Quoted 8 lines
Quoted 7 lines

However, using a self-built GCC 7.0 (20161127) and binutils 2.27 I can reproduce the problem you see using the instructions Guenter provided in the reply to Sven.

I'll try to dig a bit deeper from here on. Any suggestions on what to look out for wrt the differences between the gcc/binutils version are welcome of course.

Quoted 8 lines

This message doesn't give me enough context to know what is going on, especially without seeing the rest of the thread. Generally speaking, Mentor recommends you use one of our stable releases instead of trying to roll your own from mainline sources. As an upstream binutils and gcc maintainer I do try my best to look at bug reports for those components, but I need a reproducible standalone testcase and specific versions of the different components involved.

The problem is also seen with Sourcery CodeBench Lite 2016.11-32 (gcc 6.2.0, binutils 2.26.51). I can provide additional details if needed, but we don't have a well enough understanding of the problem to be able to provide a reduced size test case. The test used to reproduce the problem is available at https://github.com/groeck/linux-build-test/tree/master/rootfs/nios2,
run on the ToT linux kernel.

Quoted 7 lines

Just a suggestion: can you try binutils trunk, too? Alan Modra and I just tracked down and fixed a bug with the linker creating bad executables that the kernel's ELF loader couldn't properly map into memory. IIUC it only affected programs that use dynamic libraries, but maybe there was more to it than that. In any case it would be good to know if the problem has already been fixed before investigating further.

Thanks for the suggestion.

Just tried it with a kernel compiled with binutils trunk as of today (2.28.51.20170302) and latest gcc snapshot (7.0.1 20170226). Unfortunately, the issue still persists.

Tobias