Skip to contentSkip to conversation

Status of 'cris' architecture support in Linux kernel

DISCUSSION Guenter Roeck 7 participants 15 messages Aug 31, 2014 – Sep 21, 2014

lore
Share & tools
lore
#0 lore raw

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

After spending a number of days (and nights) on it, the results don't look very encouraging.

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Anyway, below are my individual findings. If there is an ongoing effort to improve cris support in the upstream kernel, specifically support for crisv32, please let me know. I'll be happy to test the resulting kernels.

Thanks,
Guenter

---------------------

Individual findings:

headers_install

make ARCH=cris INSTALL_HDR_PATH=/tmp headers_install

results in:

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

-----------

Trying to enable architecture specific drivers:

Enabling ETRAX_I2C or ETRAX_STREAMCOPROC results in compile errors. The errors date back as far as 2.6.27 and are thus not the result of later API changes. I did not attempt to enable any other functionality.

-----------

qemu: Attempts to load images in qemu fail.

Command line:

qemu-system-cris -serial stdio -kernel vmlinux -monitor none \
	-nographic -append "console=ttyS0,115200,N,8 rdinit=/sbin/init" \
	-initrd busybox-cris.img

This command yields no output; using arch/cris/boot/Image has the same result.

With arch/cris/boot/zImage, the result is:

Uncompressing Linux...

invalid compressed format (err=1)

-- System halted

---

Research shows that there is a working image and root file system at the qemu web site. Further research shows that this image includes code which was never merged upstream. The missing code includes (at least) early console support as well as support for the crisv32 serial interface.

After some digging, I found at least some of the code in out-of-tree sources. After adding early console support, there is console output, but the image crashes with the following log message.

...
NET: Registered protocol family 16
Switched to clocksource crisv32_rotime

Unable to handle kernel NULL pointer dereferenceOops: 0000
CPU: 0
ERP: c001029e SRP: c00108ce  CCS: 00028008 USP: 00000000 MOF: 00000000
 r0: c0230660  r1: c1ff3e7c   r2: 00000014  r3: 00000001
 r4: c1ff3e88  r5: c1ffffff   r6: c0102b94  r7: 00000000
 r8: c1ff3e58  r9: c1ff3e80  r10: c1ff3e7c r11: c0230660
r12: 00000001 r13: 80000200 oR10: c1ff3e7c acr: 00000018
 sp: c1ff3dd4
       Data MMU Cause: 00000100

Instruction MMU Cause: 00000000
Process swapper (pid: 1, stackpage=c1ff1c40)

Stack from c1ff3cbc:

       c0004a44 c01e84e6 c1ff3e20 c1ff3e28 00000000 c1ffffff c1ff3cf8 c000589c 
       00000018 00000000 c1ff3dd4 00000000 00000000 00000000 c1ff3e88 c1ff3d04 
       c0004b6e c0293bb0 c1ff3dcc c0005220 00000000 c0230660 c1ff3e7c 00000014 
Call Trace: [<c0004a44>] [<c01e84e6>] [<c000589c>] [<c0004b6e>] [<c0005220>] [<c0102b94>] [<c00fb92c>] 
       [<c002922c>] [<c00fbd58>] [<c00fb858>] [<c00fbc02>] [<c002922c>] [<c0008896>] [<c0102b94>] [<c00108ce>] 
       [<c001029e>] [<c0010250>] [<c00be2d6>] [<c00203f4>] [<c00108ce>] [<c00ff4d8>] [<c00b81b4>] [<c00be3b0>] 
       [<c00ff4d8>] [<c0004236>] [<c00041bc>] [<c00205ae>] [<c00ff4d8>] [<c00203f4>] [<c01e6f22>] [<c01e6f36>] 
       [<c000547e>] 
Code: 04 29 0f 00 43 36 68 30 18 21 14 22 (62) 2a c0 22 32 30 b0 05 0c 21 6f fa 
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

Manual backtrace:

c0004a44 show_stack
c01e84e6 printk
c000589c show_registers
c0004b6e die_if_kernel
c0005220 get_mmu_context
c0102b94 strcmp
c00fb92c idr_get_empty_slot
c002922c __init_waitqueue_head
c00fbd58 ida_get_new_above
c00fb858 ida_pre_get
c00fbc02 ida_get_new_above
c002922c __init_waitqueue_head
c0008896 d_mmu_refill
c0102b94 strcmp
c00108ce walk_system_ram_range
c001029e find_next_iomem_res
c0010250 find_next_iomem_res
c00be2d6 kclist_add_private
c00203f4 parse_args
c00108ce walk_system_ram_range
c00ff4d8 strcpy
c00b81b4 proc_register
c00be3b0 kcore_update_ram
c00ff4d8 strcpy
c0004236 do_one_initcall
c00041bc do_one_initcall
c00205ae parse_args
c00ff4d8 strcpy
c00203f4 parse_args
c01e6f22 kernel_init
c01e6f36 kernel_init

----------------
I did not attempt to bisect the problem.

I got an earlier kernel, 2.6.27, to proceed further, though I did not attempt to load initramfs (the kernel had a missing symbol when I tried to enable initramfs support).

I was able to find the serial driver for crisv32 in an out-of-kernel tree and port it to 2.6.27 to the point where it loads. I did not attempt to do this with the current upstream kernel, as it crashed before it gets to the point of trying to load the driver.

The crisv32 serial driver (or at least the version I found) is far from acceptable for upstream integration and would require major cleanup or even rewrite effort if it was to be merged upstream. The version I found is from a 2.6.26 kernel, while the kernel version at the qemu web site (binary only) is 2.6.33, so the driver I worked with is not the latest version and may have been improved later. However, the result was not published, or I was unable to find it.

-----------------

Toolchain

Creating a toolchain from upstream sources required a number of patches.

Linux headers:
- Fix the basic headers_install problem mentioned above
- Export ptrace.h
- Some post-processing after installing the headers; specifically, create a couple of symlinks in the headers directory usr/include/arch-v10/arch -> usr/include/arch [for crisv10] usr/include/arch-v32/arch -> usr/include/arch [for crisv32]

gcc:
- binutils 2.24
- uclibc 0.9.33.2
- gcc 4.7.4
Requires a backport from upstream gcc to compile.
Later versions of gcc (4.8.3, 4.9.1) fail with internal compiler errors even after patching.

Toolchain was built successfully with buildroot (after adding cris support to it) with above patches / changes.

#1 lore raw
Quoted 6 lines · Guenter Roeck

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

After spending a number of days (and nights) on it, the results don't look very encouraging.

Quoted 7 lines

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Anyway, below are my individual findings. If there is an ongoing effort to improve cris support in the upstream kernel, specifically support for crisv32, please let me know. I'll be happy to test the resulting kernels.

Jesper is on parental leave so do not expect any prompt reply from him.

Quoted 5 lines

Individual findings:

headers_install

Quoted 7 lines

make ARCH=cris INSTALL_HDR_PATH=/tmp headers_install

results in:

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

I have already sent a fix for this that Michael said he wanted to deal with.

Sam

#2 lore raw
Quoted 22 lines · Sam Ravnborg
Quoted 6 lines · Guenter Roeck

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

After spending a number of days (and nights) on it, the results don't look very encouraging.

Quoted 7 lines

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Anyway, below are my individual findings. If there is an ongoing effort to improve cris support in the upstream kernel, specifically support for crisv32, please let me know. I'll be happy to test the resulting kernels.

Jesper is on parental leave so do not expect any prompt reply from him.

Quoted 5 lines

Individual findings:

headers_install

Quoted 9 lines
Quoted 7 lines

make ARCH=cris INSTALL_HDR_PATH=/tmp headers_install

results in:

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

I have already sent a fix for this that Michael said he wanted to deal with.

Hi Sam,

That is great. Note there is a second part besides the failure itself:
The resulting installed headers don't work and need a post-installation patch, as mentioned in my previous e-mail. Would be great if you can look into this as well (if you did not do so already).

Thanks,
Guenter

#3 lore raw
In reply to Sam Ravnborg

Jesper is actually back at work today but give him a few days to catch up.

Quoted 34 lines

31 aug 2014 kl. 20:51 skrev "Sam Ravnborg" <[email protected]>:

Quoted 7 lines · Guenter Roeck

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

After spending a number of days (and nights) on it, the results don't look very encouraging.

Quoted 7 lines

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Anyway, below are my individual findings. If there is an ongoing effort to improve cris support in the upstream kernel, specifically support for crisv32, please let me know. I'll be happy to test the resulting kernels.

Jesper is on parental leave so do not expect any prompt reply from him.

Quoted 5 lines

Individual findings:

headers_install

Quoted 7 lines

make ARCH=cris INSTALL_HDR_PATH=/tmp headers_install

results in:

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

I have already sent a fix for this that Michael said he wanted to deal with.

Sam

#4 lore raw
In reply to Guenter Roeck

Hi Guenter,

Sorry for not answering earlier, like some have said in
the thread followups, I have been on parental leave for quite some time.

Quoted 2 lines · Guenter Roeck

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

A very ambitious goal. :-)

Quoted 6 lines

After spending a number of days (and nights) on it, the results don't look very encouraging.

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Your conclusion is not completely off, but it could be better than it looks. :-)

I'll try to explain the state of the CRIS port as it currently stands:

(Background: CRIS port supports 3 different SoC:s, where the CRISv10 is older and subsequently less used. The other two are ETRAX FS and ARTPEC-3, which in principle share the same CPU-core (CRISv32) but contain some changes to the peripheral hardware IPs)

CRISv10 is only supported by me as a hobby project, AXIS does not have any current shipping units with this SoC, thus support for this is waning fast. QEMU support is not available for this SoC.
Additionally, newer gcc assumes TLS support, which CRISv10 does not have, and older gcc:s yields an ICE (internal compiler error) on newer kernels.

Units with ETRAX FS is no longer actively upgraded by AXIS with newer kernels, and I have a problem testing anything other than the AXIS 88 developer board (Last relase of the SDK was a couple of years ago with the 2.6.32 kernel IIRC) which is not up to date, but at least have a lot in common with ARTPEC-3 and so is not so hard to support.

ARTPEC-3 support is not complete as some drivers are not included in upstream (ethernet and serial are the most notable ones) but is actually in best shape, we have 3.16 booting on real hardware in house.

I'll add the missing drivers and current patches we have locally to a git tree on git-hub, I'll get back to you on that.

It is also the ARTPEC-3 SoC that is implemented in QEMU unless I'm mistaken.

Quoted 3 lines

Anyway, below are my individual findings. If there is an ongoing effort to improve cris support in the upstream kernel, specifically support for crisv32, please let me know. I'll be happy to test the resulting kernels.

Thank you for your work, I'll try to add some more information in the hope that this will at least help people google for more information.

Feel free to send me an email if you want to pick this up again.

Quoted 9 lines

Thanks,
Guenter

---------------------

Individual findings:

headers_install

Quoted 7 lines

make ARCH=cris INSTALL_HDR_PATH=/tmp headers_install

results in:

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

Yes, known error, I believe that Sam Ravnborg's patch will correct this.

On a tangential note, my automatic (minimal) builds does not do make install, so I haven't seen this error. :-P

Quoted 7 lines

-----------

Trying to enable architecture specific drivers:

Enabling ETRAX_I2C or ETRAX_STREAMCOPROC results in compile errors. The errors date back as far as 2.6.27 and are thus not the result of later API changes. I did not attempt to enable any other functionality.

Thanks, I don't have these included in my automatic builds,
and I note that the cryptocop.c which is gated with ETRAX_STREAMCOPROC has a large number of changes in our internal tree as compared to upstream.

Quoted 10 lines

-----------

qemu: Attempts to load images in qemu fail.

Command line:

qemu-system-cris -serial stdio -kernel vmlinux -monitor none \
	-nographic -append "console=ttyS0,115200,N,8 rdinit=/sbin/init" \
	-initrd busybox-cris.img
Quoted 5 lines

This command yields no output; using arch/cris/boot/Image has the same result.

With arch/cris/boot/zImage, the result is:

Uncompressing Linux...

... which at least indicates that the uncompress code is running...

Quoted 5 lines

invalid compressed format (err=1)

-- System halted

... but I believe that the image might need to be massaged a bit further to be able to boot in QEMU, however it's been some time since I last did this. I'll have to get back to you on this.

Quoted 6 lines

---

Research shows that there is a working image and root file system at the qemu web site. Further research shows that this image includes code which was never merged upstream. The missing code includes (at least) early console support as well as support for the crisv32 serial interface.

Confirmed, the serial driver isn't upstream.

Quoted 58 lines

After some digging, I found at least some of the code in out-of-tree sources. After adding early console support, there is console output, but the image crashes with the following log message.

...
NET: Registered protocol family 16
Switched to clocksource crisv32_rotime

Unable to handle kernel NULL pointer dereferenceOops: 0000
CPU: 0
ERP: c001029e SRP: c00108ce  CCS: 00028008 USP: 00000000 MOF: 00000000
 r0: c0230660  r1: c1ff3e7c   r2: 00000014  r3: 00000001
 r4: c1ff3e88  r5: c1ffffff   r6: c0102b94  r7: 00000000
 r8: c1ff3e58  r9: c1ff3e80  r10: c1ff3e7c r11: c0230660
r12: 00000001 r13: 80000200 oR10: c1ff3e7c acr: 00000018
 sp: c1ff3dd4
       Data MMU Cause: 00000100

Instruction MMU Cause: 00000000
Process swapper (pid: 1, stackpage=c1ff1c40)

Stack from c1ff3cbc:

       c0004a44 c01e84e6 c1ff3e20 c1ff3e28 00000000 c1ffffff c1ff3cf8 c000589c 
       00000018 00000000 c1ff3dd4 00000000 00000000 00000000 c1ff3e88 c1ff3d04 
       c0004b6e c0293bb0 c1ff3dcc c0005220 00000000 c0230660 c1ff3e7c 00000014 
Call Trace: [<c0004a44>] [<c01e84e6>] [<c000589c>] [<c0004b6e>] [<c0005220>] [<c0102b94>] [<c00fb92c>] 
       [<c002922c>] [<c00fbd58>] [<c00fb858>] [<c00fbc02>] [<c002922c>] [<c0008896>] [<c0102b94>] [<c00108ce>] 
       [<c001029e>] [<c0010250>] [<c00be2d6>] [<c00203f4>] [<c00108ce>] [<c00ff4d8>] [<c00b81b4>] [<c00be3b0>] 
       [<c00ff4d8>] [<c0004236>] [<c00041bc>] [<c00205ae>] [<c00ff4d8>] [<c00203f4>] [<c01e6f22>] [<c01e6f36>] 
       [<c000547e>] 
Code: 04 29 0f 00 43 36 68 30 18 21 14 22 (62) 2a c0 22 32 30 b0 05 0c 21 6f fa 
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

Manual backtrace:

c0004a44 show_stack
c01e84e6 printk
c000589c show_registers
c0004b6e die_if_kernel
c0005220 get_mmu_context
c0102b94 strcmp
c00fb92c idr_get_empty_slot
c002922c __init_waitqueue_head
c00fbd58 ida_get_new_above
c00fb858 ida_pre_get
c00fbc02 ida_get_new_above
c002922c __init_waitqueue_head
c0008896 d_mmu_refill
c0102b94 strcmp
c00108ce walk_system_ram_range
c001029e find_next_iomem_res
c0010250 find_next_iomem_res
c00be2d6 kclist_add_private
c00203f4 parse_args
c00108ce walk_system_ram_range
c00ff4d8 strcpy
c00b81b4 proc_register
c00be3b0 kcore_update_ram

Quoted 8 lines

c00ff4d8 strcpy
c0004236 do_one_initcall
c00041bc do_one_initcall
c00205ae parse_args
c00ff4d8 strcpy
c00203f4 parse_args
c01e6f22 kernel_init
c01e6f36 kernel_init

No clue on this though...

Quoted 12 lines

----------------
I did not attempt to bisect the problem.

I got an earlier kernel, 2.6.27, to proceed further, though I did not attempt to load initramfs (the kernel had a missing symbol when I tried to enable initramfs support).

I was able to find the serial driver for crisv32 in an out-of-kernel tree and port it to 2.6.27 to the point where it loads. I did not attempt to do this with the current upstream kernel, as it crashed before it gets to the point of trying to load the driver.

Quoted 7 lines

The crisv32 serial driver (or at least the version I found) is far from acceptable for upstream integration and would require major cleanup or even rewrite effort if it was to be merged upstream. The version I found is from a 2.6.26 kernel, while the kernel version at the qemu web site (binary only) is 2.6.33, so the driver I worked with is not the latest version and may have been improved later. However, the result was not published, or I was unable to find it.

Unfortunately, the code quality was the original problem why it wasn't upstreamed at the same time as the rest of the port, and it still hasn't changed in any markable respect. :-(

Quoted 14 lines

-----------------

Toolchain

Creating a toolchain from upstream sources required a number of patches.

Linux headers:
- Fix the basic headers_install problem mentioned above
- Export ptrace.h
- Some post-processing after installing the headers; specifically, create a couple of symlinks in the headers directory usr/include/arch-v10/arch -> usr/include/arch [for crisv10] usr/include/arch-v32/arch -> usr/include/arch [for crisv32]

Quoted 7 lines

gcc:
- binutils 2.24
- uclibc 0.9.33.2
- gcc 4.7.4
Requires a backport from upstream gcc to compile.
Later versions of gcc (4.8.3, 4.9.1) fail with internal compiler errors even after patching.

I'm using gcc 4.7.2, but of course, that is a locally built version that might have patches not upstream (although I doubt it knowing the gcc CRIS maintainer :-)

Quoted 2 lines

Toolchain was built successfully with buildroot (after adding cris support to it) with above patches / changes.

Thanks again,

/^JN - Jesper Nilsson

Signature · 1 line
               Jesper Nilsson -- [email protected]
#5 lore raw

Following up on Jesper's reply...

Quoted 11 lines

From: Jesper Nilsson <[email protected]>
Date: Mon, 15 Sep 2014 16:52:03 +0200

Quoted 8 lines

Toolchain

Creating a toolchain from upstream sources required a number of patches.

- gcc 4.7.4
Requires a backport from upstream gcc to compile.
Later versions of gcc (4.8.3, 4.9.1) fail with internal compiler errors even after patching.

This sounds like PR61737 (fixed; use 4.9.2 not released yet), but of course with the above sentence serving as your complete bug-report (or did I miss one?) that will only be a guess.

Quoted 3 lines

I'm using gcc 4.7.2, but of course, that is a locally built version that might have patches not upstream (although I doubt it knowing the gcc CRIS maintainer :-)

(oops. *blush*) The TLS stuff is unfortunately not upstreamed at this time. Unimportant when building the kernel, but mentioned for completeness. (The bad excuse is that I originally awaited pre-v32 kernels to get the corresponding TLS support so I wouldn't have to do ugly pre-vs.-v32 conditionals, then the whole thing lost priority, and now I avoid adding code I can't/won't test - the glibc end isn't upstreamed either. Sorry.)

brgds, H-P

#6 lore raw
Quoted 9 lines · Hans-Peter Nilsson

Following up on Jesper's reply...

Quoted 6 lines

From: Jesper Nilsson <[email protected]>
Date: Mon, 15 Sep 2014 16:52:03 +0200

Quoted 3 lines

Toolchain

Creating a toolchain from upstream sources required a number of patches.

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

- gcc 4.7.4
Requires a backport from upstream gcc to compile.
Later versions of gcc (4.8.3, 4.9.1) fail with internal compiler errors even after patching.

This sounds like PR61737 (fixed; use 4.9.2 not released yet), but of course with the above sentence serving as your complete bug-report (or did I miss one?) that will only be a guess.

Confirmed; gcc 4.9.1 for crisv32 builds after applying the patch fixing PR61737 (and the patch to fic config.gcc) on top of gcc 4.9.1.

Guenter

#7 lore raw
Quoted 15 lines · Guenter Roeck
Quoted 14 lines · Hans-Peter Nilsson

Following up on Jesper's reply...

Quoted 11 lines

From: Jesper Nilsson <[email protected]>
Date: Mon, 15 Sep 2014 16:52:03 +0200

Quoted 8 lines

Toolchain

Creating a toolchain from upstream sources required a number of patches.

- gcc 4.7.4
Requires a backport from upstream gcc to compile.
Later versions of gcc (4.8.3, 4.9.1) fail with internal compiler errors even after patching.

Quoted 7 lines
Quoted 4 lines

This sounds like PR61737 (fixed; use 4.9.2 not released yet), but of course with the above sentence serving as your complete bug-report (or did I miss one?) that will only be a guess.

Confirmed; gcc 4.9.1 for crisv32 builds after applying the patch fixing PR61737 (and the patch to fic config.gcc) on top of gcc 4.9.1.

Not-so-good update: I got the toolchain with gcc 4.9.1 to build, but the resulting image hangs early during boot. This does not happen if I use the toolchain with gcc 4.7.4. Guess I'll stick with that.

Guenter

#8 lore raw
In reply to Jesper Nilsson
Quoted 39 lines · Jesper Nilsson

Hi Guenter,

Sorry for not answering earlier, like some have said in
the thread followups, I have been on parental leave for quite some time.

Quoted 2 lines · Guenter Roeck

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

A very ambitious goal. :-)

Quoted 6 lines

After spending a number of days (and nights) on it, the results don't look very encouraging.

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Your conclusion is not completely off, but it could be better than it looks. :-)

I'll try to explain the state of the CRIS port as it currently stands:

(Background: CRIS port supports 3 different SoC:s, where the CRISv10 is older and subsequently less used. The other two are ETRAX FS and ARTPEC-3, which in principle share the same CPU-core (CRISv32) but contain some changes to the peripheral hardware IPs)

CRISv10 is only supported by me as a hobby project, AXIS does not have any current shipping units with this SoC, thus support for this is waning fast. QEMU support is not available for this SoC.
Additionally, newer gcc assumes TLS support, which CRISv10 does not have, and older gcc:s yields an ICE (internal compiler error) on newer kernels.

Units with ETRAX FS is no longer actively upgraded by AXIS with newer kernels, and I have a problem testing anything other than the AXIS 88 developer board (Last relase of the SDK was a couple of years ago with the 2.6.32 kernel IIRC) which is not up to date, but at least have a lot in common with ARTPEC-3 and so is not so hard to support.

Quoted 7 lines

ARTPEC-3 support is not complete as some drivers are not included in upstream (ethernet and serial are the most notable ones) but is actually in best shape, we have 3.16 booting on real hardware in house.

I'll add the missing drivers and current patches we have locally to a git tree on git-hub, I'll get back to you on that.

Would be great.

Quoted 2 lines

It is also the ARTPEC-3 SoC that is implemented in QEMU unless I'm mistaken.

The only machine supported in qemu is axis-88.

qemu-system-cris -M ?
Supported machines are:
none empty machine
axis-dev88 AXIS devboard 88 (default)

Does that include or imply ARTPEC-3 ?

Quoted 9 lines
Quoted 3 lines

Anyway, below are my individual findings. If there is an ongoing effort to improve cris support in the upstream kernel, specifically support for crisv32, please let me know. I'll be happy to test the resulting kernels.

Thank you for your work, I'll try to add some more information in the hope that this will at least help people google for more information.

Feel free to send me an email if you want to pick this up again.

Definitely. Being able to build a kernel is great, but it is just as important to ensure that it is actually working. Otherwise any effort on the architecture would be just a waste of time.

Quoted 11 lines
Quoted 11 lines

Thanks,
Guenter

---------------------

Individual findings:

headers_install

make ARCH=cris INSTALL_HDR_PATH=/tmp headers_install

Quoted 8 lines
Quoted 5 lines

results in:

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

Yes, known error, I believe that Sam Ravnborg's patch will correct this.

Is that patch available in public ? Sam's response seems to suggest that he sent it to Michael (Mikael ?) who wanted to handle it, but I don't find a matching patch in public archives.

Quoted 3 lines

On a tangential note, my automatic (minimal) builds does not do make install, so I haven't seen this error. :-P

Mine not either ;-). I only found it because I tried to build an initramfs which needs installed headers.

Quoted 40 lines
Quoted 7 lines

-----------

Trying to enable architecture specific drivers:

Enabling ETRAX_I2C or ETRAX_STREAMCOPROC results in compile errors. The errors date back as far as 2.6.27 and are thus not the result of later API changes. I did not attempt to enable any other functionality.

Thanks, I don't have these included in my automatic builds,
and I note that the cryptocop.c which is gated with ETRAX_STREAMCOPROC has a large number of changes in our internal tree as compared to upstream.

Quoted 10 lines

-----------

qemu: Attempts to load images in qemu fail.

Command line:

qemu-system-cris -serial stdio -kernel vmlinux -monitor none \
	-nographic -append "console=ttyS0,115200,N,8 rdinit=/sbin/init" \
	-initrd busybox-cris.img
Quoted 5 lines

This command yields no output; using arch/cris/boot/Image has the same result.

With arch/cris/boot/zImage, the result is:

Uncompressing Linux...

... which at least indicates that the uncompress code is running...

Quoted 5 lines

invalid compressed format (err=1)

-- System halted

... but I believe that the image might need to be massaged a bit further to be able to boot in QEMU, however it's been some time since I last did this. I'll have to get back to you on this.

Quoted 9 lines
Quoted 6 lines

---

Research shows that there is a working image and root file system at the qemu web site. Further research shows that this image includes code which was never merged upstream. The missing code includes (at least) early console support as well as support for the crisv32 serial interface.

Confirmed, the serial driver isn't upstream.

:-(

Quoted 89 lines
Quoted 58 lines

After some digging, I found at least some of the code in out-of-tree sources. After adding early console support, there is console output, but the image crashes with the following log message.

...
NET: Registered protocol family 16
Switched to clocksource crisv32_rotime

Unable to handle kernel NULL pointer dereferenceOops: 0000
CPU: 0
ERP: c001029e SRP: c00108ce  CCS: 00028008 USP: 00000000 MOF: 00000000
 r0: c0230660  r1: c1ff3e7c   r2: 00000014  r3: 00000001
 r4: c1ff3e88  r5: c1ffffff   r6: c0102b94  r7: 00000000
 r8: c1ff3e58  r9: c1ff3e80  r10: c1ff3e7c r11: c0230660
r12: 00000001 r13: 80000200 oR10: c1ff3e7c acr: 00000018
 sp: c1ff3dd4
       Data MMU Cause: 00000100

Instruction MMU Cause: 00000000
Process swapper (pid: 1, stackpage=c1ff1c40)

Stack from c1ff3cbc:

       c0004a44 c01e84e6 c1ff3e20 c1ff3e28 00000000 c1ffffff c1ff3cf8 c000589c 
       00000018 00000000 c1ff3dd4 00000000 00000000 00000000 c1ff3e88 c1ff3d04 
       c0004b6e c0293bb0 c1ff3dcc c0005220 00000000 c0230660 c1ff3e7c 00000014 
Call Trace: [<c0004a44>] [<c01e84e6>] [<c000589c>] [<c0004b6e>] [<c0005220>] [<c0102b94>] [<c00fb92c>] 
       [<c002922c>] [<c00fbd58>] [<c00fb858>] [<c00fbc02>] [<c002922c>] [<c0008896>] [<c0102b94>] [<c00108ce>] 
       [<c001029e>] [<c0010250>] [<c00be2d6>] [<c00203f4>] [<c00108ce>] [<c00ff4d8>] [<c00b81b4>] [<c00be3b0>] 
       [<c00ff4d8>] [<c0004236>] [<c00041bc>] [<c00205ae>] [<c00ff4d8>] [<c00203f4>] [<c01e6f22>] [<c01e6f36>] 
       [<c000547e>] 
Code: 04 29 0f 00 43 36 68 30 18 21 14 22 (62) 2a c0 22 32 30 b0 05 0c 21 6f fa 
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

Manual backtrace:

c0004a44 show_stack
c01e84e6 printk
c000589c show_registers
c0004b6e die_if_kernel
c0005220 get_mmu_context
c0102b94 strcmp
c00fb92c idr_get_empty_slot
c002922c __init_waitqueue_head
c00fbd58 ida_get_new_above
c00fb858 ida_pre_get
c00fbc02 ida_get_new_above
c002922c __init_waitqueue_head
c0008896 d_mmu_refill
c0102b94 strcmp
c00108ce walk_system_ram_range
c001029e find_next_iomem_res
c0010250 find_next_iomem_res
c00be2d6 kclist_add_private
c00203f4 parse_args
c00108ce walk_system_ram_range
c00ff4d8 strcpy
c00b81b4 proc_register
c00be3b0 kcore_update_ram

Quoted 8 lines

c00ff4d8 strcpy
c0004236 do_one_initcall
c00041bc do_one_initcall
c00205ae parse_args
c00ff4d8 strcpy
c00203f4 parse_args
c01e6f22 kernel_init
c01e6f36 kernel_init

No clue on this though...

Quoted 12 lines

----------------
I did not attempt to bisect the problem.

I got an earlier kernel, 2.6.27, to proceed further, though I did not attempt to load initramfs (the kernel had a missing symbol when I tried to enable initramfs support).

I was able to find the serial driver for crisv32 in an out-of-kernel tree and port it to 2.6.27 to the point where it loads. I did not attempt to do this with the current upstream kernel, as it crashed before it gets to the point of trying to load the driver.

Quoted 7 lines

The crisv32 serial driver (or at least the version I found) is far from acceptable for upstream integration and would require major cleanup or even rewrite effort if it was to be merged upstream. The version I found is from a 2.6.26 kernel, while the kernel version at the qemu web site (binary only) is 2.6.33, so the driver I worked with is not the latest version and may have been improved later. However, the result was not published, or I was unable to find it.

Quoted 4 lines

Unfortunately, the code quality was the original problem why it wasn't upstreamed at the same time as the rest of the port, and it still hasn't changed in any markable respect. :-(

Is your latest kernel (if possible one running with qemu) available anywhere) ? It would be very useful to be able to access a kernel source which runs with qemu to have at least a baseline for comparison, even if not everything in it is in the upstream kernel.

Quoted 22 lines
Quoted 14 lines

-----------------

Toolchain

Creating a toolchain from upstream sources required a number of patches.

Linux headers:
- Fix the basic headers_install problem mentioned above
- Export ptrace.h
- Some post-processing after installing the headers; specifically, create a couple of symlinks in the headers directory usr/include/arch-v10/arch -> usr/include/arch [for crisv10] usr/include/arch-v32/arch -> usr/include/arch [for crisv32]

Quoted 7 lines

gcc:
- binutils 2.24
- uclibc 0.9.33.2
- gcc 4.7.4
Requires a backport from upstream gcc to compile.
Later versions of gcc (4.8.3, 4.9.1) fail with internal compiler errors even after patching.

Quoted 4 lines

I'm using gcc 4.7.2, but of course, that is a locally built version that might have patches not upstream (although I doubt it knowing the gcc CRIS maintainer :-)

I got 4.9.1 to work (or let's say build) as well after applying another patch from upstream gcc. Of course that does not help much if I don't get to a point where I can actually run it.

Guenter

#9 lore raw

Hi all,

Quoted 51 lines · Guenter Roeck
Quoted 39 lines · Jesper Nilsson

Hi Guenter,

Sorry for not answering earlier, like some have said in
the thread followups, I have been on parental leave for quite some time.

Quoted 2 lines · Guenter Roeck

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

A very ambitious goal. :-)

Quoted 6 lines

After spending a number of days (and nights) on it, the results don't look very encouraging.

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Your conclusion is not completely off, but it could be better than it looks. :-)

I'll try to explain the state of the CRIS port as it currently stands:

(Background: CRIS port supports 3 different SoC:s, where the CRISv10 is older and subsequently less used. The other two are ETRAX FS and ARTPEC-3, which in principle share the same CPU-core (CRISv32) but contain some changes to the peripheral hardware IPs)

CRISv10 is only supported by me as a hobby project, AXIS does not have any current shipping units with this SoC, thus support for this is waning fast. QEMU support is not available for this SoC.
Additionally, newer gcc assumes TLS support, which CRISv10 does not have, and older gcc:s yields an ICE (internal compiler error) on newer kernels.

Units with ETRAX FS is no longer actively upgraded by AXIS with newer kernels, and I have a problem testing anything other than the AXIS 88 developer board (Last relase of the SDK was a couple of years ago with the 2.6.32 kernel IIRC) which is not up to date, but at least have a lot in common with ARTPEC-3 and so is not so hard to support.

Quoted 7 lines

ARTPEC-3 support is not complete as some drivers are not included in upstream (ethernet and serial are the most notable ones) but is actually in best shape, we have 3.16 booting on real hardware in house.

I'll add the missing drivers and current patches we have locally to a git tree on git-hub, I'll get back to you on that.

Would be great.

Quoted 2 lines

It is also the ARTPEC-3 SoC that is implemented in QEMU unless I'm mistaken.

Quoted 8 lines

The only machine supported in qemu is axis-88.

qemu-system-cris -M ?
Supported machines are:
none empty machine
axis-dev88 AXIS devboard 88 (default)

Does that include or imply ARTPEC-3 ?

The machine/board available in upstream QEMU is an ETRAX-FS based one.

I might still have the kernel sources for the image on the QEMU wiki around. Will see if I can find the code and let you know.

Would be awesome if we could get a simple console driver for the ETRAX-FS upstream to at least have the kernel booting to a prompt on QEMU.

Cheers,
Edgar

#10 lore raw
In reply to Guenter Roeck

Hi Günther,

Quoted 9 lines · Guenter Roeck
Quoted 6 lines
Quoted 3 lines

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

Yes, known error, I believe that Sam Ravnborg's patch will correct this.

Is that patch available in public ? Sam's response seems to suggest that he sent it to Michael (Mikael ?) who wanted to handle it, but I don't find a matching patch in public archives.

https://lkml.org/lkml/2014/7/14/424

Quoted 5 lines
Quoted 3 lines

On a tangential note, my automatic (minimal) builds does not do make install, so I haven't seen this error. :-P

Mine not either ;-). I only found it because I tried to build an initramfs which needs installed headers.

http://kisskb.ellerman.id.au/kisskb/buildresult/11881673/

We need an Auto-Builder Consolidation Project ;-)

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#11 lore raw
Quoted 7 lines · Geert Uytterhoeven

Hi Günther,

Quoted 4 lines · Guenter Roeck
Quoted 4 lines
Quoted 3 lines

./scripts/Makefile.headersinst:14: arch/cris/include/uapi/asm/arch-v10/Kbuild:
No such file or directory
make[2]: *** No rule to make target `arch/cris/include/uapi/asm/arch-v10/Kbuild'. Stop.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Yes, known error, I believe that Sam Ravnborg's patch will correct this.

Is that patch available in public ? Sam's response seems to suggest that he sent it to Michael (Mikael ?) who wanted to handle it, but I don't find a matching patch in public archives.

https://lkml.org/lkml/2014/7/14/424

Must be blind. Looked at the series for a long time but didn't see it.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

On a tangential note, my automatic (minimal) builds does not do make install, so I haven't seen this error. :-P

Mine not either ;-). I only found it because I tried to build an initramfs which needs installed headers.

http://kisskb.ellerman.id.au/kisskb/buildresult/11881673/

We need an Auto-Builder Consolidation Project ;-)

More likely we need someone to actually look at build results. Like all the -next builds that are currently failing, and I bet those responsible are not even aware of it. Any ideas ?

Guenter

#12 lore raw
In reply to Jesper Nilsson
Quoted 48 lines · Jesper Nilsson

Hi Guenter,

Sorry for not answering earlier, like some have said in
the thread followups, I have been on parental leave for quite some time.

Quoted 2 lines · Guenter Roeck

The idea was to create a crisv32 kernel and initramfs to work with qemu for the ongoing Linux kernel test project.

A very ambitious goal. :-)

Quoted 6 lines

After spending a number of days (and nights) on it, the results don't look very encouraging.

My overall conclusion is that 'cris' architecture support in the Linux kernel is in bad shape, does not work anymore, and would require substantial effort to get it into working state.

Your conclusion is not completely off, but it could be better than it looks. :-)

I'll try to explain the state of the CRIS port as it currently stands:

(Background: CRIS port supports 3 different SoC:s, where the CRISv10 is older and subsequently less used. The other two are ETRAX FS and ARTPEC-3, which in principle share the same CPU-core (CRISv32) but contain some changes to the peripheral hardware IPs)

CRISv10 is only supported by me as a hobby project, AXIS does not have any current shipping units with this SoC, thus support for this is waning fast. QEMU support is not available for this SoC.
Additionally, newer gcc assumes TLS support, which CRISv10 does not have, and older gcc:s yields an ICE (internal compiler error) on newer kernels.

Units with ETRAX FS is no longer actively upgraded by AXIS with newer kernels, and I have a problem testing anything other than the AXIS 88 developer board (Last relase of the SDK was a couple of years ago with the 2.6.32 kernel IIRC) which is not up to date, but at least have a lot in common with ARTPEC-3 and so is not so hard to support.

ARTPEC-3 support is not complete as some drivers are not included in upstream (ethernet and serial are the most notable ones) but is actually in best shape, we have 3.16 booting on real hardware in house.

I'll add the missing drivers and current patches we have locally to a git tree on git-hub, I'll get back to you on that.

It is also the ARTPEC-3 SoC that is implemented in QEMU unless I'm mistaken.

Quoted 9 lines
Quoted 3 lines

Anyway, below are my individual findings. If there is an ongoing effort to improve cris support in the upstream kernel, specifically support for crisv32, please let me know. I'll be happy to test the resulting kernels.

Thank you for your work, I'll try to add some more information in the hope that this will at least help people google for more information.

Feel free to send me an email if you want to pick this up again.

Just to give you an update. I "kind of" got an image to run with qemu after applying the following patches.

8119d33 cris: Add basic qemu_defconfig
40d078b cris: time.c: Add missing include file to fix compile error a4f2390 cris: Add dummy free_initrd_mem
88585aa cris: Add serial driver for Cris v32
a38d868 cris: Add support for early console

Let me know if you would like a copy of those patches. Out of those, 40d078b, a4f2390, and a38d868 should be pretty much acceptable for upstream integration. 88585aa would need a lot of work (and probably a rewrite).

Compiler version is 4.7.4; 4.9.1 builds even after applying a number of upstream patches, but the resulting image hangs. The weekly gcc snapshot fails to build.

The configuration is (I believe) derived from the configuration used for the image available from the qemu web site. The same is true for the crisv32 serial driver and early console support. time.c needed an additional include file (<linux/mm.h>). free_initrd_mem is needed to be able to build an image with initrd support; it currently does nothing.

With this, I am able to get into a shell using a built-in initrd.

/ # uname -a
Linux (none) 3.16.2-00005-g8119d33 #40 Wed Sep 17 11:49:31 PDT 2014 crisv32 GNU/Linux

However, I see the following traceback.

------------[ cut here ]------------
WARNING: CPU: 0 PID: 186 at kernel/softirq.c:146 0xc000fd72()
Modules linked in:
CPU: 0 PID: 186 Comm: init Not tainted 3.16.2-00005-g8119d33 #40

Stack from c1cf9eb0:

       00000000 c02569d0 00000009 c0279f80 c1fda770 c1fa3680 c02574d6 c000cbb8
       00000092 c000fd72 00000200 c02d0d94 00000000 00000000 c000fd72 00000092
       c000cbea 00000000 c000fd72 c1f862d2 c1cdbc20 c01ff768 c1f862d2 c1f862d6
Call Trace: [<c02569d0>] [<c02574d6>] [<c000cbb8>] [<c000fd72>] [<c000fd72>] [<c000cbea>] [<c000fd72>]
       [<c01ff768>] [<c01ff908>] [<c016f83c>] [<c016fb10>] [<c006cfc0>] [<c025855a>] [<c006d0f4>] [<c001f212>]
       [<c0004b80>] [<c00054f0>]
---[ end trace 5bb306335a1f3b62 ]---

Manual symbol decode (this is from a 3.10 traceback, so the addresses are a bit different):

c0234d8a printk
c0012b0e local_bh_enable
c0236004 dump_stack
c000ca4a warn_slowpath_common
c000ca78 warn_slowpath_null
c0012b0e local_bh_enable
c01e3c16 unix_release_sock
c01e3dae unix_release
c015fb9a sock_release
c015fe68 sock_close
c0067fae __fput
c0237a20 _cond_resched
c0068168 ____fput
c0022062 task_work_run
c0004954 do_notify_resume
c0005324 _work_notifysig

--> Further debugging shows that interrupts are disabled when this happens.

This traceback is seen from 3.10 onwards. 3.4 did not show the traceback; I did not test any releases in between.

Guenter

#13 lore raw
Quoted 9 lines · Guenter Roeck

Just to give you an update. I "kind of" got an image to run with qemu after applying the following patches.

8119d33 cris: Add basic qemu_defconfig
40d078b cris: time.c: Add missing include file to fix compile error a4f2390 cris: Add dummy free_initrd_mem
88585aa cris: Add serial driver for Cris v32
a38d868 cris: Add support for early console

Quoted 3 lines

Let me know if you would like a copy of those patches. Out of those, 40d078b, a4f2390, and a38d868 should be pretty much acceptable for upstream integration. 88585aa would need a lot of work (and probably a rewrite).

Yes please, that would be helpful.

Quoted 33 lines

Compiler version is 4.7.4; 4.9.1 builds even after applying a number of upstream patches, but the resulting image hangs. The weekly gcc snapshot fails to build.

The configuration is (I believe) derived from the configuration used for the image available from the qemu web site. The same is true for the crisv32 serial driver and early console support. time.c needed an additional include file (<linux/mm.h>). free_initrd_mem is needed to be able to build an image with initrd support; it currently does nothing.

With this, I am able to get into a shell using a built-in initrd.

/ # uname -a
Linux (none) 3.16.2-00005-g8119d33 #40 Wed Sep 17 11:49:31 PDT 2014 crisv32 GNU/Linux

However, I see the following traceback.

------------[ cut here ]------------
WARNING: CPU: 0 PID: 186 at kernel/softirq.c:146 0xc000fd72()
Modules linked in:
CPU: 0 PID: 186 Comm: init Not tainted 3.16.2-00005-g8119d33 #40

Stack from c1cf9eb0:

       00000000 c02569d0 00000009 c0279f80 c1fda770 c1fa3680 c02574d6 c000cbb8
       00000092 c000fd72 00000200 c02d0d94 00000000 00000000 c000fd72 00000092
       c000cbea 00000000 c000fd72 c1f862d2 c1cdbc20 c01ff768 c1f862d2 c1f862d6
Call Trace: [<c02569d0>] [<c02574d6>] [<c000cbb8>] [<c000fd72>] [<c000fd72>] [<c000cbea>] [<c000fd72>]
       [<c01ff768>] [<c01ff908>] [<c016f83c>] [<c016fb10>] [<c006cfc0>] [<c025855a>] [<c006d0f4>] [<c001f212>]
       [<c0004b80>] [<c00054f0>]
---[ end trace 5bb306335a1f3b62 ]---

Manual symbol decode (this is from a 3.10 traceback, so the addresses are a bit different):

Quoted 16 lines

c0234d8a printk
c0012b0e local_bh_enable
c0236004 dump_stack
c000ca4a warn_slowpath_common
c000ca78 warn_slowpath_null
c0012b0e local_bh_enable
c01e3c16 unix_release_sock
c01e3dae unix_release
c015fb9a sock_release
c015fe68 sock_close
c0067fae __fput
c0237a20 _cond_resched
c0068168 ____fput
c0022062 task_work_run
c0004954 do_notify_resume
c0005324 _work_notifysig

I'm not sure if this has happened here, but please note that some of the symbols in the stack backtrace might be false, since the CRIS does not have true stack unwinding, the backtrace code only checks the stack for any pointers in the kernel address space and reports them too.

Quoted 4 lines

--> Further debugging shows that interrupts are disabled when this happens.

This traceback is seen from 3.10 onwards. 3.4 did not show the traceback; I did not test any releases in between.

Ok thanks, will try to reproduce here.

Quoted 1 line

Guenter

/^JN - Jesper Nilsson

Signature · 1 line
               Jesper Nilsson -- [email protected]
#14 lore raw
Quoted 73 lines · Jesper Nilsson
Quoted 9 lines · Guenter Roeck

Just to give you an update. I "kind of" got an image to run with qemu after applying the following patches.

8119d33 cris: Add basic qemu_defconfig
40d078b cris: time.c: Add missing include file to fix compile error a4f2390 cris: Add dummy free_initrd_mem
88585aa cris: Add serial driver for Cris v32
a38d868 cris: Add support for early console

Quoted 3 lines

Let me know if you would like a copy of those patches. Out of those, 40d078b, a4f2390, and a38d868 should be pretty much acceptable for upstream integration. 88585aa would need a lot of work (and probably a rewrite).

Yes please, that would be helpful.

Quoted 33 lines

Compiler version is 4.7.4; 4.9.1 builds even after applying a number of upstream patches, but the resulting image hangs. The weekly gcc snapshot fails to build.

The configuration is (I believe) derived from the configuration used for the image available from the qemu web site. The same is true for the crisv32 serial driver and early console support. time.c needed an additional include file (<linux/mm.h>). free_initrd_mem is needed to be able to build an image with initrd support; it currently does nothing.

With this, I am able to get into a shell using a built-in initrd.

/ # uname -a
Linux (none) 3.16.2-00005-g8119d33 #40 Wed Sep 17 11:49:31 PDT 2014 crisv32 GNU/Linux

However, I see the following traceback.

------------[ cut here ]------------
WARNING: CPU: 0 PID: 186 at kernel/softirq.c:146 0xc000fd72()
Modules linked in:
CPU: 0 PID: 186 Comm: init Not tainted 3.16.2-00005-g8119d33 #40

Stack from c1cf9eb0:

        00000000 c02569d0 00000009 c0279f80 c1fda770 c1fa3680 c02574d6 c000cbb8
        00000092 c000fd72 00000200 c02d0d94 00000000 00000000 c000fd72 00000092
        c000cbea 00000000 c000fd72 c1f862d2 c1cdbc20 c01ff768 c1f862d2 c1f862d6
Call Trace: [<c02569d0>] [<c02574d6>] [<c000cbb8>] [<c000fd72>] [<c000fd72>] [<c000cbea>] [<c000fd72>]
        [<c01ff768>] [<c01ff908>] [<c016f83c>] [<c016fb10>] [<c006cfc0>] [<c025855a>] [<c006d0f4>] [<c001f212>]
        [<c0004b80>] [<c00054f0>]
---[ end trace 5bb306335a1f3b62 ]---

Manual symbol decode (this is from a 3.10 traceback, so the addresses are a bit different):

Quoted 16 lines

c0234d8a printk
c0012b0e local_bh_enable
c0236004 dump_stack
c000ca4a warn_slowpath_common
c000ca78 warn_slowpath_null
c0012b0e local_bh_enable
c01e3c16 unix_release_sock
c01e3dae unix_release
c015fb9a sock_release
c015fe68 sock_close
c0067fae __fput
c0237a20 _cond_resched
c0068168 ____fput
c0022062 task_work_run
c0004954 do_notify_resume
c0005324 _work_notifysig

I'm not sure if this has happened here, but please note that some of the symbols in the stack backtrace might be false, since the CRIS does not have true stack unwinding, the backtrace code only checks the stack for any pointers in the kernel address space and reports them too.

Quoted 7 lines
Quoted 4 lines

--> Further debugging shows that interrupts are disabled when this happens.

This traceback is seen from 3.10 onwards. 3.4 did not show the traceback; I did not test any releases in between.

Ok thanks, will try to reproduce here.

I just sent the series. My suspicion is that you might have some relevant changes in architecture code which never made it upstream.

Guenter