Random memory corruption with v5.2
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Do you have some examples of crashes?
I am running 5.2-rc7 on a large number QEMU instances for builders, and I see some strange behavior, but I haven't noticed any issues on the board using OpenEmbedded build with the final 5.2 yet.
[17983.074847] Unable to handle kernel paging request at virtual address 0fffffdff5e14700 [17983.085132] Oops [#1]
[133953.710130] kernel BUG at include/linux/mm.h:1023! [133953.718204] Kernel BUG [#1]
[165770.567652] Unable to handle kernel NULL pointer dereference at
virtual address 0000000000000010
[148578.912479] kernel BUG at lib/list_debug.c:51! [148578.917701] Kernel BUG [#1]
[163756.869949] EXT4-fs (vda2): pa 00000000e9971722: logic 512, phys. 2558464, len 512 [163756.889549] EXT4-fs error (device vda2): ext4_mb_release_inode_pa:3837: group 78, free 0, pa_free 149 [163757.757600] EXT4-fs (vda2): pa 0000000066b479c3: logic 32, phys.
2558368, len 96
sbi_trap_error: hart1: misaligned store handler failed (error -10)sbi_trap_error: hart1: mcause=0x0000000000000006 mtval=0x00000000000002c3sbi_trap_error: hart1: mepc=0xffffffe0009dc1f4 mstatus=0x0000000000000802sbi_trap_error: hart1: ra=0xffffffe0009dc1ee sp=0xffffffe1f3c17be0
[178876.406122] Unable to handle kernel paging request at virtual address 0000000000012a28 [178876.423941] Oops [#1]
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Quoted 6 lines
Do you have some examples of crashes?
I am running 5.2-rc7 on a large number QEMU instances for builders, and I see some strange behavior, but I haven't noticed any issues on the board using OpenEmbedded build with the final 5.2 yet.
Looking at the timestamps, these seems to be different crashes in different instances. Is there any particular workload you were running or just happens randomly if you run long enough ?
If you have complete dmesg and/or vmlinux that will help as well.
Quoted 34 lines
[17983.074847] Unable to handle kernel paging request at virtual address 0fffffdff5e14700 [17983.085132] Oops [#1][133953.710130] kernel BUG at include/linux/mm.h:1023! [133953.718204] Kernel BUG [#1][165770.567652] Unable to handle kernel NULL pointer dereference atvirtual address 0000000000000010
[148578.912479] kernel BUG at lib/list_debug.c:51! [148578.917701] Kernel BUG [#1][163756.869949] EXT4-fs (vda2): pa 00000000e9971722: logic 512, phys. 2558464, len 512 [163756.889549] EXT4-fs error (device vda2): ext4_mb_release_inode_pa:3837: group 78, free 0, pa_free 149 [163757.757600] EXT4-fs (vda2): pa 0000000066b479c3: logic 32, phys.2558368, len 96
sbi_trap_error: hart1: misaligned store handler failed (error -10)sbi_trap_error: hart1: mcause=0x0000000000000006 mtval=0x00000000000002c3sbi_trap_error: hart1: mepc=0xffffffe0009dc1f4 mstatus=0x0000000000000802sbi_trap_error: hart1: ra=0xffffffe0009dc1ee sp=0xffffffe1f3c17be0[178876.406122] Unable to handle kernel paging request at virtual address 0000000000012a28 [178876.423941] Oops [#1]_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Signature · 7 lines
Regards, Atish _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 7 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT
/glibc-2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134
While building gcc:
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here
+===========================GNAT BUG DETECTED==============================+ | 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) | | Storage_Error stack overflow or erroneous memory access | | Error detected at output.ads:39:8 |
realloc(): invalid pointer
raised PROGRAM_ERROR : unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140: ada/exp_ch3.o] Error 1
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 18 lines · Andreas Schwab
Quoted 7 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1--library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef$flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc-2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134While building gcc:
Quoted 8 lines
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here+===========================GNAT BUG DETECTED==============================+ | 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) | | Storage_Error stack overflow or erroneous memory access | | Error detected at output.ads:39:8 |
realloc(): invalid pointer
I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.
Reflecting on the recent commits, there weren't too many recent RISC-V-specific changes that could have an impact here. So if these problems are relatively repeatable, and they didn't happen with v5.1, there are a few patches that might be worth reverting to see if the situation improves. Here would be my short list:
- Commit bf587caae305ae3b4393077fb22c98478ee55755 ("riscv: mm: synchronize MMU after pte change")
- Commit 6dd91e0eacff0a5c822ca37565d6b5740c4d2a80 ("RISC-V: defconfig:
Enable NO_HZ_IDLE and HIGH_RES_TIMERS")
- Commit 671f9a3e2e24cdeb2d2856abee7422f093e23e29 ("RISC-V: Setup initial page tables in two stages")
Of course, it's also possible that someone could have made a change outside arch/riscv that are causing these problems. If that's the case, we're probably stuck bisecting it.
- Paul
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 2 lines · Paul Walmsley
- Commit
bf587caae305ae3b4393077fb22c98478ee55755("riscv: mm: synchronize MMU after pte change")
That would be my favorite.
Quoted 2 lines
- Commit
6dd91e0eacff0a5c822ca37565d6b5740c4d2a80("RISC-V: defconfig:
EnableNO_HZ_IDLEandHIGH_RES_TIMERS")
I had these enabled forever already.
Quoted 2 lines
- Commit
671f9a3e2e24cdeb2d2856abee7422f093e23e29("RISC-V: Setup initial page tables in two stages")
I don't think a one-time initial setup can have such a subtle effect.
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 11 lines · Andreas Schwab
Quoted 2 lines · Paul Walmsley
- Commit
bf587caae305ae3b4393077fb22c98478ee55755("riscv: mm: synchronize MMU after pte change")That would be my favorite.
Quoted 2 lines
- Commit
6dd91e0eacff0a5c822ca37565d6b5740c4d2a80("RISC-V: defconfig:
EnableNO_HZ_IDLEandHIGH_RES_TIMERS")
Quoted 6 lines
I had these enabled forever already.
Quoted 2 lines
- Commit
671f9a3e2e24cdeb2d2856abee7422f093e23e29("RISC-V: Setup initial page tables in two stages")I don't think a one-time initial setup can have such a subtle effect.
The initial page table setup patch is not present in 5.2. It was merged in 5.3.
Regards,
Anup
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 2 lines · Paul Walmsley
- Commit
bf587caae305ae3b4393077fb22c98478ee55755("riscv: mm: synchronize MMU after pte change")
When I revert that commit, I'm getting soft lockups. Doesn't that point to some deeper issue with TLB flushes?
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 13 lines · Andreas Schwab
Quoted 2 lines · Paul Walmsley
- Commit
bf587caae305ae3b4393077fb22c98478ee55755("riscv: mm: synchronize MMU after pte change")When I revert that commit, I'm getting soft lockups. Doesn't that point to some deeper issue with TLB flushes?
Andreas.
What are you using for filesystem/storage? Is it the SDcard, network, or something else?
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 1 line · Troy Benjegerdes
What are you using for filesystem/storage?
NFS.
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Andreas Schwab
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1--library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef$flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDR
OOT/glibc-2.29-0.riscv64 $locale
Quoted 13 lines
Quoted 4 lines
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134
While building gcc:
Quoted 8 lines
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here+===========================GNAT BUG DETECTED==============================+ | 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) | | Storage_Error stack overflow or erroneous memory access | | Error detected at output.ads:39:8 |
realloc(): invalid pointer
Quoted 2 lines
I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.
I think there is some fundamental problem with SBI_REMOTE_SFENCE_VMA or the kernel interface to it.
For exmaple, flush_tlb_page is defined as:
#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)
But the third argument of flush_tlb_range is supposed to be the end address, so this should actually be:
#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)
Alas, that doesn't fix the crashes.
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 6 lines · On Wed, 31 Jul 2019 00:39:10 PDT (-0700), [email protected] wrote:
Quoted 2 lines · Paul Walmsley
- Commit
bf587caae305ae3b4393077fb22c98478ee55755("riscv: mm: synchronize MMU after pte change")That would be my favorite.
If that patch causes memory corruption then something scary is going on. I haven't been following the thread closely enough to know how easy this is to reproduce, but do you mind trying a kernel with a reverted version of that commit?
This is also availiable on the "for-andreas" branch of git.kernel.org/palmer/linux.git
commit 07d45256aa8641057c141f1a661bb29dd99eb32e
gpg: Signature made Thu 01 Aug 2019 12:46:22 PM PDT
gpg: using RSA key 00CE76D1834960DFCE886DF8EF4CA1502CCBAB41
gpg: issuer "[email protected]"
gpg: Good signature from "Palmer Dabbelt <[email protected]>" [ultimate]
gpg: aka "Palmer Dabbelt <[email protected]>" [ultimate]
Author: Palmer Dabbelt <[email protected]>
Date: Thu Aug 1 12:45:12 2019 -0700
Revert "riscv: mm: synchronize MMU after pte change"
Andreas Schwab is seeing some random memory corruption with 5.2, and he thinks the reverted comit is the most likely candidate. The commit itself doesn't revert cleanly, but that's just because getting the comment right took two commits.
If this does fix the issue then we're in a bit of trouble, as this TLB flush should be pretty safe.
This reverts commit bf587caae305ae3b4393077fb22c98478ee55755.
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index f960c3f4ce47..28dccb072255 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -16,7 +16,6 @@
#include <asm/pgalloc.h>
#include <asm/ptrace.h> -#include <asm/tlbflush.h>
/*
* This routine handles page faults. It determines the address and the
@@ -267,14 +266,6 @@ asmlinkage void do_page_fault(struct pt_regs *regs)
if (!pte_present(*pte_k))
goto no_context;- /*
- * The kernel assumes that TLBs don't cache invalid
- * entries, but in RISC-V, SFENCE.VMA specifies an
- * ordering constraint, not a cache flush; it is
- * necessary even after writing invalid entries.
- */
- local_flush_tlb_page(addr);
-
return;
}
}Quoted 4 lines
Quoted 2 lines
- Commit
6dd91e0eacff0a5c822ca37565d6b5740c4d2a80("RISC-V: defconfig:
EnableNO_HZ_IDLEandHIGH_RES_TIMERS")I had these enabled forever already.
IIRC that was the argument for enabling them in defconfig :)
Quoted 4 lines
Quoted 2 lines
- Commit
671f9a3e2e24cdeb2d2856abee7422f093e23e29("RISC-V: Setup initial page tables in two stages")I don't think a one-time initial setup can have such a subtle effect.
As per Anup, it's not in 5.2.
Quoted 2 lines
Andreas.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 12 lines · On Thu, 01 Aug 2019 11:32:33 PDT (-0700), [email protected] wrote:
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Andreas Schwab
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1--library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef$flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILD
ROOT/glibc-2.29-0.riscv64
Quoted 22 lines
Quoted 14 lines
Quoted 5 lines
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134
While building gcc:
Quoted 8 lines
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here+===========================GNAT BUG DETECTED==============================+ | 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) | | Storage_Error stack overflow or erroneous memory access | | Error detected at output.ads:39:8 |
realloc(): invalid pointerQuoted 2 lines
I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.
I think there is some fundamental problem with
SBI_REMOTE_SFENCE_VMAor the kernel interface to it.For exmaple,
flush_tlb_pageis defined as:
Quoted 8 lines
#define
flush_tlb_page(vma, addr)flush_tlb_range(vma, addr, 0)But the third argument of
flush_tlb_rangeis supposed to be the end address, so this should actually be:#define
flush_tlb_page(vma, addr)flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)Alas, that doesn't fix the crashes.
This line of reasoning smells like it'd find the issue: BBL just flushes the entire TLB every time, but IIRC OpenSBI respects the ranges. It looks like
Fixes: 90cb4917b584 ("lib: Implement sfence.vma correctly.")is what introduced the new behavior in OpenSBI, which may have triggered a lot of latent bugs in Linux. If you have an easy way to compile OpenSBI, does something like
$ git diff | cat
diff --git a/lib/sbi/sbi_tlb.c b/lib/sbi/sbi_tlb.c
index cffda52d66ab..007266b1f970 100644
--- a/lib/sbi/sbi_tlb.c
+++ b/lib/sbi/sbi_tlb.c
@@ -133,50 +133,12 @@ static void sbi_tlb_flush_all(void) static void sbi_tlb_fifo_sfence_vma(struct sbi_tlb_info *tinfo) {
- unsigned long start = tinfo->start;
- unsigned long size = tinfo->size;
- unsigned long i;
-
- if ((start == 0 && size == 0) || (size == SBI_TLB_FLUSH_ALL)) {
- sbi_tlb_flush_all();
- return;
- }
-
- for (i = 0; i < size; i += PAGE_SIZE) {
- __asm__ __volatile__("sfence.vma %0"
- :
- : "r"(start + i)
- : "memory");
- }
+ sbi_tlb_flush_all();
}
static void sbi_tlb_fifo_sfence_vma_asid(struct sbi_tlb_info *tinfo) {
- unsigned long start = tinfo->start;
- unsigned long size = tinfo->size;
- unsigned long asid = tinfo->asid;
- unsigned long i;
-
- if (start == 0 && size == 0) {
- sbi_tlb_flush_all();
- return;
- }
-
- /* Flush entire MM context for a given ASID */
- if (size == SBI_TLB_FLUSH_ALL) {
- __asm__ __volatile__("sfence.vma x0, %0"
- :
- : "r"(asid)
- : "memory");
- return;
- }
-
- for (i = 0; i < size; i += PAGE_SIZE) {
- __asm__ __volatile__("sfence.vma %0, %1"
- :
- : "r"(start + i), "r"(asid)
- : "memory");
- }
+ sbi_tlb_flush_all();
}
void sbi_tlb_fifo_process(struct sbi_scratch *scratch, u32 event)
cause the issue to go away? If so, then I'd bet we need to scour Linux for broken TLB flushing, as given the one you found is pretty obvious I'd bet there's a lot more...
Quoted 2 lines
Andreas.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 13 lines · Andreas Schwab
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Andreas Schwab
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1--library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef$flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUIL
DROOT/glibc-2.29-0.riscv64 $locale
Quoted 19 lines
Quoted 13 lines
Quoted 4 lines
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134
While building gcc:
Quoted 8 lines
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here+===========================GNAT BUG DETECTED==============================+ | 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) | | Storage_Error stack overflow or erroneous memory access | | Error detected at output.ads:39:8 |
realloc(): invalid pointerQuoted 2 lines
I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.
I think there is some fundamental problem with
SBI_REMOTE_SFENCE_VMAor the kernel interface to it.
Quoted 8 lines
For exmaple,
flush_tlb_pageis defined as:#define
flush_tlb_page(vma, addr)flush_tlb_range(vma, addr, 0)But the third argument of
flush_tlb_rangeis supposed to be the end address, so this should actually be:#define
flush_tlb_page(vma, addr)flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)
Instead of this can you try -1UL as the size:
#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, -1UL)
If above works for you then there is some issue with range of virtual memory we flush.
Regards,
Anup
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
I was able to build glibc, and run most of the test suite, on v5.3-rc2 with BBL, with no problems so far. (The test suite is still running.) CPU at 1GHz.
Will try gcc in a few hours.
- Paul
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 1 line · Paul Walmsley
I was able to build glibc, and run most of the test suite, on v5.3-rc2
Did you run install-locales?
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 5 lines · Andreas Schwab
Quoted 1 line · Paul Walmsley
I was able to build glibc, and run most of the test suite, on v5.3-rc2
Did you run install-locales?
I just ran "make -j4", "make -j4 check", "make -j4 xcheck". This is with rootfs on microSD, rather than on NFS.
Do you still see the failures if you only run the above commands, or does the failure only appear with install-locales?
- Paul
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 2 lines · Paul Walmsley
Do you still see the failures if you only run the above commands, or does the failure only appear with install-locales?
Only during install-locales.
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 2 lines · Anup Patel
Instead of this can you try -1UL as the size:
#defineflush_tlb_page(vma, addr)flush_tlb_range(vma, addr, -1UL)
That doesn't help either.
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
But this does help:
arch/riscv/include/asm/tlbflush.h +1 -1
@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,cpumask_clear (&hmask);riscv_cpuid_to_hartid_mask (cmask, &hmask); -sbi_remote_sfence_vma (hmask.bits, start, size); +sbi_remote_sfence_vma (hmask.bits,0 ,-1 ); }#define flush_tlb_all () sbi_remote_sfence_vma(NULL, 0, -1)
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
This helps too:
arch/riscv/include/asm/tlbflush.h +2 -1
@@ -50,10 +50,11 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,cpumask_clear (&hmask);riscv_cpuid_to_hartid_mask (cmask, &hmask);sbi_remote_sfence_vma (hmask.bits, start, size); +local_flush_tlb_all (); }#define flush_tlb_all () sbi_remote_sfence_vma(NULL, 0, -1) -#define flush_tlb_page (vma, addr) flush_tlb_range(vma, addr, 0) +#define flush_tlb_page (vma, addr) flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)#define flush_tlb_range (vma, start, end) \ remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start))#define flush_tlb_mm (mm) \
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 13 lines · Andreas Schwab
But this does help:
arch/riscv/include/asm/tlbflush.h +1
-1@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,cpumask_clear (&hmask);riscv_cpuid_to_hartid_mask (cmask, &hmask); -sbi_remote_sfence_vma (hmask.bits, start, size); +sbi_remote_sfence_vma (hmask.bits,0 ,-1 ); }
Quoted 2 lines
#define
flush_tlb_all()sbi_remote_sfence_vma(NULL, 0, -1)
I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.
(quoted diff) +0 -0
b/arch/riscv/include/asm/tlbflush.h
index 687dd19735a7..29b2bd7c9923 100644
arch/riscv/include/asm/tlbflush.h +0 -0
@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(struct cpumask
*cmask, unsigned long start,
#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1) #define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0) #define flush_tlb_range(vma, start, end) \
- remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) -
(start))
+ remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)
#define flush_tlb_mm(mm) \
remote_sfence_vma(mm_cpumask(mm), 0, -1)Can you please verify at your end?
While your fix flushes the entire tlb for every type of remote tlb flush, this fix proves that the issue is with flush_tlb_range call only.
I am looking at the OpenSBI/Kernel implementation to figure out if it is an OpenSBI issue or something changed in kernel recently to trigger this.
Additionally, do you know if a particular or group of locale install is causing this issue?
It takes more than hour to finish the full install-locales on unleashe board which makes it bit difficult to try out possible fixes.
Quoted 2 lines
Andreas.
Signature · 6 lines
Regards, Atish _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 51 lines · Atish Patra
Quoted 13 lines · Andreas Schwab
But this does help:
arch/riscv/include/asm/tlbflush.h +1
-1@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,cpumask_clear (&hmask);riscv_cpuid_to_hartid_mask (cmask, &hmask); -sbi_remote_sfence_vma (hmask.bits, start, size); +sbi_remote_sfence_vma (hmask.bits,0 ,-1 ); }Quoted 2 lines
#define
flush_tlb_all()sbi_remote_sfence_vma(NULL, 0, -1)I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.
arch/riscv/include/asm/tlbflush.h +1
-1@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,#define flush_tlb_all () sbi_remote_sfence_vma(NULL, 0, -1)#define flush_tlb_page (vma, addr) flush_tlb_range(vma, addr, 0)#define flush_tlb_range (vma, start, end) \ - remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start)) + remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)#define flush_tlb_mm (mm) \remote_sfence_vma(mm_cpumask(mm), 0, -1)Can you please verify at your end?
While your fix flushes the entire tlb for every type of remote tlb flush, this fix proves that the issue is with
flush_tlb_rangecall only.I am looking at the OpenSBI/Kernel implementation to figure out if it is an OpenSBI issue or something changed in kernel recently to trigger this.
Quoted 6 lines
Additionally, do you know if a particular or group of locale install is causing this issue?
It takes more than hour to finish the full install-locales on unleashe board which makes it bit difficult to try out possible fixes.
Did you reproduce with SDcard, or NFS?
Quoted 11 lines
Quoted 3 lines
Andreas.
Signature · 6 lines
Regards, Atish _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 58 lines · Troy Benjegerdes
Quoted 58 lines
On Aug 5, 2019, at 5:34 PM, Atish Patra <[email protected]> wrote:
Quoted 14 lines · Andreas Schwab
But this does help:
arch/riscv/include/asm/tlbflush.h +0
-0@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(structcpumask
*cmask, unsigned long start,cpumask_clear(&hmask); riscv_cpuid_to_hartid_mask(cmask, &hmask);arch/riscv/include/asm/tlbflush.h +1
-1-sbi_remote_sfence_vma (hmask.bits, start, size); +sbi_remote_sfence_vma (hmask.bits,0 ,-1 );}
Quoted 2 lines
#define
flush_tlb_all()sbi_remote_sfence_vma(NULL, 0, -1)I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.
arch/riscv/include/asm/tlbflush.h +0
-0@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(structcpumask
*cmask, unsigned long start,
#defineflush_tlb_all()sbi_remote_sfence_vma(NULL, 0, -1)#defineflush_tlb_page(vma, addr)flush_tlb_range(vma, addr, 0)#defineflush_tlb_range(vma, start, end)\
- remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) -
(start))
+remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)#define flush_tlb_mm(mm) \ remote_sfence_vma(mm_cpumask(mm), 0, -1)Can you please verify at your end?
While your fix flushes the entire tlb for every type of remote tlb flush, this fix proves that the issue is with
flush_tlb_rangecall only.I am looking at the OpenSBI/Kernel implementation to figure out if it
is an OpenSBI issue or something changed in kernel recently to trigger
this.Additionally, do you know if a particular or group of locale install
is causing this issue?
Quoted 7 lines
Quoted 4 lines
It takes more than hour to finish the full install-locales on unleashe
board which makes it bit difficult to try out possible fixes.Did you reproduce with SDcard, or NFS?
I am running it on a nvme SSD attached to Microsemi expansion board.
Kernel version: 5.3-rc2
OpenSBI/U-Boot: Latest master
Regards,
Atish
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Andreas.
Signature · 6 lines
Regards, Atish _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 2 lines · Atish Patra
It takes more than hour to finish the full install-locales on unleashe board which makes it bit difficult to try out possible fixes.
When it fails it usually fails pretty fast. Did you run it in parallel?
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 36 lines · Atish Patra
Quoted 13 lines · Andreas Schwab
But this does help:
arch/riscv/include/asm/tlbflush.h +1
-1@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,cpumask_clear (&hmask);riscv_cpuid_to_hartid_mask (cmask, &hmask); -sbi_remote_sfence_vma (hmask.bits, start, size); +sbi_remote_sfence_vma (hmask.bits,0 ,-1 ); }Quoted 2 lines
#define
flush_tlb_all()sbi_remote_sfence_vma(NULL, 0, -1)I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.
arch/riscv/include/asm/tlbflush.h +1
-1@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,#define flush_tlb_all () sbi_remote_sfence_vma(NULL, 0, -1)#define flush_tlb_page (vma, addr) flush_tlb_range(vma, addr, 0)#define flush_tlb_range (vma, start, end) \ - remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start)) + remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)#define flush_tlb_mm (mm) \remote_sfence_vma(mm_cpumask(mm), 0, -1)
Quoted 1 line
Can you please verify at your end?
This is equivalent to my patch since all other uses already pass 0,-1.
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 52 lines · Andreas Schwab
Quoted 8 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc.
Perhaps missing TLB flushes?Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped)
I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/iconvdata LC_ALL=C/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef$flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc- 2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134While building gcc:
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here +===========================GNAT BUG DETECTED==============================+Quoted 6 lines
10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64- suse-linux) |
Storage_Errorstack overflow or erroneous memory
access |
Error detected at
output.ads:39:8 |
Quoted 8 lines
realloc(): invalid pointerraised
PROGRAM_ERROR: unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140:ada/exp_ch3.o] Error 1Andreas.
Can you give it a try with following patch in OpenSBI & Kernel ?
Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.html
OpenSBI:
http://lists.infradead.org/pipermail/opensbi/2019-August/000386.html
In my testing, I no longer the stress-ng error or glibc local install issue if I use following command.
sudo make -j8 localedata/install-locale-files
DESTDIR=/home/atish/glibc/build/install
I still see segmentation fault if I use a archieve locale install command.
sudo make -j8 localedata/install-locales
DESTDIR=/home/atish/glibc/build/install
But the error dump doesn't contain remap() error. Just a segmentation fault which may be due to userspace or just different version of old tlbflush problem.
Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 78 lines · Atish Patra
Quoted 53 lines · Andreas Schwab
Quoted 8 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc.
Perhaps missing TLB flushes?Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped)
I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/iconvdata LC_ALL=C/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef$flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc- 2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error
134While building gcc:
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here +===========================GNAT BUG DETECTED==============================+Quoted 6 lines
10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64- suse-linux) |
Storage_Errorstack overflow or erroneous memory
access |
Error detected at
output.ads:39:8 |Quoted 8 lines
realloc(): invalid pointerraised
PROGRAM_ERROR: unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140:ada/exp_ch3.o] Error 1Andreas.
Can you give it a try with following patch in OpenSBI & Kernel ?
Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.htmlOpenSBI:
http://lists.infradead.org/pipermail/opensbi/2019-August/000386.htmlIn my testing, I no longer the stress-ng error or glibc local install issue if I use following command.
sudo make -j8 localedata/install-locale-files
DESTDIR=/home/atish/glibc/build/install
Quoted 6 lines
I still see segmentation fault if I use a archieve locale install command.
sudo make -j8 localedata/install-locales
DESTDIR=/home/atish/glibc/build/install
I am also able to run above archive locale install command successfully multiple times after removing the corrupted locale-archive files present in install path.
Let me know if it works for you as well.
I am now running stress-ng & parallel glibc locale install together to fully stress the system.
Regards,
Atish
Quoted 8 lines
But the error dump doesn't contain
remap()error. Just a segmentation fault which may be due to userspace or just different version of old tlbflush problem.Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 94 lines · Atish Patra
Quoted 78 lines · Atish Patra
Quoted 53 lines · Andreas Schwab
Quoted 8 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab
Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc.
Perhaps missing TLB flushes?Do you have some examples of crashes?
While building glibc:
an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped)
I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/iconvdata LC_ALL=C/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef$flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc- 2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error
134While building gcc:
../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here +===========================GNAT BUG DETECTED==============================+Quoted 6 lines
10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64- suse-linux) |
Storage_Errorstack overflow or erroneous memory
access |
Error detected at
output.ads:39:8 |Quoted 8 lines
realloc(): invalid pointerraised
PROGRAM_ERROR: unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140:ada/exp_ch3.o] Error 1Andreas.
Can you give it a try with following patch in OpenSBI & Kernel ?
Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.htmlOpenSBI:
http://lists.infradead.org/pipermail/opensbi/2019-August/000386.htmlIn my testing, I no longer the stress-ng error or glibc local install issue if I use following command.
sudo make -j8 localedata/install-locale-files
DESTDIR=/home/atish/glibc/build/installQuoted 6 lines
I still see segmentation fault if I use a archieve locale install command.
sudo make -j8 localedata/install-locales
DESTDIR=/home/atish/glibc/build/installI am also able to run above archive locale install command successfully multiple times after removing the corrupted locale-archive files present in install path.
Quoted 17 lines
Let me know if it works for you as well.
I am now running stress-ng & parallel glibc locale install together to fully stress the system.
Regards,
AtishQuoted 8 lines
But the error dump doesn't contain
remap()error. Just a segmentation fault which may be due to userspace or just different version of old tlbflush problem.Regards,
Atish
Is this with stock linux-5.2.8 release, with no additional patches, or is there something we need to look at backporting?
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 2 lines · Atish Patra
Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.html
I've been using that patch, without any changes to openSBI, to run bootstrap/regtest on gcc and to build glibc without issues.
Andreas.
Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected] GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv