Skip to contentSkip to conversation

Random memory corruption with v5.2

DISCUSSION Andreas Schwab 7 participants 30 messages Jul 29, 2019 – Aug 19, 2019 RISC-V

lore
Share & tools
lore
#0 lore raw

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#1 lore raw
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Do you have some examples of crashes?

I am running 5.2-rc7 on a large number QEMU instances for builders, and I see some strange behavior, but I haven't noticed any issues on the board using OpenEmbedded build with the final 5.2 yet.

[17983.074847] Unable to handle kernel paging request at virtual
address 0fffffdff5e14700
[17983.085132] Oops [#1]
[133953.710130] kernel BUG at include/linux/mm.h:1023!
[133953.718204] Kernel BUG [#1]
[165770.567652] Unable to handle kernel NULL pointer dereference at

virtual address 0000000000000010

[148578.912479] kernel BUG at lib/list_debug.c:51!
[148578.917701] Kernel BUG [#1]
[163756.869949] EXT4-fs (vda2): pa 00000000e9971722: logic 512, phys.
2558464, len 512
[163756.889549] EXT4-fs error (device vda2):
ext4_mb_release_inode_pa:3837: group 78, free 0, pa_free 149
[163757.757600] EXT4-fs (vda2): pa 0000000066b479c3: logic 32, phys.

2558368, len 96

sbi_trap_error: hart1: misaligned store handler failed (error -10)
sbi_trap_error: hart1: mcause=0x0000000000000006 mtval=0x00000000000002c3
sbi_trap_error: hart1: mepc=0xffffffe0009dc1f4 mstatus=0x0000000000000802
sbi_trap_error: hart1: ra=0xffffffe0009dc1ee sp=0xffffffe1f3c17be0

[178876.406122] Unable to handle kernel paging request at virtual
address 0000000000012a28
[178876.423941] Oops [#1]

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#2 lore raw
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Quoted 6 lines

Do you have some examples of crashes?

I am running 5.2-rc7 on a large number QEMU instances for builders, and I see some strange behavior, but I haven't noticed any issues on the board using OpenEmbedded build with the final 5.2 yet.

Looking at the timestamps, these seems to be different crashes in different instances. Is there any particular workload you were running or just happens randomly if you run long enough ?

If you have complete dmesg and/or vmlinux that will help as well.

Quoted 34 lines
[17983.074847] Unable to handle kernel paging request at virtual
address 0fffffdff5e14700
[17983.085132] Oops [#1]
[133953.710130] kernel BUG at include/linux/mm.h:1023!
[133953.718204] Kernel BUG [#1]
[165770.567652] Unable to handle kernel NULL pointer dereference at

virtual address 0000000000000010

[148578.912479] kernel BUG at lib/list_debug.c:51!
[148578.917701] Kernel BUG [#1]
[163756.869949] EXT4-fs (vda2): pa 00000000e9971722: logic 512, phys.
2558464, len 512
[163756.889549] EXT4-fs error (device vda2):
ext4_mb_release_inode_pa:3837: group 78, free 0, pa_free 149
[163757.757600] EXT4-fs (vda2): pa 0000000066b479c3: logic 32, phys.

2558368, len 96

sbi_trap_error: hart1: misaligned store handler failed (error -10)
sbi_trap_error: hart1: mcause=0x0000000000000006 mtval=0x00000000000002c3
sbi_trap_error: hart1: mepc=0xffffffe0009dc1f4 mstatus=0x0000000000000802
sbi_trap_error: hart1: ra=0xffffffe0009dc1ee sp=0xffffffe1f3c17be0

[178876.406122] Unable to handle kernel paging request at virtual
address 0000000000012a28
[178876.423941] Oops [#1]

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

Signature · 7 lines
Regards,
Atish

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#3 lore raw
In reply to David Abdurachmanov
Quoted 7 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT
/glibc-2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134

While building gcc:

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here

+===========================GNAT BUG DETECTED==============================+
| 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) |
| Storage_Error stack overflow or erroneous memory access                  |
| Error detected at output.ads:39:8                                        |

realloc(): invalid pointer

raised PROGRAM_ERROR : unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140: ada/exp_ch3.o] Error 1

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#4 lore raw
Quoted 18 lines · Andreas Schwab
Quoted 7 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc-2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134

While building gcc:

Quoted 8 lines

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here

+===========================GNAT BUG DETECTED==============================+
| 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) |
| Storage_Error stack overflow or erroneous memory access                  |
| Error detected at output.ads:39:8                                        |

realloc(): invalid pointer

I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.

Reflecting on the recent commits, there weren't too many recent RISC-V-specific changes that could have an impact here. So if these problems are relatively repeatable, and they didn't happen with v5.1, there are a few patches that might be worth reverting to see if the situation improves. Here would be my short list:

- Commit bf587caae305ae3b4393077fb22c98478ee55755 ("riscv: mm: synchronize MMU after pte change")

- Commit 6dd91e0eacff0a5c822ca37565d6b5740c4d2a80 ("RISC-V: defconfig:
Enable NO_HZ_IDLE and HIGH_RES_TIMERS")

- Commit 671f9a3e2e24cdeb2d2856abee7422f093e23e29 ("RISC-V: Setup initial page tables in two stages")

Of course, it's also possible that someone could have made a change outside arch/riscv that are causing these problems. If that's the case, we're probably stuck bisecting it.

- Paul

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#5 lore raw
Quoted 2 lines · Paul Walmsley

- Commit bf587caae305ae3b4393077fb22c98478ee55755 ("riscv: mm: synchronize MMU after pte change")

That would be my favorite.

Quoted 2 lines

- Commit 6dd91e0eacff0a5c822ca37565d6b5740c4d2a80 ("RISC-V: defconfig:
Enable NO_HZ_IDLE and HIGH_RES_TIMERS")

I had these enabled forever already.

Quoted 2 lines

- Commit 671f9a3e2e24cdeb2d2856abee7422f093e23e29 ("RISC-V: Setup initial page tables in two stages")

I don't think a one-time initial setup can have such a subtle effect.

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#6 lore raw
Quoted 11 lines · Andreas Schwab
Quoted 2 lines · Paul Walmsley

- Commit bf587caae305ae3b4393077fb22c98478ee55755 ("riscv: mm: synchronize MMU after pte change")

That would be my favorite.

Quoted 2 lines

- Commit 6dd91e0eacff0a5c822ca37565d6b5740c4d2a80 ("RISC-V: defconfig:
Enable NO_HZ_IDLE and HIGH_RES_TIMERS")

Quoted 6 lines

I had these enabled forever already.

Quoted 2 lines

- Commit 671f9a3e2e24cdeb2d2856abee7422f093e23e29 ("RISC-V: Setup initial page tables in two stages")

I don't think a one-time initial setup can have such a subtle effect.

The initial page table setup patch is not present in 5.2. It was merged in 5.3.

Regards,
Anup

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#7 lore raw
In reply to Andreas Schwab
Quoted 6 lines · On Wed, 31 Jul 2019 00:39:10 PDT (-0700), [email protected] wrote:
Quoted 2 lines · Paul Walmsley

- Commit bf587caae305ae3b4393077fb22c98478ee55755 ("riscv: mm: synchronize MMU after pte change")

That would be my favorite.

If that patch causes memory corruption then something scary is going on. I haven't been following the thread closely enough to know how easy this is to reproduce, but do you mind trying a kernel with a reverted version of that commit?

This is also availiable on the "for-andreas" branch of git.kernel.org/palmer/linux.git

commit 07d45256aa8641057c141f1a661bb29dd99eb32e
gpg: Signature made Thu 01 Aug 2019 12:46:22 PM PDT
gpg: using RSA key 00CE76D1834960DFCE886DF8EF4CA1502CCBAB41
gpg: issuer "[email protected]"
gpg: Good signature from "Palmer Dabbelt <[email protected]>" [ultimate]
gpg: aka "Palmer Dabbelt <[email protected]>" [ultimate]
Author: Palmer Dabbelt <[email protected]>
Date: Thu Aug 1 12:45:12 2019 -0700

Revert "riscv: mm: synchronize MMU after pte change"

Andreas Schwab is seeing some random memory corruption with 5.2, and he thinks the reverted comit is the most likely candidate. The commit itself doesn't revert cleanly, but that's just because getting the comment right took two commits.

If this does fix the issue then we're in a bit of trouble, as this TLB flush should be pretty safe.

This reverts commit bf587caae305ae3b4393077fb22c98478ee55755.

   diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
   index f960c3f4ce47..28dccb072255 100644
   --- a/arch/riscv/mm/fault.c
   +++ b/arch/riscv/mm/fault.c
   @@ -16,7 +16,6 @@
    #include <asm/pgalloc.h>
    #include <asm/ptrace.h>

-#include <asm/tlbflush.h>

/*
* This routine handles page faults. It determines the address and the
@@ -267,14 +266,6 @@ asmlinkage void do_page_fault(struct pt_regs *regs)

                   if (!pte_present(*pte_k))
                           goto no_context;

- /*

   -                * The kernel assumes that TLBs don't cache invalid
   -                * entries, but in RISC-V, SFENCE.VMA specifies an
   -                * ordering constraint, not a cache flush; it is
   -                * necessary even after writing invalid entries.
   -                */
   -               local_flush_tlb_page(addr);
   -
                   return;
           }
    }
Quoted 4 lines
Quoted 2 lines

- Commit 6dd91e0eacff0a5c822ca37565d6b5740c4d2a80 ("RISC-V: defconfig:
Enable NO_HZ_IDLE and HIGH_RES_TIMERS")

I had these enabled forever already.

IIRC that was the argument for enabling them in defconfig :)

Quoted 4 lines
Quoted 2 lines

- Commit 671f9a3e2e24cdeb2d2856abee7422f093e23e29 ("RISC-V: Setup initial page tables in two stages")

I don't think a one-time initial setup can have such a subtle effect.

As per Anup, it's not in 5.2.

Quoted 2 lines

Andreas.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#8 lore raw
In reply to Paul Walmsley
Quoted 2 lines · Paul Walmsley

- Commit bf587caae305ae3b4393077fb22c98478ee55755 ("riscv: mm: synchronize MMU after pte change")

When I revert that commit, I'm getting soft lockups. Doesn't that point to some deeper issue with TLB flushes?

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#9 lore raw
Quoted 13 lines · Andreas Schwab
Quoted 2 lines · Paul Walmsley

- Commit bf587caae305ae3b4393077fb22c98478ee55755 ("riscv: mm: synchronize MMU after pte change")

When I revert that commit, I'm getting soft lockups. Doesn't that point to some deeper issue with TLB flushes?

Andreas.

What are you using for filesystem/storage? Is it the SDcard, network, or something else?

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#10 lore raw
Quoted 1 line · Troy Benjegerdes

What are you using for filesystem/storage?

NFS.

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#11 lore raw
In reply to Paul Walmsley
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Andreas Schwab
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDR

OOT/glibc-2.29-0.riscv64 $locale

Quoted 13 lines
Quoted 4 lines

make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134

While building gcc:

Quoted 8 lines

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here

+===========================GNAT BUG DETECTED==============================+
| 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) |
| Storage_Error stack overflow or erroneous memory access                  |
| Error detected at output.ads:39:8                                        |

realloc(): invalid pointer

Quoted 2 lines

I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.

I think there is some fundamental problem with SBI_REMOTE_SFENCE_VMA or the kernel interface to it.

For exmaple, flush_tlb_page is defined as:

#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)

But the third argument of flush_tlb_range is supposed to be the end address, so this should actually be:

#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)

Alas, that doesn't fix the crashes.

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#12 lore raw
Quoted 12 lines · On Thu, 01 Aug 2019 11:32:33 PDT (-0700), [email protected] wrote:
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Andreas Schwab
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILD

ROOT/glibc-2.29-0.riscv64

Quoted 22 lines
Quoted 14 lines
Quoted 5 lines

make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134

While building gcc:

Quoted 8 lines

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here

+===========================GNAT BUG DETECTED==============================+
| 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) |
| Storage_Error stack overflow or erroneous memory access                  |
| Error detected at output.ads:39:8                                        |

realloc(): invalid pointer

Quoted 2 lines

I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.

I think there is some fundamental problem with SBI_REMOTE_SFENCE_VMA or the kernel interface to it.

For exmaple, flush_tlb_page is defined as:

Quoted 8 lines

#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)

But the third argument of flush_tlb_range is supposed to be the end address, so this should actually be:

#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)

Alas, that doesn't fix the crashes.

This line of reasoning smells like it'd find the issue: BBL just flushes the entire TLB every time, but IIRC OpenSBI respects the ranges. It looks like

    Fixes: 90cb4917b584 ("lib: Implement sfence.vma correctly.")

is what introduced the new behavior in OpenSBI, which may have triggered a lot of latent bugs in Linux. If you have an easy way to compile OpenSBI, does something like

    $ git diff | cat
    diff --git a/lib/sbi/sbi_tlb.c b/lib/sbi/sbi_tlb.c
    index cffda52d66ab..007266b1f970 100644
    --- a/lib/sbi/sbi_tlb.c
    +++ b/lib/sbi/sbi_tlb.c
    @@ -133,50 +133,12 @@ static void sbi_tlb_flush_all(void)

static void sbi_tlb_fifo_sfence_vma(struct sbi_tlb_info *tinfo) {
- unsigned long start = tinfo->start;
- unsigned long size = tinfo->size;
- unsigned long i;
-
- if ((start == 0 && size == 0) || (size == SBI_TLB_FLUSH_ALL)) {
- sbi_tlb_flush_all();
- return;
- }
-
- for (i = 0; i < size; i += PAGE_SIZE) {
- __asm__ __volatile__("sfence.vma %0"
- :
- : "r"(start + i)
- : "memory");
- }
+ sbi_tlb_flush_all();
}

static void sbi_tlb_fifo_sfence_vma_asid(struct sbi_tlb_info *tinfo) {
- unsigned long start = tinfo->start;
- unsigned long size = tinfo->size;
- unsigned long asid = tinfo->asid;
- unsigned long i;
-
- if (start == 0 && size == 0) {
- sbi_tlb_flush_all();
- return;
- }
-
- /* Flush entire MM context for a given ASID */
- if (size == SBI_TLB_FLUSH_ALL) {
- __asm__ __volatile__("sfence.vma x0, %0"
- :
- : "r"(asid)
- : "memory");
- return;
- }
-
- for (i = 0; i < size; i += PAGE_SIZE) {
- __asm__ __volatile__("sfence.vma %0, %1"
- :
- : "r"(start + i), "r"(asid)
- : "memory");
- }
+ sbi_tlb_flush_all();
}

void sbi_tlb_fifo_process(struct sbi_scratch *scratch, u32 event)

cause the issue to go away? If so, then I'd bet we need to scour Linux for broken TLB flushing, as given the one you found is pretty obvious I'd bet there's a lot more...

Quoted 2 lines

Andreas.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#13 lore raw
In reply to Andreas Schwab
Quoted 13 lines · Andreas Schwab
Quoted 10 lines · Paul Walmsley
Quoted 8 lines · Andreas Schwab
Quoted 6 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc. Perhaps missing TLB flushes?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped) I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUIL

DROOT/glibc-2.29-0.riscv64 $locale

Quoted 19 lines
Quoted 13 lines
Quoted 4 lines

make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134

While building gcc:

Quoted 8 lines

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here

+===========================GNAT BUG DETECTED==============================+
| 10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64-suse-linux) |
| Storage_Error stack overflow or erroneous memory access                  |
| Error detected at output.ads:39:8                                        |

realloc(): invalid pointer

Quoted 2 lines

I personally haven't seen these issues; but then again, I haven't done any glibc or gcc builds on v5.2. Will take a closer look.

I think there is some fundamental problem with SBI_REMOTE_SFENCE_VMA or the kernel interface to it.

Quoted 8 lines

For exmaple, flush_tlb_page is defined as:

#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)

But the third argument of flush_tlb_range is supposed to be the end address, so this should actually be:

#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)

Instead of this can you try -1UL as the size:
#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, -1UL)

If above works for you then there is some issue with range of virtual memory we flush.

Regards,
Anup

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#14 lore raw
Quoted 2 lines · Anup Patel

Instead of this can you try -1UL as the size:
#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, -1UL)

That doesn't help either.

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#15 lore raw

But this does help:

arch/riscv/include/asm/tlbflush.h +1 -1
@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,
 
 	cpumask_clear(&hmask);
 	riscv_cpuid_to_hartid_mask(cmask, &hmask);
-	sbi_remote_sfence_vma(hmask.bits, start, size);
+	sbi_remote_sfence_vma(hmask.bits, 0, -1);
 }
 
 #define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#16 lore raw

This helps too:

arch/riscv/include/asm/tlbflush.h +2 -1
@@ -50,10 +50,11 @@ static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,
 	cpumask_clear(&hmask);
 	riscv_cpuid_to_hartid_mask(cmask, &hmask);
 	sbi_remote_sfence_vma(hmask.bits, start, size);
+	local_flush_tlb_all();
 }
 
 #define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)
-#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)
+#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, (addr) + PAGE_SIZE)
 #define flush_tlb_range(vma, start, end) \
 	remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start))
 #define flush_tlb_mm(mm) \

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#17 lore raw
In reply to Andreas Schwab
Quoted 13 lines · Andreas Schwab

But this does help:

arch/riscv/include/asm/tlbflush.h +1 -1
@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask
 *cmask, unsigned long start,
 
 	cpumask_clear(&hmask);
 	riscv_cpuid_to_hartid_mask(cmask, &hmask);
-	sbi_remote_sfence_vma(hmask.bits, start, size);
+	sbi_remote_sfence_vma(hmask.bits, 0, -1);
 }
 
Quoted 2 lines

#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)

I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.

(quoted diff) +0 -0

b/arch/riscv/include/asm/tlbflush.h
index 687dd19735a7..29b2bd7c9923 100644

arch/riscv/include/asm/tlbflush.h +0 -0
@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(struct cpumask

*cmask, unsigned long start,
#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1) #define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0) #define flush_tlb_range(vma, start, end) \
- remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) -
(start))
+ remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)

 #define flush_tlb_mm(mm) \
        remote_sfence_vma(mm_cpumask(mm), 0, -1)

Can you please verify at your end?

While your fix flushes the entire tlb for every type of remote tlb flush, this fix proves that the issue is with flush_tlb_range call only.

I am looking at the OpenSBI/Kernel implementation to figure out if it is an OpenSBI issue or something changed in kernel recently to trigger this.

Additionally, do you know if a particular or group of locale install is causing this issue?

It takes more than hour to finish the full install-locales on unleashe board which makes it bit difficult to try out possible fixes.

Quoted 2 lines

Andreas.

Signature · 6 lines
Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#18 lore raw
Quoted 51 lines · Atish Patra
Quoted 13 lines · Andreas Schwab

But this does help:

arch/riscv/include/asm/tlbflush.h +1 -1
@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask
 *cmask, unsigned long start,
 
 	cpumask_clear(&hmask);
 	riscv_cpuid_to_hartid_mask(cmask, &hmask);
-	sbi_remote_sfence_vma(hmask.bits, start, size);
+	sbi_remote_sfence_vma(hmask.bits, 0, -1);
 }
 
Quoted 2 lines

#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)

I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.

arch/riscv/include/asm/tlbflush.h +1 -1
index 687dd19735a7..29b2bd7c9923 100644
@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(struct cpumask
 *cmask, unsigned long start,
 #define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)
 #define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)
 #define flush_tlb_range(vma, start, end) \
-       remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) -
 (start))
+       remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)
 #define flush_tlb_mm(mm) \
       remote_sfence_vma(mm_cpumask(mm), 0, -1)

Can you please verify at your end?

While your fix flushes the entire tlb for every type of remote tlb flush, this fix proves that the issue is with flush_tlb_range call only.

I am looking at the OpenSBI/Kernel implementation to figure out if it is an OpenSBI issue or something changed in kernel recently to trigger this.

Quoted 6 lines

Additionally, do you know if a particular or group of locale install is causing this issue?

It takes more than hour to finish the full install-locales on unleashe board which makes it bit difficult to try out possible fixes.

Did you reproduce with SDcard, or NFS?

Quoted 11 lines
Quoted 3 lines

Andreas.

Signature · 6 lines
Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#19 lore raw
Quoted 58 lines · Troy Benjegerdes
Quoted 58 lines

On Aug 5, 2019, at 5:34 PM, Atish Patra <[email protected]> wrote:

Quoted 14 lines · Andreas Schwab

But this does help:

arch/riscv/include/asm/tlbflush.h +0 -0
@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct

cpumask
*cmask, unsigned long start,

	cpumask_clear(&hmask);
	riscv_cpuid_to_hartid_mask(cmask, &hmask);
arch/riscv/include/asm/tlbflush.h +1 -1
-	sbi_remote_sfence_vma(hmask.bits, start, size);
+	sbi_remote_sfence_vma(hmask.bits, 0, -1);

}

Quoted 2 lines

#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)

I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.

arch/riscv/include/asm/tlbflush.h +0 -0
index 687dd19735a7..29b2bd7c9923 100644
@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(struct

cpumask
*cmask, unsigned long start,
#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1) #define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0) #define flush_tlb_range(vma, start, end) \
- remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) -
(start))
+ remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)

#define flush_tlb_mm(mm) \
       remote_sfence_vma(mm_cpumask(mm), 0, -1)

Can you please verify at your end?

While your fix flushes the entire tlb for every type of remote tlb flush, this fix proves that the issue is with flush_tlb_range call only.

I am looking at the OpenSBI/Kernel implementation to figure out if it
is an OpenSBI issue or something changed in kernel recently to trigger
this.

Additionally, do you know if a particular or group of locale install
is causing this issue?

Quoted 7 lines
Quoted 4 lines

It takes more than hour to finish the full install-locales on unleashe
board which makes it bit difficult to try out possible fixes.

Did you reproduce with SDcard, or NFS?

I am running it on a nvme SSD attached to Microsemi expansion board.

Kernel version: 5.3-rc2
OpenSBI/U-Boot: Latest master

Regards,
Atish

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

Andreas.

Signature · 6 lines
Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#20 lore raw
In reply to Atish Patra
Quoted 2 lines · Atish Patra

It takes more than hour to finish the full install-locales on unleashe board which makes it bit difficult to try out possible fixes.

When it fails it usually fails pretty fast. Did you run it in parallel?

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#21 lore raw
In reply to Atish Patra
Quoted 36 lines · Atish Patra
Quoted 13 lines · Andreas Schwab

But this does help:

arch/riscv/include/asm/tlbflush.h +1 -1
@@ -49,7 +49,7 @@ static inline void remote_sfence_vma(struct cpumask
 *cmask, unsigned long start,
 
 	cpumask_clear(&hmask);
 	riscv_cpuid_to_hartid_mask(cmask, &hmask);
-	sbi_remote_sfence_vma(hmask.bits, start, size);
+	sbi_remote_sfence_vma(hmask.bits, 0, -1);
 }
 
Quoted 2 lines

#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)

I am also able to reprduce the issue while doing a install-locales. Here is the temporary fix that seems to solve the issue.

arch/riscv/include/asm/tlbflush.h +1 -1
index 687dd19735a7..29b2bd7c9923 100644
@@ -55,7 +55,7 @@ static inline void remote_sfence_vma(struct cpumask
 *cmask, unsigned long start,
 #define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)
 #define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)
 #define flush_tlb_range(vma, start, end) \
-       remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) -
 (start))
+       remote_sfence_vma(mm_cpumask((vma)->vm_mm), 0, -1)
 #define flush_tlb_mm(mm) \
        remote_sfence_vma(mm_cpumask(mm), 0, -1)
Quoted 1 line

Can you please verify at your end?

This is equivalent to my patch since all other uses already pass 0,-1.

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#23 lore raw
Quoted 1 line · Paul Walmsley

I was able to build glibc, and run most of the test suite, on v5.3-rc2

Did you run install-locales?

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#24 lore raw
Quoted 5 lines · Andreas Schwab
Quoted 1 line · Paul Walmsley

I was able to build glibc, and run most of the test suite, on v5.3-rc2

Did you run install-locales?

I just ran "make -j4", "make -j4 check", "make -j4 xcheck". This is with rootfs on microSD, rather than on NFS.

Do you still see the failures if you only run the above commands, or does the failure only appear with install-locales?

- Paul

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#25 lore raw
Quoted 2 lines · Paul Walmsley

Do you still see the failures if you only run the above commands, or does the failure only appear with install-locales?

Only during install-locales.

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#26 lore raw
In reply to Andreas Schwab
Quoted 52 lines · Andreas Schwab
Quoted 8 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc.
Perhaps missing TLB flushes?

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped)
I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc- 2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error 134

While building gcc:

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here +===========================GNAT BUG DETECTED==============================+

Quoted 6 lines

10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64- suse-linux) |
Storage_Error stack overflow or erroneous memory
access |
Error detected at
output.ads:39:8 |

Quoted 8 lines

realloc(): invalid pointer

raised PROGRAM_ERROR : unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140:
ada/exp_ch3.o] Error 1

Andreas.

Can you give it a try with following patch in OpenSBI & Kernel ?

Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.html

OpenSBI:
http://lists.infradead.org/pipermail/opensbi/2019-August/000386.html

In my testing, I no longer the stress-ng error or glibc local install issue if I use following command.

sudo make -j8 localedata/install-locale-files
DESTDIR=/home/atish/glibc/build/install

I still see segmentation fault if I use a archieve locale install command.

sudo make -j8 localedata/install-locales
DESTDIR=/home/atish/glibc/build/install

But the error dump doesn't contain remap() error. Just a segmentation fault which may be due to userspace or just different version of old tlbflush problem.

Regards,
Atish

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#27 lore raw
Quoted 78 lines · Atish Patra
Quoted 53 lines · Andreas Schwab
Quoted 8 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc.
Perhaps missing TLB flushes?

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped)
I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc- 2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error
134

While building gcc:

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here +===========================GNAT BUG DETECTED==============================+

Quoted 6 lines

10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64- suse-linux) |
Storage_Error stack overflow or erroneous memory
access |
Error detected at
output.ads:39:8 |

Quoted 8 lines

realloc(): invalid pointer

raised PROGRAM_ERROR : unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140:
ada/exp_ch3.o] Error 1

Andreas.

Can you give it a try with following patch in OpenSBI & Kernel ?

Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.html

OpenSBI:
http://lists.infradead.org/pipermail/opensbi/2019-August/000386.html

In my testing, I no longer the stress-ng error or glibc local install issue if I use following command.

sudo make -j8 localedata/install-locale-files
DESTDIR=/home/atish/glibc/build/install

Quoted 6 lines

I still see segmentation fault if I use a archieve locale install command.

sudo make -j8 localedata/install-locales
DESTDIR=/home/atish/glibc/build/install

I am also able to run above archive locale install command successfully multiple times after removing the corrupted locale-archive files present in install path.

Let me know if it works for you as well.

I am now running stress-ng & parallel glibc locale install together to fully stress the system.

Regards,
Atish

Quoted 8 lines

But the error dump doesn't contain remap() error. Just a segmentation fault which may be due to userspace or just different version of old tlbflush problem.

Regards,
Atish

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#28 lore raw
Quoted 94 lines · Atish Patra
Quoted 78 lines · Atish Patra
Quoted 53 lines · Andreas Schwab
Quoted 8 lines · David Abdurachmanov
Quoted 4 lines · Andreas Schwab

Since switching to 5.2 kernels I'm seeing random crashes and misbehaviors on the HiFive, for example while building gcc or glibc.
Perhaps missing TLB flushes?

Do you have some examples of crashes?

While building glibc:

an_ES.UTF-8...realloc(): invalid pointer
/bin/sh: line 1: 7841 Aborted (core dumped)
I18NPATH=. GCONV_PATH=/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/iconvdata LC_ALL=C /home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf/ld-linux-riscv64-lp64d.so.1 --library-path /home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/math:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/elf:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/dlfcn:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nss:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/nis:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/rt:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/resolv:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/mathvec:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc- base/support:/home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/nptl /home/abuild/rpmbuild/BUILD/glibc-2.29/cc-base/locale/localedef $flags --alias-file=../intl/locale.alias -i locales/$input -f charmaps/$charset --prefix=/home/abuild/rpmbuild/BUILDROOT/glibc- 2.29-0.riscv64 $locale
make[2]: *** [Makefile:422: install-archive-an_ES.UTF-8/UTF-8] Error
134

While building gcc:

../../gcc/ada/exp_aggr.adb: In function 'Exp_Aggr.Expand_N_Aggregate':
../../gcc/ada/exp_aggr.adb:5311:21: warning: 'Csiz' may be used uninitialized in this function [-Wmaybe-uninitialized] ../../gcc/ada/exp_aggr.adb:5220:10: note: 'Csiz' was declared here +===========================GNAT BUG DETECTED==============================+

Quoted 6 lines

10.0.0 20190727 (experimental) [trunk revision 273844] (riscv64- suse-linux) |
Storage_Error stack overflow or erroneous memory
access |
Error detected at
output.ads:39:8 |

Quoted 8 lines

realloc(): invalid pointer

raised PROGRAM_ERROR : unhandled signal
make[3]: *** [../../gcc/ada/gcc-interface/Make-lang.in:140:
ada/exp_ch3.o] Error 1

Andreas.

Can you give it a try with following patch in OpenSBI & Kernel ?

Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.html

OpenSBI:
http://lists.infradead.org/pipermail/opensbi/2019-August/000386.html

In my testing, I no longer the stress-ng error or glibc local install issue if I use following command.

sudo make -j8 localedata/install-locale-files
DESTDIR=/home/atish/glibc/build/install

Quoted 6 lines

I still see segmentation fault if I use a archieve locale install command.

sudo make -j8 localedata/install-locales
DESTDIR=/home/atish/glibc/build/install

I am also able to run above archive locale install command successfully multiple times after removing the corrupted locale-archive files present in install path.

Quoted 17 lines

Let me know if it works for you as well.

I am now running stress-ng & parallel glibc locale install together to fully stress the system.

Regards,
Atish

Quoted 8 lines

But the error dump doesn't contain remap() error. Just a segmentation fault which may be due to userspace or just different version of old tlbflush problem.

Regards,
Atish

Is this with stock linux-5.2.8 release, with no additional patches, or is there something we need to look at backporting?

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#29 lore raw
In reply to Atish Patra
Quoted 2 lines · Atish Patra

Linux kernel:
http://lists.infradead.org/pipermail/linux-riscv/2019-August/005889.html

I've been using that patch, without any changes to openSBI, to run bootstrap/regtest on gcc and to build glibc without issues.

Andreas.

Signature · 8 lines
Andreas Schwab, SUSE Labs, [email protected]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv