5.6.8-rc1 review

From: Michael Ellerman <[email protected]>

commit 61da50b76b62fd815aa82d853bf82bf4f69568f5 upstream.

Currently you can enable PPC_KUAP_DEBUG when PPC_KUAP is disabled, even though the former has not effect without the latter.

Fix it so that PPC_KUAP_DEBUG can only be enabled when PPC_KUAP is enabled, not when the platform could support KUAP (PPC_HAVE_KUAP).

---
 arch/powerpc/platforms/Kconfig.cputype |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

@@ -397,7 +397,7 @@ config PPC_KUAP
 
 config PPC_KUAP_DEBUG
 	bool "Extra debugging for Kernel Userspace Access Protection"
-	depends on PPC_HAVE_KUAP && (PPC_RADIX_MMU || PPC_32)
+	depends on PPC_KUAP && (PPC_RADIX_MMU || PPC_32)
 	help
 	  Add extra debugging for Kernel Userspace Access Protection (KUAP)
 	  If you're unsure, say N.

From: Christophe Leroy <[email protected]>

commit feb8e960d780e170e992a70491eec9dd68f4dbf2 upstream.

CONFIG_PPC_KUAP_DEBUG is not selectable because it depends on PPC_32 which doesn't exists.

Fixing it leads to a deadlock due to a vital register getting clobbered in _switch().

Change dependency to PPC32 and use r0 instead of r4 in _switch()

---
 arch/powerpc/kernel/entry_32.S         |    2 +-
 arch/powerpc/platforms/Kconfig.cputype |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

@@ -710,7 +710,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_SPE)
 	stw	r10,_CCR(r1)
 	stw	r1,KSP(r3)	/* Set old stack pointer */
 
-	kuap_check r2, r4
+	kuap_check r2, r0
 #ifdef CONFIG_SMP
 	/* We need a sync somewhere here to make sure that if the
 	 * previous task gets rescheduled on another CPU, it sees all

@@ -397,7 +397,7 @@ config PPC_KUAP
 
 config PPC_KUAP_DEBUG
 	bool "Extra debugging for Kernel Userspace Access Protection"
-	depends on PPC_KUAP && (PPC_RADIX_MMU || PPC_32)
+	depends on PPC_KUAP && (PPC_RADIX_MMU || PPC32)
 	help
 	  Add extra debugging for Kernel Userspace Access Protection (KUAP)
 	  If you're unsure, say N.

From: Christian Borntraeger <[email protected]>

commit 316ec154810960052d4586b634156c54d0778f74 upstream.

A page table upgrade in a kernel section that uses secondary address mode will mess up the kernel instructions as follows:

Consider the following scenario: two threads are sharing memory. On CPU1 thread 1 does e.g. strnlen_user(). That gets to

        old_fs = enable_sacf_uaccess();
        len = strnlen_user_srst(src, size);
and
                "   la    %2,0(%1)\n"
                "   la    %3,0(%0,%1)\n"
                "   slgr  %0,%0\n"
                "   sacf  256\n"
                "0: srst  %3,%2\n"

in strnlen_user_srst(). At that point we are in secondary space mode, control register 1 points to kernel page table and instruction fetching happens via c1, rather than usual c13. Interrupts are not disabled, for obvious reasons.

On CPU2 thread 2 does MAP_FIXED mmap(), forcing the upgrade of page table from 3-level to e.g. 4-level one. We'd allocated new top-level table, set it up and now we hit this:

                notify = 1;
                spin_unlock_bh(&mm->page_table_lock);
        }
        if (notify)
                on_each_cpu(__crst_table_upgrade, mm, 0);

OK, we need to actually change over to use of new page table and we need that to happen in all threads that are currently running. Which happens to include the thread 1. IPI is delivered and we have static void __crst_table_upgrade(void *arg)

{
        struct mm_struct *mm = arg;

        if (current->active_mm == mm)
                set_user_asce(mm);
        __tlb_flush_local();
}
run on CPU1.  That does
static inline void set_user_asce(struct mm_struct *mm)
{
        S390_lowcore.user_asce = mm->context.asce;
OK, user page table address updated...
        __ctl_load(S390_lowcore.user_asce, 1, 1);
... and control register 1 set to it.
        clear_cpu_flag(CIF_ASCE_PRIMARY);
}

IPI is run in home space mode, so it's fine - insns are fetched using c13, which always points to kernel page table. But as soon as we return from the interrupt, previous PSW is restored, putting CPU1 back into secondary space mode, at which point we no longer get the kernel instructions from the kernel mapping.

The fix is to only fixup the control registers that are currently in use for user processes during the page table update. We must also disable interrupts in enable_sacf_uaccess to synchronize the cr and thread.mm_segment updates against the on_each-cpu.

---
 arch/s390/lib/uaccess.c |    4 ++++
 arch/s390/mm/pgalloc.c  |   16 ++++++++++++++--
 2 files changed, 18 insertions(+), 2 deletions(-)

@@ -64,10 +64,13 @@ mm_segment_t enable_sacf_uaccess(void)
 {
 	mm_segment_t old_fs;
 	unsigned long asce, cr;
+	unsigned long flags;
 
 	old_fs = current->thread.mm_segment;
 	if (old_fs & 1)
 		return old_fs;
+	/* protect against a concurrent page table upgrade */
+	local_irq_save(flags);
 	current->thread.mm_segment |= 1;
 	asce = S390_lowcore.kernel_asce;
 	if (likely(old_fs == USER_DS)) {

@@ -83,6 +86,7 @@ mm_segment_t enable_sacf_uaccess(void)
 		__ctl_load(asce, 7, 7);
 		set_cpu_flag(CIF_ASCE_SECONDARY);
 	}
+	local_irq_restore(flags);
 	return old_fs;
 }
 EXPORT_SYMBOL(enable_sacf_uaccess);

@@ -70,8 +70,20 @@ static void __crst_table_upgrade(void *a
 {
 	struct mm_struct *mm = arg;
 
-	if (current->active_mm == mm)
-		set_user_asce(mm);
+	/* we must change all active ASCEs to avoid the creation of new TLBs */
+	if (current->active_mm == mm) {
+		S390_lowcore.user_asce = mm->context.asce;
+		if (current->thread.mm_segment == USER_DS) {
+			__ctl_load(S390_lowcore.user_asce, 1, 1);
+			/* Mark user-ASCE present in CR1 */
+			clear_cpu_flag(CIF_ASCE_PRIMARY);
+		}
+		if (current->thread.mm_segment == USER_DS_SACF) {
+			__ctl_load(S390_lowcore.user_asce, 7, 7);
+			/* enable_sacf_uaccess does all or nothing */
+			WARN_ON(!test_cpu_flag(CIF_ASCE_SECONDARY));
+		}
+	}
 	__tlb_flush_local();
 }
 

Compiled and booted on my test system. No dmesg regressions.

thanks,
-- Shuah

Great, thanks for testing these and letting me know.

greg k-h

Results from Linaro’s test farm.
No regressions on arm64, arm, x86_64, and i386.

NOTE:
This kernel panic seems to be platform specific.
However, I am sharing a few kernel panic logs here.
While running LTP cve[1] and libhugetlbfs[2] test suite on nxp ls2088 device the kernel panic noticed with different kernel dump
and unfortunately it is not easily reproducible.
At this point it is unclear whether this problem
started happening from this stable rc review or not.
Because a different type of kernel panic noticed on Linus 's mainline tree (5.7.0-rc2) version kernel while running LTP containers tests.

[   49.677646] SError Interrupt on CPU5, code 0xbf000002 -- SError
[   49.677648] CPU: 5 PID: 0 Comm: swapper/5 Not tainted
5.7.0-rc2-00243-g5ef58e290782 #1
[   49.677649] Hardware name: Freescale Layerscape 2088A RDB Board (DT)
[   49.677650] pstate: 40000085 (nZcv daIf -PAN -UAO)
[   49.677651] pc : slabinfo_write+0x2a0/0x4fc
[   49.677652] lr : get_partial_node.isra.0.part.0+0x4c/0x338

steps to reproduce: (Not always reproducible)

# cd /opt/ltp
# ./runltp -f cve

# cd  /opt/libhugetlbfs/tests/
# ./run_tests.py -b 64

[ 1469.013456] Internal error: synchronous external abort: 96000210
[#1] PREEMPT SMP
[ 1469.020934] Modules linked in: algif_hash rfkill caam_jr
caamhash_desc caamalg_desc rng_core caam error crct10dif_ce lm90
ina2xx qoriq_thermal fuse
[ 1469.034154] CPU: 5 PID: 0 Comm: swapper/5 Not tainted
5.6.8-rc1-00164-g86cfba65ced0 #1
[ 1469.042063] Hardware name: Freescale Layerscape 2088A RDB Board (DT)
[ 1469.048409] pstate: 80000005 (Nzcv daif -PAN -UAO)
[ 1469.053196] pc : cgroup_free+0xc/0x60
[ 1469.056851] lr : __put_task_struct+0x34/0x1b0
[ 1469.061198] sp : ffff80001002be00
[ 1469.064504] x29: ffff80001002be00 x28: 0000000000000000
[ 1469.069809] x27: ffffcdb8d17f3140 x26: 0000000000000000
[ 1469.075113] x25: ffff80001002bec0 x24: ffff0082fe3b7b50
[ 1469.080417] x23: ffff0082ee5fd400 x22: 000000000000000a
[ 1469.085721] x21: ffff0082ee5fd400 x20: ffffcdb8d17f32c0
[ 1469.091024] x19: ffff0082edda1c00 x18: 0000000000000000
[ 1469.096328] x17: 0000000000000000 x16: 0000000000000000
[ 1469.101631] x15: 0000000000000000 x14: 00000000000001ec
[ 1469.106934] x13: 0000000000000000 x12: 0000000000000000
[ 1469.112238] x11: 0000000000000002 x10: 0000000000000000
[ 1469.117541] x9 : 0000000000000000 x8 : ffff0082c8340900
[ 1469.122845] x7 : 0000000044042000 x6 : 0000000000210d00
[ 1469.128150] x5 : ffff0082edda1c28 x4 : 0000000000000000
[ 1469.133454] x3 : 0000000000000000 x2 : 0000000000000001
[ 1469.138757] x1 : 0000000000000001 x0 : ffff0082edda1c00
[ 1469.144061] Call trace:
[ 1469.146498]  cgroup_free+0xc/0x60
[ 1469.149804]  __put_task_struct+0x34/0x1b0
[ 1469.153806]  delayed_put_task_struct+0x44/0x90
[ 1469.158242]  rcu_core+0x2ac/0x488
[ 1469.161549]  rcu_core_si+0xc/0x18
[ 1469.164856]  efi_header_end+0x120/0x23c
[ 1469.168685]  irq_exit+0xb8/0xd8
[ 1469.171818]  __handle_domain_irq+0x64/0xb8
[ 1469.175907]  gic_handle_irq+0x5c/0x148
[ 1469.179647]  el1_irq+0xb8/0x180
[ 1469.182780]  cpuidle_enter_state+0x88/0x2f0
[ 1469.186954]  cpuidle_enter+0x34/0x48
[ 1469.190521]  call_cpuidle+0x18/0x38
[ 1469.194000]  do_idle+0x1e8/0x280
[ 1469.197219]  cpu_startup_entry+0x24/0x40
[ 1469.201134]  secondary_start_kernel+0x154/0x190
[ 1469.205659] Code: 97fd4ce5 a9be7bfd 910003fd a90153f3 (f943ec14)
[ 1469.211752] ---[ end trace 69fc7de7c9d4a349 ]---
[ 1469.216361] Kernel panic - not syncing: Fatal exception in interrupt
[ 1469.222708] SMP: stopping secondary CPUs
[ 1469.226627] Kernel Offset: 0x4db8bfe00000 from 0xffff800010000000
[ 1469.232711] PHYS_OFFSET: 0xfffffc1f00000000
[ 1469.236886] CPU features: 0x10002,21806008
[ 1469.240972] Memory Limit: none
[ 1469.244021] ---[ end Kernel panic - not syncing: Fatal exception in

interrupt ]---

libhugetlbfs test run caused kernel panic,

[   43.809972] Internal error: synchronous external abort: 96000210
[#1] PREEMPT SMP
[   43.817451] Modules linked in: rfkill caam_jr caamhash_desc
caamalg_desc rng_core caam error crct10dif_ce lm90 ina2xx
qoriq_thermal fuse
[   43.829716] CPU: 2 PID: 1097 Comm: mmap-gettest Not tainted
5.6.8-rc1-00168-g853ae83af7cc #1
[   43.838146] Hardware name: Freescale Layerscape 2088A RDB Board (DT)
[   43.844493] pstate: 60000085 (nZCv daIf -PAN -UAO)
[   43.849281] pc : ktime_get_update_offsets_now+0x78/0xf8
[   43.854498] lr : ktime_get_update_offsets_now+0x58/0xf8
[   43.859714] sp : ffff800010013e90
[   43.863020] x29: ffff800010013e90 x28: ffff0082fe36ea38
[   43.868323] x27: 0000000a1faeb368 x26: 0000000000002e66
[   43.873627] x25: ffff0082fe36e98c x24: ffffac25d1c00f00
[   43.878931] x23: 0000000000000001 x22: ffffffffc4653600
[   43.884234] x21: ffff0082fe36ea78 x20: ffff0082fe36eab8
[   43.889538] x19: ffff0082fe36ea38 x18: 0000000000000000
[   43.894842] x17: 0000000000000000 x16: 0000000000000000
[   43.900146] x15: 0000000000000000 x14: 0000000000000000
[   43.905450] x13: 0000000000000000 x12: 0000000000000000
[   43.910754] x11: 0000000000000000 x10: 0000000000000040
[   43.916058] x9 : ffffac25d19f2250 x8 : ffffac25d19f2248
[   43.921361] x7 : ffff0082ee000288 x6 : 0000000000000000
[   43.926664] x5 : ffff545d2cd84000 x4 : 000b5f2e9a47be6c
[   43.931969] x3 : 0000000027fffac2 x2 : 0000000000000003
[   43.937272] x1 : 000af6b237f94080 x0 : 0000000000000018
[   43.942576] Call trace:
[   43.945015]  ktime_get_update_offsets_now+0x78/0xf8
[   43.949885]  hrtimer_interrupt+0x7c/0x240
[   43.953887]  arch_timer_handler_phys+0x30/0x40
[   43.958324]  handle_percpu_devid_irq+0x80/0x140
[   43.962846]  generic_handle_irq+0x24/0x38
[   43.966846]  __handle_domain_irq+0x60/0xb8
[   43.970935]  gic_handle_irq+0x5c/0x148
[   43.974009] SError Interrupt on CPU0, code 0xbf000002 -- SError
[   43.974011] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
5.6.8-rc1-00168-g853ae83af7cc #1
[   43.974012] Hardware name: Freescale Layerscape 2088A RDB Board (DT)
[   43.974013] pstate: 80000085 (Nzcv daIf -PAN -UAO)
[   43.974014] pc : _raw_spin_unlock_irqrestore+0x34/0x40
[   43.974015] lr : timekeeping_advance+0x374/0x550
[   43.974015] sp : ffff800010003e70
[   43.974016] x29: ffff800010003e70 x28: 0000000000000000
[   43.974018] x27: 003b9aca00000000 x26: ffff800010004000
[   43.974020] x25: 003b9aca00000000 x24: 0000000000011628
[   43.974021] x23: ffffac25d19e2fc0 x22: 003d090000000000
[   43.974023] x21: 0000000000000000 x20: ffffac25d1c00f00
[   43.974025] x19: ffffac25d1c01070 x18: 0000000000000000
[   43.974026] x17: 0000000000000000 x16: 0000000000000000
[   43.974028] x15: 0000000000000000 x14: 003d090000000000
[   43.974030] x13: 00003d08f8004540 x12: 0000000007ffbac0
[   43.974031] x11: 00003d08f8004540 x10: 00000000000186a0
[   43.974033] x9 : 0000000000000000 x8 : 0000000000000008
[   43.974034] x7 : 0000000000000000 x6 : ffffac25d1c01060
[   43.974036] x5 : 0000000a02ffee00 x4 : 0000000000000000
[   43.974038] x3 : 0000000000000000 x2 : 0000000000000000
[   43.974040] x1 : ffffac25d19e2fc0 x0 : 0000000100000201
[   43.974042] Kernel panic - not syncing: Asynchronous SError Interrupt
[   44.098603]  el0_irq_naked+0x4c/0x54
[   44.102171] Code: b9406700 8a020084 b9400322 9b030484 (b940f701)
[   44.108258] ---[ end trace e237e948351682ae ]---
[   44.112873] SMP: stopping secondary CPUs
[   44.112874] Kernel Offset: 0x2c25c0000000 from 0xffff800010000000
[   44.112875] PHYS_OFFSET: 0xffffce7680000000
[   44.112875] CPU features: 0x10002,21806008
[   44.112876] Memory Limit: none

LTP test suite
[1] https://github.com/linux-test-project/ltp

libhugetlbfs test repo link
[2] https://github.com/libhugetlbfs/libhugetlbfs

full test run:
https://qa-reports.linaro.org/lkft/linux-stable-rc-5.6-oe/build/v5.6.7-164-g86cfba65ced0/testrun/14378/log

Build details and kernel config:
https://qa-reports.linaro.org/lkft/linux-stable-rc-5.6-oe/build/v5.6.7-164-g86cfba65ced0/testrun/14378/
https://qa-reports.linaro.org/lkft/linux-stable-rc-5.6-oe/build/v5.6.7-168-g853ae83af7cc/testrun/14736/log
https://qa-reports.linaro.org/lkft/linux-stable-rc-5.6-oe/build/v5.6.7-168-g853ae83af7cc/testrun/14736

kernel config:
https://builds.tuxbuild.com/nYYJB1EJpavHsEQt6aGP1w/kernel.config

kernel panic log on Linus 's mainline (5.7.0-rc2) kernel
https://qa-reports.linaro.org/lkft/linux-mainline-oe/build/v5.7-rc2-243-g5ef58e290782/testrun/14053/log

Summary
------------------------------------------------------------------------

kernel: 5.6.8-rc1
git repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
git branch: linux-5.6.y
git commit: 853ae83af7cc63bd4dc0a44370e4f0b3c9fa57f9
git describe: v5.6.7-168-g853ae83af7cc6
Test details: https://qa-reports.linaro.org/lkft/linux-stable-rc-5.6-oe/build/v5.6.7-168-g853ae83af7cc6

No regressions (compared to build v5.6.7)

No fixes (compared to build v5.6.7)

Ran 36070 total tests in the following environments and test suites.

Environments
--------------
- dragonboard-410c
- hi6220-hikey
- i386
- juno-r2
- juno-r2-compat
- juno-r2-kasan
- nxp-ls2088
- qemu_arm
- qemu_arm64
- qemu_i386
- qemu_x86_64
- x15
- x86
- x86-kasan

Test Suites
-----------
* build
* install-android-platform-tools-r2600
* kselftest
* kselftest/drivers
* kselftest/filesystems
* libgpiod
* linux-log-parser
* perf
* network-basic-tests
* kselftest/net
* kselftest/networking
* libhugetlbfs
* ltp-cap_bounds-tests
* ltp-commands-tests
* ltp-containers-tests
* ltp-cpuhotplug-tests
* ltp-cve-tests
* ltp-dio-tests
* ltp-fcntl-locktests-tests
* ltp-filecaps-tests
* ltp-fs-tests
* ltp-fs_bind-tests
* ltp-fs_perms_simple-tests
* ltp-fsx-tests
* ltp-hugetlb-tests
* ltp-io-tests
* ltp-ipc-tests
* ltp-math-tests
* ltp-mm-tests
* ltp-nptl-tests
* ltp-pty-tests
* ltp-sched-tests
* ltp-securebits-tests
* ltp-syscalls-tests
* spectre-meltdown-checker-test
* v4l2-compliance
* kselftest-vsyscall-mode-native
* kselftest-vsyscall-mode-native/drivers
* kselftest-vsyscall-mode-native/filesystems
* kselftest-vsyscall-mode-native/net
* kselftest-vsyscall-mode-native/networking
* kselftest-vsyscall-mode-none
* kselftest-vsyscall-mode-none/drivers
* kselftest-vsyscall-mode-none/filesystems
* kselftest-vsyscall-mode-none/net
* kselftest-vsyscall-mode-none/networking

Linaro LKFT
https://lkft.linaro.org

As per the initial investigation these kernel panic (s) noticed from a single machine out of five machines under test.

- Naresh

Great, thanks fro testing these and letting me know.

If you end up narrowing this down to an offending commit, that would be great.

thanks,

greg k-h

All tests are passing for Tegra ...

Test results for stable-v5.6:

    13 builds:	13 pass, 0 fail
    24 boots:	24 pass, 0 fail
    40 tests:	40 pass, 0 fail

Linux version: 5.6.8-rc1-g853ae83af7cc
Boards tested: tegra124-jetson-tk1, tegra186-p2771-0000,

                tegra194-p2972-0000, tegra20-ventana,
                tegra210-p2371-2180, tegra210-p3450-0000,
                tegra30-cardhu-a04

Cheers
Jon

nvpublic

Thanks for testing all of these and letting me know.

greg k-h

Build results:

	total: 155 pass: 155 fail: 0
Qemu test results:
	total: 428 pass: 428 fail: 0

Guenter

Thanks for testing all of them and letting me know.

greg k-h