selftests/pstore: add pstore test script

These scripts include test cases which check pstore behavior. This is useful to avoid regressions of pstore.

Changes since v1:
* Exit with 1 in pstore/Makefile when pstore_crash_test failed.
* Create helper functions to make the tests much more readable.
* Use /sys/module/pstore/parameters/backend to check pstore backend is registered.
* Show content of /sys/module/.../backend and /proc/cmdline for debug
* Give UUID to each execution of test script.
* Write unique test string with UUID into pmsg.
* Check only one test string pattern appears in pmsg after crash.
* Check UUID written to pmsg matches across crash.
* Don't touch panic_on_oops because it is not necessary for reboot on crash

(v1: http://www.kernelhub.org/?msg=831044&p=2)

I also confirmed that these scripts work fine with kdump reboot with kernel boot parameter 'crash_kexec_post_notifiers'.

---

Hiraku Toyooka (2):
selftests/pstore: add pstore test script for pre-reboot selftests/pstore: add pstore test scripts going with reboot

 tools/testing/selftests/Makefile                   |    4 +
 tools/testing/selftests/pstore/Makefile            |   15 ++++
 tools/testing/selftests/pstore/common_tests        |   83 ++++++++++++++++++++
 tools/testing/selftests/pstore/pstore_crash_test   |   30 +++++++
 .../selftests/pstore/pstore_post_reboot_tests      |   77 +++++++++++++++++++
 tools/testing/selftests/pstore/pstore_tests        |   30 +++++++
 6 files changed, 239 insertions(+)
 create mode 100644 tools/testing/selftests/pstore/Makefile
 create mode 100755 tools/testing/selftests/pstore/common_tests
 create mode 100755 tools/testing/selftests/pstore/pstore_crash_test
 create mode 100755 tools/testing/selftests/pstore/pstore_post_reboot_tests
 create mode 100755 tools/testing/selftests/pstore/pstore_tests

--
Hiraku Toyooka

The pstore_tests script includes test cases which check pstore's behavior before crash (and reboot).

The test cases are currently following.

- Check pstore backend is registered
- Check pstore console is registered
- Check /dev/pmsg0 exists
- Write unique string to /dev/pmsg0

The unique string written to /dev/pmsg includes UUID. The UUID is also left in 'uuid' file in order to enable us to check if the pmsg keeps the string correctly after reboot.

Example usage is following.

  # cd /path/to/selftests
  # make run_tests -C pstore (or just .pstore/pstore_tests)
  make: Entering directory '/path/to/selftests/pstore'
  === Pstore unit tests (pstore_tests) ===
  UUID=b49b02cf-b0c2-4309-be43-b08c3971e37f
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  Checking pstore console is registered ... ok
  Checking /dev/pmsg0 exists ... ok
  Writing unique string to /dev/pmsg0 ... ok
  selftests: pstore_tests [PASS]
  make: Leaving directory '/path/to/selftests/pstore'

We can also see test logs later.

  # cat pstore/logs/20151001-072718_b49b02cf-b0c2-4309-be43-b08c3971e37f/pstore_tests.log
  Thu Oct  1 07:27:18 UTC 2015
  === Pstore unit tests (pstore_tests) ===
  UUID=b49b02cf-b0c2-4309-be43-b08c3971e37f
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  Checking pstore console is registered ... ok
  Checking /dev/pmsg0 exists ... ok
  Writing unique string to /dev/pmsg0 ... ok

---
 tools/testing/selftests/Makefile            |    1 
 tools/testing/selftests/pstore/Makefile     |   12 ++++++
 tools/testing/selftests/pstore/common_tests |   55 +++++++++++++++++++++++++++
 tools/testing/selftests/pstore/pstore_tests |   30 +++++++++++++++
 4 files changed, 98 insertions(+)
 create mode 100644 tools/testing/selftests/pstore/Makefile
 create mode 100755 tools/testing/selftests/pstore/common_tests
 create mode 100755 tools/testing/selftests/pstore/pstore_tests

index cfe1213..1a8fb99 100644
@@ -13,6 +13,7 @@ TARGETS += mount
 TARGETS += mqueue
 TARGETS += net
 TARGETS += powerpc
+TARGETS += pstore
 TARGETS += ptrace
 TARGETS += seccomp
 TARGETS += size

new file mode 100644
index 0000000..48623f7
@@ -0,0 +1,12 @@
+# Makefile for pstore selftests.
+# Expects pstore backend is registered.
+
+all:
+
+TEST_PROGS := pstore_tests
+TEST_FILES := common_tests
+
+include ../lib.mk
+
+clean:
+	rm -rf logs/* *uuid

new file mode 100755
index 0000000..f25d2a3
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+# pstore_tests - Check pstore's behavior before crash/reboot
+#
+# Copyright (C) Hitachi Ltd., 2015
+#  Written by Hiraku Toyooka <[email protected]>
+#
+# Released under the terms of the GPL v2.
+
+. ./common_tests
+
+prlog -n "Checking pstore console is registered ... "
+dmesg | grep -q "console \[pstore"
+show_result $?
+
+prlog -n "Checking /dev/pmsg0 exists ... "
+test -e /dev/pmsg0
+show_result $?
+
+prlog -n "Writing unique string to /dev/pmsg0 ... "
+if [ -e "/dev/pmsg0" ]; then
+    echo "${TEST_STRING_PATTERN}""$UUID" > /dev/pmsg0
+    show_result $?
+    echo "$UUID" > $TOP_DIR/uuid
+else
+    prlog "FAIL"
+    rc=1
+fi
+
+exit $rc

+1

To test pstore in earnest, we have to cause kernel crash and check pstore filesystem after reboot.

We add two scripts:
- pstore_crash_test
This script causes kernel crash and reboot. It is executed by 'make run_pstore_crash' in selftests. It can also be used with kdump.
- pstore_post_reboot_tests
This script includes test cases which check pstore's behavior after crash and reboot. It is executed together with pstore_tests by 'make run_tests [-C pstore]' in selftests.

The test cases in pstore_post_reboot_tests are currently following.

- Check pstore backend is registered
- Mount pstore filesystem
- Check dmesg/console/pmsg files exist in pstore filesystem
- Check dmesg/console files contain oops end marker
- Check pmsg file properly keeps the content written before crash
- Remove all files in pstore filesystem

Example usage is following.

(before reboot)

  # cd /path/to/selftests
  # make run_tests -C pstore
  === Pstore unit tests (pstore_tests) ===
  UUID=b49b02cf-b0c2-4309-be43-b08c3971e37f
  ...
  selftests: pstore_tests [PASS]
  === Pstore unit tests (pstore_post_reboot_tests) ===
  UUID=953eb1bc-8e03-48d7-b27a-6552b24c5b7e
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  pstore_crash_test has not been executed yet. we skip further tests.
  selftests: pstore_post_reboot_tests [PASS]

  # make run_pstore_crash
  === Pstore unit tests (pstore_crash_test) ===
  UUID=93c8972d-1466-430b-8c4a-28d8681e74c6
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  Causing kernel crash ...
  (kernel crash and reboot)
  ...

(after reboot)

  # make run_tests -C pstore
  === Pstore unit tests (pstore_tests) ===
  UUID=8e511e77-2285-499f-8bc0-900d9af1fbcc
  ...
  selftests: pstore_tests [PASS]
  === Pstore unit tests (pstore_post_reboot_tests) ===
  UUID=2dcc2132-4f3c-45aa-a38f-3b54bff8cef1
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  Mounting pstore filesystem ... ok
  Checking dmesg files exist in pstore filesystem ... ok
          dmesg-ramoops-0
          dmesg-ramoops-1
  Checking console files exist in pstore filesystem ... ok
          console-ramoops-0
  Checking pmsg files exist in pstore filesystem ... ok
          pmsg-ramoops-0
  Checking dmesg files contain oops end marker
          dmesg-ramoops-0 ... ok
          dmesg-ramoops-1 ... ok
  Checking console file contains oops end marker ... ok
  Checking pmsg file properly keeps the content written before crash ... ok
  Removing all files in pstore filesystem
          console-ramoops-0 ... ok
          dmesg-ramoops-0 ... ok
          dmesg-ramoops-1 ... ok
          pmsg-ramoops-0 ... ok
  selftests: pstore_post_reboot_tests [PASS]

---
 tools/testing/selftests/Makefile                   |    3 +
 tools/testing/selftests/pstore/Makefile            |    7 +-
 tools/testing/selftests/pstore/common_tests        |   28 +++++++
 tools/testing/selftests/pstore/pstore_crash_test   |   30 ++++++++
 .../selftests/pstore/pstore_post_reboot_tests      |   77 ++++++++++++++++++++
 5 files changed, 143 insertions(+), 2 deletions(-)
 create mode 100755 tools/testing/selftests/pstore/pstore_crash_test
 create mode 100755 tools/testing/selftests/pstore/pstore_post_reboot_tests

index 1a8fb99..2458288 100644
@@ -66,6 +66,9 @@ clean_hotplug:
 		make -C $$TARGET clean; \
 	done;
 
+run_pstore_crash:
+	make -C pstore run_crash
+
 INSTALL_PATH ?= install
 INSTALL_PATH := $(abspath $(INSTALL_PATH))
 ALL_SCRIPT := $(INSTALL_PATH)/run_kselftest.sh

index 48623f7..bd7abe2 100644
@@ -3,10 +3,13 @@
 
 all:
 
-TEST_PROGS := pstore_tests
-TEST_FILES := common_tests
+TEST_PROGS := pstore_tests pstore_post_reboot_tests
+TEST_FILES := common_tests pstore_crash_test
 
 include ../lib.mk
 
+run_crash:
+	@sh pstore_crash_test || { echo "pstore_crash_test: [FAIL]"; exit 1; }
+
 clean:
 	rm -rf logs/* *uuid

index b1c3757..3ea64d7 100755
@@ -26,11 +26,39 @@ show_result() { # result_value
     fi
 }
 
+check_files_exist() { # type of pstorefs file
+    if [ -e ${1}-${backend}-0 ]; then
+	prlog "ok"
+	for f in `ls ${1}-${backend}-*`; do
+            prlog -e "\t${f}"
+	done
+    else
+	prlog "FAIL"
+	rc=1
+    fi
+}
+
+operate_files() { # tested value, files, operation
+    if [ $1 -eq 0 ]; then
+	prlog
+	for f in $2; do
+	    prlog -ne "\t${f} ... "
+	    # execute operation
+	    $3 $f
+	    show_result $?
+	done
+    else
+	prlog " ... FAIL"
+	rc=1
+    fi
+}
+
 # Parameters
 TEST_STRING_PATTERN="Testing pstore: uuid="
 UUID=`cat /proc/sys/kernel/random/uuid`
 TOP_DIR=`absdir $0`
 LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`_${UUID}/
+REBOOT_FLAG=$TOP_DIR/reboot_flag
 
 # Preparing logs
 LOG_FILE=$LOG_DIR/`basename $0`.log

new file mode 100755
index 0000000..1a4afe5
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+# pstore_crash_test - Pstore test shell script which causes crash and reboot
+#
+# Copyright (C) Hitachi Ltd., 2015
+#  Written by Hiraku Toyooka <[email protected]>
+#
+# Released under the terms of the GPL v2.
+
+# exit if pstore backend is not registered
+. ./common_tests
+
+prlog "Causing kernel crash ..."
+
+# enable all functions triggered by sysrq
+echo 1 > /proc/sys/kernel/sysrq
+# setting to reboot in 3 seconds after panic
+echo 3 > /proc/sys/kernel/panic
+
+# save uuid file by different name because next test execution will replace it.
+mv $TOP_DIR/uuid $TOP_DIR/prev_uuid
+
+# create a file as reboot flag
+touch $REBOOT_FLAG
+sync
+
+# cause crash
+# Note: If you use kdump and want to see kmesg-* files after reboot, you should
+#       specify 'crash_kexec_post_notifiers' in 1st kernel's cmdline.
+echo c > /proc/sysrq-trigger

+1

Thanks. Applied to linux-kselftest next for 4.4

thanks,
-- Shuah

Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
[email protected] | (970) 217-8978