Skip to contentSkip to conversation

πŸ’₯ PANICKED: Test report for kernel 5.7.5-d1c48db.cki (stable-queue)

DISCUSSION CKI Project 3 participants 3 messages Jun 22, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Commit: d1c48dba15e7 - pinctrl: qcom: ipq6018 Add missing pins in qpic pin group

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/06/22/607452

One or more kernel tests failed:

s390x:
πŸ’₯ selinux-policy: serge-testsuite
❌ stress: stress-ng
πŸ’₯ Podman system integration test - as root

ppc64le:
πŸ’₯ selinux-policy: serge-testsuite
πŸ’₯ Podman system integration test - as root

aarch64:
πŸ’₯ Podman system integration test - as root
πŸ’₯ selinux-policy: serge-testsuite

x86_64:
πŸ’₯ Podman system integration test - as root
πŸ’₯ selinux-policy: serge-testsuite

We hope that these logs can help you find the problem quickly. For the full detail on our testing procedures, please scroll to the bottom of this message.

Please reply to this email if you have any questions about the tests that we ran or if you have any suggestions on how to make future tests more effective.

        ,-.   ,-.
       ( C ) ( K )  Continuous
        `-',-.`-'   Kernel
          ( I )     Integration
           `-'

______________________________________________________________________________

Compile testing
---------------

We compiled the kernel for 4 architectures:

aarch64:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

ppc64le:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

s390x:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

x86_64:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

Hardware testing
----------------
We booted each kernel and ran the following tests:

aarch64:
Host 1:
βœ… Boot test
πŸ’₯ Podman system integration test - as root ⚑⚑⚑ Podman system integration test - as user ⚑⚑⚑ LTP
⚑⚑⚑ Loopdev Sanity
⚑⚑⚑ Memory function: memfd_create
⚑⚑⚑ AMTU (Abstract Machine Test Utility) ⚑⚑⚑ Networking bridge: sanity
⚑⚑⚑ Ethernet drivers sanity
⚑⚑⚑ Networking socket: fuzz
⚑⚑⚑ Networking: igmp conformance test ⚑⚑⚑ Networking route: pmtu
⚑⚑⚑ Networking route_func - local
⚑⚑⚑ Networking route_func - forward ⚑⚑⚑ Networking TCP: keepalive test
⚑⚑⚑ Networking UDP: socket
⚑⚑⚑ Networking tunnel: geneve basic test ⚑⚑⚑ Networking tunnel: gre basic
⚑⚑⚑ L2TP basic test
⚑⚑⚑ Networking tunnel: vxlan basic
⚑⚑⚑ Networking ipsec: basic netns - transport ⚑⚑⚑ Networking ipsec: basic netns - tunnel ⚑⚑⚑ Libkcapi AF_ALG test
⚑⚑⚑ pciutils: update pci ids test
⚑⚑⚑ ALSA PCM loopback test
⚑⚑⚑ ALSA Control (mixer) Userspace Element test ⚑⚑⚑ storage: SCSI VPD
🚧 ⚑⚑⚑ CIFS Connectathon
🚧 ⚑⚑⚑ POSIX pjd-fstest suites
🚧 ⚑⚑⚑ jvm - DaCapo Benchmark Suite
🚧 ⚑⚑⚑ jvm - jcstress tests
🚧 ⚑⚑⚑ Memory function: kaslr
🚧 ⚑⚑⚑ Networking firewall: basic netfilter test 🚧 ⚑⚑⚑ audit: audit testsuite test
🚧 ⚑⚑⚑ trace: ftrace/tracer
🚧 ⚑⚑⚑ kdump - kexec_boot

Host 2:

⚑ Internal infrastructure issues prevented one or more tests (marked with ⚑⚑⚑) from running on this architecture.
This is not the fault of the kernel that was tested.

⚑⚑⚑ Boot test
⚑⚑⚑ xfstests - ext4
⚑⚑⚑ xfstests - xfs
⚑⚑⚑ selinux-policy: serge-testsuite
⚑⚑⚑ storage: software RAID testing
⚑⚑⚑ stress: stress-ng
🚧 ⚑⚑⚑ IPMI driver test
🚧 ⚑⚑⚑ IPMItool loop stress test
🚧 ⚑⚑⚑ Storage blktests

Host 3:

⚑ Internal infrastructure issues prevented one or more tests (marked with ⚑⚑⚑) from running on this architecture.
This is not the fault of the kernel that was tested.

βœ… Boot test
βœ… xfstests - ext4
βœ… xfstests - xfs
πŸ’₯ selinux-policy: serge-testsuite
βœ… storage: software RAID testing
🚧 ❌ IPMI driver test
🚧 βœ… IPMItool loop stress test
🚧 ⚑⚑⚑ Storage blktests

ppc64le:
Host 1:
βœ… Boot test
🚧 ❌ kdump - sysrq-c

Host 2:

⚑ Internal infrastructure issues prevented one or more tests (marked with ⚑⚑⚑) from running on this architecture.
This is not the fault of the kernel that was tested.

βœ… Boot test
βœ… xfstests - ext4
βœ… xfstests - xfs
πŸ’₯ selinux-policy: serge-testsuite
βœ… storage: software RAID testing
🚧 ❌ IPMI driver test
🚧 βœ… IPMItool loop stress test
🚧 ⚑⚑⚑ Storage blktests

Host 3:
βœ… Boot test
πŸ’₯ Podman system integration test - as root ⚑⚑⚑ Podman system integration test - as user ⚑⚑⚑ LTP
⚑⚑⚑ Loopdev Sanity
⚑⚑⚑ Memory function: memfd_create
⚑⚑⚑ AMTU (Abstract Machine Test Utility) ⚑⚑⚑ Networking bridge: sanity
⚑⚑⚑ Ethernet drivers sanity
⚑⚑⚑ Networking socket: fuzz
⚑⚑⚑ Networking route: pmtu
⚑⚑⚑ Networking route_func - local
⚑⚑⚑ Networking route_func - forward ⚑⚑⚑ Networking TCP: keepalive test
⚑⚑⚑ Networking UDP: socket
⚑⚑⚑ Networking tunnel: geneve basic test ⚑⚑⚑ Networking tunnel: gre basic
⚑⚑⚑ L2TP basic test
⚑⚑⚑ Networking tunnel: vxlan basic
⚑⚑⚑ Networking ipsec: basic netns - tunnel ⚑⚑⚑ Libkcapi AF_ALG test
⚑⚑⚑ pciutils: update pci ids test
⚑⚑⚑ ALSA PCM loopback test
⚑⚑⚑ ALSA Control (mixer) Userspace Element test 🚧 ⚑⚑⚑ CIFS Connectathon
🚧 ⚑⚑⚑ POSIX pjd-fstest suites
🚧 ⚑⚑⚑ jvm - DaCapo Benchmark Suite
🚧 ⚑⚑⚑ jvm - jcstress tests
🚧 ⚑⚑⚑ Memory function: kaslr
🚧 ⚑⚑⚑ Networking firewall: basic netfilter test 🚧 ⚑⚑⚑ audit: audit testsuite test
🚧 ⚑⚑⚑ trace: ftrace/tracer

s390x:
Host 1:

⚑ Internal infrastructure issues prevented one or more tests (marked with ⚑⚑⚑) from running on this architecture.
This is not the fault of the kernel that was tested.

⚑⚑⚑ Boot test
⚑⚑⚑ Podman system integration test - as root ⚑⚑⚑ Podman system integration test - as user ⚑⚑⚑ LTP
⚑⚑⚑ Loopdev Sanity
⚑⚑⚑ Memory function: memfd_create
⚑⚑⚑ Networking bridge: sanity
⚑⚑⚑ Ethernet drivers sanity
⚑⚑⚑ Networking route: pmtu
⚑⚑⚑ Networking route_func - local
⚑⚑⚑ Networking route_func - forward ⚑⚑⚑ Networking TCP: keepalive test
⚑⚑⚑ Networking UDP: socket
⚑⚑⚑ Networking tunnel: geneve basic test ⚑⚑⚑ Networking tunnel: gre basic
⚑⚑⚑ L2TP basic test
⚑⚑⚑ Networking tunnel: vxlan basic
⚑⚑⚑ Networking ipsec: basic netns - transport ⚑⚑⚑ Networking ipsec: basic netns - tunnel ⚑⚑⚑ Libkcapi AF_ALG test
🚧 ⚑⚑⚑ CIFS Connectathon
🚧 ⚑⚑⚑ POSIX pjd-fstest suites
🚧 ⚑⚑⚑ jvm - DaCapo Benchmark Suite
🚧 ⚑⚑⚑ jvm - jcstress tests
🚧 ⚑⚑⚑ Memory function: kaslr
🚧 ⚑⚑⚑ Networking firewall: basic netfilter test 🚧 ⚑⚑⚑ audit: audit testsuite test
🚧 ⚑⚑⚑ trace: ftrace/tracer
🚧 ⚑⚑⚑ kdump - kexec_boot

Host 2:

⚑ Internal infrastructure issues prevented one or more tests (marked with ⚑⚑⚑) from running on this architecture.
This is not the fault of the kernel that was tested.

βœ… Boot test
🚧 ⚑⚑⚑ kdump - sysrq-c

Host 3:

⚑ Internal infrastructure issues prevented one or more tests (marked with ⚑⚑⚑) from running on this architecture.
This is not the fault of the kernel that was tested.

βœ… Boot test
πŸ’₯ selinux-policy: serge-testsuite
❌ stress: stress-ng
🚧 ⚑⚑⚑ Storage blktests

Host 4:
βœ… Boot test
πŸ’₯ Podman system integration test - as root ⚑⚑⚑ Podman system integration test - as user ⚑⚑⚑ LTP
⚑⚑⚑ Loopdev Sanity
⚑⚑⚑ Memory function: memfd_create
⚑⚑⚑ Networking bridge: sanity
⚑⚑⚑ Ethernet drivers sanity
⚑⚑⚑ Networking route: pmtu
⚑⚑⚑ Networking route_func - local
⚑⚑⚑ Networking route_func - forward ⚑⚑⚑ Networking TCP: keepalive test
⚑⚑⚑ Networking UDP: socket
⚑⚑⚑ Networking tunnel: geneve basic test ⚑⚑⚑ Networking tunnel: gre basic
⚑⚑⚑ L2TP basic test
⚑⚑⚑ Networking tunnel: vxlan basic
⚑⚑⚑ Networking ipsec: basic netns - transport ⚑⚑⚑ Networking ipsec: basic netns - tunnel ⚑⚑⚑ Libkcapi AF_ALG test
🚧 ⚑⚑⚑ CIFS Connectathon
🚧 ⚑⚑⚑ POSIX pjd-fstest suites
🚧 ⚑⚑⚑ jvm - DaCapo Benchmark Suite
🚧 ⚑⚑⚑ jvm - jcstress tests
🚧 ⚑⚑⚑ Memory function: kaslr
🚧 ⚑⚑⚑ Networking firewall: basic netfilter test 🚧 ⚑⚑⚑ audit: audit testsuite test
🚧 ⚑⚑⚑ trace: ftrace/tracer
🚧 ⚑⚑⚑ kdump - kexec_boot

x86_64:
Host 1:
βœ… Boot test
πŸ’₯ Podman system integration test - as root ⚑⚑⚑ Podman system integration test - as user ⚑⚑⚑ LTP
⚑⚑⚑ Loopdev Sanity
⚑⚑⚑ Memory function: memfd_create
⚑⚑⚑ AMTU (Abstract Machine Test Utility) ⚑⚑⚑ Networking bridge: sanity
⚑⚑⚑ Ethernet drivers sanity
⚑⚑⚑ Networking socket: fuzz
⚑⚑⚑ Networking: igmp conformance test ⚑⚑⚑ Networking route: pmtu
⚑⚑⚑ Networking route_func - local
⚑⚑⚑ Networking route_func - forward ⚑⚑⚑ Networking TCP: keepalive test
⚑⚑⚑ Networking UDP: socket
⚑⚑⚑ Networking tunnel: geneve basic test ⚑⚑⚑ Networking tunnel: gre basic
⚑⚑⚑ L2TP basic test
⚑⚑⚑ Networking tunnel: vxlan basic
⚑⚑⚑ Networking ipsec: basic netns - transport ⚑⚑⚑ Networking ipsec: basic netns - tunnel ⚑⚑⚑ Libkcapi AF_ALG test
⚑⚑⚑ pciutils: sanity smoke test
⚑⚑⚑ pciutils: update pci ids test
⚑⚑⚑ ALSA PCM loopback test
⚑⚑⚑ ALSA Control (mixer) Userspace Element test ⚑⚑⚑ storage: SCSI VPD
🚧 ⚑⚑⚑ CIFS Connectathon
🚧 ⚑⚑⚑ POSIX pjd-fstest suites
🚧 ⚑⚑⚑ jvm - DaCapo Benchmark Suite
🚧 ⚑⚑⚑ jvm - jcstress tests
🚧 ⚑⚑⚑ Memory function: kaslr
🚧 ⚑⚑⚑ Networking firewall: basic netfilter test 🚧 ⚑⚑⚑ audit: audit testsuite test
🚧 ⚑⚑⚑ trace: ftrace/tracer
🚧 ⚑⚑⚑ kdump - kexec_boot

Host 2:

⚑ Internal infrastructure issues prevented one or more tests (marked with ⚑⚑⚑) from running on this architecture.
This is not the fault of the kernel that was tested.

βœ… Boot test
βœ… xfstests - ext4
βœ… xfstests - xfs
πŸ’₯ selinux-policy: serge-testsuite
βœ… storage: software RAID testing
⚑⚑⚑ stress: stress-ng
🚧 ❌ CPU: Frequency Driver Test
🚧 βœ… CPU: Idle Test
🚧 βœ… IOMMU boot test
🚧 βœ… IPMI driver test
🚧 βœ… IPMItool loop stress test
🚧 ⚑⚑⚑ Storage blktests

Host 3:
βœ… Boot test
🚧 ❌ kdump - sysrq-c

Test sources: https://github.com/CKI-project/tests-beaker
πŸ’š Pull requests are welcome for new tests or improvements to existing tests!

Aborted tests
-------------
Tests that didn't complete running successfully are marked with ⚑⚑⚑. If this was caused by an infrastructure issue, we try to mark that explicitly in the report.

Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are executed but their results are not taken into account. Tests are waived when their results are not reliable enough, e.g. when they're just introduced or are being fixed.

Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't finished running yet are marked with ⏱.

#1 lore raw
Quoted 29 lines Β· CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Commit: d1c48dba15e7 - pinctrl: qcom: ipq6018 Add missing pins in qpic pin group

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/06/22/607452

One or more kernel tests failed:

s390x:
selinux-policy: serge-testsuite
❌ stress: stress-ng
Podman system integration test - as root

ppc64le:
selinux-policy: serge-testsuite
Podman system integration test - as root

Quoted 7 lines

aarch64:
Podman system integration test - as root
selinux-policy: serge-testsuite

x86_64:
Podman system integration test - as root
selinux-policy: serge-testsuite

It seems the panics are caused by this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/commit/fs?h=linux-5.7.y&id=7bdb075348e33d757bd8cdf017a255a090e87b8f

Apparently, the "if (flags & ~(O_ACCMODE | O_LARGEFILE))" check is being hit during mount(2):
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/tree/fs/overlayfs/util.c?h=d1c48db#n465

[ 8149.093006] ------------[ cut here ]------------
[ 8149.097614] kernel BUG at fs/overlayfs/util.c:466!
[ 8149.102386] Internal error: Oops - BUG: 0 [#1] SMP
[ 8149.107155] Modules linked in: overlay xt_CONNSECMARK xt_SECMARK

nft_counter xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
nft_compat nf_tables nfnetlink ah6 ah4 sctp dm_log_writes dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rfkill
sunrpc xgene_enet xgene_hwmon xgene_edac vfat fat at803x mdio_xgene xgene_rng mailbox_xgene_slimpro drm ip_tables xfs libcrc32c sdhci_of_arasan
sdhci_pltfm gpio_dwapb i2c_xgene_slimpro crct10dif_ce sdhci cqhci gpio_xgene_sb xhci_plat_hcd gpio_keys aes_neon_bs

[ 8149.154907] CPU: 5 PID: 498222 Comm: mount Kdump: loaded Not
tainted 5.7.5-d1c48db.cki #1
[ 8149.163043] Hardware name: AmpereComputing(R) Mustang/Mustang, BIOS
0ACDY027 12/12/2018
[ 8149.171008] pstate: 80400005 (Nzcv daif +PAN -UAO)
[ 8149.175786] pc : ovl_path_open+0xbc/0xc0 [overlay]
[ 8149.180558] lr : ovl_check_d_type_supported+0x48/0x100 [overlay]
[ 8149.186533] sp : ffff800013463ab0
[ 8149.189829] x29: ffff800013463ab0 x28: ffff000379622040
[ 8149.195114] x27: 0000000000000000 x26: 0000000000000000
[ 8149.200399] x25: ffff000376d80240 x24: ffff000376d80b40
[ 8149.205684] x23: ffff0003da083e20 x22: ffff800013463cd8
[ 8149.210969] x21: ffff00039cae0540 x20: ffff800013463cd8
[ 8149.216252] x19: 0000000000004000 x18: 0000000000000010
[ 8149.221536] x17: 0000000000000000 x16: 0000000000000000
[ 8149.226820] x15: 095a041701101c00 x14: 745f73656c69665f
[ 8149.232104] x13: 7265746e756f6d5f x12: 79616c7265766f5f
[ 8149.237387] x11: 747365743a725f74 x10: 00000000000fffff
[ 8149.242672] x9 : ffff80000943df20 x8 : ffff0003e2f7f188
[ 8149.247956] x7 : 0000000000000000 x6 : ffff800011d65be8
[ 8149.253240] x5 : ffff0003d20cfbc8 x4 : 61c8864680b583eb
[ 8149.258523] x3 : ffff800011d56180 x2 : ffff80000943c470
[ 8149.263808] x1 : ffff000376d80b40 x0 : 0000000000004000
[ 8149.269091] Call trace:
[ 8149.271531]  ovl_path_open+0xbc/0xc0 [overlay]
[ 8149.275958]  ovl_check_d_type_supported+0x48/0x100 [overlay]
[ 8149.281593]  ovl_make_workdir+0xd0/0x460 [overlay]
[ 8149.286364]  ovl_fill_super+0x298/0xb98 [overlay]
[ 8149.291047]  mount_nodev+0x58/0xb8
[ 8149.294435]  ovl_mount+0x24/0x30 [overlay]
[ 8149.298512]  legacy_get_tree+0x38/0x68
[ 8149.302241]  vfs_get_tree+0x30/0xf8
[ 8149.305712]  do_mount+0x55c/0xa80
[ 8149.309010]  __arm64_sys_mount+0x94/0x100
[ 8149.312998]  el0_svc_common.constprop.0+0x7c/0x188
[ 8149.317763]  do_el0_svc+0x2c/0x98
[ 8149.321061]  el0_sync_handler+0x184/0x204
[ 8149.325049]  el0_sync+0x17c/0x180
[ 8149.328349] Code: a8c37bfd d50323bf d65f03c0 d4210000 (d4210000)
[ 8149.334413] ---[ end trace bc6ed2af25e97165 ]---
[ 8149.339375] ------------[ cut here ]------------

--
Ondrej Mosnacek
Software Engineer, Platform Security - SELinux kernel Red Hat, Inc.

maintainer
#2 lore raw
Quoted 38 lines Β· Ondrej Mosnacek
Quoted 29 lines Β· CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Commit: d1c48dba15e7 - pinctrl: qcom: ipq6018 Add missing pins in qpic pin group

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/06/22/607452

One or more kernel tests failed:

s390x:
selinux-policy: serge-testsuite
❌ stress: stress-ng
Podman system integration test - as root

ppc64le:
selinux-policy: serge-testsuite
Podman system integration test - as root

Quoted 7 lines

aarch64:
Podman system integration test - as root
selinux-policy: serge-testsuite

x86_64:
Podman system integration test - as root
selinux-policy: serge-testsuite

Quoted 6 lines

It seems the panics are caused by this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/commit/fs?h=linux-5.7.y&id=7bdb075348e33d757bd8cdf017a255a090e87b8f

Apparently, the "if (flags & ~(O_ACCMODE | O_LARGEFILE))" check is being hit during mount(2):
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/tree/fs/overlayfs/util.c?h=d1c48db#n465

I'll drop it for now, thanks!

Signature Β· 2 lines
Thanks,
Sasha