Skip to contentSkip to conversation

💥 PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)

LTP CKI Project 3 participants 8 messages Mar 24, 2020 Stable branch

lore
Share & tools
lore

CKI Project

author
#0 lore raw

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP

    x86_64:
     ? xfstests - ext4

We hope that these logs can help you find the problem quickly. For the full detail on our testing procedures, please scroll to the bottom of this message.

Please reply to this email if you have any questions about the tests that we ran or if you have any suggestions on how to make future tests more effective.

        ,-.   ,-.
       ( C ) ( K )  Continuous
        `-',-.`-'   Kernel
          ( I )     Integration
           `-'

______________________________________________________________________________

Compile testing
---------------

We compiled the kernel for 4 architectures:

aarch64:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

ppc64le:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

s390x:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

x86_64:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg

Hardware testing
----------------
We booted each kernel and ran the following tests:

aarch64:
Host 1:
? Boot test
? Podman system integration test - as root ? Podman system integration test - as user ? LTP
? Loopdev Sanity
? Memory function: memfd_create
? AMTU (Abstract Machine Test Utility) ? Networking bridge: sanity
? Ethernet drivers sanity
? Networking MACsec: sanity
? Networking socket: fuzz
? Networking sctp-auth: sockopts test ? Networking: igmp conformance test
? Networking route: pmtu
? Networking route_func - local
? Networking route_func - forward
? Networking TCP: keepalive test
? Networking UDP: socket
? Networking tunnel: geneve basic test ? Networking tunnel: gre basic
? L2TP basic test
? Networking tunnel: vxlan basic
? Networking ipsec: basic netns - transport ? Networking ipsec: basic netns - tunnel ? httpd: mod_ssl smoke sanity
? tuned: tune-processes-through-perf ? ALSA PCM loopback test
? ALSA Control (mixer) Userspace Element test ? Usex - version 1.9-29
? storage: SCSI VPD
? trace: ftrace/tracer
? ? CIFS Connectathon
? ? POSIX pjd-fstest suites
? ? jvm - DaCapo Benchmark Suite
? ? jvm - jcstress tests
? ? Memory function: kaslr
? ? LTP: openposix test suite
? ? Networking vnic: ipvlan/basic
? ? audit: audit testsuite test
? ? iotop: sanity
? ? storage: dm/common

Host 2:

? Internal infrastructure issues prevented one or more tests (marked with ???) from running on this architecture.
This is not the fault of the kernel that was tested.

? Boot test
??? xfstests - ext4
??? xfstests - xfs
??? selinux-policy: serge-testsuite
??? lvm thinp sanity
??? storage: software RAID testing
??? stress: stress-ng
? ??? IPMI driver test
? ??? IPMItool loop stress test
? ??? Storage blktests

Host 3:
? Boot test
? xfstests - ext4
? xfstests - xfs
? selinux-policy: serge-testsuite
? lvm thinp sanity
? storage: software RAID testing
? ? Storage blktests

ppc64le:
Host 1:
? Boot test
? Podman system integration test - as root ? Podman system integration test - as user ? LTP
? Loopdev Sanity
? Memory function: memfd_create
? AMTU (Abstract Machine Test Utility) ? Networking bridge: sanity
? Ethernet drivers sanity
? Networking MACsec: sanity
? Networking socket: fuzz
? Networking sctp-auth: sockopts test ? Networking route: pmtu
? Networking route_func - local
? Networking route_func - forward
? Networking TCP: keepalive test
? Networking UDP: socket
? Networking tunnel: geneve basic test ? Networking tunnel: gre basic
? L2TP basic test
? Networking tunnel: vxlan basic
? Networking ipsec: basic netns - tunnel ? httpd: mod_ssl smoke sanity
? tuned: tune-processes-through-perf ? ALSA PCM loopback test
? ALSA Control (mixer) Userspace Element test ? Usex - version 1.9-29
? trace: ftrace/tracer
? ? CIFS Connectathon
? ? POSIX pjd-fstest suites
? ? jvm - DaCapo Benchmark Suite
? ? jvm - jcstress tests
? ? Memory function: kaslr
? ? LTP: openposix test suite
? ? Networking vnic: ipvlan/basic
? ? audit: audit testsuite test
? ? iotop: sanity
? ? storage: dm/common

Host 2:
? Boot test
? xfstests - ext4
??? xfstests - xfs
??? selinux-policy: serge-testsuite
??? lvm thinp sanity
??? storage: software RAID testing
? ??? IPMI driver test
? ??? IPMItool loop stress test
? ??? Storage blktests

s390x:
Host 1:
? Boot test
? Podman system integration test - as root ? Podman system integration test - as user ? Loopdev Sanity
? Memory function: memfd_create
? Networking bridge: sanity
? Ethernet drivers sanity
? Networking MACsec: sanity
? Networking sctp-auth: sockopts test ? Networking route: pmtu
? Networking route_func - local
? Networking route_func - forward
? Networking TCP: keepalive test
? Networking UDP: socket
? Networking tunnel: geneve basic test ? Networking tunnel: gre basic
? L2TP basic test
? Networking tunnel: vxlan basic
? Networking ipsec: basic netns - transport ? Networking ipsec: basic netns - tunnel ? httpd: mod_ssl smoke sanity
? tuned: tune-processes-through-perf ? trace: ftrace/tracer
? ? CIFS Connectathon
? ? POSIX pjd-fstest suites
? ? jvm - DaCapo Benchmark Suite
? ? jvm - jcstress tests
? ? Memory function: kaslr
? ? LTP: openposix test suite
? ? Networking vnic: ipvlan/basic
? ? audit: audit testsuite test
? ? iotop: sanity
? ? storage: dm/common

Host 2:
? Boot test
? selinux-policy: serge-testsuite
? ? Storage blktests

x86_64:
Host 1:
? Boot test
? Storage SAN device stress - megaraid_sas

Host 2:
? Boot test
? xfstests - ext4
??? xfstests - xfs
??? selinux-policy: serge-testsuite ??? lvm thinp sanity
??? storage: software RAID testing ??? stress: stress-ng
? ??? IOMMU boot test
? ??? IPMI driver test
? ??? IPMItool loop stress test
? ??? power-management: cpupower/sanity test ? ??? Storage blktests

Host 3:
? Boot test
? Podman system integration test - as root ? Podman system integration test - as user ? LTP
? Loopdev Sanity
? Memory function: memfd_create
? AMTU (Abstract Machine Test Utility) ? Networking bridge: sanity
? Ethernet drivers sanity
? Networking MACsec: sanity
? Networking socket: fuzz
? Networking sctp-auth: sockopts test ? Networking: igmp conformance test
? Networking route: pmtu
? Networking route_func - local
? Networking route_func - forward
? Networking TCP: keepalive test
? Networking UDP: socket
? Networking tunnel: geneve basic test ? Networking tunnel: gre basic
? L2TP basic test
? Networking tunnel: vxlan basic
? Networking ipsec: basic netns - transport ? Networking ipsec: basic netns - tunnel ? httpd: mod_ssl smoke sanity
? tuned: tune-processes-through-perf ? pciutils: sanity smoke test
? ALSA PCM loopback test
? ALSA Control (mixer) Userspace Element test ? Usex - version 1.9-29
? storage: SCSI VPD
? trace: ftrace/tracer
? ? CIFS Connectathon
? ? POSIX pjd-fstest suites
? ? jvm - DaCapo Benchmark Suite
? ? jvm - jcstress tests
? ? Memory function: kaslr
? ? LTP: openposix test suite
? ? Networking vnic: ipvlan/basic
? ? audit: audit testsuite test
? ? iotop: sanity
? ? storage: dm/common

Host 4:
? Boot test
? Storage SAN device stress - mpt3sas driver

Test sources: https://github.com/CKI-project/tests-beaker
? Pull requests are welcome for new tests or improvements to existing tests!

Aborted tests
-------------
Tests that didn't complete running successfully are marked with ???. If this was caused by an infrastructure issue, we try to mark that explicitly in the report.

Waived tests
------------
If the test run included waived tests, they are marked with ?. Such tests are executed but their results are not taken into account. Tests are waived when their results are not reliable enough, e.g. when they're just introduced or are being fixed.

Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't finished running yet are marked with ?.

#1 lore raw
Quoted 21 lines · CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

Quoted 8 lines
    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP

    x86_64:
     ? xfstests - ext4

Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(

Remember, if something starts breaking, I need some way to find out what caused it to break...

greg k-h

#2 lore raw

----- Original Message -----

Quoted 37 lines

From: "Greg KH" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 7:22:13 AM
Subject: Re: ? PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 29 lines · CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
            Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP
Quoted 9 lines
Quoted 2 lines

x86_64:
? xfstests - ext4

Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(

Remember, if something starts breaking, I need some way to find out what caused it to break...

Hi Greg,

do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?

[0]: While these are new ones and not matching what we saw previously
(that's why they weren't filtered out), the following commits from block tree should likely fix them:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?id=fd1bb3ae54a9a2e0c42709de861c69aa146b8955
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?id=c8997736650060594845e42c5d01d3118aec8d25
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?id=576682fa52cbd95deb3773449566274f206acc58
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?id=4d38a87fbb77fb9ff2ff4e914162a8ae6453eff5

Veronika

Quoted 4 lines

greg k-h

#3 lore raw

[LTP] 💥 PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 50 lines · Veronika Kabatova

----- Original Message -----

Quoted 37 lines

From: "Greg KH" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 7:22:13 AM
Subject: Re: ? PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 29 lines · CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
            Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP
Quoted 9 lines
Quoted 2 lines

x86_64:
? xfstests - ext4

Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(

Remember, if something starts breaking, I need some way to find out what caused it to break...

Quoted 6 lines

Hi Greg,

do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?

Bisection to find the offending commit would be best.

thanks,

greg k-h

#4 lore raw

[LTP] 💥 PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)

----- Original Message -----

Quoted 62 lines

From: "Greg KH" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "CKI Project" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 12:18:19 PM
Subject: Re: ? PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 53 lines · Veronika Kabatova

----- Original Message -----

Quoted 49 lines

From: "Greg KH" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 7:22:13 AM
Subject: Re: ? PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki
(stable)

Quoted 23 lines · CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
            Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

Quoted 8 lines
    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP

    x86_64:
     ? xfstests - ext4

Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(

Remember, if something starts breaking, I need some way to find out what caused it to break...

Quoted 9 lines
Quoted 6 lines

Hi Greg,

do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?

Bisection to find the offending commit would be best.

This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!

Veronika

Quoted 6 lines

thanks,

greg k-h

#5 lore raw

[LTP] 💥 PANICKED: Test report?for?kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 72 lines · Veronika Kabatova

----- Original Message -----

Quoted 69 lines

From: "Greg KH" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "CKI Project" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 12:18:19 PM
Subject: Re: ? PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 59 lines · Veronika Kabatova

----- Original Message -----

Quoted 40 lines

From: "Greg KH" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 7:22:13 AM
Subject: Re: ? PANICKED: Test report for kernel 5.5.12-rc1-8b841eb.cki
(stable)

Quoted 29 lines · CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
            Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP
Quoted 9 lines
Quoted 2 lines

x86_64:
? xfstests - ext4

Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(

Remember, if something starts breaking, I need some way to find out what caused it to break...

Hi Greg,

do you have any specific suggestions about what to include to help you out? The linked console logs contain call traces for the panics [0]. Is there anything else that would help you with debugging those? We're planning on releasing core dumps, would those be helpful?

Quoted 5 lines
Quoted 2 lines

Bisection to find the offending commit would be best.

This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!

I got about 8 "failed" emails today, it doesn't sound like it is hard to reproduce.

And if you can't reproduce it, why would you expect me to know what is going on? :)

thanks,

greg k-h

#6 lore raw

[LTP] 💥 PANICKED: Test report?for?kernel 5.5.12-rc1-8b841eb.cki (stable)

----- Original Message -----

Quoted 90 lines

From: "Greg KH" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "CKI Project" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 12:47:27 PM
Subject: Re: ? PANICKED: Test report?for?kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 81 lines · Veronika Kabatova

----- Original Message -----

Quoted 78 lines

From: "Greg KH" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "CKI Project" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing
List" <[email protected]>
Sent: Tuesday, March 24, 2020 12:18:19 PM
Subject: Re: ? PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki
(stable)

Quoted 65 lines · Veronika Kabatova

----- Original Message -----

Quoted 45 lines

From: "Greg KH" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 7:22:13 AM
Subject: Re: ? PANICKED: Test report for kernel
5.5.12-rc1-8b841eb.cki
(stable)

Quoted 24 lines · CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
            Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download
here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

Quoted 8 lines
    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP

    x86_64:
     ? xfstests - ext4
Quoted 7 lines

Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(

Remember, if something starts breaking, I need some way to find out what
caused it to break...

Hi Greg,

do you have any specific suggestions about what to include to help you out?
The linked console logs contain call traces for the panics [0]. Is there
anything else that would help you with debugging those? We're planning on
releasing core dumps, would those be helpful?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Bisection to find the offending commit would be best.

This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!

I got about 8 "failed" emails today, it doesn't sound like it is hard to reproduce.

I meant in general, as we do catch a bunch of such problems. Those usually require some input and testing from people who understand the subsystem to create a reliable reproducer.

For this particular failure, do you want to try out the patches I linked in the first email or should we waive them for now?

Quoted 4 lines

And if you can't reproduce it, why would you expect me to know what is going on? :)

Well you do know the kernel pretty well :) We also cc the test maintainers to help out if it's needed since they understand the failed area, though given the current situation around the world people's reactions might understandably be slower.

Veronika

Quoted 6 lines

thanks,

greg k-h

#7 lore raw

[LTP] 💥 PANICKED: Test?report?for?kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 102 lines · Veronika Kabatova

----- Original Message -----

Quoted 90 lines

From: "Greg KH" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "CKI Project" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 12:47:27 PM
Subject: Re: ? PANICKED: Test report?for?kernel 5.5.12-rc1-8b841eb.cki (stable)

Quoted 81 lines · Veronika Kabatova

----- Original Message -----

Quoted 78 lines

From: "Greg KH" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "CKI Project" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing
List" <[email protected]>
Sent: Tuesday, March 24, 2020 12:18:19 PM
Subject: Re: ? PANICKED: Test report for?kernel 5.5.12-rc1-8b841eb.cki
(stable)

Quoted 65 lines · Veronika Kabatova

----- Original Message -----

Quoted 45 lines

From: "Greg KH" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "Memory Management" <[email protected]>, "Ondrej Mosnacek" <[email protected]>, "Linux Stable maillist" <[email protected]>, "Jan Stancek" <[email protected]>, "LTP Mailing List" <[email protected]>
Sent: Tuesday, March 24, 2020 7:22:13 AM
Subject: Re: ? PANICKED: Test report for kernel
5.5.12-rc1-8b841eb.cki
(stable)

Quoted 24 lines · CKI Project

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
            Commit: 8b841eb697e1 - Linux 5.5.12-rc1

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: PANICKED

All kernel binaries, config files, and logs are available for download
here:

https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/23/502039

One or more kernel tests failed:

Quoted 8 lines
    ppc64le:
     ? xfstests - ext4

    aarch64:
     ? LTP

    x86_64:
     ? xfstests - ext4
Quoted 7 lines

Ok, it's time I start just blacklisting this report again, it's not being helpful in any way :(

Remember, if something starts breaking, I need some way to find out what
caused it to break...

Hi Greg,

do you have any specific suggestions about what to include to help you out?
The linked console logs contain call traces for the panics [0]. Is there
anything else that would help you with debugging those? We're planning on
releasing core dumps, would those be helpful?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Bisection to find the offending commit would be best.

This is going to be really tricky for hard to reproduce bugs but we'll do some research on it, thanks!

I got about 8 "failed" emails today, it doesn't sound like it is hard to reproduce.

Quoted 6 lines

I meant in general, as we do catch a bunch of such problems. Those usually require some input and testing from people who understand the subsystem to create a reliable reproducer.

For this particular failure, do you want to try out the patches I linked in the first email or should we waive them for now?

As this must be something new introduced by my current queue, why not find out what patch in there is causing the problem?

If this is caused by something NOT in my queue, and is also a problem upstream in Linus's tree, then you need to notify the upstream developers as there's nothing I can do about it, right?

If this is not upstream, but is a problem in the stable tree, then please submit the patches that fix the issue as documented in the stable kernel rules document for how to submit them.

To throw this horrible "something is BROKEN!" type email at me all the time just makes me want to delete it as I have no context at all.

thanks,

greg k-h