❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

We ran automated tests on a recent commit from this kernel tree:

Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature' into for-kernelci

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: FAILED

All kernel binaries, config files, and logs are available for download here:

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431

One or more kernel tests failed:

aarch64:
❌ Boot test

Host 2:
❌ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ xfstests - btrfs

🚧 ⚡⚡⚡ IPMI driver test
🚧 ⚡⚡⚡ IPMItool loop stress test
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage block - filesystem fio test 🚧 ⚡⚡⚡ Storage block - queue scheduler test 🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test 🚧 ⚡⚡⚡ stress: stress-ng

From: "Will Deacon" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "catalin marinas" <[email protected]>, [email protected]
Sent: Wednesday, February 10, 2021 10:29:46 AM
Subject: Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

Hi CKI folks,

Please can you help me with the error below?

Quoted 23 lines · CKI Project

We ran automated tests on a recent commit from this kernel tree:

Kernel repo:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature' into for-kernelci

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: FAILED

All kernel binaries, config files, and logs are available for download here:

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431

One or more kernel tests failed:

aarch64:
❌ Boot test

[...]

Quoted 7 lines

Host 2:
❌ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ xfstests - btrfs

Quoted 8 lines

🚧 ⚡⚡⚡ IPMI driver test
🚧 ⚡⚡⚡ IPMItool loop stress test
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage block - filesystem fio test 🚧 ⚡⚡⚡ Storage block - queue scheduler test 🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test 🚧 ⚡⚡⚡ stress: stress-ng

Which system (e.g. soc) is host 2 and are there are known infra issues at the moment? I did push some changes which affect the early boot path, so we may well be running into a kernel bug, but I'd just like to make sure before we dive in trying to debug that, especially as we haven't seen failures on other systems (and host 1 seems ok).

In the meantime, I'll push a new branch without those early changes to see if that does any better.

Thanks,

Will

From: "Veronika Kabatova" <[email protected]>
To: "Will Deacon" <[email protected]>
Cc: "catalin marinas" <[email protected]>, "CKI Project" <[email protected]>, [email protected]
Sent: Wednesday, February 10, 2021 1:01:36 PM
Subject: Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

----- Original Message -----

Quoted 55 lines

From: "Will Deacon" <[email protected]>
To: "CKI Project" <[email protected]>
Cc: "catalin marinas" <[email protected]>,
[email protected]
Sent: Wednesday, February 10, 2021 10:29:46 AM
Subject: Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

Hi CKI folks,

Please can you help me with the error below?

Quoted 24 lines · CKI Project

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
            Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature'
            into
            for-kernelci

The results of these automated tests are provided below.

Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: FAILED

All kernel binaries, config files, and logs are available for download here:

https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431

One or more kernel tests failed:

aarch64:
❌ Boot test

[...]

Quoted 7 lines

Host 2:
❌ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ xfstests - btrfs

Quoted 8 lines

🚧 ⚡⚡⚡ IPMI driver test
🚧 ⚡⚡⚡ IPMItool loop stress test
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage block - filesystem fio test 🚧 ⚡⚡⚡ Storage block - queue scheduler test 🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test 🚧 ⚡⚡⚡ stress: stress-ng

Quoted 7 lines

Which system (e.g. soc) is host 2 and are there are known infra issues at the moment? I did push some changes which affect the early boot path, so we may well be running into a kernel bug, but I'd just like to make sure before
we dive in trying to debug that, especially as we haven't seen failures on other systems (and host 1 seems ok).

Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable kernel just a few days back okay. The last messages I can see in the raw console log from this run are:

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

and then it times out after hour and half. I'm not aware of any ongoing issues, however sometimes the link between the lab controller and the machines can sometimes go wrong after reboot and lead to a similarly looking problem.

I'll resubmit the test job on that same machine to check if that was the case and let you know right after it boots.

Veronika

Quoted 9 lines

In the meantime, I'll push a new branch without those early changes to see if that does any better.

Thanks,

Will

Quoted 39 lines

Quoted 17 lines · CKI Project

Quoted 7 lines

Host 2:
❌ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ xfstests - btrfs

Quoted 8 lines

🚧 ⚡⚡⚡ IPMI driver test
🚧 ⚡⚡⚡ IPMItool loop stress test
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage block - filesystem fio test 🚧 ⚡⚡⚡ Storage block - queue scheduler test 🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test 🚧 ⚡⚡⚡ stress: stress-ng

Quoted 7 lines

Which system (e.g. soc) is host 2 and are there are known infra issues at the moment? I did push some changes which affect the early boot path, so we may well be running into a kernel bug, but I'd just like to make sure before
we dive in trying to debug that, especially as we haven't seen failures on other systems (and host 1 seems ok).

Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable kernel just a few days back okay. The last messages I can see in the raw console log from this run are:

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

and then it times out after hour and half. I'm not aware of any ongoing issues, however sometimes the link between the lab controller and the machines can sometimes go wrong after reboot and lead to a similarly looking problem.

Quoted 3 lines

I'll resubmit the test job on that same machine to check if that was the case and let you know right after it boots.

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot

- tried the version from last week: boots ok

There is an extra message from the run that managed to boot, which is not present with any of the runs that failed:

EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value

But this message is not present with the stable run that I mentioned previously.

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, "CKI Project" <[email protected]>, [email protected]
Sent: Wednesday, February 10, 2021 5:09:37 PM
Subject: Re: ❌ FAIL: Test report for kernel?5.11.0-rc7 (arm-next)

Hi Veronika,

Thanks for the help with this.

Quoted 43 lines · Veronika Kabatova

Quoted 43 lines

Quoted 17 lines · CKI Project

Quoted 7 lines

Host 2:
❌ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ xfstests - btrfs

Quoted 8 lines

🚧 ⚡⚡⚡ IPMI driver test
🚧 ⚡⚡⚡ IPMItool loop stress test
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage block - filesystem fio test 🚧 ⚡⚡⚡ Storage block - queue scheduler test 🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test 🚧 ⚡⚡⚡ stress: stress-ng

Quoted 10 lines

Which system (e.g. soc) is host 2 and are there are known infra issues at
the moment? I did push some changes which affect the early boot path, so we
may well be running into a kernel bug, but I'd just like to make sure before
we dive in trying to debug that, especially as we haven't seen failures on
other systems (and host 1 seems ok).

Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
kernel just a few days back okay. The last messages I can see in the raw console log from this run are:

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

and then it times out after hour and half. I'm not aware of any ongoing issues, however sometimes the link between the lab controller and the machines can sometimes go wrong after reboot and lead to a similarly looking problem.

Quoted 8 lines

Quoted 3 lines

I'll resubmit the test job on that same machine to check if that was the case and let you know right after it boots.

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot

That's odd, as I just received a pass report for that branch!

https://lore.kernel.org/r/[email protected]

Is it just flakey, perhaps? Obviously, that's not great either, but it will make bisection more challenging.

Quoted 5 lines

- tried the version from last week: boots ok

There is an extra message from the run that managed to boot, which is not present with any of the runs that failed:

Quoted 5 lines

EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value

But this message is not present with the stable run that I mentioned previously.

Interesting. Are those messages in the logs anywhere? It would be handy to include them, if possible.

Cheers,

Will

Quoted 6 lines · Veronika Kabatova

Quoted 4 lines

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot

Quoted 7 lines

That's odd, as I just received a pass report for that branch!

https://lore.kernel.org/r/[email protected]

Is it just flakey, perhaps? Obviously, that's not great either, but it will make bisection more challenging.

We have a large number of machines (both physical and virtual) and it's impossible to run all tests on all of them, so they are randomly picked as long as they fit the distro and test requirements. The distribution for ARM tree is 1 physical and 1 "any" machine (which usually ends up being virtual). The jobs from the report you linked ran on different machines and didn't pick the one that failed to boot previously, so I manually forced my testing to pick that machine to eliminate some variables.

The machine in question can on course be somewhat flaky (hard to eliminate that possibility completely), but I checked our historical data and it didn't fail to boot a single time other than with these two new kernels.

Quoted 5 lines

Quoted 5 lines

- tried the version from last week: boots ok

There is an extra message from the run that managed to boot, which is not present with any of the runs that failed:

Quoted 9 lines

Quoted 5 lines

EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value

But this message is not present with the stable run that I mentioned previously.

Interesting. Are those messages in the logs anywhere? It would be handy to include them, if possible.

The messages are from before the kernel boot banner which is the marker we use for log inclusion (to reduce console log spam from distro installation which uses a different kernel and thus makes debugging less straightforward). The same EFI messages are present before the kernel banner in the new report you linked, and with the passing job from the previous runs as well:

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1] 
[    0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021 
[    0.000000] efi: EFI v2.70 by EDK II 
[    0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI 2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18  
[    0.000000] efi: seeding entropy pool 

....

and

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1] 
[    0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021 
[    0.000000] efi: EFI v2.70 by American Megatrends 

....

The failing machine/kernel combos get stuck right after that last EFI line before the kernel messages come in.

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, "CKI Project" <[email protected]>, [email protected]
Sent: Wednesday, February 10, 2021 6:31:34 PM
Subject: Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)

Quoted 15 lines · Veronika Kabatova

Quoted 6 lines · Veronika Kabatova

Quoted 4 lines

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot

Quoted 8 lines

That's odd, as I just received a pass report for that branch!

https://lore.kernel.org/r/[email protected]

Is it just flakey, perhaps? Obviously, that's not great either, but it will
make bisection more challenging.

Quoted 7 lines

We have a large number of machines (both physical and virtual) and it's impossible to run all tests on all of them, so they are randomly picked as long as they fit the distro and test requirements. The distribution for ARM tree is 1 physical and 1 "any" machine (which usually ends up being virtual). The jobs from the report you linked ran on different machines and didn't pick the one that failed to boot previously, so I manually forced my testing to pick that machine to eliminate some variables.

Ah thanks, I hadn't twigged that it was a different set of hosts each time. Makes sense.

Quoted 3 lines

The machine in question can on course be somewhat flaky (hard to eliminate that possibility completely), but I checked our historical data and it didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails on the machine causing us problems. If it does, then the arm64 queue for 5.12 is out of the equation, if not then we can try a targetted bisection.

Would you be able to try v5.11-rc7 please?

Quoted 53 lines

Quoted 10 lines

Quoted 10 lines

- tried the version from last week: boots ok

There is an extra message from the run that managed to boot, which is not
present with any of the runs that failed:

EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value

Quoted 7 lines

Quoted 2 lines

But this message is not present with the stable run that I mentioned previously.

Interesting. Are those messages in the logs anywhere? It would be handy to
include them, if possible.

The messages are from before the kernel boot banner which is the marker we use for log inclusion (to reduce console log spam from distro installation which uses a different kernel and thus makes debugging less straightforward). The same EFI messages are present before the kernel banner in the new report you linked, and with the passing job from the previous runs as well:

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[    0.000000] Linux version 5.11.0-rc7
(cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
(GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI
2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18
[    0.000000] efi: seeding entropy pool

....

and

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[    0.000000] Linux version 5.11.0-rc7
(cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
(GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[    0.000000] efi: EFI v2.70 by American Megatrends

....

Quoted 2 lines

The failing machine/kernel combos get stuck right after that last EFI line before the kernel messages come in.

Sorry, just to be clear here: do we always fail when we have the "KASLR will be disabled" message, or do some machines pass with that?

Thanks again,

Will

----- Original Message -----

Quoted 38 lines

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, "CKI Project" <[email protected]>, [email protected]
Sent: Wednesday, February 10, 2021 6:31:34 PM
Subject: Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)

Quoted 15 lines · Veronika Kabatova

Quoted 6 lines · Veronika Kabatova

Quoted 4 lines

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot

Quoted 8 lines

That's odd, as I just received a pass report for that branch!

https://lore.kernel.org/r/[email protected]

Is it just flakey, perhaps? Obviously, that's not great either, but it will
make bisection more challenging.

Quoted 7 lines

We have a large number of machines (both physical and virtual) and it's impossible to run all tests on all of them, so they are randomly picked as long as they fit the distro and test requirements. The distribution for ARM tree is 1 physical and 1 "any" machine (which usually ends up being virtual). The jobs from the report you linked ran on different machines and didn't pick the one that failed to boot previously, so I manually forced my testing to pick that machine to eliminate some variables.

Ah thanks, I hadn't twigged that it was a different set of hosts each time. Makes sense.

Quoted 3 lines

The machine in question can on course be somewhat flaky (hard to eliminate that possibility completely), but I checked our historical data and it didn't fail to boot a single time other than with these two new kernels.

Quoted 5 lines

So the first thing we should probably try is whether vanilla -rc7 fails on the machine causing us problems. If it does, then the arm64 queue for 5.12 is out of the equation, if not then we can try a targetted bisection.

Would you be able to try v5.11-rc7 please?

Can do.

Someone snatched up the machine in the meanwhile and appears to have it reserved till next Monday :/ I sincerely hope that they'll release it sooner and have queued up the test job with high priority. Will let you know right as I get the results.

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, "CKI Project" <[email protected]>, [email protected]
Sent: Wednesday, February 10, 2021 7:56:03 PM
Subject: Re: ❌ FAIL: Test report?for?kernel?5.11.0-rc7 (arm-next)

Quoted 53 lines · Veronika Kabatova

----- Original Message -----

Quoted 50 lines

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, "CKI Project" <[email protected]>, [email protected]
Sent: Wednesday, February 10, 2021 6:31:34 PM
Subject: Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)

Quoted 16 lines · Veronika Kabatova

Quoted 6 lines · Veronika Kabatova

Quoted 4 lines

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot

Quoted 9 lines

That's odd, as I just received a pass report for that branch!

https://lore.kernel.org/r/[email protected]

Is it just flakey, perhaps? Obviously, that's not great either, but it
will
make bisection more challenging.

Quoted 8 lines

We have a large number of machines (both physical and virtual) and it's impossible to run all tests on all of them, so they are randomly picked as
long as they fit the distro and test requirements. The distribution for ARM tree is 1 physical and 1 "any" machine (which usually ends up being virtual). The jobs from the report you linked ran on different machines and didn't pick the one that failed to boot previously, so I manually forced my testing to pick that machine to eliminate some variables.

Ah thanks, I hadn't twigged that it was a different set of hosts each time.
Makes sense.

Quoted 5 lines

The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.

Quoted 3 lines

Quoted 1 line

Would you be able to try v5.11-rc7 please?

Can do.

Brill, thanks.

Quoted 5 lines

Someone snatched up the machine in the meanwhile and appears to have it reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right as I get the results.

Just tell them the machine is broken and they really don't want to use it for anything important ;)

Will

Quoted 24 lines

Quoted 12 lines

Quoted 12 lines

Quoted 5 lines

The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.

Quoted 3 lines

Quoted 1 line

Would you be able to try v5.11-rc7 please?

Can do.

Brill, thanks.

Quoted 5 lines

Someone snatched up the machine in the meanwhile and appears to have it reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right as I get the results.

Quoted 3 lines

Just tell them the machine is broken and they really don't want to use it for anything important ;)

Our wishes were granted by the lab fairy and the machine was returned rather quickly :)

The 5.11-rc7 kernel boots.

Quoted 28 lines · Veronika Kabatova

Quoted 27 lines

Quoted 12 lines

Quoted 12 lines

Quoted 5 lines

The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.

Quoted 3 lines

Quoted 1 line

Would you be able to try v5.11-rc7 please?

Can do.

Brill, thanks.

Quoted 5 lines

Someone snatched up the machine in the meanwhile and appears to have it reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right as I get the results.

Just tell them the machine is broken and they really don't want to use it for anything important ;)

Quoted 4 lines

Our wishes were granted by the lab fairy and the machine was returned rather quickly :)

The 5.11-rc7 kernel boots.

Fantastic! Then it's something in the arm64 for-next/core tree that was added since then. The diff isn't huge and one change stands out for me, so let me try reverting that and I'll update the branch...

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, [email protected], "CKI Project" <[email protected]>
Sent: Wednesday, February 10, 2021 9:32:06 PM
Subject: Re: ❌ FAIL: Test?report?for?kernel?5.11.0-rc7 (arm-next)

Quoted 36 lines · Will Deacon

Quoted 35 lines · Veronika Kabatova

Quoted 34 lines

Quoted 16 lines

Quoted 16 lines

Quoted 6 lines

The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails
on
the machine causing us problems. If it does, then the arm64 queue for
5.12
is out of the equation, if not then we can try a targetted bisection.

Quoted 3 lines

Quoted 1 line

Would you be able to try v5.11-rc7 please?

Can do.

Brill, thanks.

Quoted 7 lines

Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.

Just tell them the machine is broken and they really don't want to use it
for anything important ;)

Quoted 8 lines

Quoted 4 lines

Our wishes were granted by the lab fairy and the machine was returned rather quickly :)

The 5.11-rc7 kernel boots.

Fantastic! Then it's something in the arm64 for-next/core tree that was added since then. The diff isn't huge and one change stands out for me, so let me try reverting that and I'll update the branch...

Ok, I updated for-kernelci so that it contains the arm64 for-next/core branch merged into -rc7, but with a couple of patches reverted on top.

HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?

Cheers,

Will

Quoted 49 lines · Will Deacon

Quoted 36 lines · Veronika Kabatova

Quoted 35 lines

Quoted 34 lines

Quoted 16 lines

Quoted 16 lines

Quoted 6 lines

The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails
on
the machine causing us problems. If it does, then the arm64 queue for
5.12
is out of the equation, if not then we can try a targetted bisection.

Quoted 3 lines

Quoted 1 line

Would you be able to try v5.11-rc7 please?

Can do.

Brill, thanks.

Quoted 7 lines

Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.

Just tell them the machine is broken and they really don't want to use it
for anything important ;)

Quoted 8 lines

Quoted 4 lines

Our wishes were granted by the lab fairy and the machine was returned rather quickly :)

The 5.11-rc7 kernel boots.

Fantastic! Then it's something in the arm64 for-next/core tree that was added since then. The diff isn't huge and one change stands out for me, so let me try reverting that and I'll update the branch...

Ok, I updated for-kernelci so that it contains the arm64 for-next/core branch merged into -rc7, but with a couple of patches reverted on top.

Quoted 3 lines

HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?

I didn't actually have to, the autopick picked the machine for the testing and it happily booted :) \o/

Apparently IT cut off our email sending access because we're sending too many emails so that's why you don't have the report yet. When we work around it I'll make sure to retrigger the sending so you have it.

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, [email protected], "CKI Project" <[email protected]>
Sent: Thursday, February 11, 2021 12:50:50 PM
Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)

Quoted 63 lines · Veronika Kabatova

Quoted 62 lines · Will Deacon

Quoted 48 lines · Veronika Kabatova

Quoted 47 lines

Quoted 38 lines

Quoted 21 lines

Quoted 21 lines

Quoted 9 lines

The machine in question can on course be somewhat flaky (hard to
eliminate
that possibility completely), but I checked our historical data
and it
didn't fail to boot a single time other than with these two new
kernels.

So the first thing we should probably try is whether vanilla -rc7
fails
on
the machine causing us problems. If it does, then the arm64 queue
for
5.12
is out of the equation, if not then we can try a targetted bisection.

Quoted 3 lines

Quoted 1 line

Would you be able to try v5.11-rc7 please?

Can do.

Brill, thanks.

Quoted 10 lines

Someone snatched up the machine in the meanwhile and appears to have
it
reserved till next Monday :/ I sincerely hope that they'll release it
sooner
and have queued up the test job with high priority. Will let you know
right
as I get the results.

Quoted 5 lines

Just tell them the machine is broken and they really don't want to use
it
for anything important ;)

Our wishes were granted by the lab fairy and the machine was returned rather quickly :)

Quoted 6 lines

Quoted 1 line

The 5.11-rc7 kernel boots.

Fantastic! Then it's something in the arm64 for-next/core tree that was added since then. The diff isn't huge and one change stands out for me, so
let me try reverting that and I'll update the branch...

Ok, I updated for-kernelci so that it contains the arm64 for-next/core branch merged into -rc7, but with a couple of patches reverted on top.

HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?

Quoted 2 lines

I didn't actually have to, the autopick picked the machine for the testing and it happily booted :) \o/

Phew, so I'll drop those from linux-next as well. Do you know if "earlycon" works on the problematic machine? If possible, it would be helpful to try booting the bad kernel with earlycon on the cmdline to see if it manages to say anything in its dying breath.

Quoted 3 lines

Apparently IT cut off our email sending access because we're sending too many emails so that's why you don't have the report yet. When we work around it I'll make sure to retrigger the sending so you have it.

That's nice of them...

Will

From: "Veronika Kabatova" <[email protected]>
To: "Will Deacon" <[email protected]>
Cc: "catalin marinas" <[email protected]>, [email protected], "CKI Project" <[email protected]>
Sent: Thursday, February 11, 2021 1:25:34 PM
Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)

----- Original Message -----

Quoted 79 lines

From: "Will Deacon" <[email protected]>
To: "Veronika Kabatova" <[email protected]>
Cc: "catalin marinas" <[email protected]>, [email protected], "CKI Project" <[email protected]>
Sent: Thursday, February 11, 2021 12:50:50 PM
Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)

Quoted 70 lines · Veronika Kabatova

Quoted 69 lines · Will Deacon

Quoted 52 lines · Veronika Kabatova

Quoted 51 lines

Quoted 40 lines

Quoted 22 lines

Quoted 22 lines

Quoted 10 lines

The machine in question can on course be somewhat flaky (hard
to
eliminate
that possibility completely), but I checked our historical data
and it
didn't fail to boot a single time other than with these two new
kernels.

So the first thing we should probably try is whether vanilla -rc7
fails
on
the machine causing us problems. If it does, then the arm64 queue
for
5.12
is out of the equation, if not then we can try a targetted bisection.

Quoted 3 lines

Quoted 1 line

Would you be able to try v5.11-rc7 please?

Can do.

Brill, thanks.

Quoted 11 lines

Someone snatched up the machine in the meanwhile and appears to have
it
reserved till next Monday :/ I sincerely hope that they'll release it
sooner
and have queued up the test job with high priority. Will let you
know
right
as I get the results.

Quoted 6 lines

Just tell them the machine is broken and they really don't want to
use
it
for anything important ;)

Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)

Quoted 8 lines

Quoted 1 line

The 5.11-rc7 kernel boots.

Fantastic! Then it's something in the arm64 for-next/core tree that was
added since then. The diff isn't huge and one change stands out for me,
so
let me try reverting that and I'll update the branch...

Ok, I updated for-kernelci so that it contains the arm64 for-next/core branch merged into -rc7, but with a couple of patches reverted on top.

HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for
ZONE_DEVICE
based memory""). Please can you try that?

Quoted 9 lines

Quoted 3 lines

I didn't actually have to, the autopick picked the machine for the testing
and it happily booted :) \o/

Phew, so I'll drop those from linux-next as well. Do you know if "earlycon" works on the problematic machine? If possible, it would be helpful to try booting the bad kernel with earlycon on the cmdline to see if it manages to say anything in its dying breath.

No idea, we don't have the option enabled and I can't find any information about it. I submitted a new job with the option and let's see whether it works or not. The machine is reserved again so there may be some delays.

Quoted 6 lines

Quoted 3 lines

Apparently IT cut off our email sending access because we're sending too many emails so that's why you don't have the report yet. When we work around it I'll make sure to retrigger the sending so you have it.

That's nice of them...

Update for the drama loving folks: IT is actually innocent in this. Someone deployed a misconfigured application in the same cluster as our reporting system is in, and this application is sending out emails every 3 seconds. We're collateral damage. Working on resolution, but right now it seems that the situation is under control.

Veronika

Quoted 3 lines

Will

Quoted 21 lines

Quoted 20 lines · Veronika Kabatova

Quoted 5 lines

Quoted 5 lines · Will Deacon

Quoted 1 line · Veronika Kabatova

Ok, I updated for-kernelci so that it contains the arm64 for-next/core branch merged into -rc7, but with a couple of patches reverted on top.

Quoted 8 lines

Quoted 4 lines

HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for
ZONE_DEVICE
based memory""). Please can you try that?

I didn't actually have to, the autopick picked the machine for the testing
and it happily booted :) \o/

Phew, so I'll drop those from linux-next as well. Do you know if "earlycon" works on the problematic machine? If possible, it would be helpful to try booting the bad kernel with earlycon on the cmdline to see if it manages to say anything in its dying breath.

Quoted 4 lines

No idea, we don't have the option enabled and I can't find any information about it. I submitted a new job with the option and let's see whether it works or not. The machine is reserved again so there may be some delays.

I finally got access to the machine and don't have any good news. The hang is from before early printk takes effect. The output is identical to what I sent before, with nothing resembling kernel lines in there.