KernelCI reports

Hi!

First... what I am currently using for testing: gitlab; I simply watch for green ticks there. It takes ~15 minutes to get results. Red cross in compile stage is pretty sure sign of build error. Red cross in testing means I need to look at the results, but it is pretty easy to see "network timeout" or "board unavailable" problems, and mostly I can just hit retry and get the green tick.

What wories me about gitlab is that we have single board for 4.4-st, and just two for 4.4-cip.

Now, kernelci.

https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html

I got this link in the irc meeting:

https://linux.kernelci.org/job/cip/

Lets start with build results.

linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3

I click on "3" and get build error summary:

Errors Summary
2 arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
2 arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

...but those are not kernel problems, that's buildsystem incompatible with config or something like that. Nothing for me to solve.

Let me go through more:

linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120

That's 19 errors total + outlier of v4.19.239-cip72 with 120 errors. None of the 19 errors is real. KernelCI needs to fix their configs/build environment.

120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as

cm_x2xx_defconfig ‐ arm ‐ gcc-10 0 warnings 0 errors. When I click for details I get "No test results found".

https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/

Nothing for me to debug, either, agreed? So lets look at test results.

Branch Kernel Commit Build Status Test Results Date
linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3 2457 256 21 2022-06-12

21 failures.

If I click that, I can see that most regressions (11) are at baseline test plan. The machine that looks relevant to us is qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --

https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/

New regression, last pass: v5.10.112-cip6

I get to
https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
which looks quite useful, it lists successful run and failed run, v5.10.121-cip9 fails while v5.10.112-cip6 works.

Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .

And sorry, no I don't see anything I could debug there. Did the docker fail to start? Hmm. Lets compare it with the older results.

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.

Lets try one more.

rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10

https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/

It gets me to:

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

I see some errors... Like missing firmware (not a kernel problem). At the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?

1368 12:21:39.212881 /usr/bin/tpm2_getcap
1369 12:21:39.251829 ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
1370 12:21:39.261387 ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error

    1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
     1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
      1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap

Userland seems to say that it does not support this device. I guess I could compare it with older boot?

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

Aha, no, could not, 404.

Now, let's go through Jan's
example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
. Again, errors look like missconfigured build system, with

arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.

I spent quite some time trying to click around to find an error I could debug. I could not find one. All of above looks like kernelCI configuration problems. Can they be fixed and/or filtered out?

Hi Pavel,

Quoted 65 lines · Pavel Machek

Hi!

First... what I am currently using for testing: gitlab; I simply watch for green ticks there. It takes ~15 minutes to get results. Red cross in compile stage is pretty sure sign of build error. Red cross in testing means I need to look at the results, but it is pretty easy to see "network timeout" or "board unavailable" problems, and mostly I can just hit retry and get the green tick.

What wories me about gitlab is that we have single board for 4.4-st, and just two for 4.4-cip.

Now, kernelci.

https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html

I got this link in the irc meeting:

https://linux.kernelci.org/job/cip/

Lets start with build results.

linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3

I click on "3" and get build error summary:

Errors Summary
2 arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
2 arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

...but those are not kernel problems, that's buildsystem incompatible with config or something like that. Nothing for me to solve.

Let me go through more:

linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120

That's 19 errors total + outlier of v4.19.239-cip72 with 120 errors. None of the 19 errors is real. KernelCI needs to fix their configs/build environment.

120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as

cm_x2xx_defconfig ‐ arm ‐ gcc-10 0 warnings 0 errors. When I click for details I get "No test results found".

https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/

Nothing for me to debug, either, agreed? So lets look at test results.

Branch Kernel Commit Build Status Test Results Date
linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3 2457 256 21 2022-06-12

21 failures.

If I click that, I can see that most regressions (11) are at baseline test plan. The machine that looks relevant to us is qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --

Quoted 8 lines

https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/

New regression, last pass: v5.10.112-cip6

I get to
https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
which looks quite useful, it lists successful run and failed run, v5.10.121-cip9 fails while v5.10.112-cip6 works.

Yes, checking for regressions is quite useful.

Quoted 7 lines

Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .

And sorry, no I don't see anything I could debug there. Did the docker fail to start? Hmm. Lets compare it with the older results.

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.

The garbage collector removes everything that's older than 28 days.

Quoted 21 lines

Lets try one more.

rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10

https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/

It gets me to:

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

I see some errors... Like missing firmware (not a kernel problem). At the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?

1368 12:21:39.212881 /usr/bin/tpm2_getcap
1369 12:21:39.251829 ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
1370 12:21:39.261387 ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error

    1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
     1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
      1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap

Quoted 5 lines

Userland seems to say that it does not support this device. I guess I could compare it with older boot?

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

Aha, no, could not, 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 9 lines

Now, let's go through Jan's
example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
. Again, errors look like missconfigured build system, with

arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 5 lines

I spent quite some time trying to click around to find an error I could debug. I could not find one. All of above looks like kernelCI configuration problems. Can they be fixed and/or filtered out?

If you can write which configurations you want to filter out (because they are not needed for CIP)
to KernelCI, KernelCI could blocklist such configurations.

for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#8754): https://lists.cip-project.org/g/cip-dev/message/8754
Mute This Topic: https://lists.cip-project.org/mt/91949361/5792637
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/9882880/5792637/2090341516/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Quoted 136 lines · Alice Ferrazzi via lists.cip- project.org

Hi Pavel,

Quoted 65 lines · Pavel Machek

Hi!

First... what I am currently using for testing: gitlab; I simply watch for green ticks there. It takes ~15 minutes to get results. Red cross in compile stage is pretty sure sign of build error. Red cross in testing means I need to look at the results, but it is pretty easy to see "network timeout" or "board unavailable" problems, and mostly I can just hit retry and get the green tick.

What wories me about gitlab is that we have single board for 4.4-st, and just two for 4.4-cip.

Now, kernelci.

https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html

I got this link in the irc meeting:

https://linux.kernelci.org/job/cip/

Lets start with build results.

linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3

I click on "3" and get build error summary:

Errors Summary
2 arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
2 arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

...but those are not kernel problems, that's buildsystem incompatible with config or something like that. Nothing for me to solve.

Let me go through more:

linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120

That's 19 errors total + outlier of v4.19.239-cip72 with 120 errors. None of the 19 errors is real. KernelCI needs to fix their configs/build environment.

120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as

cm_x2xx_defconfig ‐ arm ‐ gcc-10 0 warnings 0 errors. When I click for details I get "No test results found".

https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/

Nothing for me to debug, either, agreed? So lets look at test results.

Branch Kernel Commit Build Status Test Results Date
linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3 2457 256 21 2022-06-12

21 failures.

If I click that, I can see that most regressions (11) are at baseline test plan. The machine that looks relevant to us is qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --

Quoted 8 lines

https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/

New regression, last pass: v5.10.112-cip6

I get to
https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
which looks quite useful, it lists successful run and failed run, v5.10.121-cip9 fails while v5.10.112-cip6 works.

Yes, checking for regressions is quite useful.

Quoted 7 lines

Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .

And sorry, no I don't see anything I could debug there. Did the docker fail to start? Hmm. Lets compare it with the older results.

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.

The garbage collector removes everything that's older than 28 days.

Quoted 21 lines

Lets try one more.

rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10

https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/

It gets me to:

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

I see some errors... Like missing firmware (not a kernel problem). At the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?

1368 12:21:39.212881 /usr/bin/tpm2_getcap
1369 12:21:39.251829 ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
1370 12:21:39.261387 ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error

    1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
     1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
      1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap

Quoted 5 lines

Userland seems to say that it does not support this device. I guess I could compare it with older boot?

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

Aha, no, could not, 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 9 lines

Now, let's go through Jan's
example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
. Again, errors look like missconfigured build system, with

arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 5 lines

I spent quite some time trying to click around to find an error I could debug. I could not find one. All of above looks like kernelCI configuration problems. Can they be fixed and/or filtered out?

Quoted 5 lines

If you can write which configurations you want to filter out (because they are not needed for CIP)
to KernelCI, KernelCI could blocklist such configurations.

for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725

Jan asked me to check the kernelci related issues you summarized here related to the 4.4-cip branch. I tried to find the latest 4.4 test report that failed. Can someone point me there?

With some guidance from Alice it should be possible for me to come up with a exclude/blocklist to bring 4.4 back to a "green" state. Afterwards we would have to sort out all the problems (decoupled from Pavel's testing)

Quoted 145 lines · On Thu, Jul 14, 2022 at 6:45 PM [email protected]…

Quoted 136 lines · Alice Ferrazzi via lists.cip- project.org

Hi Pavel,

Quoted 65 lines · Pavel Machek

Hi!

First... what I am currently using for testing: gitlab; I simply watch for green ticks there. It takes ~15 minutes to get results. Red cross in compile stage is pretty sure sign of build error. Red cross in testing means I need to look at the results, but it is pretty easy to see "network timeout" or "board unavailable" problems, and mostly I can just hit retry and get the green tick.

What wories me about gitlab is that we have single board for 4.4-st, and just two for 4.4-cip.

Now, kernelci.

https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html

I got this link in the irc meeting:

https://linux.kernelci.org/job/cip/

Lets start with build results.

linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3

I click on "3" and get build error summary:

Errors Summary
2 arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
2 arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

...but those are not kernel problems, that's buildsystem incompatible with config or something like that. Nothing for me to solve.

Let me go through more:

linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120

That's 19 errors total + outlier of v4.19.239-cip72 with 120 errors. None of the 19 errors is real. KernelCI needs to fix their configs/build environment.

120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as

cm_x2xx_defconfig ‐ arm ‐ gcc-10 0 warnings 0 errors. When I click for details I get "No test results found".

https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/

Nothing for me to debug, either, agreed? So lets look at test results.

Branch Kernel Commit Build Status Test Results Date
linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3 2457 256 21 2022-06-12

21 failures.

If I click that, I can see that most regressions (11) are at baseline test plan. The machine that looks relevant to us is qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --

Quoted 8 lines

https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/

New regression, last pass: v5.10.112-cip6

I get to
https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
which looks quite useful, it lists successful run and failed run, v5.10.121-cip9 fails while v5.10.112-cip6 works.

Yes, checking for regressions is quite useful.

Quoted 7 lines

Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .

And sorry, no I don't see anything I could debug there. Did the docker fail to start? Hmm. Lets compare it with the older results.

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.

The garbage collector removes everything that's older than 28 days.

Quoted 21 lines

Lets try one more.

rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10

https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/

It gets me to:

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

I see some errors... Like missing firmware (not a kernel problem). At the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?

1368 12:21:39.212881 /usr/bin/tpm2_getcap
1369 12:21:39.251829 ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
1370 12:21:39.261387 ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error

    1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
     1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
      1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap

Quoted 5 lines

Userland seems to say that it does not support this device. I guess I could compare it with older boot?

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

Aha, no, could not, 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 9 lines

Now, let's go through Jan's
example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
. Again, errors look like missconfigured build system, with

arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 5 lines

I spent quite some time trying to click around to find an error I could debug. I could not find one. All of above looks like kernelCI configuration problems. Can they be fixed and/or filtered out?

Quoted 5 lines

If you can write which configurations you want to filter out (because they are not needed for CIP)
to KernelCI, KernelCI could blocklist such configurations.

for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725

Quoted 9 lines

Jan asked me to check the kernelci related issues you summarized here related to the 4.4-cip branch. I tried to find the latest 4.4 test report that failed. Can someone point me there?

With some guidance from Alice it should be possible for me to come up with a exclude/blocklist to bring 4.4 back to a "green" state. Afterwards we would have to sort out all the problems (decoupled from Pavel's testing)

Hello Florian

Latest build 4.4 cip is here
https://linux.kernelci.org/test/job/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip69-296-g52de9431d314/
is a really old build as the latest 4.4 cip tag is v4.4.302-cip69 tagged on date 2022-04-04 13:18:30 +0900
Unfortunately as I previously said KernelCI garbage collector removes everything that's older than 28 days.
You can find CIP builds here: https://linux.kernelci.org/job/cip/

Thanks,
Alicef

Quoted 159 lines · Alice Ferrazzi

Quoted 145 lines · On Thu, Jul 14, 2022 at 6:45 PM [email protected]…

Quoted 136 lines · Alice Ferrazzi via lists.cip- project.org

Hi Pavel,

Quoted 65 lines · Pavel Machek

Hi!

First... what I am currently using for testing: gitlab; I simply watch for green ticks there. It takes ~15 minutes to get results. Red cross in compile stage is pretty sure sign of build error. Red cross in testing means I need to look at the results, but it is pretty easy to see "network timeout" or "board unavailable" problems, and mostly I can just hit retry and get the green tick.

What wories me about gitlab is that we have single board for 4.4-st, and just two for 4.4-cip.

Now, kernelci.

https://ircbot.wl.linuxfoundation.org/meetings/cip/2022/06/cip.2022-06-23-12.01.html

I got this link in the irc meeting:

https://linux.kernelci.org/job/cip/

Lets start with build results.

linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3

I click on "3" and get build error summary:

Errors Summary
2 arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3m
2 arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

...but those are not kernel problems, that's buildsystem incompatible with config or something like that. Nothing for me to solve.

Let me go through more:

linux-4.19.y-cip        v4.19.246-cip75 38ce181ac6ca9f3decdd389cbfd0a38fecdb28c7  111 8 1
linux-4.19.y-cip        v4.19.245-cip74 0be9d6774be9b192bcea6fddada917ae3d22f22e  106 7 1
linux-4.19.y-cip-rt     v4.19.245-cip74-rt25 c80ee3077a4ce344cf1cffa23af767c9fd75d52c  181 16 6
linux-4.4.y-cip v4.4.302-cip69-296-g52de9431d314 52de9431d314a28c4aa66d8f81144c9958c11168 164 18 5
linux-5.10.y-cip-rt     v5.10.109-cip5-rt4 b41186882eaf0137277045da13ea649577c42c11 177 8 3
linux-5.10.y-cip        v5.10.112-cip6 c8f6747dcd02a8f87731c324f4bc37bd120a8982 180 7 3
linux-4.19.y-cip        v4.19.239-cip72 fdf770b2db8080de95dcf74cc7547319034a4a56 0 0 120

That's 19 errors total + outlier of v4.19.239-cip72 with 120 errors. None of the 19 errors is real. KernelCI needs to fix their configs/build environment.

120 errors are reported at https://linux.kernelci.org/build/cip/branch/linux-4.19.y-cip/kernel/v4.19.239-cip72/ . Wow. Except that I don't see that many errors there. I see lines such as

cm_x2xx_defconfig ‐ arm ‐ gcc-10 0 warnings 0 errors. When I click for details I get "No test results found".

https://linux.kernelci.org/build/id/6266a495d40dfd0dacff948c/

Nothing for me to debug, either, agreed? So lets look at test results.

Branch Kernel Commit Build Status Test Results Date
linux-5.10.y-cip v5.10.121-cip9 bd24696dd7ad23517733728da675a00cd3073916 169 7 3 2457 256 21 2022-06-12

21 failures.

If I click that, I can see that most regressions (11) are at baseline test plan. The machine that looks relevant to us is qemu_arm64-virt-gicv2 defconfig+arm64-chromebook --

Quoted 8 lines

https://linux.kernelci.org/test/job/cip/branch/linux-5.10.y-cip/kernel/v5.10.121-cip9/plan/baseline/

New regression, last pass: v5.10.112-cip6

I get to
https://linux.kernelci.org/test/case/id/62a333452713980006a39c13/
which looks quite useful, it lists successful run and failed run, v5.10.121-cip9 fails while v5.10.112-cip6 works.

Yes, checking for regressions is quite useful.

Quoted 7 lines

Eventually I get to https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html .

And sorry, no I don't see anything I could debug there. Did the docker fail to start? Hmm. Lets compare it with the older results.

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.112-cip6/arm64/defconfig+arm64-chromebook/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2.html . Aha, 404. No comparison possible.

The garbage collector removes everything that's older than 28 days.

Quoted 21 lines

Lets try one more.

rk3399-gru-kevin defconfig+arm64-chromebook ‐ arm64 ‐ gcc-10

https://linux.kernelci.org/test/plan/id/62a337700aadba677da39bdd/

It gets me to:

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.121-cip9/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

I see some errors... Like missing firmware (not a kernel problem). At the end there's "tpm-chip-is-online: fail" . Is that what I should be debugging?

1368 12:21:39.212881 /usr/bin/tpm2_getcap
1369 12:21:39.251829 ERROR:sys:src/tss2-sys/api/Tss2_Sys_Execute.c:114:Tss2_Sys_ExecuteFinish() Unsupported device. The device is a TPM 1.2
1370 12:21:39.261387 ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:307:Esys_GetCapability_Finish() Received a non-TPM Error

    1371 12:21:39.272254  ERROR:esys:src/tss2-esys/api/Esys_GetCapability.c:107:Esys_GetCapability() Esys Finish ErrorCode (0x00080001)
     1372 12:21:39.281854  ERROR: Esys_GetCapability(0x80001) - sys:Catch all for all errors not otherwise specified
      1373 12:21:39.285316  ERROR: Unable to run tpm2_getcap

Quoted 5 lines

Userland seems to say that it does not support this device. I guess I could compare it with older boot?

https://storage.kernelci.org/cip/linux-5.10.y-cip/v5.10.83-cip1-178-g2cf1d12aab81/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-rk3399-gru-kevin.html

Aha, no, could not, 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 9 lines

Now, let's go through Jan's
example. https://linux.kernelci.org/build/id/61fbabe0cb6f27a6ce5d6f03/logs/?
. Again, errors look like missconfigured build system, with

arm-linux-gnueabihf-gcc: error: unrecognized -march target: armv3
arm-linux-gnueabihf-gcc: error: missing argument to ‘-march=’

Full log https://storage.kernelci.org/stable-rc/linux-4.4.y/v4.4.302/arm/rpc_defconfig/gcc-10/logs gives 404.

same here, the garbage collector removes everything that's older than 28 days.

Quoted 5 lines

I spent quite some time trying to click around to find an error I could debug. I could not find one. All of above looks like kernelCI configuration problems. Can they be fixed and/or filtered out?

Quoted 5 lines

If you can write which configurations you want to filter out (because they are not needed for CIP)
to KernelCI, KernelCI could blocklist such configurations.

for reference: https://github.com/kernelci/kernelci-core/issues/1053#issuecomment-1050679725

Quoted 9 lines

Jan asked me to check the kernelci related issues you summarized here related to the 4.4-cip branch. I tried to find the latest 4.4 test report that failed. Can someone point me there?

With some guidance from Alice it should be possible for me to come up with a exclude/blocklist to bring 4.4 back to a "green" state. Afterwards we would have to sort out all the problems (decoupled from Pavel's testing)

Hello Florian

Quoted 7 lines

Latest build 4.4 cip is here
https://linux.kernelci.org/test/job/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip69-296-g52de9431d314/
is a really old build as the latest 4.4 cip tag is v4.4.302-cip69 tagged on date 2022-04-04 13:18:30 +0900
Unfortunately as I previously said KernelCI garbage collector removes everything that's older than 28 days.
You can find CIP builds here: https://linux.kernelci.org/job/cip/

Thanks a lot Alice!

Any chance for me to trigger a re-build?