Skip to contentSkip to conversation

gitlab test failures -- job failed ... pods not found

DISCUSSION Pavel Machek 3 participants 12 messages Sep 30, 2022 – Oct 7, 2022 2 messages not in the archive

lore
Share & tools
lore

Pavel Machek

author
#0 lore raw

Hi!

I have test failures all over the place:

-----------------
44Creating test job
45-----------------
46Version: Image_ctj_zynqmp_defconfig_5.10.145-cip17-rt7_411cd76b5 47Arch: arm64
48Config: ctj_zynqmp_defconfig
49Device: zynqmp-zcu102
50Kernel: Image
51DTB: zynqmp-zcu102-rev1.0.dtb
52Modules: N/A
53Test: smc
54------------------
55Uploading binaries
56------------------
58
Uploading artifacts for failed job
00:00
60
Cleaning up project directory and file based variables 00:00
62ERROR: Job failed (system failure): pods "runner-bwzp7ahx-project-2678032-concurrent-0pstk2" not found

Initially everything failed, so I resubmitted everything, then three succeeded but no more luck with resubmitting.

Example is
https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3109431179,
I'm trying to test 5.10-rt release candidate
https://gitlab.com/cip-project/cip-kernel/linux-cip/-/pipelines/654967302
.

Any ideas?

Best regards,
Pavel

Signature · 2 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#1 lore raw

Hello Pavel,

Quoted 30 lines

From: Pavel Machek <[email protected]>
Sent: 30 September 2022 11:51

Hi!

I have test failures all over the place:

-----------------
44Creating test job
45-----------------
46Version: Image_ctj_zynqmp_defconfig_5.10.145-cip17-rt7_411cd76b5 47Arch: arm64
48Config: ctj_zynqmp_defconfig
49Device: zynqmp-zcu102
50Kernel: Image
51DTB: zynqmp-zcu102-rev1.0.dtb
52Modules: N/A
53Test: smc
54------------------
55Uploading binaries
56------------------
58
Uploading artifacts for failed job
00:00
60
Cleaning up project directory and file based variables 00:00
62ERROR: Job failed (system failure): pods "runner-bwzp7ahx-project-2678032-concurrent-0pstk2" not found

Quoted 2 lines

Initially everything failed, so I resubmitted everything, then three succeeded but no more luck with resubmitting.

Thank you for reporting.
We recently upgraded the gitlab runner versions, but that was last week so it's a bit strange we're seeing issues now. I've resubmitted a few of the jobs and as you say, some seem to work, others don't.

@Adler, Michael, could you take a look when you get a chance? Could this be caused by the recent upgrade?

Thanks, Chris

Quoted 14 lines

Example is
https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3109431179,
I'm trying to test 5.10-rt release candidate
https://gitlab.com/cip-project/cip-kernel/linux-cip/-/pipelines/654967302
.

Any ideas?

Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

Hi Pavel,

Quoted 39 lines

From: [email protected] <[email protected]> On Behalf Of Chris Paterson via lists.cip-project.org
Sent: 30 September 2022 12:37

Hello Pavel,

Quoted 30 lines

From: Pavel Machek <[email protected]>
Sent: 30 September 2022 11:51

Hi!

I have test failures all over the place:

-----------------
44Creating test job
45-----------------
46Version: Image_ctj_zynqmp_defconfig_5.10.145-cip17-rt7_411cd76b5 47Arch: arm64
48Config: ctj_zynqmp_defconfig
49Device: zynqmp-zcu102
50Kernel: Image
51DTB: zynqmp-zcu102-rev1.0.dtb
52Modules: N/A
53Test: smc
54------------------
55Uploading binaries
56------------------
58
Uploading artifacts for failed job
00:00
60
Cleaning up project directory and file based variables 00:00
62ERROR: Job failed (system failure): pods "runner-bwzp7ahx-project-2678032-concurrent-0pstk2" not found

Quoted 2 lines

Initially everything failed, so I resubmitted everything, then three succeeded but no more luck with resubmitting.

Quoted 8 lines

Thank you for reporting.
We recently upgraded the gitlab runner versions, but that was last week so it's a bit strange we're seeing issues now.
I've resubmitted a few of the jobs and as you say, some seem to work, others don't.

@Adler, Michael, could you take a look when you get a chance? Could this be caused by the recent upgrade?

FYI if you need some test results sooner, kernelci's test results on the same code are here:
https://linux.kernelci.org/test/job/cip-gitlab/branch/ci%2Fpavel%2Flinux-test/kernel/v5.10.145-cip17-422-g411cd76b5afe8/

The history of all builds/tests for your linux-test branch can be seen here:
https://linux.kernelci.org/job/cip-gitlab/branch/ci%2Fpavel%2Flinux-test/

Or results from all of our "CI" branches here:
https://linux.kernelci.org/job/cip-gitlab/

Kind regards, Chris

Quoted 34 lines

Thanks, Chris

Quoted 3 lines

Example is

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
.com%2Fcip-project%2Fcip-kernel%2Flinux-cip%2F- %2Fjobs%2F3109431179&amp;data=05%7C01%7Cchris.paterson2%40renesas .com%7C71678297f9c24981513908daa2d81661%7C53d82571da1947e49cb4625 a166a4a2a%7C0%7C0%7C638001346214414952%7CUnknown%7CTWFpbGZsb 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 %3D%7C3000%7C%7C%7C&amp;sdata=3ABojWTs1UqFfSh%2F2ZkYPvcPX2pk 6zuOYQYJYGGm8BU%3D&amp;reserved=0,

Quoted 2 lines

I'm trying to test 5.10-rt release candidate

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
.com%2Fcip-project%2Fcip-kernel%2Flinux-cip%2F- %2Fpipelines%2F654967302&amp;data=05%7C01%7Cchris.paterson2%40ren esas.com%7C71678297f9c24981513908daa2d81661%7C53d82571da1947e49cb 4625a166a4a2a%7C0%7C0%7C638001346214414952%7CUnknown%7CTWFpb GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=q9behdYj1bp38YqpKXWk8nvt9 UEvbpSnF7nC92W5kTk%3D&amp;reserved=0

Quoted 9 lines

.

Any ideas?

Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

#4 lore raw

Hi all,

Quoted 2 lines
Quoted 2 lines

@Adler, Michael, could you take a look when you get a chance? Could this be caused by the recent upgrade?

not sure. For now, I have downgraded the Gitlab runner to the previous version. Please try again and let me know how that works out.

Kind Regards,
Michael

Signature · 8 lines
Michael Adler

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
author
#5 lore raw

Hi!

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

@Adler, Michael, could you take a look when you get a chance? Could this be caused by the recent upgrade?

not sure. For now, I have downgraded the Gitlab runner to the previous version. Please try again and let me know how that works out.

Still there, I'd say:

https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3123103311

94upload: output/Image_renesas_defconfig_5.10.145-cip17-rt6_0d804ef4a/arm64/renesas_defconfig/dtb/rcar_du_of_lvds_r8a7796.dtb to s3://download2.cip-project.org/cip-testing/linux-cip/Image_renesas_defconfig_5.10.145-cip17-rt6_0d804ef4a/arm64/renesas_defconfig/dtb/rcar_du_of_lvds_r8a7796.dtb 95Completed 10.3 MiB/~22.2 MiB (2.0 MiB/s) with ~1 file(s) remaining (calculating...)
96Completed 15.2 MiB/~22.2 MiB (2.7 MiB/s) with ~1 file(s) remaining (calculating...)
97Completed 22.2 MiB/~22.2 MiB (3.4 MiB/s) with ~1 file(s) remaining (calculating...)
98upload: output/Image_renesas_defconfig_5.10.145-cip17-rt6_0d804ef4a/arm64/renesas_defconfig/kernel/Image to s3://download2.cip-project.org/cip-testing/linux-cip/Image_renesas_defconfig_5.10.145-cip17-rt6_0d804ef4a/arm64/renesas_defconfig/kernel/Image 99Completed 22.2 MiB/~22.2 MiB (3.4 MiB/s) with ~0 file(s) remaining (calculating...)
101
Uploading artifacts for failed job
00:00
103
Cleaning up project directory and file based variables
00:00
105ERROR: Job failed (system failure): pods
"runner-kmcmucmn-project-2678032-concurrent-4jcrtw" not found

https://gitlab.com/cip-project/cip-kernel/linux-cip/-/pipelines/657708852

Best regards,
Pavel

Signature · 2 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#6 lore raw
Quoted 3 lines

Still there, I'd say:

https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3123103311

Ok, I have restored our previous cluster (v9). Please try again and tell me if the problem still persists. If it does, then I'd blame Gitlab I/O otherwise it could be a regression somewhere in the K8s stack - in that case, our best bet might be to sit it out.

Kind Regards,
Michael

Signature · 8 lines
Michael Adler

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
#7 lore raw

Hi Michael,

Quoted 15 lines

From: Michael Adler <[email protected]>
Sent: 07 October 2022 07:58

Quoted 3 lines

Still there, I'd say:

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
.com%2Fcip-project%2Fcip-kernel%2Flinux-cip%2F- %2Fjobs%2F3123103311&amp;data=05%7C01%7CChris.Paterson2%40renesa s.com%7C6f9f909bd6244ce82bc908daa83148bd%7C53d82571da1947e49cb462 5a166a4a2a%7C0%7C0%7C638007226871934080%7CUnknown%7CTWFpbGZs b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn 0%3D%7C3000%7C%7C%7C&amp;sdata=zxHVVXOY0Xdh4qmQMVWTK27n47 NeaESoLEaW2ho9Eyw%3D&amp;reserved=0

Quoted 2 lines

Ok, I have restored our previous cluster (v9). Please try again and tell me if the problem still persists. If it does,

I haven't seen the issue Pavel reported yet, but it'll take time to be sure.

The issue I was having building docker containers for linux-cip-ci _is_ still present though :( The jobs get stuck on "Cleaning up project directory and file based variables" for some reason. e.g. https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/jobs/3140519507

From some debugging it looks like this is only happening when we use a kaniko container. I've tried a handful of versions in case kaniko has broken something recently.
We use kankio for building the docker containers as dind builds using docker weren't working in our setup.

Quoted 3 lines

then I'd blame Gitlab I/O otherwise it could be a regression somewhere in the K8s stack - in that case, our best bet
might be to sit it out.

We need to find some sort of solution for building our CI build containers.

Kind regards, Chris

Quoted 17 lines

Kind Regards,
Michael

--
Michael Adler

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322

#8 lore raw
Quoted 1 line

We need to find some sort of solution for building our CI build containers.

Why not use the shared runners provided by Gitlab? Apparently, there's even a dind capable runner (tag gitlab-org-docker). If the Gitlab runners work, the next step would be to make sure we are using the same version as Gitlab. Since we're back to using the exact same versions as in the previous cluster (which did work for us!), my guess is that the culprit is outside of our cluster, e.g. Gitlab or (less likely) kaniko.

Kind Regards,
Michael

Signature · 8 lines
Michael Adler

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
#9 lore raw

Hello,

Quoted 7 lines

From: Michael Adler <[email protected]>
Sent: 07 October 2022 10:27

Quoted 1 line

We need to find some sort of solution for building our CI build containers.

Why not use the shared runners provided by Gitlab? Apparently, there's even a dind capable runner (tag gitlab-org-docker).

Yea I was actually just looking into that :)

Quoted 5 lines

If the Gitlab runners work, the next step would be to make sure we are using the same version as Gitlab.
Since we're back to using the exact same versions as in the previous cluster (which did work for us!), my guess is that
the culprit is outside of our cluster, e.g. Gitlab or (less likely) kaniko.

Agreed.

Thanks, Chris

Quoted 17 lines

Kind Regards,
Michael

--
Michael Adler

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw
Quoted 9 lines

From: [email protected] <[email protected]> On Behalf Of Chris Paterson via lists.cip-project.org
Sent: 07 October 2022 10:39

Hello,

Quoted 3 lines

From: Michael Adler <[email protected]>
Sent: 07 October 2022 10:27

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

We need to find some sort of solution for building our CI build containers.

Why not use the shared runners provided by Gitlab? Apparently, there's even a dind capable runner (tag gitlab-org-docker).

Yea I was actually just looking into that :)

And indeed it seems to work fine:
https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/jobs/3141334688

I haven't tested a complete pipeline yet, but alas I need to go offline for most the day now. I'll look again this evening.

Quoted 4 lines
Quoted 2 lines

If the Gitlab runners work, the next step would be to make sure we are

using

Quoted 1 line

the same version as Gitlab.

Working version from above is gitlab-runner 15.4.0~beta.5.gdefc7017 (defc7017).

Kind regards, Chris

Quoted 27 lines
Quoted 3 lines

Since we're back to using the exact same versions as in the previous cluster (which did work for us!), my guess is that
the culprit is outside of our cluster, e.g. Gitlab or (less likely) kaniko.

Agreed.

Thanks, Chris

Quoted 7 lines

Kind Regards,
Michael

--
Michael Adler

Quoted 7 lines

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik

Neike,

Quoted 1 line

Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft:

Berlin

Quoted 2 lines

und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322

#12 lore raw
Quoted 3 lines

And indeed it seems to work fine:
https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/jobs/3141334688

Good! Using shared runners saves us money as well. Is the performance good enough? If that's the case, I'd suggest to use them permanently.

Quoted 1 line

Working version from above is gitlab-runner 15.4.0~beta.5.gdefc7017 (defc7017).

Hmm, that's a bit older than the latest released version (tag v15.4.0). I wonder why they haven't upgraded from beta 5 to the final version? Seems kinda odd, I would have guessed that Gitlab does CI/CD. Anyhow, I have upgraded to the officially released version (and will purge the 15.2.0 runners once they are idle). Could you give this another try, Chris?

Kind Regards,
Michael

Signature · 8 lines
Michael Adler

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
author
#13 lore raw
In reply to Michael Adler

Hi!

Quoted 7 lines
Quoted 3 lines

Still there, I'd say:

https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3123103311

Ok, I have restored our previous cluster (v9). Please try again and tell me if the problem still persists. If it does, then I'd blame Gitlab I/O otherwise it could be a regression somewhere in the K8s stack - in that case, our best bet might be to sit it out.

It may be a bit better.

https://gitlab.com/cip-project/cip-kernel/linux-cip/-/pipelines/660868595

Still see bbb failure, but that's different one:

https://lava.ciplatform.org/scheduler/job/756260

Thanks,
Pavel

Signature · 2 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany