Skip to contentSkip to conversation

-stable-rc tests failing

cip-dev Pavel Machek 3 participants 7 messages Aug 11, 2021 – Aug 16, 2021 1 message not in the archive

lore
Share & tools
lore

Pavel Machek

author
#0 lore raw

Hi!

-stable-rc testing seems to be failing more than usual.

This is the "usual" failure:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/jobs/1493077012

WARNING: Retrying... error=invalid argument
2170ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/1492901612/artifacts?: Get https://gitlab.com/api/v4/jobs/1492901612/artifacts?: dial tcp: i/o timeout id=1492901612 token=9gNGsuaZ 2171FATAL: invalid argument
2173
Uploading artifacts for failed job

But I don't usually see this one:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/jobs/1492901660

Job #371616: Submitted
219Health: Unknown
220Device Type: r8a7743-iwg20d-q7
221Device: None
222Test: boot
223URL: https://lava.ciplatform.org/scheduler/job/371616
224
226ERROR: Job failed: execution took longer than 2h0m0s seconds

Links where all the failures can be seen are:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-4.4.y
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-4.19.y
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-5.10.y

I'll try hitting some resubmit buttons, but help would be welcome.

Best regards,
Pavel

Signature · 2 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6670): https://lists.cip-project.org/g/cip-dev/message/6670
Mute This Topic: https://lists.cip-project.org/mt/84811238/4520388
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/10495289/4520388/727948398/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#1 lore raw

Hello Pavel,

Quoted 26 lines

From: Pavel Machek <[email protected]>
Sent: 11 August 2021 07:40

Hi!

-stable-rc testing seems to be failing more than usual.

This is the "usual" failure:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-
/jobs/1493077012

WARNING: Retrying...                                error=invalid argument 2170ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/1492901612/artifacts?: Get
https://gitlab.com/api/v4/jobs/1492901612/artifacts?: dial tcp: i/o timeout id=1492901612 token=9gNGsuaZ
2171FATAL: invalid argument
2173
Uploading artifacts for failed job

But I don't usually see this one:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-
/jobs/1492901660

Quoted 8 lines

Job #371616: Submitted
219Health: Unknown
220Device Type: r8a7743-iwg20d-q7
221Device: None
222Test: boot
223URL: https://lava.ciplatform.org/scheduler/job/371616
224
226ERROR: Job failed: execution took longer than 2h0m0s seconds

It looks like the LAVA server has stopped processing jobs. Our queue has become very large! I'll have to see what died and no doubt reboot it and the labs.

Sorry for the pain.

Kind regards, Chris

Quoted 10 lines

Links where all the failures can be seen are:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-
4.4.y
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-
4.19.y
https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/tree/linux-
5.10.y

Quoted 7 lines

I'll try hitting some resubmit buttons, but help would be welcome.

Best regards,
                                                              Pavel --
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6671): https://lists.cip-project.org/g/cip-dev/message/6671
Mute This Topic: https://lists.cip-project.org/mt/84811238/4520388
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/10495289/4520388/727948398/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

author
#2 lore raw

Hi!

Quoted 23 lines
Quoted 14 lines

WARNING: Retrying...                                error=invalid argument 2170ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/1492901612/artifacts?: Get
https://gitlab.com/api/v4/jobs/1492901612/artifacts?: dial tcp: i/o timeout id=1492901612 token=9gNGsuaZ
2171FATAL: invalid argument
2173
Uploading artifacts for failed job

But I don't usually see this one:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-
/jobs/1492901660

Quoted 8 lines

Job #371616: Submitted
219Health: Unknown
220Device Type: r8a7743-iwg20d-q7
221Device: None
222Test: boot
223URL: https://lava.ciplatform.org/scheduler/job/371616
224
226ERROR: Job failed: execution took longer than 2h0m0s seconds

Quoted 4 lines

It looks like the LAVA server has stopped processing jobs. Our queue has become very large! I'll have to see what died and no doubt reboot it and the labs.

Sorry for the pain.

Thank you, it seems to be better now, and it looks like ctj_zynqmp is available, too (I have not yet seen successful test, but it at least tries).

But I still see problems with x86_qemu... the "2h timeout".

Best regards,
Pavel

Signature · 2 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6678): https://lists.cip-project.org/g/cip-dev/message/6678
Mute This Topic: https://lists.cip-project.org/mt/84811238/4520388
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/10495289/4520388/727948398/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw

Hi,

Quoted 35 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Pavel Machek
Sent: Saturday, August 14, 2021 4:47 AM
To: Chris Paterson <[email protected]>
Cc: Pavel Machek <[email protected]>; [email protected]
Subject: Re: [cip-dev] -stable-rc tests failing

Hi!

Quoted 26 lines
Quoted 14 lines

WARNING: Retrying...                                error=invalid argument 2170ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/1492901612/artifacts?: Get
https://gitlab.com/api/v4/jobs/1492901612/artifacts?: dial tcp: i/o timeout id=1492901612 token=9gNGsuaZ
2171FATAL: invalid argument
2173
Uploading artifacts for failed job

But I don't usually see this one:

https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-
/jobs/1492901660

Quoted 8 lines

Job #371616: Submitted
219Health: Unknown
220Device Type: r8a7743-iwg20d-q7
221Device: None
222Test: boot
223URL: https://lava.ciplatform.org/scheduler/job/371616
224
226ERROR: Job failed: execution took longer than 2h0m0s seconds

It looks like the LAVA server has stopped processing jobs. Our queue has become very large! I'll have to see what died and no doubt reboot it and the labs.

Quoted 7 lines
Quoted 1 line

Sorry for the pain.

Thank you, it seems to be better now, and it looks like ctj_zynqmp is available, too (I have not yet seen successful test, but it at least tries).

But I still see problems with x86_qemu... the "2h timeout".

QEMU's queue is full. And QEMU of each lab is not working and it seems that it can not be processed correctly.

Best regards,
Nobuhiro

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6679): https://lists.cip-project.org/g/cip-dev/message/6679
Mute This Topic: https://lists.cip-project.org/mt/84811238/4520388
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/10495289/4520388/727948398/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw

Hello,

Quoted 76 lines

From: [email protected] <[email protected]> On Behalf Of Nobuhiro Iwamatsu via lists.cip-project.org
Sent: 16 August 2021 00:15

Hi,

Quoted 2 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]]

On Behalf Of Pavel Machek

Quoted 9 lines

Sent: Saturday, August 14, 2021 4:47 AM
To: Chris Paterson <[email protected]>
Cc: Pavel Machek <[email protected]>; [email protected]
Subject: Re: [cip-dev] -stable-rc tests failing

Hi!

Quoted 2 lines
Quoted 2 lines

WARNING: Retrying...                                error=invalid argument 2170ERROR: Downloading artifacts from coordinator... error couldn't

execute

Quoted 1 line
Quoted 1 line
Quoted 1 line

GET against

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
.com%2Fapi%2Fv4%2Fjobs%2F1492901612%2Fartifacts&amp;data=04%7C01 %7Cchris.paterson2%40renesas.com%7C4479c8657b2a40a46e3108d960428bd 8%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0%7C6376466612029063 02%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=qwhRPtRSueYf9 7CTKC1JVJbJGJWHpfwkWq9GebqwLs4%3D&amp;reserved=0?: Get

Quoted 1 line
Quoted 1 line
Quoted 1 line

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
.com%2Fapi%2Fv4%2Fjobs%2F1492901612%2Fartifacts&amp;data=04%7C01 %7Cchris.paterson2%40renesas.com%7C4479c8657b2a40a46e3108d960428bd 8%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0%7C6376466612029063 02%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=qwhRPtRSueYf9 7CTKC1JVJbJGJWHpfwkWq9GebqwLs4%3D&amp;reserved=0?: dial tcp: i/o timeout

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

id=1492901612 token=9gNGsuaZ
2171FATAL: invalid argument
2173
Uploading artifacts for failed job

But I don't usually see this one:

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab
.com%2Fcip-project%2Fcip-testing%2Flinux-stable-rc-ci%2F- &amp;data=04%7C01%7Cchris.paterson2%40renesas.com%7C4479c8657b2a4 0a46e3108d960428bd8%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0% 7C637646661202906302%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sda ta=7bk5d%2FyboUuOw0l12NBKvIDvsnVLmzwvyVSniEUOjI4%3D&amp;reserv ed=0

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

/jobs/1492901660

Job #371616: Submitted
219Health: Unknown
220Device Type: r8a7743-iwg20d-q7
221Device: None
222Test: boot
223URL:

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flava.c
iplatform.org%2Fscheduler%2Fjob%2F371616&amp;data=04%7C01%7Cchris. paterson2%40renesas.com%7C4479c8657b2a40a46e3108d960428bd8%7C53d 82571da1947e49cb4625a166a4a2a%7C0%7C0%7C637646661202906302%7CUn known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6 Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Mkr7BDJRwjM%2F1Oeg4Q GEOpN%2Bj6%2B8C2%2BZ8mtbOppdhY0%3D&amp;reserved=0

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

224
226ERROR: Job failed: execution took longer than 2h0m0s seconds

It looks like the LAVA server has stopped processing jobs. Our queue has

become very large!

Quoted 4 lines
Quoted 3 lines

I'll have to see what died and no doubt reboot it and the labs.

Sorry for the pain.

Quoted 8 lines
Quoted 5 lines

Thank you, it seems to be better now, and it looks like ctj_zynqmp is available, too (I have not yet seen successful test, but it at least tries).

But I still see problems with x86_qemu... the "2h timeout".

QEMU's queue is full. And QEMU of each lab is not working and it seems that it can not be processed correctly.

All of the QEMU machines have somehow got stuck in the "reserved" state. I'm trying to fix it. Will update you asap.

Chris

Quoted 3 lines

Best regards,
Nobuhiro

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6680): https://lists.cip-project.org/g/cip-dev/message/6680
Mute This Topic: https://lists.cip-project.org/mt/84811238/4520388
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/10495289/4520388/727948398/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Hello,

Quoted 5 lines

From: [email protected] <[email protected]> On Behalf Of Chris Paterson via lists.cip-project.org
Sent: 16 August 2021 21:39

Hello,

[...]

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

But I still see problems with x86_qemu... the "2h timeout".

QEMU's queue is full. And QEMU of each lab is not working and it seems that it can not be processed correctly.

All of the QEMU machines have somehow got stuck in the "reserved" state. I'm trying to fix it. Will update you asap.

I've got these devices un-stuck now by editing their state in the database directly.
However, in the process of debugging I rebooted the LAVA server instance, so we'll now need to reboot the various labs, which is dependent on the local admins. Hopefully normal service will be resumed soon.

Kind regards, Chris

Quoted 6 lines

Chris

Quoted 3 lines

Best regards,
Nobuhiro

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6681): https://lists.cip-project.org/g/cip-dev/message/6681
Mute This Topic: https://lists.cip-project.org/mt/84811238/4520388
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/10495289/4520388/727948398/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#7 lore raw

Hi,

Quoted 21 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chris Paterson
Sent: Tuesday, August 17, 2021 5:58 AM
To: [email protected]
Subject: Re: [cip-dev] -stable-rc tests failing

Hello,

Quoted 5 lines

From: [email protected] <[email protected]> On Behalf Of Chris Paterson via lists.cip-project.org
Sent: 16 August 2021 21:39

Hello,

[...]

Quoted 5 lines
Quoted 4 lines
Quoted 1 line

But I still see problems with x86_qemu... the "2h timeout".

QEMU's queue is full. And QEMU of each lab is not working and it seems that it can not be processed correctly.

Quoted 7 lines
Quoted 2 lines

All of the QEMU machines have somehow got stuck in the "reserved" state. I'm trying to fix it. Will update you asap.

I've got these devices un-stuck now by editing their state in the database directly.
However, in the process of debugging I rebooted the LAVA server instance, so we'll now need to reboot the various labs, which is dependent on the local admins.
Hopefully normal service will be resumed soon.

Thanks for your work!

Quoted 6 lines

Kind regards, Chris

Quoted 3 lines

Chris

Best regards,
Nobuhiro

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#6682): https://lists.cip-project.org/g/cip-dev/message/6682
Mute This Topic: https://lists.cip-project.org/mt/84811238/4520388
Group Owner: [email protected]
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/10495289/4520388/727948398/xyzzy [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-