Skip to contentSkip to conversation

intel-ci: Add gem_exec_suspend/basic-S3/S4-devices to BAT

PATCH In review Imre Deak 6 participants 10 messages Oct 17, 2016 – Nov 25, 2016

lore
Share & tools
lore

Imre Deak

author
#0 lore raw

PATCH 1/1 intel-ci: Add gem_exec_suspend/basic-S3/S4-devices to BAT

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests to BAT. At the same time remove basic-s4 from the list, which is atm implicitly disabled via HIBERNATION=n in kconfig. We would need at least basic S4 coverage provided by basic-s4-devices, which requires HIBERNATION=y.

---
 tests/intel-ci/fast-feedback.testlist | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/intel-ci/fast-feedback.testlist +2 -1
index f59ec98..efa7e86 100644
@@ -73,8 +73,9 @@ igt@gem_exec_store@basic-default
 igt@gem_exec_store@basic-render
 igt@gem_exec_store@basic-vebox
 igt@gem_exec_suspend@basic
+igt@gem_exec_suspend@basic-s3-devices
 igt@gem_exec_suspend@basic-s3
-igt@gem_exec_suspend@basic-s4
+igt@gem_exec_suspend@basic-s4-devices
 igt@gem_flink_basic@bad-flink
 igt@gem_flink_basic@bad-open
 igt@gem_flink_basic@basic
Signature · 6 lines
2.5.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw
Quoted 7 lines · Imre Deak

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests to BAT. At the same time remove basic-s4 from the list, which is atm implicitly disabled via HIBERNATION=n in kconfig. We would need at least basic S4 coverage provided by basic-s4-devices, which requires HIBERNATION=y.

What's the impact on BAT runtime with this? Afaik we're already over budget ... Where do you safe the time to afford this? -Daniel

Quoted 26 lines
---
 tests/intel-ci/fast-feedback.testlist | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/intel-ci/fast-feedback.testlist +2 -1
index f59ec98..efa7e86 100644
@@ -73,8 +73,9 @@ igt@gem_exec_store@basic-default
 igt@gem_exec_store@basic-render
 igt@gem_exec_store@basic-vebox
 igt@gem_exec_suspend@basic
+igt@gem_exec_suspend@basic-s3-devices
 igt@gem_exec_suspend@basic-s3
-igt@gem_exec_suspend@basic-s4
+igt@gem_exec_suspend@basic-s4-devices
 igt@gem_flink_basic@bad-flink
 igt@gem_flink_basic@bad-open
 igt@gem_flink_basic@basic
Signature · 6 lines
2.5.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#2 lore raw
Quoted 9 lines · Daniel Vetter
Quoted 8 lines · Imre Deak

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests to
BAT. At the same time remove basic-s4 from the list, which is atm implicitly disabled via HIBERNATION=n in kconfig. We would need at least
basic S4 coverage provided by basic-s4-devices, which requires HIBERNATION=y.

Quoted 3 lines
Quoted 1 line

What's the impact on BAT runtime with this?

I measured 8 sec for S3-devices and 9 sec for S4-devices on my APL.

Quoted 2 lines

Afaik we're already over budget ... Where do you safe the time to afford this?

I didn't, but we don't have any S4 coverage in CI atm and it's the minimum that can be added. The S3-devices subtest is not critical, although it would be useful for cases where we wouldn't get any logs for the full S3 test.

--Imre

Quoted 25 lines

-Daniel

Quoted 23 lines

---
 tests/intel-ci/fast-feedback.testlist | 3 ++-

 1 file changed, 2 insertions(+), 1 deletion(-)
tests/intel- renamed from tests/intel-ci/fast-feedback.testlist +0 -0

ci/fast-feedback.testlist
index f59ec98..efa7e86 100644

tests/intel-ci/fast-feedback.testlist +0 -0
@@ -73,8 +73,9 @@ igt@gem_exec_store@basic-default

 igt@gem_exec_store@basic-render
 igt@gem_exec_store@basic-vebox
 igt@gem_exec_suspend@basic
+igt@gem_exec_suspend@basic-s3-devices
 igt@gem_exec_suspend@basic-s3

tests/intel-ci/fast-feedback.testlist +1 -1
-igt@gem_exec_suspend@basic-s4
+igt@gem_exec_suspend@basic-s4-devices

 igt@gem_flink_basic@bad-flink
 igt@gem_flink_basic@bad-open
 igt@gem_flink_basic@basic

Signature · 1 line
2.5.0
Quoted 5 lines
Quoted 4 lines

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw
Quoted 16 lines · Imre Deak
Quoted 9 lines · Daniel Vetter
Quoted 8 lines · Imre Deak

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests to
BAT. At the same time remove basic-s4 from the list, which is atm implicitly disabled via HIBERNATION=n in kconfig. We would need at least
basic S4 coverage provided by basic-s4-devices, which requires HIBERNATION=y.

Quoted 3 lines
Quoted 1 line

What's the impact on BAT runtime with this?

I measured 8 sec for S3-devices and 9 sec for S4-devices on my APL.

Quoted 7 lines
Quoted 2 lines

Afaik we're already over budget ... Where do you safe the time to afford this?

I didn't, but we don't have any S4 coverage in CI atm and it's the minimum that can be added. The S3-devices subtest is not critical, although it would be useful for cases where we wouldn't get any logs for the full S3 test.

If our CI was sufficiently clever, it could detect these patches to change the test list, and let us all know what the impact is...

BR,
Jani.

Quoted 37 lines

--Imre

Quoted 25 lines

-Daniel

Quoted 23 lines

---
 tests/intel-ci/fast-feedback.testlist | 3 ++-

 1 file changed, 2 insertions(+), 1 deletion(-)
tests/intel- renamed from tests/intel-ci/fast-feedback.testlist +0 -0

ci/fast-feedback.testlist
index f59ec98..efa7e86 100644

tests/intel-ci/fast-feedback.testlist +0 -0
@@ -73,8 +73,9 @@ igt@gem_exec_store@basic-default

 igt@gem_exec_store@basic-render
 igt@gem_exec_store@basic-vebox
 igt@gem_exec_suspend@basic
+igt@gem_exec_suspend@basic-s3-devices
 igt@gem_exec_suspend@basic-s3

tests/intel-ci/fast-feedback.testlist +1 -1
-igt@gem_exec_suspend@basic-s4
+igt@gem_exec_suspend@basic-s4-devices

 igt@gem_flink_basic@bad-flink
 igt@gem_flink_basic@bad-open
 igt@gem_flink_basic@basic

Signature · 1 line
2.5.0
Quoted 5 lines
Quoted 4 lines

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw
In reply to Imre Deak

HI,

Quoted 9 lines

-----Original Message-----
From: Intel-gfx [mailto:[email protected]] On Behalf Of Imre Deak
Sent: Monday, October 17, 2016 5:47 PM
To: Daniel Vetter <[email protected]>
Cc: [email protected]
Subject: Re: [Intel-gfx] [PATCH] intel-ci: Add gem_exec_suspend/basic- S3/S4-devices to BAT

Quoted 11 lines
Quoted 10 lines · Daniel Vetter
Quoted 7 lines · Imre Deak

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests to BAT. At the same time remove basic-s4 from the list, which is atm implicitly disabled via HIBERNATION=n in kconfig. We would need at least basic S4 coverage provided by basic-s4-devices, which requires HIBERNATION=y.

What's the impact on BAT runtime with this?

Are these also stable enough to be added so that we are not causing flip-flops?

Quoted 28 lines

I measured 8 sec for S3-devices and 9 sec for S4-devices on my APL.

Quoted 2 lines

Afaik we're already over budget ... Where do you safe the time to afford this?

I didn't, but we don't have any S4 coverage in CI atm and it's the minimum that can be added. The S3-devices subtest is not critical, although it would be useful for cases where we wouldn't get any logs for the full S3 test.

--Imre

Quoted 16 lines

-Daniel

Quoted 14 lines

---
 tests/intel-ci/fast-feedback.testlist | 3 ++-

 1 file changed, 2 insertions(+), 1 deletion(-)
tests/intel- renamed from tests/intel-ci/fast-feedback.testlist +0 -0

ci/fast-feedback.testlist index f59ec98..efa7e86 100644

tests/intel-ci/fast-feedback.testlist +0 -0
@@ -73,8 +73,9 @@ igt@gem_exec_store@basic-default

 igt@gem_exec_store@basic-render
 igt@gem_exec_store@basic-vebox
 igt@gem_exec_suspend@basic
+igt@gem_exec_suspend@basic-s3-devices
 igt@gem_exec_suspend@basic-s3

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
tests/intel-ci/fast-feedback.testlist +1 -1
-igt@gem_exec_suspend@basic-s4
+igt@gem_exec_suspend@basic-s4-devices

 igt@gem_flink_basic@bad-flink
 igt@gem_flink_basic@bad-open
 igt@gem_flink_basic@basic
--
2.5.0

Jani Saarinen
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#5 lore raw
Quoted 24 lines · Jani

HI,

Quoted 23 lines

-----Original Message-----
From: Intel-gfx [mailto:[email protected]] On Behalf
Of Imre Deak
Sent: Monday, October 17, 2016 5:47 PM
To: Daniel Vetter <[email protected]>
Cc: [email protected]
Subject: Re: [Intel-gfx] [PATCH] intel-ci: Add gem_exec_suspend/basic-
S3/S4-devices to BAT

Quoted 11 lines · Daniel Vetter
Quoted 10 lines · Imre Deak

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests
to BAT. At the same time remove basic-s4 from the list, which is atm
implicitly disabled via HIBERNATION=n in kconfig. We would need at
least basic S4 coverage provided by basic-s4-devices, which requires
HIBERNATION=y.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

What's the impact on BAT runtime with this?

Are these also stable enough to be added so that we are not causing flip-flops?

Based on my tests, they are.

Quoted 34 lines
Quoted 34 lines

I measured 8 sec for S3-devices and 9 sec for S4-devices on my APL.

Quoted 2 lines

Afaik we're already over budget ... Where do you safe the time to afford this?

I didn't, but we don't have any S4 coverage in CI atm and it's the minimum
that can be added. The S3-devices subtest is not critical, although it would be
useful for cases where we wouldn't get any logs for the full S3 test.

--Imre

Quoted 20 lines

-Daniel

Quoted 18 lines

---
 tests/intel-ci/fast-feedback.testlist | 3 ++-

 1 file changed, 2 insertions(+), 1 deletion(-)
tests/intel- renamed from tests/intel-ci/fast-feedback.testlist +0 -0

ci/fast-feedback.testlist index f59ec98..efa7e86 100644

tests/intel-ci/fast-feedback.testlist +0 -0
@@ -73,8 +73,9 @@ igt@gem_exec_store@basic-default

 igt@gem_exec_store@basic-render
 igt@gem_exec_store@basic-vebox
 igt@gem_exec_suspend@basic
+igt@gem_exec_suspend@basic-s3-devices
 igt@gem_exec_suspend@basic-s3

tests/intel-ci/fast-feedback.testlist +1 -1
-igt@gem_exec_suspend@basic-s4
+igt@gem_exec_suspend@basic-s4-devices

 igt@gem_flink_basic@bad-flink

Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

 igt@gem_flink_basic@bad-open
 igt@gem_flink_basic@basic
--
2.5.0

Jani Saarinen
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#6 lore raw
In reply to Imre Deak
Quoted 16 lines · Imre Deak
Quoted 9 lines · Daniel Vetter
Quoted 8 lines · Imre Deak

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests to
BAT. At the same time remove basic-s4 from the list, which is atm implicitly disabled via HIBERNATION=n in kconfig. We would need at least
basic S4 coverage provided by basic-s4-devices, which requires HIBERNATION=y.

Quoted 3 lines
Quoted 1 line

What's the impact on BAT runtime with this?

I measured 8 sec for S3-devices and 9 sec for S4-devices on my APL.

Quoted 7 lines
Quoted 2 lines

Afaik we're already over budget ... Where do you safe the time to afford this?

I didn't, but we don't have any S4 coverage in CI atm and it's the minimum that can be added. The S3-devices subtest is not critical, although it would be useful for cases where we wouldn't get any logs for the full S3 test.

Yeah, I don't think a reduced S3 test in BAT is useful if we have the full one already. It's a good test for debugging, but not within the very tight constraints we have for BAT.

Re S4: If it indeed improves coverage (i.e. calls our shutdown hooks and all these S4-only callbacks) then adding it to BAT sounds reasonable. Still there's the issue of where to get the machine time from. I really do think you need to first trade in some speed-up here (or throw out some other tests) before you can add more tests.

And yes I know that's not fun, but until we have CI running a more complete test set (after BAT has done the initial sanity checking, to avoid wasting machine time on broken patches), that's the reality we have :(

Cheers, Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#7 lore raw
Quoted 5 lines · Daniel Vetter

Re S4: If it indeed improves coverage (i.e. calls our shutdown hooks and all these S4-only callbacks) then adding it to BAT sounds reasonable. Still there's the issue of where to get the machine time from. I really do think you need to first trade in some speed-up here (or throw out some other tests) before you can add more tests.

For the record, the runtimes of all tests in CI can be found in the result-runtimes.log file, within the test log directory.

Over 10s runtimes on a randomly selected machine are:

31.92 igt@gem_ctx_switch@basic-default-heavy 22.91 igt@gem_busy@basic-hang-default
21.03 igt@kms_flip@basic-flip-vs-dpms
20.96 igt@kms_flip@basic-flip-vs-modeset 20.39 igt@gem_ctx_switch@basic-default
20.30 igt@gem_exec_create@basic
20.11 igt@gem_ctx_create@basic-files
19.64 igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a 18.93 igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b 18.91 igt@kms_pipe_crc_basic@suspend-read-crc-pipe-c 18.76 igt@gem_exec_suspend@basic-s3
17.98 igt@kms_pipe_crc_basic@hang-read-crc-pipe-c 16.70 igt@kms_flip@basic-flip-vs-wf_vblank 14.17 igt@kms_pipe_crc_basic@hang-read-crc-pipe-a 14.14 igt@gem_exec_nop@basic-parallel
14.12 igt@gem_exec_nop@basic-series
13.76 igt@kms_pipe_crc_basic@hang-read-crc-pipe-b 12.70 igt@gem_ringfill@basic-default-hang 11.34 igt@drv_hangman@error-state-basic 10.91 igt@gem_sync@basic-many-each
10.81 igt@gem_close_race@basic-threads
10.73 igt@kms_flip@basic-plain-flip
10.33 igt@gem_sync@basic-store-each
10.29 igt@gem_sync@basic-store-all
10.11 igt@gem_sync@basic-each
10.08 igt@gem_sync@basic-all

Does any of these strike as a low-hanging fruit?

--
Petri Latvala
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw
Quoted 39 lines · Petri Latvala
Quoted 5 lines · Daniel Vetter

Re S4: If it indeed improves coverage (i.e. calls our shutdown hooks and all these S4-only callbacks) then adding it to BAT sounds reasonable. Still there's the issue of where to get the machine time from. I really do think you need to first trade in some speed-up here (or throw out some other tests) before you can add more tests.

For the record, the runtimes of all tests in CI can be found in the result-runtimes.log file, within the test log directory.

Over 10s runtimes on a randomly selected machine are:

31.92 igt@gem_ctx_switch@basic-default-heavy 22.91 igt@gem_busy@basic-hang-default
21.03 igt@kms_flip@basic-flip-vs-dpms
20.96 igt@kms_flip@basic-flip-vs-modeset 20.39 igt@gem_ctx_switch@basic-default
20.30 igt@gem_exec_create@basic
20.11 igt@gem_ctx_create@basic-files
19.64 igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a 18.93 igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b 18.91 igt@kms_pipe_crc_basic@suspend-read-crc-pipe-c 18.76 igt@gem_exec_suspend@basic-s3
17.98 igt@kms_pipe_crc_basic@hang-read-crc-pipe-c 16.70 igt@kms_flip@basic-flip-vs-wf_vblank 14.17 igt@kms_pipe_crc_basic@hang-read-crc-pipe-a 14.14 igt@gem_exec_nop@basic-parallel
14.12 igt@gem_exec_nop@basic-series
13.76 igt@kms_pipe_crc_basic@hang-read-crc-pipe-b 12.70 igt@gem_ringfill@basic-default-hang 11.34 igt@drv_hangman@error-state-basic 10.91 igt@gem_sync@basic-many-each
10.81 igt@gem_close_race@basic-threads
10.73 igt@kms_flip@basic-plain-flip
10.33 igt@gem_sync@basic-store-each
10.29 igt@gem_sync@basic-store-all
10.11 igt@gem_sync@basic-each
10.08 igt@gem_sync@basic-all

Quoted 1 line

Does any of these strike as a low-hanging fruit?

s/15/5/ for the S3 delay I'd say. If there's a machine that won't work with 5 seconds, then we find out what's the lowest safe value and use that instead. 15 seconds is excessive on all the machines I've seen. And maybe make these delays configurabe via env variables so that I don't have to keep patching igt when I want to test suspendy stuff ;)

Signature · 6 lines
Ville Syrjälä
Intel OTC
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#9 lore raw
In reply to Imre Deak
Quoted 16 lines · Imre Deak

Add gem_exec_suspend/basic-s3-devices and basic-s4-devices subtests to BAT. At the same time remove basic-s4 from the list, which is atm implicitly disabled via HIBERNATION=n in kconfig. We would need at least basic S4 coverage provided by basic-s4-devices, which requires HIBERNATION=y.

---
 tests/intel-ci/fast-feedback.testlist | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/intel-ci/fast-feedback.testlist +0 -0
index f59ec98..efa7e86 100644

tests/intel-ci/fast-feedback.testlist igt@gem_exec_store@basic-default · line 81

Quoted 10 lines
tests/intel-ci/fast-feedback.testlist +2 -1
@@ -73,8 +73,9 @@ igt@gem_exec_store@basic-default
 igt@gem_exec_store@basic-render
 igt@gem_exec_store@basic-vebox
 igt@gem_exec_suspend@basic
+igt@gem_exec_suspend@basic-s3-devices
 igt@gem_exec_suspend@basic-s3
-igt@gem_exec_suspend@basic-s4
+igt@gem_exec_suspend@basic-s4-devices
 igt@gem_flink_basic@bad-flink
 igt@gem_flink_basic@bad-open
 igt@gem_flink_basic@basic

I pushed this commit, but without adding basic-s3-devices. The needed kernel config changes will be deployed after CI is using IGT builds with this commit in, next week.

--
Petri Latvala
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx