Skip to contentSkip to conversation

tests/kms_flip: Adjust tolerance when counting frames

PATCH Superseded Gabriel Feceoru 5 participants 13 messages May 10, 2016 – May 17, 2016

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/kms_flip: Adjust tolerance when counting frames

Comparing 2 numbers with 1% accuracy depends on which one is the reference. If count == 100 and expected == 99 this condition fails, although it should pass.

---
 tests/kms_flip.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/kms_flip.c +2 -1
index eda2fcc..938b32d 100644
@@ -1187,7 +1187,8 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
 		count *= o->seq_step;
 		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f((count >= expected * 99/100 && count <= expected * 101/100) ||
+			     (expected >= count * 99/100 && expected <= count * 101/100),
 			     "dropped frames, expected %d, counted %d, encoder type %d\n",
 			     expected, count, o->kencoder[0]->encoder_type);
 	}
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw
Quoted 3 lines · Gabriel Feceoru

Comparing 2 numbers with 1% accuracy depends on which one is the reference. If count == 100 and expected == 99 this condition fails, although it should pass.

Well, the expectation should be the reference. If you expect 50 at 50% tolerance, 25..75 is okay. 100 is clearly out of tolerance, but your method would accept it too.

Would it help to round the lower limit down and upper limit up? I think that would be more acceptable.

BR,
Jani.

Quoted 19 lines
---
 tests/kms_flip.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/kms_flip.c +2 -1
index eda2fcc..938b32d 100644
@@ -1187,7 +1187,8 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
 		count *= o->seq_step;
 		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f((count >= expected * 99/100 && count <= expected * 101/100) ||
+			     (expected >= count * 99/100 && expected <= count * 101/100),
 			     "dropped frames, expected %d, counted %d, encoder type %d\n",
 			     expected, count, o->kencoder[0]->encoder_type);
 	}
Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw
Quoted 11 lines · Jani Nikula
Quoted 3 lines · Gabriel Feceoru

Comparing 2 numbers with 1% accuracy depends on which one is the reference. If count == 100 and expected == 99 this condition fails, although it should pass.

Well, the expectation should be the reference. If you expect 50 at 50% tolerance, 25..75 is okay. 100 is clearly out of tolerance, but your method would accept it too.

Would it help to round the lower limit down and upper limit up? I think that would be more acceptable.

Yes, you are right. I'll adjust it to 98/100 and 102/100.

Thanks,
Gabriel.

Quoted 15 lines

BR,
Jani.

Quoted 11 lines
---
  tests/kms_flip.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
tests/kms_flip.c +0 -0
index eda2fcc..938b32d 100644
@@ -1187,7 +1187,8 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
Quoted 9 lines

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1194

Quoted 8 lines
tests/kms_flip.c +2 -1
  		count *= o->seq_step;
  		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f((count >= expected * 99/100 && count <= expected * 101/100) ||
+			     (expected >= count * 99/100 && expected <= count * 101/100),
  			     "dropped frames, expected %d, counted %d, encoder type %d\n",
  			     expected, count, o->kencoder[0]->encoder_type);
  	}

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw
Quoted 8 lines · Gabriel Feceoru
Quoted 5 lines · Jani Nikula
Quoted 3 lines · Gabriel Feceoru

Comparing 2 numbers with 1% accuracy depends on which one is the reference. If count == 100 and expected == 99 this condition fails, although it should pass.

Quoted 8 lines
Quoted 6 lines

Well, the expectation should be the reference. If you expect 50 at 50% tolerance, 25..75 is okay. 100 is clearly out of tolerance, but your method would accept it too.

Would it help to round the lower limit down and upper limit up? I think that would be more acceptable.

Yes, you are right. I'll adjust it to 98/100 and 102/100.

Maybe also extract the computation for lower/upper limit into local variables, to make the code less dense and easier to understand. -Daniel

Quoted 28 lines

Thanks,
Gabriel.

Quoted 24 lines

BR,
Jani.

Quoted 19 lines
---
 tests/kms_flip.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tests/kms_flip.c +2 -1
index eda2fcc..938b32d 100644
@@ -1187,7 +1187,8 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
 		count *= o->seq_step;
 		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f((count >= expected * 99/100 && count <= expected * 101/100) ||
+			     (expected >= count * 99/100 && expected <= count * 101/100),
 			     "dropped frames, expected %d, counted %d, encoder type %d\n",
 			     expected, count, o->kencoder[0]->encoder_type);
 	}
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw
Quoted 13 lines · Daniel Vetter
Quoted 12 lines · Gabriel Feceoru
Quoted 9 lines · Jani Nikula
Quoted 3 lines · Gabriel Feceoru

Comparing 2 numbers with 1% accuracy depends on which one is the reference. If count == 100 and expected == 99 this condition fails, although it should pass.

Well, the expectation should be the reference. If you expect 50 at 50% tolerance, 25..75 is okay. 100 is clearly out of tolerance, but your method would accept it too.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Would it help to round the lower limit down and upper limit up? I think that would be more acceptable.

Yes, you are right. I'll adjust it to 98/100 and 102/100.

Maybe also extract the computation for lower/upper limit into local variables, to make the code less dense and easier to understand.

Just because I have this lying around (untested):

tests/kms_flip.c +6 -6
index eda2fcc..22b9c9d 100644
@@ -1182,14 +1182,14 @@ static void check_final_state(struct test_output *o, struct event_state *es,
        /* Verify we drop no frames, but only if it's not a TV encoder, since
         * those use some funny fake timings behind userspace's back. */
        if (o->flags & TEST_CHECK_TS && !analog_tv_connector(o)) {
-               int expected;
-               int count = es->count;
+               int count = es->count * o->seq_step;
+               unsigned int min = frame_time(o) * (count - 1);
+               unsigned int max = frame_time(o) * (count + 1);
 
-               count *= o->seq_step;
-               expected = elapsed / frame_time(o);
-               igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+               igt_assert_f(elapsed > min && elapsed < max,
                             "dropped frames, expected %d, counted %d, encoder type %d\n",
-                            expected, count, o->kencoder[0]->encoder_type);
+                            elapsed / frame_time(o), count,
+                            o->kencoder[0]->encoder_type);
        }
 }
 

-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#5 lore raw

PATCH v2 1/1 tests/kms_flip: Adjust tolerance when counting frames

In reply to Gabriel Feceoru

If count == 100 and expected == 99 this condition fails (99*101/100 = 99.99).

(v2): Increased the tolerance range, as suggested by Jani.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 tests/kms_flip.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/kms_flip.c +1 -1
index eda2fcc..ceb0e0b 100644
@@ -1187,7 +1187,7 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
 		count *= o->seq_step;
 		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f(count >= expected * 98/100 && count <= expected * 102/100,
 			     "dropped frames, expected %d, counted %d, encoder type %d\n",
 			     expected, count, o->kencoder[0]->encoder_type);
 	}
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw
Quoted 16 lines · Gabriel Feceoru

If count == 100 and expected == 99 this condition fails (99*101/100 = 99.99).

(v2): Increased the tolerance range, as suggested by Jani.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 tests/kms_flip.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/kms_flip.c +0 -0
index eda2fcc..ceb0e0b 100644

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1190

Quoted 5 lines
tests/kms_flip.c +1 -1
@@ -1187,7 +1187,7 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
 		count *= o->seq_step;
 		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f(count >= expected * 98/100 && count <= expected * 102/100,

I was thinking of

#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))

igt_assert_f(count >= expected * 99 / 100 &&

             count <= DIV_ROUND_UP(expected * 101, 100));

but maybe someone who knows how accurate this should really be could chime in.

BR,
Jani.

Quoted 3 lines
tests/kms_flip.c +0 -0
 			     "dropped frames, expected %d, counted %d, encoder type %d\n",
 			     expected, count, o->kencoder[0]->encoder_type);
 	}
Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#7 lore raw
Quoted 27 lines · Jani Nikula
Quoted 16 lines · Gabriel Feceoru

If count == 100 and expected == 99 this condition fails (99*101/100 = 99.99).

(v2): Increased the tolerance range, as suggested by Jani.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
  tests/kms_flip.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/kms_flip.c +0 -0
index eda2fcc..ceb0e0b 100644

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1190

Quoted 5 lines
tests/kms_flip.c +1 -1
@@ -1187,7 +1187,7 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
  		count *= o->seq_step;
  		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f(count >= expected * 98/100 && count <= expected * 102/100,

I was thinking of

#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))

Quoted 8 lines

igt_assert_f(count >= expected * 99 / 100 &&

              count <= DIV_ROUND_UP(expected * 101, 100));

but maybe someone who knows how accurate this should really be could chime in.

BR,
Jani.

I've just realized that frame_time(0) is a float, so converting to int in that division will lose precision.

Proposing this now:

tests/kms_flip.c +0 -0
index eda2fcc..c2d3929 100644
@@ -1182,13 +1182,13 @@ static void check_final_state(struct test_output *o, struct event_state *es,
        /* Verify we drop no frames, but only if it's not a TV encoder, 

since
* those use some funny fake timings behind userspace's back. */
if (o->flags & TEST_CHECK_TS && !analog_tv_connector(o)) {
- int expected;

+               double expected;
                int count = es->count;

                count *= o->seq_step;

- expected = elapsed / frame_time(o);
- igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
- "dropped frames, expected %d, counted %d, encoder type %d\n",
+ expected = (double)elapsed / frame_time(o);
+ igt_assert_f(fabs((double)count - expected)/expected < 0.01,
+ "dropped frames, expected %f, counted %d, encoder type %d\n",
expected, count,

o->kencoder[0]->encoder_type);
        }

}

Regards,
Gabriel.

Quoted 6 lines

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1193

Quoted 5 lines
tests/kms_flip.c +0 -0
  			     "dropped frames, expected %d, counted %d, encoder type %d\n",
  			     expected, count, o->kencoder[0]->encoder_type);
  	}

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw
Quoted 62 lines · Gabriel Feceoru
Quoted 27 lines · Jani Nikula
Quoted 16 lines · Gabriel Feceoru

If count == 100 and expected == 99 this condition fails (99*101/100 = 99.99).

(v2): Increased the tolerance range, as suggested by Jani.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
  tests/kms_flip.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/kms_flip.c +0 -0
index eda2fcc..ceb0e0b 100644

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1190

Quoted 5 lines
tests/kms_flip.c +1 -1
@@ -1187,7 +1187,7 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 
  		count *= o->seq_step;
  		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
+		igt_assert_f(count >= expected * 98/100 && count <= expected * 102/100,

I was thinking of

#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))

Quoted 8 lines

igt_assert_f(count >= expected * 99 / 100 &&

              count <= DIV_ROUND_UP(expected * 101, 100));

but maybe someone who knows how accurate this should really be could chime in.

BR,
Jani.

I've just realized that frame_time(0) is a float, so converting to int in that division will lose precision.

Proposing this now:

tests/kms_flip.c +0 -0
index eda2fcc..c2d3929 100644
@@ -1182,13 +1182,13 @@ static void check_final_state(struct test_output 
 *o, struct event_state *es,
         /* Verify we drop no frames, but only if it's not a TV encoder, 

since
* those use some funny fake timings behind userspace's back. */
if (o->flags & TEST_CHECK_TS && !analog_tv_connector(o)) {

tests/kms_flip.c +1 -3
-               int expected;
+               double expected;
                 int count = es->count;
 
                 count *= o->seq_step;
-               expected = elapsed / frame_time(o);
-               igt_assert_f(count >= expected * 99/100 && count <= 

expected * 101/100,
- "dropped frames, expected %d, counted %d,
encoder type %d\n",

tests/kms_flip.c

Quoted 3 lines
tests/kms_flip.c +3 -0
+               expected = (double)elapsed / frame_time(o);
+               igt_assert_f(fabs((double)count - expected)/expected < 0.01,
+                            "dropped frames, expected %f, counted %d, 

Heh, I've been working on the kernel so long that I was trying to avoid floats in my suggestion above.

This seems fine to me, except the condition should be <= not < I guess.

Please send a proper patch.

BR,
Jani.

Quoted 15 lines

encoder type %d\n",
expected, count,

o->kencoder[0]->encoder_type);
         }

}

Regards,
Gabriel.

Quoted 6 lines
Quoted 5 lines
tests/kms_flip.c +0 -0
  			     "dropped frames, expected %d, counted %d, encoder type %d\n",
  			     expected, count, o->kencoder[0]->encoder_type);
  	}
Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#9 lore raw

PATCH v3 1/1 tests/kms_flip: Adjust tolerance when counting frames

In reply to Gabriel Feceoru

basic-flip-vs-wf_vblank subtest sometimes fails asserting counted frames to be aproximately equal with the estimated number.

This is a false negative, one of the reasons being the precision lost when truncating a fractional number.

Fixed this by using floating point arithmetic.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 tests/kms_flip.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/kms_flip.c +4 -4
index eda2fcc..6ec97d0 100644
@@ -1182,13 +1182,13 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 	/* Verify we drop no frames, but only if it's not a TV encoder, since
 	 * those use some funny fake timings behind userspace's back. */
 	if (o->flags & TEST_CHECK_TS && !analog_tv_connector(o)) {
-		int expected;
+		double expected;
 		int count = es->count;
 
 		count *= o->seq_step;
-		expected = elapsed / frame_time(o);
-		igt_assert_f(count >= expected * 99/100 && count <= expected * 101/100,
-			     "dropped frames, expected %d, counted %d, encoder type %d\n",
+		expected = (double)elapsed / frame_time(o);
+		igt_assert_f(fabs((double)count - expected)/expected <= 0.01,
+			     "dropped frames, expected %f, counted %d, encoder type %d\n",
 			     expected, count, o->kencoder[0]->encoder_type);
 	}
 }
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#10 lore raw
Quoted 20 lines · Gabriel Feceoru

basic-flip-vs-wf_vblank subtest sometimes fails asserting counted frames to be aproximately equal with the estimated number.

This is a false negative, one of the reasons being the precision lost when truncating a fractional number.

Fixed this by using floating point arithmetic.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 tests/kms_flip.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/kms_flip.c +0 -0
index eda2fcc..6ec97d0 100644

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1189

Quoted 9 lines
tests/kms_flip.c +1 -2
@@ -1182,13 +1182,13 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 	/* Verify we drop no frames, but only if it's not a TV encoder, since
 	 * those use some funny fake timings behind userspace's back. */
 	if (o->flags & TEST_CHECK_TS && !analog_tv_connector(o)) {
-		int expected;
+		double expected;
 		int count = es->count;
 
 		count *= o->seq_step;
-		expected = elapsed / frame_time(o);

int expected = count * frame_time(o);
igt_assert_f(100 * expected >= elasped * 99 && 100 * count <= expected * 101,

	     "dropped frames, expected %d, counted %d, encoder type %d\n",
	     elapsed / frame_time(o), count, o->kencoder[0]->encoder_type);

if I understood your concern correctly, as the only loss of precison there would be from calculating 'expected'.
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#11 lore raw
Quoted 28 lines · Chris Wilson
Quoted 27 lines · Gabriel Feceoru

basic-flip-vs-wf_vblank subtest sometimes fails asserting counted frames to be aproximately equal with the estimated number.

This is a false negative, one of the reasons being the precision lost when truncating a fractional number.

Fixed this by using floating point arithmetic.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 tests/kms_flip.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/kms_flip.c +1 -1
index eda2fcc..6ec97d0 100644
@@ -1182,13 +1182,13 @@ static void check_final_state(struct test_output *o, struct event_state *es,
 	/* Verify we drop no frames, but only if it's not a TV encoder, since
 	 * those use some funny fake timings behind userspace's back. */
 	if (o->flags & TEST_CHECK_TS && !analog_tv_connector(o)) {
-		int expected;
+		double expected;
 		int count = es->count;
 
Quoted 5 lines

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1189

Quoted 2 lines
tests/kms_flip.c +0 -1
 		count *= o->seq_step;
-		expected = elapsed / frame_time(o);

int expected = count * frame_time(o);
igt_assert_f(100 * expected >= elasped * 99 && 100 * count <= expected * 101,

igt_assert_f(100 * expected >= elasped * 99 && 100 * expected <= elasped * 101, -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#12 lore raw
In reply to Chris Wilson
Quoted 33 lines · Chris Wilson
Quoted 20 lines · Gabriel Feceoru

basic-flip-vs-wf_vblank subtest sometimes fails asserting counted frames to be aproximately equal with the estimated number.

This is a false negative, one of the reasons being the precision lost when truncating a fractional number.

Fixed this by using floating point arithmetic.

Cc Jani Nikula <[email protected]>
Cc Daniel Vetter <[email protected]>
---
  tests/kms_flip.c | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
tests/kms_flip.c +0 -0
index eda2fcc..6ec97d0 100644

tests/kms_flip.c static void check_final_state(struct test_output *o, struct event_state *es, · line 1189

Quoted 9 lines
tests/kms_flip.c +1 -2
@@ -1182,13 +1182,13 @@ static void check_final_state(struct test_output *o, struct event_state *es,
  	/* Verify we drop no frames, but only if it's not a TV encoder, since
  	 * those use some funny fake timings behind userspace's back. */
  	if (o->flags & TEST_CHECK_TS && !analog_tv_connector(o)) {
-		int expected;
+		double expected;
  		int count = es->count;
 
  		count *= o->seq_step;
-		expected = elapsed / frame_time(o);

int expected = count * frame_time(o);
igt_assert_f(100 * expected >= elasped * 99 && 100 * count <= expected * 101,

Quoted 5 lines
	     "dropped frames, expected %d, counted %d, encoder type %d\n",
	     elapsed / frame_time(o), count, o->kencoder[0]->encoder_type);

if I understood your concern correctly, as the only loss of precison there would be from calculating 'expected'.

That would also do it, but with care where to use the cast:

int expected = (int)((double)count * frame_time(o)); vs
int expected = count * (int)frame_time(o);

Gabriel.

Quoted 2 lines

-Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx