Skip to contentSkip to conversation

Pixel-perfect frame checks in IGT Chamelium tests and CRC

DISCUSSION Paul Kocialkowski 4 participants 8 messages Jun 15, 2017 – Jun 16, 2017

lore
Share & tools
lore
#0 lore raw

Hi,

So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium and compares it pixel-to-pixel with the cairo reference, which works well for DP/HDMI.

For VGA, this is probably not the case (because the link is analogue). In that case, I will look into implementing some fuzzy testing, probably inspired by what piglit (probably) does to compare output frames with references.

For pixel-perfect testing, grabbing a full frame and testing it with memcmp comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium also provides a CRC mechanism that is faster and does not require retrieving the frame, that IGT currently also supports. It compares the CRC calculated by the Chamelium (implemented in the HDL) with a hardcoded reference value.

This approach currently fails for me (the values I get don't match the hardcoded reference). There are reasons why it is not really reasonable: fonts rendering may change between machines (e.g. use of anti-aliasing) and cairo version changes could introduce slight rendering changes too (not to mention changes in the test pattern itself). So instead of comparing the CRC with a hardcoded reference value, I think it would make a lot more sense to actually calculate the CRC based on the cairo image that is the actual reference (and that we should assume may change between runs/machines).

I am currently looking into the CRC calculation mechanism used by the Chamelium and trying to reproduce it in C code. Is this a known algorithm for which a reference/optimized implementation exists, or something custom that the folks over at Google came up with?

Any thoughts, comments or suggestions?

Cheers!

Signature · 6 lines
Paul Kocialkowski <[email protected]>
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

JFYI: Hardcoded CRCs are fine I'm pretty sure, but I might be wrong. As well, put the chamelium in a metal box. The way the IO board is hooked up is not really the way something running DP should be hooked up, so it's very susceptible to electromagnetic interference. This will usually cause CRC mismatches with DP.

This being said however, I think we should have some better functions for the chamelium for doing frame comparisons, mainly something that does fuzzy frame comparisons for stuff like VGA.

Quoted 50 lines · Paul Kocialkowski

Hi,

So far, there are two ways of testing for pixel-perfect frames using the
Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.

For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.

For pixel-perfect testing, grabbing a full frame and testing it with memcmp
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.

This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version
changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded
reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we
should assume may change between runs/machines).

I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?

Quoted 4 lines

Any thoughts, comments or suggestions?

Cheers!

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#2 lore raw
Quoted 3 lines · Lyude Paul

This being said however, I think we should have some better functions for the chamelium for doing frame comparisons, mainly something that does fuzzy frame comparisons for stuff like VGA.

Something that takes into account dithering might be of use too.

BR,
Jani.

Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw
In reply to Lyude Paul
Quoted 5 lines · Lyude Paul

JFYI: Hardcoded CRCs are fine I'm pretty sure, but I might be wrong. As well, put the chamelium in a metal box. The way the IO board is hooked up is not really the way something running DP should be hooked up, so it's very susceptible to electromagnetic interference. This will usually cause CRC mismatches with DP.

I don't think that is the case here, as pixel-to-pixel comparison succeeds reliably. I also get the same hashes (that don't match the hardcoded ones) everytime I run the test. I think the reference picture I have is just different from the one you hardcoded the CRCs for, as I pointed out (font rendering cannot be trusted, among other things that can change).

This is why I think we should calculate the CRC instead of using hardcoded ones.

Quoted 3 lines

This being said however, I think we should have some better functions for the chamelium for doing frame comparisons, mainly something that does fuzzy frame comparisons for stuff like VGA.

Agreed. I will most likely look into that next week.

Quoted 55 lines · Paul Kocialkowski
Quoted 54 lines

Hi,

So far, there are two ways of testing for pixel-perfect frames using the
Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.

For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.

For pixel-perfect testing, grabbing a full frame and testing it with memcmp
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.

This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version
changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded
reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we
should assume may change between runs/machines).

I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?

Any thoughts, comments or suggestions?

Cheers!

Signature · 6 lines
Paul Kocialkowski <[email protected]>
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw
In reply to Paul Kocialkowski

Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:

Quoted 47 lines

Hi,

So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.

For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.

For pixel-perfect testing, grabbing a full frame and testing it with memcmp
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.

This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we should assume may change between runs/machines).

I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?

Quoted 2 lines

Any thoughts, comments or suggestions?

I feel bad about the stupid hash algorithm I came up with, but here is the document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-OXtITfiomCb1HBKN07T-POlY/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the whole frame (i.e. cropping before hashing).

Thanks,
Chih-Chung Chang

Quoted 7 lines

Cheers!

Signature · 2 lines
Paul Kocialkowski <[email protected]> 
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo 

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#5 lore raw

Hi,

Quoted 45 lines · On Thu, 2017-06-15 at 21:08 -0700, [email protected] wrote:

Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:

Quoted 44 lines

Hi,

So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.

For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.

For pixel-perfect testing, grabbing a full frame and testing it with memcmp comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.

This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we should assume may change between runs/machines).

I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a reference/optimized implementation exists, or something custom that the folks
over at Google came up with?

Quoted 8 lines
Quoted 1 line

Any thoughts, comments or suggestions?

I feel bad about the stupid hash algorithm I came up with, but here is the document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-OXtITfiomCb1HBKN07T-POl
Y/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the whole frame (i.e. cropping before hashing).

Thanks for the link to the relevant documentation! I have been following it closely and came up with the following implementation, which does not produce the same result. Would you have any idea of what I'm doing wrong?

That function is called with m = 4 and k = { 0, 1, 2, 3 }. I am using a single- color-frame and the 4 hash16 I get from the Chamelium are identical. They are however different from the 4 (also identical) that I get from :

int hash16(unsigned char *buffer, int width, int height, int k, int m)

{
	unsigned char r, g, b;
	long int sum = 0;
	int count = 0;
	int index;
	int hash;
	int value;
	int i;

	for (i=0; i < width * height; i++) {
		if ((i % m) != k)
			continue;

		index = i * 3;

		r = buffer[index + 0];
		g = buffer[index + 1];
		b = buffer[index + 2];

		value = r | (g << 8) | (b << 16);
		sum += ++count * value;
	}

	hash = ((sum >> 48) ^ (sum >> 32) ^ (sum >> 16) ^ (sum >> 0)) & 0xffff;

	return hash;
}

I am certain that the r, g, b values are correct.

Thanks!

Signature · 6 lines
Paul Kocialkowski <[email protected]>
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw
Quoted 97 lines · Paul Kocialkowski

Hi,

Quoted 4 lines · On Thu, 2017-06-15 at 21:08 -0700, [email protected] wrote:

Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:

Quoted 3 lines

Hi,

So far, there are two ways of testing for pixel-perfect frames using

the

Quoted 3 lines
Quoted 3 lines

Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works

well

Quoted 4 lines
Quoted 4 lines

for
DP/HDMI.

For VGA, this is probably not the case (because the link is analogue).

In

Quoted 2 lines
Quoted 2 lines

that
case, I will look into implementing some fuzzy testing, probably

inspired

Quoted 4 lines
Quoted 4 lines

by
what piglit (probably) does to compare output frames with references.

For pixel-perfect testing, grabbing a full frame and testing it with

memcmp

Quoted 3 lines
Quoted 3 lines

comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require

retrieving

Quoted 2 lines
Quoted 2 lines

the
frame, that IGT currently also supports. It compares the CRC

calculated by

Quoted 8 lines
Quoted 8 lines

the
Chamelium (implemented in the HDL) with a hardcoded reference value.

This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo

version

Quoted 1 line
Quoted 1 line

changes could introduce slight rendering changes too (not to mention

changes

Quoted 2 lines
Quoted 2 lines

in
the test pattern itself). So instead of comparing the CRC with a

hardcoded

Quoted 3 lines
Quoted 3 lines

reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and

that we

Quoted 5 lines
Quoted 5 lines

should assume may change between runs/machines).

I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for

which a

Quoted 7 lines
Quoted 5 lines

reference/optimized implementation exists, or something custom that the folks
over at Google came up with?

Any thoughts, comments or suggestions?

I feel bad about the stupid hash algorithm I came up with, but here is

the

Quoted 2 lines

document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-

OXtITfiomCb1HBKN07T-POl

Quoted 2 lines

Y/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the whole

frame

Quoted 1 line

(i.e. cropping before hashing).

Thanks for the link to the relevant documentation! I have been following it closely and came up with the following implementation, which does not produce
the same result. Would you have any idea of what I'm doing wrong?

That function is called with m = 4 and k = { 0, 1, 2, 3 }. I am using a single-
color-frame and the 4 hash16 I get from the Chamelium are identical. They are
however different from the 4 (also identical) that I get from :

int hash16(unsigned char *buffer, int width, int height, int k, int m)

{
        unsigned char r, g, b;
        long int sum = 0;
        int count = 0;
        int index;
        int hash;
        int value;
        int i;

        for (i=0; i < width * height; i++) {
                if ((i % m) != k)
                        continue;
Quoted 9 lines
                index = i * 3;

                r = buffer[index + 0];
                g = buffer[index + 1];
                b = buffer[index + 2];

                value = r | (g << 8) | (b << 16);
                sum += ++count * value;

Maybe this multiplication overflows?

Quoted 16 lines
        }

        hash = ((sum >> 48) ^ (sum >> 32) ^ (sum >> 16) ^ (sum >> 0)) &

0xffff;

        return hash;
}

I am certain that the r, g, b values are correct.

Thanks!

--
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

Quoted 15 lines

--
N.B. This list is shared with folks outside Google so please make sure no confidential information is being discussed.
---
You received this message because you are subscribed to a topic in the Google Groups "chamelium-external" group.
To unsubscribe from this topic, visit https://groups.google.com/a/
google.com/d/topic/chamelium-external/ITCjBMJHfGM/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected].
To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/
google.com/d/msgid/chamelium-external/1497605300.1451.9. camel%40linux.intel.com.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#7 lore raw

Hey,

Quoted 99 lines · Chih-Chung Chang
Quoted 97 lines · Paul Kocialkowski

Hi,

Quoted 7 lines · On Thu, 2017-06-15 at 21:08 -0700, [email protected] wrote:

Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:

Quoted 6 lines

Hi,

So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works

well

Quoted 4 lines
Quoted 4 lines

for
DP/HDMI.

For VGA, this is probably not the case (because the link is analogue).

In

Quoted 2 lines
Quoted 2 lines

that
case, I will look into implementing some fuzzy testing, probably

inspired

Quoted 4 lines
Quoted 4 lines

by
what piglit (probably) does to compare output frames with references.

For pixel-perfect testing, grabbing a full frame and testing it with

memcmp

Quoted 3 lines
Quoted 3 lines

comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require

retrieving

Quoted 2 lines
Quoted 2 lines

the
frame, that IGT currently also supports. It compares the CRC calculated

by

Quoted 8 lines
Quoted 8 lines

the
Chamelium (implemented in the HDL) with a hardcoded reference value.

This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo

version

Quoted 1 line
Quoted 1 line

changes could introduce slight rendering changes too (not to mention

changes

Quoted 2 lines
Quoted 2 lines

in
the test pattern itself). So instead of comparing the CRC with a

hardcoded

Quoted 3 lines
Quoted 3 lines

reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that

we

Quoted 5 lines
Quoted 5 lines

should assume may change between runs/machines).

I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for

which a

Quoted 9 lines
Quoted 5 lines

reference/optimized implementation exists, or something custom that the folks
over at Google came up with?

Any thoughts, comments or suggestions?

I feel bad about the stupid hash algorithm I came up with, but here is the document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-OXtITfiomCb1HBKN07T

-POl

Quoted 2 lines

Y/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the whole

frame

Quoted 1 line

(i.e. cropping before hashing).

Thanks for the link to the relevant documentation! I have been following it closely and came up with the following implementation, which does not produce
the same result. Would you have any idea of what I'm doing wrong?

That function is called with m = 4 and k = { 0, 1, 2, 3 }. I am using a single-
color-frame and the 4 hash16 I get from the Chamelium are identical. They are
however different from the 4 (also identical) that I get from :

int hash16(unsigned char *buffer, int width, int height, int k, int m)

{
        unsigned char r, g, b;
        long int sum = 0;
        int count = 0;
        int index;
        int hash;
        int value;
        int i;

        for (i=0; i < width * height; i++) {
                if ((i % m) != k)
                        continue;

                index = i * 3;
Quoted 8 lines
Quoted 6 lines
                r = buffer[index + 0];
                g = buffer[index + 1];
                b = buffer[index + 2];

                value = r | (g << 8) | (b << 16);
                sum += ++count * value;

Maybe this multiplication overflows?

Good pick, that was it! Now the CRC is the same as the one obtained from the Chamelium.

Thanks a lot!

Quoted 31 lines
Quoted 29 lines
        }

        hash = ((sum >> 48) ^ (sum >> 32) ^ (sum >> 16) ^ (sum >> 0)) &

0xffff;

        return hash;
}

I am certain that the r, g, b values are correct.

Thanks!

--
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo

--
N.B. This list is shared with folks outside Google so please make sure no confidential information is being discussed.
---
You received this message because you are subscribed to a topic in the Google Groups "chamelium-external" group.
To unsubscribe from this topic, visit https://groups.google.com/a/google.com
/d/topic/chamelium-external/ITCjBMJHfGM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chameliu [email protected].
To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/google.
com/d/msgid/chamelium-external/1497605300.1451.9.camel%40linux.intel.com.

Signature · 6 lines
Paul Kocialkowski <[email protected]>
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx