Pixel-perfect frame checks in IGT Chamelium tests and CRC
Hi,
So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium and compares it pixel-to-pixel with the cairo reference, which works well for DP/HDMI.
For VGA, this is probably not the case (because the link is analogue). In that case, I will look into implementing some fuzzy testing, probably inspired by what piglit (probably) does to compare output frames with references.
For pixel-perfect testing, grabbing a full frame and testing it with memcmp comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium also provides a CRC mechanism that is faster and does not require retrieving the frame, that IGT currently also supports. It compares the CRC calculated by the Chamelium (implemented in the HDL) with a hardcoded reference value.
This approach currently fails for me (the values I get don't match the hardcoded reference). There are reasons why it is not really reasonable: fonts rendering may change between machines (e.g. use of anti-aliasing) and cairo version changes could introduce slight rendering changes too (not to mention changes in the test pattern itself). So instead of comparing the CRC with a hardcoded reference value, I think it would make a lot more sense to actually calculate the CRC based on the cairo image that is the actual reference (and that we should assume may change between runs/machines).
I am currently looking into the CRC calculation mechanism used by the Chamelium and trying to reproduce it in C code. Is this a known algorithm for which a reference/optimized implementation exists, or something custom that the folks over at Google came up with?
Any thoughts, comments or suggestions?
Cheers!
Signature · 6 lines
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
JFYI: Hardcoded CRCs are fine I'm pretty sure, but I might be wrong. As well, put the chamelium in a metal box. The way the IO board is hooked up is not really the way something running DP should be hooked up, so it's very susceptible to electromagnetic interference. This will usually cause CRC mismatches with DP.
This being said however, I think we should have some better functions for the chamelium for doing frame comparisons, mainly something that does fuzzy frame comparisons for stuff like VGA.
Quoted 50 lines · Paul Kocialkowski
Hi,
So far, there are two ways of testing for pixel-perfect frames using the
Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.For pixel-perfect testing, grabbing a full frame and testing it with memcmp
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version
changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded
reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we
should assume may change between runs/machines).I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?
Quoted 4 lines
Any thoughts, comments or suggestions?
Cheers!
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 3 lines · Lyude Paul
This being said however, I think we should have some better functions for the chamelium for doing frame comparisons, mainly something that does fuzzy frame comparisons for stuff like VGA.
Something that takes into account dithering might be of use too.
BR,
Jani.
Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:
Quoted 47 lines
Hi,
So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.For pixel-perfect testing, grabbing a full frame and testing it with memcmp
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we should assume may change between runs/machines).I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?
Quoted 2 lines
Any thoughts, comments or suggestions?
I feel bad about the stupid hash algorithm I came up with, but here is the document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-OXtITfiomCb1HBKN07T-POlY/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the whole frame (i.e. cropping before hashing).
Thanks,
Chih-Chung Chang
Quoted 7 lines
Cheers!
Signature · 2 lines
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 5 lines · Lyude Paul
JFYI: Hardcoded CRCs are fine I'm pretty sure, but I might be wrong. As well, put the chamelium in a metal box. The way the IO board is hooked up is not really the way something running DP should be hooked up, so it's very susceptible to electromagnetic interference. This will usually cause CRC mismatches with DP.
I don't think that is the case here, as pixel-to-pixel comparison succeeds reliably. I also get the same hashes (that don't match the hardcoded ones) everytime I run the test. I think the reference picture I have is just different from the one you hardcoded the CRCs for, as I pointed out (font rendering cannot be trusted, among other things that can change).
This is why I think we should calculate the CRC instead of using hardcoded ones.
Quoted 3 lines
This being said however, I think we should have some better functions for the chamelium for doing frame comparisons, mainly something that does fuzzy frame comparisons for stuff like VGA.
Agreed. I will most likely look into that next week.
Quoted 55 lines · Paul Kocialkowski
Quoted 54 lines
Hi,
So far, there are two ways of testing for pixel-perfect frames using the
Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.For pixel-perfect testing, grabbing a full frame and testing it with memcmp
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version
changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded
reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we
should assume may change between runs/machines).I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?Any thoughts, comments or suggestions?
Cheers!
Signature · 6 lines
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Hi,
Quoted 45 lines · On Thu, 2017-06-15 at 21:08 -0700, [email protected] wrote:
Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:
Quoted 44 lines
Hi,
So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which works well for
DP/HDMI.For VGA, this is probably not the case (because the link is analogue). In that
case, I will look into implementing some fuzzy testing, probably inspired by
what piglit (probably) does to compare output frames with references.For pixel-perfect testing, grabbing a full frame and testing it with memcmp comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not require retrieving the
frame, that IGT currently also supports. It compares the CRC calculated by the
Chamelium (implemented in the HDL) with a hardcoded reference value.This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairo version changes could introduce slight rendering changes too (not to mention changes in
the test pattern itself). So instead of comparing the CRC with a hardcoded reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and that we should assume may change between runs/machines).I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm for which a reference/optimized implementation exists, or something custom that the folks
over at Google came up with?
Quoted 8 lines
Quoted 1 line
Any thoughts, comments or suggestions?
I feel bad about the stupid hash algorithm I came up with, but here is the document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-OXtITfiomCb1HBKN07T-POl
Y/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the whole frame (i.e. cropping before hashing).
Thanks for the link to the relevant documentation! I have been following it closely and came up with the following implementation, which does not produce the same result. Would you have any idea of what I'm doing wrong?
That function is called with m = 4 and k = { 0, 1, 2, 3 }. I am using a single- color-frame and the 4 hash16 I get from the Chamelium are identical. They are however different from the 4 (also identical) that I get from :
int hash16(unsigned char *buffer, int width, int height, int k, int m)
{
unsigned char r, g, b;
long int sum = 0;
int count = 0;
int index;
int hash;
int value;
int i;
for (i=0; i < width * height; i++) {
if ((i % m) != k)
continue;
index = i * 3;
r = buffer[index + 0];
g = buffer[index + 1];
b = buffer[index + 2];
value = r | (g << 8) | (b << 16);
sum += ++count * value;
}
hash = ((sum >> 48) ^ (sum >> 32) ^ (sum >> 16) ^ (sum >> 0)) & 0xffff;
return hash;
}I am certain that the r, g, b values are correct.
Thanks!
Signature · 6 lines
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 97 lines · Paul Kocialkowski
Hi,
Quoted 4 lines · On Thu, 2017-06-15 at 21:08 -0700, [email protected] wrote:
Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:
Quoted 3 lines
Hi,
So far, there are two ways of testing for pixel-perfect frames using
the
Quoted 3 lines
Quoted 3 lines
Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which workswell
Quoted 4 lines
Quoted 4 lines
for
DP/HDMI.For VGA, this is probably not the case (because the link is analogue).
In
Quoted 2 lines
Quoted 2 lines
that
case, I will look into implementing some fuzzy testing, probablyinspired
Quoted 4 lines
Quoted 4 lines
by
what piglit (probably) does to compare output frames with references.For pixel-perfect testing, grabbing a full frame and testing it with
memcmp
Quoted 3 lines
Quoted 3 lines
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not requireretrieving
Quoted 2 lines
Quoted 2 lines
the
frame, that IGT currently also supports. It compares the CRCcalculated by
Quoted 8 lines
Quoted 8 lines
the
Chamelium (implemented in the HDL) with a hardcoded reference value.This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairoversion
Quoted 1 line
Quoted 1 line
changes could introduce slight rendering changes too (not to mention
changes
Quoted 2 lines
Quoted 2 lines
in
the test pattern itself). So instead of comparing the CRC with ahardcoded
Quoted 3 lines
Quoted 3 lines
reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (andthat we
Quoted 5 lines
Quoted 5 lines
should assume may change between runs/machines).
I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm forwhich a
Quoted 7 lines
Quoted 5 lines
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?Any thoughts, comments or suggestions?
I feel bad about the stupid hash algorithm I came up with, but here is
the
Quoted 2 lines
document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-OXtITfiomCb1HBKN07T-POl
Quoted 2 lines
Y/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the wholeframe
Quoted 1 line
(i.e. cropping before hashing).
Thanks for the link to the relevant documentation! I have been following it closely and came up with the following implementation, which does not produce
the same result. Would you have any idea of what I'm doing wrong?That function is called with m = 4 and k = { 0, 1, 2, 3 }. I am using a single-
color-frame and the 4 hash16 I get from the Chamelium are identical. They are
however different from the 4 (also identical) that I get from :int
hash16(unsigned char *buffer, int width, int height, int k, int m){ unsigned char r, g, b; long int sum = 0; int count = 0; int index; int hash; int value; int i; for (i=0; i < width * height; i++) { if ((i % m) != k) continue;
Quoted 9 lines
index = i * 3;
r = buffer[index + 0];
g = buffer[index + 1];
b = buffer[index + 2];
value = r | (g << 8) | (b << 16);
sum += ++count * value;Maybe this multiplication overflows?
Quoted 16 lines
} hash = ((sum >> 48) ^ (sum >> 32) ^ (sum >> 16) ^ (sum >> 0)) &
0xffff;return hash; }I am certain that the r, g, b values are correct.
Thanks!
--
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
Quoted 15 lines
--
N.B. This list is shared with folks outside Google so please make sure no confidential information is being discussed.
---
You received this message because you are subscribed to a topic in the Google Groups "chamelium-external" group.
To unsubscribe from this topic, visit https://groups.google.com/a/
google.com/d/topic/chamelium-external/ITCjBMJHfGM/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected].
To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/
google.com/d/msgid/chamelium-external/1497605300.1451.9. camel%40linux.intel.com.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Hey,
Quoted 99 lines · Chih-Chung Chang
Quoted 97 lines · Paul Kocialkowski
Hi,
Quoted 7 lines · On Thu, 2017-06-15 at 21:08 -0700, [email protected] wrote:
Paul Kocialkowski於 2017年6月15日星期四 UTC+8下午9時57分09秒寫道:
Quoted 6 lines
Hi,
So far, there are two ways of testing for pixel-perfect frames using the Chamelium that are in IGT. The first one grabs a full frame from the Chamelium
and compares it pixel-to-pixel with the cairo reference, which workswell
Quoted 4 lines
Quoted 4 lines
for
DP/HDMI.For VGA, this is probably not the case (because the link is analogue).
In
Quoted 2 lines
Quoted 2 lines
that
case, I will look into implementing some fuzzy testing, probablyinspired
Quoted 4 lines
Quoted 4 lines
by
what piglit (probably) does to compare output frames with references.For pixel-perfect testing, grabbing a full frame and testing it with
memcmp
Quoted 3 lines
Quoted 3 lines
comes with a significant time penalty (about 2 seconds for 1080p). The Chamelium
also provides a CRC mechanism that is faster and does not requireretrieving
Quoted 2 lines
Quoted 2 lines
the
frame, that IGT currently also supports. It compares the CRC calculatedby
Quoted 8 lines
Quoted 8 lines
the
Chamelium (implemented in the HDL) with a hardcoded reference value.This approach currently fails for me (the values I get don't match the hardcoded
reference). There are reasons why it is not really reasonable: fonts rendering
may change between machines (e.g. use of anti-aliasing) and cairoversion
Quoted 1 line
Quoted 1 line
changes could introduce slight rendering changes too (not to mention
changes
Quoted 2 lines
Quoted 2 lines
in
the test pattern itself). So instead of comparing the CRC with ahardcoded
Quoted 3 lines
Quoted 3 lines
reference value, I think it would make a lot more sense to actually calculate
the CRC based on the cairo image that is the actual reference (and thatwe
Quoted 5 lines
Quoted 5 lines
should assume may change between runs/machines).
I am currently looking into the CRC calculation mechanism used by the Chamelium
and trying to reproduce it in C code. Is this a known algorithm forwhich a
Quoted 9 lines
Quoted 5 lines
reference/optimized implementation exists, or something custom that the folks
over at Google came up with?Any thoughts, comments or suggestions?
I feel bad about the stupid hash algorithm I came up with, but here is the document:
https://docs.google.com/document/d/1_HjEMA8fBoHkUbpUZq-OXtITfiomCb1HBKN07T-POl
Quoted 2 lines
Y/edit#heading=h.jqek3kkh9qjm
You can also ask it to hash just part of the frame instead of the wholeframe
Quoted 1 line
(i.e. cropping before hashing).
Thanks for the link to the relevant documentation! I have been following it closely and came up with the following implementation, which does not produce
the same result. Would you have any idea of what I'm doing wrong?That function is called with m = 4 and k = { 0, 1, 2, 3 }. I am using a single-
color-frame and the 4 hash16 I get from the Chamelium are identical. They are
however different from the 4 (also identical) that I get from :int
hash16(unsigned char *buffer, int width, int height, int k, int m){ unsigned char r, g, b; long int sum = 0; int count = 0; int index; int hash; int value; int i; for (i=0; i < width * height; i++) { if ((i % m) != k) continue; index = i * 3;
Quoted 8 lines
Quoted 6 lines
r = buffer[index + 0]; g = buffer[index + 1]; b = buffer[index + 2]; value = r | (g << 8) | (b << 16); sum += ++count * value;Maybe this multiplication overflows?
Good pick, that was it! Now the CRC is the same as the one obtained from the Chamelium.
Thanks a lot!
Quoted 31 lines
Quoted 29 lines
} hash = ((sum >> 48) ^ (sum >> 32) ^ (sum >> 16) ^ (sum >> 0)) &
0xffff;return hash; }I am certain that the r, g, b values are correct.
Thanks!
--
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo--
N.B. This list is shared with folks outside Google so please make sure no confidential information is being discussed.
---
You received this message because you are subscribed to a topic in the Google Groups "chamelium-external" group.
To unsubscribe from this topic, visit https://groups.google.com/a/google.com
/d/topic/chamelium-external/ITCjBMJHfGM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chameliu [email protected].
To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/google.
com/d/msgid/chamelium-external/1497605300.1451.9.camel%40linux.intel.com.
Signature · 6 lines
Paul Kocialkowski <[email protected]> Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx