vnc: Fix tight_detect_smooth_image() for lossless case
PATCH 1/1 vnc: Fix tight_detect_smooth_image() for lossless case
VncTight member uint8_t quality is either (uint8_t)-1 for lossless or less than 10 for lossy.
tight_detect_smooth_image() first promotes it to int, then compares with -1. Always unequal, so we always execute the lossy code. Reads beyond tight_conf[] and returns crap when quality is actually lossless.
Compare to (uint8_t)-1 instead, like we do elsewhere.
Spotted by Coverity.
--- ui/vnc-enc-tight.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ui/vnc-enc-tight.c +1 -1
@@ -330,7 +330,7 @@ tight_detect_smooth_image(VncState *vs, int w, int h) }else { errors =tight_detect_smooth_image16 (vs, w, h); } -if (quality !=-1 ) { +if (quality != (uint8_t )-1 ) {return (errors < tight_conf[quality].jpeg_threshold); }return (errors < tight_conf[compression].gradient_threshold);
Signature · 1 line
1.8.1.4
Ping?
Quoted 30 lines · Markus Armbruster
VncTight member
uint8_tquality is either (uint8_t)-1 for lossless or less than 10 for lossy.
tight_detect_smooth_image()first promotes it to int, then compares with -1. Always unequal, so we always execute the lossy code. Reads beyond tight_conf[] and returns crap when quality is actually lossless.Compare to (uint8_t)-1 instead, like we do elsewhere.
Spotted by Coverity.
Signed-off-by Markus Armbruster <[email protected]>--- ui/vnc-enc-tight.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)ui/vnc-enc-tight.c +1
-1@@ -330,7 +330,7 @@ tight_detect_smooth_image(VncState *vs, int w, int h) }else { errors =tight_detect_smooth_image16 (vs, w, h); } -if (quality !=-1 ) { +if (quality != (uint8_t )-1 ) {return (errors < tight_conf[quality].jpeg_threshold); }return (errors < tight_conf[compression].gradient_threshold);
Quoted 1 line · Markus Armbruster
Ping?
picked up this (and some other vnc bits) vnc pull req will go out soon.
cheers,
Gerd