Skip to contentSkip to conversation

vnc: Fix tight_detect_smooth_image() for lossless case

PATCH In review Markus Armbruster 2 participants 3 messages Feb 21, 2014 – Mar 10, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vnc: Fix tight_detect_smooth_image() for lossless case

VncTight member uint8_t quality is either (uint8_t)-1 for lossless or less than 10 for lossy.

tight_detect_smooth_image() first promotes it to int, then compares with -1. Always unequal, so we always execute the lossy code. Reads beyond tight_conf[] and returns crap when quality is actually lossless.

Compare to (uint8_t)-1 instead, like we do elsewhere.

Spotted by Coverity.

---
 ui/vnc-enc-tight.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ui/vnc-enc-tight.c +1 -1
index e6966ae..59b59c0 100644
@@ -330,7 +330,7 @@ tight_detect_smooth_image(VncState *vs, int w, int h)
     } else {
         errors = tight_detect_smooth_image16(vs, w, h);
     }
-    if (quality != -1) {
+    if (quality != (uint8_t)-1) {
         return (errors < tight_conf[quality].jpeg_threshold);
     }
     return (errors < tight_conf[compression].gradient_threshold);
Signature · 1 line
1.8.1.4
#1 lore raw

Ping?

Quoted 30 lines · Markus Armbruster

VncTight member uint8_t quality is either (uint8_t)-1 for lossless or less than 10 for lossy.

tight_detect_smooth_image() first promotes it to int, then compares with -1. Always unequal, so we always execute the lossy code. Reads beyond tight_conf[] and returns crap when quality is actually lossless.

Compare to (uint8_t)-1 instead, like we do elsewhere.

Spotted by Coverity.

---
 ui/vnc-enc-tight.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ui/vnc-enc-tight.c +1 -1
index e6966ae..59b59c0 100644
@@ -330,7 +330,7 @@ tight_detect_smooth_image(VncState *vs, int w, int h)
     } else {
         errors = tight_detect_smooth_image16(vs, w, h);
     }
-    if (quality != -1) {
+    if (quality != (uint8_t)-1) {
         return (errors < tight_conf[quality].jpeg_threshold);
     }
     return (errors < tight_conf[compression].gradient_threshold);
#2 lore raw
Quoted 1 line · Markus Armbruster

Ping?

picked up this (and some other vnc bits) vnc pull req will go out soon.

cheers,
Gerd