Skip to contentSkip to conversation

vnc: support some new extensions.

PATCH In review Gerd Hoffmann 3 participants 9 messages Jan 12, 2021 – Jan 14, 2021

lore
Share & tools
lore
#0 lore raw

v3 cover vnc: support some new extensions.

The rfb standard keeps envolving. While the official spec is kind of frozen since a decade or so the community maintains an updated version of the spec at:
https://github.com/rfbproto/rfbproto/

This series adds support for two new extensions from that spec: alpha cursor and extended desktop resize.

alpha cursor allows a full alpha channel for the cursor image (unlike the rich cursor extension which has only a bitmask for transparency).

extended desktop resize makes the desktop-resize work both ways, i.e. we can not only signal guest display resolution changes to the vnc client but also vnc client window size changes to the guest.

Tested with tigervnc.

Support for gtk-vnc is in progress.

v3:
- dropped cursor patches (already merged).
- fix client initialization (reply to update request not set encoding).

v2:
- dropped qxl bits (will be a separate patch series).
- use error codes for desktop resize responses.
- little tweaks here and there.
- pick up some review tags.

Gerd Hoffmann (3):
vnc: move check into vnc_cursor_define
vnc: move initialization to framebuffer_update_request
vnc: add support for extended desktop resize

 ui/vnc.h |  2 ++
 ui/vnc.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 74 insertions(+), 13 deletions(-)
Signature · 1 line
2.29.2
#1 lore raw

PATCH v3 1/3 vnc: move check into vnc_cursor_define

Move the check whenever a cursor exists into the vnc_cursor_define() function so callers don't have to do it.

---
 ui/vnc.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
ui/vnc.c +6 -6
index 7452ac7df2ce..84c4972b895b 100644
@@ -792,9 +792,7 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
     QTAILQ_FOREACH(vs, &vd->clients, next) {
         vnc_colordepth(vs);
         vnc_desktop_resize(vs);
-        if (vs->vd->cursor) {
-            vnc_cursor_define(vs);
-        }
+        vnc_cursor_define(vs);
         memset(vs->dirty, 0x00, sizeof(vs->dirty));
         vnc_set_area_dirty(vs->dirty, vd, 0, 0,
                            vnc_width(vd),
@@ -928,6 +926,10 @@ static int vnc_cursor_define(VncState *vs)
     QEMUCursor *c = vs->vd->cursor;
     int isize;
 
+    if (!vs->vd->cursor) {
+        return -1;
+    }
+
     if (vnc_has_feature(vs, VNC_FEATURE_ALPHA_CURSOR)) {
         vnc_lock_output(vs);
         vnc_write_u8(vs,  VNC_MSG_SERVER_FRAMEBUFFER_UPDATE);
@@ -2137,9 +2139,7 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
     vnc_desktop_resize(vs);
     check_pointer_type_change(&vs->mouse_mode_notifier, NULL);
     vnc_led_state_change(vs);
-    if (vs->vd->cursor) {
-        vnc_cursor_define(vs);
-    }
+    vnc_cursor_define(vs);
 }
 
 static void set_pixel_conversion(VncState *vs)
Signature · 1 line
2.29.2
Reviewed-by on v3 1/3
#2 lore raw

Re: [PATCH v3 1/3] vnc: move check into vnc_cursor_define

Quoted 6 lines · Gerd Hoffmann

Move the check whenever a cursor exists into the vnc_cursor_define() function so callers don't have to do it.

thanks

---

Quoted 45 lines
 ui/vnc.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
ui/vnc.c +6 -6
index 7452ac7df2ce..84c4972b895b 100644
@@ -792,9 +792,7 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
     QTAILQ_FOREACH(vs, &vd->clients, next) {
         vnc_colordepth(vs);
         vnc_desktop_resize(vs);
-        if (vs->vd->cursor) {
-            vnc_cursor_define(vs);
-        }
+        vnc_cursor_define(vs);
         memset(vs->dirty, 0x00, sizeof(vs->dirty));
         vnc_set_area_dirty(vs->dirty, vd, 0, 0,
                            vnc_width(vd),
@@ -928,6 +926,10 @@ static int vnc_cursor_define(VncState *vs)
     QEMUCursor *c = vs->vd->cursor;
     int isize;
 
+    if (!vs->vd->cursor) {
+        return -1;
+    }
+
     if (vnc_has_feature(vs, VNC_FEATURE_ALPHA_CURSOR)) {
         vnc_lock_output(vs);
         vnc_write_u8(vs,  VNC_MSG_SERVER_FRAMEBUFFER_UPDATE);
@@ -2137,9 +2139,7 @@ static void set_encodings(VncState *vs, int32_t
 *encodings, size_t n_encodings)
     vnc_desktop_resize(vs);
     check_pointer_type_change(&vs->mouse_mode_notifier, NULL);
     vnc_led_state_change(vs);
-    if (vs->vd->cursor) {
-        vnc_cursor_define(vs);
-    }
+    vnc_cursor_define(vs);
 }
 

static void set_pixel_conversion(VncState *vs)
--
2.29.2

Reviewed-by on v3 1/3
#3 lore raw

Re: [PATCH v3 1/3] vnc: move check into vnc_cursor_define

In reply to Gerd Hoffmann
Quoted 8 lines · Gerd Hoffmann

Move the check whenever a cursor exists into the vnc_cursor_define() function so callers don't have to do it.

---
 ui/vnc.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
Quoted 17 lines
ui/vnc.c +1 -3
index 7452ac7df2ce..84c4972b895b 100644
@@ -792,9 +792,7 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
     QTAILQ_FOREACH(vs, &vd->clients, next) {
         vnc_colordepth(vs);
         vnc_desktop_resize(vs);
-        if (vs->vd->cursor) {
-            vnc_cursor_define(vs);
-        }
+        vnc_cursor_define(vs);
         memset(vs->dirty, 0x00, sizeof(vs->dirty));
         vnc_set_area_dirty(vs->dirty, vd, 0, 0,
                            vnc_width(vd),

ui/vnc.c static int vnc_cursor_define(VncState *vs) · line 931

Quoted 6 lines
ui/vnc.c +3 -0
@@ -928,6 +926,10 @@ static int vnc_cursor_define(VncState *vs)
     QEMUCursor *c = vs->vd->cursor;
     int isize;
 
+    if (!vs->vd->cursor) {
+        return -1;
+    }

None of the callers check the return value, so we could just make this function void, as a future cleanup patch.

Quoted 19 lines
ui/vnc.c +2 -3
+
     if (vnc_has_feature(vs, VNC_FEATURE_ALPHA_CURSOR)) {
         vnc_lock_output(vs);
         vnc_write_u8(vs,  VNC_MSG_SERVER_FRAMEBUFFER_UPDATE);
@@ -2137,9 +2139,7 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
     vnc_desktop_resize(vs);
     check_pointer_type_change(&vs->mouse_mode_notifier, NULL);
     vnc_led_state_change(vs);
-    if (vs->vd->cursor) {
-        vnc_cursor_define(vs);
-    }
+    vnc_cursor_define(vs);
 }
 
 static void set_pixel_conversion(VncState *vs)
Signature · 1 line
2.29.2

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw

PATCH v3 2/3 vnc: move initialization to framebuffer_update_request

In reply to Gerd Hoffmann

qemu sends various state info like current cursor shape to newly connected clients in response to a set_encoding message. This is not correct according to the rfb spec. Send that information in response to a full (incremental=0) framebuffer update request instead. Also send the resize information unconditionally, not only in case of an actual server-side change.

This makes the qemu vnc server conform to the spec and allows clients to request the complete vnc server state without reconnect.

---
 ui/vnc.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
ui/vnc.c +4 -7
index 84c4972b895b..8df63b349b38 100644
@@ -660,10 +660,6 @@ static void vnc_desktop_resize(VncState *vs)
     if (vs->ioc == NULL || !vnc_has_feature(vs, VNC_FEATURE_RESIZE)) {
         return;
     }
-    if (vs->client_width == pixman_image_get_width(vs->vd->server) &&
-        vs->client_height == pixman_image_get_height(vs->vd->server)) {
-        return;
-    }
 
     assert(pixman_image_get_width(vs->vd->server) < 65536 &&
            pixman_image_get_width(vs->vd->server) >= 0);
@@ -2013,6 +2009,10 @@ static void framebuffer_update_request(VncState *vs, int incremental,
     } else {
         vs->update = VNC_STATE_UPDATE_FORCE;
         vnc_set_area_dirty(vs->dirty, vs->vd, x, y, w, h);
+        vnc_colordepth(vs);
+        vnc_desktop_resize(vs);
+        vnc_led_state_change(vs);
+        vnc_cursor_define(vs);
     }
 }
 
@@ -2136,10 +2136,7 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
             break;
         }
     }
-    vnc_desktop_resize(vs);
     check_pointer_type_change(&vs->mouse_mode_notifier, NULL);
-    vnc_led_state_change(vs);
-    vnc_cursor_define(vs);
 }
 
 static void set_pixel_conversion(VncState *vs)
Signature · 1 line
2.29.2
Reviewed-by on v3 2/3
#5 lore raw

Re: [PATCH v3 2/3] vnc: move initialization to framebuffer_update_request

Quoted 30 lines · Gerd Hoffmann

qemu sends various state info like current cursor shape to newly connected clients in response to a set_encoding message. This is not correct according to the rfb spec. Send that information in response to a full (incremental=0) framebuffer update request instead. Also send the resize information unconditionally, not only in case of an actual server-side change.

This makes the qemu vnc server conform to the spec and allows clients to request the complete vnc server state without reconnect.

---
 ui/vnc.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
ui/vnc.c +0 -4
index 84c4972b895b..8df63b349b38 100644
@@ -660,10 +660,6 @@ static void vnc_desktop_resize(VncState *vs)
     if (vs->ioc == NULL || !vnc_has_feature(vs, VNC_FEATURE_RESIZE)) {
         return;
     }
-    if (vs->client_width == pixman_image_get_width(vs->vd->server) &&
-        vs->client_height == pixman_image_get_height(vs->vd->server)) {
-        return;
-    }
 
     assert(pixman_image_get_width(vs->vd->server) < 65536 &&
            pixman_image_get_width(vs->vd->server) >= 0);

ui/vnc.c static void framebuffer_update_request(VncState *vs, int incremental, · line 2015

Quoted 7 lines
ui/vnc.c +4 -0
@@ -2013,6 +2009,10 @@ static void framebuffer_update_request(VncState *vs, int incremental,
     } else {
         vs->update = VNC_STATE_UPDATE_FORCE;
         vnc_set_area_dirty(vs->dirty, vs->vd, x, y, w, h);
+        vnc_colordepth(vs);
+        vnc_desktop_resize(vs);
+        vnc_led_state_change(vs);
+        vnc_cursor_define(vs);

If I'm nit-picking this still isn't spec compliant as it is sending 4 separate framebuffer update messages each with 1 rectangle, in response to 1 single framebuffer update request.

The strictly spec compliant approach would be to put these on a queue and send them as rectangles in the frame buffer update message that's sending the framebuffer contents. IIUC, this is the approach tigervnc uses. This would require a major change in the way QEMU sends FB update messages though.

In reality I doubt there are VNC clients that really care about this distinction, and this change is still more spec compliant that the current impl.

Quoted 4 lines
ui/vnc.c +0 -0
     }
 }
 

ui/vnc.c static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings) · line 2142

Quoted 10 lines
ui/vnc.c +0 -3
@@ -2136,10 +2136,7 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
             break;
         }
     }
-    vnc_desktop_resize(vs);
     check_pointer_type_change(&vs->mouse_mode_notifier, NULL);
-    vnc_led_state_change(vs);
-    vnc_cursor_define(vs);
 }
 
 static void set_pixel_conversion(VncState *vs)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw

PATCH v3 3/3 vnc: add support for extended desktop resize

In reply to Gerd Hoffmann

The extended desktop resize encoding adds support for (a) clients sending resize requests to the server, and (b) multihead support.

This patch implements (a). All resize requests are rejected by qemu. Qemu can't resize the framebuffer on its own, this is in the hands of the guest, so all qemu can do is forward the request to the guest. Should the guest actually resize the framebuffer we can notify the vnc client later with a separate message.

This requires support in the display device. Works with virtio-gpu.

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#extendeddesktopsize-pseudo-encoding

---
 ui/vnc.h |  2 ++
 ui/vnc.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 1 deletion(-)
ui/vnc.h +2 -0
index c8d3ad9ec496..77a310947bd6 100644
@@ -442,6 +442,7 @@ enum {
 
 enum VncFeatures {
     VNC_FEATURE_RESIZE,
+    VNC_FEATURE_RESIZE_EXT,
     VNC_FEATURE_HEXTILE,
     VNC_FEATURE_POINTER_TYPE_CHANGE,
     VNC_FEATURE_WMVI,
@@ -456,6 +457,7 @@ enum VncFeatures {
 };
 
 #define VNC_FEATURE_RESIZE_MASK              (1 << VNC_FEATURE_RESIZE)
+#define VNC_FEATURE_RESIZE_EXT_MASK          (1 << VNC_FEATURE_RESIZE_EXT)
 #define VNC_FEATURE_HEXTILE_MASK             (1 << VNC_FEATURE_HEXTILE)
 #define VNC_FEATURE_POINTER_TYPE_CHANGE_MASK (1 << VNC_FEATURE_POINTER_TYPE_CHANGE)
 #define VNC_FEATURE_WMVI_MASK                (1 << VNC_FEATURE_WMVI)
ui/vnc.c +63 -1

91 lines not on this page · raw message

Signature · 1 line
2.29.2
Reviewed-by on v3 3/3
#7 lore raw

Re: [PATCH v3 3/3] vnc: add support for extended desktop resize

Quoted 11 lines · Gerd Hoffmann

The extended desktop resize encoding adds support for (a) clients sending resize requests to the server, and (b) multihead support.

This patch implements (a). All resize requests are rejected by qemu. Qemu can't resize the framebuffer on its own, this is in the hands of the guest, so all qemu can do is forward the request to the guest. Should the guest actually resize the framebuffer we can notify the vnc client later with a separate message.

This requires support in the display device. Works with virtio-gpu.

Quoted 7 lines

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#extendeddesktopsize-pseudo-encoding

---
 ui/vnc.h |  2 ++
 ui/vnc.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 1 deletion(-)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw
In reply to Gerd Hoffmann
Quoted 11 lines · Gerd Hoffmann

The rfb standard keeps envolving. While the official spec is kind of frozen since a decade or so the community maintains an updated version of the spec at:
https://github.com/rfbproto/rfbproto/

This series adds support for two new extensions from that spec: alpha cursor and extended desktop resize.

alpha cursor allows a full alpha channel for the cursor image (unlike the rich cursor extension which has only a bitmask for transparency).

Quoted 7 lines

extended desktop resize makes the desktop-resize work both ways, i.e. we can not only signal guest display resolution changes to the vnc client but also vnc client window size changes to the guest.

Tested with tigervnc.

Support for gtk-vnc is in progress.

FWIW, basic support (ie no multi-head) is implemented and testable via the examples/gvncviewer program in gtk-vnc.

virt-viewer/remote-viewer are still todo.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|