Fix VNC server unbounded memory usage
cover Fix VNC server unbounded memory usage
In the 2.11 release we fixed CVE-2017-15268, which allowed the VNC websockets server to consume arbitrary memory when a slow client was connected. I have since discovered that this same type of problem can be triggered in several other ways in the regular (non-websockets) VNC server. This patch series attempts to fix this problem by limiting framebuffer updates and other data sent from server to client. The mitigating factor is that you need to have successfully authenticated with the VNC server to trigger these new flaws. This new more general flaw is assigned CVE-2017-15124 by the Red Hat security team.
The key patches containing the security fix are 9, 10, 11.
Since this code is incredibly subtle & hard to understand though, the first 8 patches do a bunch of independant cleanups/refactoring to make the security fixes clearer. The last two patches are just some extra cleanup / help for future maint.
Daniel P. Berrange (13):
ui: remove 'sync' parametr from vnc_update_client
ui: remove unreachable code in vnc_update_client
ui: remove redundant indentation in vnc_client_update
ui: avoid pointless VNC updates if framebuffer isn't dirty
ui: track how much decoded data we consumed when doing SASL encoding
ui: introduce enum to track VNC client framebuffer update request
state
ui: correctly reset framebuffer update state after processing dirty
regions
ui: refactor code for determining if an update should be sent to the
client
ui: fix VNC client throttling when audio capture is active
ui: fix VNC client throttling when forced update is requested
ui: place a hard cap on VNC server output buffer size
ui: add trace events related to VNC client throttling
ui: mix misleading comments & return types of VNC I/O helper methods
ui/trace-events | 7 ++ ui/vnc-auth-sasl.c | 16 ++- ui/vnc-auth-sasl.h | 5 +- ui/vnc-jobs.c | 5 + ui/vnc.c | 320 ++++++++++++++++++++++++++++++++++++++--------------- ui/vnc.h | 28 ++++- 6 files changed, 277 insertions(+), 104 deletions(-)
Signature · 1 line
2.14.3
PATCH 1/13 ui: remove 'sync' parametr from vnc_update_client
There is only one caller of vnc_update_client and that always passes false for the 'sync' parameter.
--- ui/vnc.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
ui/vnc.c +3 -8
@@ -596,7 +596,7 @@ VncInfo2List *qmp_query_vnc_servers(Error **errp)3) resolutions > 1024 */ -static int vnc_update_client (VncState *vs,int has_dirty,bool sync); +static int vnc_update_client (VncState *vs,int has_dirty);static void vnc_disconnect_start (VncState *vs);static void vnc_colordepth (VncState *vs);
@@ -961,7 +961,7 @@ static int find_and_clear_dirty_height(VncState *vs,return h; } -static int vnc_update_client (VncState *vs,int has_dirty,bool sync) +static int vnc_update_client (VncState *vs,int has_dirty) {if (vs->disconnecting) {vnc_disconnect_finish (vs);
@@ -1025,9 +1025,6 @@ static int vnc_update_client(VncState *vs, int has_dirty, bool sync) }vnc_job_push (job); -if (sync) { -vnc_jobs_join (vs); - } vs->force_update =0 ; vs->has_dirty =0 ;return n;
@@ -1035,8 +1032,6 @@ static int vnc_update_client(VncState *vs, int has_dirty, bool sync)if (vs->disconnecting) {vnc_disconnect_finish (vs); - }else if (sync) { -vnc_jobs_join (vs); }return 0 ;
@@ -2863,7 +2858,7 @@ static void vnc_refresh(DisplayChangeListener *dcl)vnc_unlock_display (vd); QTAILQ_FOREACH_SAFE(vs, &vd->clients, next, vn) { - rects +=vnc_update_client (vs, has_dirty, false); + rects +=vnc_update_client (vs, has_dirty);/* vs might be free()ed here */ }
Signature · 1 line
2.14.3
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [Qemu-devel] [PATCH v1 01/13] ui: remove 'sync' parametr from vnc_update_client
Just cc'ing qemu-devel back on the mail since it was accidentally dropped.
Quoted 69 lines · Darren Kenny
Small nit on the subject s/parametr/parameter/
Otherwise, looks good.
Reviewed-by Darren Kenny <[email protected]>Thanks,
Darren.
Quoted 58 lines · Daniel P. Berrange
There is only one caller of
vnc_update_clientand that always passes false for the 'sync' parameter.Signed-off-by Daniel P. Berrange <[email protected]>---
ui/vnc.c| 11 +++--------1 file changed, 3 insertions(+), 8 deletions(-)ui/vnc.c +3
-8@@ -596,7 +596,7 @@ VncInfo2List *qmp_query_vnc_servers(Error **errp)3) resolutions > 1024 */ -static int vnc_update_client (VncState *vs,int has_dirty,bool sync); +static int vnc_update_client (VncState *vs,int has_dirty);static void vnc_disconnect_start (VncState *vs);static void vnc_colordepth (VncState *vs);@@ -961,7 +961,7 @@ static int find_and_clear_dirty_height(VncState *vs,return h; } -static int vnc_update_client (VncState *vs,int has_dirty,bool sync) +static int vnc_update_client (VncState *vs,int has_dirty) {if (vs->disconnecting) {vnc_disconnect_finish (vs);@@ -1025,9 +1025,6 @@ static int vnc_update_client(VncState *vs, int has_dirty, bool sync) }vnc_job_push (job); -if (sync) { -vnc_jobs_join (vs); - } vs->force_update =0 ; vs->has_dirty =0 ;return n;@@ -1035,8 +1032,6 @@ static int vnc_update_client(VncState *vs, int has_dirty, bool sync)if (vs->disconnecting) {vnc_disconnect_finish (vs); - }else if (sync) { -vnc_jobs_join (vs); }return 0 ;@@ -2863,7 +2858,7 @@ static void vnc_refresh(DisplayChangeListener *dcl)vnc_unlock_display (vd); QTAILQ_FOREACH_SAFE(vs, &vd->clients, next, vn) { - rects +=vnc_update_client (vs, has_dirty, false); + rects +=vnc_update_client (vs, has_dirty);/* vs might be free()ed here */ }
Quoted 4 lines
Quoted 4 lines
Signature · 1 line
2.14.3
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v1 01/13] ui: remove 'sync' parametr from vnc_update_client
In reply to Daniel P. BerrangeQuoted 2 lines · Daniel P. Berrange
There is only one caller of
vnc_update_clientand that always passes false for the 'sync' parameter.
Unused since commit 50628d3479e4f9aa97e323506856e394fe7ad7a6. Good catch.
What is the state of this series btw (/me busy wading through my email backlog ...) ?
cheers,
Gerd
Re: [Qemu-devel] [PATCH v1 01/13] ui: remove 'sync' parametr from vnc_update_client
Quoted 9 lines · Gerd Hoffmann
Quoted 2 lines · Daniel P. Berrange
There is only one caller of
vnc_update_clientand that always passes false for the 'sync' parameter.Unused since commit
50628d3479e4f9aa97e323506856e394fe7ad7a6. Good catch.What is the state of this series btw (/me busy wading through my email backlog ...) ?
Marc-André reviewed it, but I was assuming you would want to review and (if acceptable) queue it, since you're the designated VNC maintainer. There's just one nit-pick in the Subject of this mail so far.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v1 01/13] ui: remove 'sync' parametr from vnc_update_client
Quoted 6 lines
Quoted 2 lines
What is the state of this series btw (/me busy wading through my email backlog ...) ?
Marc-André reviewed it, but I was assuming you would want to review and (if acceptable) queue it, since you're the designated VNC maintainer. There's just one nit-pick in the Subject of this mail so far.
Ok, queued, fixed $subject tyops, test builds running ...
thanks,
Gerd
PATCH 2/13 ui: remove unreachable code in vnc_update_client
In reply to Daniel P. BerrangeA previous commit:
commit 5a8be0f73d6f60ff08746377eb09ca459f39deab
Author: Gerd Hoffmann <[email protected]>
Date: Wed Jul 13 12:21:20 2016 +0200
vnc: make sure we finish disconnect
Added a check for vs->disconnecting at the very start of the vnc_update_client method. This means that the very next "if" statement check for !vs->disconnecting always evaluates true, and is thus redundant. This in turn means the vs->disconnecting check at the very end of the method never evaluates true, and is thus unreachable code.
--- ui/vnc.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
ui/vnc.c +1 -5
@@ -969,7 +969,7 @@ static int vnc_update_client(VncState *vs, int has_dirty) } vs->has_dirty += has_dirty; -if (vs->need_update && !vs->disconnecting) { +if (vs->need_update) {VncDisplay *vd = vs->vd;VncJob *job;int y;
@@ -1030,10 +1030,6 @@ static int vnc_update_client(VncState *vs, int has_dirty)return n; } -if (vs->disconnecting) { -vnc_disconnect_finish (vs); - } -return 0 ; }
Signature · 1 line
2.14.3
PATCH 3/13 ui: remove redundant indentation in vnc_client_update
In reply to Daniel P. BerrangeNow that previous dead / unreachable code has been removed, we can simplify the indentation in the vnc_client_update method.
--- ui/vnc.c | 112 ++++++++++++++++++++++++++++++++------------------------------- 1 file changed, 57 insertions(+), 55 deletions(-)
ui/vnc.c +57 -55
@@ -963,74 +963,76 @@ static int find_and_clear_dirty_height(VncState *vs,static int vnc_update_client (VncState *vs,int has_dirty) { +VncDisplay *vd = vs->vd; +VncJob *job; +int y; +int height, width; +int n =0 ; +if (vs->disconnecting) {vnc_disconnect_finish (vs);return 0 ; } vs->has_dirty += has_dirty; -if (vs->need_update) { -VncDisplay *vd = vs->vd; -VncJob *job; -int y; -int height, width; -int n =0 ; - -if (vs->output.offset && !vs->audio_cap && !vs->force_update) -/* kernel send buffers are full -> drop frames to throttle */ -return 0 ; +if (!vs->need_update) { +return 0 ; + } -if (!vs->has_dirty && !vs->audio_cap && !vs->force_update) -return 0 ; +if (vs->output.offset && !vs->audio_cap && !vs->force_update) { +/* kernel send buffers are full -> drop frames to throttle */ +return 0 ; + } -/* -* Send screen updates to the vnc client using the server -* surface and server dirty map. guest surface updates -* happening in parallel don't disturb us, the next pass will -* send them to the client. -*/ - job =vnc_job_new (vs); - - height =pixman_image_get_height (vd->server); - width =pixman_image_get_width (vd->server); - - y =0 ; -for (;;) { -int x, h; -unsigned long x2; -unsigned long offset =find_next_bit ((unsigned long *) &vs->dirty, - height *VNC_DIRTY_BPL (vs), - y *VNC_DIRTY_BPL (vs)); -if (offset == height *VNC_DIRTY_BPL (vs)) { -/* no more dirty bits */ +if (!vs->has_dirty && !vs->audio_cap && !vs->force_update) { +return 0 ; + } + +/* +* Send screen updates to the vnc client using the server +* surface and server dirty map. guest surface updates +* happening in parallel don't disturb us, the next pass will +* send them to the client. +*/ + job =vnc_job_new (vs); + + height =pixman_image_get_height (vd->server); + width =pixman_image_get_width (vd->server); + + y =0 ; +for (;;) { +int x, h; +unsigned long x2; +unsigned long offset =find_next_bit ((unsigned long *) &vs->dirty, + height *VNC_DIRTY_BPL (vs), + y *VNC_DIRTY_BPL (vs)); +if (offset == height *VNC_DIRTY_BPL (vs)) { +/* no more dirty bits */ +break ; + } + y = offset /VNC_DIRTY_BPL (vs); + x = offset %VNC_DIRTY_BPL (vs); + x2 =find_next_zero_bit ((unsigned long *) &vs->dirty[y], +VNC_DIRTY_BPL (vs), x); +bitmap_clear (vs->dirty[y], x, x2 - x); + h =find_and_clear_dirty_height (vs, y, x, x2, height); + x2 =MIN (x2, width /VNC_DIRTY_PIXELS_PER_BIT ); +if (x2 > x) { + n +=vnc_job_add_rect (job, x *VNC_DIRTY_PIXELS_PER_BIT , y, + (x2 - x) *VNC_DIRTY_PIXELS_PER_BIT , h); + } +if (!x && x2 == width /VNC_DIRTY_PIXELS_PER_BIT ) { + y += h; +if (y == height) {break ; } - y = offset /VNC_DIRTY_BPL (vs); - x = offset %VNC_DIRTY_BPL (vs); - x2 =find_next_zero_bit ((unsigned long *) &vs->dirty[y], -VNC_DIRTY_BPL (vs), x); -bitmap_clear (vs->dirty[y], x, x2 - x); - h =find_and_clear_dirty_height (vs, y, x, x2, height); - x2 =MIN (x2, width /VNC_DIRTY_PIXELS_PER_BIT ); -if (x2 > x) { - n +=vnc_job_add_rect (job, x *VNC_DIRTY_PIXELS_PER_BIT , y, - (x2 - x) *VNC_DIRTY_PIXELS_PER_BIT , h); - } -if (!x && x2 == width /VNC_DIRTY_PIXELS_PER_BIT ) { - y += h; -if (y == height) { -break ; - } - } } - -vnc_job_push (job); - vs->force_update =0 ; - vs->has_dirty =0 ; -return n; } -return 0 ; +vnc_job_push (job); + vs->force_update =0 ; + vs->has_dirty =0 ; +return n; }/* audio */
Signature · 1 line
2.14.3
PATCH 4/13 ui: avoid pointless VNC updates if framebuffer isn't dirty
In reply to Daniel P. BerrangeThe vnc_update_client() method checks the 'has_dirty' flag to see if there are dirty regions that are pending to send to the client. Regardless of this flag, if a forced update is requested, updates must be sent. For unknown reasons though, the code also tries to sent updates if audio capture is enabled. This makes no sense as audio capture state does not impact framebuffer contents, so this check is removed.
--- ui/vnc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ui/vnc.c +1 -1
@@ -984,7 +984,7 @@ static int vnc_update_client(VncState *vs, int has_dirty)return 0 ; } -if (!vs->has_dirty && !vs->audio_cap && !vs->force_update) { +if (!vs->has_dirty && !vs->force_update) {return 0 ; }
Signature · 1 line
2.14.3
PATCH 5/13 ui: track how much decoded data we consumed when doing SASL encoding
In reply to Daniel P. BerrangeWhen we encode data for writing with SASL, we encode the entire pending output buffer. The subsequent write, however, may not be able to send the full encoded data in one go though, particularly with a slow network. So we delay setting the output buffer offset back to zero until all the SASL encoded data is sent.
Between encoding the data and completing sending of the SASL encoded data, however, more data might have been placed on the pending output buffer. So it is not valid to set offset back to zero. Instead we must keep track of how much data we consumed during encoding and subtract only that amount.
With the current bug we would be throwing away some pending data without having sent it at all. By sheer luck this did not previously cause any serious problem because appending data to the send buffer is always an atomic action, so we only ever throw away complete RFB protocol messages. In the case of frame buffer updates we'd catch up fairly quickly, so no obvious problem was visible.
--- ui/vnc-auth-sasl.c | 3 ++- ui/vnc-auth-sasl.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
ui/vnc-auth-sasl.c +2 -1
@@ -67,6 +67,7 @@ long vnc_client_write_sasl(VncState *vs)if (err !=SASL_OK )return vnc_client_io_error (vs,-1 ,NULL ); + vs->sasl.encodedRawLength = vs->output.offset; vs->sasl.encodedOffset =0 ; }
@@ -78,7 +79,7 @@ long vnc_client_write_sasl(VncState *vs) vs->sasl.encodedOffset += ret;if (vs->sasl.encodedOffset == vs->sasl.encodedLength) { - vs->output.offset =0 ; + vs->output.offset -= vs->sasl.encodedRawLength; vs->sasl.encoded =NULL ; vs->sasl.encodedOffset = vs->sasl.encodedLength =0 ; }
ui/vnc-auth-sasl.h +1 -0
@@ -53,6 +53,7 @@ struct VncStateSASL {*/ const uint8_t *encoded;unsigned int encodedLength; +unsigned int encodedRawLength;unsigned int encodedOffset;char *username;char *mechlist;
Signature · 1 line
2.14.3
PATCH 6/13 ui: introduce enum to track VNC client framebuffer update request state
In reply to Daniel P. BerrangeCurrently the VNC servers tracks whether a client has requested an incremental or forced update with two boolean flags. There are only really 3 distinct states to track, so create an enum to more accurately reflect permitted states.
--- ui/vnc.c | 21 +++++++++++---------- ui/vnc.h | 9 +++++++-- 2 files changed, 18 insertions(+), 12 deletions(-)
ui/vnc.c +11 -10
@@ -975,16 +975,17 @@ static int vnc_update_client(VncState *vs, int has_dirty) } vs->has_dirty += has_dirty; -if (!vs->need_update) { +if (vs->update ==VNC_STATE_UPDATE_NONE ) {return 0 ; } -if (vs->output.offset && !vs->audio_cap && !vs->force_update) { +if (vs->output.offset && !vs->audio_cap && + vs->update !=VNC_STATE_UPDATE_FORCE ) {/* kernel send buffers are full -> drop frames to throttle */ return 0 ; } -if (!vs->has_dirty && !vs->force_update) { +if (!vs->has_dirty && vs->update !=VNC_STATE_UPDATE_FORCE ) {return 0 ; }
@@ -1030,7 +1031,7 @@ static int vnc_update_client(VncState *vs, int has_dirty) }vnc_job_push (job); - vs->force_update =0 ; + vs->update =VNC_STATE_UPDATE_INCREMENTAL ; vs->has_dirty =0 ;return n; }
@@ -1869,14 +1870,14 @@ static void ext_key_event(VncState *vs, int down,static void framebuffer_update_request (VncState *vs,int incremental,int x,int y,int w,int h) { - vs->need_update =1 ; -if (incremental) { -return ; +if (vs->update !=VNC_STATE_UPDATE_FORCE ) { + vs->update =VNC_STATE_UPDATE_INCREMENTAL ; + } + }else { + vs->update =VNC_STATE_UPDATE_FORCE ; +vnc_set_area_dirty (vs->dirty, vs->vd, x, y, w, h); } - - vs->force_update =1 ; -vnc_set_area_dirty (vs->dirty, vs->vd, x, y, w, h); }static void send_ext_key_event_ack (VncState *vs)
ui/vnc.h +7 -2
@@ -252,6 +252,12 @@ struct VncJobQTAILQ_ENTRY (VncJob) next; }; +typedef enum { +VNC_STATE_UPDATE_NONE , +VNC_STATE_UPDATE_INCREMENTAL , +VNC_STATE_UPDATE_FORCE , +}VncStateUpdate ; +struct VncState {QIOChannelSocket *sioc;/* The underlying socket */
@@ -264,8 +270,7 @@ struct VncState* vnc-jobs-async.c */ VncDisplay *vd; -int need_update; -int force_update; +VncStateUpdate update;/* Most recent pending request from client */ int has_dirty;uint32_t features;int absolute;
Signature · 1 line
2.14.3
PATCH 7/13 ui: correctly reset framebuffer update state after processing dirty regions
In reply to Daniel P. BerrangeAccording to the RFB protocol, a client sends one or more framebuffer update requests to the server. The server can reply with a single framebuffer update response, that covers all previously received requests. Once the client has read this update from the server, it may send further framebuffer update requests to monitor future changes. The client is free to delay sending the framebuffer update request if it needs to throttle the amount of data it is reading from the server.
The QEMU VNC server, however, has never correctly handled the framebuffer update requests. Once QEMU has received an update request, it will continue to send client updates forever, even if the client hasn't asked for further updates. This prevents the client from throttling back data it gets from the server. This change fixes the flawed logic such that after a set of updates are sent out, QEMU waits for a further update request before sending more data.
--- ui/vnc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ui/vnc.c +1 -1
@@ -1031,7 +1031,7 @@ static int vnc_update_client(VncState *vs, int has_dirty) }vnc_job_push (job); - vs->update =VNC_STATE_UPDATE_INCREMENTAL ; + vs->update =VNC_STATE_UPDATE_NONE ; vs->has_dirty =0 ;return n; }
Signature · 1 line
2.14.3
PATCH 8/13 ui: refactor code for determining if an update should be sent to the client
In reply to Daniel P. BerrangeThe logic for determining if it is possible to send an update to the client will become more complicated shortly, so pull it out into a separate method for easier extension later.
--- ui/vnc.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)
ui/vnc.c +20 -7
@@ -961,6 +961,25 @@ static int find_and_clear_dirty_height(VncState *vs,return h; } +static bool vnc_should_update (VncState *vs) +{ +switch (vs->update) { +case VNC_STATE_UPDATE_NONE : +break ; +case VNC_STATE_UPDATE_INCREMENTAL : +/* Only allow incremental updates if the output buffer +* is empty, or if audio capture is enabled. +*/ +if (!vs->output.offset || vs->audio_cap) { +return true; + } +break ; +case VNC_STATE_UPDATE_FORCE : +return true; + } +return false; +} +static int vnc_update_client (VncState *vs,int has_dirty) {VncDisplay *vd = vs->vd;
@@ -975,13 +994,7 @@ static int vnc_update_client(VncState *vs, int has_dirty) } vs->has_dirty += has_dirty; -if (vs->update ==VNC_STATE_UPDATE_NONE ) { -return 0 ; - } - -if (vs->output.offset && !vs->audio_cap && - vs->update !=VNC_STATE_UPDATE_FORCE ) { -/* kernel send buffers are full -> drop frames to throttle */ +if (!vnc_should_update (vs)) {return 0 ; }
Signature · 1 line
2.14.3
PATCH 9/13 ui: fix VNC client throttling when audio capture is active
In reply to Daniel P. BerrangeThe VNC server must throttle data sent to the client to prevent the 'output' buffer size growing without bound, if the client stops reading data off the socket (either maliciously or due to stalled/slow network connection).
The current throttling is very crude because it simply checks whether the output buffer offset is zero. This check must be disabled if audio capture is enabled, because when streaming audio the output buffer offset will rarely be zero due to queued audio data, and so this would starve framebuffer updates.
As a result, the VNC client can cause QEMU to allocate arbitrary amounts of RAM. They can first start something in the guest that triggers lots of framebuffer updates eg play a youtube video. Then enable audio capture, and simply never read data back from the server. This can easily make QEMU's VNC server send buffer consume 100MB of RAM per second, until the OOM killer starts reaping processes (hopefully the rogue QEMU process, but it might pick others...).
To address this we make the throttling more intelligent, so we can throttle when audio capture is active too. To determine how to throttle incremental updates or audio data, we calculate a size threshold. Normally the threshold is the approximate number of bytes associated with a single complete framebuffer update. ie width * height * bytes per pixel. We'll send incremental updates until we hit this threshold, at which point we'll stop sending updates until data has been written to the wire, causing the output buffer offset to fall back below the threshold.
If audio capture is enabled, we increase the size of the threshold to also allow for upto 1 seconds worth of audio data samples. ie nchannels * bytes per sample * frequency. This allows the output buffer to have a mixture of incremental framebuffer updates and audio data queued, but once the threshold is exceeded, audio data will be dropped and incremental updates will be throttled.
This unbounded memory growth affects all VNC server configurations supported by QEMU, with no workaround possible. The mitigating factor is that it can only be triggered by a client that has authenticated with the VNC server, and who is able to trigger a large quantity of framebuffer updates or audio samples from the guest OS. Mostly they'll just succeed in getting the OOM killer to kill their own QEMU process, but its possible other processes can get taken out as collateral damage.
This is a more general variant of the similar unbounded memory usage flaw in the websockets server, that was previously assigned CVE-2017-15268, and fixed in 2.11 by:
commit a7b20a8efa28e5f22c26c06cd06c2f12bc863493
Author: Daniel P. Berrange <[email protected]>
Date: Mon Oct 9 14:43:42 2017 +0100
io: monitor encoutput buffer size from websocket GSource
This new general memory usage flaw has been assigned CVE-2017-15124, and is partially fixed by this patch.
--- ui/vnc.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------- ui/vnc.h | 6 ++++++ 2 files changed, 70 insertions(+), 8 deletions(-)
ui/vnc.c +64 -8
@@ -60,6 +60,7 @@ static QTAILQ_HEAD(, VncDisplay) vnc_displays =static int vnc_cursor_define (VncState *vs);static void vnc_release_modifiers (VncState *vs); +static void vnc_update_throttle_offset (VncState *vs);static void vnc_set_share_mode (VncState *vs,VncShareMode mode) {
@@ -766,6 +767,7 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,vnc_set_area_dirty (vs->dirty, vd,0 ,0 ,vnc_width (vd),vnc_height (vd)); +vnc_update_throttle_offset (vs); } }
@@ -961,16 +963,67 @@ static int find_and_clear_dirty_height(VncState *vs,return h; } +/* +* Figure out how much pending data we should allow in the output +* buffer before we throttle incremental display updates, and/or +* drop audio samples. +* +* We allow for equiv of 1 full display's worth of FB updates, +* and 1 second of audio samples. If audio backlog was larger +* than that the client would already suffering awful audio +* glitches, so dropping samples is no worse really). +*/ +static void vnc_update_throttle_offset (VncState *vs) +{ +size_t offset = + vs->client_width * vs->client_height * vs->client_pf.bytes_per_pixel; + +if (vs->audio_cap) { +int freq = vs->as.freq; +/* We don't limit freq when reading settings from client, so +* it could be upto MAX_INT in size. 48khz is a sensible +* upper bound for trustworthy clients */ +int bps; +if (freq >48000 ) { + freq =48000 ; + } +switch (vs->as.fmt) { +default : +case AUD_FMT_U8 : +case AUD_FMT_S8 : + bps =1 ; +break ; +case AUD_FMT_U16 : +case AUD_FMT_S16 : + bps =2 ; +break ; +case AUD_FMT_U32 : +case AUD_FMT_S32 : + bps =4 ; +break ; + } + offset += freq * bps * vs->as.nchannels; + } + +/* Put a floor of 1MB on offset, so that if we have a large pending +* buffer and the display is resized to a small size & back again +* we don't suddenly apply a tiny send limit +*/ + offset =MAX (offset,1024 *1024 ); + + vs->throttle_output_offset = offset; +} +static bool vnc_should_update (VncState *vs) {switch (vs->update) {case VNC_STATE_UPDATE_NONE :break ;case VNC_STATE_UPDATE_INCREMENTAL : -/* Only allow incremental updates if the output buffer -* is empty, or if audio capture is enabled. +/* Only allow incremental updates if the pending send queue +* is less than the permitted threshold */ -if (!vs->output.offset || vs->audio_cap) { +if (vs->output.offset < vs->throttle_output_offset) {return true; }break ;
@@ -1084,11 +1137,13 @@ static void audio_capture(void *opaque, void *buf, int size)VncState *vs = opaque;vnc_lock_output (vs); -vnc_write_u8 (vs,VNC_MSG_SERVER_QEMU ); -vnc_write_u8 (vs,VNC_MSG_SERVER_QEMU_AUDIO ); -vnc_write_u16 (vs,VNC_MSG_SERVER_QEMU_AUDIO_DATA ); -vnc_write_u32 (vs, size); -vnc_write (vs, buf, size); +if (vs->output.offset < vs->throttle_output_offset) { +vnc_write_u8 (vs,VNC_MSG_SERVER_QEMU ); +vnc_write_u8 (vs,VNC_MSG_SERVER_QEMU_AUDIO ); +vnc_write_u16 (vs,VNC_MSG_SERVER_QEMU_AUDIO_DATA ); +vnc_write_u32 (vs, size); +vnc_write (vs, buf, size); + }vnc_unlock_output (vs);vnc_flush (vs); }
@@ -2288,6 +2343,7 @@ static int protocol_client_msg(VncState *vs, uint8_t *data, size_t len)break ; } +vnc_update_throttle_offset (vs);vnc_read_when (vs, protocol_client_msg,1 );return 0 ; }
ui/vnc.h +6 -0
@@ -298,6 +298,12 @@ struct VncStateVncClientInfo *info; +/* We allow multiple incremental updates or audio capture +* samples to be queued in output buffer, provided the +* buffer size doesn't exceed this threshold. The value +* is calculating dynamically based on framebuffer size +* and audio sample settings in vnc_update_throttle_offset() */ +size_t throttle_output_offset;Buffer output;Buffer input;/* current output mode information */
Signature · 1 line
2.14.3
PATCH 10/13 ui: fix VNC client throttling when forced update is requested
In reply to Daniel P. BerrangeThe VNC server must throttle data sent to the client to prevent the 'output' buffer size growing without bound, if the client stops reading data off the socket (either maliciously or due to stalled/slow network connection).
The current throttling is very crude because it simply checks whether the output buffer offset is zero. This check is disabled if the client has requested a forced update, because we want to send these as soon as possible.
As a result, the VNC client can cause QEMU to allocate arbitrary amounts of RAM. They can first start something in the guest that triggers lots of framebuffer updates eg play a youtube video. Then repeatedly send full framebuffer update requests, but never read data back from the server. This can easily make QEMU's VNC server send buffer consume 100MB of RAM per second, until the OOM killer starts reaping processes (hopefully the rogue QEMU process, but it might pick others...).
To address this we make the throttling more intelligent, so we can throttle full updates. When we get a forced update request, we keep track of exactly how much data we put on the output buffer. We will not process a subsequent forced update request until this data has been fully sent on the wire. We always allow one forced update request to be in flight, regardless of what data is queued for incremental updates or audio data. The slight complication is that we do not initially know how much data an update will send, as this is done in the background by the VNC job thread. So we must track the fact that the job thread has an update pending, and not process any further updates until this job is has been completed & put data on the output buffer.
This unbounded memory growth affects all VNC server configurations supported by QEMU, with no workaround possible. The mitigating factor is that it can only be triggered by a client that has authenticated with the VNC server, and who is able to trigger a large quantity of framebuffer updates or audio samples from the guest OS. Mostly they'll just succeed in getting the OOM killer to kill their own QEMU process, but its possible other processes can get taken out as collateral damage.
This is a more general variant of the similar unbounded memory usage flaw in the websockets server, that was previously assigned CVE-2017-15268, and fixed in 2.11 by:
commit a7b20a8efa28e5f22c26c06cd06c2f12bc863493
Author: Daniel P. Berrange <[email protected]>
Date: Mon Oct 9 14:43:42 2017 +0100
io: monitor encoutput buffer size from websocket GSource
This new general memory usage flaw has been assigned CVE-2017-15124, and is partially fixed by this patch.
--- ui/vnc-auth-sasl.c | 5 +++++ ui/vnc-jobs.c | 5 +++++ ui/vnc.c | 28 ++++++++++++++++++++++++---- ui/vnc.h | 7 +++++++ 4 files changed, 41 insertions(+), 4 deletions(-)
ui/vnc-auth-sasl.c +5 -0
@@ -79,6 +79,11 @@ long vnc_client_write_sasl(VncState *vs) vs->sasl.encodedOffset += ret;if (vs->sasl.encodedOffset == vs->sasl.encodedLength) { +if (vs->sasl.encodedRawLength >= vs->force_update_offset) { + vs->force_update_offset =0 ; + }else { + vs->force_update_offset -= vs->sasl.encodedRawLength; + } vs->output.offset -= vs->sasl.encodedRawLength; vs->sasl.encoded =NULL ; vs->sasl.encodedOffset = vs->sasl.encodedLength =0 ;
ui/vnc-jobs.c +5 -0
@@ -152,6 +152,11 @@ void vnc_jobs_consume_buffer(VncState *vs) vs->ioc, G_IO_IN | G_IO_OUT, vnc_client_io, vs, NULL); }buffer_move (&vs->output, &vs->jobs_buffer); + +if (vs->job_update ==VNC_STATE_UPDATE_FORCE ) { + vs->force_update_offset = vs->output.offset; + } + vs->job_update =VNC_STATE_UPDATE_NONE ; } flush = vs->ioc !=NULL && vs->abort != true;vnc_unlock_output (vs);
ui/vnc.c +24 -4
@@ -1021,14 +1021,28 @@ static bool vnc_should_update(VncState *vs)break ;case VNC_STATE_UPDATE_INCREMENTAL :/* Only allow incremental updates if the pending send queue -* is less than the permitted threshold +* is less than the permitted threshold, and the job worker +* is completely idle. */ -if (vs->output.offset < vs->throttle_output_offset) { +if (vs->output.offset < vs->throttle_output_offset && + vs->job_update ==VNC_STATE_UPDATE_NONE ) {return true; }break ;case VNC_STATE_UPDATE_FORCE : -return true; +/* Only allow forced updates if the pending send queue +* does not contain a previous forced update, and the +* job worker is completely idle. +* +* Note this means we'll queue a forced update, even if +* the output buffer size is otherwise over the throttle +* output limit. +*/ +if (vs->force_update_offset ==0 && + vs->job_update ==VNC_STATE_UPDATE_NONE ) { +return true; + } +break ; }return false; }
@@ -1096,8 +1110,9 @@ static int vnc_update_client(VncState *vs, int has_dirty) } } -vnc_job_push (job); + vs->job_update = vs->update; vs->update =VNC_STATE_UPDATE_NONE ; +vnc_job_push (job); vs->has_dirty =0 ;return n; }
@@ -1332,6 +1347,11 @@ static ssize_t vnc_client_write_plain(VncState *vs)if (!ret)return 0 ; +if (ret >= vs->force_update_offset) { + vs->force_update_offset =0 ; + }else { + vs->force_update_offset -= ret; + }buffer_advance (&vs->output, ret);if (vs->output.offset ==0 ) {
ui/vnc.h +7 -0
@@ -271,6 +271,7 @@ struct VncStateVncDisplay *vd;VncStateUpdate update;/* Most recent pending request from client */ +VncStateUpdate job_update;/* Currently processed by job thread */ int has_dirty;uint32_t features;int absolute;
@@ -298,6 +299,12 @@ struct VncStateVncClientInfo *info; +/* Job thread bottom half has put data for a forced update +* into the output buffer. This offset points to the end of +* the update data in the output buffer. This lets us determine +* when a force update is fully sent to the client, allowing +* us to process further forced updates. */ +size_t force_update_offset;/* We allow multiple incremental updates or audio capture * samples to be queued in output buffer, provided the * buffer size doesn't exceed this threshold. The value
Signature · 1 line
2.14.3
Re: [Qemu-devel] [PATCH v1 10/13] ui: fix VNC client throttling when forced update is requested
Hi
Quoted 125 lines · Daniel P. Berrange
The VNC server must throttle data sent to the client to prevent the 'output' buffer size growing without bound, if the client stops reading data off the socket (either maliciously or due to stalled/slow network connection).
The current throttling is very crude because it simply checks whether the output buffer offset is zero. This check is disabled if the client has requested a forced update, because we want to send these as soon as possible.
As a result, the VNC client can cause QEMU to allocate arbitrary amounts of RAM. They can first start something in the guest that triggers lots of framebuffer updates eg play a youtube video. Then repeatedly send full framebuffer update requests, but never read data back from the server. This can easily make QEMU's VNC server send buffer consume 100MB of RAM per second, until the OOM killer starts reaping processes (hopefully the rogue QEMU process, but it might pick others...).
To address this we make the throttling more intelligent, so we can throttle full updates. When we get a forced update request, we keep track of exactly how much data we put on the output buffer. We will not process a subsequent forced update request until this data has been fully sent on the wire. We always allow one forced update request to be in flight, regardless of what data is queued for incremental updates or audio data. The slight complication is that we do not initially know how much data an update will send, as this is done in the background by the VNC job thread. So we must track the fact that the job thread has an update pending, and not process any further updates until this job is has been completed & put data on the output buffer.
This unbounded memory growth affects all VNC server configurations supported by QEMU, with no workaround possible. The mitigating factor is that it can only be triggered by a client that has authenticated with the VNC server, and who is able to trigger a large quantity of framebuffer updates or audio samples from the guest OS. Mostly they'll just succeed in getting the OOM killer to kill their own QEMU process, but its possible other processes can get taken out as collateral damage.
This is a more general variant of the similar unbounded memory usage flaw in the websockets server, that was previously assigned CVE-2017-15268, and fixed in 2.11 by:
commit
a7b20a8efa28e5f22c26c06cd06c2f12bc863493
Author: Daniel P. Berrange <[email protected]>
Date: Mon Oct 9 14:43:42 2017 +0100io: monitor encoutput buffer size from websocket GSource
This new general memory usage flaw has been assigned CVE-2017-15124, and is partially fixed by this patch.
Signed-off-by Daniel P. Berrange <[email protected]>--- ui/vnc-auth-sasl.c | 5 +++++ ui/vnc-jobs.c | 5 +++++ ui/vnc.c | 28 ++++++++++++++++++++++++---- ui/vnc.h | 7 +++++++ 4 files changed, 41 insertions(+), 4 deletions(-)ui/vnc-auth-sasl.c +5
-0@@ -79,6 +79,11 @@ long vnc_client_write_sasl(VncState *vs) vs->sasl.encodedOffset += ret;if (vs->sasl.encodedOffset == vs->sasl.encodedLength) { +if (vs->sasl.encodedRawLength >= vs->force_update_offset) { + vs->force_update_offset =0 ; + }else { + vs->force_update_offset -= vs->sasl.encodedRawLength; + } vs->output.offset -= vs->sasl.encodedRawLength; vs->sasl.encoded =NULL ; vs->sasl.encodedOffset = vs->sasl.encodedLength =0 ;ui/vnc-jobs.c +5
-0@@ -152,6 +152,11 @@ void vnc_jobs_consume_buffer(VncState *vs) vs->ioc, G_IO_IN | G_IO_OUT, vnc_client_io, vs, NULL); }buffer_move (&vs->output, &vs->jobs_buffer); + +if (vs->job_update ==VNC_STATE_UPDATE_FORCE ) { + vs->force_update_offset = vs->output.offset; + } + vs->job_update =VNC_STATE_UPDATE_NONE ; } flush = vs->ioc !=NULL && vs->abort != true;vnc_unlock_output (vs);ui/vnc.c +17
-3@@ -1021,14 +1021,28 @@ static bool vnc_should_update(VncState *vs)break ;case VNC_STATE_UPDATE_INCREMENTAL :/* Only allow incremental updates if the pending send queue -* is less than the permitted threshold +* is less than the permitted threshold, and the job worker +* is completely idle. */ -if (vs->output.offset < vs->throttle_output_offset) { +if (vs->output.offset < vs->throttle_output_offset && + vs->job_update ==VNC_STATE_UPDATE_NONE ) {return true; }break ;case VNC_STATE_UPDATE_FORCE : -return true; +/* Only allow forced updates if the pending send queue +* does not contain a previous forced update, and the +* job worker is completely idle. +* +* Note this means we'll queue a forced update, even if +* the output buffer size is otherwise over the throttle +* output limit. +*/ +if (vs->force_update_offset ==0 && + vs->job_update ==VNC_STATE_UPDATE_NONE ) { +return true; + } +break ; }return false; }
ui/vnc.c static int vnc_update_client(VncState *vs, int has_dirty) · line 1113
Quoted 5 lines
ui/vnc.c +1
-1@@ -1096,8 +1110,9 @@ static int vnc_update_client(VncState *vs, int has_dirty) } } -vnc_job_push (job); + vs->job_update = vs->update;
How is this going to match the buffer job in vnc_jobs_consume_buffer() ?
(isn't this potentially taking the next job to finish as a force-update?)
Quoted 46 lines
ui/vnc.c +6
-0vs->update =VNC_STATE_UPDATE_NONE ; +vnc_job_push (job); vs->has_dirty =0 ;return n; }@@ -1332,6 +1347,11 @@ static ssize_t vnc_client_write_plain(VncState *vs)if (!ret)return 0 ; +if (ret >= vs->force_update_offset) { + vs->force_update_offset =0 ; + }else { + vs->force_update_offset -= ret; + }buffer_advance (&vs->output, ret);if (vs->output.offset ==0 ) {ui/vnc.h +7
-1@@ -271,6 +271,7 @@ struct VncStateVncDisplay *vd;VncStateUpdate update;/* Most recent pending request from client */ +VncStateUpdate job_update;/* Currently processed by job thread */ int has_dirty;uint32_t features;int absolute;@@ -298,6 +299,12 @@ struct VncStateVncClientInfo *info; +/* Job thread bottom half has put data for a forced update +* into the output buffer. This offset points to the end of +* the update data in the output buffer. This lets us determine +* when a force update is fully sent to the client, allowing +* us to process further forced updates. */ +size_t force_update_offset;/* We allow multiple incremental updates or audio capture * samples to be queued in output buffer, provided the * buffer size doesn't exceed this threshold. The value -- 2.14.3
Signature · 1 line
Marc-André Lureau
Re: [Qemu-devel] [PATCH v1 10/13] ui: fix VNC client throttling when forced update is requested
Quoted 131 lines · Marc-André Lureau
Hi
Quoted 128 lines · Daniel P. Berrange
The VNC server must throttle data sent to the client to prevent the 'output' buffer size growing without bound, if the client stops reading data off the socket (either maliciously or due to stalled/slow network connection).
The current throttling is very crude because it simply checks whether the output buffer offset is zero. This check is disabled if the client has requested a forced update, because we want to send these as soon as possible.
As a result, the VNC client can cause QEMU to allocate arbitrary amounts of RAM. They can first start something in the guest that triggers lots of framebuffer updates eg play a youtube video. Then repeatedly send full framebuffer update requests, but never read data back from the server. This can easily make QEMU's VNC server send buffer consume 100MB of RAM per second, until the OOM killer starts reaping processes (hopefully the rogue QEMU process, but it might pick others...).
To address this we make the throttling more intelligent, so we can throttle full updates. When we get a forced update request, we keep track of exactly how much data we put on the output buffer. We will not process a subsequent forced update request until this data has been fully sent on the wire. We always allow one forced update request to be in flight, regardless of what data is queued for incremental updates or audio data. The slight complication is that we do not initially know how much data an update will send, as this is done in the background by the VNC job thread. So we must track the fact that the job thread has an update pending, and not process any further updates until this job is has been completed & put data on the output buffer.
This unbounded memory growth affects all VNC server configurations supported by QEMU, with no workaround possible. The mitigating factor is that it can only be triggered by a client that has authenticated with the VNC server, and who is able to trigger a large quantity of framebuffer updates or audio samples from the guest OS. Mostly they'll just succeed in getting the OOM killer to kill their own QEMU process, but its possible other processes can get taken out as collateral damage.
This is a more general variant of the similar unbounded memory usage flaw in the websockets server, that was previously assigned CVE-2017-15268, and fixed in 2.11 by:
commit
a7b20a8efa28e5f22c26c06cd06c2f12bc863493
Author: Daniel P. Berrange <[email protected]>
Date: Mon Oct 9 14:43:42 2017 +0100io: monitor encoutput buffer size from websocket GSource
This new general memory usage flaw has been assigned CVE-2017-15124, and is partially fixed by this patch.
Signed-off-by Daniel P. Berrange <[email protected]>--- ui/vnc-auth-sasl.c | 5 +++++ ui/vnc-jobs.c | 5 +++++ ui/vnc.c | 28 ++++++++++++++++++++++++---- ui/vnc.h | 7 +++++++ 4 files changed, 41 insertions(+), 4 deletions(-)ui/vnc-auth-sasl.c +5
-0@@ -79,6 +79,11 @@ long vnc_client_write_sasl(VncState *vs) vs->sasl.encodedOffset += ret;if (vs->sasl.encodedOffset == vs->sasl.encodedLength) { +if (vs->sasl.encodedRawLength >= vs->force_update_offset) { + vs->force_update_offset =0 ; + }else { + vs->force_update_offset -= vs->sasl.encodedRawLength; + } vs->output.offset -= vs->sasl.encodedRawLength; vs->sasl.encoded =NULL ; vs->sasl.encodedOffset = vs->sasl.encodedLength =0 ;ui/vnc-jobs.c +5
-0@@ -152,6 +152,11 @@ void vnc_jobs_consume_buffer(VncState *vs) vs->ioc, G_IO_IN | G_IO_OUT, vnc_client_io, vs, NULL); }buffer_move (&vs->output, &vs->jobs_buffer); + +if (vs->job_update ==VNC_STATE_UPDATE_FORCE ) { + vs->force_update_offset = vs->output.offset; + } + vs->job_update =VNC_STATE_UPDATE_NONE ; } flush = vs->ioc !=NULL && vs->abort != true;vnc_unlock_output (vs);ui/vnc.c +17
-3@@ -1021,14 +1021,28 @@ static bool vnc_should_update(VncState *vs)break ;case VNC_STATE_UPDATE_INCREMENTAL :/* Only allow incremental updates if the pending send queue -* is less than the permitted threshold +* is less than the permitted threshold, and the job worker +* is completely idle. */ -if (vs->output.offset < vs->throttle_output_offset) { +if (vs->output.offset < vs->throttle_output_offset && + vs->job_update ==VNC_STATE_UPDATE_NONE ) {return true; }break ;case VNC_STATE_UPDATE_FORCE : -return true; +/* Only allow forced updates if the pending send queue +* does not contain a previous forced update, and the +* job worker is completely idle. +* +* Note this means we'll queue a forced update, even if +* the output buffer size is otherwise over the throttle +* output limit. +*/ +if (vs->force_update_offset ==0 && + vs->job_update ==VNC_STATE_UPDATE_NONE ) { +return true; + } +break ; }return false; }@@ -1096,8 +1110,9 @@ static int vnc_update_client(VncState *vs, int has_dirty) } }
Quoted 6 lines
ui/vnc.c static int vnc_update_client(VncState *vs, int has_dirty) · line 1113
Quoted 2 lines
ui/vnc.c +1
-1-vnc_job_push (job); + vs->job_update = vs->update;How is this going to match the buffer job in
vnc_jobs_consume_buffer()?(isn't this potentially taking the next job to finish as a force-update?)
Earlier in this method we check vnc_should_update() and that only returns true if job_update == VNC_STATE_UPDATE_NONE (ie no currently processed job in flight).
(this is the slight change from the previous patch version you looked at off list where I allowed a force job to be queued even if a incremental job was inflight. I decided that didnt really have any benefit from what the client gets, and it complicated tracking)
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 11/13 ui: place a hard cap on VNC server output buffer size
In reply to Daniel P. BerrangeThe previous patches fix problems with throttling of forced framebuffer updates and audio data capture that would cause the QEMU output buffer size to grow without bound. Those fixes are graceful in that once the client catches up with reading data from the server, everything continues operating normally.
There is some data which the server sends to the client that is impractical to throttle. Specifically there are various pseudo framebuffer update encodings to inform the client of things like desktop resizes, pointer changes, audio playback start/stop, LED state and so on. These generally only involve sending a very small amount of data to the client, but a malicious guest might be able to do things that trigger these changes at a very high rate. Throttling them is not practical as missed or delayed events would cause broken behaviour for the client.
This patch thus takes a more forceful approach of setting an absolute upper bound on the amount of data we permit to be present in the output buffer at any time. The previous patch set a threshold for throttling the output buffer by allowing an amount of data equivalent to one complete framebuffer update and one seconds worth of audio data. On top of this it allowed for one further forced framebuffer update to be queued.
To be conservative, we thus take that throttling threshold and multiply it by 5 to form an absolute upper bound. If this bound is hit during vnc_write() we forceably disconnect the client, refusing to queue further data. This limit is high enough that it should never be hit unless a malicious client is trying to exploit the sever, or the network is completely saturated preventing any sending of data on the socket.
This completes the fix for CVE-2017-15124 started in the previous patches.
--- ui/vnc.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
ui/vnc.c +29 -0
@@ -1521,8 +1521,37 @@ gboolean vnc_client_io(QIOChannel *ioc G_GNUC_UNUSED, } +/* +* Scale factor to apply to vs->throttle_output_offset when checking for +* hard limit. Worst case normal usage could be x2, if we have a complete +* incremental update and complete forced update in the output buffer. +* So x3 should be good enough, but we pick x5 to be conservative and thus +* (hopefully) never trigger incorrectly. +*/ +#define VNC_THROTTLE_OUTPUT_LIMIT_SCALE 5 +void vnc_write (VncState *vs,const void *data,size_t len) { +if (vs->disconnecting) { +return ; + } +/* Protection against malicious client/guest to prevent our output +* buffer growing without bound if client stops reading data. This +* should rarely trigger, because we have earlier throttling code +* which stops issuing framebuffer updates and drops audio data +* if the throttle_output_offset value is exceeded. So we only reach +* this higher level if a huge number of pseudo-encodings get +* triggered while data can't be sent on the socket. +* +* NB throttle_output_offset can be zero during early protocol +* handshake, or from the job thread's VncState clone +*/ +if (vs->throttle_output_offset !=0 && + vs->output.offset > (vs->throttle_output_offset * +VNC_THROTTLE_OUTPUT_LIMIT_SCALE )) { +vnc_disconnect_start (vs); +return ; + }buffer_reserve (&vs->output, len);if (vs->ioc !=NULL &&buffer_empty (&vs->output)) {
Signature · 1 line
2.14.3
Re: [Qemu-devel] [PATCH v1 11/13] ui: place a hard cap on VNC server output buffer size
Hi
Quoted 62 lines · Daniel P. Berrange
The previous patches fix problems with throttling of forced framebuffer updates and audio data capture that would cause the QEMU output buffer size to grow without bound. Those fixes are graceful in that once the client catches up with reading data from the server, everything continues operating normally.
There is some data which the server sends to the client that is impractical to throttle. Specifically there are various pseudo framebuffer update encodings to inform the client of things like desktop resizes, pointer changes, audio playback start/stop, LED state and so on. These generally only involve sending a very small amount of data to the client, but a malicious guest might be able to do things that trigger these changes at a very high rate. Throttling them is not practical as missed or delayed events would cause broken behaviour for the client.
This patch thus takes a more forceful approach of setting an absolute upper bound on the amount of data we permit to be present in the output buffer at any time. The previous patch set a threshold for throttling the output buffer by allowing an amount of data equivalent to one complete framebuffer update and one seconds worth of audio data. On top of this it allowed for one further forced framebuffer update to be queued.
To be conservative, we thus take that throttling threshold and multiply it by 5 to form an absolute upper bound. If this bound is hit during
vnc_write()we forceably disconnect the client, refusing to queue further data. This limit is high enough that it should never be hit unless a malicious client is trying to exploit the sever, or the network is completely saturated preventing any sending of data on the socket.This completes the fix for CVE-2017-15124 started in the previous patches.
Signed-off-by Daniel P. Berrange <[email protected]>--- ui/vnc.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)ui/vnc.c +17
-0@@ -1521,8 +1521,37 @@ gboolean vnc_client_io(QIOChannel *ioc G_GNUC_UNUSED, } +/* +* Scale factor to apply to vs->throttle_output_offset when checking for +* hard limit. Worst case normal usage could be x2, if we have a complete +* incremental update and complete forced update in the output buffer. +* So x3 should be good enough, but we pick x5 to be conservative and thus +* (hopefully) never trigger incorrectly. +*/ +#define VNC_THROTTLE_OUTPUT_LIMIT_SCALE 5 +void vnc_write (VncState *vs,const void *data,size_t len) { +if (vs->disconnecting) { +return ; + } +/* Protection against malicious client/guest to prevent our output +* buffer growing without bound if client stops reading data. This +* should rarely trigger, because we have earlier throttling code +* which stops issuing framebuffer updates and drops audio data +* if the throttle_output_offset value is exceeded. So we only reach
ui/vnc.c gboolean vnc_client_io(QIOChannel *ioc G_GNUC_UNUSED, · line 1552
Quoted 10 lines
ui/vnc.c +10
-0+* this higher level if a huge number of pseudo-encodings get +* triggered while data can't be sent on the socket. +* +* NB throttle_output_offset can be zero during early protocol +* handshake, or from the job thread's VncState clone +*/ +if (vs->throttle_output_offset !=0 && + vs->output.offset > (vs->throttle_output_offset * +VNC_THROTTLE_OUTPUT_LIMIT_SCALE )) { +vnc_disconnect_start (vs);
It seems to me that the main source of data, the display, bypass this check.
The vnc_worker_thread_loop() uses a local VncState & buffer. The result is moved to the vs->jobs_buffer, which is later moved in vnc_jobs_consume_buffer() to the vs->output in bottom-half.
So in theory, it seems it would be possible for a client to make several update-request (assuming guest display content changed), and have several vnc jobs queued. In the unlikely events they would be consumed together, they would not respect the hard cap. I am not sure how the vnc-job queueing should be improved, just wanted to raise some concerns around that code and the fact it doesn't really respect the hard limits apparently. Am I wrong?
Perhaps the hard limit should also be put in vnc_jobs_consume_buffer() ? Then I can imagine synchronization issues if the hard limit changes before the job buffer are consumed.
May be we should limit the amount of jobs that can be queued? If we can estimate the max result buffer size of a job buffer, vnc_should_update() could take that into account?
Quoted 9 lines
ui/vnc.c +2
-1+return ; + }buffer_reserve (&vs->output, len);if (vs->ioc !=NULL &&buffer_empty (&vs->output)) { --2.14.3
Signature · 1 line
Marc-André Lureau
Re: [Qemu-devel] [PATCH v1 11/13] ui: place a hard cap on VNC server output buffer size
Quoted 90 lines · Marc-André Lureau
Hi
Quoted 62 lines · Daniel P. Berrange
The previous patches fix problems with throttling of forced framebuffer updates and audio data capture that would cause the QEMU output buffer size to grow without bound. Those fixes are graceful in that once the client catches up with reading data from the server, everything continues operating normally.
There is some data which the server sends to the client that is impractical to throttle. Specifically there are various pseudo framebuffer update encodings to inform the client of things like desktop resizes, pointer changes, audio playback start/stop, LED state and so on. These generally only involve sending a very small amount of data to the client, but a malicious guest might be able to do things that trigger these changes at a very high rate. Throttling them is not practical as missed or delayed events would cause broken behaviour for the client.
This patch thus takes a more forceful approach of setting an absolute upper bound on the amount of data we permit to be present in the output buffer at any time. The previous patch set a threshold for throttling the output buffer by allowing an amount of data equivalent to one complete framebuffer update and one seconds worth of audio data. On top of this it allowed for one further forced framebuffer update to be queued.
To be conservative, we thus take that throttling threshold and multiply it by 5 to form an absolute upper bound. If this bound is hit during
vnc_write()we forceably disconnect the client, refusing to queue further data. This limit is high enough that it should never be hit unless a malicious client is trying to exploit the sever, or the network is completely saturated preventing any sending of data on the socket.This completes the fix for CVE-2017-15124 started in the previous patches.
Signed-off-by Daniel P. Berrange <[email protected]>--- ui/vnc.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)ui/vnc.c +17
-0@@ -1521,8 +1521,37 @@ gboolean vnc_client_io(QIOChannel *ioc G_GNUC_UNUSED, } +/* +* Scale factor to apply to vs->throttle_output_offset when checking for +* hard limit. Worst case normal usage could be x2, if we have a complete +* incremental update and complete forced update in the output buffer. +* So x3 should be good enough, but we pick x5 to be conservative and thus +* (hopefully) never trigger incorrectly. +*/ +#define VNC_THROTTLE_OUTPUT_LIMIT_SCALE 5 +void vnc_write (VncState *vs,const void *data,size_t len) { +if (vs->disconnecting) { +return ; + } +/* Protection against malicious client/guest to prevent our output +* buffer growing without bound if client stops reading data. This +* should rarely trigger, because we have earlier throttling code +* which stops issuing framebuffer updates and drops audio data +* if the throttle_output_offset value is exceeded. So we only reach ui/vnc.c gboolean vnc_client_io(QIOChannel *ioc G_GNUC_UNUSED, · line 1552
Quoted 10 lines
ui/vnc.c +10
-0+* this higher level if a huge number of pseudo-encodings get +* triggered while data can't be sent on the socket. +* +* NB throttle_output_offset can be zero during early protocol +* handshake, or from the job thread's VncState clone +*/ +if (vs->throttle_output_offset !=0 && + vs->output.offset > (vs->throttle_output_offset * +VNC_THROTTLE_OUTPUT_LIMIT_SCALE )) { +vnc_disconnect_start (vs);It seems to me that the main source of data, the display, bypass this check.
The
vnc_worker_thread_loop()uses a local VncState & buffer. The result is moved to thevs->jobs_buffer, which is later moved invnc_jobs_consume_buffer()to thevs->outputin bottom-half.So in theory, it seems it would be possible for a client to make several update-request (assuming guest display content changed), and have several vnc jobs queued. In the unlikely events they would be consumed together, they would not respect the hard cap. I am not sure how the vnc-job queueing should be improved, just wanted to raise some concerns around that code and the fact it doesn't really respect the hard limits apparently. Am I wrong?
Quoted 7 lines
Perhaps the hard limit should also be put in
vnc_jobs_consume_buffer()? Then I can imagine synchronization issues if the hard limit changes before the job buffer are consumed.May be we should limit the amount of jobs that can be queued? If we can estimate the max result buffer size of a job buffer,
vnc_should_update()could take that into account?
The vnc_should_update() already prevents there being more than 1 queued job for the worker thread. So even if the client reuqests more updates, we won't start processing them until the worker thread as copied the job_buffer into output in the vnc_jobs_consume_buffer bottom half. So no matter what the client requests, and how frequently the guest display updates, we're still limiting output buffer size in the vnc_update_client method. This vnc_write protection only needs to cope with non-display updates, for things like psuedo-encoding messages.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 12/13 ui: add trace events related to VNC client throttling
In reply to Daniel P. BerrangeThe VNC client throttling is quite subtle so will benefit from having trace points available for live debugging.
--- ui/trace-events | 7 +++++++ ui/vnc.c | 23 +++++++++++++++++++++++ 2 files changed, 30 insertions(+)
ui/trace-events +7 -0
@@ -35,6 +35,13 @@ vnc_client_connect(void *state, void *ioc) "VNC client connect state=%p ioc=%p" vnc_client_disconnect_start(void *state, void *ioc) "VNC client disconnect start state=%p ioc=%p" vnc_client_disconnect_finish(void *state, void *ioc) "VNC client disconnect finish state=%p ioc=%p" vnc_client_io_wrap(void *state, void *ioc, const char *type) "VNC client I/O wrap state=%p ioc=%p type=%s" +vnc_client_throttle_threshold(void *state, void *ioc, size_t oldoffset, size_t offset, int client_width, int client_height, int bytes_per_pixel, void *audio_cap) "VNC client throttle threshold state=%p ioc=%p oldoffset=%zu newoffset=%zu width=%d height=%d bpp=%d audio=%p" +vnc_client_throttle_incremental(void *state, void *ioc, int job_update, size_t offset) "VNC client throttle incremental state=%p ioc=%p job-update=%d offset=%zu" +vnc_client_throttle_forced(void *state, void *ioc, int job_update, size_t offset) "VNC client throttle forced state=%p ioc=%p job-update=%d offset=%zu" +vnc_client_throttle_audio(void *state, void *ioc, size_t offset) "VNC client throttle audio state=%p ioc=%p offset=%zu" +vnc_client_unthrottle_forced(void *state, void *ioc) "VNC client unthrottle forced offset state=%p ioc=%p" +vnc_client_unthrottle_incremental(void *state, void *ioc, size_t offset) "VNC client unthrottle incremental state=%p ioc=%p offset=%zu" +vnc_client_output_limit(void *state, void *ioc, size_t offset, size_t threshold) "VNC client output limit state=%p ioc=%p offset=%zu threshold=%zu" vnc_auth_init(void *display, int websock, int auth, int subauth) "VNC auth init state=%p websock=%d auth=%d subauth=%d" vnc_auth_start(void *state, int method) "VNC client auth start state=%p method=%d" vnc_auth_pass(void *state, int method) "VNC client auth passed state=%p method=%d"
ui/vnc.c +23 -0
@@ -1011,6 +1011,12 @@ static void vnc_update_throttle_offset(VncState *vs)*/ offset =MAX (offset,1024 *1024 ); +if (vs->throttle_output_offset != offset) { +trace_vnc_client_throttle_threshold ( + vs, vs->ioc, vs->throttle_output_offset, offset, vs->client_width, + vs->client_height, vs->client_pf.bytes_per_pixel, vs->audio_cap); + } + vs->throttle_output_offset = offset; }
@@ -1028,6 +1034,8 @@ static bool vnc_should_update(VncState *vs) vs->job_update == VNC_STATE_UPDATE_NONE) {return true; } +trace_vnc_client_throttle_incremental ( + vs, vs->ioc, vs->job_update, vs->output.offset);break ;case VNC_STATE_UPDATE_FORCE :/* Only allow forced updates if the pending send queue
@@ -1042,6 +1050,8 @@ static bool vnc_should_update(VncState *vs) vs->job_update == VNC_STATE_UPDATE_NONE) {return true; } +trace_vnc_client_throttle_forced ( + vs, vs->ioc, vs->job_update, vs->force_update_offset);break ; }return false;
@@ -1158,6 +1168,8 @@ static void audio_capture(void *opaque, void *buf, int size)vnc_write_u16 (vs,VNC_MSG_SERVER_QEMU_AUDIO_DATA );vnc_write_u32 (vs, size);vnc_write (vs, buf, size); + }else { +trace_vnc_client_throttle_audio (vs, vs->ioc, vs->output.offset); }vnc_unlock_output (vs);vnc_flush (vs);
@@ -1328,6 +1340,7 @@ ssize_t vnc_client_write_buf(VncState *vs, const uint8_t *data, size_t datalen)*/ static ssize_t vnc_client_write_plain (VncState *vs) { +size_t offset;ssize_t ret;#ifdef CONFIG_VNC_SASL
@@ -1348,11 +1361,19 @@ static ssize_t vnc_client_write_plain(VncState *vs)return 0 ;if (ret >= vs->force_update_offset) { +if (vs->force_update_offset !=0 ) { +trace_vnc_client_unthrottle_forced (vs, vs->ioc); + } vs->force_update_offset =0 ; }else { vs->force_update_offset -= ret; } + offset = vs->output.offset;buffer_advance (&vs->output, ret); +if (offset >= vs->throttle_output_offset && + vs->output.offset < vs->throttle_output_offset) { +trace_vnc_client_unthrottle_incremental (vs, vs->ioc, vs->output.offset); + }if (vs->output.offset ==0 ) {if (vs->ioc_tag) {
@@ -1549,6 +1570,8 @@ void vnc_write(VncState *vs, const void *data, size_t len)if (vs->throttle_output_offset !=0 && vs->output.offset > (vs->throttle_output_offset *VNC_THROTTLE_OUTPUT_LIMIT_SCALE )) { +trace_vnc_client_output_limit (vs, vs->ioc, vs->output.offset, + vs->throttle_output_offset);vnc_disconnect_start (vs);return ; }
Signature · 1 line
2.14.3
PATCH 13/13 ui: mix misleading comments & return types of VNC I/O helper methods
In reply to Daniel P. BerrangeWhile the QIOChannel APIs for reading/writing data return ssize_t, with negative value indicating an error, the VNC code passes this return value through the vnc_client_io_error() method. This detects the error condition, disconnects the client and returns 0 to indicate error. Thus all the VNC helper methods should return size_t (unsigned), and misleading comments which refer to the possibility of negative return values need fixing.
--- ui/vnc-auth-sasl.c | 8 ++++---- ui/vnc-auth-sasl.h | 4 ++-- ui/vnc.c | 29 +++++++++++++++-------------- ui/vnc.h | 6 +++--- 4 files changed, 24 insertions(+), 23 deletions(-)
ui/vnc-auth-sasl.c +4 -4
@@ -48,9 +48,9 @@ void vnc_sasl_client_cleanup(VncState *vs) } -long vnc_client_write_sasl (VncState *vs) +size_t vnc_client_write_sasl (VncState *vs) { -long ret; +size_t ret; VNC_DEBUG("Write SASL: Pending output %p size %zd offset %zd " "Encoded: %p size %d offset %d\n",
@@ -106,9 +106,9 @@ long vnc_client_write_sasl(VncState *vs) } -long vnc_client_read_sasl (VncState *vs) +size_t vnc_client_read_sasl (VncState *vs) { -long ret; +size_t ret;uint8_t encoded[4096 ];const char *decoded;unsigned int decodedLen;
ui/vnc-auth-sasl.h +2 -2
@@ -65,8 +65,8 @@ struct VncDisplaySASL {void vnc_sasl_client_cleanup (VncState *vs); -long vnc_client_read_sasl (VncState *vs); -long vnc_client_write_sasl (VncState *vs); +size_t vnc_client_read_sasl (VncState *vs); +size_t vnc_client_write_sasl (VncState *vs);void start_auth_sasl (VncState *vs);
ui/vnc.c +15 -14
@@ -1272,7 +1272,7 @@ void vnc_disconnect_finish(VncState *vs)g_free (vs); } -ssize_t vnc_client_io_error (VncState *vs,ssize_t ret,Error **errp) +size_t vnc_client_io_error (VncState *vs,ssize_t ret,Error **errp) {if (ret <=0 ) {if (ret ==0 ) {
@@ -1315,9 +1315,9 @@ void vnc_client_error(VncState *vs)* * Returns the number of bytes written, which may be less than * the requested 'datalen' if the socket would block. Returns -* -1 on error, and disconnects the client socket. +* 0 on I/O error, and disconnects the client socket. */ -ssize_t vnc_client_write_buf (VncState *vs,const uint8_t *data,size_t datalen) +size_t vnc_client_write_buf (VncState *vs,const uint8_t *data,size_t datalen) {Error *err =NULL ;ssize_t ret;
@@ -1335,13 +1335,13 @@ ssize_t vnc_client_write_buf(VncState *vs, const uint8_t *data, size_t datalen)* will switch the FD poll() handler back to read monitoring. * * Returns the number of bytes written, which may be less than -* the buffered output data if the socket would block. Returns -* -1 on error, and disconnects the client socket. +* the buffered output data if the socket would block. Returns +* 0 on I/O error, and disconnects the client socket. */ -static ssize_t vnc_client_write_plain (VncState *vs) +static size_t vnc_client_write_plain (VncState *vs) {size_t offset; -ssize_t ret; +size_t ret;#ifdef CONFIG_VNC_SASL VNC_DEBUG("Write Plain: Pending output %p size %zd offset %zd. Wait SSF %d\n",
@@ -1442,9 +1442,9 @@ void vnc_read_when(VncState *vs, VncReadEvent *func, size_t expecting)* * Returns the number of bytes read, which may be less than * the requested 'datalen' if the socket would block. Returns -* -1 on error, and disconnects the client socket. +* 0 on I/O error or EOF, and disconnects the client socket. */ -ssize_t vnc_client_read_buf (VncState *vs,uint8_t *data,size_t datalen) +size_t vnc_client_read_buf (VncState *vs,uint8_t *data,size_t datalen) {ssize_t ret;Error *err =NULL ;
@@ -1460,12 +1460,13 @@ ssize_t vnc_client_read_buf(VncState *vs, uint8_t *data, size_t datalen)* when not using any SASL SSF encryption layers. Will read as much * data as possible without blocking. * -* Returns the number of bytes read. Returns -1 on error, and -* disconnects the client socket. +* Returns the number of bytes read, which may be less than +* the requested 'datalen' if the socket would block. Returns +* 0 on I/O error or EOF, and disconnects the client socket. */ -static ssize_t vnc_client_read_plain (VncState *vs) +static size_t vnc_client_read_plain (VncState *vs) { -ssize_t ret; +size_t ret;VNC_DEBUG ("Read plain %p size %zd offset %zd\n" , vs->input.buffer, vs->input.capacity, vs->input.offset);buffer_reserve (&vs->input,4096 );
@@ -1491,7 +1492,7 @@ static void vnc_jobs_bh(void *opaque)*/ static int vnc_client_read (VncState *vs) { -ssize_t ret; +size_t ret;#ifdef CONFIG_VNC_SASL if (vs->sasl.conn && vs->sasl.runSSF)
ui/vnc.h +3 -3
@@ -524,8 +524,8 @@ gboolean vnc_client_io(QIOChannel *ioc,GIOCondition condition, void *opaque); -ssize_t vnc_client_read_buf (VncState *vs,uint8_t *data,size_t datalen); -ssize_t vnc_client_write_buf (VncState *vs,const uint8_t *data,size_t datalen); +size_t vnc_client_read_buf (VncState *vs,uint8_t *data,size_t datalen); +size_t vnc_client_write_buf (VncState *vs,const uint8_t *data,size_t datalen);/* Protocol I/O functions */ void vnc_write (VncState *vs,const void *data,size_t len);
@@ -544,7 +544,7 @@ uint32_t read_u32(uint8_t *data, size_t offset);/* Protocol stage functions */ void vnc_client_error (VncState *vs); -ssize_t vnc_client_io_error (VncState *vs,ssize_t ret,Error **errp); +size_t vnc_client_io_error (VncState *vs,ssize_t ret,Error **errp);void start_client_init (VncState *vs);void start_auth_vnc (VncState *vs);
Signature · 1 line
2.14.3
Hi Daniel,
For the series:
With one small nit on patch 1.
Thanks,
Darren.
Quoted 47 lines · Daniel P. Berrange
In the 2.11 release we fixed CVE-2017-15268, which allowed the VNC websockets server to consume arbitrary memory when a slow client was connected. I have since discovered that this same type of problem can be triggered in several other ways in the regular (non-websockets) VNC server. This patch series attempts to fix this problem by limiting framebuffer updates and other data sent from server to client. The mitigating factor is that you need to have successfully authenticated with the VNC server to trigger these new flaws. This new more general flaw is assigned CVE-2017-15124 by the Red Hat security team.
The key patches containing the security fix are 9, 10, 11.
Since this code is incredibly subtle & hard to understand though, the first 8 patches do a bunch of independant cleanups/refactoring to make the security fixes clearer. The last two patches are just some extra cleanup / help for future maint.
Daniel P. Berrange (13):
ui: remove 'sync' parametr fromvnc_update_client
ui: remove unreachable code invnc_update_client
ui: remove redundant indentation invnc_client_update
ui: avoid pointless VNC updates if framebuffer isn't dirty
ui: track how much decoded data we consumed when doing SASL encoding
ui: introduce enum to track VNC client framebuffer update request
state
ui: correctly reset framebuffer update state after processing dirty
regions
ui: refactor code for determining if an update should be sent to the
client
ui: fix VNC client throttling when audio capture is active
ui: fix VNC client throttling when forced update is requested
ui: place a hard cap on VNC server output buffer size
ui: add trace events related to VNC client throttling
ui: mix misleading comments & return types of VNC I/O helper methodsui/trace-events | 7 ++ ui/vnc-auth-sasl.c | 16 ++- ui/vnc-auth-sasl.h | 5 +- ui/vnc-jobs.c | 5 + ui/vnc.c | 320 ++++++++++++++++++++++++++++++++++++++--------------- ui/vnc.h | 28 ++++- 6 files changed, 277 insertions(+), 104 deletions(-)Signature · 1 line
2.14.3
Hi
----- Original Message -----
Quoted 10 lines
In the 2.11 release we fixed CVE-2017-15268, which allowed the VNC websockets server to consume arbitrary memory when a slow client was connected. I have since discovered that this same type of problem can be triggered in several other ways in the regular (non-websockets) VNC server. This patch series attempts to fix this problem by limiting framebuffer updates and other data sent from server to client. The mitigating factor is that you need to have successfully authenticated with the VNC server to trigger these new flaws. This new more general flaw is assigned CVE-2017-15124 by the Red Hat security team.
Quoted 6 lines
The key patches containing the security fix are 9, 10, 11.
Since this code is incredibly subtle & hard to understand though, the first 8 patches do a bunch of independant cleanups/refactoring to make the security fixes clearer. The last two patches are just some extra cleanup / help for future maint.
I already reviewed the series privately, and in general, I'd ack it.
Did you manage to run some throttle tests? (I tried to trigger them manually by slowing artificially network or calling framebuffer_update_request() in gdb, I didn't manage yet :-/)
Quoted 19 lines
Daniel P. Berrange (13):
ui: remove 'sync' parametr fromvnc_update_client
ui: remove unreachable code invnc_update_client
ui: remove redundant indentation invnc_client_update
ui: avoid pointless VNC updates if framebuffer isn't dirty
ui: track how much decoded data we consumed when doing SASL encoding
ui: introduce enum to track VNC client framebuffer update request
state
ui: correctly reset framebuffer update state after processing dirty
regions
ui: refactor code for determining if an update should be sent to the
client
ui: fix VNC client throttling when audio capture is active
ui: fix VNC client throttling when forced update is requested
ui: place a hard cap on VNC server output buffer size
ui: add trace events related to VNC client throttling
ui: mix misleading comments & return types of VNC I/O helper methods
Quoted 9 lines
ui/trace-events | 7 ++ ui/vnc-auth-sasl.c | 16 ++- ui/vnc-auth-sasl.h | 5 +- ui/vnc-jobs.c | 5 + ui/vnc.c | 320 ++++++++++++++++++++++++++++++++++++++--------------- ui/vnc.h | 28 ++++- 6 files changed, 277 insertions(+), 104 deletions(-)
Series:
Quoted 4 lines
--
2.14.3
Quoted 20 lines · Marc-André Lureau
Hi
----- Original Message -----
Quoted 10 lines
In the 2.11 release we fixed CVE-2017-15268, which allowed the VNC websockets server to consume arbitrary memory when a slow client was connected. I have since discovered that this same type of problem can be triggered in several other ways in the regular (non-websockets) VNC server. This patch series attempts to fix this problem by limiting framebuffer updates and other data sent from server to client. The mitigating factor is that you need to have successfully authenticated with the VNC server to trigger these new flaws. This new more general flaw is assigned CVE-2017-15124 by the Red Hat security team.
Quoted 6 lines
The key patches containing the security fix are 9, 10, 11.
Since this code is incredibly subtle & hard to understand though, the first 8 patches do a bunch of independant cleanups/refactoring to make the security fixes clearer. The last two patches are just some extra cleanup / help for future maint.
Quoted 5 lines
I already reviewed the series privately, and in general, I'd ack it.
Did you manage to run some throttle tests? (I tried to trigger them manually by slowing artificially network or calling
framebuffer_update_request()in gdb, I didn't manage yet :-/)
I used two approaches. First a MITM socket server that just throttles the rate at which is reads from QEMU. Second, I wrote an intentionally malicious VNC client to artificially exercise the codepaths & verify they are fixed.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 35 lines · Daniel P. Berrange
In the 2.11 release we fixed CVE-2017-15268, which allowed the VNC websockets server to consume arbitrary memory when a slow client was connected. I have since discovered that this same type of problem can be triggered in several other ways in the regular (non-websockets) VNC server. This patch series attempts to fix this problem by limiting framebuffer updates and other data sent from server to client. The mitigating factor is that you need to have successfully authenticated with the VNC server to trigger these new flaws. This new more general flaw is assigned CVE-2017-15124 by the Red Hat security team.
The key patches containing the security fix are 9, 10, 11.
Since this code is incredibly subtle & hard to understand though, the first 8 patches do a bunch of independant cleanups/refactoring to make the security fixes clearer. The last two patches are just some extra cleanup / help for future maint.
Daniel P. Berrange (13):
ui: remove 'sync' parametr fromvnc_update_client
ui: remove unreachable code invnc_update_client
ui: remove redundant indentation invnc_client_update
ui: avoid pointless VNC updates if framebuffer isn't dirty
ui: track how much decoded data we consumed when doing SASL encoding
ui: introduce enum to track VNC client framebuffer update request
state
ui: correctly reset framebuffer update state after processing dirty
regions
ui: refactor code for determining if an update should be sent to the
client
ui: fix VNC client throttling when audio capture is active
ui: fix VNC client throttling when forced update is requested
ui: place a hard cap on VNC server output buffer size
ui: add trace events related to VNC client throttling
ui: mix misleading comments & return types of VNC I/O helper methods
Quoted 8 lines
ui/trace-events | 7 ++ ui/vnc-auth-sasl.c | 16 ++- ui/vnc-auth-sasl.h | 5 +- ui/vnc-jobs.c | 5 + ui/vnc.c | 320 ++++++++++++++++++++++++++++++++++++++--------------- ui/vnc.h | 28 ++++- 6 files changed, 277 insertions(+), 104 deletions(-)
For the series:
Signature · 1 line
Marc-André Lureau