opengl rendering in the sdl window

This patch comes from xen-unstable and adds opengl support for rendering the guest framebuffer in the SDL window.

Signed-off-by Stefano Stabellini <[email protected]>

This patch comes from xen-unstable and adds opengl support for rendering the guest framebuffer in the SDL window.
SDL is needed anyway to open the window and handle the events. Opengl rendering is optional and can be turned off at both compile time and run time (--disable-opengl).
Some of the benefits of using opengl are:

-faster rendering, less CPU intensive, especially with good graphic cards;

-makes the window resizing possible and hardware accelerated, thus very efficient and smooth;

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.

Quoted 10 lines · Stefano Stabellini

This patch comes from xen-unstable and adds opengl support for rendering the guest framebuffer in the SDL window.
SDL is needed anyway to open the window and handle the events. Opengl rendering is optional and can be turned off at both compile time and run time (--disable-opengl).
Some of the benefits of using opengl are:

-faster rendering, less CPU intensive, especially with good graphic cards;

Have you measured this or is this just intuition? I've measured it with gtk-vnc and I did not observe any CPU usage decrease in using OpenGL for rendering verses an XShmImage.

Quoted 3 lines

-makes the window resizing possible and hardware accelerated, thus very efficient and smooth;

This is neat, but, I'm unsure if the right way to support OpenGL is through SDL. For instance, there were Cocoa OpenGL patches posted a bit ago that would be largely similar. It may make more sense to have an OpenGL front-end that has conditional code for SDL/Cocoa/X/etc.

Then again, I've been kicking around the idea of doing a GTK front-end. An obvious thing to do here would be a glext based OpenGL version (as we do in gtk-vnc).

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.

Quoted 4 lines

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.

As long as that's optional, because in a server deployment scenario like oVirt I don't want to pull in the GTK stack just to run QEMU vms. We currently have a minimal OS image target of < 64 MB in size. Adding GTK and its deps will totally blow that limit.

Am 04.09.2008 um 11:42 schrieb Daniel P. Berrange:

Quoted 7 lines · Anthony Liguori

Quoted 5 lines

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a
proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.

Quoted 7 lines

As long as that's optional, because in a server deployment scenario like
oVirt I don't want to pull in the GTK stack just to run QEMU vms. We currently
have a minimal OS image target of < 64 MB in size. Adding GTK and its deps
will totally blow that limit.

All graphical frontends have been optional (--disable-sdl, undocumented --disable-gfx-check).

Quoted 18 lines

Am 04.09.2008 um 11:42 schrieb Daniel P. Berrange:

Quoted 14 lines · Anthony Liguori

Quoted 5 lines

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a
proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.

As long as that's optional, because in a server deployment scenario like
oVirt I don't want to pull in the GTK stack just to run QEMU vms. We currently
have a minimal OS image target of < 64 MB in size. Adding GTK and its deps
will totally blow that limit.

Quoted 2 lines

All graphical frontends have been optional (--disable-sdl, undocumented --disable-gfx-check).

It gets a little more complicated when you have to ship this still in binary packages though. Users typically expect us to enable all the compile time options which are a available for a particular OS distro, so they'd expect GTK enabled by default if we had that. At the same time many people won't want QEMU to have a dep on GTK.

It gets a little more complicated when you have to ship this still in binary packages though. Users typically expect us to enable all the compile time options which are a available for a particular OS distro,
so they'd expect GTK enabled by default if we had that. At the same time many people won't want QEMU to have a dep on GTK. I don't want to end up building multiple binary packages of QEMU, with & without
the frontend. It might warrant having a libqemu, and separate graphical
frontends, one GTK, one VNC,... That of course raises a question if
ABI for libqemu - it could be put in a private lib directory just for
the official frontends to use, and declared 'not for public use'.

On Wed, Sep 03, 2008 at 10:00:31PM -0500, Anthony Liguori wrote:

Actually I'm not so sure this was a good idea in the end. I'm seriously considering re-writing the GTK-VNC stuff to use Cairo, which in turn can use 2-d hardware acceleration primitives - it really doesn't need the full 3-d acceleration stack just for scaling.

Quoted 2 lines

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.

As long as that's optional, because in a server deployment scenario like oVirt I don't want to pull in the GTK stack just to run QEMU vms. We currently have a minimal OS image target of < 64 MB in size. Adding GTK and its deps will totally blow that limit.

Daniel

Quoted 7 lines · Daniel P. Berrange

On Wed, Sep 03, 2008 at 10:00:31PM -0500, Anthony Liguori wrote:

Actually I'm not so sure this was a good idea in the end. I'm seriously considering re-writing the GTK-VNC stuff to use Cairo, which in turn can use 2-d hardware acceleration primitives - it really doesn't need the full 3-d acceleration stack just for scaling.

I tried to originally write the GTK-VNC scaling stuff in Cairo. Could not get it to perform well at all. I'd be really interested if you had better luck with it.

Quoted 10 lines · Anthony Liguori

Quoted 8 lines · Daniel P. Berrange

On Wed, Sep 03, 2008 at 10:00:31PM -0500, Anthony Liguori wrote:

Actually I'm not so sure this was a good idea in the end. I'm seriously considering re-writing the GTK-VNC stuff to use Cairo, which in turn can use 2-d hardware acceleration primitives - it really doesn't need the full 3-d acceleration stack just for scaling.

Quoted 2 lines

I tried to originally write the GTK-VNC scaling stuff in Cairo. Could not get it to perform well at all. I'd be really interested if you had better luck with it.

I've just posted patches to the GTK-VNC devel list demonstrating use of Cairo for all rendering. I can't notice any serious drop in performance when enabling scaling with Cairo. If you confirm my tests, it'd be worth evaluating Cairo as an alternative to OpenGL in QEMU too.

Regards,
Daniel

Quoted 7 lines · Stefano Stabellini

This patch comes from xen-unstable and adds opengl support for rendering the guest framebuffer in the SDL window.
SDL is needed anyway to open the window and handle the events. Opengl rendering is optional and can be turned off at both compile time and run time (--disable-opengl).
Some of the benefits of using opengl are:

Quoted 3 lines

-faster rendering, less CPU intensive, especially with good graphic cards;

Have you measured this or is this just intuition? I've measured it with gtk-vnc and I did not observe any CPU usage decrease in using OpenGL for rendering verses an XShmImage.

Quoted 8 lines · Anthony Liguori

Quoted 7 lines · Stefano Stabellini

This patch comes from xen-unstable and adds opengl support for rendering the guest framebuffer in the SDL window.
SDL is needed anyway to open the window and handle the events. Opengl rendering is optional and can be turned off at both compile time and run time (--disable-opengl).
Some of the benefits of using opengl are:

Quoted 7 lines

Quoted 3 lines

-faster rendering, less CPU intensive, especially with good graphic cards;

Have you measured this or is this just intuition? I've measured it with gtk-vnc and I did not observe any CPU usage decrease in using OpenGL for rendering verses an XShmImage.

I wrote that because before my patches all the colour conversions were done in vga_template; after the sdl shared buffer patch the colour conversions are done by SDL that sadly most of the time still implements them using CPU based algorithms.
The opengl code should be able to offload all the colour conversions to the GPU.
So I was referring to SDL vs. OpenGL.

Then there is the video memory versus system memory discussion: the opengl patch loads the guest framebuffer into video memory then does the conversion and blitting video->video that should be accelerated. SDL_CreateRGBSurfaceFrom creates the surface in system memory so the blitting shouldn't be accelerated.
I am saying should and shouldn't because it also depends on the video card drivers and the opengl implementation.

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it.

Quoted 2 lines · Jamie Lokier, le Fri 05 Sep 2008 13:02:15 +0100, a écrit :

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it.

It is being deprecated and probably never works nowadays anyway :)

Quoted 5 lines · Samuel Thibault

Quoted 2 lines · Jamie Lokier, le Fri 05 Sep 2008 13:02:15 +0100, a écrit :

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it.

It is being deprecated and probably never works nowadays anyway :)

That's a shame as the alternatives are all slower on old video cards.

Quoted 5 lines · Jamie Lokier, le Sun 07 Sep 2008 00:27:23 +0100, a écrit :

Quoted 4 lines · Samuel Thibault

Quoted 2 lines · Jamie Lokier, le Fri 05 Sep 2008 13:02:15 +0100, a écrit :

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it.

Quoted 3 lines

Quoted 1 line

It is being deprecated and probably never works nowadays anyway :)

That's a shame as the alternatives are all slower on old video cards.

Yes, and because alternatives don't provide so much performance. Ideally, once switched to fullscreen you could just let the guest directly write to the actual video memory (particularly interesting for Xen and KVM).

Quoted 3 lines

Ideally, once switched to fullscreen you could just let the guest directly write to the actual video memory (particularly interesting for Xen and KVM).

You really don't want to have the guest writing directly to host video ram. Video ram tends to be high-latency, so you want to write to regular memory, then use a wide block transfer or DMA to copy to video ram.

If you really need zero-copy then you should have the renderer read the data directly out of the guest framebuffer.

Quoted 7 lines · Paul Brook, le Sun 07 Sep 2008 15:36:51 +0100, a écrit :

Quoted 3 lines

Ideally, once switched to fullscreen you could just let the guest directly write to the actual video memory (particularly interesting for Xen and KVM).

You really don't want to have the guest writing directly to host video ram. Video ram tends to be high-latency, so you want to write to regular memory, then use a wide block transfer or DMA to copy to video ram.

How often?

Which part(s) of the framebuffer?

Quoted 5 lines · Samuel Thibault

Quoted 4 lines · Paul Brook, le Sun 07 Sep 2008 15:36:51 +0100, a écrit :

Quoted 3 lines

Ideally, once switched to fullscreen you could just let the guest directly write to the actual video memory (particularly interesting for Xen and KVM).

Quoted 5 lines

Quoted 3 lines

You really don't want to have the guest writing directly to host video ram. Video ram tends to be high-latency, so you want to write to regular memory, then use a wide block transfer or DMA to copy to video ram.

How often?

You probably want to sync the updates to the vertical refresh. Updating more often than that is pointless.

Quoted 1 line

Which part(s) of the framebuffer?

Whichever parts have changed.

Quoted 3 lines

Quoted 1 line

Which part(s) of the framebuffer?

Whichever parts have changed.

Sure, how do you detect that?
The previous Xen memcmp method was eating 10% cpu for a 30Hz refresh, just for the memcmp. Now it uses the pagetable dirty bits and after some idleness, trapping, but having the guest just write to actual video memory (when it _has_ to be shown on the screen anyway) seems the fastest way.

It also depends what testcase you're benchmarking. A few pixels changing every frame is IMHO not a particularly interesting case.

The interesting cases are guest completely idle,

and large screen updates.

Quoted 1 line

and large screen updates.

In that case, why blitting from main memory to video memory? Sure video memory is slow. But the guest already does tricky stuff to cope with that...

Quoted 2 lines · Paul Brook, le Sun 07 Sep 2008 16:35:54 +0100, a écrit :

It also depends what testcase you're benchmarking. A few pixels changing every frame is IMHO not a particularly interesting case.

But we shouldn't spend time on it. For qemu it's trivial to get the dirtyness of course. For Xen/KVM it's far less.

Quoted 6 lines · Samuel Thibault

Quoted 2 lines · Paul Brook, le Sun 07 Sep 2008 16:35:54 +0100, a écrit :

It also depends what testcase you're benchmarking. A few pixels changing every frame is IMHO not a particularly interesting case.

But we shouldn't spend time on it. For qemu it's trivial to get the dirtyness of course. For Xen/KVM it's far less.

It's actually pretty simple with KVM.

FWIW, I don't see guests consume a tremendous amount of CPU when idle with KVM. Can ya'll post some numbers with OpenGL and QEMU that show reduced CPU consumption?

Quoted 10 lines · Samuel Thibault

Quoted 8 lines · Jamie Lokier, le Sun 07 Sep 2008 00:27:23 +0100, a écrit :

Quoted 5 lines · Samuel Thibault

Quoted 2 lines · Jamie Lokier, le Fri 05 Sep 2008 13:02:15 +0100, a écrit :

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it.

It is being deprecated and probably never works nowadays anyway :)

That's a shame as the alternatives are all slower on old video cards.

Quoted 4 lines

Yes, and because alternatives don't provide so much performance. Ideally, once switched to fullscreen you could just let the guest directly write to the actual video memory (particularly interesting for Xen and KVM).

You really don't want to have the guest writing directly to host video ram. Video ram tends to be high-latency, so you want to write to regular memory, then use a wide block transfer or DMA to copy to video ram.

If you really need zero-copy then you should have the renderer read the data directly out of the guest framebuffer.

Quoted 7 lines · Paul Brook

Quoted 6 lines · Samuel Thibault

Quoted 5 lines · Jamie Lokier, le Sun 07 Sep 2008 00:27:23 +0100, a écrit :

Quoted 4 lines · Samuel Thibault

Quoted 2 lines · Jamie Lokier, le Fri 05 Sep 2008 13:02:15 +0100, a écrit :

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it.

It is being deprecated and probably never works nowadays anyway :)

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

That's a shame as the alternatives are all slower on old video cards.

Yes, and because alternatives don't provide so much performance. Ideally, once switched to fullscreen you could just let the guest directly write to the actual video memory (particularly interesting for Xen and KVM).

You really don't want to have the guest writing directly to host video ram. Video ram tends to be high-latency, so you want to write to regular memory, then use a wide block transfer or DMA to copy to video ram.

Latency applies mainly when reading.

A good quality guest _expects_ that reading its video RAM will be slow, and only writes to its video RAM, and possible by preparing images off-screen then wide block transfers. :-)

Quoted 3 lines

A good quality guest _expects_ that reading its video RAM will be slow, and only writes to its video RAM, and possible by preparing images off-screen then wide block transfers. :-)

That is a really good point.
Usually the safe way to handle the video ram VS main memory problem is to write to main memory then copy the whole buffer to video ram every so often.
But in our case the guest thinks that he is writing to video ram already, so in theory he is already using all precautions needed to handle the latency problem.
It is actually probable that the guest is handling another buffer in his own memory and syncing that one with what he thinks to be the video ram.

Fwiw, in my experience with Xine and mplayer, drawing video updates to the screen using OpenGL was quite a lot slower than drawing them with XShmImage. That needs colour conversion as video doesn't even use RGB.

Probably OpenGL is faster on some hardware, and slower on some hardware. Don't assume it's always faster.

Yes it very much depends on the drivers and opengl implementation. It isn't always possible to map the guest framebuffer into video (texture) memory, and also some opengl implementations are not particularly fast at copying textures from CPU memory to texture memory.

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it. I think VMware uses it in full-screen mode.

Quoted 3 lines · Jamie Lokier

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it. I think VMware uses it in full-screen mode.

I went with OpenGL because I wanted the accelerated window resize feature, and because I think it will give us more opportunities for further improvements (for example using PBOs as soon as they are supported by at least one open source graphic card driver).

The problem with OpenGL is that it has horrible rendering issues if you are also using a compositing window manager like Compiz. You need an X server & driver which supports Redirected Direct Rendering [1]. No distro currently has any official support for this yet. AFAIK, the next Fedora 10 will be the first to support it, and even then only on Intel hardware[2]. ATI is still a fair way off, and NVidia support is nowhere to be seen.

For
accelerated resize of windows though you shouldn't need 3d - the graphics driver 2-d hardware acceleration support ought to be sufficient I'd imagine

Quoted 6 lines · Daniel P. Berrange

The problem with OpenGL is that it has horrible rendering issues if you
are also using a compositing window manager like Compiz. You need an X
server & driver which supports Redirected Direct Rendering [1]. No distro currently has any official support for this yet. AFAIK, the next Fedora 10 will be the first to support it, and even then only on Intel hardware[2].
ATI is still a fair way off, and NVidia support is nowhere to be seen.

I don't think this should impact our choice, these are all issues that will be solved sooner or later (hopefully sooner).

Quoted 6 lines

For
accelerated resize of windows though you shouldn't need 3d - the graphics driver 2-d hardware acceleration support ought to be sufficient I'd imagine

SDL doesn't offer this feature, and OpenGL is not just for 3D.

I'd like to see if it really makes a difference performance wise to render with OpenGL. I've had good results in the past using software bilinear interpolation to do scaling. If OpenGL doesn't provide a measurable performance advantage, doing it in software may be the right thing to do.

I'd like to see if it really makes a difference performance wise to render with OpenGL. I've had good results in the past using software bilinear interpolation to do scaling. If OpenGL doesn't provide a measurable performance advantage, doing it in software may be the right thing to do.

Quoted 5 lines · Anthony Liguori

I'd like to see if it really makes a difference performance wise to render with OpenGL. I've had good results in the past using software bilinear interpolation to do scaling. If OpenGL doesn't provide a measurable performance advantage, doing it in software may be the right thing to do.

Software interpolation, especially bilinear or other filter, is relatively slow if you're expanding a lot of pixels e.g. 640x384 to 1920x1200. OpenGL will surely beat it.

On the other hand, OpenGL can to be relatively slow (in my experience) if there is no colour conversion and scaling to do and X11-SHM would be a straight 2d blit on the server side.

In principle OpenGL should as fast as the 2d blit or faster on implementations where the image in client app's memory is a memory-mapped texture, but that seems to be still a work in progress on BSD and Linux. DRI2 was dropped this summer because the TTM texture memory manager was ousted in favour of the GEM texture memory manager... and they haven't caught up with each other yet. You need some kind of good shared texture memory to get great speed out of OpenGL displaying a texture which is being constantly updated by the client app.

Imho, if there is an OpenGL backend (overlaid on SDL, Xlib, or whatever), it should be a switch, for the next few years probably defaulting to X11-SHM (or plain X11 when remote), and changing the default to OpenGL in a few years when that's working really well everywhere.

Practically speaking, it's a royal pain. It means it has to be disabled by default. What's most annoying is that Ubuntu enables compiz by default so there are a lot of users that will be affected.

Quoted 3 lines · Anthony Liguori

Practically speaking, it's a royal pain. It means it has to be disabled by default. What's most annoying is that Ubuntu enables compiz by default so there are a lot of users that will be affected.

I'm under the impression that the Compiz problem in stable distros is that OpenGL rendering isn't redirected by the compositor, so it doesn't follow the transformed window in rotating-cube desktops, faded or wobbly windows etc. (Much like Xv for video players sits over everything).

The rotating-cube issue is a one annoyance - the bigger problem is that it'll render over the top of other windows, even when it doesn't have input focus, and when you move the window, the area where it used to be doesn't get refreshed. This makes overlapping window management pretty unusable :-(

Beside that I still think it would be a good idea to separate qemu and the gui into two separate processes, so you can close the GUI window and keep the VM running. It also solves the dependency issue for distros as the gtk frontend with all the dependencies can just go into a separate sub-package.

Quoted 5 lines · Gerd Hoffmann

Beside that I still think it would be a good idea to separate qemu and the gui into two separate processes, so you can close the GUI window and keep the VM running. It also solves the dependency issue for distros as the gtk frontend with all the dependencies can just go into a separate sub-package.

Extending VNC with a "shared memory" extension, similar to Xlib's MIT-SHM extension, and implementing it in QEMU and Gtk-VNC would be a nice way to do this.

Quoted 7 lines · Jamie Lokier

Quoted 5 lines · Gerd Hoffmann

Beside that I still think it would be a good idea to separate qemu and the gui into two separate processes, so you can close the GUI window and keep the VM running. It also solves the dependency issue for distros as the gtk frontend with all the dependencies can just go into a separate sub-package.

Quoted 3 lines

Extending VNC with a "shared memory" extension, similar to Xlib's MIT-SHM extension, and implementing it in QEMU and Gtk-VNC would be a nice way to do this.

Or use the monitor, which the gui needs anyway to implement the nice control buttons for the user. Which would it make easier to re-attach the gui to the vm as all you need to know is where the monitor is ...

Quoted 3 lines · Gerd Hoffmann

Or use the monitor, which the gui needs anyway to implement the nice control buttons for the user. Which would it make easier to re-attach the gui to the vm as all you need to know is where the monitor is ...

You want to transmit the image or the image-blit-please protocol over the monitor connection?

I think the monitor would need a few
improvements before it could do that well.

A nice thing about VNC-SHM would be the GUI could run on a different machine to the emulator (SHM not used then), since it only needs VNC and monitor, which can both be remote.

Quoted 9 lines · Jamie Lokier

Quoted 4 lines · Gerd Hoffmann

Or use the monitor, which the gui needs anyway to implement the nice control buttons for the user. Which would it make easier to re-attach the gui to the vm as all you need to know is where the monitor is ...

You want to transmit the image or the image-blit-please protocol over the monitor connection?

Negotiate a shared memory segment somehow, then send just notifications. Have to think about how to to that in detail, ideally this would be a shared memory segment shared by all three instances involved: the gui process, the qemu process and the X-Server. So qemu can render directly into a XShmImage, although it has no (direct) connection to the X-Server.

Quoted 5 lines · Gerd Hoffmann

Negotiate a shared memory segment somehow, then send just notifications. Have to think about how to to that in detail, ideally this would be a shared memory segment shared by all three instances involved: the gui process, the qemu process and the X-Server. So qemu can render directly into a XShmImage, although it has no (direct) connection to the X-Server.

This is exactly how my VNC Shm patches worked. The protocol extension is generic but it happens to work very well with an XShmImage.

Quoted 3 lines

A nice thing about VNC-SHM would be the GUI could run on a different machine to the emulator (SHM not used then), since it only needs VNC and monitor, which can both be remote.

For the remote case it certainly doesn't make sense to reinvent VNC.

But you'll probably use some management layer like libvirt anyway then,

ssh'ing into the remote machine for starting / stopping VM's isn't very handy.

I'm targeting the local desktop use case. But maybe vnc + shmem-extention works equally well here.

Quoted 5 lines · Gerd Hoffmann

Beside that I still think it would be a good idea to separate qemu and the gui into two separate processes, so you can close the GUI window and keep the VM running. It also solves the dependency issue for distros as the gtk frontend with all the dependencies can just go into a separate sub-package.

Extending VNC with a "shared memory" extension, similar to Xlib's MIT-SHM extension, and implementing it in QEMU and Gtk-VNC would be a nice way to do this.

Quoted 8 lines · Jamie Lokier

Quoted 6 lines · Gerd Hoffmann

Beside that I still think it would be a good idea to separate qemu and the gui into two separate processes, so you can close the GUI window and keep the VM running. It also solves the dependency issue for distros as the gtk frontend with all the dependencies can just go into a separate sub-package.

Quoted 4 lines

Extending VNC with a "shared memory" extension, similar to Xlib's MIT-SHM extension, and implementing it in QEMU and Gtk-VNC would be a nice way to do this.

Funny you should mention that. Anthony had code todo exactly that against
for both QEMU and GTK-VNC a while back. We had it in the GTK-VNC for a short while but removed it due to some race conditions in its impl, and it interracted badly with our OpenGL impl. We could easily revisit this
idea if it were thought to be important / useful.

Daniel

Quoted 15 lines · Daniel P. Berrange

Quoted 8 lines · Jamie Lokier

Quoted 6 lines · Gerd Hoffmann

Beside that I still think it would be a good idea to separate qemu and the gui into two separate processes, so you can close the GUI window and keep the VM running. It also solves the dependency issue for distros as the gtk frontend with all the dependencies can just go into a separate sub-package.

Quoted 4 lines

Extending VNC with a "shared memory" extension, similar to Xlib's MIT-SHM extension, and implementing it in QEMU and Gtk-VNC would be a nice way to do this.

Quoted 5 lines

Funny you should mention that. Anthony had code todo exactly that against
for both QEMU and GTK-VNC a while back. We had it in the GTK-VNC for a short while but removed it due to some race conditions in its impl, and it interracted badly with our OpenGL impl. We could easily revisit this
idea if it were thought to be important / useful.

The difficulty is deleting the shared memory segment reliably.

Normally, what you want to do is:

shmget()
shmat()
shmctl(IPC_RMID)

...

shmdt()

And the first sequence has to be as reliable and quick as possible because if you exit before the shmctl(), you'll leak the shared memory.

Unfortunately, you have to shmget() in the client and shmat() in the server and client, so that means:

shmget()
shmat()

...

notify server of key
wait for server to confirm shmat()

...

shmctl(IPC_RMID)

..

shmdt()

Which leaves a huge window open where bad things can happen. The client can exit, the client can crash. This is particularly troublesome in a library because it's really not nice to try and register an atexit() handler or something like that from a library.

It would be nice to come up with a solution to this.

Regards,

Anthony Liguori

Quoted 2 lines

Daniel

Hi,

Some OS (especially *nix) versions have pretty small size restrictions on shmem.
OS X for instance has a restriction of total 4mb for all processes.

Quoted 5 lines · Mike Kronenberg

Hi,

Some OS (especially *nix) versions have pretty small size restrictions on shmem.
OS X for instance has a restriction of total 4mb for all processes.

I bet that can be configured somewhere.

Linux had that too long ago (2.2.x or so?), and there was a knob to tune that limit.

But even shmem allocation failing shouldn't be a fundamental problem, vnc can just fallback to non-shm mode then.

cheers,
Gerd

The difficulty is deleting the shared memory segment reliably.

Normally, what you want to do is:

shmget()
shmat()
shmctl(IPC_RMID)

...

shmdt()

And the first sequence has to be as reliable and quick as possible because if you exit before the shmctl(), you'll leak the shared memory.

shmget()
shmat()

...

notify server of key
wait for server to confirm shmat()

...

shmctl(IPC_RMID)

..

shmdt()

Which leaves a huge window open where bad things can happen. The client can exit, the client can crash. This is particularly troublesome in a library because it's really not nice to try and register an atexit() handler or something like that from a library.

Quoted 10 lines · Anthony Liguori

shmget()
shmat()

...

notify server of key
wait for server to confirm shmat()

...

Quoted 8 lines

shmctl(IPC_RMID)

..

shmdt()

Which leaves a huge window open where bad things can happen. The client can exit, the client can crash. This is particularly troublesome in a library because it's really not nice to try and register an atexit() handler or something like that from a library.

Another way to do it, which ought to be quite reliable, is to fork() (and maybe exec) a child helper process which calls shmget() then prints out the ID, for the parent to read, then waits on its input. There is also a pipe from parent to child. If the parent closes the pipe to child or dies, the child kills the shm ID immediately. This catches all kinds of parent crashing and deliberate kills, and you can still close the pipe as soon as the parent confirms everyone is attached to the segment.

What do Gtk and/or Qt and/or Gecko do? They're likely to have thought about it.

-- Jamie

But you still leak if both processes are killed at once (think killall qemu-system-x86_64). An exec would be a little safer but it's pretty hideous.

Quoted 4 lines

What do Gtk and/or Qt and/or Gecko do? They're likely to have thought about it.

It's not a problem for Gtk because the window of possibly leaking is so short. It creates the shmid, send it to the x server, shmat, then shmctl. The key point is that it interacts with the X server synchronously so not much bad can happen while it's talking to the X server.

It's probably not a huge problem in practice, but I've never liked this. Sys V shared memory is really just crappy.

Quoted 10 lines · Jamie Lokier

Quoted 5 lines · Gerd Hoffmann

Beside that I still think it would be a good idea to separate qemu and the gui into two separate processes, so you can close the GUI window and keep the VM running. It also solves the dependency issue for distros as the gtk frontend with all the dependencies can just go into a separate sub-package.

Extending VNC with a "shared memory" extension, similar to Xlib's MIT-SHM extension, and implementing it in QEMU and Gtk-VNC would be a nice way to do this.

Funny you should mention that. Anthony had code todo exactly that against for both QEMU and GTK-VNC a while back. We had it in the GTK-VNC for a short while but removed it due to some race conditions in its impl, and it interracted badly with our OpenGL impl. We could easily revisit this idea if it were thought to be important / useful.

Just thought I'd mention one other thing - it is not actually clearcut / or guarenteed that XShm + VNC would improve performance in all circumstances. This is because X pixmaps created via XShm apparently cannot live in video memory, and thus it is not so easy to get full performance advantage of hardware acceleration - this could impact scaling for example.

The need to
perform XSync operations when updating the pixmaps may also negate the benefit of avoiding the socket I/O.

Quoted 6 lines · Daniel P. Berrange

Just thought I'd mention one other thing - it is not actually clearcut / or guarenteed that XShm + VNC would improve performance in all circumstances. This is because X pixmaps created via XShm apparently cannot live in video memory, and thus it is not so easy to get full performance advantage of hardware acceleration - this could impact scaling for example.

I haven't seen any scaling with VNC so far, but it would be nice sometimes.

How does sending the image over the wire then asking the server to scale it using RENDER improve on sending it over XShm then asking the server to xscale the pixel using RENDER? Or is there something else you can do when you don't use XShm?

But copying data isn't super expensive assuming you're hitting the cpu cache (which you probably are). So I don't know how much it saves in real life.

Quoted 2 lines · Anthony Liguori

Practically speaking, it's a royal pain. It means it has to be disabled by default. What's most annoying is that Ubuntu enables compiz by default so there are a lot of users that will be affected.

I'm under the impression that the Compiz problem in stable distros is that OpenGL rendering isn't redirected by the compositor, so it doesn't follow the transformed window in rotating-cube desktops, faded or wobbly windows etc. (Much like Xv for video players sits over everything).

If that's all (not crashes etc.) that might be acceptable.

-- Jamie

Quoted 6 lines · Stefano Stabellini

Quoted 4 lines · Jamie Lokier

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it. I think VMware uses it in full-screen mode.

Quoted 5 lines

I went with OpenGL because I wanted the accelerated window resize feature, and because I think it will give us more opportunities for further improvements (for example using PBOs as soon as they are supported by at least one open source graphic card driver).

The problem with OpenGL is that it has horrible rendering issues if you are also using a compositing window manager like Compiz.

You need an X
server & driver which supports Redirected Direct Rendering [1]. No distro currently has any official support for this yet.

AFAIK, the next Fedora 10
will be the first to support it, and even then only on Intel hardware[2]. ATI is still a fair way off, and NVidia support is nowhere to be seen. For accelerated resize of windows though you shouldn't need 3d - the graphics driver 2-d hardware acceleration support ought to be sufficient I'd imagine

Quoted 14 lines · Daniel P. Berrange

Quoted 11 lines · Stefano Stabellini

Quoted 4 lines · Jamie Lokier

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it. I think VMware uses it in full-screen mode.

I went with OpenGL because I wanted the accelerated window resize feature, and because I think it will give us more opportunities for further improvements (for example using PBOs as soon as they are supported by at least one open source graphic card driver).

Quoted 2 lines

The problem with OpenGL is that it has horrible rendering issues if you are also using a compositing window manager like Compiz.

I would say that this is a compiz problem.

Quoted 3 lines

You need an X
server & driver which supports Redirected Direct Rendering [1]. No distro currently has any official support for this yet.

Or a distro problem.

Quoted 6 lines

AFAIK, the next Fedora 10
will be the first to support it, and even then only on Intel hardware[2]. ATI is still a fair way off, and NVidia support is nowhere to be seen. For accelerated resize of windows though you shouldn't need 3d - the graphics driver 2-d hardware acceleration support ought to be sufficient I'd imagine

I don't see how distros can enable compiz if it breaks opengl applications like this.

Quoted 5 lines

Quoted 3 lines

The problem with OpenGL is that it has horrible rendering issues if you are also using a compositing window manager like Compiz.

I would say that this is a compiz problem.

Not really. Its a limitation of current X server driver architecture which is being addressed by DRI2 / Redirected direct rendering work

Quoted 2 lines

Or a distro problem.

The distros can't do anything about it, until X driver architecture work is done.

This is exactly why Fedora does not yet enable compiz by default. Many users still enable it though.

Quoted 4 lines

Daniel P. Berrange wrote:
The distros can't do anything about it, until X driver architecture work is done.

The distros can avoid shipping compiz until it is resolved one way or another. Are wobbly windows worth breaking applications?

Quoted 3 lines

This is exactly why Fedora does not yet enable compiz by default. Many users still enable it though.

Ah, okay then. The gui should probably say something about voiding the warranty, etc.

I completely agree with Avi: Opengl has been around since 1992, and now you tell me that it is better not to use it because it breaks Compiz?! To say the least it would be a bad long term engineering decision.

Quoted 3 lines · Stefano Stabellini

I completely agree with Avi: Opengl has been around since 1992, and now you tell me that it is better not to use it because it breaks Compiz?! To say the least it would be a bad long term engineering decision.

It would be a bad decision *if* OpenGL were the only technical solution to the problem. There are a variety of possible solutions with various pluses & minuses. If it is possible to do an implementation which uses hardware acceleration for scaling to get the same level of performance as the OpenGL proposal, it is good sense to evaluate it. Blindly going for OpenGL when it has known problems without evaluating alternatives would be the bad engineering decision.

Quoted 5 lines · Daniel P. Berrange

Quoted 3 lines · Stefano Stabellini

I completely agree with Avi: Opengl has been around since 1992, and now you tell me that it is better not to use it because it breaks Compiz?! To say the least it would be a bad long term engineering decision.

Quoted 7 lines

It would be a bad decision *if* OpenGL were the only technical solution to the problem. There are a variety of possible solutions with various pluses & minuses. If it is possible to do an implementation which uses hardware acceleration for scaling to get the same level of performance as the OpenGL proposal, it is good sense to evaluate it. Blindly going for OpenGL when it has known problems without evaluating alternatives would be the bad engineering decision.

You are right that we have a variety of possible solution each one with its own pros and cons.
I think removing SDL(+Opengl) for Gtk and Cairo is not a good idea *but* we could add another Gtk based fronted.

Quoted 26 lines

Quoted 9 lines · Daniel P. Berrange

Quoted 6 lines · Stefano Stabellini

I completely agree with Avi: Opengl has been around since 1992, and now
you tell me that it is better not to use it because it breaks Compiz?!
To say the least it would be a bad long term engineering decision.

Quoted 13 lines

It would be a bad decision *if* OpenGL were the only technical solution
to the problem. There are a variety of possible solutions with various
pluses & minuses. If it is possible to do an implementation which uses
hardware acceleration for scaling to get the same level of performance
as the OpenGL proposal, it is good sense to evaluate it. Blindly going
for OpenGL when it has known problems without evaluating alternatives
would be the bad engineering decision.

Quoted 6 lines

You are right that we have a variety of possible solution each one with
its own pros and cons.
I think removing SDL(+Opengl) for Gtk and Cairo is not a good idea *but*
we could add another Gtk based fronted.

Sorry, I wasn't being entirely clear in my suggestion. Cairo itself is not
tied to any toolkit - its dependancy is on libX11 / the RENDER extension.
So, use of Cairo is rather tangential to question of GTK vs SDL - you can
use Cairo with either of them. That said, it may turn out that SDL does
not expose enough of its X11 internals to allow an efficient use of Cairo.
It would need some experimentation to determine for sure, because the docs
aren't entirely clear on this question.

Quoted 5 lines

Sorry, I wasn't being entirely clear in my suggestion. Cairo itself is not
tied to any toolkit - its dependancy is on libX11 / the RENDER extension.
So, use of Cairo is rather tangential to question of GTK vs SDL - you

Quoted 8 lines

can
use Cairo with either of them. That said, it may turn out that SDL does
not expose enough of its X11 internals to allow an efficient use of Cairo.
It would need some experimentation to determine for sure, because the docs
aren't entirely clear on this question.

It would definitely make things much less portable... Cairo itself is quite a beast to port I heard... (not everyone uses X11...)

It would definitely make things much less portable...
Cairo itself is quite a beast to port I heard... (not everyone uses X11...)

François.

Quoted 5 lines

It would definitely make things much less portable... Cairo itself is quite a beast to port I heard... (not everyone uses X11...)

It compiles fine on Windows.

Quoted 7 lines · Jamie Lokier

Fwiw, in my experience with Xine and mplayer, drawing video updates to the screen using OpenGL was quite a lot slower than drawing them with XShmImage. That needs colour conversion as video doesn't even use RGB.

Probably OpenGL is faster on some hardware, and slower on some hardware. Don't assume it's always faster.

Yes, good point.

Quoted 4 lines

Yes it very much depends on the drivers and opengl implementation. It isn't always possible to map the guest framebuffer into video (texture) memory, and also some opengl implementations are not particularly fast at copying textures from CPU memory to texture memory.

This is also true.

Quoted 3 lines

Another way to map guest framebuffer to video memory is the XF86DGA extension, which is great when you can use it. I think VMware uses it in full-screen mode.

I went with OpenGL because I wanted the accelerated window resize feature, and because I think it will give us more opportunities for further improvements (for example using PBOs as soon as they are supported by at least one open source graphic card driver).

This is neat, but, I'm unsure if the right way to support OpenGL is through SDL. For instance, there were Cocoa OpenGL patches posted a bit ago that would be largely similar. It may make more sense to have an OpenGL front-end that has conditional code for SDL/Cocoa/X/etc.

Then again, I've been kicking around the idea of doing a GTK front-end. An obvious thing to do here would be a glext based OpenGL version (as we do in gtk-vnc).

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.

Quoted 6 lines · Anthony Liguori

This is neat, but, I'm unsure if the right way to support OpenGL is

through SDL. For instance, there were Cocoa OpenGL patches posted a bit ago that would be largely similar. It may make more sense to have an OpenGL front-end that has conditional code for SDL/Cocoa/X/etc.

Quoted 3 lines

Then again, I've been kicking around the idea of doing a GTK front-end. An obvious thing to do here would be a glext based OpenGL version (as we do in gtk-vnc).

I don't think GTK support would add much to what qemu already has (aside from more dependencies :).

Quoted 4 lines · Stefano Stabellini

-makes the window resizing possible and hardware accelerated, thus very
efficient and smooth;

This is neat, but, I'm unsure if the right way to support OpenGL is through SDL. For instance, there were Cocoa OpenGL patches posted a bit ago that would be largely similar. It may make more sense to have an OpenGL front-end that has conditional code for SDL/Cocoa/X/etc.

Then again, I've been kicking around the idea of doing a GTK front-end. An obvious thing to do here would be a glext based OpenGL version (as we do in gtk-vnc).

I think we need to have some discussion about what the long term front-end should be for QEMU. Otherwise, we're going to end up with a proliferation of front-ends. Personally, I'd rather move from SDL to GTK so that we can build a proper user interface.