Skip to contentSkip to conversation

Implement a virtio GPU transport

PATCH In review Ian Molton 6 participants 34 messages Oct 6, 2010 – Nov 12, 2010

Subsystems (3)
lore
Share & tools
lore

Ian Molton

author
#0 lore raw

PATCH 1/1 Implement a virtio GPU transport

This patch implements a virtio-based transport for use by a virtualised OpenGL passthrough implementation.

The libGL and qemu-gl code to support this patch are available here:

http://gitorious.org/vm-gl-accel/qemu-gl
http://gitorious.org/vm-gl-accel/qemu-libgl

Comments please!

drivers/gpu/Makefile +1 -1
index 30879df..35699a0 100644
@@ -1 +1 @@
-obj-y			+= drm/ vga/
+obj-y			+= drm/ vga/ misc/
drivers/gpu/misc/Kconfig new file +8 -0
new file mode 100644
index 0000000..50043d3
@@ -0,0 +1,8 @@
+config VIRTIOGL
+        tristate "Virtio userspace memory transport"
+        depends on VIRTIO_PCI
+        default n
+        help
+          A Driver to facilitate transferring data from userspace to a
+          hypervisor (eg. qemu)
+
drivers/gpu/misc/Makefile new file +1 -0
new file mode 100644
index 0000000..d9ab333
@@ -0,0 +1 @@
+obj-$(CONFIG_VIRTIOGL)          += virtio-gl.o
drivers/gpu/misc/virtio-gl.c new file +312 -0

312 lines not on this page · raw message

drivers/video/Kconfig +1 -0
index 3d94a14..9a9a6cc 100644
@@ -16,6 +16,7 @@ source "drivers/char/agp/Kconfig"
 source "drivers/gpu/vga/Kconfig"
 
 source "drivers/gpu/drm/Kconfig"
+source "drivers/gpu/misc/Kconfig"
 
 config VGASTATE
        tristate
include/linux/virtio_ids.h +1 -0
index 06660c0..663b496 100644
@@ -12,6 +12,7 @@
 #define VIRTIO_ID_CONSOLE	3 /* virtio console */
 #define VIRTIO_ID_RNG		4 /* virtio ring */
 #define VIRTIO_ID_BALLOON	5 /* virtio balloon */
+#define VIRTIO_ID_GL		6 /* virtio usermem */
 #define VIRTIO_ID_9P		9 /* 9p virtio console */
 
 #endif /* _LINUX_VIRTIO_IDS_H */
#1 lore raw
Quoted 9 lines · Ian Molton

This patch implements a virtio-based transport for use by a virtualised OpenGL passthrough implementation.

The libGL and qemu-gl code to support this patch are available here:

http://gitorious.org/vm-gl-accel/qemu-gl
http://gitorious.org/vm-gl-accel/qemu-libgl

Comments please!

1. copy qemu-devel an virtualization@, many virtio developers live there.

2. should start with a patch to the virtio-pci spec to document what you're doing

Quoted 10 lines
drivers/gpu/misc/virtio-gl.c new file +10 -0
+static int put_data(struct virtio_gl_data *gldata)
+{
+	struct scatterlist *sg, *sg_list;
+	unsigned int count, ret, o_page, i_page, sg_entries;
+	struct virtio_gl_header *header =
+				(struct virtio_gl_header *)gldata->buffer;
+
+	ret = header->buf_size;
+
+	o_page = (header->buf_size + PAGE_SIZE-1)>>  PAGE_SHIFT;

...

Quoted 8 lines
drivers/gpu/misc/virtio-gl.c new file +8 -0
+	sg_init_table(sg_list, sg_entries);
+
+	sg = vmalloc_to_sg(sg_list, gldata->buffer, o_page);
+	sg = vmalloc_to_sg(sg, gldata->buffer, i_page);
+
+	if (!sg) {
+		ret = -EIO;
+		goto out_free;

drivers/gpu/misc/virtio-gl.c line 150

Quoted 10 lines
drivers/gpu/misc/virtio-gl.c new file +10 -0
+	}
+
+	/* Transfer data */
+	if (virtqueue_add_buf(vq, sg_list, o_page, i_page, (void *)1)>= 0) {
+		virtqueue_kick(vq);
+		/* Chill out until it's done with the buffer. */
+		while (!virtqueue_get_buf(vq,&count))
+			cpu_relax();
+	}
+

This is pretty gross, and will burn lots of cpu if the hypervisor processes the queue asynchronously.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#2 lore raw
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Ian Molton

This patch implements a virtio-based transport for use by a virtualised OpenGL passthrough implementation.

The libGL and qemu-gl code to support this patch are available here:

Quoted 7 lines
Quoted 5 lines

http://gitorious.org/vm-gl-accel/qemu-gl
http://gitorious.org/vm-gl-accel/qemu-libgl

Comments please!

1. copy qemu-devel

Ok, will do.

Quoted 1 line

an virtualization@, many virtio developers live there.

you mean [email protected] ?

Quoted 2 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Where can I find that spec?

Quoted 11 lines
Quoted 8 lines

+ /* Transfer data */
+ if (virtqueue_add_buf(vq, sg_list, o_page, i_page, (void *)1)>= 0) {
+ virtqueue_kick(vq);
+ /* Chill out until it's done with the buffer. */
+ while (!virtqueue_get_buf(vq,&count))
+ cpu_relax();
+ }
+

This is pretty gross, and will burn lots of cpu if the hypervisor processes the queue asynchronously.

It doesnt, at present... It could be changed fairly easily ithout breaking anything if that happens though.

-Ian

#3 lore raw
Quoted 4 lines · Ian Molton
Quoted 2 lines

an virtualization@, many virtio developers live there.

you mean [email protected] ?

Yes.

Quoted 5 lines
Quoted 3 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Where can I find that spec?

http://ozlabs.org/~rusty/virtio-spec/

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

+ /* Transfer data */
+ if (virtqueue_add_buf(vq, sg_list, o_page, i_page, (void *)1)>= 0) {
+ virtqueue_kick(vq);
+ /* Chill out until it's done with the buffer. */
+ while (!virtqueue_get_buf(vq,&count))
+ cpu_relax();
+ }
+

Quoted 4 lines
Quoted 2 lines

This is pretty gross, and will burn lots of cpu if the hypervisor processes the queue asynchronously.

It doesnt, at present... It could be changed fairly easily ithout breaking anything if that happens though.

The hypervisor and the guest can be changed independently. The driver should be coded so that it doesn't depend on hypervisor implementation details.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#4 lore raw
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Ian Molton
Quoted 2 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Where can I find that spec?

http://ozlabs.org/~rusty/virtio-spec/

Ok, but I'm not patching that until theres been some review.

There are links to the associated qemu and guest OS changes in my original email.

Quoted 6 lines
Quoted 2 lines

It doesnt, at present... It could be changed fairly easily ithout breaking anything if that happens though.

The hypervisor and the guest can be changed independently. The driver should be coded so that it doesn't depend on hypervisor implementation details.

Fixed - updated patch tested and attached.

-Ian

drivers/gpu/Makefile +1 -1
index 30879df..35699a0 100644
@@ -1 +1 @@
-obj-y			+= drm/ vga/
+obj-y			+= drm/ vga/ misc/
drivers/gpu/misc/Kconfig new file +8 -0
new file mode 100644
index 0000000..50043d3
@@ -0,0 +1,8 @@
+config VIRTIOGL
+        tristate "Virtio userspace memory transport"
+        depends on VIRTIO_PCI
+        default n
+        help
+          A Driver to facilitate transferring data from userspace to a
+          hypervisor (eg. qemu)
+
drivers/gpu/misc/Makefile new file +1 -0
new file mode 100644
index 0000000..d9ab333
@@ -0,0 +1 @@
+obj-$(CONFIG_VIRTIOGL)          += virtio-gl.o
drivers/gpu/misc/virtio-gl.c new file +325 -0

325 lines not on this page · raw message

drivers/video/Kconfig +1 -0
index 3d94a14..9a9a6cc 100644
@@ -16,6 +16,7 @@ source "drivers/char/agp/Kconfig"
 source "drivers/gpu/vga/Kconfig"
 
 source "drivers/gpu/drm/Kconfig"
+source "drivers/gpu/misc/Kconfig"
 
 config VGASTATE
        tristate
include/linux/virtio_ids.h +1 -0
index 06660c0..663b496 100644
@@ -12,6 +12,7 @@
 #define VIRTIO_ID_CONSOLE	3 /* virtio console */
 #define VIRTIO_ID_RNG		4 /* virtio ring */
 #define VIRTIO_ID_BALLOON	5 /* virtio balloon */
+#define VIRTIO_ID_GL		6 /* virtio usermem */
 #define VIRTIO_ID_9P		9 /* 9p virtio console */
 
 #endif /* _LINUX_VIRTIO_IDS_H */
#5 lore raw
Quoted 11 lines · Ian Molton
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Ian Molton
Quoted 2 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Where can I find that spec?

http://ozlabs.org/~rusty/virtio-spec/

Ok, but I'm not patching that until theres been some review.

Well, I like to review an implementation against a spec.

Quoted 11 lines

There are links to the associated qemu and guest OS changes in my original email.

Quoted 6 lines
Quoted 2 lines

It doesnt, at present... It could be changed fairly easily ithout breaking anything if that happens though.

The hypervisor and the guest can be changed independently. The driver should be coded so that it doesn't depend on hypervisor implementation details.

Fixed - updated patch tested and attached.

drivers/gpu/misc/virtio-gl.c line 162

Quoted 8 lines
drivers/gpu/misc/virtio-gl.c new file +8 -0
+
+	/* Transfer data */
+	if (virtqueue_add_buf(vq, sg_list, o_page, i_page, gldata)>= 0) {
+		virtqueue_kick(vq);
+		/* Chill out until it's done with the buffer. */
+		wait_for_completion(&gldata->done);
+	}
+

Better, but still unsatisfying. If the server is busy, the caller would block. I guess it's expected since it's called from ->fsync(). I'm not sure whether that's the best interface, perhaps aio_writev is better.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#6 lore raw
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Ian Molton
Quoted 5 lines · Avi Kivity
Quoted 3 lines · Ian Molton
Quoted 2 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Where can I find that spec?

http://ozlabs.org/~rusty/virtio-spec/

Ok, but I'm not patching that until theres been some review.

Well, I like to review an implementation against a spec.

True, but then all that would prove is that I can write a spec to match the code.

The code is proof of concept. the kernel bit is pretty simple, but I'd like to get some idea of whether the rest of the code will be accepted given that theres not much point in having any one (or two) of these components exist without the other.

Quoted 3 lines

Better, but still unsatisfying. If the server is busy, the caller would block. I guess it's expected since it's called from ->fsync(). I'm not sure whether that's the best interface, perhaps aio_writev is better.

The caller is intended to block as the host must perform GL rendering before allowing the guests process to continue.

The only real bottleneck is that processes will block trying to submit data if another process is performing rendering, but that will only be solved when the renderer is made multithreaded. The same would happen on a real GPU if it had only one queue too.

If you look at the host code, you can see that the data is already buffered per-process, in a pretty sensible way. if the renderer itself were made a seperate thread, then this problem magically disappears (the queuing code on the host is pretty fast).

In testing, the overhead of this was pretty small anyway. Running a few dozen glxgears and a copy of ioquake3 simultaneously on an intel video card managed the same framerate with the same CPU utilisation, both with the old code and the version I just posted. Contention during rendering just isn't much of an issue.

-Ian

#7 lore raw
Quoted 4 lines · Ian Molton
Quoted 1 line

Well, I like to review an implementation against a spec.

True, but then all that would prove is that I can write a spec to match the code.

It would also allow us to check that the spec matches the requirements. Those two steps are easier than checking that the code matches the requirements.

Quoted 1 line

The code is proof of concept. the kernel bit is pretty simple, but I'd like to get some idea of whether the rest of the code will be accepted given that theres not much point in having any one (or two) of these components exist without the other.

I guess some graphics people need to be involved.

Quoted 6 lines
Quoted 4 lines

Better, but still unsatisfying. If the server is busy, the caller would block. I guess it's expected since it's called from ->fsync(). I'm not sure whether that's the best interface, perhaps aio_writev is better.

The caller is intended to block as the host must perform GL rendering before allowing the guests process to continue.

Why is that? Can't we pipeline the process?

Quoted 4 lines

The only real bottleneck is that processes will block trying to submit data if another process is performing rendering, but that will only be solved when the renderer is made multithreaded. The same would happen on a real GPU if it had only one queue too.

If you look at the host code, you can see that the data is already buffered per-process, in a pretty sensible way. if the renderer itself were made a seperate thread, then this problem magically disappears (the queuing code on the host is pretty fast).

Well, this is out of my area of expertise. I don't like it, but if it's acceptable to the gpu people, okay.

Quoted 2 lines

In testing, the overhead of this was pretty small anyway. Running a few dozen glxgears and a copy of ioquake3 simultaneously on an intel video card managed the same framerate with the same CPU utilisation, both with the old code and the version I just posted. Contention during rendering just isn't much of an issue.

Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw
Quoted 7 lines · Avi Kivity
Quoted 4 lines · Ian Molton
Quoted 1 line

Well, I like to review an implementation against a spec.

True, but then all that would prove is that I can write a spec to match the code.

It would also allow us to check that the spec matches the requirements. Those two steps are easier than checking that the code matches the requirements.

I'm extremely sceptical of any GL passthrough proposal. There have literally been half a dozen over the years and they never seem to leave proof-of-concept phase. My (limited) understanding is that it's a fundamentally hard problem that no one has adequately solved yet.

A specifically matters an awful lot less than an explanation of how the problem is being solved in a robust fashion such that it can be reviewed by people with a deeper understanding of the problem space.

Regards,

Anthony Liguori

Quoted 23 lines
Quoted 1 line

The code is proof of concept. the kernel bit is pretty simple, but I'd like to get some idea of whether the rest of the code will be accepted given that theres not much point in having any one (or two) of these components exist without the other.

I guess some graphics people need to be involved.

Quoted 6 lines
Quoted 4 lines

Better, but still unsatisfying. If the server is busy, the caller would block. I guess it's expected since it's called from ->fsync(). I'm not sure whether that's the best interface, perhaps aio_writev is better.

The caller is intended to block as the host must perform GL rendering before allowing the guests process to continue.

Why is that? Can't we pipeline the process?

Quoted 4 lines

The only real bottleneck is that processes will block trying to submit data if another process is performing rendering, but that will only be solved when the renderer is made multithreaded. The same would happen on a real GPU if it had only one queue too.

If you look at the host code, you can see that the data is already buffered per-process, in a pretty sensible way. if the renderer itself were made a seperate thread, then this problem magically disappears (the queuing code on the host is pretty fast).

Well, this is out of my area of expertise. I don't like it, but if it's acceptable to the gpu people, okay.

Quoted 2 lines

In testing, the overhead of this was pretty small anyway. Running a few dozen glxgears and a copy of ioquake3 simultaneously on an intel video card managed the same framerate with the same CPU utilisation, both with the old code and the version I just posted. Contention during rendering just isn't much of an issue.

author
#9 lore raw
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Avi Kivity
Quoted 2 lines · Ian Molton

True, but then all that would prove is that I can write a spec to match the code.

It would also allow us to check that the spec matches the requirements. Those two steps are easier than checking that the code matches the requirements.

There was no formal spec for this. The code was written to replace nasty undefined-instruction based data transport hacks in the (already existing) GL passthrough code.

Quoted 4 lines

I'm extremely sceptical of any GL passthrough proposal. There have literally been half a dozen over the years and they never seem to leave proof-of-concept phase. My (limited) understanding is that it's a fundamentally hard problem that no one has adequately solved yet.

The code in this case has been presented as a patch to qemu nearly 3 years ago. I've taken the patches and refactored them to use virtio rather than an undefined instruction (which fails under KVM, unlike my approach).

Its in use testing meego images and seems to be fairly reliable. it can handle compositing window managers, games, video, etc. We're currently supporting OpenGL 1.4 including shaders.

Quoted 3 lines

A specifically matters an awful lot less than an explanation of how the problem is being solved in a robust fashion such that it can be reviewed by people with a deeper understanding of the problem space.

I'm not sure there is a way to prevent nefarious tasks upsetting the hosts OpenGL with carefully crafted strings of commands, short of inspecting every single command, which is insane.

Really this needs to be done at a lower level by presenting a virtual GPU to the guest OS but I am not in a position to code that right now.

The code as it is is useful, and can always be superceeded by a virtual GPU implementation in future.

At least this breaks the chicken / egg cycle of people wanting GL support on virtual machines, but not writing stuff to take advantage of it because the support isn't there. its also a neatly encapsulated solution - if you dont want people to have access to the passthrough, simply tell qemu not to present the virtio-gl device to the guest, via qemus existing commandline options.

If this code was invasive to qemus core, I'd say 'no way' but its just not. and as the GL device is versioned, we can keep using it even if the passthrough is replaced by a virtual GPU.

author
#12 lore raw
In reply to Avi Kivity
Quoted 4 lines · Avi Kivity
Quoted 2 lines

The caller is intended to block as the host must perform GL rendering before allowing the guests process to continue.

Why is that? Can't we pipeline the process?

No, not really. the guest may call for the scene to be rendered at any time and we have to wait for that to happen before we can return the data to it.

-Ian

#10 lore raw
In reply to Ian Molton
Quoted 50 lines · Ian Molton
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Avi Kivity
Quoted 2 lines · Ian Molton

True, but then all that would prove is that I can write a spec to match the code.

It would also allow us to check that the spec matches the requirements. Those two steps are easier than checking that the code matches the requirements.

There was no formal spec for this. The code was written to replace nasty undefined-instruction based data transport hacks in the (already existing) GL passthrough code.

Quoted 4 lines

I'm extremely sceptical of any GL passthrough proposal. There have literally been half a dozen over the years and they never seem to leave proof-of-concept phase. My (limited) understanding is that it's a fundamentally hard problem that no one has adequately solved yet.

The code in this case has been presented as a patch to qemu nearly 3 years ago. I've taken the patches and refactored them to use virtio rather than an undefined instruction (which fails under KVM, unlike my approach).

Its in use testing meego images and seems to be fairly reliable. it can handle compositing window managers, games, video, etc. We're currently supporting OpenGL 1.4 including shaders.

Quoted 3 lines

A specifically matters an awful lot less than an explanation of how the problem is being solved in a robust fashion such that it can be reviewed by people with a deeper understanding of the problem space.

I'm not sure there is a way to prevent nefarious tasks upsetting the hosts OpenGL with carefully crafted strings of commands, short of inspecting every single command, which is insane.

Really this needs to be done at a lower level by presenting a virtual GPU to the guest OS but I am not in a position to code that right now.

The code as it is is useful, and can always be superceeded by a virtual GPU implementation in future.

At least this breaks the chicken / egg cycle of people wanting GL support on virtual machines, but not writing stuff to take advantage of it because the support isn't there. its also a neatly encapsulated solution - if you dont want people to have access to the passthrough, simply tell qemu not to present the virtio-gl device to the guest, via qemus existing commandline options.

Quoted 3 lines

If this code was invasive to qemus core, I'd say 'no way' but its just not. and as the GL device is versioned, we can keep using it even if the passthrough is replaced by a virtual GPU.

The virtio-gl implementation is basically duplicating virtio-serial. It looks like ti creates a totally separate window for the GL session. In the current form, is there really any advantage to having the code in QEMU? It could just as easily live outside of QEMU.

Regards,

Anthony Liguori

author
#11 lore raw
Quoted 8 lines · Anthony Liguori
Quoted 3 lines

If this code was invasive to qemus core, I'd say 'no way' but its just not. and as the GL device is versioned, we can keep using it even if the passthrough is replaced by a virtual GPU.

The virtio-gl implementation is basically duplicating virtio-serial. It looks like ti creates a totally separate window for the GL session. In the current form, is there really any advantage to having the code in QEMU? It could just as easily live outside of QEMU.

you could say much the same about any driver in qemu... you could serialise up the registers and ship the data off to be processed on another PC if you wanted to...

The code does not, however, create a seperate window for the GL session. the GL scene is rendered offscreen, and then piped back to the guest for display, so that it is fully composited into the guests graphical environment. From a user perspective, its as if the guest had hardware 3D. Performance is very reasonable, around 40fps in ioquake3 on modest (host) hardware.

In theory, the code *could* use a serial transport and render in a seperate process, but then that would make it much harder to evolve it into a GPU-like system in future.

-Ian

#19 lore raw
In reply to Ian Molton
Quoted 11 lines · Ian Molton
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Ian Molton
Quoted 2 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Where can I find that spec?

http://ozlabs.org/~rusty/virtio-spec/

Ok, but I'm not patching that until theres been some review.

Fair enough; it's a bit of a PITA to patch, so it makes sense to get the details nailed down first.

Quoted 11 lines

There are links to the associated qemu and guest OS changes in my original email.

Quoted 6 lines
Quoted 2 lines

It doesnt, at present... It could be changed fairly easily ithout breaking anything if that happens though.

The hypervisor and the guest can be changed independently. The driver should be coded so that it doesn't depend on hypervisor implementation details.

Fixed - updated patch tested and attached.

OK. FWIW, I think this is an awesome idea. I understand others are skeptical, but this seems simple and if it works and you're happy to maintain it I'm happy to let you do it :)

Cheers,
Rusty.

#20 lore raw
Quoted 26 lines · On Friday 29 October 2010 07:18:00 Rusty Russell wrote:
Quoted 11 lines · Ian Molton
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Ian Molton
Quoted 2 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Where can I find that spec?

http://ozlabs.org/~rusty/virtio-spec/

Ok, but I'm not patching that until theres been some review.

Fair enough; it's a bit of a PITA to patch, so it makes sense to get the details nailed down first.

Quoted 10 lines

There are links to the associated qemu and guest OS changes in my original email.

Quoted 6 lines
Quoted 2 lines

It doesnt, at present... It could be changed fairly easily ithout breaking anything if that happens though.

The hypervisor and the guest can be changed independently. The driver should be coded so that it doesn't depend on hypervisor implementation details.

Quoted 5 lines
Quoted 1 line

Fixed - updated patch tested and attached.

OK. FWIW, I think this is an awesome idea. I understand others are skeptical, but this seems simple and if it works and you're happy to maintain it I'm happy to let you do it :)

From a kvm user's perspective this is a GREAT idea. Looks like Ian and friends have though about better solutions down the line and have made their code easy to superceed. If this is really the case I vote to get this in qemu/kvm asap - software opengl can be a real pain.

Thanks
Ed Tomlinson

#21 lore raw
In reply to Rusty Russell
Quoted 3 lines · Rusty Russell
Quoted 2 lines

Fixed - updated patch tested and attached.

OK. FWIW, I think this is an awesome idea.

Paravirtual OpenGL or the actual proposed implementation? Have you looked at the actual code?

If I understand correctly, the implementation is an RPC of opengl commands across virtio that are then rendered on the host to an offscreen buffer. The buffer is then sent back to the guest in rendered form.

But it's possible to mess around with other guests because the opengl commands are not scrubbed. There have been other proposals in the past that include a mini JIT that would translate opengl commands into a safe form.

Exposing just an opengl RPC transport doesn't seem that useful either. It ought to be part of a VGA card in order for it to be integrated with the rest of the graphics system without a round trip.

The alternative proposal is Spice which so far noone has mentioned. Right now, Spice seems to be taking the right approach to guest 3d support.

Regards,

Anthony Liguori

Quoted 8 lines

I understand others are skeptical,
but this seems simple and if it works and you're happy to maintain it I'm happy to let you do it :)

Cheers,
Rusty.

#13 lore raw
In reply to Ian Molton
Quoted 7 lines · Ian Molton
Quoted 4 lines · Avi Kivity
Quoted 2 lines

The caller is intended to block as the host must perform GL rendering before allowing the guests process to continue.

Why is that? Can't we pipeline the process?

No, not really. the guest may call for the scene to be rendered at any time and we have to wait for that to happen before we can return the data to it.

Waiting for a response is fine, but can't the guest issue a second batch while waiting for the first?

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#22 lore raw
In reply to Anthony Liguori
Quoted 24 lines · ----- "Anthony Liguori
Quoted 3 lines · Rusty Russell
Quoted 2 lines

Fixed - updated patch tested and attached.

OK. FWIW, I think this is an awesome idea.

Paravirtual OpenGL or the actual proposed implementation? Have you looked at the actual code?

If I understand correctly, the implementation is an RPC of opengl commands across virtio that are then rendered on the host to an offscreen buffer. The buffer is then sent back to the guest in rendered
form.

But it's possible to mess around with other guests because the opengl

commands are not scrubbed. There have been other proposals in the past

that include a mini JIT that would translate opengl commands into a safe
form.

Exposing just an opengl RPC transport doesn't seem that useful either.

Quoted 8 lines

It ought to be part of a VGA card in order for it to be integrated with
the rest of the graphics system without a round trip.

The alternative proposal is Spice which so far noone has mentioned. Right now, Spice seems to be taking the right approach to guest 3d support.

While we (speaking as part of the SPICE developers) want to have the same support in our virtual GPU for 3d as we have for 2d, we just don't at this point of time.

Quoted 13 lines

Regards,

Anthony Liguori

Quoted 2 lines

I understand others are skeptical,
but this seems simple and if it works and you're happy to maintain

it I'm

Quoted 6 lines

happy to let you do it :)

Cheers,
Rusty.

#14 lore raw
In reply to Avi Kivity
Quoted 10 lines · Avi Kivity
Quoted 7 lines · Ian Molton
Quoted 4 lines · Avi Kivity
Quoted 2 lines

The caller is intended to block as the host must perform GL rendering before allowing the guests process to continue.

Why is that? Can't we pipeline the process?

No, not really. the guest may call for the scene to be rendered at any time and we have to wait for that to happen before we can return the data to it.

Waiting for a response is fine, but can't the guest issue a second batch while waiting for the first?

In a threaded application I think you mean but all RPCs are dispatched holding a global lock so even within a threaded application, only a single GL call will be executed at a time.

The other scenario would be multiple applications trying to use GL but AFAICT, this is not supported in the current model.

Regards,

Anthony Liguori

#23 lore raw
In reply to Alon Levy
Quoted 9 lines · Alon Levy
Quoted 5 lines

The alternative proposal is Spice which so far noone has mentioned. Right now, Spice seems to be taking the right approach to guest 3d support.

While we (speaking as part of the SPICE developers) want to have the same support in our virtual GPU for 3d as we have for 2d, we just don't at this point of time.

Yes, but I think the point is that are two general approaches to supporting 3d that are being proposed. One approach is to an RPC layer at the OpenGL level which essentially passes through the host OpenGL stack. That's what virtio-gl is. This has existed for quite a while and there are multiple transports for it. It supports serial ports, TCP sockets, a custom ISA extension for x86, and now a custom virtio transport.

Another approach would be to have a virtual GPU and to implement GPU-level commands for 3d. I have been repeated told that much of the complexity of Spice is absolutely needed for 3d and that that's a major part of the design.

GPU-level support for 3d operations has a number of advantages mainly that it's more reasonably portable to things like Windows since the 3d commands can be a superset of both OpenGL and Direct3d.

Also, Spice has an abstraction layer that doesn't simply passthrough graphics commands, but translates/sanitizes them first. That's another advantage over OpenGL passthrough. Without a layer to sanitize commands, a guest can do funky things with the host or other guests.

I think a Spice-like approach is the best thing long term. In the short term, I think doing the GL marshalling over virtio-serial makes a ton of sense since the kernel driver is already present upstream. It exists exactly for things like this.

In the very, very short term, I think an external backend to QEMU also makes a lot of sense because that's something that Just Works today. I think we can consider integrating it into QEMU (or at least simplifying the execution of the backend) but integrating into QEMU is going to require an awful lot of the existing code to be rewritten. Keeping it separate has the advantage of allowing something to Just Work as an interim solution as we wait for proper support in Spice.

Regards,

Anthony Liguori

Quoted 18 lines
Quoted 18 lines

Regards,

Anthony Liguori

Quoted 3 lines

I understand others are skeptical,
but this seems simple and if it works and you're happy to maintain

it I'm

Quoted 7 lines

happy to let you do it :)

Cheers,
Rusty.

author
#15 lore raw
In reply to Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
Quoted 1 line · Ian Molton

On 28/10/10 15:24, Avi Kivity wrote:

Waiting for a response is fine, but can't the guest issue a second batch while waiting for the first?

The other scenario would be multiple applications trying to use GL but AFAICT, this is not supported in the current model.

It very much is. It supports fully visually integrated rendering (no overlay windows) and even compositing GL window managers work fine, even if running 3D apps under them.

-Ian

author
#18 lore raw
In reply to Avi Kivity
Quoted 6 lines · Avi Kivity
Quoted 3 lines

No, not really. the guest may call for the scene to be rendered at any time and we have to wait for that to happen before we can return the data to it.

Waiting for a response is fine, but can't the guest issue a second batch while waiting for the first?

No. Userspace doesnt do or expect that.

Multiple applications work fine, however.

-Ian

#16 lore raw
In reply to Ian Molton
Quoted 5 lines · Ian Molton
Quoted 3 lines · Anthony Liguori
Quoted 2 lines · Avi Kivity
Quoted 1 line · Ian Molton

On 28/10/10 15:24, Avi Kivity wrote:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Waiting for a response is fine, but can't the guest issue a second batch while waiting for the first?

The other scenario would be multiple applications trying to use GL but AFAICT, this is not supported in the current model.

It very much is. It supports fully visually integrated rendering (no overlay windows) and even compositing GL window managers work fine, even if running 3D apps under them.

Does the kernel track userspace pid and pass that information to qemu?

Regards,

Anthony Liguori

Quoted 1 line

-Ian

author
#17 lore raw
Quoted 5 lines · Anthony Liguori
Quoted 3 lines

It very much is. It supports fully visually integrated rendering (no overlay windows) and even compositing GL window managers work fine, even if running 3D apps under them.

Does the kernel track userspace pid and pass that information to qemu?

Yes. And the qemu code tracks the PIDs and keeps multiple queues (one per pid).

-Ian

author
#33 lore raw
In reply to Rusty Russell
Quoted 7 lines · Rusty Russell
Quoted 6 lines · Ian Molton
Quoted 5 lines · Avi Kivity
Quoted 3 lines · Ian Molton
Quoted 2 lines

2. should start with a patch to the virtio-pci spec to document what you're doing

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Where can I find that spec?

http://ozlabs.org/~rusty/virtio-spec/

Ok, but I'm not patching that until theres been some review.

Fair enough; it's a bit of a PITA to patch, so it makes sense to get the details nailed down first.

I thought so :)

Quoted 5 lines
Quoted 1 line

Fixed - updated patch tested and attached.

OK. FWIW, I think this is an awesome idea. I understand others are skeptical, but this seems simple and if it works and you're happy to maintain it I'm happy to let you do it :)

Awesoe, thanks Rusty :)

author
#24 lore raw
In reply to Anthony Liguori
Quoted 5 lines · Anthony Liguori
Quoted 3 lines · Alon Levy

While we (speaking as part of the SPICE developers) want to have the same support in our virtual GPU for 3d as we have for 2d, we just don't at this point of time.

Would it be helpful to you to have /something/ that works in the interim? I'm happy to work with you guys so that we dont need to reinvent the wheel ;-)

Quoted 5 lines

Yes, but I think the point is that are two general approaches to supporting 3d that are being proposed. One approach is to an RPC layer at the OpenGL level which essentially passes through the host OpenGL stack. That's what virtio-gl is. This has existed for quite a while and there are multiple transports for it.

Well, sort of. this version is heavily modified and only the virtio transport is supported in it. Its quite a large code cleanup.

Quoted 1 line

It supports serial ports, TCP sockets, a custom ISA extension for x86,

The custom ISA idea is cut too since it fails to play nice with KVM. Virtio-gl offers better security too.

Quoted 4 lines

Another approach would be to have a virtual GPU and to implement GPU-level commands for 3d. I have been repeated told that much of the complexity of Spice is absolutely needed for 3d and that that's a major part of the design.

I've not seen any implementations of this type that are even close to useable.

Quoted 3 lines

GPU-level support for 3d operations has a number of advantages mainly that it's more reasonably portable to things like Windows since the 3d commands can be a superset of both OpenGL and Direct3d.

Agreed.

Quoted 3 lines

Also, Spice has an abstraction layer that doesn't simply passthrough graphics commands, but translates/sanitizes them first. That's another advantage over OpenGL passthrough.

I'm not sure that thats actually needed if you are careful about what commands you implement on the virtual GPU (and how)

Quoted 2 lines

Without a layer to sanitize commands,
a guest can do funky things with the host or other guests.

It shouldnt be possible for the guest to hurt other guests actual GL rendering (or the hosts) as each PID is handed a context (or contexts) of its own. It is possible for the guest to cause the host problems though. this is a design flaw in the whole idea of GL RPC. It *isnt* reasonable to make it secure, but it certainly is useful right now since theres no alternative available.

Quoted 4 lines

I think a Spice-like approach is the best thing long term. In the short term, I think doing the GL marshalling over virtio-serial makes a ton of sense since the kernel driver is already present upstream. It exists exactly for things like this.

The virtio driver enfoces the PID field and understands the packet format used. Its better than using serial. Its also just one driver - which doesnt have any special interdependencies and can be extended or got rid of in future if and when better things come along.

Quoted 2 lines

In the very, very short term, I think an external backend to QEMU also makes a lot of sense because that's something that Just Works today.

Whos written that? The 2007 patch I've been working on and updating simply fails to work altogether without huge alterations on current qemu.

My current patch touches a tiny part of the qemu sources. It works today.

Quoted 6 lines

I
think we can consider integrating it into QEMU (or at least simplifying the execution of the backend) but integrating into QEMU is going to require an awful lot of the existing code to be rewritten. Keeping it separate has the advantage of allowing something to Just Work as an interim solution as we wait for proper support in Spice.

I dont know why you think integrating it into qemu is hard? I've already done it. I added one virtio driver and a seperate offscreen renderer. it touches the qemu code in *one* place. There should be no need to rewrite anything.

#25 lore raw
Quoted 2 lines · Ian Molton

The virtio driver enfoces the PID field and understands the packet format used. Its better than using serial. Its also just one driver - which doesnt have any special interdependencies and can be extended or got rid of in future if and when better things come along.

Why is it better than using virtio-serial?

Quoted 7 lines
Quoted 3 lines

In the very, very short term, I think an external backend to QEMU also makes a lot of sense because that's something that Just Works today.

Whos written that? The 2007 patch I've been working on and updating simply fails to work altogether without huge alterations on current qemu.

My current patch touches a tiny part of the qemu sources. It works today.

But it's not at all mergable in the current form. If you want to do the work of getting it into a mergable state (cleaning up the coding style, moving it to hw/, etc.) than I'm willing to consider it. But I don't think a custom virtio transport is the right thing to do here.

However, if you want something that Just Works with the least amount of code possible, just split it into a separate process and we can stick it in a contrib/ directory or something.

Quoted 9 lines
Quoted 7 lines

I
think we can consider integrating it into QEMU (or at least simplifying the execution of the backend) but integrating into QEMU is going to require an awful lot of the existing code to be rewritten. Keeping it separate has the advantage of allowing something to Just Work as an interim solution as we wait for proper support in Spice.

I dont know why you think integrating it into qemu is hard? I've already done it.

Adding a file that happens to compile as part of qemu even though it doesn't actually integrate with qemu in any meaningful way is not integrating. That's just build system manipulation.

Regards,

Anthony Liguori

Quoted 1 line

I added one virtio driver and a seperate offscreen renderer. it touches the qemu code in *one* place. There should be no need to rewrite anything.

#31 lore raw
In reply to Ian Molton
Quoted 11 lines · Ian Molton
Quoted 5 lines · Anthony Liguori
Quoted 3 lines · Alon Levy

While we (speaking as part of the SPICE developers) want to have the same support in our virtual GPU for 3d as we have for 2d, we just don't at this point of time.

Would it be helpful to you to have /something/ that works in the interim? I'm happy to work with you guys so that we dont need to reinvent the wheel ;-)

In case it wasn't clear, I think putting virtio-gl is a good idea, exactly because it works right now.

[Snip]

author
#32 lore raw
Quoted 8 lines · Alon Levy
Quoted 7 lines · Ian Molton
Quoted 5 lines · Anthony Liguori
Quoted 3 lines · Alon Levy

While we (speaking as part of the SPICE developers) want to have the same support in our virtual GPU for 3d as we have for 2d, we just don't at this point of time.

Quoted 5 lines
Quoted 3 lines

Would it be helpful to you to have /something/ that works in the interim? I'm happy to work with you guys so that we dont need to reinvent the wheel ;-)

In case it wasn't clear, I think putting virtio-gl is a good idea, exactly because it works right now.

Thanks :)

-Ian

author
#26 lore raw
In reply to Anthony Liguori
Quoted 3 lines · Anthony Liguori

On 11/03/2010 01:03 PM, Ian Molton wrote:

Why is it better than using virtio-serial?

For one thing, it enforces the PID in kernel so the guests processes cant screw each other over by forging the PID passed to qemu.

Quoted 6 lines
Quoted 1 line

My current patch touches a tiny part of the qemu sources. It works today.

But it's not at all mergable in the current form. If you want to do the work of getting it into a mergable state (cleaning up the coding style, moving it to hw/, etc.) than I'm willing to consider it. But I don't think a custom virtio transport is the right thing to do here.

Hm, I thought I'd indented everything in qemus odd way... Is there a codingstyle document or a checkpatch-like tool for qemu?

I'm happy to make the code meet qemus coding style.

Quoted 3 lines

However, if you want something that Just Works with the least amount of code possible, just split it into a separate process and we can stick it in a contrib/ directory or something.

I dont see what splitting it into a seperate process buys us given we still need the virtio-gl driver in order to enforce the PID. The virtio driver is probably quite a bit more efficient at marshalling the data too, given that it was designed alongside the userspace code.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I
think we can consider integrating it into QEMU (or at least simplifying the execution of the backend) but integrating into QEMU is going to require an awful lot of the existing code to be rewritten.

Why? aside from codingstyle, whats massively wrong with it thats going to demand a total rewrite?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Keeping it
separate has the advantage of allowing something to Just Work as an interim solution as we wait for proper support in Spice.

Why does keeping it seperate make life easier? qemu is in a git repo. when the time comes, if it reall is a total replacement, git-rm will nuke it all.

Quoted 6 lines
Quoted 2 lines

I dont know why you think integrating it into qemu is hard? I've already done it.

Adding a file that happens to compile as part of qemu even though it doesn't actually integrate with qemu in any meaningful way is not integrating. That's just build system manipulation.

Uh? Either it compiles and works as part of qemu (which it does) or it doesnt. How is that not integrated? I've added it to the configure script too.

-Ian

author
#27 lore raw

Ping ?

Quoted 59 lines · Ian Molton
Quoted 3 lines · Anthony Liguori

On 11/03/2010 01:03 PM, Ian Molton wrote:

Why is it better than using virtio-serial?

For one thing, it enforces the PID in kernel so the guests processes cant screw each other over by forging the PID passed to qemu.

Quoted 7 lines
Quoted 2 lines

My current patch touches a tiny part of the qemu sources. It works today.

But it's not at all mergable in the current form. If you want to do the work of getting it into a mergable state (cleaning up the coding style, moving it to hw/, etc.) than I'm willing to consider it. But I don't think a custom virtio transport is the right thing to do here.

Hm, I thought I'd indented everything in qemus odd way... Is there a codingstyle document or a checkpatch-like tool for qemu?

I'm happy to make the code meet qemus coding style.

Quoted 3 lines

However, if you want something that Just Works with the least amount of code possible, just split it into a separate process and we can stick it in a contrib/ directory or something.

I dont see what splitting it into a seperate process buys us given we still need the virtio-gl driver in order to enforce the PID. The virtio driver is probably quite a bit more efficient at marshalling the data too, given that it was designed alongside the userspace code.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I
think we can consider integrating it into QEMU (or at least simplifying the execution of the backend) but integrating into QEMU is going to require an awful lot of the existing code to be rewritten.

Why? aside from codingstyle, whats massively wrong with it thats going to demand a total rewrite?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Keeping it
separate has the advantage of allowing something to Just Work as an interim solution as we wait for proper support in Spice.

Why does keeping it seperate make life easier? qemu is in a git repo. when the time comes, if it reall is a total replacement, git-rm will nuke it all.

Quoted 6 lines
Quoted 2 lines

I dont know why you think integrating it into qemu is hard? I've already done it.

Adding a file that happens to compile as part of qemu even though it doesn't actually integrate with qemu in any meaningful way is not integrating. That's just build system manipulation.

Uh? Either it compiles and works as part of qemu (which it does) or it doesnt. How is that not integrated? I've added it to the configure script too.

-Ian

#28 lore raw
Quoted 1 line · Ian Molton

Ping ?

I think the best way forward is to post patches.

To summarize what I was trying to express in the thread, I think this is not the right long term architecture but am not opposed to it as a short term solution. I think having a new virtio device is a bad design choice but am not totally opposed to it.

My advice is that using virtio-serial + an external tool is probably the least amount of work to get something working and usable with QEMU. If you want to go for the path of integration, you're going to have to fix all of the coding style issues and make the code fit into QEMU. Dropping a bunch of junk into target-i386/ is not making the code fit into QEMU.

If you post just what you have now in patch form, I can try to provide more concrete advice ignoring the coding style problems.

Regards,

Anthony Liguori

author
#29 lore raw
Quoted 4 lines · Anthony Liguori
Quoted 1 line · Ian Molton

Ping ?

I think the best way forward is to post patches.

I posted links to the git trees. I can post patches, but they are *large*. Do you really want me to post them?

Quoted 4 lines

To summarize what I was trying to express in the thread, I think this is not the right long term architecture but am not opposed to it as a short term solution. I think having a new virtio device is a bad design choice but am not totally opposed to it.

Ok! (I agree (that this should be a short term solution) :) )

Quoted 3 lines

you want to go for the path of integration, you're going to have to fix all of the coding style issues and make the code fit into QEMU. Dropping a bunch of junk into target-i386/ is not making the code fit into QEMU.

I agree. how about hw/gl for the renderer and hw/ for the virtio module?

Quoted 2 lines

If you post just what you have now in patch form, I can try to provide more concrete advice ignoring the coding style problems.

I can post patches, although I dont think LKML would appreciate the volume! I can post them to the qemu list if you do.

-Ian

#30 lore raw
Quoted 7 lines · Ian Molton
Quoted 4 lines · Anthony Liguori
Quoted 1 line · Ian Molton

Ping ?

I think the best way forward is to post patches.

I posted links to the git trees. I can post patches, but they are *large*. Do you really want me to post them?

Yes, and they have to be split up into something reviewable.

Quoted 6 lines
Quoted 5 lines

To summarize what I was trying to express in the thread, I think this is not the right long term architecture but am not opposed to it as a short term solution. I think having a new virtio device is a bad design choice but am not totally opposed to it.

Quoted 7 lines

Ok! (I agree (that this should be a short term solution) :) )

Quoted 3 lines

you want to go for the path of integration, you're going to have to fix all of the coding style issues and make the code fit into QEMU. Dropping a bunch of junk into target-i386/ is not making the code fit into QEMU.

I agree. how about hw/gl for the renderer and hw/ for the virtio module?

That would be fine.

Quoted 4 lines
Quoted 2 lines

If you post just what you have now in patch form, I can try to provide more concrete advice ignoring the coding style problems.

I can post patches, although I dont think LKML would appreciate the volume! I can post them to the qemu list if you do.

Yes, qemu is where I was suggesting you post them.

Regards,

Anthony Liguori

Quoted 1 line

-Ian