Skip to contentSkip to conversation

Reading the virtio code...

RFC PATCH In review Rob Landley 3 participants 3 messages Apr 23, 2011 – Apr 27, 2011 virtio core

lore
Share & tools
lore

Rob Landley

author
#0 lore raw

RFC PATCH 1/1 Reading the virtio code...

From: Rob Landley <[email protected]>

Going indirect for only two buffers isn't likely to be a performance win because the kmalloc/kfree overhead for the indirect block can't be cheaper than one extra linked list traversal.

Properly "tuning" the threshold would probably be workload-specific. (One big downside of not going indirect is extra pressure on the table entries, and table size varies.) But I think that in the general case, 2 is a defensible minimum?

Signed-off-by Rob Landley <[email protected]>

---

 drivers/virtio/virtio_ring.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/virtio/virtio_ring.c +1 -1
index b0043fb..2b69441 100644
@@ -173,7 +173,7 @@ int virtqueue_add_buf_gfp(struct virtqueue *_vq,
 
 	/* If the host supports indirect descriptor tables, and we have multiple
 	 * buffers, then go indirect. FIXME: tune this threshold */
-	if (vq->indirect && (out + in) > 1 && vq->num_free) {
+	if (vq->indirect && (out + in) > 2 && vq->num_free) {
 		head = vring_add_indirect(vq, sg, out, in, gfp);
 		if (likely(head >= 0))
 			goto add_head;
#1 lore raw
Quoted 5 lines · Rob Landley

From: Rob Landley <[email protected]>

Going indirect for only two buffers isn't likely to be a performance win because the kmalloc/kfree overhead for the indirect block can't be cheaper than one extra linked list traversal.

Unfortunately it's not completely clear. QEMU sets fairly small rings, and the virtio-net driver uses 2 descriptors minimum. The effect can be a real bottleneck for small packets.

Now, virtio-net could often stuff the virtio_net_hdr in the space before the packet data (saving a descriptor) but I think that will need a feature bit since qemu (incorrectly) used to insist on a separate descriptor for that header.

Quoted 4 lines

Properly "tuning" the threshold would probably be workload-specific. (One big downside of not going indirect is extra pressure on the table entries, and table size varies.) But I think that in the general case, 2 is a defensible minimum?

I'd be tempted to say that once we fill the ring, we should drop the threshold.

Michael?

Thanks,
Rusty.

maintainer
#2 lore raw
Quoted 21 lines · Rusty Russell
Quoted 5 lines · Rob Landley

From: Rob Landley <[email protected]>

Going indirect for only two buffers isn't likely to be a performance win because the kmalloc/kfree overhead for the indirect block can't be cheaper than one extra linked list traversal.

Unfortunately it's not completely clear. QEMU sets fairly small rings, and the virtio-net driver uses 2 descriptors minimum. The effect can be a real bottleneck for small packets.

Now, virtio-net could often stuff the virtio_net_hdr in the space before the packet data (saving a descriptor) but I think that will need a feature bit since qemu (incorrectly) used to insist on a separate descriptor for that header.

Quoted 4 lines

Properly "tuning" the threshold would probably be workload-specific. (One big downside of not going indirect is extra pressure on the table entries, and table size varies.) But I think that in the general case, 2 is a defensible minimum?

Quoted 7 lines

I'd be tempted to say that once we fill the ring, we should drop the threshold.

Michael?

Thanks,
Rusty.

Yes, one idea is to use part of a ring (e.g. 1/4 of a ring) for direct entries, and the rest for indirect. So we end up with a threshold like

	max(1, vq->num_free - in - out - vq->num * N)

(above is pseudo-code, must take care of unsigned vs signed etc) and I think I'd try with N = 3/4 or maybe N = 1/2

Signature · 1 line
MST