Skip to contentSkip to conversation

block io lost in the guest , possible related to qemu?

RFC In review Jack Wang 3 participants 6 messages Oct 25, 2013 – Oct 30, 2013

lore
Share & tools
lore

Jack Wang

author
#0 lore raw

RFC 1/1 block io lost in the guest , possible related to qemu?

Hi Experts,

We've seen guest block io lost in a VM.any response will be helpful

environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk

the exported vdb has very high infight io over 300. Some times later a lot io process in D state, looks a lot requests is lost in below storage stack.

We're use qemu-kvm 1.0, host kernel 3.4.51

In qemu log of virtio-blk.c
I found below commit, I wonder is it possible the workload generate some unknown reqests to qemu that lost in virtio_blk_handle_read?
I do some fio test myself, I cann't generate so call unknown request type.

Any response will be helpful.

Jack

commit 9e72c45033770b81b536ac6091e91807247cc25a

Author: Alexey Zaytsev <[email protected]>
Date:   Thu Dec 13 09:03:43 2012 +0200

virtio-blk: Return UNSUPP for unknown request types

Currently, all unknown requests are treated as VIRTIO_BLK_T_IN

hw/virtio-blk.c +0 -0
index 92c745a..df57b35 100644
@@ -398,10 +398,14 @@ static void

virtio_blk_handle_request(VirtIOBlockReq *req,

         qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
                                  req->elem.out_num - 1);
         virtio_blk_handle_write(req, mrb);
-    } else {
+    } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
+        /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
         qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
                                  req->elem.in_num - 1);
         virtio_blk_handle_read(req);
+    } else {
+        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
+        g_free(req);
     }

}

author
#1 lore raw

Hello Kevin & Stefan

Any comments or wild guess about the bug?

Regards,
Jack

Quoted 56 lines · Jack Wang

Hi Experts,

We've seen guest block io lost in a VM.any response will be helpful

environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk

the exported vdb has very high infight io over 300. Some times later a lot io process in D state, looks a lot requests is lost in below storage stack.

We're use qemu-kvm 1.0, host kernel 3.4.51

In qemu log of virtio-blk.c
I found below commit, I wonder is it possible the workload generate some unknown reqests to qemu that lost in virtio_blk_handle_read?
I do some fio test myself, I cann't generate so call unknown request type.

Any response will be helpful.

Jack

commit 9e72c45033770b81b536ac6091e91807247cc25a

Author: Alexey Zaytsev <[email protected]>
Date:   Thu Dec 13 09:03:43 2012 +0200

virtio-blk: Return UNSUPP for unknown request types

Currently, all unknown requests are treated as VIRTIO_BLK_T_IN

hw/virtio-blk.c +5 -1
index 92c745a..df57b35 100644
@@ -398,10 +398,14 @@ static void
 virtio_blk_handle_request(VirtIOBlockReq *req,
         qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
                                  req->elem.out_num - 1);
         virtio_blk_handle_write(req, mrb);
-    } else {
+    } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
+        /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
         qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
                                  req->elem.in_num - 1);
         virtio_blk_handle_read(req);
+    } else {
+        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
+        g_free(req);
     }
 }
#2 lore raw

Hey.

I very much doubt this commit could be causing the problem, as qemu would never set wrong request type in the first place. You can easily check by either reverting it, or adding a printk() before virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP).

Quoted 65 lines · Jack Wang

Hello Kevin & Stefan

Any comments or wild guess about the bug?

Regards,
Jack

Quoted 56 lines · Jack Wang

Hi Experts,

We've seen guest block io lost in a VM.any response will be helpful

environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk

the exported vdb has very high infight io over 300. Some times later a lot io process in D state, looks a lot requests is lost in below storage stack.

We're use qemu-kvm 1.0, host kernel 3.4.51

In qemu log of virtio-blk.c
I found below commit, I wonder is it possible the workload generate some unknown reqests to qemu that lost in virtio_blk_handle_read?
I do some fio test myself, I cann't generate so call unknown request type.

Any response will be helpful.

Jack

commit 9e72c45033770b81b536ac6091e91807247cc25a

Author: Alexey Zaytsev <[email protected]>
Date:   Thu Dec 13 09:03:43 2012 +0200

virtio-blk: Return UNSUPP for unknown request types

Currently, all unknown requests are treated as VIRTIO_BLK_T_IN

hw/virtio-blk.c +5 -1
index 92c745a..df57b35 100644
@@ -398,10 +398,14 @@ static void
 virtio_blk_handle_request(VirtIOBlockReq *req,
         qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
                                  req->elem.out_num - 1);
         virtio_blk_handle_write(req, mrb);
-    } else {
+    } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
+        /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
         qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
                                  req->elem.in_num - 1);
         virtio_blk_handle_read(req);
+    } else {
+        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
+        g_free(req);
     }
 }
author
#3 lore raw
Quoted 6 lines · Alexey Zaytsev

Hey.

I very much doubt this commit could be causing the problem, as qemu would never set wrong request type in the first place. You can easily check by either reverting it, or adding a printk() before virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP).

Hi Alexey,

Thanks for you input.
According to my test results, yes, as you said, virtio-blk never generate wrong request type. So the commit is only a small cosmetic extra check:(

As there's nothing abnormal in host server and storage, there must be some hidden bug somewhere, damn it.

Regards,
Jack

Quoted 67 lines · Jack Wang
Quoted 65 lines

Hello Kevin & Stefan

Any comments or wild guess about the bug?

Regards,
Jack

Quoted 56 lines · Jack Wang

Hi Experts,

We've seen guest block io lost in a VM.any response will be helpful

environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk

the exported vdb has very high infight io over 300. Some times later a lot io process in D state, looks a lot requests is lost in below storage stack.

We're use qemu-kvm 1.0, host kernel 3.4.51

In qemu log of virtio-blk.c
I found below commit, I wonder is it possible the workload generate some unknown reqests to qemu that lost in virtio_blk_handle_read?
I do some fio test myself, I cann't generate so call unknown request type.

Any response will be helpful.

Jack

commit 9e72c45033770b81b536ac6091e91807247cc25a

Author: Alexey Zaytsev <[email protected]>
Date:   Thu Dec 13 09:03:43 2012 +0200

virtio-blk: Return UNSUPP for unknown request types

Currently, all unknown requests are treated as VIRTIO_BLK_T_IN

hw/virtio-blk.c +5 -1
index 92c745a..df57b35 100644
@@ -398,10 +398,14 @@ static void
 virtio_blk_handle_request(VirtIOBlockReq *req,
         qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1],
                                  req->elem.out_num - 1);
         virtio_blk_handle_write(req, mrb);
-    } else {
+    } else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) {
+        /* VIRTIO_BLK_T_IN is 0, so we can't just & it. */
         qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0],
                                  req->elem.in_num - 1);
         virtio_blk_handle_read(req);
+    } else {
+        virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP);
+        g_free(req);
     }
 }
#4 lore raw
In reply to Jack Wang
Quoted 9 lines · Jack Wang

We've seen guest block io lost in a VM.any response will be helpful

environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk

the exported vdb has very high infight io over 300. Some times later a lot io process in D state, looks a lot requests is lost in below storage stack.

Is the image file on a local file system or are you using a network storage system (e.g. NFS, Gluster, Ceph, Sheepdog)?

If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O activity? If that number is very low or zero then there may be a starvation problem. If that number is reasonable then the workload is simply bottlenecked on disk I/O.

virtio-blk only has 128 descriptors available so it's not possible to have 300 requests pending at the virtio-blk layer.

If you suspect QEMU, try building qemu.git/master from source in case the bug has already been fixed.

If you want to trace I/O requests, you might find this blog post on writing trace analysis scripts useful:
http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html

Stefan

author
#5 lore raw
Quoted 22 lines · Stefan Hajnoczi
Quoted 9 lines · Jack Wang

We've seen guest block io lost in a VM.any response will be helpful

environment is:
guest os: Ubuntu 1304
running busy database workload with xfs on a disk export with virtio-blk

the exported vdb has very high infight io over 300. Some times later a lot io process in D state, looks a lot requests is lost in below storage stack.

Is the image file on a local file system or are you using a network storage system (e.g. NFS, Gluster, Ceph, Sheepdog)?

If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O activity? If that number is very low or zero then there may be a starvation problem. If that number is reasonable then the workload is simply bottlenecked on disk I/O.

virtio-blk only has 128 descriptors available so it's not possible to have 300 requests pending at the virtio-blk layer.

Quoted 9 lines

If you suspect QEMU, try building qemu.git/master from source in case the bug has already been fixed.

If you want to trace I/O requests, you might find this blog post on writing trace analysis scripts useful:
http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html

Stefan

Thanks Stefan for your valuable input.

The image is on device exported with InfiniBand srp/srpt. Will follow your suggestions to do further investigation.

The 300 infight ios I memtioned is from the /proc/diskstats Field 9 --

# of I/Os currently in progress.

Jack