Skip to contentSkip to conversation

xen-blkfront: weird behavior of "iostat" after VM live-migrate which xen-blkfront module has indirect descriptors

DISCUSSION Ouyang Zhaowei (Charles) 2 participants 7 messages Jan 23, 2015 – Jan 30, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi Roger,

We are testing the indirect feature of xen-blkfront module these days.
And we found that, after VM live-migrate a couple of times, the "%util" of iostat keeps being 100%, and there are several requests stock in "avgqu-sz". We have checked some later version of Linux, and it happens on Ubuntu 14.04, Ubuntu 14.10 and RHEL 7.0.

The iostat shows like below:

avg-cpu: %user %nice %system %iowait %steal %idle

           0.00    0.00    0.00    0.00    0.00  100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     4.00    0.00    0.00    0.00   0.00 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Could you tell us that why is this happening, is this a bug?

Thanks!

Ouyang Zhaowei

#1 lore raw

Hello,

El 23/01/15 a les 8.59, Ouyang Zhaowei (Charles) ha escrit:

Quoted 9 lines

Hi Roger,

We are testing the indirect feature of xen-blkfront module these days.
And we found that, after VM live-migrate a couple of times, the "%util" of iostat keeps being 100%, and there are several requests stock in "avgqu-sz". We have checked some later version of Linux, and it happens on Ubuntu 14.04, Ubuntu 14.10 and RHEL 7.0.

The iostat shows like below:

avg-cpu: %user %nice %system %iowait %steal %idle

Quoted 8 lines
           0.00    0.00    0.00    0.00    0.00  100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     4.00    0.00    0.00    0.00   0.00 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Could you tell us that why is this happening, is this a bug?

It is a bug indeed, thanks for reporting it. The problem seems to be that blk_put_request (which is used to discard the old requests before requeuing them) doesn't update the queue statistics. The following patch solves the problem for me, could you try it and report back?

---
commit bb4317c051ca81a2906edb7ccc505cbd6d1d80c7

Author: Roger Pau Monne <[email protected]>
Date:   Fri Jan 23 12:10:51 2015 +0100

xen-blkfront: fix accounting of reqs when migrating

Current migration code uses blk_put_request in order to finish a request before requeuing it. This function doesn't update the statistics of the queue, which completely screws accounting. Use blk_end_request_all instead which properly updates the statistics of the queue.

drivers/block/xen-blkfront.c +2 -2
index 5ac312f..aac41c1 100644
@@ -1493,7 +1493,7 @@ static int blkif_recover(struct blkfront_info *info)
 		merge_bio.tail = copy[i].request->biotail;
 		bio_list_merge(&bio_list, &merge_bio);
 		copy[i].request->bio = NULL;
-		blk_put_request(copy[i].request);
+		blk_end_request_all(copy[i].request, 0);
 	}
 
 	kfree(copy);
@@ -1516,7 +1516,7 @@ static int blkif_recover(struct blkfront_info *info)
 		req->bio = NULL;
 		if (req->cmd_flags & (REQ_FLUSH | REQ_FUA))
 			pr_alert("diskcache flush request found!\n");
-		__blk_put_request(info->rq, req);
+		__blk_end_request_all(req, 0);
 	}
 	spin_unlock_irq(&info->io_lock);
 

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#2 lore raw
In reply to Ouyang Zhaowei (Charles)

Hello,

El 23/01/15 a les 8.59, Ouyang Zhaowei (Charles) ha escrit:

Quoted 9 lines

Hi Roger,

We are testing the indirect feature of xen-blkfront module these days.
And we found that, after VM live-migrate a couple of times, the "%util" of iostat keeps being 100%, and there are several requests stock in "avgqu-sz". We have checked some later version of Linux, and it happens on Ubuntu 14.04, Ubuntu 14.10 and RHEL 7.0.

The iostat shows like below:

avg-cpu: %user %nice %system %iowait %steal %idle

Quoted 8 lines
           0.00    0.00    0.00    0.00    0.00  100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     4.00    0.00    0.00    0.00   0.00 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Could you tell us that why is this happening, is this a bug?

It is a bug indeed, thanks for reporting it. The problem seems to be that blk_put_request (which is used to discard the old requests before requeuing them) doesn't update the queue statistics. The following patch solves the problem for me, could you try it and report back?

---
commit bb4317c051ca81a2906edb7ccc505cbd6d1d80c7

Author: Roger Pau Monne <[email protected]>
Date:   Fri Jan 23 12:10:51 2015 +0100

xen-blkfront: fix accounting of reqs when migrating

Current migration code uses blk_put_request in order to finish a request before requeuing it. This function doesn't update the statistics of the queue, which completely screws accounting. Use blk_end_request_all instead which properly updates the statistics of the queue.

drivers/block/xen-blkfront.c +2 -2
index 5ac312f..aac41c1 100644
@@ -1493,7 +1493,7 @@ static int blkif_recover(struct blkfront_info *info)
 		merge_bio.tail = copy[i].request->biotail;
 		bio_list_merge(&bio_list, &merge_bio);
 		copy[i].request->bio = NULL;
-		blk_put_request(copy[i].request);
+		blk_end_request_all(copy[i].request, 0);
 	}
 
 	kfree(copy);
@@ -1516,7 +1516,7 @@ static int blkif_recover(struct blkfront_info *info)
 		req->bio = NULL;
 		if (req->cmd_flags & (REQ_FLUSH | REQ_FUA))
 			pr_alert("diskcache flush request found!\n");
-		__blk_put_request(info->rq, req);
+		__blk_end_request_all(req, 0);
 	}
 	spin_unlock_irq(&info->io_lock);
 
#3 lore raw
Quoted 55 lines · Roger Pau Monné

Hello,

El 23/01/15 a les 8.59, Ouyang Zhaowei (Charles) ha escrit:

Quoted 9 lines

Hi Roger,

We are testing the indirect feature of xen-blkfront module these days.
And we found that, after VM live-migrate a couple of times, the "%util" of iostat keeps being 100%, and there are several requests stock in "avgqu-sz". We have checked some later version of Linux, and it happens on Ubuntu 14.04, Ubuntu 14.10 and RHEL 7.0.

The iostat shows like below:

avg-cpu: %user %nice %system %iowait %steal %idle

Quoted 8 lines
           0.00    0.00    0.00    0.00    0.00  100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     4.00    0.00    0.00    0.00   0.00 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Could you tell us that why is this happening, is this a bug?

It is a bug indeed, thanks for reporting it. The problem seems to be that blk_put_request (which is used to discard the old requests before requeuing them) doesn't update the queue statistics. The following patch solves the problem for me, could you try it and report back?

---
commit bb4317c051ca81a2906edb7ccc505cbd6d1d80c7

Author: Roger Pau Monne <[email protected]>
Date:   Fri Jan 23 12:10:51 2015 +0100

xen-blkfront: fix accounting of reqs when migrating

Current migration code uses blk_put_request in order to finish a request before requeuing it. This function doesn't update the statistics of the queue, which completely screws accounting. Use blk_end_request_all instead which properly updates the statistics of the queue.

drivers/block/xen-blkfront.c +1 -1
index 5ac312f..aac41c1 100644
@@ -1493,7 +1493,7 @@ static int blkif_recover(struct blkfront_info *info)
 		merge_bio.tail = copy[i].request->biotail;
 		bio_list_merge(&bio_list, &merge_bio);
 		copy[i].request->bio = NULL;
-		blk_put_request(copy[i].request);
+		blk_end_request_all(copy[i].request, 0);
 	}
 
 	kfree(copy);

drivers/block/xen-blkfront.c static int blkif_recover(struct blkfront_info *info) · line 1522

Quoted 8 lines
drivers/block/xen-blkfront.c +1 -1
@@ -1516,7 +1516,7 @@ static int blkif_recover(struct blkfront_info *info)
 		req->bio = NULL;
 		if (req->cmd_flags & (REQ_FLUSH | REQ_FUA))
 			pr_alert("diskcache flush request found!\n");
-		__blk_put_request(info->rq, req);
+		__blk_end_request_all(req, 0);
 	}
 	spin_unlock_irq(&info->io_lock);
 

Hi Roger,

Thanks for answering this question. Sure, I'll try this patch and test VM migrating, so far it seems this patch has solved this bug (after 10 times migrate). I'll keep testing it for more times and will let you know if it's OK.

Regards,
Ouyang Zhaowei

#4 lore raw
Quoted 21 lines · Ouyang Zhaowei (Charles)
Quoted 19 lines · Roger Pau Monné

Hello,

El 23/01/15 a les 8.59, Ouyang Zhaowei (Charles) ha escrit:

Quoted 16 lines

Hi Roger,

We are testing the indirect feature of xen-blkfront module these days.
And we found that, after VM live-migrate a couple of times, the "%util" of iostat keeps being 100%, and there are several requests stock in "avgqu-sz". We have checked some later version of Linux, and it happens on Ubuntu 14.04, Ubuntu 14.10 and RHEL 7.0.

The iostat shows like below:

avg-cpu: %user %nice %system %iowait %steal %idle

           0.00    0.00    0.00    0.00    0.00  100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     4.00    0.00    0.00    0.00   0.00 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Could you tell us that why is this happening, is this a bug?

It is a bug indeed, thanks for reporting it. The problem seems to be that blk_put_request (which is used to discard the old requests before requeuing them) doesn't update the queue statistics. The following patch solves the problem for me, could you try it and report back?

Hi Roger,

After near 1000 times migrate test, the "%util" of iostat did not become 100% anymore, seems like the patch fix this bug

Thanks

Ouyang Zhaowei

Quoted 55 lines
Quoted 30 lines

---
commit bb4317c051ca81a2906edb7ccc505cbd6d1d80c7

Author: Roger Pau Monne <[email protected]>
Date:   Fri Jan 23 12:10:51 2015 +0100

xen-blkfront: fix accounting of reqs when migrating

Current migration code uses blk_put_request in order to finish a request before requeuing it. This function doesn't update the statistics of the queue, which completely screws accounting. Use blk_end_request_all instead which properly updates the statistics of the queue.

drivers/block/xen-blkfront.c +1 -1
index 5ac312f..aac41c1 100644
@@ -1493,7 +1493,7 @@ static int blkif_recover(struct blkfront_info *info)
 		merge_bio.tail = copy[i].request->biotail;
 		bio_list_merge(&bio_list, &merge_bio);
 		copy[i].request->bio = NULL;
-		blk_put_request(copy[i].request);
+		blk_end_request_all(copy[i].request, 0);
 	}
 
 	kfree(copy);

drivers/block/xen-blkfront.c static int blkif_recover(struct blkfront_info *info) · line 1522

Quoted 8 lines
drivers/block/xen-blkfront.c +1 -1
@@ -1516,7 +1516,7 @@ static int blkif_recover(struct blkfront_info *info)
 		req->bio = NULL;
 		if (req->cmd_flags & (REQ_FLUSH | REQ_FUA))
 			pr_alert("diskcache flush request found!\n");
-		__blk_put_request(info->rq, req);
+		__blk_end_request_all(req, 0);
 	}
 	spin_unlock_irq(&info->io_lock);
 

Hi Roger,

Thanks for answering this question. Sure, I'll try this patch and test VM migrating, so far it seems this patch has solved this bug (after 10 times migrate). I'll keep testing it for more times and will let you know if it's OK.

Regards,
Ouyang Zhaowei

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

.

#5 lore raw
In reply to Ouyang Zhaowei (Charles)
Quoted 21 lines · Ouyang Zhaowei (Charles)
Quoted 19 lines · Roger Pau Monné

Hello,

El 23/01/15 a les 8.59, Ouyang Zhaowei (Charles) ha escrit:

Quoted 16 lines

Hi Roger,

We are testing the indirect feature of xen-blkfront module these days.
And we found that, after VM live-migrate a couple of times, the "%util" of iostat keeps being 100%, and there are several requests stock in "avgqu-sz". We have checked some later version of Linux, and it happens on Ubuntu 14.04, Ubuntu 14.10 and RHEL 7.0.

The iostat shows like below:

avg-cpu: %user %nice %system %iowait %steal %idle

           0.00    0.00    0.00    0.00    0.00  100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     4.00    0.00    0.00    0.00   0.00 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Could you tell us that why is this happening, is this a bug?

It is a bug indeed, thanks for reporting it. The problem seems to be that blk_put_request (which is used to discard the old requests before requeuing them) doesn't update the queue statistics. The following patch solves the problem for me, could you try it and report back?

Hi Roger,

After near 1000 times migrate test, the "%util" of iostat did not become 100% anymore, seems like the patch fix this bug

Thanks

Ouyang Zhaowei

Quoted 47 lines
Quoted 30 lines

---
commit bb4317c051ca81a2906edb7ccc505cbd6d1d80c7

Author: Roger Pau Monne <[email protected]>
Date:   Fri Jan 23 12:10:51 2015 +0100

xen-blkfront: fix accounting of reqs when migrating

Current migration code uses blk_put_request in order to finish a request before requeuing it. This function doesn't update the statistics of the queue, which completely screws accounting. Use blk_end_request_all instead which properly updates the statistics of the queue.

drivers/block/xen-blkfront.c +1 -1
index 5ac312f..aac41c1 100644
@@ -1493,7 +1493,7 @@ static int blkif_recover(struct blkfront_info *info)
 		merge_bio.tail = copy[i].request->biotail;
 		bio_list_merge(&bio_list, &merge_bio);
 		copy[i].request->bio = NULL;
-		blk_put_request(copy[i].request);
+		blk_end_request_all(copy[i].request, 0);
 	}
 
 	kfree(copy);

drivers/block/xen-blkfront.c static int blkif_recover(struct blkfront_info *info) · line 1522

Quoted 8 lines
drivers/block/xen-blkfront.c +1 -1
@@ -1516,7 +1516,7 @@ static int blkif_recover(struct blkfront_info *info)
 		req->bio = NULL;
 		if (req->cmd_flags & (REQ_FLUSH | REQ_FUA))
 			pr_alert("diskcache flush request found!\n");
-		__blk_put_request(info->rq, req);
+		__blk_end_request_all(req, 0);
 	}
 	spin_unlock_irq(&info->io_lock);
 

Hi Roger,

Thanks for answering this question. Sure, I'll try this patch and test VM migrating, so far it seems this patch has solved this bug (after 10 times migrate). I'll keep testing it for more times and will let you know if it's OK.

Regards,
Ouyang Zhaowei

Quoted 8 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#6 lore raw
In reply to Roger Pau Monné
Quoted 55 lines · Roger Pau Monné

Hello,

El 23/01/15 a les 8.59, Ouyang Zhaowei (Charles) ha escrit:

Quoted 9 lines

Hi Roger,

We are testing the indirect feature of xen-blkfront module these days.
And we found that, after VM live-migrate a couple of times, the "%util" of iostat keeps being 100%, and there are several requests stock in "avgqu-sz". We have checked some later version of Linux, and it happens on Ubuntu 14.04, Ubuntu 14.10 and RHEL 7.0.

The iostat shows like below:

avg-cpu: %user %nice %system %iowait %steal %idle

Quoted 8 lines
           0.00    0.00    0.00    0.00    0.00  100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

xvda              0.00     0.00    0.00    0.00     0.00     0.00     0.00     4.00    0.00    0.00    0.00   0.00 100.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Could you tell us that why is this happening, is this a bug?

It is a bug indeed, thanks for reporting it. The problem seems to be that blk_put_request (which is used to discard the old requests before requeuing them) doesn't update the queue statistics. The following patch solves the problem for me, could you try it and report back?

---
commit bb4317c051ca81a2906edb7ccc505cbd6d1d80c7

Author: Roger Pau Monne <[email protected]>
Date:   Fri Jan 23 12:10:51 2015 +0100

xen-blkfront: fix accounting of reqs when migrating

Current migration code uses blk_put_request in order to finish a request before requeuing it. This function doesn't update the statistics of the queue, which completely screws accounting. Use blk_end_request_all instead which properly updates the statistics of the queue.

drivers/block/xen-blkfront.c +1 -1
index 5ac312f..aac41c1 100644
@@ -1493,7 +1493,7 @@ static int blkif_recover(struct blkfront_info *info)
 		merge_bio.tail = copy[i].request->biotail;
 		bio_list_merge(&bio_list, &merge_bio);
 		copy[i].request->bio = NULL;
-		blk_put_request(copy[i].request);
+		blk_end_request_all(copy[i].request, 0);
 	}
 
 	kfree(copy);

drivers/block/xen-blkfront.c static int blkif_recover(struct blkfront_info *info) · line 1522

Quoted 8 lines
drivers/block/xen-blkfront.c +1 -1
@@ -1516,7 +1516,7 @@ static int blkif_recover(struct blkfront_info *info)
 		req->bio = NULL;
 		if (req->cmd_flags & (REQ_FLUSH | REQ_FUA))
 			pr_alert("diskcache flush request found!\n");
-		__blk_put_request(info->rq, req);
+		__blk_end_request_all(req, 0);
 	}
 	spin_unlock_irq(&info->io_lock);
 

Hi Roger,

Thanks for answering this question. Sure, I'll try this patch and test VM migrating, so far it seems this patch has solved this bug (after 10 times migrate). I'll keep testing it for more times and will let you know if it's OK.

Regards,
Ouyang Zhaowei

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel