Skip to contentSkip to conversation

xen/blkfront: increase the default value of xen_blkif_max_segments

PATCH Superseded Bob Liu 4 participants 11 messages Dec 16, 2014 – Dec 17, 2014

Subsystems (3)
lore
Share & tools
lore

Bob Liu

author
#0 lore raw

PATCH 1/1 xen/blkfront: increase the default value of xen_blkif_max_segments

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

---
 drivers/block/xen-blkfront.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/block/xen-blkfront.c +2 -2
index 2236c6f..1bf2429 100644
@@ -94,9 +94,9 @@ static const struct block_device_operations xlvbd_block_fops;
  * by the backend driver.
  */
 
-static unsigned int xen_blkif_max_segments = 32;
+static unsigned int xen_blkif_max_segments = 128;
 module_param_named(max, xen_blkif_max_segments, int, S_IRUGO);
-MODULE_PARM_DESC(max, "Maximum amount of segments in indirect requests (default is 32)");
+MODULE_PARM_DESC(max, "Maximum amount of segments in indirect requests (default is 128)");
 
 #define BLK_RING_SIZE __CONST_RING_SIZE(blkif, PAGE_SIZE)
 
Signature · 1 line
1.7.10.4
#1 lore raw

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 8 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the default amount of persistent grants blkback can handle, so the LRU in blkback will kick in.

Roger.

#2 lore raw
In reply to Bob Liu

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 8 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the default amount of persistent grants blkback can handle, so the LRU in blkback will kick in.

Roger.

author
#3 lore raw
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Quoted 3 lines

default amount of persistent grants blkback can handle, so the LRU in blkback will kick in.

Sounds good.

Signature · 2 lines
Regards,
-Bob
author
#4 lore raw
In reply to Roger Pau Monné
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Quoted 3 lines

default amount of persistent grants blkback can handle, so the LRU in blkback will kick in.

Sounds good.

Signature · 2 lines
Regards,
-Bob
#5 lore raw
Quoted 29 lines · Bob Liu
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 8 lines
Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Or potentially making the request and response be seperate rings - and the response ring entries not tied in to the request. As in right now if we have an request at say slot 1,5, and 7, we expect the response to be at slot 1,5, and 7 as well. If we made the response ring producer index not be tied to the request we could put the responses on the first available slot - so say at 1, 2, and 3 (if all three responses came at the same time).

Quoted 15 lines
Quoted 4 lines

default amount of persistent grants blkback can handle, so the LRU in blkback will kick in.

Sounds good.

Signature · 7 lines
Regards,
-Bob

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
#6 lore raw
In reply to Bob Liu
Quoted 29 lines · Bob Liu
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 8 lines
Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Or potentially making the request and response be seperate rings - and the response ring entries not tied in to the request. As in right now if we have an request at say slot 1,5, and 7, we expect the response to be at slot 1,5, and 7 as well. If we made the response ring producer index not be tied to the request we could put the responses on the first available slot - so say at 1, 2, and 3 (if all three responses came at the same time).

Quoted 15 lines
Quoted 4 lines

default amount of persistent grants blkback can handle, so the LRU in blkback will kick in.

Sounds good.

Signature · 7 lines
Regards,
-Bob

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
#7 lore raw
Quoted 39 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Bob Liu
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 8 lines
Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Quoted 4 lines

Or potentially making the request and response be seperate rings - and the response ring entries not tied in to the request. As in right now if we have an request at say slot 1,5, and 7, we expect the response to be at slot 1,5, and 7 as well.

No. Responses are placed in the first available slot. The response is associated with the original request by the ID field.

See make_response().

David

#8 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 39 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Bob Liu
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 8 lines
Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Quoted 4 lines

Or potentially making the request and response be seperate rings - and the response ring entries not tied in to the request. As in right now if we have an request at say slot 1,5, and 7, we expect the response to be at slot 1,5, and 7 as well.

No. Responses are placed in the first available slot. The response is associated with the original request by the ID field.

See make_response().

David

#9 lore raw
Quoted 45 lines · David Vrabel
Quoted 39 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Bob Liu
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 8 lines
Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Quoted 4 lines

Or potentially making the request and response be seperate rings - and the response ring entries not tied in to the request. As in right now if we have an request at say slot 1,5, and 7, we expect the response to be at slot 1,5, and 7 as well.

Quoted 4 lines

No. Responses are placed in the first available slot. The response is associated with the original request by the ID field.

See make_response().

You are right! Thank you for the update.

Quoted 2 lines

David

#10 lore raw
In reply to David Vrabel
Quoted 45 lines · David Vrabel
Quoted 39 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Bob Liu
Quoted 5 lines · Roger Pau Monné

El 16/12/14 a les 11.11, Bob Liu ha escrit:

Quoted 4 lines

The default maximum value of segments in indirect requests was 32, IO operations with bigger block size(>32*4k) would be split and performance start to drop.

Quoted 7 lines
Quoted 4 lines

Nowadays backend device usually support 512k max_sectors_kb on desktop, and may larger on server machines with high-end storage system.
The default size 128k was not very appropriate, this patch increases the default maximum value to 128(128*4k=512k).

This looks fine, do you have any data/graphs to backup your reasoning?

I only have some results for 1M block size FIO test but I think that's enough.

xen_blkfront.max Rate (MB/s) Percent of Dom-0
32 11.1 31.0%
48 15.3 42.7%
64 19.8 55.3%
80 19.9 55.6%
96 23.0 64.2%
112 23.7 66.2%
128 31.6 88.3%

The rates above are compared against the dom-0 rate of 35.8 MB/s.

Quoted 8 lines
Quoted 2 lines

I would also add to the commit message that this change implies we can now have 32*128+32 = 4128 in-flight grants, which greatly surpasses the

The number could be larger if using more pages as the xen-blkfront/backend ring based on Wei Liu's patch "xenbus_client:
extend interface to suppurt multi-page ring", it helped improve the IO performance a lot on our system connected with high-end storage. I'm preparing resend related patches.

Quoted 4 lines

Or potentially making the request and response be seperate rings - and the response ring entries not tied in to the request. As in right now if we have an request at say slot 1,5, and 7, we expect the response to be at slot 1,5, and 7 as well.

Quoted 4 lines

No. Responses are placed in the first available slot. The response is associated with the original request by the ID field.

See make_response().

You are right! Thank you for the update.

Quoted 2 lines

David