-b vs. -n
Has anybody experimented with increasing the _number_ of buffers rather than the _size_ of the buffers when confronted with drops? I'm finding on a large(ish) system that it is better to have lots of small buffers handled by relay rather than fewer larger buffers. In my particular case:
16 CPUs
96 devices
running some dd's against all the devices...
-b 1024 or -b 2048 still results in drops
but:
-n 512 -b 16 allows things to run smoother.
I _think_ this may have to do with the way relay reports POLLIN: it does it only when a buffer switch happens as opposed to when there is data ready. Need to look at this some more, but just wondering if others out there have found similar things in their testing...
Alan
Quoted 11 lines · Alan D. Brunelle
Has anybody experimented with increasing the _number_ of buffers rather than the _size_ of the buffers when confronted with drops? I'm finding on a large(ish) system that it is better to have lots of small buffers handled by relay rather than fewer larger buffers. In my particular case:
16 CPUs
96 devices
running some dd's against all the devices...-b 1024 or -b 2048 still results in drops
Quoted 8 lines
but:
-n 512 -b 16 allows things to run smoother.
I _think_ this may have to do with the way relay reports POLLIN: it does it only when a buffer switch happens as opposed to when there is data ready. Need to look at this some more, but just wondering if others out there have found similar things in their testing...
That's interesting. The reason why I exposed both parameters was mainly that I didn't have the equipment to do adequate testing on what would be the best setup for this. So perhaps we can change the README to reflect that it is usually better to bump the number of buffers instead of the size, if you run into overflow problems?
Signature · 1 line
Jens Axboe
Quoted 16 lines · Jens Axboe
Quoted 15 lines · Alan D. Brunelle
Has anybody experimented with increasing the _number_ of buffers rather than the _size_ of the buffers when confronted with drops? I'm finding on a large(ish) system that it is better to have lots of small buffers handled by relay rather than fewer larger buffers. In my particular case:
16 CPUs
96 devices
running some dd's against all the devices...-b 1024 or -b 2048 still results in drops
but:
-n 512 -b 16 allows things to run smoother.
Quoted 11 lines
Quoted 4 lines
I _think_ this may have to do with the way relay reports POLLIN: it does it only when a buffer switch happens as opposed to when there is data ready. Need to look at this some more, but just wondering if others out there have found similar things in their testing...
That's interesting. The reason why I exposed both parameters was mainly that I didn't have the equipment to do adequate testing on what would be the best setup for this. So perhaps we can change the README to reflect that it is usually better to bump the number of buffers instead of the size, if you run into overflow problems?
It's not clear - still running tests. [I know for SMALLER numbers of disks increasing the buffers has worked just fine.] I'm still fighting (part time) with version 2.0 of blktrace, so _that_ may have something to do with it! :-)
Alan
Quoted 28 lines · Alan D. Brunelle
Quoted 16 lines · Jens Axboe
Quoted 15 lines · Alan D. Brunelle
Has anybody experimented with increasing the _number_ of buffers rather than the _size_ of the buffers when confronted with drops? I'm finding on a large(ish) system that it is better to have lots of small buffers handled by relay rather than fewer larger buffers. In my particular case:
16 CPUs
96 devices
running some dd's against all the devices...-b 1024 or -b 2048 still results in drops
but:
-n 512 -b 16 allows things to run smoother.
Quoted 10 lines
Quoted 4 lines
I _think_ this may have to do with the way relay reports POLLIN: it does it only when a buffer switch happens as opposed to when there is data ready. Need to look at this some more, but just wondering if others out there have found similar things in their testing...
That's interesting. The reason why I exposed both parameters was mainly that I didn't have the equipment to do adequate testing on what would be the best setup for this. So perhaps we can change the README to reflect that it is usually better to bump the number of buffers instead of the size, if you run into overflow problems?
Quoted 6 lines
It's not clear - still running tests. [I know for SMALLER numbers of disks increasing the buffers has worked just fine.] I'm still fighting (part time) with version 2.0 of blktrace, so _that_ may have something to do with it! :-)
Alan
Did some testing over the weekend: purposefully "bad" set up:
o 48 FC disks having Ext2 file systems created on them (I've found mkfs really stresses systems :-) )
o 5 CCISS disks in LVM w/ an XFS file system used to capture the traces.
I then did 30 passes for a select set of permutations of -n & -b parameters. Here's the averages:
-b 4 8 16 32 64 128 256 512 1024 2048 4096
-n |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- 4| 77.9 83.2 88.1 86.7 8| 77.9 73.7 16| 86.1 65.5 32| 86.0 64.7 64| 85.6 128| 85.8 256| 85.6 512| 86.9 1024| 79.8 2048| 66.1 4096| 70.8
The values in the chart are percent of traces dropped* (you can see that this is a bad set up - >60% drops in all case). (-b is across, and -n is down).
Looking at increasing -b from the default (512) to 4096 whilst keep -n @ 4 shows _more_ drops: 77.9 83.2 88.1 86.7
Looking at increasing -n from the default (4) to 2048 whilst keeping -b @ 512 shows _fewer_ drops: 77.9, 73.7, 65.5 and then 64.7
(Doing this with a small buffer size - 4KiB - was a bit inconclusive:
86.9 -> 79.8 -> 66.1 but then up to 70.8.)
The diagonal numbers represent the results from trying to keep the total memory consumption level - 4 buffers @ 512K up to 512 buffers @ 4K. Not too conclusive, but it seems that there's a benefit to having smaller numbers of larger buffers keeping the same memory footprint.
Anyways, none of this looks too convincing overall - and as noted, it's a "bad" case - way too many drops.
-----------------
I'm re-doing this using a more balanced configuration: I'm using the 48 fibre channel disks to hold the traces, and using 48 CCISS disks to do the mkfs operations on. (Preliminary results show we're around the hairy edge here - a few % drops on some case (<2.0%).)
-----------------
* I have modified blktrace to output the total number of drops, the percentage of drops, and changed the suggestion line to read:
You have 254436128 ( 81.3%) dropped events
Consider using a larger buffer size (-b) and/or more buffers (-n)
Quoted 91 lines · Alan D. Brunelle
Quoted 27 lines · Alan D. Brunelle
Quoted 16 lines · Jens Axboe
Quoted 15 lines · Alan D. Brunelle
Has anybody experimented with increasing the _number_ of buffers rather than the _size_ of the buffers when confronted with drops? I'm finding on a large(ish) system that it is better to have lots of small buffers handled by relay rather than fewer larger buffers. In my particular case:
16 CPUs
96 devices
running some dd's against all the devices...-b 1024 or -b 2048 still results in drops
but:
-n 512 -b 16 allows things to run smoother.
Quoted 10 lines
Quoted 4 lines
I _think_ this may have to do with the way relay reports POLLIN: it does it only when a buffer switch happens as opposed to when there is data ready. Need to look at this some more, but just wondering if others out there have found similar things in their testing...
That's interesting. The reason why I exposed both parameters was mainly that I didn't have the equipment to do adequate testing on what would be the best setup for this. So perhaps we can change the README to reflect that it is usually better to bump the number of buffers instead of the size, if you run into overflow problems?
Quoted 6 lines
It's not clear - still running tests. [I know for SMALLER numbers of disks increasing the buffers has worked just fine.] I'm still fighting (part time) with version 2.0 of blktrace, so _that_ may have something to do with it! :-)
Alan
Did some testing over the weekend: purposefully "bad" set up:
o 48 FC disks having Ext2 file systems created on them (I've found mkfs really stresses systems :-) )
o 5 CCISS disks in LVM w/ an XFS file system used to capture the traces.I then did 30 passes for a select set of permutations of -n & -b parameters. Here's the averages:
-b 4 8 16 32 64 128 256 512 1024 2048 4096
-n |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- 4| 77.9 83.2 88.1 86.7 8| 77.9 73.7 16| 86.1 65.5 32| 86.0 64.7 64| 85.6 128| 85.8 256| 85.6 512| 86.9 1024| 79.8 2048| 66.1 4096| 70.8The values in the chart are percent of traces dropped* (you can see that this is a bad set up - >60% drops in all case). (-b is across, and -n is down).
Looking at increasing -b from the default (512) to 4096 whilst keep -n @ 4 shows _more_ drops: 77.9 83.2 88.1 86.7
Looking at increasing -n from the default (4) to 2048 whilst keeping -b @ 512 shows _fewer_ drops: 77.9, 73.7, 65.5 and then 64.7
(Doing this with a small buffer size - 4KiB - was a bit inconclusive:
86.9 -> 79.8 -> 66.1 but then up to 70.8.)The diagonal numbers represent the results from trying to keep the total memory consumption level - 4 buffers @ 512K up to 512 buffers @ 4K. Not too conclusive, but it seems that there's a benefit to having smaller numbers of larger buffers keeping the same memory footprint.
Anyways, none of this looks too convincing overall - and as noted, it's a "bad" case - way too many drops.
-----------------
I'm re-doing this using a more balanced configuration: I'm using the 48 fibre channel disks to hold the traces, and using 48 CCISS disks to do the mkfs operations on. (Preliminary results show we're around the hairy edge here - a few % drops on some case (<2.0%).)
-----------------
* I have modified blktrace to output the total number of drops, the percentage of drops, and changed the suggestion line to read:
Quoted 7 lines
You have 254436128 ( 81.3%) dropped events
Consider using a larger buffer size (-b) and/or more buffers (-n) --
To unsubscribe from this list: send the line "unsubscribe linux-btrace" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Well, the "balanced configuration" didn't help much: going in either direction (more buffers or larger buffers) both helped. (This is with 48 FC disks set up in an LV & capturing traces generated whilst doing mkfs on 48 CCISS disks.)
-b 4 8 16 32 64 128 256 512 1024 2048 4096
-n |----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
4| 4.4 0.0 0.0 0.0
8| 1.5 0.0
16| 0.1 0.0
32| 0.8 0.0
64| 1.1
128| 0.8
256| 2.6
512| 2.3
1024| 0.5
2048| 0.1
4096| 0.0I _think_ having the error message reflect the possibility of upping either buffer size OR number of buffers makes sense. I'll also back-port the "You have 254436128 ( 81.3%) dropped events" formatted message code too.
Alan