LVM and I/O block size (max_sectors_kb)
Hi all,
it seems that, when using LVM, I/O block transfer size has an hard limit at about 512 KB/iop. Large I/O transfers can be crucial for performance, so I am trying to understand if I can change that.
Some info: uname -a (CentOS 6.5 x86_64)
Linux blackhole.assyoma.it 2.6.32-431.20.3.el6.x86_64 #1 SMP Thu Jun 19 21:14:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
From my understanding, maximum I/O block size for physical device can be tuned from /sys/block/sdX/queue/max_sectors_kb.
Some quick iostat -k -x 1 show that, for physical device, the tunable works:
# max_sectors_kb=512 (default)
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct
iostat -x -k 1 /dev/sdd: avgrq-sz=1024 (1024 sectors = 512KB = max_sectors_kb)
# max_sectors_kb=2048
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct iostat -x -k 1 /dev/sdd: avgrq-sz=2048 (2048 sectors = 1024 KB)
dd if=/dev/zero of=/dev/sdd1 bs=4M count=16 oflag=direct iostat -x -k 1 /dev/sdd: avgrq-sz=2730.67 (2730 sectors = ~1350 KB)
As you can see, I can't always reach 100% efficienty but I came reasonably close.
The problem is that, when using LVM2 (on software raid10), I can not really increase I/O transfer size. Setting both sd*, md* and dm-* to max_sectors_kb=2048 leads to _no_ increase in I/O blocks, while decreasing the max_sectors_kb works properly:
# max_sectors_kb=2048
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct iostat -x -k 1 /dev/vg_kvm/TEST: avgrq-sz=1024.00 (it remains as 512KB)
# max_sectors_kb=256
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct
iostat -x -k 1 /dev/vg_kvm/TEST: avgrq-sz=512.00 (512 sectors = 256KB = max_sectors_kb)
Now, two questions:
1) why I see this hard limit?
2) can I change this behavior?
Thank you very much.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 45 lines · Gionatan Danti
Hi all,
it seems that, when using LVM, I/O block transfer size has an hard limit at about 512 KB/iop. Large I/O transfers can be crucial for performance, so I am trying to understand if I can change that.Some info: uname -a (CentOS 6.5
x86_64)
Linux blackhole.assyoma.it 2.6.32-431.20.3.el6.x86_64 #1 SMP Thu Jun 19 21:14:45 UTC 2014x86_64x86_64x86_64GNU/LinuxFrom my understanding, maximum I/O block size for physical device can
be tuned from/sys/block/sdX/queue/max_sectors_kb.Some quick iostat -k -x 1 show that, for physical device, the tunable works:
# max_sectors_kb=512 (default)
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct iostat -x -k 1/dev/sdd: avgrq-sz=1024 (1024 sectors = 512KB =max_sectors_kb)# max_sectors_kb=2048
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct iostat -x -k 1/dev/sdd: avgrq-sz=2048 (2048 sectors = 1024 KB)dd if=/dev/zero of=/dev/sdd1 bs=4M count=16 oflag=direct iostat -x -k 1
/dev/sdd: avgrq-sz=2730.67 (2730 sectors = ~1350 KB)As you can see, I can't always reach 100% efficienty but I came reasonably close.
The problem is that, when using LVM2 (on software raid10), I can not really increase I/O transfer size. Setting both sd*, md* and dm-* to max_sectors_kb=2048 leads to _no_ increase in I/O blocks, while decreasing the
max_sectors_kbworks properly:# max_sectors_kb=2048
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct iostat -x -k 1/dev/vg_kvm/TEST: avgrq-sz=1024.00 (it remains as 512KB)# max_sectors_kb=256
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct iostat -x -k 1/dev/vg_kvm/TEST: avgrq-sz=512.00 (512 sectors = 256KB =max_sectors_kb)
Quoted 6 lines
Now, two questions:
1) why I see this hard limit?
2) can I change this behavior?Thank you very much.
Anyone with some ideas? :)
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 43 lines
Hi all,
it seems that, when using LVM, I/O block transfer size has an hard limit at about 512 KB/iop. Large I/O transfers can be crucial for performance, so I am trying to understand if I can change that.Some info: uname -a (CentOS 6.5
x86_64)
Linux blackhole.assyoma.it 2.6.32-431.20.3.el6.x86_64 #1 SMP Thu Jun 19 21:14:45 UTC 2014x86_64x86_64x86_64GNU/LinuxFrom my understanding, maximum I/O block size for physical device can be tuned from
/sys/block/sdX/queue/max_sectors_kb.Some quick iostat -k -x 1 show that, for physical device, the tunable works:
# max_sectors_kb=512 (default)
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct
iostat -x -k 1/dev/sdd: avgrq-sz=1024 (1024 sectors = 512KB =max_sectors_kb)# max_sectors_kb=2048
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct iostat -x -k 1/dev/sdd: avgrq-sz=2048 (2048 sectors = 1024 KB)dd if=/dev/zero of=/dev/sdd1 bs=4M count=16 oflag=direct iostat -x -k 1
/dev/sdd: avgrq-sz=2730.67 (2730 sectors = ~1350 KB)As you can see, I can't always reach 100% efficienty but I came reasonably close.
The problem is that, when using LVM2 (on software raid10), I can not really increase I/O transfer size. Setting both sd*, md* and dm-* to max_sectors_kb=2048 leads to _no_ increase in I/O blocks, while decreasing the
max_sectors_kbworks properly:# max_sectors_kb=2048
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct iostat -x -k 1/dev/vg_kvm/TEST: avgrq-sz=1024.00 (it remains as 512KB)# max_sectors_kb=256
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct iostat -x -k 1/dev/vg_kvm/TEST: avgrq-sz=512.00 (512 sectors = 256KB =max_sectors_kb)
Quoted 5 lines
Now, two questions:
1) why I see this hard limit?
2) can I change this behavior?Thank you very much.
Hi all,
sorry for bumping up this thread.
Anyone has an explanation for what I see? I am missing something?
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 40 lines · Gionatan Danti
Quoted 40 lines
Hi all,
it seems that, when using LVM, I/O block transfer size has an hard limit at about 512 KB/iop. Large I/O transfers can be crucial for performance, so I am trying to understand if I can change that.Some info: uname -a (CentOS 6.5
x86_64)
Linux blackhole.assyoma.it 2.6.32-431.20.3.el6.x86_64 #1 SMP Thu Jun 19 21:14:45 UTC 2014x86_64x86_64x86_64GNU/LinuxFrom my understanding, maximum I/O block size for physical device can be tuned from
/sys/block/sdX/queue/max_sectors_kb.Some quick iostat -k -x 1 show that, for physical device, the tunable works:
# max_sectors_kb=512 (default)
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct iostat -x -k 1/dev/sdd: avgrq-sz=1024 (1024 sectors = 512KB =max_sectors_kb)# max_sectors_kb=2048
dd if=/dev/zero of=/dev/sdd1 bs=2M count=16 oflag=direct iostat -x -k 1/dev/sdd: avgrq-sz=2048 (2048 sectors = 1024 KB)dd if=/dev/zero of=/dev/sdd1 bs=4M count=16 oflag=direct iostat -x -k 1
/dev/sdd: avgrq-sz=2730.67 (2730 sectors = ~1350 KB)As you can see, I can't always reach 100% efficienty but I came reasonably close.
The problem is that, when using LVM2 (on software raid10), I can not really increase I/O transfer size. Setting both sd*, md* and dm-* to max_sectors_kb=2048 leads to _no_ increase in I/O blocks, while decreasing the
max_sectors_kbworks properly:# max_sectors_kb=2048
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct iostat -x -k 1/dev/vg_kvm/TEST: avgrq-sz=1024.00 (it remains as 512KB)
Quoted 7 lines
Quoted 7 lines
# max_sectors_kb=256
dd if=/dev/zero of=/dev/vg_kvm/TEST bs=2M count=64 oflag=direct iostat -x -k 1/dev/vg_kvm/TEST: avgrq-sz=512.00 (512 sectors = 256KB =max_sectors_kb)Now, two questions:
1) why I see this hard limit?
1. As there are more disks in the VG, can them all work with larger max_sectors_kb?
2. Does it work for md device without stacking LVM on top of that?
3. What's your Physical extent size?
vgs -ovg_extent_size vg_kvm # or simple vgs -v
4. If all above is fine then it may be related to LVM.
-- Marian
Quoted 11 lines
Quoted 3 lines
2) can I change this behavior?
Thank you very much.
Hi all,
sorry for bumping up this thread.Anyone has an explanation for what I see? I am missing something?
Thanks.
Hi Marian,
Quoted 2 lines
1. As there are more disks in the VG, can them all work with larger
max_sectors_kb?
All four disks and relative MD device has max_sectors_kb set to the required (large) 2048 value. They all are recent (circa 2010) SATA disks, albeit from different vendors.
Quoted 1 line
2. Does it work for md device without stacking LVM on top of that?
I did not try to directly use the MD device. Let me test this and I will report here.
Quoted 3 lines
3. What's your Physical extent size?
vgs -ovg_extent_size
vg_kvm# or simple vgs -v
PE size is at default (4MiB).
Quoted 1 line
4. If all above is fine then it may be related to LVM.
I think so, because I did some test with a _single_ disk (an SSD, actually) with and without LVM on top. Without LVM, I see the normal and expected behavior - IO transfer size increase with max_sectors_kb. With LVM on top, I see the exact same behavior I am reporting: max IO transfer size seems capped at 512 KB, regardless the max_sectors_kb setting.
It really seems something related to LVM layer.
Quoted 1 line
-- Marian
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
my understanding was that this is an OLD problem (as designed?) Conceivably LVM could be written to be smart enough to query all of the underlying PV and arrive at the lowest common value <= PhysicalExtent-Size. But it was probably more expedient to just hard code it to the native BIO value.