write performance with active snapshot
Hi,
for an application I am just working on it looks like lvm snapshots would be just what I need as far as functionality is concerned. Unfortunately, I am experiencing such a massive degradation in performance, that the result is almost useless.
I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; LVM version:2.02.39 (2008-06-27)
Library version: 1.02.27 (2008-06-25) Driver version: 4.13.0)
Sequentially writing to a file (ext3) on a logical volume, I get a sustained performance of ~ 250 MB/sec. When I create a snapshot volume, the write throughput drops to 7-8 MB/secs (on the original volume; writing to the snapshot I see a significant degradation, but not nearly, as bad; read performance is o.k.).Is this "normal" or is there anything I can do to about it?
I looked in this list and searched the WWW but couldn't find any concrete information on the performance impact of snapshots (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3 of the original throughput, because every write to the source volume causes 3 I/O operations plus some overhead for meta data. More difficult to estimate would be the time lost by additional head movements. Still, a throughput degradation by a factor of 30 seems pretty extreme.
Any ideas?
Regards,
Peter Daum
My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the parent volume is on one array, the snapshot(s) on the other?
Larry Dickson
Cutting Edge Networked Storage
Quoted 43 lines · Peter Daum
Hi,
for an application I am just working on it looks like lvm snapshots would be just what I need as far as functionality is concerned. Unfortunately, I am experiencing such a massive degradation in performance, that the result is almost useless.
I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; LVM version:2.02.39 (2008-06-27)
Library version: 1.02.27 (2008-06-25) Driver version: 4.13.0)Sequentially writing to a file (ext3) on a logical volume, I get a sustained performance of ~ 250 MB/sec. When I create a snapshot volume, the write throughput drops to 7-8 MB/secs (on the original volume; writing to the snapshot I see a significant degradation, but not nearly, as bad; read performance is o.k.).Is this "normal" or is there anything I can do to about it?
I looked in this list and searched the WWW but couldn't find any concrete information on the performance impact of snapshots (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3 of the original throughput, because every write to the source volume causes 3 I/O operations plus some overhead for meta data. More difficult to estimate would be the time lost by additional head movements. Still, a throughput degradation by a factor of 30 seems pretty extreme.Any ideas?
Regards,
Peter Daum_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 4 lines · Larry Dickson
My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the parent volume is on one array, the snapshot(s) on the other?
At the expense of memory, LVM could copy larger pieces to snapshot on write. This would improve sequential write performance with snapshots on the same spindle. Is there a tuning knob for that? Does read_ahead affect COW?
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Hi,
Quoted 1 line · Larry Dickson
My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the parent volume is on one array, the snapshot(s) on the other?
That was my suspicion, too (although I could not imagine such an extreme impact). Just for testing I added a single disk to the same volume group and put the snapshot onto that disk - amazingly it made hardly any difference (Actually, I'm almost glad about that, because the combination of a 12-disk-array with a single disk would be under almost all other aspects foolish).
One thing that does improve the performance a little (actually by 100%, which in this case meens still pretty lousy 16 MB/sec) is to increase the chunk size to 512kb. (I don't know yet, how this might affect performance when dealing with many small files) ...
Regards,
Peter
Quoted 37 lines · On 11/9/08, *Peter Daum* <[email protected] <mailto:[email protected]>> wrote:
Hi,
for an application I am just working on it looks like lvm snapshots would
be just what I need as far as functionality is concerned. Unfortunately, I am experiencing such a massive degradation in performance, that the result is almost useless.I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; LVM version:2.02.39 (2008-06-27)
Library version: 1.02.27 (2008-06-25)
Driver version: 4.13.0)Sequentially writing to a file (ext3) on a logical volume, I get a sustained performance of ~ 250 MB/sec. When I create a snapshot volume, the write throughput drops to 7-8 MB/secs (on the original volume; writing to the snapshot I see a significant degradation, but not nearly, as bad; read performance is o.k.).Is this "normal" or is there anything I can do to about it?
I looked in this list and searched the WWW but couldn't find any concrete information on the performance impact of snapshots (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3 of the original throughput, because every write to the source volume causes 3 I/O operations plus some overhead for meta data. More difficult to estimate would be the time lost by additional head movements. Still, a throughput degradation by a factor of 30 seems pretty extreme.Any ideas?
Regards,
Peter Daum
Hi Peter,
I am following your thread on this topic...have any solutions emerged? I as well have seen miserably performance when snapshots are active.
Thank you in advance for your feedback.
Regards,
Thomas
-----Original Message-----
From: Peter Daum <[email protected]>
To: [email protected]
Sent: Tue, 11 Nov 2008 8:04 am
Subject: [linux-lvm] Re: write performance with active snapshot
Hi,
Quoted 3 lines · Larry Dickson
My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the
parent
Quoted 1 line
volume is on one array, the snapshot(s) on the other?
That was my suspicion, too (although I could not imagine such an extreme
impact). Just for testing I added a single disk to the same volume group
and put the snapshot onto that disk - amazingly it made hardly any
difference (Actually, I'm almost glad about that, because the combination
of a 12-disk-array with a single disk would be under almost all other
aspects foolish).
One thing that does improve the performance a little (actually by 100%,
which in this case meens still pretty lousy 16 MB/sec) is to increase
the chunk size to 512kb. (I don't know yet, how this might affect
performance when dealing with many small files) ...
Regards,
Peter
Quoted 1 line
On 11/9/08, *Peter Daum* <[email protected]
<mailto:[email protected]>>
Quoted 7 lines
wrote:
Hi,
for an application I am just working on it looks like lvm
snapshots
Quoted 3 lines
would
be just what I need as far as functionality is concerned.
Unfortunately,
Quoted 1 line
I am experiencing such a massive degradation in performance, that
the
Quoted 4 lines
result is almost useless.
I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a
big
Quoted 10 lines
hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64;
LVM version:2.02.39 (2008-06-27)
Library version: 1.02.27 (2008-06-25)
Driver version: 4.13.0)
Sequentially writing to a file (ext3) on a logical volume, I get a
Quoted 3 lines
sustained performance of ~ 250 MB/sec. When I create a snapshot
volume, the write throughput drops to 7-8 MB/secs (on the
original
Quoted 3 lines
volume; writing to the snapshot I see a significant degradation,
but not nearly, as bad; read performance is o.k.).Is this
"normal"
Quoted 1 line
or is there a
nything I can do to about it?
Quoted 20 lines
I looked in this list and searched the WWW but couldn't find any
concrete information on the performance impact of snapshots
(except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3
of the original throughput, because every write to the source
volume causes 3 I/O operations plus some overhead for meta data.
More difficult to estimate would be the time lost by additional
head movements. Still, a throughput degradation by a factor of 30
seems pretty extreme.
Quoted 6 lines
Any ideas?
Regards,
Peter Daum
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Hi
Hi Thomas,
Quoted 1 line · [email protected] wrote:
I am following your thread on this topic...have any solutions emerged? I as well have seen miserably performance when snapshots are active.
I am sorry, at least I still don't know any solution (except avoiding snapshots wherever performance matters )-:
Regards,
Peter
Quoted 155 lines
-----Original Message-----
From: Peter Daum <[email protected]>
To: [email protected]
Sent: Tue, 11 Nov 2008 8:04 am
Subject: [linux-lvm] Re: write performance with active snapshotHi,
Quoted 3 lines · Larry Dickson
My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the
parent
Quoted 1 line
volume is on one array, the snapshot(s) on the other?
That was my suspicion, too (although I could not imagine such an extreme
impact). Just for testing I added a single disk to the same volume group
and put the snapshot onto that disk - amazingly it made hardly any
difference (Actually, I'm almost glad about that, because the combination
of a 12-disk-array with a single disk would be under almost all other
aspects foolish).
One thing that does improve the performance a little (actually by 100%,
which in this case meens still pretty lousy 16 MB/sec) is to increase
the chunk size to 512kb. (I don't know yet, how this might affect
performance when dealing with many small files) ...
Regards,
Peter
Quoted 1 line
On 11/9/08, *Peter Daum* <[email protected]
<mailto:[email protected]>>
Quoted 7 lines
wrote:
Hi,
for an application I am just working on it looks like lvm
snapshots
Quoted 3 lines
would
be just what I need as far as functionality is concerned.
Unfortunately,
Quoted 1 line
I am experiencing such a massive degradation in performance, that
the
Quoted 4 lines
result is almost useless.
I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a
big
Quoted 10 lines
hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; LVM version:2.02.39 (2008-06-27) Library version: 1.02.27 (2008-06-25) Driver version: 4.13.0) Sequentially writing to a file (ext3) on a logical volume, I geta
Quoted 3 lines
sustained performance of ~ 250 MB/sec. When I create a snapshot
volume, the write throughput drops to 7-8 MB/secs (on the
original
Quoted 3 lines
volume; writing to the snapshot I see a significant degradation,
but not nearly, as bad; read performance is o.k.).Is this
"normal"
Quoted 1 line
or is there a
nything I can do to about it?
Quoted 20 lines
I looked in this list and searched the WWW but couldn't find any
concrete information on the performance impact of snapshots
(except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3
of the original throughput, because every write to the source
volume causes 3 I/O operations plus some overhead for meta data.
More difficult to estimate would be the time lost by additional
head movements. Still, a throughput degradation by a factor of 30
seems pretty extreme.
Quoted 6 lines
Any ideas?
Regards,
Peter Daum
_______________________________________________
linux-lvm mailing list
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Hi
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/