Skip to contentSkip to conversation

write performance with active snapshot

linux-lvm Peter Daum 4 participants 6 messages Nov 9, 2008 – Jan 31, 2009

lore
Share & tools
lore

Peter Daum

author
#0 lore raw

Hi,

for an application I am just working on it looks like lvm snapshots would be just what I need as far as functionality is concerned. Unfortunately, I am experiencing such a massive degradation in performance, that the result is almost useless.

I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; LVM version:2.02.39 (2008-06-27)

Library version: 1.02.27 (2008-06-25)
Driver version:  4.13.0)

Sequentially writing to a file (ext3) on a logical volume, I get a sustained performance of ~ 250 MB/sec. When I create a snapshot volume, the write throughput drops to 7-8 MB/secs (on the original volume; writing to the snapshot I see a significant degradation, but not nearly, as bad; read performance is o.k.).Is this "normal" or is there anything I can do to about it?

I looked in this list and searched the WWW but couldn't find any concrete information on the performance impact of snapshots (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3 of the original throughput, because every write to the source volume causes 3 I/O operations plus some overhead for meta data. More difficult to estimate would be the time lost by additional head movements. Still, a throughput degradation by a factor of 30 seems pretty extreme.

Any ideas?

Regards,
Peter Daum

#1 lore raw

My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the parent volume is on one array, the snapshot(s) on the other?

Larry Dickson
Cutting Edge Networked Storage

Quoted 43 lines · Peter Daum

Hi,

for an application I am just working on it looks like lvm snapshots would be just what I need as far as functionality is concerned. Unfortunately, I am experiencing such a massive degradation in performance, that the result is almost useless.

I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; LVM version:2.02.39 (2008-06-27)

Library version: 1.02.27 (2008-06-25)
Driver version:  4.13.0)

Sequentially writing to a file (ext3) on a logical volume, I get a sustained performance of ~ 250 MB/sec. When I create a snapshot volume, the write throughput drops to 7-8 MB/secs (on the original volume; writing to the snapshot I see a significant degradation, but not nearly, as bad; read performance is o.k.).Is this "normal" or is there anything I can do to about it?

I looked in this list and searched the WWW but couldn't find any concrete information on the performance impact of snapshots (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3 of the original throughput, because every write to the source volume causes 3 I/O operations plus some overhead for meta data. More difficult to estimate would be the time lost by additional head movements. Still, a throughput degradation by a factor of 30 seems pretty extreme.

Any ideas?

Regards,
Peter Daum

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
Quoted 4 lines · Larry Dickson

My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the parent volume is on one array, the snapshot(s) on the other?

At the expense of memory, LVM could copy larger pieces to snapshot on write. This would improve sequential write performance with snapshots on the same spindle. Is there a tuning knob for that? Does read_ahead affect COW?

Signature · 4 lines
	      Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.
author
#3 lore raw
In reply to Larry Dickson

Hi,

Quoted 1 line · Larry Dickson

My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the parent volume is on one array, the snapshot(s) on the other?

That was my suspicion, too (although I could not imagine such an extreme impact). Just for testing I added a single disk to the same volume group and put the snapshot onto that disk - amazingly it made hardly any difference (Actually, I'm almost glad about that, because the combination of a 12-disk-array with a single disk would be under almost all other aspects foolish).

One thing that does improve the performance a little (actually by 100%, which in this case meens still pretty lousy 16 MB/sec) is to increase the chunk size to 512kb. (I don't know yet, how this might affect performance when dealing with many small files) ...

Regards,
Peter

Quoted 37 lines · On 11/9/08, *Peter Daum* <[email protected] <mailto:[email protected]>> wrote:

Hi,

for an application I am just working on it looks like lvm snapshots would
be just what I need as far as functionality is concerned. Unfortunately, I am experiencing such a massive degradation in performance, that the result is almost useless.

I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a big hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; LVM version:2.02.39 (2008-06-27)
Library version: 1.02.27 (2008-06-25)
Driver version: 4.13.0)

Sequentially writing to a file (ext3) on a logical volume, I get a sustained performance of ~ 250 MB/sec. When I create a snapshot volume, the write throughput drops to 7-8 MB/secs (on the original volume; writing to the snapshot I see a significant degradation, but not nearly, as bad; read performance is o.k.).Is this "normal" or is there anything I can do to about it?

I looked in this list and searched the WWW but couldn't find any concrete information on the performance impact of snapshots (except http://www.nikhef.nl/~dennisvd/lvmcrap.html).
It seems like write performance should probably be less then 1/3 of the original throughput, because every write to the source volume causes 3 I/O operations plus some overhead for meta data. More difficult to estimate would be the time lost by additional head movements. Still, a throughput degradation by a factor of 30 seems pretty extreme.

Any ideas?

Regards,
Peter Daum

#4 lore raw

Hi Peter,

I am following your thread on this topic...have any solutions emerged? I as well have seen miserably performance when snapshots are active.

Thank you in advance for your feedback.

Regards,
Thomas

-----Original Message-----
From: Peter Daum <[email protected]>
To: [email protected]
Sent: Tue, 11 Nov 2008 8:04 am
Subject: [linux-lvm] Re: write performance with active snapshot

Hi,

Quoted 3 lines · Larry Dickson

My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the

parent

Quoted 1 line

volume is on one array, the snapshot(s) on the other?

That was my suspicion, too (although I could not imagine such an extreme

impact). Just for testing I added a single disk to the same volume group

and put the snapshot onto that disk - amazingly it made hardly any

difference (Actually, I'm almost glad about that, because the combination

of a 12-disk-array with a single disk would be under almost all other

aspects foolish).

One thing that does improve the performance a little (actually by 100%,

which in this case meens still pretty lousy 16 MB/sec) is to increase

the chunk size to 512kb. (I don't know yet, how this might affect

performance when dealing with many small files) ...

Regards,

  Peter

Quoted 1 line

On 11/9/08, *Peter Daum* <[email protected]

<mailto:[email protected]>>

Quoted 7 lines

wrote:

Hi,

for an application I am just working on it looks like lvm

snapshots

Quoted 3 lines

would

be just what I need as far as functionality is concerned.

Unfortunately,

Quoted 1 line

I am experiencing such a massive degradation in performance, that

the

Quoted 4 lines

result is almost useless.

I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a

big

Quoted 10 lines
    hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; 

    LVM version:2.02.39 (2008-06-27) 

    Library version: 1.02.27 (2008-06-25) 

    Driver version:  4.13.0) 


    Sequentially writing to a file (ext3) on a logical volume, I get 

a

Quoted 3 lines

sustained performance of ~ 250 MB/sec. When I create a snapshot

volume, the write throughput drops to 7-8 MB/secs (on the

original

Quoted 3 lines

volume; writing to the snapshot I see a significant degradation,

but not nearly, as bad; read performance is o.k.).Is this

"normal"

Quoted 1 line

or is there a

nything I can do to about it?

Quoted 20 lines

I looked in this list and searched the WWW but couldn't find any

concrete information on the performance impact of snapshots

(except http://www.nikhef.nl/~dennisvd/lvmcrap.html).

It seems like write performance should probably be less then 1/3

of the original throughput, because every write to the source

volume causes 3 I/O operations plus some overhead for meta data.

More difficult to estimate would be the time lost by additional

head movements. Still, a throughput degradation by a factor of 30

seems pretty extreme.

Quoted 6 lines

Any ideas?

Regards,

Peter Daum

_______________________________________________

linux-lvm mailing list

[email protected]

https://www.redhat.com/mailman/listinfo/linux-lvm

read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Hi

author
#5 lore raw

Hi Thomas,

Quoted 1 line · [email protected] wrote:

I am following your thread on this topic...have any solutions emerged? I as well have seen miserably performance when snapshots are active.

I am sorry, at least I still don't know any solution (except avoiding snapshots wherever performance matters )-:

Regards,
Peter

Quoted 155 lines

-----Original Message-----
From: Peter Daum <[email protected]>
To: [email protected]
Sent: Tue, 11 Nov 2008 8:04 am
Subject: [linux-lvm] Re: write performance with active snapshot

Hi,

Quoted 3 lines · Larry Dickson

My guess is that you are getting the typical seek overhead. Have you tried making a volume group out of two separate RAID arrays (driving different spindles), and using lvdisplay --maps to make sure the

parent

Quoted 1 line

volume is on one array, the snapshot(s) on the other?

That was my suspicion, too (although I could not imagine such an extreme

impact). Just for testing I added a single disk to the same volume group

and put the snapshot onto that disk - amazingly it made hardly any

difference (Actually, I'm almost glad about that, because the combination

of a 12-disk-array with a single disk would be under almost all other

aspects foolish).

One thing that does improve the performance a little (actually by 100%,

which in this case meens still pretty lousy 16 MB/sec) is to increase

the chunk size to 512kb. (I don't know yet, how this might affect

performance when dealing with many small files) ...

Regards,

Peter

Quoted 1 line

On 11/9/08, *Peter Daum* <[email protected]

<mailto:[email protected]>>

Quoted 7 lines

wrote:

Hi,

for an application I am just working on it looks like lvm

snapshots

Quoted 3 lines

would

be just what I need as far as functionality is concerned.

Unfortunately,

Quoted 1 line

I am experiencing such a massive degradation in performance, that

the

Quoted 4 lines

result is almost useless.

I'm working on a fairly fast machine (Quadcore, 8GB RAM) with a

big

Quoted 10 lines
    hardware RAID array and lvm2 (Debian Lenny; Linux 2.6.26-1-amd64; 

    LVM version:2.02.39 (2008-06-27) 

    Library version: 1.02.27 (2008-06-25) 

    Driver version:  4.13.0) 


    Sequentially writing to a file (ext3) on a logical volume, I get 

a

Quoted 3 lines

sustained performance of ~ 250 MB/sec. When I create a snapshot

volume, the write throughput drops to 7-8 MB/secs (on the

original

Quoted 3 lines

volume; writing to the snapshot I see a significant degradation,

but not nearly, as bad; read performance is o.k.).Is this

"normal"

Quoted 1 line

or is there a

nything I can do to about it?

Quoted 20 lines

I looked in this list and searched the WWW but couldn't find any

concrete information on the performance impact of snapshots

(except http://www.nikhef.nl/~dennisvd/lvmcrap.html).

It seems like write performance should probably be less then 1/3

of the original throughput, because every write to the source

volume causes 3 I/O operations plus some overhead for meta data.

More difficult to estimate would be the time lost by additional

head movements. Still, a throughput degradation by a factor of 30

seems pretty extreme.

Quoted 6 lines

Any ideas?

Regards,

Peter Daum

_______________________________________________

linux-lvm mailing list

[email protected]

https://www.redhat.com/mailman/listinfo/linux-lvm

read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Hi

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/