LVM and *bad* performance (no striping)

I tried to use LVM today using Linux 2.4.2 and it mostly works like I expected. But the performance is unacceptably bad:

    isnogud:/root# fdisk -l /dev/sda|grep sda5
    /dev/sda5            80       399   2570368+  8e  Linux LVM
    isnogud:/root# pvcreate /dev/sda5
    pvcreate -- physical volume "/dev/sda5" successfully created

    isnogud:/root# vgcreate vg0 /dev/sda5
    vgcreate -- INFO: using default physical extent size 4 MB
    vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
    vgcreate -- doing automatic backup of volume group "vg0"
    vgcreate -- volume group "vg0" successfully created and activated

    isnogud:/root# lvcreate -n test /dev/vg0 -L 300M
    lvcreate -- doing automatic backup of "vg0"
    lvcreate -- logical volume "/dev/vg0/test" successfully created

    isnogud:/root# time dd if=/dev/sda5 of=/dev/null count=100000
    100000+0 records in
    100000+0 records out

    real    0m6.720s
    user    0m0.300s
    sys     0m0.970s
    isnogud:/root# time dd if=/dev/vg0/test of=/dev/null count=100000
    100000+0 records in
    100000+0 records out

    real    0m41.851s
    user    0m0.350s
    sys     0m2.250s

When making an ext2 filesystem on /dev/sda5 or /dev/vg0/test respectivley, the performance of cp'ing, grep'ing all files and rm'ing all files, seems to be comparable. But why is the performance of reading the block device with dd so bad?

urs

Every now and then someone posts to the list claiming LVM performance is really bad. We go off and try it and find the overhead is typically less than 1%.

Normally people are trying to stripe to two partitions on the same device, but you aren't.

Is any one else on the list seeing a similar performance hit ?

- Joe

here are my stats... (be your owen judge....)

Jorg

[root] /root > time dd if=/dev/hde of=/dev/null count=100000 100000+0 records in
100000+0 records out
0.200u 1.050s 0:03.95 31.6% 0+0k 0+0io 116pf+0w

[root] /root > time dd if=/dev/vg00/vm of=/dev/null count=100000 100000+0 records in
100000+0 records out
0.160u 0.860s 0:02.92 34.9% 0+0k 0+0io 116pf+0w

Interesting, so LVM actually takes a full second LESS to read in the same data ...

Maybe LVM does some device-level readahead to speed things up ?

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

I've always seen bad performance doing dd on LVM volumes but very good performance in "real world" performance or bonnie++ tests.

Stop worrying about it, how many times do you *need* to dd the start of a disk at high speed eh? :-)

patrick

I did some more tests to ensure that this is not a caching issue. Also, I found the performance of dd on LVM depends much on the block size.

I still have the effect that reading from LVM with small block size (smaller than 8k on my system) is *much* slower than reading the physical disk directly. IMHO it'd be worth looking into it and possibly fixing this performance hit. Could it be possible, that the read syscall on the LVM block device has so much overhead that reading small blocks can't cope with SCSI disk's transfer rate, so that additional disk revolutions are needed to read all the physical blocks? I find it noticable that when doubling the block size from 1k to 8k the time for reading quite exactly halves until it reaches the performance of reading /dev/sda5 directly.

In case it matters, the machine I did the test on is a

	ASUS P2L97-S mainboard
	PII 333MHz
	128 MB RAM
	Adaptec AIC-7880U onboard UW-SCSI controller
	/dev/sda is a IBM DCAS-34330W
	kernel: Linux 2.4.2

I've put the commands into a script that uses "set -x" to show what is exec'ed. Here's the result:

Initialisation:

+ pvcreate /dev/sda5
pvcreate -- physical volume "/dev/sda5" successfully created

+ vgcreate vg0 /dev/sda5
vgcreate -- INFO: using default physical extent size 4 MB vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte vgcreate -- doing automatic backup of volume group "vg0" vgcreate -- volume group "vg0" successfully created and activated

+ lvcreate -n test /dev/vg0 -L 500M
lvcreate -- doing automatic backup of "vg0"
lvcreate -- logical volume "/dev/vg0/test" successfully created

The first test shows, that, with small block sizes, LVM over sda5 is much slower than reading sda5 directly. I do the reading three times to see if any caching affects the numbers. This is obviously not the case:

    + dd if=/dev/sda5 of=/dev/null count=128k bs=512
    131072+0 records in
    131072+0 records out
    
    real    0m8.862s
    user    0m0.120s
    sys     0m1.690s
    + dd if=/dev/sda5 of=/dev/null count=128k bs=512
    131072+0 records in
    131072+0 records out
    
    real    0m8.860s
    user    0m0.200s
    sys     0m1.670s
    + dd if=/dev/sda5 of=/dev/null count=128k bs=512
    131072+0 records in
    131072+0 records out
    
    real    0m8.860s
    user    0m0.110s
    sys     0m1.800s
    + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
    131072+0 records in
    131072+0 records out
    
    real    0m54.860s
    user    0m0.150s
    sys     0m2.780s
    + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
    131072+0 records in
    131072+0 records out
    
    real    0m54.810s
    user    0m0.170s
    sys     0m2.660s
    + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
    131072+0 records in
    131072+0 records out
    
    real    0m54.936s
    user    0m0.200s
    sys     0m3.010s

The next test shows that the performance of LVM is very dependend on the blocksize but that of sda5 is not, which I find surprising since the same physical device is involved.

    + dd if=/dev/sda5 of=/dev/null count=512 bs=128k
    512+0 records in
    512+0 records out
    
    real    0m8.857s
    user    0m0.010s
    sys     0m1.440s
    + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
    131072+0 records in
    131072+0 records out
    
    real    0m55.120s
    user    0m0.190s
    sys     0m2.760s
    + dd if=/dev/vg0/test of=/dev/null count=64k bs=1k
    65536+0 records in
    65536+0 records out
    
    real    0m55.195s
    user    0m0.170s
    sys     0m2.970s
    + dd if=/dev/vg0/test of=/dev/null count=32k bs=2k
    32768+0 records in
    32768+0 records out
    
    real    0m28.470s
    user    0m0.090s
    sys     0m1.830s
    + dd if=/dev/vg0/test of=/dev/null count=16k bs=4k
    16384+0 records in
    16384+0 records out
    
    real    0m15.209s
    user    0m0.010s
    sys     0m1.860s
    + dd if=/dev/vg0/test of=/dev/null count=8k bs=8k
    8192+0 records in
    8192+0 records out
    
    real    0m8.859s
    user    0m0.010s
    sys     0m1.670s
    + dd if=/dev/vg0/test of=/dev/null count=4k bs=16k
    4096+0 records in
    4096+0 records out
    
    real    0m8.860s
    user    0m0.010s
    sys     0m1.570s
    + dd if=/dev/vg0/test of=/dev/null count=2k bs=32k
    2048+0 records in
    2048+0 records out
    
    real    0m8.873s
    user    0m0.010s
    sys     0m1.390s
    + dd if=/dev/vg0/test of=/dev/null count=1k bs=64k
    1024+0 records in
    1024+0 records out
    
    real    0m8.883s
    user    0m0.000s
    sys     0m1.380s
    + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
    512+0 records in
    512+0 records out
    
    real    0m8.862s
    user    0m0.000s
    sys     0m1.370s

Performance on filessystems on LVM is better, since the buffer cache can be used. I thought that the buffer cache works on block devices. But this seems to suggest that it only works with files systems mounted on the block device. Is this really so and why?

    + mke2fs -q /dev/vg0/test
    mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
    + mount /dev/vg0/test /mnt
    + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
    512+0 records in
    512+0 records out
    
    real    0m8.792s
    user    0m0.000s
    sys     0m1.280s
    + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
    512+0 records in
    512+0 records out
    
    real    0m1.044s
    user    0m0.000s
    sys     0m1.050s

For tests like this it would be useful to be able to completely flush the kernel's cache. Is there a tool to do this?

urs

Hi Urs,

I can't reproduce at all what you are seeing with your configuration.

My test configuration is:

- Dual Pentium II/350
- 256MB RAM
- Adaptec 2940UW
- HDD Vendor: QUANTUM Model: XP34550W Rev: LXY4 (/dev/sdb)
- first 300M of the LV involved allocated contiguous at the beginning of /dev/sdb1
- LVM 0.9.1 Beta 5
- Linux 2.4.2
- dd (GNU fileutils) 4.0

This table records test runs which where run one after the other *without* any additional load on the machine using dd to copy 300m into /dev/null. Each run performed twice with little difference + build average.

		1k	2k	4k	8k	16k

-----------------------------------------------------------
sdb1 32.63 32.8 32,7 32.55 32.9
vg00/u1 32.58 33.1 32.63 32.4 32.4

This gives *no* hint to me that there's any basical overhead invloved with LVM as oposed to the native device beeing used at all.

Therefore I guess that the allocation policy of LVM might have an impact here.

LVM calculates the offset of the first PE from the end of the PV. There could be drives used which suffer from that placement with regard to drive internal caching, latency or other model specific issues.

Could everybody who posts with regard to this performance subject please provide

- "lvdisplay -v" of the test LV
- the drive vendor and model
- the host adaptor used to access it
- the motherboard and CPU(s)
- the RAM size

in order to get a better understanding.

BTW: Rik van Riel's assumption in another mail of this thread, that LVM
does some sort of read ahead is correct (as you can see with the above numbers).
Even though we still only have a per block major read ahead in the kernel *and* multithreading will make the actual read ahead implementation fail with regard parallel access LVs (as Andrea Arcangeli pointed out a while ago), it hits quite good on not too heavy loaded systems. Taking into account that a lot of users most likely don't change the read ahead default of LVs, it won't have an impact at all.

Regards,
Heinz -- The LVM Guy --

*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf Germany
[email protected] +49 2626 141200

                                                       FAX 924446

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Today, I've found some time to make some further performance tests on my system.

I find surprinsing that we get so different performance results since my hardware is very similar to yours. Here is it, again:

Mainboard: ASUS P2L97-S with UW-SCSI controller onboard
RAM: 128 MB
CPU: from /proc/cpuinfo:

    model name      : Pentium II (Deschutes)
    stepping        : 1
    cpu MHz         : 334.095

/dev/sda: from /proc/scsi/scsi:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: DCAS-34330W Rev: S65A
Type: Direct-Access ANSI SCSI revision: 02
SCSI controller: from /proc/scsi/aic7xxx/0:
Adaptec AIC7xxx driver version: 6.1.5
aic7880: Wide Channel A, SCSI Id=7, 16/255 SCBs Channel A Target 0 Negotiation Settings
User: 40.000MB/s transfers (20.000MHz, offset 255, 16bit)
Goal: 40.000MB/s transfers (20.000MHz, offset 8, 16bit)
Curr: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) Channel A Target 0 Lun 0 Settings
Commands Queued 13557298
Commands Active 0
Command Openings 64
Max Tagged Openings 64
Device Queue Frozen Count 0
kernel: Linux-2.4.3

The size of /dev/sda5 is

    # fdisk -l /dev/sda | egrep '^[^/]|sda5'
    Disk /dev/sda: 255 heads, 63 sectors, 527 cylinders
    Units = cylinders of 16065 * 512 bytes
       Device Boot    Start       End    Blocks   Id  System
    /dev/sda5            80       399   2570368+  8e  Linux LVM

i.e. 627 PEs of 4MB each. After

pvcreate /dev/sda5; vgcreate vg0 /dev/sda5; lvcreate -n test vg0 -l 627

I have

    # vgdisplay -v
    --- Volume group ---
    VG Name               vg0
    VG Access             read/write
    VG Status             available/resizable
    VG #                  0
    MAX LV                256
    Cur LV                1
    Open LV               0
    MAX LV Size           255.99 GB
    Max PV                256
    Cur PV                1
    Act PV                1
    VG Size               2.45 GB
    PE Size               4 MB
    Total PE              627
    Alloc PE / Size       627 / 2.45 GB
    Free  PE / Size       0 / 0
    VG UUID               zmdKrC-eis5-rgWV-C5rz-TcJE-8tTI-iUUeBj

--- Logical volume ---
LV Name /dev/vg0/test
VG Name vg0
LV Write Access read/write
LV Status available
LV # 1
# open 0
LV Size 2.45 GB
Current LE 627
Allocated LE 627
Allocation next free
Read ahead sectors 120
Block device 58:0

--- Physical volumes ---
PV Name (#) /dev/sda5 (1)
PV Status available / allocatable
Total PE / Free PE 627 / 0

and lvdisplay -v /dev/vg0/test shows a mapping of

--- Logical extents ---

   LE    PV                        PE     reads      writes
   00000 /dev/sda5                 00000  278528     2        
   00001 /dev/sda5                 00001  266240     0        
   00002 /dev/sda5                 00002  266240     0        
   00003 /dev/sda5                 00003  266240     0        
   ...
   00625 /dev/sda5                 00625  266240     0        
   00626 /dev/sda5                 00626  266240     0       

For me, numbers look quite different. I have then run the following test, which reads all the 627 PEs (2.45 GB) from /dev/vg0/test and the same amount from /dev/sda5 at different block sizes. The system was otherwise almost idle when I ran this test.

    #!/bin/sh
    
    function doit() {
    
    	time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
    	time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
    	time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
    	time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
    
    	time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
    	time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
    	time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
    	time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
    }
    
    set -x
    
    doit   40128 64k
    doit   80256 32k
    doit  160512 16k
    doit  321024  8k
    doit  642048  4k
    doit 1284096  2k
    doit 2568192  1k
    doit 5136384 512

The elapsed times of the 4 runs of each test are averaged, where for each test the variations among the 4 times were less than 1% of the average time (times in seconds):

		  64k	  32k	  16k	  8k	  4k	  2k	  1k	  512

------------------------------------------------------------------------------
/dev/sda5 389.4 389.4 388.9 388.7 388.5 388.6 389.1 388.9
/dev/vg0/test 389.3 389.6 390.9 393.1 607.5 1147.2 2194.9 2195.8

I think you mean that when I read a number of blocks from /dev/sda5 I read them from the beginning of /dev/sda5 while when reading the same number of blocks from /dev/vg0/test I read from the end of the PV /dev/sda5 since the PE are allocated from the end, right.

But in my test, I read the whole /dev/sda5 and the whole LV which has allocated all PEs of /dev/sda5 to it. So about the same sectors are read, just the order is different. Do you think that can reduce the performance from 2.45GB/390s = 6.43MB/s to 2.45GB/2195s = 1.14MB/s? I find this very unlikely.

Is there any other explanation for this performance degradation?

urs

Urs, you write:

No, PEs are allocated from the start, but are only aligned with the end of the disk/partition (to give maximum room to LVM for metadata). If you have contiguous LE allocation (which you appear to have), the read order for LVM should basically be exactly the same as for a raw disk. Effectively all that LVM is doing is adding a small offset to the disk blocks to leave room for the VGDA, nothing more.

No, shouldn't be a problem with placement, because you are reading 4MB contiguous chunks (with default PE size). This should be enough to minimize seek impact.

Do you notice the sound of frantic head seeking when you are running with small block sizes?

Other than some strange seek pattern, I can't think of anything. If you look at the (non-snapshot and non-blocking because of pv_move) path of lvm_map, it boils down to:

pe_num = lvm_block / pe_size;
new_dev = pe[pe_num].dev;
new_block = (old_block % pe_size) + pe[pe_num].start;

So basically a few memory accesses and a few math operations. Nothing _should_ affect disk performance at all. With < 2k I/O you are forcing read-modify-write on pages, but this shouldn't be an issue, because you are also doing the same for the raw disk.

One thing that now comes to mind is if the LVM I/O is not aligned, because LVM PEs are aligned with the end of the PV and not the start, this _may_ cause performance problems because of misaligned read-modify-write causing extra seeking (purely speculation).

Try the following patch, which will align the PE structs to 64kB boundaries from the start of the disk, as well as making the other large VGDA structs start on 4k boundaries. I have tested the new alignment, and it works fine with existing LVM code. However, the patch is extracted from 8000+ lines of diff (yuk), so there may be a few cases where it doesn't apply cleanly or I missed something in a non-critical file. Basically, if it compiles it will work OK.

Cheers, Andreas
======================= lvm-0.9.1b7-align.diff ===========================

diff -u -u -r1.5.2.38 lvm.h
@@ -246,39 +246,34 @@
  */
 
 /* DONT TOUCH THESE !!! */
+#define LVM_VGDA_ALIGN		(4096UL - 1)
+#define LVM_VGDA_ROUND(size)	(((size) + LVM_VGDA_ALIGN) & ~LVM_VGDA_ALIGN)
+
 /* base of PV structure in disk partition */
-#define	LVM_PV_DISK_BASE  	0L
+#define	LVM_PV_DISK_BASE	0L
 
 /* size reserved for PV structure on disk */
-#define	LVM_PV_DISK_SIZE  	1024L
+#define	LVM_PV_DISK_SIZE	1024L
 
 /* base of VG structure in disk partition */
-#define	LVM_VG_DISK_BASE  	LVM_PV_DISK_SIZE
+#define	LVM_VG_DISK_BASE	LVM_VGDA_ROUND ( LVM_PV_DISK_BASE + \
+						 LVM_PV_DISK_SIZE)
 
 /* size reserved for VG structure */
-#define	LVM_VG_DISK_SIZE  	( 9 * 512L)
-
-/* size reserved for timekeeping */
-#define	LVM_TIMESTAMP_DISK_BASE	( LVM_VG_DISK_BASE +  LVM_VG_DISK_SIZE)
-#define	LVM_TIMESTAMP_DISK_SIZE	512L	/* reserved for timekeeping */
+#define	LVM_VG_DISK_SIZE	sizeof ( vg_disk_t)
 
 /* name list of physical volumes on disk */
-#define	LVM_PV_UUIDLIST_DISK_BASE ( LVM_TIMESTAMP_DISK_BASE + \
-                                    LVM_TIMESTAMP_DISK_SIZE)
+#define	LVM_PV_UUIDLIST_DISK_BASE LVM_VGDA_ROUND ( LVM_VG_DISK_BASE + \
+						   LVM_VG_DISK_SIZE)
 
 /* now for the dynamically calculated parts of the VGDA */
 #define	LVM_LV_DISK_OFFSET(a, b) ( (a)->lv_on_disk.base + \
                                    sizeof ( lv_disk_t) * b)
-#define	LVM_DISK_SIZE(pv) 	 ( (pv)->pe_on_disk.base + \
+#define	LVM_VGDA_SIZE(pv)	 ( (pv)->pe_on_disk.base + \
                                    (pv)->pe_on_disk.size)
 #define	LVM_PE_DISK_OFFSET(pe, pv)	( pe * pv->pe_size + \
-					  ( LVM_DISK_SIZE ( pv) / SECTOR_SIZE))
-#define	LVM_PE_ON_DISK_BASE(pv) \
-   { int rest; \
-     pv->pe_on_disk.base = pv->lv_on_disk.base + pv->lv_on_disk.size; \
-     if ( ( rest = pv->pe_on_disk.base % SECTOR_SIZE) != 0) \
-        pv->pe_on_disk.base += ( SECTOR_SIZE - rest); \
-   }
+					  ( LVM_VGDA_SIZE ( pv) / SECTOR_SIZE))
+#define LVM_PE_ALIGN		(65536UL / SECTOR_SIZE - 1)
 /* END default disk spaces and offsets for PVs */
 
 

diff -u -u -r1.2.2.8 vgcreate.c
@@ -101,8 +101,8 @@
    int opt_v = 0;
    int p = 0;
    int p1 = 0;
-   int max_lv = MAX_LV;
-   int max_pv = MAX_PV;
+   int max_lv = MAX_LV - 1;
+   int max_pv = MAX_PV - 1;
    int min_pv_index = 0;
    ulong max_pv_size = 0;
    ulong min_pv_size = -1;

diff -u -u -r1.5.2.17 liblvm.h
@@ -99,13 +99,9 @@
 
 extern struct config_file *config_file;
 
-#define	SECTOR_SIZE	512
 #define	BLKGETSIZE	_IO(0x12,96)	/* return device size */ 
 #define	BLKRASET	_IO(0x12,98)	/* Set read ahead */
 #define	EXT2_SUPER_MAGIC	0xEF53
-#ifndef BLOCK_SIZE
-#define	BLOCK_SIZE	1024
-#endif
 
 #define	LVM_ID		"HM"      /* Identifier PV (id in pv_t) */
 #define	EXPORTED	"PV_EXP"  /* Identifier exported PV (system_id in pv_t) */

diff -u -u -r1.2.2.2 pv_read_pe.c
@@ -58,8 +57,7 @@
    *pe = NULL;
 
    size = pv->pe_total * sizeof ( pe_disk_t);
-   if ( size + pv->pe_on_disk.base > 
-        LVM_DISK_SIZE ( pv)) {
+   if ( size + pv->pe_on_disk.base > LVM_VGDA_SIZE ( pv)) {
       ret = -LVM_EPV_READ_PE_SIZE;
       goto pv_read_pe_end;
    }

diff -u -u -r1.1.2.1 pv_release_pe.c
@@ -82,8 +82,7 @@
             goto pv_release_pe_end;
          }
          pe_index = ( vg->lv[l]->lv_current_pe[p].pe - 
-                      LVM_DISK_SIZE ( vg->pv[pv_num]) /
-                      SECTOR_SIZE) /
+                      LVM_VGDA_SIZE ( vg->pv[pv_num]) / SECTOR_SIZE) /
                       vg->pe_size;
          debug ( "pv_release_pe -- pv_name: %s  pe: %lu  sector: %lu\n",
                   vg->pv[pv_num]->pv_name,

diff -u -u -r1.2.2.1 pv_write_pe.c
@@ -50,7 +50,7 @@
         pv == NULL) ret = -LVM_EPARAM;
    else {
       size = pv->pe_total * sizeof ( pe_disk_t);
-      if ( size + pv->pe_on_disk.base > LVM_DISK_SIZE ( pv))
+      if ( size + pv->pe_on_disk.base > LVM_VGDA_SIZE ( pv))
          ret = -LVM_EPV_WRITE_PE_SIZE;;
       if ( ( pv_handle = open ( pv_name, O_WRONLY)) == -1)
          ret = -LVM_EPV_WRITE_PE_OPEN;

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Yes, I found that out after my posting by writing a 32 Mb ( = 8 PEs) pattern to the beginning of /dev/vg0/test and searching for it in /dev/sda5. In my case the offset is of the first PE is 4352 sectors, since /dev/sda5 has 627 * 4 MB + 4352 * 512 bytes. This small offset shouldn't have an impact on the performance when reading 2.45 GB.

No, sounds very similar and only very moderate head seeking (probably from syslogd).

What has that all to do with pages? Isn't it the hardsect_size that is relevant here? And why is < 2k the relevant limit? If the page size was relevant this is 4k. If the hardsect_size is relevant, this is 1k in 2.4.3 (after initialization, but can grow when mounting a FS with larger block size, due to the bug I reported on this list) and if I read the src in 0.9.1b7 correctly Andrea's patch makes the hardsect_size equal to the maximum sector size of all PVs, i.e. 512 bytes with most disks.

I don't understand, what you're saying here (and above). I was *reading* with dd if=/dev/vg0/test of=/dev/null. So why is there an RMW involved?

What does happen, when I do a write(2) of a 512-byte block to /dev/vg0/test, when the hardsect_size of the PV /dev/sda5 is 512 bytes and the hardsect_size of the LV /dev/vg0/test is 1k? Does the kernel read 1k from LVM, modify 512 bytes and (later, when syncing) write 1k back to LVM? If so, the fact that hardsect_size of the LV is larger than the physical sector size could explain the performance impact. The kernel would have to modify only parts of a LV block, so (I assume) it would RMW. With

    hardsect_size == sector size == size of the write(2) call

the kernel could probably just write to LVM without prior reading. But this would still not explain the performance degradation in my tests where I was *reading* from the LV.

Could you explain the interaction of the kernel, LVM and the physical device a little bit or give a pointer to some text?

Does it align to the start of the disk (as you write) or to the start of the PV? And is this a patch that will go into further releases anyway or is it just to find my reported performance problem? Since alignment is changed, I think I will have to recreate the PV, VG and LV with pvcreate, vgcreate and lvcreate, right? The patched tools and kernel would not work with my existing LV, would they?

I don't expect this change in alignment to cause a change in performance. For testing, I have written my own very simple version of dd which can seek (not only skip) on the input file (I called it di since it understands only if= but not of=, it always writes to stdout). With this I have read from /dev/sda5 exactly the same sectors that /dev/vg0/test allocates (the first PE in vg0 starts at sector 4352 and /dev/vg0/test allocates PEs from vg0 contigously starting at PE no 0, see my last mail):

    # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 >/dev/null
    131072+0

    real    0m9.345s
    user    0m0.180s
    sys     0m1.250s
    # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 seek=4352 >/dev/null
    131072+0

    real    0m8.955s
    user    0m0.130s
    sys     0m1.570s
    # time ~urs/src/di bs=512 count=131072 if=/dev/vg0/test >/dev/null
    131072+0

    real    0m54.611s
    user    0m0.110s
    sys     0m2.480s

Anyway, I will try your patch, probably today. I have not applied patches from the lvm-0.9... to the kernel before. AFAICS, I have to apply your patch to the lvm-0.9.1beta7 package, then ./configure; cd PATCHES; make and then apply lvm-0.9.1_beta7-2.4.3.patch to my kernel src, right?

Also, does lvm-0.9.1_beta7-2.4.3.patch affect files other than lvm-mod.o, i.e. do I have to restart my kernel or only reload the LVM module? I have already made the lvm-0.9.1_beta7-2.4.3.patch file and see lvm.h is changed, but haven't already checked where it is included.

urs

Urs, you write:

Sorry, too sleepy I guess. Obviously don't need read-modify-write when only reading pages.

Ok, the chain of events is as follows (hopefully correct):

Non-LVM read LVM read
------------ --------
sys_read sys_read
block_read block_read
ll_rw_block ll_rw_block
submit_bh submit_bh
generic_make_request generic_make_request
lvm_make_request_fn lvm_map
__make_request __make_request

The rest should be the same because you have the same IDE driver below.

You are correct, it aligns to the start of the PV, and not the disk.

I'm hoping this patch will make it into the stock LVM because it aligns all of the large VGDA structs to PAGE_SIZE (at least 4k pages), and the PE alignment matches MD RAID device alignment.

Yes, you will need to re-create the PV and VG and LV for this to take effect. However, the kernel currently knows nothing at all about on-disk layout of VGDA data. Also, the tools all use the VGDA locations stored in the PV header (they are written at pvcreate and vgcreate time), so even if you create your VG/LV with this patch and it doesn't make it into the stock kernel, it is 100% compatible with unpatched tools.

Neither do I. Just something that I was thinking about when I had writing on my mind, not reading.

Very strange. I just tested the same on my system, and LVM ends up _faster_ than raw disk access in each of the 3 tests I did:

dd if=<dev> of=/dev/null bs=512 count=131072

LVM /dev/lvtest/vgtest: 11.099s, 11.393s, 10.785s
IDE /dev/hda9:          12.073s, 11.822s, 11.843s

This may have something to do with how my LV is set up. The first 5 PEs are actually closer to the end of the PV, and then the remaining PEs are contiguous from the start of the PV.

Yes, but you really don't need to change the kernel code to use my patch. The VGDA alignment is purely a user-space issue. The kernel never does I/O to the VGDA - it is all read in user-space and passed to the kernel via ioctl.

Actually, I think you should be OK with only rebuilding the LVM module. However, you should be able to see from the patch itself if it affects anything outside lvm*. AFAIK, noting includes lvm.h in the kernel that would be affected.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

I'm going to stick my head in here and say thanks for that info. I really need to _read_ the code. *sigh*

okay, I only have the option of testing this on an old IDE drive, and the times where within 7% of eachother (LVM being the slower of the two on all runs). it's a bigger difference then I'd like to see, but it varies with runs, and the box is low on memory and being used at the same time. the only thing they're really saying is that LVM isn't a bottleneck.

not that it'll help much in Urs' case though. :/

Terje

Sorry, I hadn't time to test your patch earlier. I wanted to try it yesterday but lvm-0.9.1-beta7 with your patch applied doesn't compile. There are undefined symbols

    $ make
    ...
    gcc  -L../tools/lib -llvm-10  -o vgchange vgchange.o 
    vgchange.o: In function `main':
    /home/urs/tmp/LVM/0.9.1_beta7-dilger/tools/vgchange.c:635: undefined reference to `LVM_PE_ON_DISK_BASE'
    /home/urs/tmp/LVM/0.9.1_beta7-dilger/tools/vgchange.c:638: undefined reference to `LVM_DISK_SIZE'

When I have done my performance tests I had a stock linux-2.4.3 and user space tools from lvm-0.9 running. I now repeated the tests with lvm-0.9.1-beta7, both user space tools and kernel (recreated all PVs, VGs and LVs). However, the results with these versions are the same.

I also did some tests with another hard disk drive. My previous tests were with a IBM DCAS-34330W SCSI-U2W. I now did some tests with a NEC DSE2100S SCSI2 drive. Again, LVM is much slower with the standard 512 byte block size:

    # pvcreate /dev/sdc4
    pvcreate -- physical volume "/dev/sdc4" successfully created

    # vgcreate vg1 /dev/sdc4
    vgcreate -- INFO: using default physical extent size 4 MB
    vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
    vgcreate -- doing automatic backup of volume group "vg1"
    vgcreate -- volume group "vg1" successfully created and activated

    # lvcreate -n test vg1 -l31
    lvcreate -- doing automatic backup of "vg1"
    lvcreate -- logical volume "/dev/vg1/test" successfully created

    # time dd if=/dev/sdc4 of=/dev/null
    257040+0 records in
    257040+0 records out

    real    0m37.205s
    user    0m0.840s
    sys     0m3.780s
    # time dd if=/dev/vg1/test of=/dev/null
    253952+0 records in
    253952+0 records out

    real    2m52.368s
    user    0m1.330s
    sys     0m12.390s
    # time dd if=/dev/vg1/test of=/dev/null bs=8k
    15872+0 records in
    15872+0 records out

    real    0m43.501s
    user    0m0.050s
    sys     0m4.600s
    # time dd if=/dev/vg1/test of=/dev/null bs=16k
    7936+0 records in
    7936+0 records out

    real    0m40.806s
    user    0m0.070s
    sys     0m4.290s
    # time dd if=/dev/vg1/test of=/dev/null bs=32k
    3968+0 records in
    3968+0 records out

    real    0m38.777s
    user    0m0.030s
    sys     0m4.290s

What I find surprising is that the system time increases to 12.390s with LVM compared to 3.780s on /dev/sdc4. That is approx. 34ms of overhead per 512-block which I find a lot for the simple mapping of block numbers. But this still does not explain the much longer real elapsed time.

Is there another patch (maybe to produce some debugging output) I could try to find the reason for this performnace hit?

And am I really the only one who sees this? I would really be interested to hear from some more people about performance for their LVM.

urs

I think one problem is that your "benchmarks" are not very realistic. In real world use the overhead of LVM is very small. I would respectfully suggest that you try something like bonnie++ before getting too worried about the performance of LVM on your machine - unless you spend most of your day doing "dd" to your disks :-)

Here are some bonnie++ benchmarks of a disk partition and and LV (same disk) using reiserfs* on my machine:

Disk Partition:
Version 1.00g ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
tyke           496M 10462  98 20675  53  7699  20 11707  96 17808  16 257.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  8426 100 +++++ +++ 12622 100  8031 100 +++++ +++  9635 100

Logical Volume:
Version 1.00g ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
tyke           496M 10577 100 20631  52  7739  20 11676  96 17622  17 260.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  8521 100 +++++ +++ 12661  99  8007 100 +++++ +++  9555  99

patrick

* ext2 is comparable, just the numbers are different.

The reason I tested dd on the partition and on the LV is that I thought when the performance of the block device is bad (like it is on my LV) this will have an impact on a mounted file system on that LV. If also thought, the performance of dd if=/dev/vg0/test will be approximately equal to dd if=/mnt/foo od md5sum foo where foo is large file on the LV.

I've done some tests with a file system on /dev/sda5 and the on /dev/vg0/test (which is a LV on /dev/sda5) and I am really surprised:

First, I repeated my previous tests:

						         $dev

                                                /dev/sda5    /dev/vg0/test
---------------------------------------------+-------------+----------------
dd if=$dev of=/dev/null bs=512 count=1048576 |    71.2s    |   122.7s
dd if=$dev of=/dev/null bs=64k count=8192    |    71.5s    |    71.7s

Then I made a ext2 filesystem with 1K blocksize on /dev/sda5, mounted it to /mnt, copied a 512MB random file to it, md5sum'med that fie and dd'ed that file to /dev/null. Then I recreated /dev/vg0/test and did the same tests and that block device:

---------------------------------------------+-------------+----------------

cp -a /usr/local/foo /mnt                    |    96.6s    |   123.4s
md5sum /mnt/foo                              |    89.8s    |    92.4s
dd if=/mnt/bar of=/dev/null bs=512           |    71.8s    |    72.1s
dd if=/mnt/bar of=/dev/null bs=64k           |    71.7s    |    72.5s

When I use a 4K blocksize on the ext2 filesystem the times for /dev/sda5 and /dev/vg0/test are even closer.

Result: Although the block device /dev/vg0/test is significantly slower than /dev/sda5 (where /dev/vg0/test is located on /dev/sda5) the *read* performance of the file system on top the block device is roughly the same. This is probably not some effect of buffer caching since I did a blockdev --flushbufs $dev before each test, and /usr/local/bar is 512MB in size which is twice as large as the main memory size on that machine.

However, writing to an LV is still much slower than to a normal partition.

urs

OK, I have now reproduced this problem on my SMP test system. This is on a 2xP5-200 + 128MB RAM, Advansys ABP940U SCSI, and (old) IBM 0662-S12 and IBM 0663-L12 1GB SCSI drives. I do NOT have this problem on my laptop AFAICS (P2-500 + 128MB RAM, Intel PIIX4 IDE, IBM-DARA-218000).

Doing similar tests to the above (dd if=$dev of=/dev/null, bs=512 count=100k) I get the following results (50MB I/O, 100MB LV, Linux 2.2.18 + LVM 0.9):

/dev/sda = real 12.290s user 0.110s sys 2.890s
/dev/sda = real 12.269s user 0.100s sys 2.900s
/dev/sda1 = real 12.317s user 0.140s sys 2.980s
/dev/sda1 = real 12.251s user 0.130s sys 2.900s /dev/vgtest/lvtest = real 66.909s user 0.440s sys 10.420 *** 5x as slow /dev/vgtest/lvtest = real 70.135s user 0.520s sys 11.390 *** 5x as slow

/dev/sdb = real 11.614s user 0.060s sys 3.090s
/dev/sdb = real 11.597s user 0.120s sys 2.910s
/dev/sdb1 = real 11.857s user 0.150s sys 4.020s
/dev/sdb1 = real 11.749s user 0.110s sys 2.800s /dev/vgtest3/lvtest = real 63.552s user 0.510s sys 10.290s *** 6x as slow /dev/vgtest3/lvtest = real 62.354s user 0.610s sys 10.020s *** 6x as slow

/dev/hda6 = real 10.848s user 0.180s sys 4.380s
/dev/hda6 = real 10.803s user 0.210s sys 4.300s /dev/vgtest2/lvtest = real 15.258s user 0.480s sys 9.500s *** 50% slower /dev/vgtest2/lvtest = real 15.827s user 0.360s sys 9.830s *** 50% slower

I'm not sure if it is an SMP or SCSI issue, or what. Urs, do you also have an SMP system? Maybe there are locking slowdowns under SMP and not under UP?

The problem is definitely more noticable on the SCSI disks, but at least on the IDE disk the increased wall-clock time is wholly accounted for by increased system time. On the SCSI disks, it would guess that "dd" is waiting for I/O completions under LVM, and not on the raw disk???

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

This is similar to the worst case numbers I have measured.

No. It's a single Pentium II, 333MHz, kernel 2.4.3 with lvm-0.9.1_beta7 compiled without SMP support:

isnogud:urs$ grep SMP /usr/src/linux-2.4.3-ut/.config # CONFIG_SMP is not set

The SCSI controller is an Adaptec AIC 7880 on an ASUS P2L97-S mainboard (onboard). The SCSI disks are

Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: DCAS-34330W Rev: S65A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: DCAS-34330W Rev: S65A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: NEC Model: DSE2100S Rev: 0306
Type: Direct-Access ANSI SCSI revision: 02

i.e. /dev/sda and /dev/sdb are UW-SCSI and /dev/sdc is a SCSI-2 disks.

How could I check? And do you have an idea, why a file system on top of such a slow LV is, if at all, only somewhat slower?

urs

I haven't got a UP kernel to test right now to confirm, but my SMP purely IDE system (specs follow) shows rough parity at 512 byte reads and 2x FASTER on the LV at 4096 byte blocks. When I switch to a striped LV I see parity at both sizes, with the LV being slightly faster. For giggles, I also checked out a RAID-1 (mirror) set as well, which smoked everything, as it should.

Device Block Real User Sys

/dev/md0                512     2.080s        0.110s        0.660s
/dev/hda1               512     5.079s        0.140s        2.040s
/dev/mitethe/nostripe   512     4.795s        0.300s        2.380s
/dev/mitethe/stripe     512     4.350s        0.090s        2.530s

/dev/md0                4096   16.482s        0.060s        6.090s
/dev/mitethe/nostripe   4096   26.278s        0.010s        7.270s
/dev/mitethe/stripe     4096   32.803s        0.170s       12.020s
/dev/hda1               4096   40.947s        0.180s       13.590s

Now comes the part where I admit we're comparing apples to oranges:

Linux 2.4.3
LVM 0.9.1_beta7
2x Pentium III 600MHz
Tyan Tiger 133 Mobo (running at 133 FSB)

Maxtor 53073U6 30GB ATA/66 hda
WDC WD300AB-00BVA0 30GB ATA/66 hdc

/dev/hda:

 multcount    = 16 (on)
 I/O support  =  1 (32-bit)
 unmaskirq    =  1 (on)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)

geometry = 59554/16/63, sectors = 60030432, start = 0

/dev/hdc:

 multcount    = 16 (on)
 I/O support  =  1 (32-bit)
 unmaskirq    =  1 (on)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)

geometry = 58168/16/63, sectors = 58633344, start = 0

$ fortune -m Kellen

Sorry, I must have missed some parts of the patch for the user-space tools. Even so, I don't think the patch will help.

One other person had a similar problem. It was because they were using PIO mode instead of DMA mode on their (IDE) drive. However, I see you are using a SCSI disk, so I don't know what to suggest...

What SCSI controller are you using?

I have never seen more than a few percent difference between LVM and non-LVM performance.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

It is an Adaptec AIC 7880 which is onboard on my
ASUS P2L97-S mainboard (see some previous posting where I did describe my hardware in more detail).

Do you have some further suggestion what I could try to find the reason for this performance hit?

urs

Maybe dd is reading in 512 byte blocks on /dev/sda5 and 4096 bytes on /dev/vg0/test ?

(I'm not sure if it is, but these blocksizes are the default blocksize on each device and you only told dd to "read 100000 blocks" ... this ratio also explains the time difference of 8 times)

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

I've checked it. The man page says if no bs is given, dd defaults to 512 bytes on any device. I've also checked it with dd|wc.

The performance numbers I've observed seem to dependant on caching.

urs