Skip to contentSkip to conversation

Suspicious performance gaps in LVM volume over 2 block devices (internally RAID 5).

linux-lvm M4N - Arjan Tijms 1 participant 1 message Jan 26, 2009

lore
Share & tools
lore
#0 lore raw

Hi,

We're building a system on Linux (kernel 2.6.26) using LVM 2.02.39, for which we stripe 2 block devices into 1 volume. Each device is internally build using a RAID 5 on a hardware RAID controller (Areca 1680Ix-12/4GB RAM). This RAID controller is rather fast (sporting a 1.2Ghz IOP 348). To each Areca we connected 6 Mtron pro 7535's. The stripe set thus spans 12 physical disks over 2 raid controllers.

Initially it seemed to work just fine, but when running the easyco benchmark (see http://easyco.com/), suspicious gaps in the performance figures appear for certain block sizes:

EasyCo# ./bm-flash /ssd/ssd-striped/test.txt
Filling 4G before testing ... 4096 MB done in 1 seconds (4096 MB/sec).

Read Tests:

Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW

     |               |               |               
512B | 21856   10.6M | 59945   29.2M | 59381   28.9M 
  1K | 32945   32.1M | 59773   58.3M | 59252   57.8M 
  2K | 32462   63.4M | 59611  116.4M | 59081  115.3M 
  4K | 31398  122.6M | 59707  233.2M | 58946  230.2M 
  8K | 28847  225.3M | 58294  455.4M | 57684  450.6M 
 16K | 24965  390.0M | 57934  905.2M | 57004  890.6M 
 32K | 19393  606.0M | 52848 1651.5M | 52520 1641.2M 
 64K | 13466  841.6M | 46626    2.8G | 46533    2.8G 
128K |  8541 1067.6M | 25962    3.1G | 27364    3.3G 256K |  7777    1.8G | 13876    3.3G | 13876    3.3G 512K |  4377    2.1G |  6942    3.3G |  6943    3.3G 
  1M |  2356    2.3G |  3472    3.3G |  3473    3.3G 
  2M |  1292    2.5G |  1735    3.3G |  1732    3.3G 
  4M |   728    2.8G |   847    3.3G |   864    3.3G 

Write Tests:

Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW

     |               |               |               
512B | 31208   15.2M | 30125   14.7M | 22332   10.9M 
  1K | 29351   28.6M | 29055   28.3M | 27419   26.7M 
  2K | 24413   47.6M | 28681   56.0M | 27832   54.3M 
  4K | 28874  112.7M | 25613  100.0M | 25060   97.8M 
  8K |    27  216.0K |   925    7.2M | 12215   95.4M 
 16K | 18230  284.8M | 20401  318.7M |    13  209.5K 
 32K | 12462  389.4M |    35 1148.7K | 13666  427.0M 
 64K |    10  684.7K |  8437  527.3M |     6  441.5K 
128K |  7030  878.7M |     2  332.7K 256K |  4523 1130.8M |     3  947.1K 512K |  4222    2.0G |     5    2.5M 
  1M |  2085    2.0G |     2    2.0M 
  2M |  1495    2.9G |    12   25.1M 
  4M |   768    3.0G |   122  488.0M |   143  573.5M

As can be seen from this report, testing with 1 thread shows no gaps, but does show numbers which are clearly too high for writing. When more threads are added to the test, gaps start to appear.

We have extensively tested the individual devices (i.e. 6 disks, 1 controller, no LVM) and there doesn't seem to be any problem. I wonder if someone could explain the behavior we're seeing when using LVM to stripe two otherwise correctly working devices into 1 volume.

Thanks in advance,
Arjan

Signature · 1 line
It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good.