Avoiding I/O bottlenecks between VM's
I am using KVM 69 and kernel 2.6.25.17 to host several VM's in a large server. So far, everything has been great. Except I'm adding a Windows Server VM that will run a SQL Server database. A few times I've noticed that I/O becomes a bottleneck for the VM and Windows VM freezes for a few seconds. Oh well, no biggie. Except, every so often, these I/O bottlenecks start to affect other VM's and they freeze as well for a few seconds. I don't really care of one VM does so much I/O that it freezes itself temporarily. I just don't want I/O bottlenecks on one VM to affect other VM's.
My questions are:
1. Is this a problem anyone else has experienced and has it been fixed in a later KVM release?
2. I'm using the CFQ scheduler. Would the deadline scheduler do a better job?
3. Any other suggestions to improve this problem?
Hi Alberto,
Quoted 12 lines · Alberto Treviño
I am using KVM 69 and kernel 2.6.25.17 to host several VM's in a large server. So far, everything has been great. Except I'm adding a Windows Server VM that will run a SQL Server database. A few times I've noticed that I/O becomes a bottleneck for the VM and Windows VM freezes for a few seconds. Oh well, no biggie. Except, every so often, these I/O bottlenecks start to affect other VM's and they freeze as well for a few seconds. I don't really care of one VM does so much I/O that it freezes itself temporarily. I just don't want I/O bottlenecks on one VM to affect other VM's.
My questions are:
Quoted 7 lines
1. Is this a problem anyone else has experienced and has it been fixed in a later KVM release?
2. I'm using the CFQ scheduler. Would the deadline scheduler do a better job?
3. Any other suggestions to improve this problem?
Are you using filesystem backed storage for the guest images or direct block device storage? I assume there's heavy write activity on the guests when these hangs happen?
ext3 with ordered data mode has latency issues on fsync.
Quoted 3 lines · you
Are you using filesystem backed storage for the guest images or direct block device storage? I assume there's heavy write activity on the guests when these hangs happen?
Yes, they happen when one VM is doing heavy writes. I'm actually using a whole stack of things:
OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-encrypted partition. Fun debugging that, heh?
In trying to figure out the problem, I tried to reconfigure DRBD to use Protocol B instead of C. However, it failed to make the switch and both nodes disconnected so now I have a split-brain. In try to fix the split brain I'm taking down on one node all the VM's one by one, copying the VM drives from one node to the next, and starting up on the other node (old- fashioned migration). Yes, I'm having *lots* of fun! Perfect way to end the week!
So, any ideas on how to solve the bottleneck? Isn't the CFQ scheduler supposed to grant every processes the same amount of I/O? Is there a way to change something in proc to avoid this situation?
Signature · 3 lines
Alberto Treviño BYU Testing Center Brigham Young University
Quoted 10 lines · Alberto Treviño
Quoted 3 lines · you
Are you using filesystem backed storage for the guest images or direct block device storage? I assume there's heavy write activity on the guests when these hangs happen?
Yes, they happen when one VM is doing heavy writes. I'm actually using a whole stack of things:
OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-encrypted partition. Fun debugging that, heh?
a not-so-wild guess might be the inter-node locking needed by any cluster FS. you'd do much better using just CLVM or EVMS-Ha
if it's a single box, it would be interesting to compare with ext3
Quoted 3 lines
So, any ideas on how to solve the bottleneck? Isn't the CFQ scheduler supposed to grant every processes the same amount of I/O? Is there a way to change something in proc to avoid this situation?
i don't think CFQ can do much to alleviate the heavy lock-dependency of a cluster FS
Signature · 1 line
Javier
Quoted 11 lines · Javier Guerra
Quoted 10 lines · Alberto Treviño
Quoted 3 lines · you
Are you using filesystem backed storage for the guest images or direct block device storage? I assume there's heavy write activity on the guests when these hangs happen?
Yes, they happen when one VM is doing heavy writes. I'm actually using a whole stack of things:
OCFS2 on DRBD (Primary-Primary) on LVM Volume (continuous) on LUKS-encrypted partition. Fun debugging that, heh?
Heh. Lots of variables there.
Quoted 7 lines
a not-so-wild guess might be the inter-node locking needed by any cluster FS. you'd do much better using just CLVM or EVMS-Ha
if it's a single box, it would be interesting to compare with ext3
Quoted 2 lines
So, any ideas on how to solve the bottleneck? Isn't the CFQ scheduler supposed to grant every processes the same amount of I/O?
Yes, but if the filesystem on top is at fault, the IO scheduler can't help (this is the case with ext3 ordered mode and fsync latency, which could last for hundreds of seconds last time I checked).
Quoted 5 lines
Quoted 2 lines
Is there a way to
change something in proc to avoid this situation?i don't think CFQ can do much to alleviate the heavy lock-dependency of a cluster FS
Perhaps isolate the problem by having the guest images directly on partitions first (or ext3 with writeback data mode).