kvm-27 vs 28 I/O speed

DISCUSSION Dave Hansen 4 participants 25 messages Jul 6, 2007 – Jul 18, 2007 14 messages not in the archive KVM

Share & tools

Dave Hansen

author

2007-07-06 19:07 UTC #0 lore raw

I've noticed that some of my tests run *MUCH* slower in kvm-28 than in
27. I'm sure that wall time is pretty wonky in the guests, but it is much slower in real-world time as well.

Here's a little test to create a 32MB zeroed file with dd. Here it is from kvm-27 (this took ~5.5 seconds on my wristwatch):
33554432 bytes transferred in 0.052050 seconds (644657845 bytes/sec) 33554432 bytes transferred in 0.062933 seconds (533176451 bytes/sec)

Here's the same thing from kvm-28 (~80 seconds on my wristwatch):
33554432 bytes transferred in 38.607065 seconds (869127 bytes/sec) 33554432 bytes transferred in 22.274318 seconds (1506418 bytes/sec)

Same host kernel, same kvm kernel modules (from kvm-28) same guest kernel, same command-line options, same disk image.

Any ideas what is going on?

-- Dave

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Avi Kivity

2007-07-08 08:14 UTC #1 lore raw

Quoted 9 lines · Dave Hansen

I've noticed that some of my tests run *MUCH* slower in kvm-28 than in
27. I'm sure that wall time is pretty wonky in the guests, but it is much slower in real-world time as well.
Here's a little test to create a 32MB zeroed file with dd. Here it is from kvm-27 (this took ~5.5 seconds on my wristwatch):
33554432 bytes transferred in 0.052050 seconds (644657845 bytes/sec) 33554432 bytes transferred in 0.062933 seconds (533176451 bytes/sec)

Quoted 9 lines

Here's the same thing from kvm-28 (~80 seconds on my wristwatch):
33554432 bytes transferred in 38.607065 seconds (869127 bytes/sec) 33554432 bytes transferred in 22.274318 seconds (1506418 bytes/sec)
Same host kernel, same kvm kernel modules (from kvm-28) same guest kernel, same command-line options, same disk image.
Any ideas what is going on?

Is this repeatable? I don't see anything in kvm-27..kvm-28 that warrants such a regression.

Things to check:

- effect of pinning the vm onto one cpu (with 'taskset')
- does any counter in kvm_stat behave differently

If you are using qcow, maybe the effect is due to the first test hitting a hole and the second being forced to read from disk. I recommend doing performance tests from a real partition or volume.

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 46909CFD.9050806-atKUWr5tajBWk0Htik3J/[email protected]

Dave Hansen

author

2007-07-09 20:33 UTC #3 lore raw

Quoted 14 lines · Avi Kivity

Quoted 13 lines · Dave Hansen
I've noticed that some of my tests run *MUCH* slower in kvm-28 than in
27. I'm sure that wall time is pretty wonky in the guests, but it is much slower in real-world time as well.
Here's a little test to create a 32MB zeroed file with dd. Here it is from kvm-27 (this took ~5.5 seconds on my wristwatch):
33554432 bytes transferred in 0.052050 seconds (644657845 bytes/sec) 33554432 bytes transferred in 0.062933 seconds (533176451 bytes/sec)
Here's the same thing from kvm-28 (~80 seconds on my wristwatch):
33554432 bytes transferred in 38.607065 seconds (869127 bytes/sec) 33554432 bytes transferred in 22.274318 seconds (1506418 bytes/sec)

Quoted 8 lines

Quoted 5 lines
Same host kernel, same kvm kernel modules (from kvm-28) same guest kernel, same command-line options, same disk image.
Any ideas what is going on?
Is this repeatable? I don't see anything in kvm-27..kvm-28 that warrants such a regression.

Right now, it's completely repeatable

Quoted 3 lines

Things to check:
- effect of pinning the vm onto one cpu (with 'taskset')

Tried this. No effect that I can see at all.

Quoted 1 line

- does any counter in kvm_stat behave differently

Nothing really stands out. The most strange thing to me is that the counters are very stationary on the slow one. It doesn't look like it is spinning on something, just that it is idle. Its counters during the dd look very much like the guest does when it is idle:

the slow one:

dave@elm3b173:~$ KVM=28 HDA=qemu.raw ./run-qemu -snapshot
idle:
 efer_reload    156984     800
 exits          395676     800
 halt_exits       7165     100
 invlpg              0       0
 io_exits       123487     700
 irq_exits        1406       0
 irq_window        817       0
 light_exits    238692       0
 mmio_exits      24716       0
 pf_fixed       130354       0
 pf_guest        57309       0
 request_irq         0       0
 signal_exit       781       0
 tlb_flush        2267       0

dd'ing (mosly idle?):

 efer_reload    197669     856
 exits          455341     856
 halt_exits      11568     100
 invlpg              0       0
 io_exits       159006     740
 irq_exits        2125      14
 irq_window        975       2
 light_exits    257672       0
 mmio_exits      24716       0
 pf_fixed       146646       0
 pf_guest        59596       0
 request_irq         0       0

signal_exit 1383 14
tlb_flush 2455 0

burst during dd:
efer_reload 205387 854
exits 465166 2953
halt_exits 12471 98
invlpg 0 0
io_exits 165605 733
irq_exits 2341 30
irq_window 985 2
light_exits 259779 2099
mmio_exits 24716 0
pf_fixed 148731 2082
pf_guest 59596 0
request_irq 0 0
33554432 bytes transferred in 20.837575 seconds (1610285 bytes/sec)

the fast one:

$ KVM=27 HDA=qemu.raw ./run-qemu -snapshot
idle:
 efer_reload    164333     800
 exits          416798     800
 halt_exits       4570     100
 invlpg              0       0
 io_exits       132855     700
 irq_exits         838       0
 irq_window       2037       0
 light_exits    252465       0
 mmio_exits      24716       0
 pf_fixed       130325       0
 pf_guest        57275       0
 request_irq         0       0
 signal_exit       127       0
 tlb_flush        1864       0

during dd:

 efer_reload    243917   29963
 exits          523608   34205
 halt_exits       6766       0
 invlpg              0       0
 io_exits       207716   28730
 irq_exits        1079      52
 irq_window       4525    1232
 light_exits    279695    4235
 mmio_exits      24716       0
 pf_fixed       148656    4177
 pf_guest        59560       0
 request_irq         0       0
 signal_exit       154       7
 tlb_flush        2043       8

33554432 bytes transferred in 0.100797 seconds (332891147 bytes/sec)

Quoted 3 lines

If you are using qcow, maybe the effect is due to the first test hitting a hole and the second being forced to read from disk. I recommend doing performance tests from a real partition or volume.

I switched over to a raw image, and used -snapshot. I don't have the disk space to give them their own partitions right now.

-- Dave

Avi Kivity

2007-07-10 05:44 UTC #4 lore raw

Quoted 38 lines · Dave Hansen

Quoted 15 lines · Avi Kivity
Quoted 13 lines · Dave Hansen
I've noticed that some of my tests run *MUCH* slower in kvm-28 than in
27. I'm sure that wall time is pretty wonky in the guests, but it is much slower in real-world time as well.
Here's a little test to create a 32MB zeroed file with dd. Here it is from kvm-27 (this took ~5.5 seconds on my wristwatch):
33554432 bytes transferred in 0.052050 seconds (644657845 bytes/sec) 33554432 bytes transferred in 0.062933 seconds (533176451 bytes/sec)
Here's the same thing from kvm-28 (~80 seconds on my wristwatch):
33554432 bytes transferred in 38.607065 seconds (869127 bytes/sec) 33554432 bytes transferred in 22.274318 seconds (1506418 bytes/sec)
Quoted 9 lines
Quoted 6 lines
Same host kernel, same kvm kernel modules (from kvm-28) same guest kernel, same command-line options, same disk image.
Any ideas what is going on?
Is this repeatable? I don't see anything in kvm-27..kvm-28 that warrants such a regression.
Right now, it's completely repeatable
Quoted 4 lines
Things to check:
- effect of pinning the vm onto one cpu (with 'taskset')
Tried this. No effect that I can see at all.

Quoted 8 lines

Quoted 2 lines
- does any counter in kvm_stat behave differently
Nothing really stands out. The most strange thing to me is that the counters are very stationary on the slow one. It doesn't look like it is spinning on something, just that it is idle. Its counters during the dd look very much like the guest does when it is idle:

What does 'top' on the host show in both cases?

There was a change in kvm userspace that affected IDE. Can you try kvm-27 userspace with a kvm-28 kernel module? If that's slow, then the only recourse is bisecting.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 46931CC9.8060106-atKUWr5tajBWk0Htik3J/[email protected]

Dave Hansen

author

2007-07-11 18:54 UTC #6 lore raw

Quoted 1 line · Avi Kivity

What does 'top' on the host show in both cases?

On the fast one (kvm-27) I just see the dd as a flash in top. It doesn't stay for long. The VM isn't very responsive during the dd.

On the slow (kvm-28) one, I see the dd in top creeping along, using ~2% of the CPU. The guest is responsive during this time

Quoted 2 lines

There was a change in kvm userspace that affected IDE. Can you try kvm-27 userspace with a kvm-28 kernel module?

That's actually what I've been doing the whole time. Sorry, my initial problem description must not have been very clear. I've been using the kvm-28 modules the entire time, and only varying the userspace.

The IDE changes do look suspect.

-- Dave

Avi Kivity

2007-07-12 05:37 UTC #7 lore raw

Quoted 16 lines · Dave Hansen

Quoted 2 lines · Avi Kivity
What does 'top' on the host show in both cases?
On the fast one (kvm-27) I just see the dd as a flash in top. It doesn't stay for long. The VM isn't very responsive during the dd.
On the slow (kvm-28) one, I see the dd in top creeping along, using ~2% of the CPU. The guest is responsive during this time
Quoted 3 lines
There was a change in kvm userspace that affected IDE. Can you try kvm-27 userspace with a kvm-28 kernel module?

Quoted 7 lines

That's actually what I've been doing the whole time. Sorry, my initial problem description must not have been very clear. I've been using the kvm-28 modules the entire time, and only varying the userspace.
The IDE changes do look suspect.

Can you confirm it by backing out that one patch?

I don't see how it can cause this slowdown, except by causing a different driver to be loaded or perhaps with different settings.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 4695BE06.6060609-atKUWr5tajBWk0Htik3J/[email protected]

Dave Hansen

author

2007-07-12 18:10 UTC #9 lore raw

Quoted 1 line · Avi Kivity

Can you confirm it by backing out that one patch?

Do you know the git commit id by chance?

-- Dave

Avi Kivity

2007-07-13 12:23 UTC #10 lore raw

Quoted 10 lines · Dave Hansen

Quoted 2 lines · Avi Kivity
Can you confirm it by backing out that one patch?
Do you know the git commit id by chance?
-- Dave

commit 55a3212bc2f5ecddcd4c5cdf2bfb37ad71e45ff2

Author: Luca Tettamanti <[email protected]>
Date:   Tue Jun 5 14:47:33 2007 +0300

kvm: qemu: initialize ata ports as enabled

this allows libata to see the qemu ata ports.

qemu/hw/ide.c +2 -0

index 190f074..ddfb9bc 100644
@@ -2586,6 +2586,8 @@ static void piix3_reset(PCIIDEState *d)
     pci_conf[0x06] = 0x80; /* FBC */
     pci_conf[0x07] = 0x02; // PCI_status_devsel_medium
     pci_conf[0x20] = 0x01; /* BMIBA: 20-23h */
+    pci_conf[0x41] = 0x80; // enable port 0
+    pci_conf[0x43] = 0x80; // enable port 1
 }
 
 void pci_piix_ide_init(PCIBus *bus, BlockDriverState **hd_table, int devfn)

I was unable to reproduce this on a FC6 i386 guest.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 46976EA7.2020202-atKUWr5tajBWk0Htik3J/[email protected]

Luca

2007-07-13 13:27 UTC #12 lore raw

Quoted 34 lines · Avi Kivity

Quoted 10 lines · Dave Hansen
Quoted 2 lines · Avi Kivity
Can you confirm it by backing out that one patch?
Do you know the git commit id by chance?
-- Dave
commit 55a3212bc2f5ecddcd4c5cdf2bfb37ad71e45ff2
Author: Luca Tettamanti <[email protected]>
Date:   Tue Jun 5 14:47:33 2007 +0300
kvm: qemu: initialize ata ports as enabled
this allows libata to see the qemu ata ports.
qemu/hw/ide.c +2 -0
index 190f074..ddfb9bc 100644
@@ -2586,6 +2586,8 @@ static void piix3_reset(PCIIDEState *d)
     pci_conf[0x06] = 0x80; /* FBC */
     pci_conf[0x07] = 0x02; // PCI_status_devsel_medium
     pci_conf[0x20] = 0x01; /* BMIBA: 20-23h */
+    pci_conf[0x41] = 0x80; // enable port 0
+    pci_conf[0x43] = 0x80; // enable port 1
 }
 
 void pci_piix_ide_init(PCIBus *bus, BlockDriverState **hd_table, int devfn)

Quoted 1 line

I was unable to reproduce this on a FC6 i386 guest.

Hum... I assume the FC6 is using the "old" IDE driver, yes? I can't see why that patch should matter though... if those bits are not set then the IDE controller is not decoding the commands (on real hw).

Luca

A message that is not in the archive (sent off-list or not yet received): 68676e00707130627p32a43a63l3c6fe647242ec3e-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Avi Kivity

2007-07-13 13:41 UTC #14 lore raw

Quoted 36 lines · Luca

Quoted 35 lines · Avi Kivity
Quoted 10 lines · Dave Hansen
Quoted 2 lines · Avi Kivity
Can you confirm it by backing out that one patch?
Do you know the git commit id by chance?
-- Dave
commit 55a3212bc2f5ecddcd4c5cdf2bfb37ad71e45ff2
Author: Luca Tettamanti <[email protected]>
Date:   Tue Jun 5 14:47:33 2007 +0300
kvm: qemu: initialize ata ports as enabled
this allows libata to see the qemu ata ports.
qemu/hw/ide.c +2 -0
index 190f074..ddfb9bc 100644
@@ -2586,6 +2586,8 @@ static void piix3_reset(PCIIDEState *d)
     pci_conf[0x06] = 0x80; /* FBC */
     pci_conf[0x07] = 0x02; // PCI_status_devsel_medium
     pci_conf[0x20] = 0x01; /* BMIBA: 20-23h */
+    pci_conf[0x41] = 0x80; // enable port 0
+    pci_conf[0x43] = 0x80; // enable port 1
 }
 
 void pci_piix_ide_init(PCIBus *bus, BlockDriverState **hd_table, int
devfn)

Quoted 6 lines

Quoted 1 line
I was unable to reproduce this on a FC6 i386 guest.
Hum... I assume the FC6 is using the "old" IDE driver, yes? I can't see why that patch should matter though... if those bits are not set then the IDE controller is not decoding the commands (on real hw).

I tested FC6 i386 about found no difference with or without the patch.

Dave, can you diff the guest dmesg with kvm-27 and kvm-28 userspace? Maybe that will tell us something.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 46978110.5090303-atKUWr5tajBWk0Htik3J/[email protected]

Dave Hansen

author

2007-07-13 18:31 UTC #16 lore raw

Quoted 1 line · Avi Kivity

I tested FC6 i386 about found no difference with or without the patch.

I've git bisected down the the same patch that you pasted above. I've also tried just applying that single patch to kvm-27 and reproduced the slow behavior.

Quoted 2 lines

Dave, can you diff the guest dmesg with kvm-27 and kvm-28 userspace? Maybe that will tell us something.

Here's kvm-27 and kvm-27+55a3212bc2...:

diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt

Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007 BIOS-provided physical RAM map:
@@ -39,7 +22,7 @@
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 512 (order: 9, 2048 bytes)
-Detected 2669.668 MHz processor.
+Detected 2669.011 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
@@ -53,7 +36,7 @@

       .data : 0xc036efcf - 0xc043911c   ( 808 kB)
       .text : 0xc0100000 - 0xc036efcf   (2491 kB)

Checking if this processor honours the WP bit even in supervisor mode... Ok.
-Calibrating delay using timer specific routine.. 26758.82 BogoMIPS (lpj=53517640) +Calibrating delay using timer specific routine.. 26800.00 BogoMIPS (lpj=53600004)
Mount-cache hash table entries: 512
CPU: L1 I cache: 8K
CPU: L2 cache: 128K
@@ -107,20 +90,21 @@
PIIX3: IDE controller at PCI slot 0000:00:01.1
PIIX3: chipset revision 0
PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+ ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+ ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
hda: QEMU HARDDISK, ATA DISK drive
hdb: QEMU HARDDISK, ATA DISK drive
-Clocksource tsc unstable (delta = 767711774 ns)
+ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
+Clocksource tsc unstable (delta = 768021683 ns)
Time: pit clocksource has been installed.
hdc: QEMU CD-ROM, ATAPI CD/DVD-ROM drive
-ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 512KiB
-hda: 10485760 sectors (5368 MB) w/256KiB Cache, CHS=10402/255/63 +hda: 10485760 sectors (5368 MB) w/256KiB Cache, CHS=10402/255/63, (U)DMA
hda: cache flushes supported
hda: hda1
hdb: max request size: 512KiB
-hdb: 262144 sectors (134 MB) w/256KiB Cache, CHS=260/255/63
+hdb: 262144 sectors (134 MB) w/256KiB Cache, CHS=260/255/63, (U)DMA
hdb: cache flushes supported
hdb: hdb1
serio: i8042 KBD port at 0x60,0x64 irq 1
@@ -146,7 +130,7 @@
Adding 128480k swap on /dev/hdb1. Priority:-1 extents:1 across:128480k Checking root file system...
fsck 1.37 (21-Mar-2005)
-/dev/hda1: clean, 46970/655360 files, 287821/1309289 blocks +/dev/hda1: clean, 46970/655360 files, 287829/1309289 blocks
EXT3 FS on hda1, internal journal
Cleaning up ifupdown...done.
Checking all file systems...

Looks like the slow versions have DMA enabled. Is it slower to emulate DMA than PIO?

-- Dave

Anthony Liguori

2007-07-13 18:49 UTC #17 lore raw

Quoted 40 lines · Dave Hansen

Quoted 2 lines · Avi Kivity
I tested FC6 i386 about found no difference with or without the patch.
I've git bisected down the the same patch that you pasted above. I've also tried just applying that single patch to kvm-27 and reproduced the slow behavior.
Quoted 3 lines
Dave, can you diff the guest dmesg with kvm-27 and kvm-28 userspace? Maybe that will tell us something.
Here's kvm-27 and kvm-27+55a3212bc2...:
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007 BIOS-provided physical RAM map:
(quoted diff) +2 -2
@@ -39,7 +22,7 @@
 Enabling unmasked SIMD FPU exception support... done.
 Initializing CPU#0
 PID hash table entries: 512 (order: 9, 2048 bytes)
-Detected 2669.668 MHz processor.
+Detected 2669.011 MHz processor.
 Console: colour VGA+ 80x25
 Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
 Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
@@ -53,7 +36,7 @@
       .data : 0xc036efcf - 0xc043911c   ( 808 kB)
       .text : 0xc0100000 - 0xc036efcf   (2491 kB)
 Checking if this processor honours the WP bit even in supervisor mode... Ok.
-Calibrating delay using timer specific routine.. 26758.82 BogoMIPS (lpj=53517640)
+Calibrating delay using timer specific routine.. 26800.00 BogoMIPS (lpj=53600004)
 Mount-cache hash table entries: 512
 CPU: L1 I cache: 8K
 CPU: L2 cache: 128K

Review line 95

Quoted 7 lines

(quoted diff) +2 -1

@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio

There's your problem. With that patch, your kernel is using PIO instead of DMA.

I'm not 100% sure what's going here other than the fact that this particular problem has cropped up a number of times on qemu-devel. It's been very difficult to get all guests to actually use DMA.

Regards,

Anthony Liguori

Quoted 36 lines

(quoted diff) +4 -4

 hda: QEMU HARDDISK, ATA DISK drive
 hdb: QEMU HARDDISK, ATA DISK drive
-Clocksource tsc unstable (delta = 767711774 ns)
+ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
+Clocksource tsc unstable (delta = 768021683 ns)
 Time: pit clocksource has been installed.
 hdc: QEMU CD-ROM, ATAPI CD/DVD-ROM drive
-ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
 ide1 at 0x170-0x177,0x376 on irq 15
 hda: max request size: 512KiB
-hda: 10485760 sectors (5368 MB) w/256KiB Cache, CHS=10402/255/63
+hda: 10485760 sectors (5368 MB) w/256KiB Cache, CHS=10402/255/63, (U)DMA
 hda: cache flushes supported
  hda: hda1
 hdb: max request size: 512KiB
-hdb: 262144 sectors (134 MB) w/256KiB Cache, CHS=260/255/63
+hdb: 262144 sectors (134 MB) w/256KiB Cache, CHS=260/255/63, (U)DMA
 hdb: cache flushes supported
  hdb: hdb1

serio: i8042 KBD port at 0x60,0x64 irq 1

(quoted diff) +1 -1

@@ -146,7 +130,7 @@
 Adding 128480k swap on /dev/hdb1.  Priority:-1 extents:1 across:128480k
 Checking root file system...
 fsck 1.37 (21-Mar-2005)
-/dev/hda1: clean, 46970/655360 files, 287821/1309289 blocks
+/dev/hda1: clean, 46970/655360 files, 287829/1309289 blocks
 EXT3 FS on hda1, internal journal
 Cleaning up ifupdown...done.
 Checking all file systems...

Looks like the slow versions have DMA enabled. Is it slower to emulate DMA than PIO?

-- Dave

Quoted 11 lines

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

A message that is not in the archive (sent off-list or not yet received): [email protected]

Avi Kivity

2007-07-14 06:27 UTC #19 lore raw

Quoted 8 lines · Anthony Liguori

Quoted 8 lines
Quoted 4 lines
Dave, can you diff the guest dmesg with kvm-27 and kvm-28 userspace? Maybe that will tell us something.
Here's kvm-27 and kvm-27+55a3212bc2...:

Quoted 16 lines

Review line 95
Quoted 12 lines
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu
4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007
BIOS-provided physical RAM map:
(quoted diff) +2 -1
@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
 
There's your problem. With that patch, your kernel is using PIO instead of DMA.

Yeah, but later:

Quoted 8 lines

Quoted 7 lines

(quoted diff) +1 -1

 hda: max request size: 512KiB
-hda: 10485760 sectors (5368 MB) w/256KiB Cache, CHS=10402/255/63
+hda: 10485760 sectors (5368 MB) w/256KiB Cache, CHS=10402/255/63,

(U)DMA
hda: cache flushes supported

So now the slow version uses dma.

My attempt at wild guessing is that if the port is disabled, Linux enables it and configures it correctly. If it is already enabled, Linux trusts the bios to have configured it and leaves it at some silly default.

Dave, can you compare the output of hdparm -v /dev/hda? Maybe more clues there.

My FC6 i386 guest says:

/dev/hda:

 multcount    = 16 (on)
 IO_support   =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 16383/255/63, sectors = 20971520, start = 0

Which seems nice and sane.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 46986CEC.7050903-atKUWr5tajBWk0Htik3J/[email protected]

Dave Hansen

author

2007-07-16 19:40 UTC #21 lore raw

Quoted 2 lines · Avi Kivity

Dave, can you compare the output of hdparm -v /dev/hda? Maybe more clues there.

slow:

qemu:~# hdparm -v /dev/hda

/dev/hda:

 multcount    = 16 (on)
 IO_support   =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 10402/255/63, sectors = 10485760, start = 0

fast:

qemu:~# hdparm -v /dev/hda

/dev/hda:

 multcount    = 16 (on)
 IO_support   =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  0 (off)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 10402/255/63, sectors = 10485760, start = 0

-- Dave

Luca

2007-07-14 17:30 UTC #22 lore raw

In reply to a message not in the archive

Quoted 52 lines · Anthony Liguori

Quoted 40 lines · Dave Hansen
Quoted 2 lines · Avi Kivity
I tested FC6 i386 about found no difference with or without the patch.
I've git bisected down the the same patch that you pasted above. I've also tried just applying that single patch to kvm-27 and reproduced the slow behavior.
Quoted 3 lines
Dave, can you diff the guest dmesg with kvm-27 and kvm-28 userspace? Maybe that will tell us something.
Here's kvm-27 and kvm-27+55a3212bc2...:
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007 BIOS-provided physical RAM map:
(quoted diff) +2 -2
@@ -39,7 +22,7 @@
 Enabling unmasked SIMD FPU exception support... done.
 Initializing CPU#0
 PID hash table entries: 512 (order: 9, 2048 bytes)
-Detected 2669.668 MHz processor.
+Detected 2669.011 MHz processor.
 Console: colour VGA+ 80x25
 Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
 Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
@@ -53,7 +36,7 @@
       .data : 0xc036efcf - 0xc043911c   ( 808 kB)
       .text : 0xc0100000 - 0xc036efcf   (2491 kB)
 Checking if this processor honours the WP bit even in supervisor mode... Ok.
-Calibrating delay using timer specific routine.. 26758.82 BogoMIPS (lpj=53517640)
+Calibrating delay using timer specific routine.. 26800.00 BogoMIPS (lpj=53600004)
 Mount-cache hash table entries: 512
 CPU: L1 I cache: 8K
 CPU: L2 cache: 128K
Review line 95
Quoted 7 lines
(quoted diff) +2 -1
@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
 
There's your problem. With that patch, your kernel is using PIO instead of DMA.

Quoted 3 lines

I'm not 100% sure what's going here other than the fact that this particular problem has cropped up a number of times on qemu-devel. It's been very difficult to get all guests to actually use DMA.

Those are the BIOS settings, which are consistent with what QEMU is doing. Linux driver is actually configuring the device for MWDMA2

Luca

2007-07-15 19:27 UTC #23 lore raw

In reply to Dave Hansen

Quoted 3 lines · Dave Hansen

diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007

Is this a vanilla 2.6.22? 32 bit?

Review line 94

Quoted 7 lines

(quoted diff) +2 -1

@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio

Hum, FC6 PIIX driver always comes up with the same mode, regardless of what I'm doing to the controller...

Luca

A message that is not in the archive (sent off-list or not yet received): 68676e00707151227hb8b8538j706d9d0ee765ed41-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Luca

2007-07-15 21:07 UTC #25 lore raw

Quoted 7 lines · Luca

Quoted 3 lines · Dave Hansen
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007
Is this a vanilla 2.6.22? 32 bit?

Quoted 10 lines

Review line 94
Quoted 7 lines
(quoted diff) +2 -1
@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
Hum, FC6 PIIX driver always comes up with the same mode, regardless of what I'm doing to the controller...

Ok, now I'm confused...

pci_config_write: PIIX3 IDE: addr=40 val=00008000 len=2
pci_config_write: PIIX3 IDE: addr=42 val=00008000 len=2

I'm using KVM-27 userspace, and the BIOS is enabling both ports. So... linux shouldn't think that the ports are disabled (yet I saw the same thing with FC7 and libata).

Luca

A message that is not in the archive (sent off-list or not yet received): 68676e00707151407j1cf758e0ya7032c8c8577b982-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Luca

2007-07-15 21:22 UTC #27 lore raw

Quoted 19 lines · Luca

Quoted 7 lines · Luca
Quoted 3 lines · Dave Hansen
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007
Is this a vanilla 2.6.22? 32 bit?
Quoted 10 lines
Review line 94
Quoted 7 lines
(quoted diff) +2 -1
@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
Hum, FC6 PIIX driver always comes up with the same mode, regardless of what I'm doing to the controller...

Quoted 8 lines

Ok, now I'm confused...
pci_config_write: PIIX3 IDE: addr=40 val=00008000 len=2
pci_config_write: PIIX3 IDE: addr=42 val=00008000 len=2
I'm using KVM-27 userspace, and the BIOS is enabling both ports. So... linux shouldn't think that the ports are disabled (yet I saw the same thing with FC7 and libata).

Oh fuck, it's obvious... If you don't supply -L on the command line KVM ends up using the BIOS in the default system directory (/usr/share{,local}/qemu); on my system this the BIOS installed by QEMU 0.9, which does *not* enable the IDE ports. OTOH the BIOS file shipped with KVM does enable the ports.
I guess that you're seeing the same thing: you used to run KVM+QEMU bios (ports disabled - fast); then KVM started to enable IDE ports (even with QEMU bios) and you saw a regression.
Can you re-test KVM 27 with it's BIOS (i.e. use something like -L ~/src/kvm-27/qemu/pc-bios)?

FYI I'm seeing a better tput with MWDMA2 (ports enabled) rather than with PIO (ports disabled).

Luca

A message that is not in the archive (sent off-list or not yet received): 68676e00707151422n2e1f0a07kc4ec10797edfa40c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Avi Kivity

2007-07-16 08:58 UTC #29 lore raw

Quoted 41 lines · Luca

Quoted 20 lines · Luca
Quoted 4 lines · Luca
Quoted 3 lines · Dave Hansen
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu
4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007
Quoted 13 lines
Is this a vanilla 2.6.22? 32 bit?
Review line 94
Quoted 7 lines
(quoted diff) +2 -1
@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
Hum, FC6 PIIX driver always comes up with the same mode, regardless of what I'm doing to the controller...
Quoted 8 lines
Ok, now I'm confused...
pci_config_write: PIIX3 IDE: addr=40 val=00008000 len=2
pci_config_write: PIIX3 IDE: addr=42 val=00008000 len=2
I'm using KVM-27 userspace, and the BIOS is enabling both ports. So... linux shouldn't think that the ports are disabled (yet I saw the same thing with FC7 and libata).
Oh fuck, it's obvious... If you don't supply -L on the command line KVM ends up using the BIOS in the default system directory (/usr/share{,local}/qemu); on my system this the BIOS installed by QEMU 0.9, which does *not* enable the IDE ports. OTOH the BIOS file shipped with KVM does enable the ports.
I guess that you're seeing the same thing: you used to run KVM+QEMU bios (ports disabled - fast); then KVM started to enable IDE ports (even with QEMU bios) and you saw a regression.
Can you re-test KVM 27 with it's BIOS (i.e. use something like -L ~/src/kvm-27/qemu/pc-bios)?

Quoted 2 lines

FYI I'm seeing a better tput with MWDMA2 (ports enabled) rather than with PIO (ports disabled).

So, would it be correct to revert the patch enabling the ports, assuming people use the kvm-supplied bios?

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 469B3351.50508-atKUWr5tajBWk0Htik3J/[email protected]

Luca

2007-07-16 19:32 UTC #31 lore raw

Quoted 42 lines · Avi Kivity

Quoted 41 lines · Luca
Quoted 20 lines · Luca
Quoted 4 lines · Luca
Quoted 3 lines · Dave Hansen
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu
4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007
Quoted 13 lines
Is this a vanilla 2.6.22? 32 bit?
Review line 94
Quoted 7 lines
(quoted diff) +2 -1
@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
Hum, FC6 PIIX driver always comes up with the same mode, regardless of what I'm doing to the controller...
Quoted 8 lines
Ok, now I'm confused...
pci_config_write: PIIX3 IDE: addr=40 val=00008000 len=2
pci_config_write: PIIX3 IDE: addr=42 val=00008000 len=2
I'm using KVM-27 userspace, and the BIOS is enabling both ports. So... linux shouldn't think that the ports are disabled (yet I saw the same thing with FC7 and libata).
Oh fuck, it's obvious... If you don't supply -L on the command line KVM ends up using the BIOS in the default system directory (/usr/share{,local}/qemu); on my system this the BIOS installed by QEMU 0.9, which does *not* enable the IDE ports. OTOH the BIOS file shipped with KVM does enable the ports.
I guess that you're seeing the same thing: you used to run KVM+QEMU bios (ports disabled - fast); then KVM started to enable IDE ports (even with QEMU bios) and you saw a regression.
Can you re-test KVM 27 with it's BIOS (i.e. use something like -L ~/src/kvm-27/qemu/pc-bios)?

Quoted 5 lines

Quoted 2 lines
FYI I'm seeing a better tput with MWDMA2 (ports enabled) rather than with PIO (ports disabled).
So, would it be correct to revert the patch enabling the ports, assuming people use the kvm-supplied bios?

I think so. Where does the BIOS come from? Latest Bochs?

Luca

A message that is not in the archive (sent off-list or not yet received): 68676e00707161232k5fbd0c2cxd8bebfc21faacc8c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Avi Kivity

2007-07-17 07:58 UTC #33 lore raw

Quoted 5 lines · Luca

Quoted 3 lines
So, would it be correct to revert the patch enabling the ports, assuming people use the kvm-supplied bios?
I think so.

Please confirm that the original test case (I think it was the FC7 installer?) works with the kvm bios.

Quoted 2 lines

Where does the BIOS come from? Latest Bochs?

Yes. See qemu/pc-bios/bochs-manifest.

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

A message that is not in the archive (sent off-list or not yet received): 469C76C0.5050101-atKUWr5tajBWk0Htik3J/[email protected]

Luca

2007-07-18 06:01 UTC #35 lore raw

Quoted 9 lines · Avi Kivity

Quoted 5 lines · Luca
Quoted 3 lines
So, would it be correct to revert the patch enabling the ports, assuming people use the kvm-supplied bios?
I think so.
Please confirm that the original test case (I think it was the FC7 installer?) works with the kvm bios.

Yes, FC7 libata driver does see the disks.

Luca

Dave Hansen

author

2007-07-16 19:49 UTC #36 lore raw

In reply to a message not in the archive

Quoted 2 lines · Luca

Can you re-test KVM 27 with it's BIOS (i.e. use something like -L ~/src/kvm-27/qemu/pc-bios)?

Doing that does appear to make it behave like kvm-28. DMA enabled, and slow I/O:

qemu:~# hdparm -v /dev/hda

/dev/hda:

 multcount    = 16 (on)
 IO_support   =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 10402/255/63, sectors = 10485760, start = 0

-- Dave

Dave Hansen

author

2007-07-16 19:41 UTC #37 lore raw

In reply to a message not in the archive

Quoted 6 lines · Luca

Quoted 3 lines · Dave Hansen
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007
Is this a vanilla 2.6.22? 32 bit?

Yep, vanilla.

Quoted 10 lines

Review line 94
Quoted 7 lines
(quoted diff) +2 -1
@@ -107,20 +90,21 @@
 PIIX3: IDE controller at PCI slot 0000:00:01.1
 PIIX3: chipset revision 0
 PIIX3: not 100% native mode: will probe irqs later
-PIIX3: neither IDE port enabled (BIOS)
+    ide0: BM-DMA at 0x1400-0x1407, BIOS settings: hda:pio, hdb:pio
+    ide1: BM-DMA at 0x1408-0x140f, BIOS settings: hdc:pio, hdd:pio
Hum, FC6 PIIX driver always comes up with the same mode, regardless of what I'm doing to the controller...

-- Dave

Dave Hansen

author

2007-07-16 19:42 UTC #38 lore raw

In reply to a message not in the archive

Quoted 6 lines · Luca

Quoted 3 lines · Dave Hansen
diff -ru kvm-fast-dmesg.txt kvm-slow-dmesg.txt
Linux version 2.6.22 (dave@kernel) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #13 Wed Jul 11 15:27:01 PDT 2007
Is this a vanilla 2.6.22? 32 bit?

And, yes 32-bit kernel. But, I'm running it on the x86_64 version kvm on an x86_64 host.

-- Dave