large page size virtio issues
Hi Rusty, I'm using a patch that changes the Linux base page size to 64K. (This is actually pretty common in ppc64 world, but I happen to be trying it on ppc32.)
I'm seeing a problem with virtio. I think at least part of it can be explained by qemu's TARGET_PAGE_BITS=12, and the guest's PAGE_SHIFT=16. The guest allocates the queue, then passes the pfn (pa
Quoted 1 line
Quoted 1 line
PAGE_SHIFT) to the virtio backend (vp_find_vq()). The backend then
calculates the pa as pfn << TARGET_PAGE_BITS.
I have to run right now, but quickly changing qemu TARGET_PAGE_BITS to 16 got me a little further but still didn't work. Any thoughts?
Thanks!
Signature · 2 lines
Hollis Blanchard IBM Linux Technology Center
Quoted 8 lines · On Wednesday 05 November 2008 09:14:20 Hollis Blanchard wrote:
Hi Rusty, I'm using a patch that changes the Linux base page size to 64K. (This is actually pretty common in ppc64 world, but I happen to be trying it on ppc32.)
I'm seeing a problem with virtio. I think at least part of it can be explained by qemu's TARGET_PAGE_BITS=12, and the guest's PAGE_SHIFT=16. The guest allocates the queue, then passes the pfn (pa
Quoted 6 lines
Quoted 1 line
Quoted 1 line
PAGE_SHIFT) to the virtio backend (vp_find_vq()). The backend thencalculates the pa as pfn <<
TARGET_PAGE_BITS.I have to run right now, but quickly changing qemu
TARGET_PAGE_BITSto 16 got me a little further but still didn't work. Any thoughts?
I see Anthony hardwired page size into the queue activation ABI for virtio_pci. I think that this should be an actual 4096 (or 12) rather than depending on guest page size:
virtio_pci.c:247:
iowrite32(virt_to_phys(info->queue) >> PAGE_SHIFT, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
Anthony?
Cheers,
Rusty.
Quoted 13 lines · Rusty Russell
Quoted 11 lines · On Wednesday 05 November 2008 09:14:20 Hollis Blanchard wrote:
Hi Rusty, I'm using a patch that changes the Linux base page size to 64K. (This is actually pretty common in ppc64 world, but I happen to be trying it on ppc32.)
I'm seeing a problem with virtio. I think at least part of it can be explained by qemu's TARGET_PAGE_BITS=12, and the guest's PAGE_SHIFT=16. The guest allocates the queue, then passes the pfn (pa
Quoted 2 lines
Quoted 2 lines
PAGE_SHIFT) to the virtio backend (vp_find_vq()). The backend then
Quoted 8 lines
Quoted 5 lines
calculates the pa as pfn <<
TARGET_PAGE_BITS.I have to run right now, but quickly changing qemu
TARGET_PAGE_BITSto 16 got me a little further but still didn't work. Any thoughts?I see Anthony hardwired page size into the queue activation ABI for
virtio_pci.
So did you FWIW, virtio-balloon passes PFNs which are computed based on PAGE_SHIFT.
Quoted 3 lines
I think that this should be an actual 4096 (or 12) rather than
depending on guest page size:
So is the issue that PPC can support 4k or 16k pages, and the guest happens to always use 16k pages? Does the guest set any global flag indicating it is using 16k pages? Is this anyway we could detect this in QEMU?
I don't much like the idea of globally hard coding it to 4k. I'd rather make it architecture specific.
Regards,
Anthony Liguori
Quoted 9 lines
virtio_pci.c:247:
iowrite32(virt_to_phys(info->queue) >> PAGE_SHIFT, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);Anthony?
Cheers,
Rusty.
Quoted 17 lines · Anthony Liguori
Quoted 16 lines · Rusty Russell
Quoted 13 lines · On Wednesday 05 November 2008 09:14:20 Hollis Blanchard wrote:
Hi Rusty, I'm using a patch that changes the Linux base page size to 64K. (This is actually pretty common in ppc64 world, but I happen to be trying it on ppc32.)
I'm seeing a problem with virtio. I think at least part of it can be explained by qemu's TARGET_PAGE_BITS==12, and the guest's PAGE_SHIFT==16. The guest allocates the queue, then passes the pfn (pa
Quoted 1 line
Quoted 1 line
PAGE_SHIFT) to the virtio backend (vp_find_vq()). The backend thencalculates the pa as pfn <<
TARGET_PAGE_BITS.I have to run right now, but quickly changing qemu
TARGET_PAGE_BITSto 16 got me a little further but still didn't work. Any thoughts?
Quoted 8 lines
Quoted 2 lines
I see Anthony hardwired page size into the queue activation ABI for
virtio_pci.So did you FWIW, virtio-balloon passes PFNs which are computed based on
PAGE_SHIFT.Quoted 2 lines
I think that this should be an actual 4096 (or 12) rather than
depending on guest page size:
I agree: it's simply a question of both sides of the interface agreeing on the units used in the interface. As the interface is defined today, both qemu and the guest virtio should use a fixed constant which is neither (Linux) PAGE_SHIFT nor (qemu) TARGET_PAGE_BITS.
Quoted 4 lines
So is the issue that PPC can support 4k or 16k pages, and the guest happens to always use 16k pages? Does the guest set any global flag indicating it is using 16k pages? Is this anyway we could detect this in QEMU?
To elaborate a little, I'm using a patch to PowerPC 440 Linux that allows you to configure the base page size at build time; choices are 4K, 16K, and 64K. (The hardware supports more sizes, and with other patches or other operating systems qemu would need to worry about 256K, 1M, 16M, and 256M pages.)
The page size is set per MMU mapping, and of course it's ridiculous to walk the TLB to see if all the page sizes are the same to "detect" the condition. In fact, regardless of the base page size, the (Linux) kernel is always mapped with 256M pages, and if you consider hugetlbfs the situation is even more fluid.
Quoted 2 lines
I don't much like the idea of globally hard coding it to 4k. I'd rather make it architecture specific.
Making the units architecture-specific doesn't solve the problem at all AFAICS. It doesn't even solve my original problem on PowerPC 440 since the guest page size can vary.
AFAIK the only reason to use a PFN in this interface in the first place is to allow for physical addresses >32 bits. A hardcoded shift of 12 gives you 44 bits of physical address space (16 TB). This actually isn't very big today, so using an architecture-specific hardcoded 4K size will become an issue anyways, *even on x86*.
Brainstorming backwards-compatible interface expansion possibilities:
1. Rename the current interface to "4K_PFN", and add another, let's say "64K_PFN". Of course, if a guest with smaller pages uses the new interface, it must properly align its queue allocation.
2. Rename the current interface to "4K_PFN". Use 64-bit writes to set VIRTIO_PCI_QUEUE_PFN. 32-bit architectures couldn't use this, which might be OK since practically speaking, I think 32-bit architectures can address at most 36 bits of physical space. I also don't know what the semantics are of 64-bit PCI writes (if it's not allowed on physical hardware) -- it looks like Linux doesn't have an iowrite64, for example.
3. Rename the current interface to "4K_PFN". Use multiple writes (high/low) to set VIRTIO_PCI_QUEUE_PFN. Not atomic. To simplify backend implementation, you could require that PFN_HIGH writes come before PFN_LOW.
4. Use multiple writes (set page size, set PFN). SET_PAGE_SIZE must precede SET_PFN. Not atomic.
5. Create a variable-sized interface (still 32-bit write), where the shift value is encoded in the value itself (I guess this is the FP mantissa+exponent approach). For example, the low 8 bits are the shift beyond 12, so a write of 0x10000004 would mean physical address 1<<(12+4).
These solutions would solve both problems: a) making "guest page size" explicit, and b) addressing more than 16TB of physical memory in the future. I think I like #3 or #4 the best.
Hardcoding the current interface to mean "4K pages" (and updating qemu and Linux to match) would solve my problem, and the 16TB limit could be addressed in the future as needed.
Signature · 7 lines
Hollis Blanchard IBM Linux Technology Center -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Hollis Blanchard
Quoted 2 lines · Anthony Liguori
I don't much like the idea of globally hard coding it to 4k. I'd rather make it architecture specific.
Making the units architecture-specific doesn't solve the problem at all AFAICS. It doesn't even solve my original problem on PowerPC 440 since the guest page size can vary.
Quoted 6 lines
AFAIK the only reason to use a PFN in this interface in the first place is to allow for physical addresses >32 bits. A hardcoded shift of 12 gives you 44 bits of physical address space (16 TB). This actually isn't very big today, so using an architecture-specific hardcoded 4K size will become an issue anyways, *even on x86*.
It's PIO on x86 so there's no way to do a 64-bit operation.
Quoted 14 lines
Brainstorming backwards-compatible interface expansion possibilities:
1. Rename the current interface to "4K_PFN", and add another, let's say "64K_PFN". Of course, if a guest with smaller pages uses the new interface, it must properly align its queue allocation.
2. Rename the current interface to "4K_PFN". Use 64-bit writes to setVIRTIO_PCI_QUEUE_PFN. 32-bit architectures couldn't use this, which might be OK since practically speaking, I think 32-bit architectures can address at most 36 bits of physical space. I also don't know what the semantics are of 64-bit PCI writes (if it's not allowed on physical hardware) -- it looks like Linux doesn't have an iowrite64, for example.
3. Rename the current interface to "4K_PFN". Use multiple writes (high/low) to setVIRTIO_PCI_QUEUE_PFN. Not atomic. To simplify backend implementation, you could require thatPFN_HIGHwrites
Quoted 9 lines
come before
PFN_LOW.
4. Use multiple writes (set page size, set PFN).SET_PAGE_SIZEmust precedeSET_PFN. Not atomic.
5. Create a variable-sized interface (still 32-bit write), where the shift value is encoded in the value itself (I guess this is the FP mantissa+exponent approach). For example, the low 8 bits are the shift beyond 12, so a write of0x10000004would mean physical address 1<<(12+4).
I think we should just have a VIRTIO_PFN_SHIFT define that is architecture specific.
On x86, it'll be 4k. You can make it whatever you want for PPC. Fixing all architectures to be 4k is going to suck for architectures with < 4k pages.
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Anthony Liguori
I think we should just have a
VIRTIO_PFN_SHIFTdefine that is architecture specific.On x86, it'll be 4k. You can make it whatever you want for PPC. Fixing all architectures to be 4k is going to suck for architectures with < 4k pages.
So we're now in agreement that it should be hardcoded. I don't see much value in making the actual value arch-specific, but either way it will solve (at least part of, and maybe all of) my problem. Rusty?
Signature · 2 lines
Hollis Blanchard IBM Linux Technology Center
Quoted 4 lines · Hollis Blanchard
Quoted 2 lines · Anthony Liguori
I think we should just have a
VIRTIO_PFN_SHIFTdefine that is architecture specific.
Quoted 9 lines
Quoted 4 lines
On x86, it'll be 4k. You can make it whatever you want for PPC. Fixing all architectures to be 4k is going to suck for architectures with < 4k pages.
So we're now in agreement that it should be hardcoded. I don't see much value in making the actual value arch-specific, but either way it will solve (at least part of, and maybe all of) my problem. Rusty?
When you balloon pages, you need to pass something that points to the beginning of a page boundary. If your page size is 2k, then if you are only able to pass 4k addresses, you can't below any page that happens to not be 4k aligned.
If you were going to make it common across all architectures, you would have to make it the smallest possible page size and that would break x86 backwards compatibility.
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Anthony Liguori
Quoted 5 lines · Hollis Blanchard
Quoted 3 lines · Anthony Liguori
I think we should just have a
VIRTIO_PFN_SHIFTdefine that is architecture specific.Quoted 9 lines
Quoted 4 lines
On x86, it'll be 4k. You can make it whatever you want for PPC. Fixing all architectures to be 4k is going to suck for architectures with < 4k pages.
So we're now in agreement that it should be hardcoded. I don't see much value in making the actual value arch-specific, but either way it will solve (at least part of, and maybe all of) my problem. Rusty?
Quoted 8 lines
When you balloon pages, you need to pass something that points to the beginning of a page boundary. If your page size is 2k, then if you are only able to pass 4k addresses, you can't below any page that happens to not be 4k aligned.
If you were going to make it common across all architectures, you would have to make it the smallest possible page size and that would break x86 backwards compatibility.
OK, makes sense to me.
Signature · 2 lines
Hollis Blanchard IBM Linux Technology Center
Quoted 16 lines · Rusty Russell
Quoted 8 lines · On Wednesday 05 November 2008 09:14:20 Hollis Blanchard wrote:
Hi Rusty, I'm using a patch that changes the Linux base page size to 64K. (This is actually pretty common in ppc64 world, but I happen to be trying it on ppc32.)
I'm seeing a problem with virtio. I think at least part of it can be explained by qemu's TARGET_PAGE_BITS==12, and the guest's PAGE_SHIFT==16. The guest allocates the queue, then passes the pfn (pa
Quoted 6 lines
Quoted 1 line
Quoted 1 line
PAGE_SHIFT) to the virtio backend (vp_find_vq()). The backend thencalculates the pa as pfn <<
TARGET_PAGE_BITS.I have to run right now, but quickly changing qemu
TARGET_PAGE_BITSto 16 got me a little further but still didn't work. Any thoughts?
Quoted 7 lines
I see Anthony hardwired page size into the queue activation ABI for
virtio_pci. I think that this should be an actual 4096 (or 12) rather than depending on guest page size:virtio_pci.c:247:
iowrite32(virt_to_phys(info->queue) >> PAGE_SHIFT, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
Actually there's an additional complication: PAGE_SIZE is entrenched in the layout of the ring structure itself (large "struct vring" comment in include/linux/virtio_ring.h). Callers of vring_size() demonstrate this problem.
I believe the vring is split across two pages so that it could be directly mapped by two untrusted guests: the producer would have RW access to the descriptors and the "available" fields, while the consumer would have only RO access. I don't think this is yet implemented; is it still planned?
It looks like vring_size() et al were carefully written to allow arbitrary page sizes, so for now I'll assume that a struct vring that is contained completely within a single guest mapping is OK and work up a patch.
It's worth noting that the PFN_SHIFT question (in the other thread) is a separate issue. I could have PFN_SHIFT=10 but define VRING_PAGE_SIZE=4K for internal alignment. I'm a little nervous about making PPC the only arch with VRING_PAGE_SIZE=1K, since that might affect my vring layout in "unusual" ways if it expands beyond 1K but stays inside 4K.
Signature · 7 lines
Hollis Blanchard IBM Linux Technology Center -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Hollis Blanchard
Actually there's an additional complication:
PAGE_SIZEis entrenched in the layout of the ring structure itself (large "struct vring" comment ininclude/linux/virtio_ring.h). Callers ofvring_size()demonstrate this problem.I believe the vring is split across two pages so that it could be directly mapped by two untrusted guests: the producer would have RW access to the descriptors and the "available" fields, while the consumer would have only RO access. I don't think this is yet implemented; is it still planned?
Rusty experimented with it but I don't think anyone has done anything seriously with it.
Quoted 5 lines
It looks like
vring_size()et al were carefully written to allow arbitrary page sizes, so for now I'll assume that a struct vring that is contained completely within a single guest mapping is OK and work up a patch.
It was written that way so that vring_size() could be used in userspace where there isn't a PAGE_SIZE defined. Rusty just passes in getpagesize() in the lguest userspace.
Pure evil if you ask me :-)
Quoted 6 lines
It's worth noting that the
PFN_SHIFTquestion (in the other thread) is a separate issue. I could have PFN_SHIFT but define VRING_PAGE_SIZE=4K for internal alignment. I'm a little nervous about making PPC the only arch with VRING_PAGE_SIZE=1K, since that might affect my vring layout in "unusual" ways if it expands beyond 1K but stays inside 4K.
This is starting to get a little hairy. Maybe 64k guest pages is asking for trouble? Are you sure other things aren't breaking all over the place with these patches in place?
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 7 lines
It's worth noting that the
PFN_SHIFTquestion (in the other thread) is a separate issue. I could have PFN_SHIFT=10 but define VRING_PAGE_SIZE=4K for internal alignment. I'm a little nervous about making PPC the only arch with VRING_PAGE_SIZE=1K, since that might affect my vring layout in "unusual" ways if it expands beyond 1K but stays inside 4K.This is starting to get a little hairy.
Yes it is.
Quoted 3 lines
Maybe 64k guest pages is asking
for trouble? Are you sure other things aren't breaking all over the place with these patches in place?
Define "other things"? Here's my current boot log from 64K/64K guest/host configuration:
bash-3.00# ./qemu-system-ppcemb -m 128 -M bamboo -nographic -kernel ~/uImage.64k -L ../pc-bios/ -append "root=/dev/vda rw debug" -net nic -net tap -drive file=/mnt/images/ramdisk_image,if=virtio,boot=onbamboo_init: START
Ram size passed is: 128 MB
Calling function ppc440_init
setup mmio
setup universal controller
trying to setup sdram controllersdram_unmap_bcr: Unmap RAM area 0000000000000000 00400000sdram_set_bcr: Map RAM area 0000000000000000 08000000
Initializing first serial portppc405_serial_init: offset 0000000000000300
Done calling ppc440_initbamboo_init: load kernel
kernel is at guest address: 0x0bamboo_init: load device tree file
device tree address is at guest address: 0x381488bamboo_init: DONE
Using PowerPC 44x Platform machine description
Linux version 2.6.28-rc2-dirty (hollisb-EGjIuKC2qUdB0N6nvOmcJFaTQe2KTcn/@public.gmane.org) (gcc version 4.2.3) #101 Tue Nov 4 14:19:54 CST 2008
Found legacy serial port 0 for /plb/opb/serial@ef600300
mem=ef600300, taddr=ef600300, irq=0, clk=11059200, speed=115200
Found legacy serial port 1 for /plb/opb/serial@ef600400
mem=ef600400, taddr=ef600400, irq=0, clk=11059200, speed=0
console [udbg0] enabledsetup_arch: bootmem
arch: exit
Top of RAM: 0x8000000, Total RAM: 0x8000000
Memory hole size: 0MB
Zone PFN ranges:
DMA 0x00000000 -> 0x00000800
Normal 0x00000800 -> 0x00000800
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0x00000000 -> 0x00000800
On node 0 totalpages: 2048free_area_init_node: node 0, pgdat c034baa8, node_mem_map c0440000
DMA zone: 1 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 2047 pages, LIFO batch:0
Normal zone: 0 pages used for memmap
Movable zone: 0 pages used for memmap
Built 1 zonelists in Zone order, mobility grouping off. Total pages: 2047
Kernel command line: root=/dev/vda rw debug
UIC0 (32 IRQ sources) at DCR 0xc0
PID hash table entries: 512 (order: 9, 2048 bytes)time_init: decrementer frequency = 666.666660 MHztime_init: processor frequency = 666.666660 MHz
clocksource: timebase mult[600000] shift[22] registered
clockevent: decrementer mult[aaaa] shift[16] cpu[0]
Dentry cache hash table entries: 16384 (order: 0, 65536 bytes)
Inode-cache hash table entries: 16384 (order: 0, 65536 bytes)
Memory: 126336k/131072k available (3136k kernel code, 4736k reserved, 256k data, 655k bss, 320k init)
SLUB: Genslabs=14, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop... 2555.90 BogoMIPS (lpj=5111808)
Mount-cache hash table entries: 8192net_namespace: 288 bytesprogram_check_exception: nip c02cad1c reason 2800000
------------[ cut here ]------------ Badness at c02cad1c [verbose debug info unavailable] NIP: c02cad1c LR: c00013e4 CTR: c02cacf0 REGS: c7e11e00 TRAP: 0700 Not tainted (2.6.28-rc2-dirty) MSR: 00029002 <EE,ME> CR: 84000042 XER: 20000000
TASK = c7e00000[1] 'swapper' THREAD: c7e10000
GPR00: c7fe0000 c7e11eb0 c7e00000 c02cacf0 00000000 00000000 c04a04bc 0000076d
GPR08: 00000036 00000001 c7e11e38 c7fdffff 44000042 00000000 00000000 00000000
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
GPR24: 00000000 00000000 00000f60 00000000 c0390360 c02cacf0 c7e10000 c032001c NIP [c02cad1c] dma_alloc_init+0x2c/0xcc
LR [c00013e4] do_one_initcall+0x38/0x19c
Call Trace: [c7e11eb0] [c02d78d0] net_ns_init+0x0/0x1e4 (unreliable) [c7e11ec0] [c00013e4] do_one_initcall+0x38/0x19c [c7e11fd0] [c02c0168] kernel_init+0x88/0xf4 [c7e11ff0] [c000d5f4] kernel_thread+0x50/0x6c
Instruction dump:4e800020 9421fff0 7c0802a6 3d20c033 93e1000c 90010014 81290334 37e9001c 41820094 8009001c 3160ffff 7d2b0110 <0f090000> 7403ffff 4182005c 3c030001
NET: Registered protocol family 16
PCI host bridge /plb/pci@ec000000 (primary) ranges:
MEM 0x00000000a0000000..0x00000000bfffffff -> 0x00000000a0000000
IO 0x00000000e8000000..0x00000000e800ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCI: Probing PCI hardware
PCI: Hiding 4xx host bridge resources 0000:00:00.0
pci 0000:00:01.0: reg 10 io port: [0x00-0x3f]
pci 0000:00:02.0: reg 10 io port: [0x00-0x1f]
bus: 00 index 0 io port: [0x00-0xffff]
bus: 00 index 1 mmio: [0xa0000000-0xbfffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 16384 (order: 0, 65536 bytes)
TCP established hash table entries: 8192 (order: 0, 65536 bytes)
TCP bind hash table entries: 8192 (order: -1, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 244
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
Serial: 8250/16550 driver4 ports, IRQ sharing enabled
serial8250.0: ttyS0 at MMIO 0xef600300 (irq = 16) is a 16550A console handover: boot [udbg0] -> real [ttyS0]
ef600300.serial: ttyS0 at MMIO 0xef600300 (irq = 16) is a 16550A
brd: module loaded
Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI
Copyright (c) 1999-2006 Intel Corporation.
PPC 4xx OCP EMAC driver, version 3.54
ZMII /plb/opb/emac-zmii@ef600d00 initialized
pcnet32.c:v1.35 21.Apr.2008 [email protected]
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk-zC7DfRvBq/JWk0Htik3J/[email protected]> virtio-pci 0000:00:01.0: enabling device (0000 -> 0001)
vda:qemu-system-ppcemb: Guest moved used index from 0 to 38277
(This is with my in-progress patches applied. Before virtio would just hang, since it was miscommunicating the vring's physical address.)
Signature · 7 lines
Hollis Blanchard IBM Linux Technology Center -- To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html