Announcing qboot, a minimal x86 firmware for QEMU
Paolo Bonzini
maintainerSome of you may have heard about the "Clear Containers" initiative from Intel, which couple KVM with various kernel tricks to create extremely lightweight virtual machines. The experimental Clear Containers setup requires only 18-20 MB to launch a virtual machine, and needs about 60 ms to boot.
Now, as all of you probably know, "QEMU is great for running Windows or legacy Linux guests, but that flexibility comes at a hefty price. Not only does all of the emulation consume memory, it also requires some form of low-level firmware in the guest as well. All of this adds quite a bit to virtual-machine startup times (500 to 700 milliseconds is not unusual)".
Right? In fact, it's for this reason that Clear Containers uses kvmtool instead of QEMU.
No, wrong! In fact, reporting bad performance is pretty much the same as throwing down the gauntlet.
Enter qboot, a minimal x86 firmware that runs on QEMU and, together with a slimmed-down QEMU configuration, boots a virtual machine in 40 milliseconds[2] on an Ivy Bridge Core i7 processor.
qboot is available at git://github.com/bonzini/qboot.git. In all the glory of its 8KB of code, it brings together various existing open source components:
* a minimal (really minimal) 16-bit BIOS runtime based on kvmtool's own BIOS
* a couple hardware initialization routines written mostly from scratch but with good help from SeaBIOS source code
* a minimal 32-bit libc based on kvm-unit-tests
* the Linux loader from QEMU itself
The repository has more information on how to achieve fast boot times, and examples of using qboot. Right now there is a limit of 8 MB for vmlinuz+initrd+cmdline, which however should be enough for initrd-less containers.
The first commit to qboot is more or less 24 hours old, so there is definitely more work to do, in particular to extract ACPI tables from QEMU and present them to the guest. This is probably another day of work or so, and it will enable multiprocessor guests with little or no impact on the boot times. SMBIOS information is also available from QEMU.
On the QEMU side, there is no support yet for persistent memory and the NFIT tables from ACPI 6.0. Once that (and ACPI support) is added, qboot will automatically start using it.
Happy hacking!
Paolo
Quoted 47 lines · Paolo Bonzini
Some of you may have heard about the "Clear Containers" initiative from Intel, which couple KVM with various kernel tricks to create extremely lightweight virtual machines. The experimental Clear Containers setup requires only 18-20 MB to launch a virtual machine, and needs about 60 ms to boot.
Now, as all of you probably know, "QEMU is great for running Windows or legacy Linux guests, but that flexibility comes at a hefty price. Not only does all of the emulation consume memory, it also requires some form of low-level firmware in the guest as well. All of this adds quite a bit to virtual-machine startup times (500 to 700 milliseconds is not unusual)".
Right? In fact, it's for this reason that Clear Containers uses kvmtool instead of QEMU.
No, wrong! In fact, reporting bad performance is pretty much the same as throwing down the gauntlet.
Enter qboot, a minimal x86 firmware that runs on QEMU and, together with a slimmed-down QEMU configuration, boots a virtual machine in 40 milliseconds[2] on an Ivy Bridge Core i7 processor.
qboot is available at
git://github.com/bonzini/qboot.git. In all the glory of its 8KB of code, it brings together various existing open source components:* a minimal (really minimal) 16-bit BIOS runtime based on kvmtool's own BIOS
* a couple hardware initialization routines written mostly from scratch but with good help from SeaBIOS source code
* a minimal 32-bit libc based on kvm-unit-tests
* the Linux loader from QEMU itself
The repository has more information on how to achieve fast boot times, and examples of using qboot. Right now there is a limit of 8 MB for vmlinuz+initrd+cmdline, which however should be enough for initrd-less containers.
The first commit to qboot is more or less 24 hours old, so there is definitely more work to do, in particular to extract ACPI tables from QEMU and present them to the guest. This is probably another day of work or so, and it will enable multiprocessor guests with little or no impact on the boot times. SMBIOS information is also available from QEMU.
Quoted 6 lines
On the QEMU side, there is no support yet for persistent memory and the NFIT tables from ACPI 6.0. Once that (and ACPI support) is added, qboot will automatically start using it.
Happy hacking!
Lovely!
Note you have memcpy.o instead of memcpy.c.
Paolo Bonzini
maintainerQuoted 3 lines · Avi Kivity
Lovely!
Note you have memcpy.o instead of
memcpy.c.
Doh, and it's not used anyway. Check the repository, and let me know if OSv boots with it (it probably needs ACPI; Linux doesn't boot virtio without ACPI).
Paolo
Quoted 9 lines · Paolo Bonzini
Quoted 3 lines · Avi Kivity
Lovely!
Note you have memcpy.o instead of
memcpy.c.Doh, and it's not used anyway. Check the repository, and let me know if OSv boots with it (it probably needs ACPI; Linux doesn't boot virtio without ACPI).
Yes, it requires ACPI. We don't implement the pre-ACPI bootstrap methods.
Quoted 47 lines · Paolo Bonzini
Some of you may have heard about the "Clear Containers" initiative from Intel, which couple KVM with various kernel tricks to create extremely lightweight virtual machines. The experimental Clear Containers setup requires only 18-20 MB to launch a virtual machine, and needs about 60 ms to boot.
Now, as all of you probably know, "QEMU is great for running Windows or legacy Linux guests, but that flexibility comes at a hefty price. Not only does all of the emulation consume memory, it also requires some form of low-level firmware in the guest as well. All of this adds quite a bit to virtual-machine startup times (500 to 700 milliseconds is not unusual)".
Right? In fact, it's for this reason that Clear Containers uses kvmtool instead of QEMU.
No, wrong! In fact, reporting bad performance is pretty much the same as throwing down the gauntlet.
Enter qboot, a minimal x86 firmware that runs on QEMU and, together with a slimmed-down QEMU configuration, boots a virtual machine in 40 milliseconds[2] on an Ivy Bridge Core i7 processor.
qboot is available at
git://github.com/bonzini/qboot.git. In all the glory of its 8KB of code, it brings together various existing open source components:* a minimal (really minimal) 16-bit BIOS runtime based on kvmtool's own BIOS
* a couple hardware initialization routines written mostly from scratch but with good help from SeaBIOS source code
* a minimal 32-bit libc based on kvm-unit-tests
* the Linux loader from QEMU itself
The repository has more information on how to achieve fast boot times, and examples of using qboot. Right now there is a limit of 8 MB for vmlinuz+initrd+cmdline, which however should be enough for initrd-less containers.
The first commit to qboot is more or less 24 hours old, so there is definitely more work to do, in particular to extract ACPI tables from QEMU and present them to the guest. This is probably another day of work or so, and it will enable multiprocessor guests with little or no impact on the boot times. SMBIOS information is also available from QEMU.
Quoted 5 lines
On the QEMU side, there is no support yet for persistent memory and the NFIT tables from ACPI 6.0. Once that (and ACPI support) is added, qboot will automatically start using it.
Happy hacking!
Incidentally, I did something similar these days to get Linux booting in Jailhouse non-root cells, i.e without BIOS and almost no hardware except memory, cpus and pci devices. Yes, requires a bit pv for Linux, but really little. Not aiming for speed (yet), just for less hypervisor work. Maybe there are some milliseconds to save when cutting off more hardware in an analogous way...
PV pat^Whacks are here:
http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/jailhouse.
The boot loader is a combination of a python script [1] (result can be saved and reused - replaces ACPI) and really few lines of code [2][3].
Jan
[1]
https://github.com/siemens/jailhouse/blob/wip/linux-x86-inmate/tools/jailhouse-cell-linux
[2]
https://github.com/siemens/jailhouse/blob/wip/linux-x86-inmate/inmates/lib/x86/header.S
[3]
https://github.com/siemens/jailhouse/blob/wip/linux-x86-inmate/inmates/tools/x86/linux-loader.c
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 4 lines · Paolo Bonzini
On the QEMU side, there is no support yet for persistent memory and the NFIT tables from ACPI 6.0. Once that (and ACPI support) is added, qboot will automatically start using it.
We are working on adding NFIT support into virtual bios.
Quoted 7 lines · Yong Wang
Quoted 4 lines · Paolo Bonzini
On the QEMU side, there is no support yet for persistent memory and the NFIT tables from ACPI 6.0. Once that (and ACPI support) is added, qboot will automatically start using it.
We are working on adding NFIT support into virtual bios.
Great. I asked about this on the #pmem (irc.oftc.net) IRC channel last week.
Which virtual bios are you targeting?
Stefan
Quoted 7 lines · Stefan Hajnoczi
Quoted 6 lines · Yong Wang
Quoted 4 lines · Paolo Bonzini
On the QEMU side, there is no support yet for persistent memory and the NFIT tables from ACPI 6.0. Once that (and ACPI support) is added, qboot will automatically start using it.
Quoted 5 lines
Quoted 1 line
We are working on adding NFIT support into virtual bios.
Great. I asked about this on the #pmem (irc.oftc.net) IRC channel last week.
Which virtual bios are you targeting?
Ping?
Interest in persistent memory is picking up and I'd like to avoid duplicating work. Which pieces do you have patches for?
1. QEMU -device pmem,file=/path/to/dax/file,id=pmem1 and fw_cfg/ACPI info that gets passed to the guest
2. SeaBIOS NFIT ACPI table
3. ACPI NVDIMM DSM (probably not much needed, most features would be disabled)
Thanks,
Stefan
Quoted 13 lines · Paolo Bonzini
Some of you may have heard about the "Clear Containers" initiative from Intel, which couple KVM with various kernel tricks to create extremely lightweight virtual machines. The experimental Clear Containers setup requires only 18-20 MB to launch a virtual machine, and needs about 60 ms to boot.
Now, as all of you probably know, "QEMU is great for running Windows or legacy Linux guests, but that flexibility comes at a hefty price. Not only does all of the emulation consume memory, it also requires some form of low-level firmware in the guest as well. All of this adds quite a bit to virtual-machine startup times (500 to 700 milliseconds is not unusual)".
Quoted 5 lines
Right? In fact, it's for this reason that Clear Containers uses kvmtool instead of QEMU.
No, wrong! In fact, reporting bad performance is pretty much the same as throwing down the gauntlet.
On the QEMU side of things I wonder if there is scope for taking AArch64's 'virt' machine type concept and duplicating it on all architectures. It would be nice to have a common minimal machine type on all architectures that discards all legacy platform stuff and focuses on the minimum needed to run modern virtual machine optimized guest OS. People would always know that a machine type called 'virt' was the minimal virtualization platform, while the others all target emulation of realworld (legacy) baremetal platforms.
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Quoted 2 lines · Daniel P. Berrange
On the QEMU side of things I wonder if there is scope for taking AArch64's 'virt' machine type concept and duplicating it on all architectures.
Experience suggests that holding the line on "minimal" is really quite tricky, though -- there's always one more thing that somebody really wants to add...
-- PMM
Quoted 7 lines · Peter Maydell
Quoted 2 lines · Daniel P. Berrange
On the QEMU side of things I wonder if there is scope for taking AArch64's 'virt' machine type concept and duplicating it on all architectures.
Experience suggests that holding the line on "minimal" is really quite tricky, though -- there's always one more thing that somebody really wants to add...
Yep, it is hard saying no - but I'd think as long as it was possible to add the extra features using -device, it ought to be practical to keep a "virt" machine types "-nodefaults -nodefconfig" base setup pretty minimal. In particular I don't see why we need to have a SATA controller and ISA/LPC bridge in every virt machine - root PCI bus only should be possible, as you can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse, floppy via PCI devices and/or by adding a USB bus in the cases where you really need one.
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Quoted 3 lines · Daniel P. Berrange
Yep, it is hard saying no - but I'd think as long as it was possible to add the extra features using -device, it ought to be practical to keep a "virt" machine types "-nodefaults -nodefconfig" base setup pretty minimal.
Mmm, but -device only works for pluggable devices really. We don't have a coherent mechanism for saying "put the PS/2 keyboard controller into the system at its usual IO ports" on the command line.
-- PMM
Quoted 8 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrange
Yep, it is hard saying no - but I'd think as long as it was possible to add the extra features using -device, it ought to be practical to keep a "virt" machine types "-nodefaults -nodefconfig" base setup pretty minimal.
Mmm, but -device only works for pluggable devices really. We don't have a coherent mechanism for saying "put the PS/2 keyboard controller into the system at its usual IO ports" on the command line.
Oh, I didn't neccessarily mean that we'd need the ability to add a ps/2 keyboard via -device. I meant that there just need to be able to add /some/ kind of keyboard. eg we have a usb-kbd device that could potentially fill that role. Likewise for mouse pointer. Serial ports, etc.
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Quoted 8 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrange
Yep, it is hard saying no - but I'd think as long as it was possible to add the extra features using -device, it ought to be practical to keep a "virt" machine types "-nodefaults -nodefconfig" base setup pretty minimal.
Mmm, but -device only works for pluggable devices really. We don't have a coherent mechanism for saying "put the PS/2 keyboard controller into the system at its usual IO ports" on the command line.
Do we need that in the first place?
You can plugin a usb keyboard today.
You'll be able to plugin a virtio keyboard soon.
I think alot of hardware where this applies to is the legacy stuff we want to get rid of for '-M virt' ...
cheers,
Gerd
Quoted 8 lines · Peter Maydell
Quoted 3 lines · Daniel P. Berrange
Yep, it is hard saying no - but I'd think as long as it was possible to add the extra features using -device, it ought to be practical to keep a "virt" machine types "-nodefaults -nodefconfig" base setup pretty minimal.
Mmm, but -device only works for pluggable devices really. We don't have a coherent mechanism for saying "put the PS/2 keyboard controller into the system at its usual IO ports" on the command line.
... yet.
The qdev long-term vision has always been to support starting with an empty board, then add device models and their wiring. Unfortunately, progress towards that goal has been slow.
Paolo Bonzini
maintainerQuoted 6 lines · Daniel P. Berrange
In
particular I don't see why we need to have a SATA controller and ISA/LPC bridge in every virt machine - root PCI bus only should be possible, as you can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse, floppy via PCI devices and/or by adding a USB bus in the cases where you really need one.
I think removing the ISA/LPC bridge is hard. It includes the real-time clock and fw_cfg, for example.
Paolo
Quoted 11 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrange
In
particular I don't see why we need to have a SATA controller and ISA/LPC bridge in every virt machine - root PCI bus only should be possible, as you can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse, floppy via PCI devices and/or by adding a USB bus in the cases where you really need one.I think removing the ISA/LPC bridge is hard. It includes the real-time clock and
fw_cfg, for example.
Could VirtIO specified replacements make sense for these peripherals?
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Paolo Bonzini
maintainerQuoted 10 lines · Christopher Covington
Quoted 9 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrange
In
particular I don't see why we need to have a SATA controller and ISA/LPC bridge in every virt machine - root PCI bus only should be possible, as you can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse, floppy via PCI devices and/or by adding a USB bus in the cases where you really need one.
Quoted 4 lines
Quoted 2 lines
I think removing the ISA/LPC bridge is hard. It includes the real-time clock and
fw_cfg, for example.Could VirtIO specified replacements make sense for these peripherals?
Not really. virtio is too heavyweight and you'd be reinventing the wheel unnecessarily.
For example, ARM's "-M virt" uses a pl011 block for the RTC, and also uses fw_cfg. Another commonly used ISA device is the UART, for which again -M virt uses a pl031.
Paolo
Quoted 12 lines · Paolo Bonzini
Quoted 10 lines · Christopher Covington
Quoted 9 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrange
In
particular I don't see why we need to have a SATA controller and ISA/LPC bridge in every virt machine - root PCI bus only should be possible, as you can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse, floppy via PCI devices and/or by adding a USB bus in the cases where you really need one.I think removing the ISA/LPC bridge is hard. It includes the real-time clock and
fw_cfg, for example.
Quoted 8 lines
Quoted 1 line
Could VirtIO specified replacements make sense for these peripherals?
Not really. virtio is too heavyweight and you'd be reinventing the wheel unnecessarily.
For example, ARM's "-M virt" uses a pl011 block for the RTC, and also uses
fw_cfg. Another commonly used ISA device is the UART, for which again -M virt uses a pl031.
The RTC can be replaced by kvmclock, the keyboard by virtio-console. Maybe we can provide an msr- or pci- based interface to fw_cfg.
Paolo Bonzini
maintainerQuoted 23 lines · Avi Kivity
Quoted 20 lines · Paolo Bonzini
Quoted 12 lines · Christopher Covington
Quoted 11 lines · Paolo Bonzini
Quoted 10 lines · Daniel P. Berrange
In
particular I don't see why we need to have a SATA controller and ISA/LPC
bridge in every virt machine - root PCI bus only should be possible, as you
can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse,
floppy via PCI devices and/or by adding a USB bus in the cases where you
really need one.Quoted 3 lines
Quoted 2 lines
I think removing the ISA/LPC bridge is hard. It includes the real-time clock and
fw_cfg, for example.Could VirtIO specified replacements make sense for these peripherals?
Not really. virtio is too heavyweight and you'd be reinventing the wheel unnecessarily.
Quoted 7 lines
Quoted 4 lines
For example, ARM's "-M virt" uses a pl011 block for the RTC, and also uses
fw_cfg. Another commonly used ISA device is the UART, for which again -M virt uses a pl031.The RTC can be replaced by kvmclock, the keyboard by virtio-console. Maybe we can provide an msr- or pci- based interface to
fw_cfg.
The RTC is used for more than clock unfortunately. S3 support uses it for example, both to tell the firmware that it's an S3 resume and for resuming when the alarm fires.
All in all, getting rid of ISA seems like chasing windmills. If you want to do that, fine, but then do not do that on a minimal firmware like qboot or SeaBIOS. For example, UEFI provides run-time services that let you access some low-level devices like this, and that's part of why the ARM virtual machine image specification mandates UEFI support. But even then, the ARM virtual machine image specification lets you choose between pl031 and a Xen pv console, and doesn't specify virtio-console...
Paolo
Paolo
Quoted 3 lines · Paolo Bonzini
For example, ARM's "-M virt" uses a pl011 block for the RTC, and also uses
fw_cfg. Another commonly used ISA device is the UART, for which again -M virt uses a pl031.
Partly we do that because there were a number of reports that trying to use virtio for the console didn't work reliably... Using the stock UART that is widely supported in UEFI/uboot/kernel was a conservative design choice.
The next thing that's likely to appear in "virt" is a PL061 GPIO device, which you need for CPU hotplug and external-shutdown-request notifications.
-- PMM
Paolo Bonzini
maintainerQuoted 4 lines · Peter Maydell
Quoted 3 lines · Paolo Bonzini
Quoted 3 lines
For example, ARM's "-M virt" uses a pl011 block for the RTC, and also uses
fw_cfg. Another commonly used ISA device is the UART, for which again -M virt uses a pl031.
Quoted 8 lines
Partly we do that because there were a number of reports that trying to use virtio for the console didn't work reliably... Using the stock UART that is widely supported in UEFI/uboot/kernel was a conservative design choice.
The next thing that's likely to appear in "virt" is a PL061 GPIO device, which you need for CPU hotplug and external-shutdown-request notifications.
Indeed, and the x86 Q35 chipset puts the ACPI registers... in the ISA/LPC bridge. :)
Paolo
Quoted 13 lines · Paolo Bonzini
Quoted 10 lines · Christopher Covington
Quoted 9 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrange
In
particular I don't see why we need to have a SATA controller and ISA/LPC bridge in every virt machine - root PCI bus only should be possible, as you can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse, floppy via PCI devices and/or by adding a USB bus in the cases where you really need one.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
I think removing the ISA/LPC bridge is hard. It includes the real-time clock and
fw_cfg, for example.Could VirtIO specified replacements make sense for these peripherals?
Not really. virtio is too heavyweight
I'd be curious to read where in your estimation this weight lies. Is it one-time initialization or recurring? Is it specific to the PCI transport or does MMIO suffer from it as well?
Quoted 1 line
and you'd be reinventing the wheel unnecessarily.
In my mind the utility of peripherals that are instruction set architecture agnostic and can work over several transports is in reducing the amount of (emulator/hypervisor, firmware, and OS) code used, and therefore in need of maintenance, for common system emulation and virtualization use cases.
Quoted 3 lines
For example, ARM's "-M virt" uses a pl011 block for the RTC, and also uses
fw_cfg. Another commonly used ISA device is the UART, for which again -M virt uses a pl031.
(UART is PL011; RTC is PL031)
Thanks,
Chris
Signature · 3 lines
Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Paolo Bonzini
maintainerQuoted 5 lines · Christopher Covington
Quoted 1 line
Not really. virtio is too heavyweight
I'd be curious to read where in your estimation this weight lies. Is it one-time initialization or recurring? Is it specific to the PCI transport or does MMIO suffer from it as well?
It's heavyweight in the sense that virtio requires you to design a communication mechanism based on ring buffers. It's much harder than a few ad-hoc registers.
And I know everyone is upset about the attack surface of QEMU these days, but effort would be much better spent adding QEMU-specific customizations to a static analysis tool (that e.g. would derive bound checks for the MemoryRegion read/write ops and be able to prove that said ops cannot access arrays out of their bounds).
Quoted 6 lines
Quoted 1 line
and you'd be reinventing the wheel unnecessarily.
In my mind the utility of peripherals that are instruction set architecture agnostic and can work over several transports is in reducing the amount of (emulator/hypervisor, firmware, and OS) code used, and therefore in need of maintenance, for common system emulation and virtualization use cases.
A fully processor-agnostic hardware architecture is a non-goal. You'll always have stuff like interrupt controllers that is extremely tied to the processor.
If you want to abstract hardware, use the firmware, Luke! Things such as UEFI and ACPI are there for exactly this reason. We will be able to reuse a lot of x86 hotplug code on ARM using ACPI. And if you don't want to use ACPI, you can always write native OS drivers for the same hotplug hardware.
Paolo
Quoted 13 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 10 lines · Christopher Covington
Quoted 9 lines · Paolo Bonzini
Quoted 6 lines · Daniel P. Berrange
In
particular I don't see why we need to have a SATA controller and ISA/LPC bridge in every virt machine - root PCI bus only should be possible, as you can provide disks via virtio-blk or virtio-scsi and serial, parallel, mouse, floppy via PCI devices and/or by adding a USB bus in the cases where you really need one.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I think removing the ISA/LPC bridge is hard. It includes the real-time clock and
fw_cfg, for example.Could VirtIO specified replacements make sense for these peripherals?
Not really. virtio is too heavyweight and you'd be reinventing the wheel unnecessarily.
I see reasons to replace some but not all these components; and there's no point in replacing the ISA/LPC bridge since it's got nothing at all in it.
Quoted 3 lines
For example, ARM's "-M virt" uses a pl011 block for the RTC, and also uses
fw_cfg. Another commonly used ISA device is the UART, for which again -M virt uses a pl031.
I don't see much point in replacing the simple PC uart with anything virtio; I can imagine that you might want to go down to something really trivial with non of the bells and whistles; but a UART is pretty simple.
The PC RTC though, it's a bit of a disaster that's had 30 years of random cruft added into it to hold random things that should never have been there.
Dave
Quoted 3 lines
Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Hi,
Quoted 1 line
qboot is available at
git://github.com/bonzini/qboot.git.
Firmware repo has packages now.
https://www.kraxel.org/repos/firmware.repo
https://www.kraxel.org/repos/jenkins/qboot/
enjoy,
Gerd
Quoted 16 lines · Paolo Bonzini
Some of you may have heard about the "Clear Containers" initiative from Intel, which couple KVM with various kernel tricks to create extremely lightweight virtual machines. The experimental Clear Containers setup requires only 18-20 MB to launch a virtual machine, and needs about 60 ms to boot.
Now, as all of you probably know, "QEMU is great for running Windows or legacy Linux guests, but that flexibility comes at a hefty price. Not only does all of the emulation consume memory, it also requires some form of low-level firmware in the guest as well. All of this adds quite a bit to virtual-machine startup times (500 to 700 milliseconds is not unusual)".
Right? In fact, it's for this reason that Clear Containers uses kvmtool instead of QEMU.
Quoted 6 lines
No, wrong! In fact, reporting bad performance is pretty much the same as throwing down the gauntlet.
Enter qboot, a minimal x86 firmware that runs on QEMU and, together with a slimmed-down QEMU configuration, boots a virtual machine in 40 milliseconds[2] on an Ivy Bridge Core i7 processor.
Hi Paolo,
I'm curious if you've tried profiling SeaBIOS to see where it is spending unnecessary time? I wonder if a stripped down SeaBIOS could obtain sufficient performance.
The page at http://seabios.org/Debugging#Timing_debug_messages
describes how to do basic profiling via timing of debug messages.
The default SeaBIOS build takes ~180ms on my (old AMD) system. But, by removing drivers and options via Kconfig I was able to bring it down to ~25ms. I suspect some additional Kconfig settings and a few optimizations would make it possible to significantly reduce this time.
Cheers,
-Kevin
Quoted 30 lines · Kevin O'Connor
Quoted 16 lines · Paolo Bonzini
Some of you may have heard about the "Clear Containers" initiative from Intel, which couple KVM with various kernel tricks to create extremely lightweight virtual machines. The experimental Clear Containers setup requires only 18-20 MB to launch a virtual machine, and needs about 60 ms to boot.
Now, as all of you probably know, "QEMU is great for running Windows or legacy Linux guests, but that flexibility comes at a hefty price. Not only does all of the emulation consume memory, it also requires some form of low-level firmware in the guest as well. All of this adds quite a bit to virtual-machine startup times (500 to 700 milliseconds is not unusual)".
Right? In fact, it's for this reason that Clear Containers uses kvmtool instead of QEMU.
Quoted 6 lines
No, wrong! In fact, reporting bad performance is pretty much the same as throwing down the gauntlet.
Enter qboot, a minimal x86 firmware that runs on QEMU and, together with a slimmed-down QEMU configuration, boots a virtual machine in 40 milliseconds[2] on an Ivy Bridge Core i7 processor.
Hi Paolo,
I'm curious if you've tried profiling SeaBIOS to see where it is spending unnecessary time? I wonder if a stripped down SeaBIOS could obtain sufficient performance.
Quoted 8 lines
The page at http://seabios.org/Debugging#Timing_debug_messages
describes how to do basic profiling via timing of debug messages.The default SeaBIOS build takes ~180ms on my (old AMD) system. But, by removing drivers and options via
KconfigI was able to bring it down to ~25ms. I suspect some additionalKconfigsettings and a few optimizations would make it possible to significantly reduce this time.
Out of curiosity, I ran some additional timing tests. With SeaBIOS fully stripped down (via Kconfig), it takes ~20ms to get to the boot phase on my old AMD system. Of that 20ms, ~7ms is to enable shadow ram, 2ms is to calibrate the cpu timestamp counter, 4ms is for pci init, and ~6ms is to make the shadow ram area read-only. The time in the remaining parts of the SeaBIOS code is so small that it's hard to measure.
The above delays could be removed without much work: It would be trivial to add kconfig options to support disabling the pci init and shadow read-only functionality. The cpu timestamp calibrations could be reworked so that the existence of the ACPI timer was checked first so as to avoid that work. The enabling of shadow ram could also be bypassed if QEMU was changed to start with a ram based shadow instead of a rom shadow (since QEMU doesn't fully implement the chipset shadow registers, it's really pointless to start in rom shadow mode anyway).
-Kevin
2015-05-23 6:55 GMT+03:00 Kevin O'Connor <[email protected]>:
Quoted 7 lines
Out of curiosity, I ran some additional timing tests. With SeaBIOS fully stripped down (via
Kconfig), it takes ~20ms to get to the boot phase on my old AMD system. Of that 20ms, ~7ms is to enable shadow ram, 2ms is to calibrate the cpu timestamp counter, 4ms is for pci init, and ~6ms is to make the shadow ram area read-only. The time in the remaining parts of the SeaBIOS code is so small that it's hard to measure.
Can you share config for seabios? As i understand i can safety to remove keybord, ps2, usb, ata/ahci and leave only virtio (in case of using qemu to boo linux/freebsd/windows systems) ?
Signature · 2 lines
Vasiliy Tolstov, e-mail: [email protected]
Quoted 9 lines · Vasiliy Tolstov
2015-05-23 6:55 GMT+03:00 Kevin O'Connor <[email protected]>:
Quoted 7 lines
Out of curiosity, I ran some additional timing tests. With SeaBIOS fully stripped down (via
Kconfig), it takes ~20ms to get to the boot phase on my old AMD system. Of that 20ms, ~7ms is to enable shadow ram, 2ms is to calibrate the cpu timestamp counter, 4ms is for pci init, and ~6ms is to make the shadow ram area read-only. The time in the remaining parts of the SeaBIOS code is so small that it's hard to measure.
Quoted 3 lines
Can you share config for seabios? As i understand i can safety to remove keybord, ps2, usb, ata/ahci and leave only virtio (in case of using qemu to boo linux/freebsd/windows systems) ?
To get to 20ms, I basically disabled everything. (Actually, after sending my last email I found I cound get to 16ms by setting CONFIG_WRITABLE_UPPERMEMORY=y - it avoids much of the shadow ram read-only cost.) It's not possible to actually boot anything with this config - I was instead timing to the point where one could add a multiboot or vmlinux boot capability to SeaBIOS.
To get to these times I disabled debug messages themselves (they take a few ms of the boot time). So, to time events without debug messages I added a few outb() calls to seabios at critical points. See patch and config below.
-Kevin
src/post.c +8 -0
@@ -167,11 +167,15 @@ platform_hardware_setup(void)// Init base pc hardware. pic_setup ();mathcp_setup (); +outb ('3' ,0x402 );outb ('\n' ,0x402 );timer_setup (); +outb ('4' ,0x402 );outb ('\n' ,0x402 );clock_setup ();// Platform specific setup +outb ('5' ,0x402 );outb ('\n' ,0x402 );qemu_platform_setup (); +outb ('6' ,0x402 );outb ('\n' ,0x402 );coreboot_platform_setup (); }
@@ -200,6 +204,7 @@ startBoot(void)// Clear low-memory allocations (required by PMM spec). memset ((void *)BUILD_STACK_ADDR ,0 ,BUILD_EBDA_MINIMUM -BUILD_STACK_ADDR ); +outb ('8' ,0x402 );outb ('\n' ,0x402 );dprintf (3 ,"Jump to int19\n" );struct bregs br;memset (&br,0 ,sizeof (br));
@@ -247,6 +252,7 @@ maininit(void)prepareboot ();// Write protect bios memory. +outb ('7' ,0x402 );outb ('\n' ,0x402 );make_bios_readonly ();// Invoke int 19 to start boot process.
@@ -331,12 +337,14 @@ handle_post(void)serial_debug_preinit ();debug_banner (); +outb ('1' ,0x402 );outb ('\n' ,0x402 );// Check if we are running under Xen. xen_preinit ();// Allow writes to modify bios area (0xf0000) make_bios_writable (); +outb ('2' ,0x402 );outb ('\n' ,0x402 );// Now that memory is read/writable - start post process. dopost ();
#
# Automatically generated file; DO NOT EDIT.
# SeaBIOS Configuration
#
#
# General Features
#
# CONFIG_COREBOOT is not setCONFIG_QEMU=y
# CONFIG_CSM is not setCONFIG_QEMU_HARDWARE=y
# CONFIG_XEN is not set
# CONFIG_THREADS is not set
# CONFIG_RELOCATE_INIT is not set
# CONFIG_BOOTMENU is not set
# CONFIG_BOOTORDER is not setCONFIG_ENTRY_EXTRASTACK=yCONFIG_MALLOC_UPPERMEMORY=yCONFIG_ROM_SIZE=0
#
# Hardware support
#
# CONFIG_ATA is not set
# CONFIG_AHCI is not set
# CONFIG_SDCARD is not set
# CONFIG_VIRTIO_BLK is not set
# CONFIG_VIRTIO_SCSI is not set
# CONFIG_PVSCSI is not set
# CONFIG_ESP_SCSI is not set
# CONFIG_LSI_SCSI is not set
# CONFIG_MEGASAS is not set
# CONFIG_FLOPPY is not set
# CONFIG_PS2PORT is not set
# CONFIG_USB is not set
# CONFIG_SERIAL is not set
# CONFIG_LPT is not set
# CONFIG_USE_SMM is not setCONFIG_MTRR_INIT=yCONFIG_PMTIMER=y
#
# BIOS interfaces
#CONFIG_DRIVES=yCONFIG_CDROM_BOOT=yCONFIG_CDROM_EMU=yCONFIG_PCIBIOS=yCONFIG_APMBIOS=yCONFIG_PNPBIOS=y
# CONFIG_OPTIONROMS is not setCONFIG_BOOT=yCONFIG_KEYBOARD=yCONFIG_KBD_CALL_INT15_4F=yCONFIG_MOUSE=yCONFIG_S3_RESUME=y
# CONFIG_VGAHOOKS is not set
# CONFIG_DISABLE_A20 is not setCONFIG_WRITABLE_UPPERMEMORY=y
# CONFIG_TCGBIOS is not set
#
# BIOS Tables
#
# CONFIG_PIRTABLE is not set
# CONFIG_MPTABLE is not set
# CONFIG_SMBIOS is not set
# CONFIG_ACPI is not set
# CONFIG_FW_ROMFILE_LOAD is not set
#
# VGA ROM
#CONFIG_NO_VGABIOS=y
# CONFIG_VGA_STANDARD_VGA is not set
# CONFIG_VGA_CIRRUS is not set
# CONFIG_VGA_BOCHS is not set
# CONFIG_VGA_GEODEGX2 is not set
# CONFIG_VGA_GEODELX is not set
# CONFIG_BUILD_VGABIOS is not setCONFIG_VGA_FIXUP_ASM=yCONFIG_VGA_EXTRA_STACK_SIZE=512
#
# Debugging
#CONFIG_DEBUG_LEVEL=0
Paolo Bonzini
maintainerQuoted 4 lines · Kevin O'Connor
Hi Paolo,
I'm curious if you've tried profiling SeaBIOS to see where it is spending unnecessary time?
No, I really wanted to get the absolute minimum time needed to get to the kernel. I announced it publicly because I think it's also instructive.
Quoted 2 lines
I wonder if a stripped down SeaBIOS could
obtain sufficient performance.
It most likely could. However, some of the features (e.g. booting vmlinuz from parallel flash or, in the future, from persistent memory) probably do not fit SeaBIOS very well. One would have to redo them otherwise, for example using a DMA interface in fw_cfg.
Paolo
Quoted 16 lines
The page at http://seabios.org/Debugging#Timing_debug_messages
describes how to do basic profiling via timing of debug messages.The default SeaBIOS build takes ~180ms on my (old AMD) system. But, by removing drivers and options via
KconfigI was able to bring it down to ~25ms. I suspect some additionalKconfigsettings and a few optimizations would make it possible to significantly reduce this time.Cheers,
-Kevin
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Paolo Bonzini
Quoted 2 lines · Kevin O'Connor
I'm curious if you've tried profiling SeaBIOS to see where it is spending unnecessary time?
No, I really wanted to get the absolute minimum time needed to get to the kernel. I announced it publicly because I think it's also instructive.
Quoted 7 lines
Quoted 2 lines
I wonder if a stripped down SeaBIOS could
obtain sufficient performance.It most likely could. However, some of the features (e.g. booting vmlinuz from parallel flash or, in the future, from persistent memory) probably do not fit SeaBIOS very well. One would have to redo them otherwise, for example using a DMA interface in
fw_cfg.
I think adding vmlinuz and/or multiboot support to SeaBIOS would be generally useful (and not difficult).
If a kernel was placed in memory (or memory-mapped flash) then I think it would be reasonable to have SeaBIOS obtain its address from (an unoptimized) fw_cfg and deploy it.
-Kevin