RFC Towards an Heterogeneous QEMU
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest8 patches 10 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/8 | backend: multi-socketbackends/ backends/ include/ | +479 | 0 replies | |
| 2/8 | backend: shared memory backendbackends/ backends/ include/ | +256 | 0 replies | |
| 3/8 | migration: add shared migration typebackends/ include/ migration/ +2 more | +46 | 0 replies | |
| 4/8 | hw/misc: IDM Devicedefault-configs/ default-configs/ default-configs/ +3 more | +540 | 0 replies | |
| 5/8 | hw/arm: sysbus-fdthw/ | +60 | 0 replies | |
| 6/8 | qemu: slave machine flaghw/ include/ qemu-options.hx +1 more | +38 | 0 replies | |
| 7/8 | hw/arm: boothw/ | +13 | 0 replies | |
| 8/8 | qemu: numanuma.c | +16 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [Qemu-devel] [RFC PATCH 0/8] Towards an Heterogeneous QEMU
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".
To recap the goal of such RFC:
The idea is to enhance the current architecture of QEMU to enable the modeling of a state of-the-art SoC with an AMP processing style, where different processing units share the same system memory and communicate through shared memory and inter-processor interrupts. An example is a multi-core ARM CPU working alongside with two Cortex-M micro controllers.
Quoted 1 line
From the user point of view there is usually an operating system booting on
the Master processor (e.g. Linux) at platform startup, while the other processors are used to offload the Master one from some computation or to deal with real-time interfaces. It is the Master OS that triggers the boot of the Slave processors, and provides them also the binary code to execute (e.g. RTOS, binary firmware) by placing it into a pre-defined memory area that is accessible to the Slaves. Usually the memory for the Slaves is carved out from the Master OS during boot. Once a Slave is booted the two processors can communicate through queues in shared memory and inter-processor interrupts (IPIs). In Linux, it is the remoteproc/rpmsg framework that enables the control (boot/shutdown) of Slave processors, and also to establish a communication channel based on virtio queues.
Currently, QEMU is not able to model such an architecture mainly because only a single processor can be emulated at one time, and the OS binary image needs to be placed in memory at model startup.
This patch series adds a set of modules and introduces minimal changes to the current QEMU code-base to implement what described above, with master and slave implemented as two different instances of QEMU. The aim of this work is to enable application and runtime programmers to test their AMP applications, or their new inter-SoC communtication protocol.
The main changes are depicted in the following diagram and involve:
- A new multi-client socket implementation that allows multiple instances of QEMU to attach to the same socket, with only one acting as a master.
- A new memory backend, the shared memory backend, based on
the file memory backend. Such new backend enables, on the master side, to allocate the whole memory as shareable (e.g. /dev/shm, or hugetlbfs). On the slave side it enables the startup of QEMU without any main memory allocated. The the slave goes in a waiting state, the same used in the case of an incoming migration, and a callback is registered on a multi-client socket shared with the master.
The waiting state ends when the master sends to the slave the file descriptor and offset to mmap and use as memory.
- A new inter-processor interrupt hardware distribution module, that is used also to trigger the boot of slave processors. Such module uses a pair of eventfd for each master-slave couple to trigger interrupts between the instances. No slave-to-slave interrupts are envisioned by the current implementation. The multi client-socket is used for the master to trigger the boot of a slave, and also for each master-slave couple to exchancge the eventd file descriptors. The IDM device can be instantiated either as a PCI or sysbus device.
Memory
(e.g. hugetlbfs)
+------------------+ +--------------+ +------------------+
| | | | | |
| QEMU MASTER | | Master | | QEMU SLAVE |
| | | Memory | | |
| +------+ +------+-+ | | +-+------+ +------+ |
| | | |SHMEM | | | |SHMEM | | | |
| | VCPU | |Backend +-----> | +----->Backend | | VCPU | |
| | | | | | | | +---> | | | |
| +--^---+ +------+-+ | | | | +-+------+ +--^---+ |
| | | | | | | | | |
| +--+ | | | | | | +---+ |
| | IRQ | | +----------+ | | | | IRQ | |
| | | | | | | | | | | |
| +----+----+ | | | Slave <------+ | | +----+---+ |
+--+ IDM +-----+ | | Memory | | | +---+ IDM +-----+
+-^----^--+ | | | | | +-^---^--+
| | | +----------+ | | | |
| | +--------------+ | | |
| | | | |
| +--------------------------------------+-----------+ |
| UNIX Domain Socket(send mem fd + offset, trigger boot) |
| |
+-----------------------------------------------------------+ eventfd
The whole code can be checked out from:
https://git.virtualopensystems.com/dev/qemu-het.git
branch:
qemu-het-rfc-v1
Patches apply to the current QEMU master branch
=========
Demo
=========
This patch series comes in the form of a demo to better understand how the changes introduced can be exploited.
At the current status the demo can be executed using an ARM target for both master and slave.
The demo shows how a master QEMU instance carves out the memory for a slave, copies inside linux kernel image and device tree blob and finally triggers the boot.
How to reproduce the demo:
In order to reproduce the demo a couple more extra elements need to be downloaded and compiled.
Binary loader
Loads the slave firmware (kernel) binary into memory and triggers the boot
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
load-bin-boot
To compile: just type "make"
Slave kernel
Compile a linux kernel image (zImage) for the virt machine model.
IDM test kernel module
Needed to trigger the boot of a slave
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
IDM-kernel-module
To compile: KDIR=kernel_path ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make
Slave DTB
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
slave-dtb
Copy binary loader, IDM kernel module, zImage and dtb inside the disk image or ramdisk of the master instance.
Run the demo:
run the master instance
./arm-softmmu/qemu-system-arm \
-kernel zImage \
-M virt -cpu cortex-a15 \
-drive if=none,file=disk.img,cache=writeback,id=foo1 \
-device virtio-blk-device,drive=foo1 \
-object multi-socket-backend,id=foo,listen,path=ms_socket \
-object memory-backend-shared,id=mem,size=1G,mem-path=/mnt/hugetlbfs,chardev=foo,master=on,prealloc=on \
-device idm_ipi,master=true,memdev=mem,socket=foo \
-numa node,memdev=mem -m 1G \
-append "root=/dev/vda rw console=ttyAMA0 mem=512M memmap=512M$0x60000000" \
-nographicrun the slave instance
./arm-softmmu/qemu-system-arm\
-M virt -cpu cortex-a15 -machine slave=on \
-drive if=none,file=disk.img,cache=writeback,id=foo1 \
-device virtio-blk-device,drive=foo1 \
-object multi-socket-backend,id=foo,path=ms_socket \
-object memory-backend-shared,id=mem,size=512M,mem-path=/mnt/hugetlbfs,chardev=foo,master=off \
-device idm_ipi,master=false,memdev=mem,socket=foo \
-incoming "shared:mem" -numa node,memdev=mem -m 512M \
-nographicFor simplicity, use a disk image for the slave instead of a ramdisk.
As visible from the kernel boot arguments, the master is booted with mem=512 so that one half of the whole memory allocated is not used by the master and reserved for the slave. Such memory starts for the virt platform from address 0x60000000.
Once the master is booted the image of the kernel and DTB can be copied in the memory carved out for the slave.
In the maser console
probe the IDM kernel module:
$ insmod idm_test_mod.ko
run the application that copies the binaries into memory and triggers the boot:
$ ./load_bin_app 1 ./zImage ./slave.dtb
On the slave console the linux kernel boot should be visible.
The present demo is intended only as a demonstration to see the patch-set at work. In the near future, boot triggering, memory carveout and binary copy might be implemented in a remoteproc driver coupled with a RPMSG driver for communication between master and slave instance.
This work has been sponsored by Huawei Technologies Duesseldorf GmbH.
Baptiste Reynal (3):
backend: multi-socket
backend: shared memory backend
migration: add shared migration type
Christian Pinto (5):
hw/misc: IDM Device
hw/arm: sysbus-fdt
qemu: slave machine flag
hw/arm: boot
qemu: numa
backends/Makefile.objs | 4 +- backends/hostmem-shared.c | 203 ++++++++++++++++++ backends/multi-socket.c | 353 +++++++++++++++++++++++++++++++ default-configs/arm-softmmu.mak | 1 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + hw/arm/boot.c | 13 ++ hw/arm/sysbus-fdt.c | 60 ++++++ hw/core/machine.c | 27 +++ hw/misc/Makefile.objs | 2 + hw/misc/idm.c | 416 +++++++++++++++++++++++++++++++++++++ include/hw/boards.h | 2 + include/hw/misc/idm.h | 119 +++++++++++ include/migration/migration.h | 2 + include/qemu/multi-socket.h | 124 +++++++++++ include/sysemu/hostmem-shared.h | 61 ++++++ migration/Makefile.objs | 2 +- migration/migration.c | 2 + migration/shared.c | 32 +++ numa.c | 17 +- qemu-options.hx | 5 +- util/qemu-config.c | 5 + 22 files changed, 1448 insertions(+), 4 deletions(-) create mode 100644 backends/hostmem-shared.c create mode 100644 backends/multi-socket.c create mode 100644 hw/misc/idm.c create mode 100644 include/hw/misc/idm.h create mode 100644 include/qemu/multi-socket.h create mode 100644 include/sysemu/hostmem-shared.h create mode 100644 migration/shared.c
Signature · 1 line
1.9.1