Towards an Heterogeneous QEMU
cover Towards an Heterogeneous QEMU
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".
To recap the goal of such RFC:
The idea is to enhance the current architecture of QEMU to enable the modeling of a state of-the-art SoC with an AMP processing style, where different processing units share the same system memory and communicate through shared memory and inter-processor interrupts. An example is a multi-core ARM CPU working alongside with two Cortex-M micro controllers.
Quoted 1 line
From the user point of view there is usually an operating system booting on
the Master processor (e.g. Linux) at platform startup, while the other processors are used to offload the Master one from some computation or to deal with real-time interfaces. It is the Master OS that triggers the boot of the Slave processors, and provides them also the binary code to execute (e.g. RTOS, binary firmware) by placing it into a pre-defined memory area that is accessible to the Slaves. Usually the memory for the Slaves is carved out from the Master OS during boot. Once a Slave is booted the two processors can communicate through queues in shared memory and inter-processor interrupts (IPIs). In Linux, it is the remoteproc/rpmsg framework that enables the control (boot/shutdown) of Slave processors, and also to establish a communication channel based on virtio queues.
Currently, QEMU is not able to model such an architecture mainly because only a single processor can be emulated at one time, and the OS binary image needs to be placed in memory at model startup.
This patch series adds a set of modules and introduces minimal changes to the current QEMU code-base to implement what described above, with master and slave implemented as two different instances of QEMU. The aim of this work is to enable application and runtime programmers to test their AMP applications, or their new inter-SoC communtication protocol.
The main changes are depicted in the following diagram and involve:
- A new multi-client socket implementation that allows multiple instances of QEMU to attach to the same socket, with only one acting as a master.
- A new memory backend, the shared memory backend, based on
the file memory backend. Such new backend enables, on the master side, to allocate the whole memory as shareable (e.g. /dev/shm, or hugetlbfs). On the slave side it enables the startup of QEMU without any main memory allocated. The the slave goes in a waiting state, the same used in the case of an incoming migration, and a callback is registered on a multi-client socket shared with the master.
The waiting state ends when the master sends to the slave the file descriptor and offset to mmap and use as memory.
- A new inter-processor interrupt hardware distribution module, that is used also to trigger the boot of slave processors. Such module uses a pair of eventfd for each master-slave couple to trigger interrupts between the instances. No slave-to-slave interrupts are envisioned by the current implementation. The multi client-socket is used for the master to trigger the boot of a slave, and also for each master-slave couple to exchancge the eventd file descriptors. The IDM device can be instantiated either as a PCI or sysbus device.
Memory
(e.g. hugetlbfs)
+------------------+ +--------------+ +------------------+
| | | | | |
| QEMU MASTER | | Master | | QEMU SLAVE |
| | | Memory | | |
| +------+ +------+-+ | | +-+------+ +------+ |
| | | |SHMEM | | | |SHMEM | | | |
| | VCPU | |Backend +-----> | +----->Backend | | VCPU | |
| | | | | | | | +---> | | | |
| +--^---+ +------+-+ | | | | +-+------+ +--^---+ |
| | | | | | | | | |
| +--+ | | | | | | +---+ |
| | IRQ | | +----------+ | | | | IRQ | |
| | | | | | | | | | | |
| +----+----+ | | | Slave <------+ | | +----+---+ |
+--+ IDM +-----+ | | Memory | | | +---+ IDM +-----+
+-^----^--+ | | | | | +-^---^--+
| | | +----------+ | | | |
| | +--------------+ | | |
| | | | |
| +--------------------------------------+-----------+ |
| UNIX Domain Socket(send mem fd + offset, trigger boot) |
| |
+-----------------------------------------------------------+ eventfd
The whole code can be checked out from:
https://git.virtualopensystems.com/dev/qemu-het.git
branch:
qemu-het-rfc-v1
Patches apply to the current QEMU master branch
=========
Demo
=========
This patch series comes in the form of a demo to better understand how the changes introduced can be exploited.
At the current status the demo can be executed using an ARM target for both master and slave.
The demo shows how a master QEMU instance carves out the memory for a slave, copies inside linux kernel image and device tree blob and finally triggers the boot.
How to reproduce the demo:
In order to reproduce the demo a couple more extra elements need to be downloaded and compiled.
Binary loader
Loads the slave firmware (kernel) binary into memory and triggers the boot
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
load-bin-boot
To compile: just type "make"
Slave kernel
Compile a linux kernel image (zImage) for the virt machine model.
IDM test kernel module
Needed to trigger the boot of a slave
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
IDM-kernel-module
To compile: KDIR=kernel_path ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make
Slave DTB
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
slave-dtb
Copy binary loader, IDM kernel module, zImage and dtb inside the disk image or ramdisk of the master instance.
Run the demo:
run the master instance
./arm-softmmu/qemu-system-arm \
-kernel zImage \
-M virt -cpu cortex-a15 \
-drive if=none,file=disk.img,cache=writeback,id=foo1 \
-device virtio-blk-device,drive=foo1 \
-object multi-socket-backend,id=foo,listen,path=ms_socket \
-object memory-backend-shared,id=mem,size=1G,mem-path=/mnt/hugetlbfs,chardev=foo,master=on,prealloc=on \
-device idm_ipi,master=true,memdev=mem,socket=foo \
-numa node,memdev=mem -m 1G \
-append "root=/dev/vda rw console=ttyAMA0 mem=512M memmap=512M$0x60000000" \
-nographicrun the slave instance
./arm-softmmu/qemu-system-arm\
-M virt -cpu cortex-a15 -machine slave=on \
-drive if=none,file=disk.img,cache=writeback,id=foo1 \
-device virtio-blk-device,drive=foo1 \
-object multi-socket-backend,id=foo,path=ms_socket \
-object memory-backend-shared,id=mem,size=512M,mem-path=/mnt/hugetlbfs,chardev=foo,master=off \
-device idm_ipi,master=false,memdev=mem,socket=foo \
-incoming "shared:mem" -numa node,memdev=mem -m 512M \
-nographicFor simplicity, use a disk image for the slave instead of a ramdisk.
As visible from the kernel boot arguments, the master is booted with mem=512 so that one half of the whole memory allocated is not used by the master and reserved for the slave. Such memory starts for the virt platform from address 0x60000000.
Once the master is booted the image of the kernel and DTB can be copied in the memory carved out for the slave.
In the maser console
probe the IDM kernel module:
$ insmod idm_test_mod.ko
run the application that copies the binaries into memory and triggers the boot:
$ ./load_bin_app 1 ./zImage ./slave.dtb
On the slave console the linux kernel boot should be visible.
The present demo is intended only as a demonstration to see the patch-set at work. In the near future, boot triggering, memory carveout and binary copy might be implemented in a remoteproc driver coupled with a RPMSG driver for communication between master and slave instance.
This work has been sponsored by Huawei Technologies Duesseldorf GmbH.
Baptiste Reynal (3):
backend: multi-socket
backend: shared memory backend
migration: add shared migration type
Christian Pinto (5):
hw/misc: IDM Device
hw/arm: sysbus-fdt
qemu: slave machine flag
hw/arm: boot
qemu: numa
backends/Makefile.objs | 4 +- backends/hostmem-shared.c | 203 ++++++++++++++++++ backends/multi-socket.c | 353 +++++++++++++++++++++++++++++++ default-configs/arm-softmmu.mak | 1 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + hw/arm/boot.c | 13 ++ hw/arm/sysbus-fdt.c | 60 ++++++ hw/core/machine.c | 27 +++ hw/misc/Makefile.objs | 2 + hw/misc/idm.c | 416 +++++++++++++++++++++++++++++++++++++ include/hw/boards.h | 2 + include/hw/misc/idm.h | 119 +++++++++++ include/migration/migration.h | 2 + include/qemu/multi-socket.h | 124 +++++++++++ include/sysemu/hostmem-shared.h | 61 ++++++ migration/Makefile.objs | 2 +- migration/migration.c | 2 + migration/shared.c | 32 +++ numa.c | 17 +- qemu-options.hx | 5 +- util/qemu-config.c | 5 + 22 files changed, 1448 insertions(+), 4 deletions(-) create mode 100644 backends/hostmem-shared.c create mode 100644 backends/multi-socket.c create mode 100644 hw/misc/idm.c create mode 100644 include/hw/misc/idm.h create mode 100644 include/qemu/multi-socket.h create mode 100644 include/sysemu/hostmem-shared.h create mode 100644 migration/shared.c
Signature · 1 line
1.9.1
RFC PATCH 1/8 backend: multi-socket
From: Baptiste Reynal <[email protected]>
This patch introduces a new socket for QEMU, called multi-socket. This socket allows multiple QEMU instances to communicate by sharing messages and file descriptors.
A socket can be instantiated with the following parameters:
-object multi-socket-backend,id=<id>,path=<socket_path>,listen=<on/off>
If listen is set, the socket will act as a listener and register new clients.
--- backends/Makefile.objs | 2 + backends/multi-socket.c | 353 ++++++++++++++++++++++++++++++++++++++++++++ include/qemu/multi-socket.h | 124 ++++++++++++++++ 3 files changed, 479 insertions(+) create mode 100644 backends/multi-socket.c create mode 100644 include/qemu/multi-socket.h
backends/Makefile.objs +2 -0
@@ -9,3 +9,5 @@ common-obj-$(CONFIG_TPM) += tpm.o common-obj-y += hostmem.o hostmem-ram.o common-obj-$(CONFIG_LINUX) += hostmem-file.o + +common-obj-y += multi-socket.o
backends/multi-socket.c new file +353 -0
353 lines not on this page · raw message
include/qemu/multi-socket.h new file +124 -0
124 lines not on this page · raw message
Signature · 1 line
1.9.1
RFC PATCH 2/8 backend: shared memory backend
In reply to Christian PintoFrom: Baptiste Reynal <[email protected]>
This patch introduces a shared memory backend, allowing to share memory between a master and many slaves.
The memory is implemented using hugetlbfs, and relies on the multi-socket backend to share informations (size and offset for the slaves).
Instantiation on the master:
-object memory-backend-shared,id=<id>,size=<memory_sizeK,M,G>,
chardev=<multi-socket_id>,master=on
Instantiation on the slave:
-object memory-backend-shared,id=<id>,size=<slave_memory_sizeK,M,G>,
chardev=<multi-socket_id>,master=off
Memory size on the slave can be smaller than on master. The master will send to the slave the size and the offset of the memory segment it can allocate in the total shared memory.
--- backends/Makefile.objs | 2 +- backends/hostmem-shared.c | 194 ++++++++++++++++++++++++++++++++++++++++ include/sysemu/hostmem-shared.h | 61 +++++++++++++ 3 files changed, 256 insertions(+), 1 deletion(-) create mode 100644 backends/hostmem-shared.c create mode 100644 include/sysemu/hostmem-shared.h
backends/Makefile.objs +1 -1
@@ -8,6 +8,6 @@ baum.o-cflags := $(SDL_CFLAGS) common-obj-$(CONFIG_TPM) += tpm.o common-obj-y += hostmem.o hostmem-ram.o -common-obj-$(CONFIG_LINUX) += hostmem-file.o +common-obj-$(CONFIG_LINUX) += hostmem-file.o hostmem-shared.o common-obj-y += multi-socket.o
backends/hostmem-shared.c new file +194 -0
194 lines not on this page · raw message
include/sysemu/hostmem-shared.h new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
1.9.1
RFC PATCH 3/8 migration: add shared migration type
In reply to Christian PintoFrom: Baptiste Reynal <[email protected]>
A QEMU instance can now wait for the instantiation of the memory by the master while shared-memory backend is used.
Use:
-incoming "shared:<shared-memory_id>"
--- backends/hostmem-shared.c | 9 +++++++++ include/migration/migration.h | 2 ++ migration/Makefile.objs | 2 +- migration/migration.c | 2 ++ migration/shared.c | 32 ++++++++++++++++++++++++++++++++ 5 files changed, 46 insertions(+), 1 deletion(-) create mode 100644 migration/shared.c
backends/hostmem-shared.c +9 -0
@@ -11,6 +11,7 @@*/ #include "sysemu/hostmem-shared.h" +#include "migration/vmstate.h" static void shared_backend_init_shm (HostMemoryBackendShared *shm,int shmd,size_t size,off_t offset) {
@@ -29,6 +30,8 @@ static void shared_backend_init_shm(HostMemoryBackendShared *shm, int shmd,memory_region_add_subregion (&backend->mr,0 , &shm->shared_region); + +vmstate_register_ram_global (&shm->shared_region); }/* Callback function if a fd is received over the socket */
@@ -46,6 +49,7 @@ static void set_shared_memory(MSClient *c, const char *message, void *opaque) }shared_backend_init_shm (shm, fd, infos[0 ], infos[1 ]); +event_notifier_set (shm->levent); } static void
@@ -87,6 +91,11 @@ shared_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) set_shared_memory, shm); } } + + shm->levent =g_new (EventNotifier,1 ); +event_notifier_init (shm->levent,0 ); + + shm->event =event_notifier_get_fd (shm->levent); } static void
include/migration/migration.h +2 -0
@@ -114,6 +114,8 @@ void migrate_fd_connect(MigrationState *s);int migrate_fd_close (MigrationState *s); +void shared_start_incoming_migration (const char *name,Error **errp); +void add_migration_state_change_notifier (Notifier *notify);void remove_migration_state_change_notifier (Notifier *notify);bool migration_in_setup (MigrationState *);
migration/Makefile.objs +1 -1
@@ -7,4 +7,4 @@ common-obj-$(CONFIG_RDMA) += rdma.o common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o common-obj-y += block.o - +common-obj-y += shared.o
migration/migration.c +2 -0
@@ -249,6 +249,8 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)deferred_incoming_migration (errp); }else if (strstart (uri,"tcp:" , &p)) {tcp_start_incoming_migration (p, errp); + }else if (strstart (uri,"shared:" , &p)) { +shared_start_incoming_migration (p, errp); #ifdef CONFIG_RDMA }else if (strstart (uri,"rdma:" , &p)) {rdma_start_incoming_migration (p, errp);
migration/shared.c new file +32 -0
@@ -0,0 +1,32 @@ +#include "qemu-common.h" +#include "qemu/main-loop.h" +#include "qemu/sockets.h" +#include "migration/migration.h" +#include "monitor/monitor.h" +#include "migration/qemu-file.h" +#include "block/block.h" +#include "sysemu/hostmem-shared.h" + +static void shared_accept_incoming_migration (void *opaque) { +QEMUFile *f = opaque; +printf ("Start !\n" ); + +qemu_set_fd_handler (qemu_get_fd (f),NULL ,NULL ,NULL ); +vm_start (); +} + +void shared_start_incoming_migration (const char *id,Error **errp) +{ +HostMemoryBackendShared *shm = (HostMemoryBackendShared *) +object_resolve_path_type (id,TYPE_MEMORY_BACKEND_SHARED ,NULL ); +QEMUFile *f; + +if (shm ==NULL ) { +printf ("Error: Cannot find shared memory %s\n" , id); +exit (-1 ); + } + + f =qemu_fdopen (shm->event,"rb" ); + +qemu_set_fd_handler (shm->event, shared_accept_incoming_migration,NULL , f); +}
Signature · 1 line
1.9.1
RFC PATCH 4/8 hw/misc: IDM Device
In reply to Christian PintoThis patch introduces the Interrupt Distribution Module (IDM) device for ARM, x86 and X86_64 architectures, as the only currently supported ones.
The IDM device is used both as and Inter-processor interrupt routing device, and to trigger the boot of a slave QEMU instance.
The IDM device can be instantiated either as a sysbus or PCI device.
-device idm_ipi
or
-device idm_ipi_pci
parameters are:
master=[true/false] - configure the IDM device as master or slave num_slaves=[slaves_number] - if master is true specifies the number of slaves memdev=[memdev_id] - id of the shared memory backend socket=[socket_id] - id of the multi-client socket
--- default-configs/arm-softmmu.mak | 1 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + hw/misc/Makefile.objs | 2 + hw/misc/idm.c | 416 +++++++++++++++++++++++++++++++++++++ include/hw/misc/idm.h | 119 +++++++++++ 6 files changed, 540 insertions(+) create mode 100644 hw/misc/idm.c create mode 100644 include/hw/misc/idm.h
default-configs/arm-softmmu.mak +1 -0
@@ -109,3 +109,4 @@ CONFIG_IOH3420=y CONFIG_I82801B11=y CONFIG_ACPI=y CONFIG_SMBIOS=y +CONFIG_IDM=y
default-configs/i386-softmmu.mak +1 -0
@@ -51,3 +51,4 @@ CONFIG_XIO3130=y CONFIG_IOH3420=y CONFIG_I82801B11=y CONFIG_SMBIOS=y +CONFIG_IDM=y
default-configs/x86_64-softmmu.mak +1 -0
@@ -49,5 +49,6 @@ CONFIG_PVPANIC=y CONFIG_MEM_HOTPLUG=y CONFIG_XIO3130=y CONFIG_IOH3420=y +CONFIG_IDM=y CONFIG_I82801B11=y CONFIG_SMBIOS=y
hw/misc/Makefile.objs +2 -0
@@ -21,6 +21,8 @@ common-obj-$(CONFIG_MACIO) += macio/ obj-$(CONFIG_IVSHMEM) += ivshmem.o +obj-$(CONFIG_IDM) += idm.o + obj-$(CONFIG_REALVIEW) += arm_sysctl.o obj-$(CONFIG_NSERIES) += cbus.o obj-$(CONFIG_ECCMEMCTL) += eccmemctl.o
hw/misc/idm.c new file +416 -0
416 lines not on this page · raw message
include/hw/misc/idm.h new file +119 -0
119 lines not on this page · raw message
Signature · 1 line
1.9.1
RFC PATCH 5/8 hw/arm: sysbus-fdt
In reply to Christian PintoAdded node creation for dynamically instantiated sysbus IDM device. Support added for all ARM machines modeling dynamic sysbus devices instantiation.
--- hw/arm/sysbus-fdt.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)
hw/arm/sysbus-fdt.c +60 -0
@@ -28,6 +28,7 @@#include "sysemu/sysemu.h" #include "hw/vfio/vfio-platform.h" #include "hw/vfio/vfio-calxeda-xgmac.h" +#include "hw/misc/idm.h" #include "hw/arm/fdt.h" /*
@@ -123,9 +124,68 @@ fail_reg:return ret; } +/** +* add_idm_fdt_node +* +* Generates a node with following properties: +* compatible string, regs, interrupts +*/ +static int add_idm_fdt_node (SysBusDevice *sbdev,void *opaque) +{ +PlatformBusFDTData *data = opaque; +PlatformBusDevice *pbus = data->pbus; +const char idm_compat[] ="idm-ipi" ; +void *fdt = data->fdt; +const char *parent_node = data->pbus_node_name; +char *nodename; +int ret =-1 ; +uint32_t *irq_attr, *reg_attr; +uint64_t mmio_base, irq_number; + + mmio_base =platform_bus_get_mmio_addr (pbus, sbdev,0 ); + nodename =g_strdup_printf ("%s/%s@%" PRIx64, parent_node, +TYPE_IDM , mmio_base); +qemu_fdt_add_subnode (fdt, nodename); + +qemu_fdt_setprop (fdt, nodename,"compatible" , idm_compat, +sizeof (idm_compat)); + +/** +* There is only one MMIO region defined for the IDM device +*/ + reg_attr =g_new (uint32_t,2 ); + reg_attr[0 ] =cpu_to_be32 (mmio_base); + reg_attr[1 ] =cpu_to_be32 (IDM_SIZE ); + ret =qemu_fdt_setprop (fdt, nodename,"reg" , reg_attr, +2 *sizeof (uint32_t )); +if (ret) { +error_report ("could not set reg property of node %s" , nodename); + goto fail; + } + + irq_attr =g_new (uint32_t,3 ); + irq_number =platform_bus_get_irqn (pbus, sbdev ,0 ) + + data->irq_start; + irq_attr[0 ] =cpu_to_be32 (GIC_FDT_IRQ_TYPE_SPI ); + irq_attr[1 ] =cpu_to_be32 (irq_number); + irq_attr[2 ] =cpu_to_be32 (GIC_FDT_IRQ_FLAGS_EDGE_LO_HI ); + ret =qemu_fdt_setprop (fdt, nodename,"interrupts" , + irq_attr,3 *sizeof (uint32_t )); +if (ret) { +error_report ("could not set interrupts property of node %s" , + nodename); + } +g_free (irq_attr); +fail: +g_free (reg_attr); +g_free (nodename); +return ret; +} +/* list of supported dynamic sysbus devices */ static const NodeCreationPair add_fdt_node_functions[] = { {TYPE_VFIO_CALXEDA_XGMAC , add_calxeda_midway_xgmac_fdt_node}, + {TYPE_IDM , add_idm_fdt_node}, {"" ,NULL },/* last element */ };
Signature · 1 line
1.9.1
RFC PATCH 6/8 qemu: slave machine flag
In reply to Christian PintoThis patch adds a new machine flag, to configure qemu as a slave instance.
Usage
-machine -slave=[on|off] (default=off)
--- hw/core/machine.c | 27 +++++++++++++++++++++++++++ include/hw/boards.h | 2 ++ qemu-options.hx | 5 ++++- util/qemu-config.c | 5 +++++ 4 files changed, 38 insertions(+), 1 deletion(-)
hw/core/machine.c +27 -0
@@ -283,6 +283,20 @@ static bool machine_get_suppress_vmdesc(Object *obj, Error **errp)return ms->suppress_vmdesc; } +static bool machine_get_slave (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->slave; +} + +static void machine_set_slave (Object *obj,bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->slave = value; +} +static int error_on_sysbus_device (SysBusDevice *sbdev,void *opaque) { error_report("Option '-device %s' cannot be handled by this machine",
@@ -335,6 +349,7 @@ static void machine_initfn(Object *obj) ms->kvm_shadow_mem =-1 ; ms->dump_guest_core = true; ms->mem_merge = true; + ms->slave = false;object_property_add_str (obj,"accel" , machine_get_accel, machine_set_accel,NULL );
@@ -437,6 +452,13 @@ static void machine_initfn(Object *obj)object_property_set_description (obj,"suppress-vmdesc" ,"Set on to disable self-describing migration" ,NULL ); +object_property_add_bool (obj,"slave" , + machine_get_slave, + machine_set_slave, +NULL ); +object_property_set_description (obj,"slave" , +"Enables a slave (remote) machine instance" , +NULL );/* Register notifier when init is done for sysbus sanity checks */ ms->sysbus_notifier.notify = machine_init_notify;
@@ -497,6 +519,11 @@ bool machine_mem_merge(MachineState *machine)return machine->mem_merge; } +bool machine_slave (MachineState *machine) +{ +return machine->slave; +} +static const TypeInfo machine_info = { .name =TYPE_MACHINE , .parent =TYPE_OBJECT ,
include/hw/boards.h +2 -0
@@ -40,6 +40,7 @@ int machine_kvm_shadow_mem(MachineState *machine);int machine_phandle_start (MachineState *machine);bool machine_dump_guest_core (MachineState *machine);bool machine_mem_merge (MachineState *machine); +bool machine_slave (MachineState *machine);/** * MachineClass:
@@ -120,6 +121,7 @@ struct MachineState {char *firmware;bool iommu;bool suppress_vmdesc; +bool slave;ram_addr_t ram_size;ram_addr_t maxram_size;
qemu-options.hx +4 -1
@@ -41,7 +41,8 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n" " aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n" " dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n" - " suppress-vmdesc=on|off disables self-describing migration (default=off)\n", + " suppress-vmdesc=on|off disables self-describing migration (default=off)\n" + " slave=on|off enables a slave (remote) machine instance (default=off)", QEMU_ARCH_ALL) STEXI @item -machine [type=]@var{name}[,prop=@var{value}[,...]]
@@ -80,6 +81,8 @@ execution of AES cryptographic functions. The default is on. Enables or disables DEA key wrapping support on s390-ccw hosts. This feature controls whether DEA wrapping keys will be created to allow execution of DEA cryptographic functions. The default is on. +@item slave=on|off +Enables a slave (remote) machine instance @end table ETEXI
util/qemu-config.c +5 -0
@@ -219,6 +219,11 @@ static QemuOptsList machine_opts = { .name = "suppress-vmdesc", .type = QEMU_OPT_BOOL, .help = "Set on to disable self-describing migration", + },{ + .name = "slave", + .type = QEMU_OPT_BOOL, + .help = "Indicates the machine is a slave instance " + "(e.g. remoteproc)", }, {/* End of list */ } }
Signature · 1 line
1.9.1
RFC PATCH 7/8 hw/arm: boot
In reply to Christian PintoModify the boot process of an ARM machine in order to check whether it is a slave, by checking the slave machine flag.
When the slave flag is on, no kernel, dtb or initrd are loaded into memory. The boot address of each core is set to the start address of the RAM, that depends on the machine model executed.
--- hw/arm/boot.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
hw/arm/boot.c +13 -0
@@ -590,6 +590,19 @@ static void arm_load_kernel_notify(Notifier *notifier, void *data)/* Load the kernel. */ if (!info->kernel_filename || info->firmware_loaded) { +if (!info->kernel_filename &&machine_slave (current_machine)) { +/* If a machine is booted as a slave instance there is no need to +* provide the DTB blob or kernel image, that will instead +* be copied into memory later by a master instance. +* The boot address is set to be at the beginning of the RAM. +*/ + info->entry = info->loader_start; +CPU_FOREACH (cs) { +ARM_CPU (cs)->env.boot_info = info; + } +return ; + } +if (have_dtb (info)) {/* If we have a device tree blob, but no kernel to supply it to (or * the kernel is supposed to be loaded by the bootloader), copy the
Signature · 1 line
1.9.1
RFC PATCH 8/8 qemu: numa
In reply to Christian PintoThis patch modifies the behavior of memory_region_allocate_system_memory, when the new shared memory backend is used from a slave qemu instance.
In such case there is not yet a valid pointer for the memory region pointed by the backend (it will be innitilized later when received from the master.) and vmstate_register would fail.
The patch skips the call to vmstate_register in case of slave shared memory backend, that will be performed later after the actual memory pointer is available.
--- numa.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
numa.c +16 -1
@@ -33,6 +33,7 @@#include "qapi/dealloc-visitor.h" #include "hw/boards.h" #include "sysemu/hostmem.h" +#include "sysemu/hostmem-shared.h" #include "qmp-commands.h" #include "hw/mem/pc-dimm.h" #include "qemu/option.h"
@@ -442,6 +443,7 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner, {uint64_t addr =0 ;int i; +bool vmstate_register = true;if (nb_numa_nodes ==0 || !have_memdevs) {allocate_system_memory_nonnuma (mr, owner, name, ram_size);
@@ -453,9 +455,18 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner,Error *local_err =NULL ;uint64_t size = numa_info[i].node_mem;HostMemoryBackend *backend = numa_info[i].node_memdev; +if (!backend) {continue ; } + +if (IS_MEMORY_BACKEND_SHARED (backend)) { +HostMemoryBackendShared *shm =MEMORY_BACKEND_SHARED (backend); +if (!shm->master) { + vmstate_register = false; + } + } +MemoryRegion *seg =host_memory_backend_get_memory (backend, &local_err);if (local_err) {error_report_err (local_err);
@@ -471,7 +482,11 @@ void memory_region_allocate_system_memory(MemoryRegion *mr, Object *owner, }memory_region_add_subregion (mr, addr, seg); -vmstate_register_ram_global (seg); + +if (vmstate_register) { +vmstate_register_ram_global (seg); + } + addr += size; } }
Signature · 1 line
1.9.1
Quoted 6 lines · Christian Pinto
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".
Quoted 6 lines
To recap the goal of such RFC:
The idea is to enhance the current architecture of QEMU to enable the modeling of a state of-the-art SoC with an AMP processing style, where different processing units share the same system memory and communicate through shared memory and inter-processor interrupts.
This might have a lot in common with a similar inter-qemu communication solution effort at Xilinx. Edgar talks about it at KVM forum:
https://www.youtube.com/watch?v=L5zG5Aukfek
Around 18:30 mark. I think it might be lower level that your proposal, remote-port is designed to export the raw hardware interfaces (busses and pins) between QEMU and some other system, another QEMU being the common use cases.
Quoted 3 lines
An example is a multi-core ARM CPU
working alongside with two Cortex-M micro controllers.
Marcin is doing something with A9+M3. It sounds like he already has a lot working (latest emails were on some finer points). What is the board/SoC in question here (if you are able to share)?
Quoted 4 lines
From the user point of view there is usually an operating system booting on the Master processor (e.g. Linux) at platform startup, while the other processors are used to offload the Master one from some computation or to deal with real-time interfaces.
I feel like this is architecting hardware based on common software use cases, rather than directly modelling the SoC in question. Can we model the hardware (e.g. the devices that are used for rpmesg and IPIs etc.) as regular devices, as it is in-SoC? That means AMP is just another guest?
Quoted 10 lines
It is the Master OS that triggers the boot of the
Slave processors, and provides them also the binary code to execute (e.g. RTOS, binary firmware) by placing it into a pre-defined memory area that is accessible to the Slaves. Usually the memory for the Slaves is carved out from the Master OS during boot. Once a Slave is booted the two processors can communicate through queues in shared memory and inter-processor interrupts (IPIs). In Linux, it is the remoteproc/rpmsg framework that enables the control (boot/shutdown) of Slave processors, and also to establish a communication channel based on virtio queues.
Quoted 2 lines
Currently, QEMU is not able to model such an architecture mainly because only a single processor can be emulated at one time,
SMP does work already. MTTCG will remove the one-run-at-a-time limitation. Multi-arch will allow you to mix multiple CPU architectures (e.g. PPC + ARM in same QEMU). But multiple heterogeneous ARMs should already just work, and there is already an in-tree precedent with the xlnx-zynqmp SoC. That SoC has 4xA53 and 2xR5 (all ARM).
Multiple system address spaces and CPUs have different views of the address space is another common snag on this effort, and is discussed on a recent thread between myself and Marcin.
Quoted 3 lines
and the OS binary image needs
to be placed in memory at model startup.
I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
Can this effort be a bootloader overhaul? Two things:
1: The bootloader needs to repeatable
2: The bootloaders need to be targetable (to certain CPUs or clusters)
Quoted 6 lines
This patch series adds a set of modules and introduces minimal changes to the current QEMU code-base to implement what described above, with master and slave implemented as two different instances of QEMU. The aim of this work is to enable application and runtime programmers to test their AMP applications, or their new inter-SoC communtication protocol.
Quoted 12 lines
The main changes are depicted in the following diagram and involve:
- A new multi-client socket implementation that allows multiple instances of QEMU to attach to the same socket, with only one acting as a master.
- A new memory backend, the shared memory backend, based on
the file memory backend. Such new backend enables, on the master side, to allocate the whole memory as shareable (e.g./dev/shm, or hugetlbfs). On the slave side it enables the startup of QEMU without any main memory allocated. The the slave goes in a waiting state, the same used in the case of an incoming migration, and a callback is registered on a multi-client socket shared with the master.
The waiting state ends when the master sends to the slave the file descriptor and offset to mmap and use as memory.
This is useful in it's own right and came up in the Xilinx implementation.
Quoted 5 lines
- A new inter-processor interrupt hardware distribution module, that is used also to trigger the boot of slave processors. Such module uses a pair of eventfd for each master-slave couple to trigger interrupts between the instances. No slave-to-slave interrupts are envisioned by the current implementation.
Wouldn't that just be a software interrupt in the local QEMU instance?
Quoted 5 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to exchancge the eventd file descriptors. The IDM device can be instantiated either as a PCI or sysbus device.
So if everything is is one QEMU, IPIs can be implemented with just a regular interrupt controller (which has a software set).
Quoted 27 lines
Memory
(e.g. hugetlbfs)+------------------+ +--------------+ +------------------+ | | | | | | | QEMU MASTER | | Master | | QEMU SLAVE | | | | Memory | | | | +------+ +------+-+ | | +-+------+ +------+ | | | | |SHMEM | | | |SHMEM | | | | | | VCPU | |Backend +-----> | +----->Backend | | VCPU | | | | | | | | | | +---> | | | | | +--^---+ +------+-+ | | | | +-+------+ +--^---+ | | | | | | | | | | | | +--+ | | | | | | +---+ | | | IRQ | | +----------+ | | | | IRQ | | | | | | | | | | | | | | | +----+----+ | | | Slave <------+ | | +----+---+ | +--+ IDM +-----+ | | Memory | | | +---+ IDM +-----+ +-^----^--+ | | | | | +-^---^--+ | | | +----------+ | | | | | | +--------------+ | | | | | | | | | +--------------------------------------+-----------+ | | UNIX Domain Socket(send mem fd + offset, trigger boot) | | | +-----------------------------------------------------------+
Quoted 2 lines
eventfd
So the slave can only see a subset of the masters memory? Is the masters memory just the full system memory and the master is doing IOMMU setup for the slave pre-boot? Or is it a hard feature of the physical SoC?
Quoted 12 lines
The whole code can be checked out from:
https://git.virtualopensystems.com/dev/qemu-het.git
branch:
qemu-het-rfc-v1Patches apply to the current QEMU master branch
=========
Demo
=========
Quoted 9 lines
This patch series comes in the form of a demo to better understand how the changes introduced can be exploited.
At the current status the demo can be executed using an ARM target for both master and slave.The demo shows how a master QEMU instance carves out the memory for a slave, copies inside linux kernel image and device tree blob and finally triggers the boot.
These processes must have underlying hardware implementation, is the master using a system controller to implement the slave boot? (setting reset and entry points via registers?). How hard are they to model as regular devs?
Quoted 81 lines
How to reproduce the demo:
In order to reproduce the demo a couple more extra elements need to be downloaded and compiled.
Binary loader
Loads the slave firmware (kernel) binary into memory and triggers the boot
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
load-bin-boot
To compile: just type "make"Slave kernel
Compile a linux kernel image (zImage) for the virt machine model.IDM test kernel module
Needed to trigger the boot of a slave
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
IDM-kernel-module
To compile: KDIR=kernel_path ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- makeSlave DTB
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
slave-dtbCopy binary loader, IDM kernel module, zImage and dtb inside the disk image or ramdisk of the master instance.
Run the demo:
run the master instance
./arm-softmmu/qemu-system-arm \ -kernel zImage \ -M virt -cpu cortex-a15 \ -drive if=none,file=disk.img,cache=writeback,id=foo1 \ -device virtio-blk-device,drive=foo1 \ -object multi-socket-backend,id=foo,listen,path=ms_socket \ -object memory-backend-shared,id=mem,size=1G,mem-path=/mnt/hugetlbfs,chardev=foo,master=on,prealloc=on \ -device idm_ipi,master=true,memdev=mem,socket=foo \ -numa node,memdev=mem -m 1G \ -append "root=/dev/vda rw console=ttyAMA0 mem=512M memmap=512M$0x60000000" \ -nographicrun the slave instance
./arm-softmmu/qemu-system-arm\ -M virt -cpu cortex-a15 -machine slave=on \ -drive if=none,file=disk.img,cache=writeback,id=foo1 \ -device virtio-blk-device,drive=foo1 \ -object multi-socket-backend,id=foo,path=ms_socket \ -object memory-backend-shared,id=mem,size=512M,mem-path=/mnt/hugetlbfs,chardev=foo,master=off \ -device idm_ipi,master=false,memdev=mem,socket=foo \ -incoming "shared:mem" -numa node,memdev=mem -m 512M \ -nographicFor simplicity, use a disk image for the slave instead of a ramdisk.
As visible from the kernel boot arguments, the master is booted with mem=512 so that one half of the whole memory allocated is not used by the master and reserved for the slave. Such memory starts for the virt platform from address
0x60000000.Once the master is booted the image of the kernel and DTB can be copied in the memory carved out for the slave.
In the maser console
probe the IDM kernel module:
$ insmod idm_test_mod.korun the application that copies the binaries into memory and triggers the boot:
$ ./load_bin_app 1 ./zImage ./slave.dtb
Quoted 7 lines
On the slave console the linux kernel boot should be visible.
The present demo is intended only as a demonstration to see the patch-set at work. In the near future, boot triggering, memory carveout and binary copy might be implemented in a remoteproc driver coupled with a RPMSG driver for communication between master and slave instance.
So are these drivers the same ones as run on the real hardware? is there value in the fact that the real IPI mechanisms are replaced with virt ones?
Regards,
Peter
Quoted 50 lines
This work has been sponsored by Huawei Technologies Duesseldorf GmbH.
Baptiste Reynal (3):
backend: multi-socket
backend: shared memory backend
migration: add shared migration typeChristian Pinto (5):
hw/misc: IDM Device
hw/arm: sysbus-fdt
qemu: slave machine flag
hw/arm: boot
qemu: numabackends/Makefile.objs | 4 +- backends/hostmem-shared.c | 203 ++++++++++++++++++ backends/multi-socket.c | 353 +++++++++++++++++++++++++++++++ default-configs/arm-softmmu.mak | 1 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + hw/arm/boot.c | 13 ++ hw/arm/sysbus-fdt.c | 60 ++++++ hw/core/machine.c | 27 +++ hw/misc/Makefile.objs | 2 + hw/misc/idm.c | 416 +++++++++++++++++++++++++++++++++++++ include/hw/boards.h | 2 + include/hw/misc/idm.h | 119 +++++++++++ include/migration/migration.h | 2 + include/qemu/multi-socket.h | 124 +++++++++++ include/sysemu/hostmem-shared.h | 61 ++++++ migration/Makefile.objs | 2 +- migration/migration.c | 2 + migration/shared.c | 32 +++ numa.c | 17 +- qemu-options.hx | 5 +- util/qemu-config.c | 5 + 22 files changed, 1448 insertions(+), 4 deletions(-) create mode 100644 backends/hostmem-shared.c create mode 100644 backends/multi-socket.c create mode 100644 hw/misc/idm.c create mode 100644 include/hw/misc/idm.h create mode 100644 include/qemu/multi-socket.h create mode 100644 include/sysemu/hostmem-shared.h create mode 100644 migration/shared.c--
1.9.1
Hello Peter,
thanks for your comments
Quoted 18 lines · Peter Crosthwaite
Quoted 6 lines · Christian Pinto
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".Quoted 6 lines
To recap the goal of such RFC:
The idea is to enhance the current architecture of QEMU to enable the modeling of a state of-the-art SoC with an AMP processing style, where different processing units share the same system memory and communicate through shared memory and inter-processor interrupts.
This might have a lot in common with a similar inter-qemu communication solution effort at Xilinx. Edgar talks about it at KVM forum:
Quoted 6 lines
https://www.youtube.com/watch?v=L5zG5Aukfek
Around 18:30 mark. I think it might be lower level that your proposal, remote-port is designed to export the raw hardware interfaces (busses and pins) between QEMU and some other system, another QEMU being the common use cases.
Thanks for pointing this out. Indeed what presented by Edgar has a lot
of similarities with our proposal, but is targeting a different scenario
where low-level modeling of the various hardware components is taken
into account.
The goal of my proposal is on the other hand to enable a set of tools
for high level early prototyping of systems with a heterogeneous
set of cores, so to model a platform that does not exist in reality but that
the user wants to experiment with.
As an example I can envision a programming model researcher willing to explore an heterogeneous system based on an X86 and a multi-core ARM accelerator sharing memory, to build a new programming paradigm on top of it. Such user would not need the specific details of the hardware nor all the various devices available in a real SoC, but only an abstract model encapsulating the main features needed for his research.
So to link also to your next comment there is no actual SoC/hardware targeted by this work.
Quoted 11 lines
Quoted 3 lines
An example is a multi-core ARM CPU
working alongside with two Cortex-M micro controllers.Marcin is doing something with A9+M3. It sounds like he already has a lot working (latest emails were on some finer points). What is the board/SoC in question here (if you are able to share)?
Quoted 4 lines
From the user point of view there is usually an operating system booting on
the Master processor (e.g. Linux) at platform startup, while the other processors are used to offload the Master one from some computation or to deal with real-time interfaces.
Quoted 5 lines
I feel like this is architecting hardware based on common software use cases, rather than directly modelling the SoC in question. Can we model the hardware (e.g. the devices that are used for rpmesg and IPIs etc.) as regular devices, as it is in-SoC? That means AMP is just another guest?
This is a set of extensions focusing more on the communication channel between the processors rather than a full SoC model. With this patch series each of the AMP processors is a different "guest".
Quoted 10 lines
Quoted 10 lines
It is the Master OS that triggers the boot of the
Slave processors, and provides them also the binary code to execute (e.g. RTOS, binary firmware) by placing it into a pre-defined memory area that is accessible to the Slaves. Usually the memory for the Slaves is carved out from the Master OS during boot. Once a Slave is booted the two processors can communicate through queues in shared memory and inter-processor interrupts (IPIs). In Linux, it is the remoteproc/rpmsg framework that enables the control (boot/shutdown) of Slave processors, and also to establish a communication channel based on virtio queues.
Quoted 8 lines
Quoted 2 lines
Currently, QEMU is not able to model such an architecture mainly because only a single processor can be emulated at one time,
SMP does work already. MTTCG will remove the one-run-at-a-time limitation. Multi-arch will allow you to mix multiple CPU architectures (e.g. PPC + ARM in same QEMU). But multiple heterogeneous ARMs should already just work, and there is already an in-tree precedent with the xlnx-zynqmp SoC. That SoC has 4xA53 and 2xR5 (all ARM).
Since Multi-arch is not yet available, with this proposal it is possible to experiment with heterogeneous processors at high level of abstraction, even beyond the ARM + ARM (e.g. X86 + ARM), exploiting the off-the-shelf QEMU.
One thing I want to add is that all the solutions mentioned in this discussion Multi-arch, Xilinx's patches, and our proposal could coexist from the code point of view, and none would prevent the others from being used.
Quoted 3 lines
Multiple system address spaces and CPUs have different views of the address space is another common snag on this effort, and is discussed on a recent thread between myself and Marcin.
Yes I have seen the discussion, but it was mostly dealing with one single QEMU instance modeling all the cores. Here the different address spaces are enforced by multiple QEMU instances.
Quoted 6 lines
Quoted 3 lines
and the OS binary image needs
to be placed in memory at model startup.I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
the problem here was to me that when we launch QEMU a binary needs to be provided and put in memory in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched. The information about memory (fd + offset for mmap) is sent only later when the boot is triggered. This is also safe since the slave will be waiting in the incoming state, and thus no corruption or errors can happen before the boot is triggered.
Quoted 4 lines
Can this effort be a bootloader overhaul? Two things:
1: The bootloader needs to repeatable
2: The bootloaders need to be targetable (to certain CPUs or clusters)
Well in this series the bootloader for the master is different from the one for the slave. In my idea the master, besides the firmware/kernel image, will copy also a bootloader for the slave.
Quoted 18 lines
Quoted 6 lines
This patch series adds a set of modules and introduces minimal changes to the current QEMU code-base to implement what described above, with master and slave implemented as two different instances of QEMU. The aim of this work is to enable application and runtime programmers to test their AMP applications, or their new inter-SoC communtication protocol.
Quoted 12 lines
The main changes are depicted in the following diagram and involve:
- A new multi-client socket implementation that allows multiple instances of QEMU to attach to the same socket, with only one acting as a master.
- A new memory backend, the shared memory backend, based on
the file memory backend. Such new backend enables, on the master side, to allocate the whole memory as shareable (e.g./dev/shm, or hugetlbfs). On the slave side it enables the startup of QEMU without any main memory allocated. The the slave goes in a waiting state, the same used in the case of an incoming migration, and a callback is registered on a multi-client socket shared with the master.
The waiting state ends when the master sends to the slave the file descriptor and offset to mmap and use as memory.
Quoted 1 line
This is useful in it's own right and came up in the Xilinx implementation.
It is also mentioned in the video you are pointing, where the Microblaze cores are instantiated as foreign QEMU instances. Is the code publicly available? There was a question about that in the video but I couldn't catch the answer.
Quoted 6 lines
Quoted 5 lines
- A new inter-processor interrupt hardware distribution module, that is used also to trigger the boot of slave processors. Such module uses a pair of eventfd for each master-slave couple to trigger interrupts between the instances. No slave-to-slave interrupts are envisioned by the current implementation.
Wouldn't that just be a software interrupt in the local QEMU instance?
Since in this proposal there will be multiple instances of QEMU running at the same time, eventfd are used to signal the event (interrupt) among the different processes. So writing to a register of the IDM will raise an interrupt to a remote QEMU instance using eventfd. Did this answer your question?
Quoted 7 lines
Quoted 5 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to exchancge the eventd file descriptors. The IDM device can be instantiated either as a PCI or sysbus device.So if everything is is one QEMU, IPIs can be implemented with just a regular interrupt controller (which has a software set).
As said there are multiple instances of QEMU running at the same time, and each of them will see the IDM in their memory map.
Even if the IDM instances will be physically different, because of the multiple processes, all together will act as a single block (e.g., a light version of a mailbox).
Quoted 28 lines
Quoted 26 lines
Memory
(e.g. hugetlbfs)+------------------+ +--------------+ +------------------+ | | | | | | | QEMU MASTER | | Master | | QEMU SLAVE | | | | Memory | | | | +------+ +------+-+ | | +-+------+ +------+ | | | | |SHMEM | | | |SHMEM | | | | | | VCPU | |Backend +-----> | +----->Backend | | VCPU | | | | | | | | | | +---> | | | | | +--^---+ +------+-+ | | | | +-+------+ +--^---+ | | | | | | | | | | | | +--+ | | | | | | +---+ | | | IRQ | | +----------+ | | | | IRQ | | | | | | | | | | | | | | | +----+----+ | | | Slave <------+ | | +----+---+ | +--+ IDM +-----+ | | Memory | | | +---+ IDM +-----+ +-^----^--+ | | | | | +-^---^--+ | | | +----------+ | | | | | | +--------------+ | | | | | | | | | +--------------------------------------+-----------+ | | UNIX Domain Socket(send mem fd + offset, trigger boot) | | | +-----------------------------------------------------------+Quoted 2 lines
eventfd
Quoted 4 lines
So the slave can only see a subset of the masters memory? Is the masters memory just the full system memory and the master is doing IOMMU setup for the slave pre-boot? Or is it a hard feature of the physical SoC?
Yes slaves can only see the memory that has been reserved for them. This is ensured by carving out the memory from the master kernel and providing the offset to such memory to the slave. Each slave will have its own memory map, and see the memory at the address defined in the machine model. There is no IOMMU modeled, but it is neither a hard feature since decided at run-time.
Quoted 16 lines
Quoted 16 lines
The whole code can be checked out from:
https://git.virtualopensystems.com/dev/qemu-het.git
branch:
qemu-het-rfc-v1Patches apply to the current QEMU master branch
=========
Demo
=========This patch series comes in the form of a demo to better understand how the changes introduced can be exploited.
At the current status the demo can be executed using an ARM target for both master and slave.
Quoted 9 lines
Quoted 4 lines
The demo shows how a master QEMU instance carves out the memory for a slave, copies inside linux kernel image and device tree blob and finally triggers the boot.
These processes must have underlying hardware implementation, is the master using a system controller to implement the slave boot? (setting reset and entry points via registers?). How hard are they to model as regular devs?
In this series the system controller is the IDM device, that through a set of registers makes the master in "control" each of the slaves. The IDM device is already seen as a regular device by each of the QEMU instances involved.
Quoted 82 lines
Quoted 82 lines
How to reproduce the demo:
In order to reproduce the demo a couple more extra elements need to be downloaded and compiled.
Binary loader
Loads the slave firmware (kernel) binary into memory and triggers the boot
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
load-bin-boot
To compile: just type "make"Slave kernel
Compile a linux kernel image (zImage) for the virt machine model.IDM test kernel module
Needed to trigger the boot of a slave
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
IDM-kernel-module
To compile: KDIR=kernel_path ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- makeSlave DTB
https://git.virtualopensystems.com/dev/qemu-het-tools.git
branch:
slave-dtbCopy binary loader, IDM kernel module, zImage and dtb inside the disk image or ramdisk of the master instance.
Run the demo:
run the master instance
./arm-softmmu/qemu-system-arm \ -kernel zImage \ -M virt -cpu cortex-a15 \ -drive if=none,file=disk.img,cache=writeback,id=foo1 \ -device virtio-blk-device,drive=foo1 \ -object multi-socket-backend,id=foo,listen,path=ms_socket \ -object memory-backend-shared,id=mem,size=1G,mem-path=/mnt/hugetlbfs,chardev=foo,master=on,prealloc=on \ -device idm_ipi,master=true,memdev=mem,socket=foo \ -numa node,memdev=mem -m 1G \ -append "root=/dev/vda rw console=ttyAMA0 mem=512M memmap=512M$0x60000000" \ -nographicrun the slave instance
./arm-softmmu/qemu-system-arm\ -M virt -cpu cortex-a15 -machine slave=on \ -drive if=none,file=disk.img,cache=writeback,id=foo1 \ -device virtio-blk-device,drive=foo1 \ -object multi-socket-backend,id=foo,path=ms_socket \ -object memory-backend-shared,id=mem,size=512M,mem-path=/mnt/hugetlbfs,chardev=foo,master=off \ -device idm_ipi,master=false,memdev=mem,socket=foo \ -incoming "shared:mem" -numa node,memdev=mem -m 512M \ -nographicFor simplicity, use a disk image for the slave instead of a ramdisk.
As visible from the kernel boot arguments, the master is booted with mem=512 so that one half of the whole memory allocated is not used by the master and reserved for the slave. Such memory starts for the virt platform from address
0x60000000.Once the master is booted the image of the kernel and DTB can be copied in the memory carved out for the slave.
In the maser console
probe the IDM kernel module:
$ insmod idm_test_mod.korun the application that copies the binaries into memory and triggers the boot:
$ ./load_bin_app 1 ./zImage ./slave.dtbOn the slave console the linux kernel boot should be visible.
Quoted 8 lines
Quoted 5 lines
The present demo is intended only as a demonstration to see the patch-set at work. In the near future, boot triggering, memory carveout and binary copy might be implemented in a remoteproc driver coupled with a RPMSG driver for communication between master and slave instance.
So are these drivers the same ones as run on the real hardware? is there value in the fact that the real IPI mechanisms are replaced with virt ones?
As for the first question, since there is no specific target hardware even the drivers are generic and thus not meant to run on a real SoC. The drivers shown for this demo are an example of how the patch series could be used, and do not prevent the user to implement its own drivers based on its own communication protocol.
Thanks,
Christian
Quoted 52 lines
Regards,
PeterQuoted 49 lines
This work has been sponsored by Huawei Technologies Duesseldorf GmbH.
Baptiste Reynal (3):
backend: multi-socket
backend: shared memory backend
migration: add shared migration typeChristian Pinto (5):
hw/misc: IDM Device
hw/arm: sysbus-fdt
qemu: slave machine flag
hw/arm: boot
qemu: numabackends/Makefile.objs | 4 +- backends/hostmem-shared.c | 203 ++++++++++++++++++ backends/multi-socket.c | 353 +++++++++++++++++++++++++++++++ default-configs/arm-softmmu.mak | 1 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + hw/arm/boot.c | 13 ++ hw/arm/sysbus-fdt.c | 60 ++++++ hw/core/machine.c | 27 +++ hw/misc/Makefile.objs | 2 + hw/misc/idm.c | 416 +++++++++++++++++++++++++++++++++++++ include/hw/boards.h | 2 + include/hw/misc/idm.h | 119 +++++++++++ include/migration/migration.h | 2 + include/qemu/multi-socket.h | 124 +++++++++++ include/sysemu/hostmem-shared.h | 61 ++++++ migration/Makefile.objs | 2 +- migration/migration.c | 2 + migration/shared.c | 32 +++ numa.c | 17 +- qemu-options.hx | 5 +- util/qemu-config.c | 5 + 22 files changed, 1448 insertions(+), 4 deletions(-)create mode 100644
backends/hostmem-shared.ccreate mode 100644backends/multi-socket.ccreate mode 100644hw/misc/idm.c
create mode 100644include/hw/misc/idm.hcreate mode 100644include/qemu/multi-socket.hcreate mode 100644include/sysemu/hostmem-shared.hcreate mode 100644migration/shared.c--
1.9.1
Quoted 78 lines · Christian Pinto
Hello Peter,
thanks for your comments
Quoted 24 lines · Peter Crosthwaite
Quoted 8 lines · Christian Pinto
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".Quoted 8 lines
To recap the goal of such RFC:
The idea is to enhance the current architecture of QEMU to enable the modeling
of a state of-the-art SoC with an AMP processing style, where different processing units share the same system memory and communicate through shared
memory and inter-processor interrupts.This might have a lot in common with a similar inter-qemu communication solution effort at Xilinx. Edgar talks about it at KVM forum:
Quoted 6 lines
https://www.youtube.com/watch?v=L5zG5Aukfek
Around 18:30 mark. I think it might be lower level that your proposal, remote-port is designed to export the raw hardware interfaces (busses and pins) between QEMU and some other system, another QEMU being the common use cases.
Thanks for pointing this out. Indeed what presented by Edgar has a lot of similarities with our proposal, but is targeting a different scenario where low-level modeling of the various hardware components is taken into account.
The goal of my proposal is on the other hand to enable a set of tools for high level early prototyping of systems with a heterogeneous set of cores, so to model a platform that does not exist in reality but that the user wants to experiment with.
As an example I can envision a programming model researcher willing to explore an
heterogeneous system based on an X86 and a multi-core ARM accelerator sharing memory,
to build a new programming paradigm on top of it. Such user would not need the specific details
of the hardware nor all the various devices available in a real SoC, but only an abstract model
encapsulating the main features needed for his research.So to link also to your next comment there is no actual SoC/hardware targeted by this
work.Quoted 15 lines
Quoted 4 lines
An example is a multi-core ARM CPU
working alongside with two Cortex-M micro controllers.Marcin is doing something with A9+M3. It sounds like he already has a lot working (latest emails were on some finer points). What is the board/SoC in question here (if you are able to share)?
Quoted 6 lines
From the user point of view there is usually an operating system booting
on
the Master processor (e.g. Linux) at platform startup, while the other processors are used to offload the Master one from some computation or to deal
with real-time interfaces.Quoted 5 lines
I feel like this is architecting hardware based on common software use cases, rather than directly modelling the SoC in question. Can we model the hardware (e.g. the devices that are used for rpmesg and IPIs etc.) as regular devices, as it is in-SoC? That means AMP is just another guest?
Quoted 4 lines
This is a set of extensions focusing more on the communication channel between the processors
rather than a full SoC model. With this patch series each of the AMP processors is a different "guest".
Ok. My issue here is that establishing a 1:1 relationship between QEMU guests and AMP peers is building use-case policy into the mechanism. I am seeing the use-case where there is just one guest that sets up AMP without any QEMU awareness of the AMPness.
Quoted 30 lines
Quoted 18 lines
Quoted 14 lines
It is the Master OS that triggers the boot of the
Slave processors, and provides them also the binary code to execute (e.g. RTOS, binary firmware) by placing it into a pre-defined memory area that is
accessible to the Slaves. Usually the memory for the Slaves is carved out from
the Master OS during boot. Once a Slave is booted the two processors can communicate through queues in shared memory and inter-processor interrupts
(IPIs). In Linux, it is the remoteproc/rpmsg framework that enables the control (boot/shutdown) of Slave processors, and also to establish a communication channel based on virtio queues.Quoted 3 lines
Currently, QEMU is not able to model such an architecture mainly because only
a single processor can be emulated at one time,Quoted 6 lines
SMP does work already. MTTCG will remove the one-run-at-a-time limitation. Multi-arch will allow you to mix multiple CPU architectures (e.g. PPC + ARM in same QEMU). But multiple heterogeneous ARMs should already just work, and there is already an in-tree precedent with the xlnx-zynqmp SoC. That SoC has 4xA53 and 2xR5 (all ARM).
Since Multi-arch is not yet available, with this proposal it is possible to experiment with heterogeneous processors at high level of abstraction, even beyond the ARM + ARM (e.g. X86 + ARM), exploiting the off-the-shelf QEMU.
Quoted 5 lines
One thing I want to add is that all the solutions mentioned in this discussion Multi-arch,
Xilinx's patches, and our proposal could coexist from the code point of view, and none
would prevent the others from being used.
We should consolidate where possible though.
Quoted 12 lines
Quoted 4 lines
Multiple system address spaces and CPUs have different views of the address space is another common snag on this effort, and is discussed on a recent thread between myself and Marcin.
Yes I have seen the discussion, but it was mostly dealing with one single QEMU instance modeling all the cores. Here the different address spaces are enforced by multiple QEMU instances.
Quoted 4 lines
Quoted 4 lines
and the OS binary image needs
to be placed in memory at model startup.
Quoted 8 lines
Quoted 3 lines
I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
the problem here was to me that when we launch QEMU a binary needs to be provided and put in memory
in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched.
But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Quoted 45 lines
The information about memory (fd + offset for mmap) is sent only later when the boot is triggered. This is also
safe since the slave will be waiting in the incoming state, and thus no corruption or errors can happen before the
boot is triggered.Quoted 5 lines
Can this effort be a bootloader overhaul? Two things:
1: The bootloader needs to repeatable
2: The bootloaders need to be targetable (to certain CPUs or clusters)Well in this series the bootloader for the master is different from the one for the slave. In my idea the master,
besides thefirmware/kernelimage, will copy also a bootloader for the slave.Quoted 30 lines
Quoted 29 lines
This patch series adds a set of modules and introduces minimal changes to the
current QEMU code-base to implement what described above, with master and slave
implemented as two different instances of QEMU. The aim of this work is to
enable application and runtime programmers to test their AMP applications, or
their new inter-SoC communtication protocol.The main changes are depicted in the following diagram and involve:
- A new multi-client socket implementation that allows multiple
instances of
QEMU to attach to the same socket, with only one acting as a
master.
- A new memory backend, the shared memory backend, based on the file memory backend. Such new backend enables, on the master
side,
to allocate the whole memory as shareable (e.g./dev/shm, or
hugetlbfs).
On the slave side it enables the startup of QEMU without any main
memory
allocated. The the slave goes in a waiting state, the same used in
the
case of an incoming migration, and a callback is registered on a multi-client socket shared with the master.
The waiting state ends when the master sends to the slave the file descriptor and offset to mmap and use as memory.
Quoted 6 lines
Quoted 1 line
This is useful in it's own right and came up in the Xilinx implementation.
It is also mentioned in the video you are pointing, where the Microblaze cores are instantiated as foreign QEMU instances.
Is the code publicly available? There was a question about that in the video but I couldn't catch the answer.
This would probably be a starting point, this is the remote port adapter, which is a generic construct for sending/receiving hardware events to/from things outside QEMU (or other QEMUs):
https://github.com/Xilinx/qemu/blob/pub/2015.2.plnx/hw/core/remote-port.c
This specific device interfaces to RP adapter and lets you send GPIOs between QEMUs:
https://github.com/Xilinx/qemu/blob/pub/2015.2.plnx/hw/core/remote-port-gpio.c
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
- A new inter-processor interrupt hardware distribution module, that
is used
also to trigger the boot of slave processors. Such module uses a
pair of
eventfd for each master-slave couple to trigger interrupts between
the
instances. No slave-to-slave interrupts are envisioned by the
current
implementation.
Quoted 8 lines
Quoted 1 line
Wouldn't that just be a software interrupt in the local QEMU instance?
Since in this proposal there will be multiple instances of QEMU running at the same time, eventfd
are used to signal the event (interrupt) among the different processes. So writing to a register of the IDM
will raise an interrupt to a remote QEMU instance using eventfd. Did this answer your question?
I was thinking more about your comment about slave-to-slave interrupts. This would just trivially be a local software-generated interrupts of some form within the slave cluster.
Quoted 103 lines
Quoted 10 lines
Quoted 8 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to
exchancge the
eventd file descriptors. The IDM device can be instantiated either
as a
PCI or sysbus device.So if everything is is one QEMU, IPIs can be implemented with just a regular interrupt controller (which has a software set).
As said there are multiple instances of QEMU running at the same time, and each of them will see the IDM in their memory map.
Even if the IDM instances will be physically different, because of the multiple processes, all together will act as a single block (e.g., a light version of a mailbox).Quoted 43 lines
Quoted 41 lines
Memory
(e.g. hugetlbfs)+------------------+ +--------------+ +------------------+ | | | | | | | QEMU MASTER | | Master | | QEMU SLAVE | | | | Memory | | | | +------+ +------+-+ | | +-+------+ +------+ | | | | |SHMEM | | | |SHMEM | | | | | | VCPU | |Backend +-----> | +----->Backend | | VCPU | | | | | | | | | | +---> | | | | | +--^---+ +------+-+ | | | | +-+------+ +--^---+ | | | | | | | | | | | | +--+ | | | | | | +---+ | | | IRQ | | +----------+ | | | | IRQ | | | | | | | | | | | | | | | +----+----+ | | | Slave <------+ | | +----+---+ | +--+ IDM +-----+ | | Memory | | | +---+ IDM +-----+ +-^----^--+ | | | | | +-^---^--+ | | | +----------+ | | | | | | +--------------+ | | | | | | | | | +--------------------------------------+-----------+ | | UNIX Domain Socket(send mem fd + offset, trigger boot) | | | +-----------------------------------------------------------+Quoted 2 lines
eventfd
Quoted 4 lines
So the slave can only see a subset of the masters memory? Is the masters memory just the full system memory and the master is doing IOMMU setup for the slave pre-boot? Or is it a hard feature of the physical SoC?
Yes slaves can only see the memory that has been reserved for them. This is ensured by carving out
the memory from the master kernel and providing the offset to such memory to the slave. Each slave
will have its own memory map, and see the memory at the address defined in the machine model.
There is no IOMMU modeled, but it is neither a hard feature since decided at run-time.Quoted 25 lines
Quoted 19 lines
The whole code can be checked out from:
https://git.virtualopensystems.com/dev/qemu-het.git
branch:
qemu-het-rfc-v1Patches apply to the current QEMU master branch
=========
Demo
=========This patch series comes in the form of a demo to better understand how the
changes introduced can be exploited.
At the current status the demo can be executed using an ARM target for both
master and slave.Quoted 6 lines
The demo shows how a master QEMU instance carves out the memory for a slave,
copies inside linux kernel image and device tree blob and finally triggers the
boot.Quoted 5 lines
These processes must have underlying hardware implementation, is the master using a system controller to implement the slave boot? (setting reset and entry points via registers?). How hard are they to model as regular devs?
Quoted 6 lines
In this series the system controller is the IDM device, that through a set of registers makes the master in
"control" each of the slaves. The IDM device is already seen as a regular device by each of the QEMU instances
involved.
I'm starting to think this series is two things that should be decoupled. One is the abstract device(s) to facilitate your AMP, the other is the inter-qemu communication. For the abstract device, I guess this would be a new virtio-idm device. We should try and involve virtio people perhaps. I can see the value in it quite separate from modelling the real sysctrl hardware. But I think the implementation should be free of any inter-QEMU awareness. E.g. from P4 of this series:
+static void send_shmem_fd(IDMState *s, MSClient *c) +{
+ int fd, len; + uint32_t *message; + HostMemoryBackend *backend = MEMORY_BACKEND(s->hostmem); + + len = strlen(SEND_MEM_FD_CMD)/4 + 3; + message = malloc(len * sizeof(uint32_t)); + strcpy((char *) message, SEND_MEM_FD_CMD); + message[len - 2] = s->pboot_size; + message[len - 1] = s->pboot_offset;
+
+ fd = memory_region_get_fd(&backend->mr);
+
+ multi_socket_send_fds_to(c, &fd, 1, (char *) message, len *sizeof(uint32_t));
The device itself is aware of shared-memory and multi-sockets. Using the device for single-QEMU AMP would require neither - can the IDM device be used in a homogeneous AMP flow in one of our existing SMP machine models (eg on a dual core A9 with one core being master and the other slave)?
Can this be architected in two phases for greater utility, with the AMP devices as just normal devices, and the inter-qemu communication as a separate feature?
Regards,
Peter
Hello Peter,
Quoted 128 lines · Peter Crosthwaite
Quoted 71 lines · Christian Pinto
Hello Peter,
thanks for your comments
Quoted 21 lines · Peter Crosthwaite
Quoted 7 lines · Christian Pinto
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".Quoted 8 lines
To recap the goal of such RFC:
The idea is to enhance the current architecture of QEMU to enable the modeling
of a state of-the-art SoC with an AMP processing style, where different processing units share the same system memory and communicate through shared
memory and inter-processor interrupts.This might have a lot in common with a similar inter-qemu communication solution effort at Xilinx. Edgar talks about it at KVM forum:
Quoted 6 lines
https://www.youtube.com/watch?v=L5zG5Aukfek
Around 18:30 mark. I think it might be lower level that your proposal, remote-port is designed to export the raw hardware interfaces (busses and pins) between QEMU and some other system, another QEMU being the common use cases.
Thanks for pointing this out. Indeed what presented by Edgar has a lot of similarities with our proposal, but is targeting a different scenario where low-level modeling of the various hardware components is taken into account.
The goal of my proposal is on the other hand to enable a set of tools for high level early prototyping of systems with a heterogeneous set of cores, so to model a platform that does not exist in reality but that the user wants to experiment with.
As an example I can envision a programming model researcher willing to explore an
heterogeneous system based on an X86 and a multi-core ARM accelerator sharing memory,
to build a new programming paradigm on top of it. Such user would not need the specific details
of the hardware nor all the various devices available in a real SoC, but only an abstract model
encapsulating the main features needed for his research.So to link also to your next comment there is no actual SoC/hardware targeted by this
work.Quoted 13 lines
Quoted 3 lines
An example is a multi-core ARM CPU
working alongside with two Cortex-M micro controllers.Marcin is doing something with A9+M3. It sounds like he already has a lot working (latest emails were on some finer points). What is the board/SoC in question here (if you are able to share)?
Quoted 6 lines
From the user point of view there is usually an operating system booting
on
the Master processor (e.g. Linux) at platform startup, while the other processors are used to offload the Master one from some computation or to deal
with real-time interfaces.Quoted 5 lines
I feel like this is architecting hardware based on common software use cases, rather than directly modelling the SoC in question. Can we model the hardware (e.g. the devices that are used for rpmesg and IPIs etc.) as regular devices, as it is in-SoC? That means AMP is just another guest?
Quoted 4 lines
This is a set of extensions focusing more on the communication channel between the processors
rather than a full SoC model. With this patch series each of the AMP processors is a different "guest".Ok. My issue here is that establishing a 1:1 relationship between QEMU guests and AMP peers is building use-case policy into the mechanism. I am seeing the use-case where there is just one guest that sets up AMP without any QEMU awareness of the AMPness.
Quoted 27 lines
Quoted 16 lines
Quoted 13 lines
It is the Master OS that triggers the boot of the
Slave processors, and provides them also the binary code to execute (e.g. RTOS, binary firmware) by placing it into a pre-defined memory area that is
accessible to the Slaves. Usually the memory for the Slaves is carved out from
the Master OS during boot. Once a Slave is booted the two processors can communicate through queues in shared memory and inter-processor interrupts
(IPIs). In Linux, it is the remoteproc/rpmsg framework that enables the control (boot/shutdown) of Slave processors, and also to establish a communication channel based on virtio queues.Quoted 3 lines
Currently, QEMU is not able to model such an architecture mainly because only
a single processor can be emulated at one time,Quoted 6 lines
SMP does work already. MTTCG will remove the one-run-at-a-time limitation. Multi-arch will allow you to mix multiple CPU architectures (e.g. PPC + ARM in same QEMU). But multiple heterogeneous ARMs should already just work, and there is already an in-tree precedent with the xlnx-zynqmp SoC. That SoC has 4xA53 and 2xR5 (all ARM).
Since Multi-arch is not yet available, with this proposal it is possible to experiment with heterogeneous processors at high level of abstraction, even beyond the ARM + ARM (e.g. X86 + ARM), exploiting the off-the-shelf QEMU.
Quoted 5 lines
One thing I want to add is that all the solutions mentioned in this discussion Multi-arch,
Xilinx's patches, and our proposal could coexist from the code point of view, and none
would prevent the others from being used.We should consolidate where possible though.
Quoted 12 lines
Quoted 3 lines
Multiple system address spaces and CPUs have different views of the address space is another common snag on this effort, and is discussed on a recent thread between myself and Marcin.
Yes I have seen the discussion, but it was mostly dealing with one single QEMU instance modeling all the cores. Here the different address spaces are enforced by multiple QEMU instances.
Quoted 6 lines
Quoted 3 lines
and the OS binary image needs
to be placed in memory at model startup.I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
Quoted 8 lines
Quoted 4 lines
the problem here was to me that when we launch QEMU a binary needs to be provided and put in memory
in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched.But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Actually the address space for a slave is built at init time, the thing that is not completely configured is the memory region modeling the RAM. Such region is configured in terms of size, but there is no pointer to the actual memory. The pointer is mmap-ed later before the slave boots.
Quoted 51 lines
Quoted 42 lines
The information about memory (fd + offset for mmap) is sent only later when the boot is triggered. This is also
safe since the slave will be waiting in the incoming state, and thus no corruption or errors can happen before the
boot is triggered.Quoted 4 lines
Can this effort be a bootloader overhaul? Two things:
1: The bootloader needs to repeatable
2: The bootloaders need to be targetable (to certain CPUs or clusters)Well in this series the bootloader for the master is different from the one for the slave. In my idea the master,
besides thefirmware/kernelimage, will copy also a bootloader for the slave.Quoted 28 lines
Quoted 28 lines
This patch series adds a set of modules and introduces minimal changes to the
current QEMU code-base to implement what described above, with master and slave
implemented as two different instances of QEMU. The aim of this work is to
enable application and runtime programmers to test their AMP applications, or
their new inter-SoC communtication protocol.The main changes are depicted in the following diagram and involve:
- A new multi-client socket implementation that allows multiple
instances of
QEMU to attach to the same socket, with only one acting as a
master.
- A new memory backend, the shared memory backend, based on the file memory backend. Such new backend enables, on the master
side,
to allocate the whole memory as shareable (e.g./dev/shm, or
hugetlbfs).
On the slave side it enables the startup of QEMU without any main
memory
allocated. The the slave goes in a waiting state, the same used in
the
case of an incoming migration, and a callback is registered on a multi-client socket shared with the master.
The waiting state ends when the master sends to the slave the file descriptor and offset to mmap and use as memory.Quoted 5 lines
Quoted 1 line
This is useful in it's own right and came up in the Xilinx implementation.
It is also mentioned in the video you are pointing, where the Microblaze cores are instantiated as foreign QEMU instances.
Is the code publicly available? There was a question about that in the video but I couldn't catch the answer.This would probably be a starting point, this is the remote port adapter, which is a generic construct for sending/receiving hardware events to/from things outside QEMU (or other QEMUs):
Quoted 6 lines
https://github.com/Xilinx/qemu/blob/pub/2015.2.plnx/hw/core/remote-port.c
This specific device interfaces to RP adapter and lets you send GPIOs between QEMUs:
https://github.com/Xilinx/qemu/blob/pub/2015.2.plnx/hw/core/remote-port-gpio.c
Thanks, quite interesting work. As you said already,
that is a lower level approach modeling details like the bus transactions. While the part of memory sharing between the qemu instances is not going through the socket but using the file backed memory, similar to what we do in our work.
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
- A new inter-processor interrupt hardware distribution module, that
is used
also to trigger the boot of slave processors. Such module uses a
pair of
eventfd for each master-slave couple to trigger interrupts between
the
instances. No slave-to-slave interrupts are envisioned by the
current
implementation.Quoted 7 lines
Quoted 1 line
Wouldn't that just be a software interrupt in the local QEMU instance?
Since in this proposal there will be multiple instances of QEMU running at the same time, eventfd
are used to signal the event (interrupt) among the different processes. So writing to a register of the IDM
will raise an interrupt to a remote QEMU instance using eventfd. Did this answer your question?
Quoted 3 lines
I was thinking more about your comment about slave-to-slave interrupts. This would just trivially be a local software-generated interrupts of some form within the slave cluster.
Sorry, I did not catch your comment at first time. You are right, if cores are in the same cluster a software generated interrupt is going to be enough. Of course the eventfd based interrupts make sense for a remote QEMU.
Quoted 105 lines
Quoted 99 lines
Quoted 9 lines
Quoted 7 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to
exchancge the
eventd file descriptors. The IDM device can be instantiated either
as a
PCI or sysbus device.So if everything is is one QEMU, IPIs can be implemented with just a regular interrupt controller (which has a software set).
As said there are multiple instances of QEMU running at the same time, and each of them will see the IDM in their memory map.
Even if the IDM instances will be physically different, because of the multiple processes, all together will act as a single block (e.g., a light version of a mailbox).Quoted 43 lines
Quoted 41 lines
Memory
(e.g. hugetlbfs)+------------------+ +--------------+ +------------------+ | | | | | | | QEMU MASTER | | Master | | QEMU SLAVE | | | | Memory | | | | +------+ +------+-+ | | +-+------+ +------+ | | | | |SHMEM | | | |SHMEM | | | | | | VCPU | |Backend +-----> | +----->Backend | | VCPU | | | | | | | | | | +---> | | | | | +--^---+ +------+-+ | | | | +-+------+ +--^---+ | | | | | | | | | | | | +--+ | | | | | | +---+ | | | IRQ | | +----------+ | | | | IRQ | | | | | | | | | | | | | | | +----+----+ | | | Slave <------+ | | +----+---+ | +--+ IDM +-----+ | | Memory | | | +---+ IDM +-----+ +-^----^--+ | | | | | +-^---^--+ | | | +----------+ | | | | | | +--------------+ | | | | | | | | | +--------------------------------------+-----------+ | | UNIX Domain Socket(send mem fd + offset, trigger boot) | | | +-----------------------------------------------------------+Quoted 2 lines
eventfd
Quoted 4 lines
So the slave can only see a subset of the masters memory? Is the masters memory just the full system memory and the master is doing IOMMU setup for the slave pre-boot? Or is it a hard feature of the physical SoC?
Yes slaves can only see the memory that has been reserved for them. This is ensured by carving out
the memory from the master kernel and providing the offset to such memory to the slave. Each slave
will have its own memory map, and see the memory at the address defined in the machine model.
There is no IOMMU modeled, but it is neither a hard feature since decided at run-time.Quoted 24 lines
Quoted 18 lines
The whole code can be checked out from:
https://git.virtualopensystems.com/dev/qemu-het.git
branch:
qemu-het-rfc-v1Patches apply to the current QEMU master branch
=========
Demo
=========This patch series comes in the form of a demo to better understand how the
changes introduced can be exploited.
At the current status the demo can be executed using an ARM target for both
master and slave.Quoted 6 lines
The demo shows how a master QEMU instance carves out the memory for a slave,
copies inside linux kernel image and device tree blob and finally triggers the
boot.Quoted 5 lines
These processes must have underlying hardware implementation, is the master using a system controller to implement the slave boot? (setting reset and entry points via registers?). How hard are they to model as regular devs?
Quoted 6 lines
In this series the system controller is the IDM device, that through a set of registers makes the master in
"control" each of the slaves. The IDM device is already seen as a regular device by each of the QEMU instances
involved.
Quoted 6 lines
I'm starting to think this series is two things that should be decoupled. One is the abstract device(s) to facilitate your AMP, the other is the inter-qemu communication. For the abstract device, I guess this would be a new virtio-idm device. We should try and involve virtio people perhaps. I can see the value in it quite separate from modelling the real sysctrl hardware.
Interesting, which other value/usage do you see in it? For me the IDM was meant to work as an abstract system controller to centralize the management
of the slaves (boot_regs and interrupts).
Quoted 27 lines
But I think the implementation
should be free of any inter-QEMU awareness. E.g. from P4 of this series:(quoted diff) +15
-0+static void send_shmem_fd(IDMState *s, MSClient *c) +{ + int fd, len; + uint32_t *message; + HostMemoryBackend *backend = MEMORY_BACKEND(s->hostmem); + + len = strlen(SEND_MEM_FD_CMD)/4 + 3; + message = malloc(len * sizeof(uint32_t)); + strcpy((char *) message, SEND_MEM_FD_CMD); + message[len - 2] = s->pboot_size; + message[len - 1] = s->pboot_offset; + + fd = memory_region_get_fd(&backend->mr); + + multi_socket_send_fds_to(c, &fd, 1, (char *) message, len *
sizeof(uint32_t));The device itself is aware of shared-memory and multi-sockets. Using the device for single-QEMU AMP would require neither - can the IDM device be used in a homogeneous AMP flow in one of our existing SMP machine models (eg on a dual core A9 with one core being master and the other slave)?
Quoted 3 lines
Can this be architected in two phases for greater utility, with the AMP devices as just normal devices, and the inter-qemu communication as a separate feature?
I see your point, and it is an interesting proposal.
What I can think here to remove the awareness of how the IDM communicates with the slaves, is to define a kind of AMP Slave interface. So there will be an instance of the interface for each of the slaves, encapsulating the communication part (being either local or based on sockets).
The AMP Slave interfaces would be what you called the AMP devices, with one device per slave.
At master side, besides the IDM, one would instantiate
as many interface devices as slaves. During the initialization the IDM will link with all those interfaces, and only call functions like: send_interrupt() or boot_slave() to interact with the slaves. The interface will be the same for both local or remote slaves, while the implementation of the methods will differ and reside in the specific AMP Slave Interface device.
On the slave side, if the slave is remote, another instance of the interface is instantiated so to connect to socket/eventfd.
So as an example the send_shmem_fd function you pointed could be hidden in the slave interface, and invoked only when the IDM will invoke the slave_boot() function of a remote slave interface.
This would higher the level of abstraction and open the door to potentially any communication mechanism between master and slave, without the need to adapt the IDM device to the specific case. Or, eventually, to mix between local and remote instances.
Thanks,
Christian
Quoted 3 lines
Regards,
Peter
Quoted 44 lines · Christian Pinto
Hello Peter,
Quoted 34 lines · Peter Crosthwaite
Quoted 22 lines · Christian Pinto
Hello Peter,
thanks for your comments
Quoted 16 lines · Peter Crosthwaite
Quoted 13 lines · Christian Pinto
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous
RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".and the OS binary image needs
to be placed in memory at model startup.Quoted 8 lines
Quoted 3 lines
I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
the problem here was to me that when we launch QEMU a binary needs to be provided and put in memory
in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched.Quoted 4 lines
But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Quoted 8 lines
Actually the address space for a slave is built at init time, the thing that is not
completely configured is the memory region modeling the RAM. Such region is configured
in terms of size, but there is no pointer to the actual memory. The pointer is mmap-ed later
before the slave boots.
based on what information? Is the master guest controlling this? If so what is the real-hardware analogue for this concept where the address map of the slave can change (i.e. be configured) at runtime?
Quoted 15 lines
Quoted 10 lines
Quoted 8 lines
The information about memory (fd + offset for mmap) is sent only later when
the boot is triggered. This is also
safe since the slave will be waiting in the incoming state, and thus no corruption or errors can happen before the
boot is triggered.Quoted 3 lines
I was thinking more about your comment about slave-to-slave interrupts. This would just trivially be a local software-generated interrupts of some form within the slave cluster.
Quoted 6 lines
Sorry, I did not catch your comment at first time. You are right, if cores are in the same cluster
a software generated interrupt is going to be enough. Of course the eventfd based interrupts
make sense for a remote QEMU.
Is eventfd a better implementation of remote-port GPIOs as in the Xilinx work?
Re the terminology, I don't like the idea of thinking of inter-qemu "interrupts" as whatever system we decide on should be able to support arbitrary signals going from one QEMU to another. I think the Xilinx work already has reset signals going between the QEMU peers.
Quoted 24 lines
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to
exchancge the
eventd file descriptors. The IDM device can be instantiated
either
as a
PCI or sysbus device.Quoted 7 lines
Quoted 1 line
So if everything is is one QEMU, IPIs can be implemented with just a
of registers makes the master in
"control" each of the slaves. The IDM device is already seen as a regular device by each of the QEMU instances
involved.Quoted 6 lines
I'm starting to think this series is two things that should be decoupled. One is the abstract device(s) to facilitate your AMP, the other is the inter-qemu communication. For the abstract device, I guess this would be a new virtio-idm device. We should try and involve virtio people perhaps. I can see the value in it quite separate from modelling the real sysctrl hardware.
Quoted 2 lines
Interesting, which other value/usage do you see in it? For me the IDM was meant to
It has value in prototyping with your abstract toolkit even with homogeneous hardware. E.g. I should be able to just use single-QEMU ARM virt machine -smp 2 and create one of these virtio-AMP setups. Homogeneous hardware with heterogenous software using your new pieces of abstract hardware.
It is also more practical for getting a merge of your work as you are targetting two different audiences with the work. People intersted in virtio can handle the new devices you create, while the core maintainers can handle your multi-QEMU work. It is two rather big new features.
Quoted 38 lines
work as an abstract system controller to centralize the management of the slaves (
boot_regsand interrupts).Quoted 27 lines
But I think the implementation
should be free of any inter-QEMU awareness. E.g. from P4 of this series:(quoted diff) +15
-0+static void send_shmem_fd(IDMState *s, MSClient *c) +{ + int fd, len; + uint32_t *message; + HostMemoryBackend *backend = MEMORY_BACKEND(s->hostmem); + + len = strlen(SEND_MEM_FD_CMD)/4 + 3; + message = malloc(len * sizeof(uint32_t)); + strcpy((char *) message, SEND_MEM_FD_CMD); + message[len - 2] = s->pboot_size; + message[len - 1] = s->pboot_offset; + + fd = memory_region_get_fd(&backend->mr); + + multi_socket_send_fds_to(c, &fd, 1, (char *) message, len *
sizeof(uint32_t));The device itself is aware of shared-memory and multi-sockets. Using the device for single-QEMU AMP would require neither - can the IDM device be used in a homogeneous AMP flow in one of our existing SMP machine models (eg on a dual core A9 with one core being master and the other slave)?
Quoted 3 lines
Can this be architected in two phases for greater utility, with the AMP devices as just normal devices, and the inter-qemu communication as a separate feature?
I see your point, and it is an interesting proposal.
Quoted 8 lines
What I can think here to remove the awareness of how the IDM communicates with
the slaves, is to define a kind of AMP Slave interface. So there will be an instance of the interface for each of the slaves, encapsulating the communication part (being either local or based on sockets). The AMP Slave interfaces would be what you called the AMP devices, with one device per slave.
Do we need this hard definition of master and slave in the hardware? Can the virtio-device be more peer-peer and the master-slave relationship is purely implemented by the guest?
Regards,
Peter
Quoted 32 lines
At master side, besides the IDM, one would instantiate
as many interface devices as slaves. During the initialization the IDM will link
with all those interfaces, and only call functions like:send_interrupt()orboot_slave()to interact with the slaves. The interface will be the same for both local or remote slaves, while the implementation of the methods will differ and reside in the specific AMP Slave Interface device.
On the slave side, if the slave is remote, another instance of the interface is instantiated so to connect to socket/eventfd.So as an example the
send_shmem_fdfunction you pointed could be hidden in the
slave interface, and invoked only when the IDM will invoke theslave_boot()function of a remote slave interface.This would higher the level of abstraction and open the door to potentially any
communication mechanism between master and slave, without the need to adapt the
IDM device to the specific case. Or, eventually, to mix between local and remote instances.Thanks,
Christian
Quoted 3 lines
Regards,
Peter
W dniu 25.10.2015 o 22:38, Peter Crosthwaite pisze:
Quoted 15 lines
Quoted 13 lines · Christian Pinto
Hello Peter,
Quoted 9 lines · Peter Crosthwaite
Quoted 7 lines · Christian Pinto
Hello Peter,
thanks for your comments
Quoted 2 lines · Peter Crosthwaite
On Tue, Sep 29, 2015 at 6:57 AM, Christian Pinto
<[email protected]> wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous
RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".
Sorry for late response, unfortunately my M3+A9 SoC can not be published (yet). But I am working on it.
Quoted 23 lines
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
and the OS binary image needs
to be placed in memory at model startup.I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
Quoted 8 lines
Quoted 4 lines
the problem here was to me that when we launch QEMU a binary needs to be provided and put in memory
in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched.But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Quoted 8 lines
Actually the address space for a slave is built at init time, the thing that is not
completely configured is the memory region modeling the RAM. Such region is configured
in terms of size, but there is no pointer to the actual memory. The pointer is mmap-ed later
before the slave boots.
Quoted 3 lines
based on what information? Is the master guest controlling this? If so what is the real-hardware analogue for this concept where the address map of the slave can change (i.e. be configured) at runtime?
I am not sure if it is the case since I haven't emulated this yet (and it has very low priority),
but I might have a real case in my M3+A9 - M3 has 256MiB window that can be moved over the 1GiB system memory at runtime.
Quoted 104 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
The information about memory (fd + offset for mmap) is sent only later when
the boot is triggered. This is also
safe since the slave will be waiting in the incoming state, and thus no corruption or errors can happen before the
boot is triggered.I was thinking more about your comment about slave-to-slave interrupts. This would just trivially be a local software-generated interrupts of some form within the slave cluster.
Quoted 6 lines
Sorry, I did not catch your comment at first time. You are right, if cores are in the same cluster
a software generated interrupt is going to be enough. Of course the eventfd based interrupts
make sense for a remote QEMU.Is eventfd a better implementation of remote-port GPIOs as in the Xilinx work?
Re the terminology, I don't like the idea of thinking of inter-qemu "interrupts" as whatever system we decide on should be able to support arbitrary signals going from one QEMU to another. I think the Xilinx work already has reset signals going between the QEMU peers.
Quoted 21 lines
Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to
exchancge the
eventd file descriptors. The IDM device can be instantiated
either
as a
PCI or sysbus device.Quoted 6 lines
Quoted 1 line
So if everything is is one QEMU, IPIs can be implemented with just a
of registers makes the master in
"control" each of the slaves. The IDM device is already seen as a regular device by each of the QEMU instances
involved.Quoted 6 lines
I'm starting to think this series is two things that should be decoupled. One is the abstract device(s) to facilitate your AMP, the other is the inter-qemu communication. For the abstract device, I guess this would be a new virtio-idm device. We should try and involve virtio people perhaps. I can see the value in it quite separate from modelling the real sysctrl hardware.
Quoted 2 lines
Interesting, which other value/usage do you see in it? For me the IDM was meant to
It has value in prototyping with your abstract toolkit even with homogeneous hardware. E.g. I should be able to just use single-QEMU ARM virt machine -smp 2 and create one of these virtio-AMP setups. Homogeneous hardware with heterogenous software using your new pieces of abstract hardware.
It is also more practical for getting a merge of your work as you are targetting two different audiences with the work. People intersted in virtio can handle the new devices you create, while the core maintainers can handle your multi-QEMU work. It is two rather big new features.
Quoted 37 lines
work as an abstract system controller to centralize the management of the slaves (
boot_regsand interrupts).Quoted 27 lines
But I think the implementation
should be free of any inter-QEMU awareness. E.g. from P4 of this series:(quoted diff) +15
-0+static void send_shmem_fd(IDMState *s, MSClient *c) +{ + int fd, len; + uint32_t *message; + HostMemoryBackend *backend = MEMORY_BACKEND(s->hostmem); + + len = strlen(SEND_MEM_FD_CMD)/4 + 3; + message = malloc(len * sizeof(uint32_t)); + strcpy((char *) message, SEND_MEM_FD_CMD); + message[len - 2] = s->pboot_size; + message[len - 1] = s->pboot_offset; + + fd = memory_region_get_fd(&backend->mr); + + multi_socket_send_fds_to(c, &fd, 1, (char *) message, len *
sizeof(uint32_t));The device itself is aware of shared-memory and multi-sockets. Using the device for single-QEMU AMP would require neither - can the IDM device be used in a homogeneous AMP flow in one of our existing SMP machine models (eg on a dual core A9 with one core being master and the other slave)?
Quoted 3 lines
Can this be architected in two phases for greater utility, with the AMP devices as just normal devices, and the inter-qemu communication as a separate feature?
I see your point, and it is an interesting proposal.
Quoted 8 lines
What I can think here to remove the awareness of how the IDM communicates with
the slaves, is to define a kind of AMP Slave interface. So there will be an instance of the interface for each of the slaves, encapsulating the communication part (being either local or based on sockets). The AMP Slave interfaces would be what you called the AMP devices, with one device per slave.
Quoted 37 lines
Do we need this hard definition of master and slave in the hardware? Can the virtio-device be more peer-peer and the master-slave relationship is purely implemented by the guest?
Regards,
PeterQuoted 30 lines
At master side, besides the IDM, one would instantiate
as many interface devices as slaves. During the initialization the IDM will link
with all those interfaces, and only call functions like:send_interrupt()orboot_slave()to interact with the slaves. The interface will be the same for both local or remote slaves, while the implementation of the methods will differ and reside in the specific AMP Slave Interface device.
On the slave side, if the slave is remote, another instance of the interface is instantiated so to connect to socket/eventfd.So as an example the
send_shmem_fdfunction you pointed could be hidden in the
slave interface, and invoked only when the IDM will invoke theslave_boot()function of a remote slave interface.This would higher the level of abstraction and open the door to potentially any
communication mechanism between master and slave, without the need to adapt the
IDM device to the specific case. Or, eventually, to mix between local and remote instances.Thanks,
Christian
Quoted 2 lines
Regards,
Peter
Regards,
Marcin
Quoted 66 lines · mar.krzeminski
W dniu 25.10.2015 o 22:38, Peter Crosthwaite pisze:
Quoted 21 lines
Quoted 18 lines · Christian Pinto
Hello Peter,
Quoted 13 lines · Peter Crosthwaite
Quoted 10 lines · Christian Pinto
Hello Peter,
thanks for your comments
Quoted 4 lines · Peter Crosthwaite
Quoted 1 line · Christian Pinto
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous
RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".Sorry for late response, unfortunately my M3+A9 SoC can not be published (yet).
But I am working on it.Quoted 31 lines
Quoted 31 lines
Quoted 14 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
and the OS binary image needs
to be placed in memory at model startup.I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
Quoted 5 lines
the problem here was to me that when we launch QEMU a binary needs to be
provided and put in memory
in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched.Quoted 4 lines
But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Actually the address space for a slave is built at init time, the thing that
is not
completely configured is the memory region modeling the RAM. Such region is
configured
in terms of size, but there is no pointer to the actual memory. The pointer
is mmap-ed later
before the slave boots.
Quoted 9 lines
Quoted 3 lines
based on what information? Is the master guest controlling this? If so what is the real-hardware analogue for this concept where the address map of the slave can change (i.e. be configured) at runtime?
I am not sure if it is the case since I haven't emulated this yet (and it has very low priority),
but I might have a real case in my M3+A9 - M3 has 256MiB window that can be moved over the 1GiB system memory at runtime.
Right, the main thing for me there, is it works at runtime. The master in Christians scheme would be able to remap it by configuring registers. The same runtime reconfigurability should apply to the slave memory mapping. Rather than having to declare the slave memory map pre machine init, it should just be runtime configured by such a device. Then there is no need for inter-qemu communication of machine-init data. The two machines are inited independently, with the communication channel only being used for runtime data. If there is hardware support for it, remapping another processors address space is a valid runtime operation for which we have support for at core layers.
Regards,
Peter
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Regards,
Marcin
Quoted 44 lines · Peter Crosthwaite
Quoted 34 lines · Christian Pinto
Hello Peter,
Quoted 22 lines · Peter Crosthwaite
Quoted 20 lines · Christian Pinto
Hello Peter,
thanks for your comments
Quoted 15 lines · Peter Crosthwaite
Quoted 10 lines · Christian Pinto
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous
RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU". and the OS binary image needs
to be placed in memory at model startup.I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
Quoted 8 lines
Quoted 4 lines
the problem here was to me that when we launch QEMU a binary needs to be provided and put in memory
in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched.But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Quoted 8 lines
Actually the address space for a slave is built at init time, the thing that is not
completely configured is the memory region modeling the RAM. Such region is configured
in terms of size, but there is no pointer to the actual memory. The pointer is mmap-ed later
before the slave boots.
Quoted 3 lines
based on what information? Is the master guest controlling this? If so what is the real-hardware analogue for this concept where the address map of the slave can change (i.e. be configured) at runtime?
Hello Peter,
The memory map of a slave is not controlled by the master guest, since it is dependent from the machine model used for the slave. The only thing the master controls is the subset of the main memory that is assigned to a slave. By
saying that the memory pointer is sent to the slave later, before the boot, it is like setting the boot address for that specific slave within the whole platform memory. So essentially the offset passed for the mmap is from beginning of master memory up to the beginning of the memory carved out for the specific slave. I see this as a way to protect the master memory from malicious accesses from the slave side, so this way the slave will only "see" the part of the memory that it got assigned.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
The information about memory (fd + offset for mmap) is sent only later when
the boot is triggered. This is also
safe since the slave will be waiting in the incoming state, and thus no corruption or errors can happen before the
boot is triggered.I was thinking more about your comment about slave-to-slave interrupts. This would just trivially be a local software-generated interrupts of some form within the slave cluster.
Quoted 7 lines
Quoted 6 lines
Sorry, I did not catch your comment at first time. You are right, if cores are in the same cluster
a software generated interrupt is going to be enough. Of course the eventfd based interrupts
make sense for a remote QEMU.Is eventfd a better implementation of remote-port GPIOs as in the Xilinx work?
Functionally I think they provide the same behavior. We went for eventfd since when designing the code of the IDM we based it on what available on upstream QEMU to signal events between processes (e.g., eventfd).
Quoted 4 lines
Re the terminology, I don't like the idea of thinking of inter-qemu "interrupts" as whatever system we decide on should be able to support arbitrary signals going from one QEMU to another. I think the Xilinx work already has reset signals going between the QEMU peers.
We used the inter-qemu interrupt term, since such signal was triggered from the IDM and is an interrupt. But I see your point and agree that such interrupt could be a generic inter-qemu signaling mechanism, that can be used as interrupt for this specific purpose.
Quoted 28 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 8 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to
exchancge the
eventd file descriptors. The IDM device can be instantiated
either
as a
PCI or sysbus device.So if everything is is one QEMU, IPIs can be implemented with just a
of registers makes the master in
"control" each of the slaves. The IDM device is already seen as a regular device by each of the QEMU instances
involved.Quoted 8 lines
Quoted 6 lines
I'm starting to think this series is two things that should be decoupled. One is the abstract device(s) to facilitate your AMP, the other is the inter-qemu communication. For the abstract device, I guess this would be a new virtio-idm device. We should try and involve virtio people perhaps. I can see the value in it quite separate from modelling the real sysctrl hardware.
Interesting, which other value/usage do you see in it? For me the IDM was meant to
It has value in prototyping with your abstract toolkit even with homogeneous hardware. E.g. I should be able to just use single-QEMU ARM virt machine -smp 2 and create one of these virtio-AMP setups. Homogeneous hardware with heterogenous software using your new pieces of abstract hardware.
Quoted 5 lines
It is also more practical for getting a merge of your work as you are targetting two different audiences with the work. People intersted in virtio can handle the new devices you create, while the core maintainers can handle your multi-QEMU work. It is two rather big new features.
This is true, too much meat on the fire for the same patch makes it difficult to get merged. Thanks.
We could split in multi-client socket work, the inter-qemu communication and virtio-idm.
Quoted 45 lines
Quoted 37 lines
work as an abstract system controller to centralize the management of the slaves (
boot_regsand interrupts).Quoted 27 lines
But I think the implementation
should be free of any inter-QEMU awareness. E.g. from P4 of this series:(quoted diff) +15
-0+static void send_shmem_fd(IDMState *s, MSClient *c) +{ + int fd, len; + uint32_t *message; + HostMemoryBackend *backend = MEMORY_BACKEND(s->hostmem); + + len = strlen(SEND_MEM_FD_CMD)/4 + 3; + message = malloc(len * sizeof(uint32_t)); + strcpy((char *) message, SEND_MEM_FD_CMD); + message[len - 2] = s->pboot_size; + message[len - 1] = s->pboot_offset; + + fd = memory_region_get_fd(&backend->mr); + + multi_socket_send_fds_to(c, &fd, 1, (char *) message, len *
sizeof(uint32_t));The device itself is aware of shared-memory and multi-sockets. Using the device for single-QEMU AMP would require neither - can the IDM device be used in a homogeneous AMP flow in one of our existing SMP machine models (eg on a dual core A9 with one core being master and the other slave)?
Quoted 3 lines
Can this be architected in two phases for greater utility, with the AMP devices as just normal devices, and the inter-qemu communication as a separate feature?
I see your point, and it is an interesting proposal.
Quoted 8 lines
What I can think here to remove the awareness of how the IDM communicates with
the slaves, is to define a kind of AMP Slave interface. So there will be an instance of the interface for each of the slaves, encapsulating the communication part (being either local or based on sockets). The AMP Slave interfaces would be what you called the AMP devices, with one device per slave.
Quoted 3 lines
Do we need this hard definition of master and slave in the hardware? Can the virtio-device be more peer-peer and the master-slave relationship is purely implemented by the guest?
I think we can architect it in a way that the virtio-idm simply connects two or more peers, and depending from the usage done by the software, behaving as master from one side and slave on the other. I used the term slave AMP interface, I should have used AMP client interface, to indicate the cores/procesors the IDM has inter-connect (being local or on another QEMU instance).
So there would be an implementation of the AMP client interface that is based on the assumption that all the processors are on the same instance, and one based on sockets for the remote instances.
to make an example, for a single qemu instance with -smp 2 you would add something like :
-smp 2
-device amp-local-client, core_id=0, id=client0
-device amp-local-client, core_id=1, id=client1
-device virtio-idm, clients=2, id=idm
while for remote qemu instances something like
(the opposite to be instantiated on the other remote instance):
-device amp-local-client, id=client0
-device amp-remote-client, chardev=chdev_id, id=client1 -device virtio-idm, clients=2, id=idm-dev
This way the idm only knows about clients (all clients are the same for the IDM). The software running on the processors will enable the interaction between the clients by writing into the IDM device registers.
At a first glance, and according to my current proposal, I see such AMP client interfaces exporting the following methods:
* raise_interrupt() function: called by the IDM to trigger an interrupt towards the destination client
* boot_trigger() function: called by the IDM to trigger the boot of the client
If the clients are remote, socket communication will be used and hidden in the AMP client interface implementation
Do you foresee a different type of interface for the use-case you have in mind? I ask because if for example the clients are cores of the same cluster (and same instance), interrupts could simply be software generated from the linux-kernel/firmware running on top of the processors and theoretically no need to go through the IDM, same I guess for the boot.
Another thing that needs to be defined clearly is the interface between the IDM and the software running on the cores.
At the moment I am using a set of registers, namely the boot and the interrupt registers. By writing the ID of a client in such registers it is possible to forward an interrupt or trigger its boot.
Thanks,
Christian
Quoted 33 lines
Regards,
PeterQuoted 29 lines
At master side, besides the IDM, one would instantiate
as many interface devices as slaves. During the initialization the IDM will link
with all those interfaces, and only call functions like:send_interrupt()orboot_slave()to interact with the slaves. The interface will be the same for both local or remote slaves, while the implementation of the methods will differ and reside in the specific AMP Slave Interface device.
On the slave side, if the slave is remote, another instance of the interface is instantiated so to connect to socket/eventfd.So as an example the
send_shmem_fdfunction you pointed could be hidden in the
slave interface, and invoked only when the IDM will invoke theslave_boot()function of a remote slave interface.This would higher the level of abstraction and open the door to potentially any
communication mechanism between master and slave, without the need to adapt the
IDM device to the specific case. Or, eventually, to mix between local and remote instances.Thanks,
Christian
Quoted 2 lines
Regards,
Peter
Hi Christian,
Sorry about the delayed response.
Quoted 62 lines · Christian Pinto
On 25/10/2015 22:38, Peter Crosthwaite wrote:
On Thu, Oct 22, 2015 at 2:21 AM, Christian Pinto<[email protected]> <[email protected]> wrote:
Hello Peter,
On 07/10/2015 17:48, Peter Crosthwaite wrote:
On Mon, Oct 5, 2015 at 8:50 AM, Christian Pinto<[email protected]> <[email protected]> wrote:
Hello Peter,
thanks for your comments
On 01/10/2015 18:26, Peter Crosthwaite wrote:
On Tue, Sep 29, 2015 at 6:57 AM, Christian Pinto<[email protected]> <[email protected]> wrote:
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous
RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".and the OS binary image needs
to be placed in memory at model startup.I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
the problem here was to me that when we launch QEMU a binary needs to be provided and put in memory
in order to be executed. In this patch series the slave doesn't have a proper memory allocated when first launched.But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Actually the address space for a slave is built at init time, the thing that is not
completely configured is the memory region modeling the RAM. Such region is configured
in terms of size, but there is no pointer to the actual memory. The pointer is mmap-ed later
before the slave boots.based on what information? Is the master guest controlling this? If so what is the real-hardware analogue for this concept where the address map of the slave can change (i.e. be configured) at runtime?
Hello Peter,
Quoted 17 lines
The memory map of a slave is not controlled by the master guest, since it is
dependent from the machine model used for the slave. The only thing the master
controls is the subset of the main memory that is assigned to a slave. By saying that the memory pointer is sent to the slave later, before the boot, it is like setting the
boot address for that specific slave within the whole platform memory. So essentially the offset passed for the mmap is from beginning of master memory up to the
beginning of the memory carved out for the specific slave. I see this as a way to
protect the master memory from malicious accesses from the slave side, so this
way the slave will only "see" the part of the memory that it got assigned.
That does sound like memory map control though. Is it simpler to just give the slave full access and implement such protections as a specific feature (probably some sort of IOMMU)?
Quoted 81 lines
The information about memory (fd + offset for mmap) is sent only later when
the boot is triggered. This is also
safe since the slave will be waiting in the incoming state, and thus no corruption or errors can happen before the
boot is triggered.I was thinking more about your comment about slave-to-slave interrupts. This would just trivially be a local software-generated interrupts of some form within the slave cluster.
Sorry, I did not catch your comment at first time. You are right, if cores are in the same cluster
a software generated interrupt is going to be enough. Of course the eventfd based interrupts
make sense for a remote QEMU.Is eventfd a better implementation of remote-port GPIOs as in the Xilinx work?
Functionally I think they provide the same behavior. We went for eventfd since
when designing the code of the IDM we based it on what available on upstream QEMU
to signal events between processes (e.g., eventfd).Re the terminology, I don't like the idea of thinking of inter-qemu "interrupts" as whatever system we decide on should be able to support arbitrary signals going from one QEMU to another. I think the Xilinx work already has reset signals going between the QEMU peers.
We used the inter-qemu interrupt term, since such signal was triggered from the IDM
and is an interrupt. But I see your point and agree that such interrupt could be a generic
inter-qemu signaling mechanism, that can be used as interrupt for this specific purpose.The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple to
exchancge the
eventd file descriptors. The IDM device can be instantiated
either
as a
PCI or sysbus device.So if everything is is one QEMU, IPIs can be implemented with just a
of registers makes the master in
"control" each of the slaves. The IDM device is already seen as a regular device by each of the QEMU instances
involved.I'm starting to think this series is two things that should be decoupled. One is the abstract device(s) to facilitate your AMP, the other is the inter-qemu communication. For the abstract device, I guess this would be a new virtio-idm device. We should try and involve virtio people perhaps. I can see the value in it quite separate from modelling the real sysctrl hardware.
Interesting, which other value/usage do you see in it? For me the IDM was meant to
It has value in prototyping with your abstract toolkit even with homogeneous hardware. E.g. I should be able to just use single-QEMU ARM virt machine -smp 2 and create one of these virtio-AMP setups. Homogeneous hardware with heterogenous software using your new pieces of abstract hardware.
It is also more practical for getting a merge of your work as you are targetting two different audiences with the work. People intersted in virtio can handle the new devices you create, while the core maintainers can handle your multi-QEMU work. It is two rather big new features.
Quoted 6 lines
This is true, too much meat on the fire for the same patch makes it difficult to get merged. Thanks.
We could split in multi-client socket work, the inter-qemu communication and virtio-idm.
OK.
Quoted 53 lines
work as an abstract system controller to centralize the management of the slaves (
boot_regsand interrupts).But I think the implementation
should be free of any inter-QEMU awareness. E.g. from P4 of this series:(quoted diff) +15
-0+static void send_shmem_fd(IDMState *s, MSClient *c) +{ + int fd, len; + uint32_t *message; + HostMemoryBackend *backend = MEMORY_BACKEND(s->hostmem); + + len = strlen(SEND_MEM_FD_CMD)/4 + 3; + message = malloc(len * sizeof(uint32_t)); + strcpy((char *) message, SEND_MEM_FD_CMD); + message[len - 2] = s->pboot_size; + message[len - 1] = s->pboot_offset; + + fd = memory_region_get_fd(&backend->mr); + + multi_socket_send_fds_to(c, &fd, 1, (char *) message, len *
sizeof(uint32_t));The device itself is aware of shared-memory and multi-sockets. Using the device for single-QEMU AMP would require neither - can the IDM device be used in a homogeneous AMP flow in one of our existing SMP machine models (eg on a dual core A9 with one core being master and the other slave)?
Can this be architected in two phases for greater utility, with the AMP devices as just normal devices, and the inter-qemu communication as a separate feature?
I see your point, and it is an interesting proposal.
What I can think here to remove the awareness of how the IDM communicates with
the slaves, is to define a kind of AMP Slave interface. So there will be an instance of the interface for each of the slaves, encapsulating the communication part (being either local or based on sockets). The AMP Slave interfaces would be what you called the AMP devices, with one device per slave.Do we need this hard definition of master and slave in the hardware? Can the virtio-device be more peer-peer and the master-slave relationship is purely implemented by the guest?
Quoted 11 lines
I think we can architect it in a way that the virtio-idm simply connects two or more peers, and depending from the usage done by the software, behaving as master from one side and slave on the other. I used the term slave AMP interface, I should have used AMP client interface, to indicate the cores/procesors the IDM has inter-connect (being local or on another QEMU instance).
So there would be an implementation of the AMP client interface that is based on the assumption that all the processors are on the same instance, and one based on sockets for the remote instances.
Do you need this dual mode? Can the IDM just have GPIOs which are then either directly connected to the local CPUs, or sent out over inter-qemu connectivity mechanism? Then the inter-qemu can be used for any GPIO communication.
Quoted 34 lines
to make an example, for a single qemu instance with -smp 2 you would add something like :
(quoted diff) +0
-4-smp 2 -device amp-local-client, core_id=0, id=client0 -device amp-local-client, core_id=1, id=client1 -device virtio-idm, clients=2, id=idmwhile for remote qemu instances something like
(the opposite to be instantiated on the other remote instance):(quoted diff) +0
-3-device amp-local-client, id=client0 -device amp-remote-client, chardev=chdev_id, id=client1 -device virtio-idm, clients=2, id=idm-devThis way the idm only knows about clients (all clients are the same for the IDM). The software running on the processors will enable the interaction between the clients by writing into the IDM device registers.
At a first glance, and according to my current proposal, I see such AMP client interfaces exporting the following methods:
-
raise_interrupt()function: called by the IDM to trigger an interrupt towards the destination client-
boot_trigger()function: called by the IDM to trigger the boot of the clientIf the clients are remote, socket communication will be used and hidden in the AMP client interface implementation
Quoted 7 lines
Do you foresee a different type of interface for the use-case you have in mind? I ask because if for example the clients are cores of the same cluster (and same instance), interrupts could simply be software generated from the linux-kernel/firmware running on top of the processors and theoretically no need to go through the IDM, same I guess for the boot.
True. But if you are developing code for IDM, you can do a crawl-before-walk test with an SMP test case.
Regards,
Peter
Quoted 50 lines
Another thing that needs to be defined clearly is the interface between the IDM and the software running on the cores.
At the moment I am using a set of registers, namely the boot and the interrupt registers. By writing the ID of a client in such registers it is possible to forward an interrupt or trigger its boot.Thanks,
Christian
Regards,
PeterAt master side, besides the IDM, one would instantiate
as many interface devices as slaves. During the initialization the IDM will link
with all those interfaces, and only call functions like:send_interrupt()orboot_slave()to interact with the slaves. The interface will be the same for both local or remote slaves, while the implementation of the methods will differ and reside in the specific AMP Slave Interface device.
On the slave side, if the slave is remote, another instance of the interface is instantiated so to connect to socket/eventfd.So as an example the
send_shmem_fdfunction you pointed could be hidden in the
slave interface, and invoked only when the IDM will invoke theslave_boot()function of a remote slave interface.This would higher the level of abstraction and open the door to potentially any
communication mechanism between master and slave, without the need to adapt the
IDM device to the specific case. Or, eventually, to mix between local and remote instances.Thanks,
Christian
Regards,
Peter
Hello Peter,
Apologies for the highly delayed response.
Quoted 6 lines · Peter Crosthwaite
Hi Christian,
Sorry about the delayed response.
On Tue, Oct 27, 2015 at 3:30 AM, Christian Pinto <
Quoted 33 lines · [email protected]> wrote:
Quoted 33 lines · Peter Crosthwaite
Quoted 29 lines
Quoted 26 lines · Christian Pinto
Hello Peter,
Quoted 21 lines · Peter Crosthwaite
Quoted 18 lines · Christian Pinto
Hello Peter,
thanks for your comments
Quoted 12 lines · Peter Crosthwaite
Quoted 9 lines · Christian Pinto
Hi all,
This RFC patch-series introduces the set of changes enabling the architectural elements to model the architecture presented in a previous
RFC
letter: "[Qemu-devel][RFC] Towards an Heterogeneous QEMU".
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
and the OS binary image needs
to be placed in memory at model startup.I don't see what this limitation is exactly. Can you explain more? I do see a need to work on the ARM bootloader for AMP flows, it is a pure SMP bootloader than assumes total control.
the problem here was to me that when we launch QEMU a binary needs
to be
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
provided and put in memory
in order to be executed. In this patch series the slave doesn't have
a
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
proper memory allocated when first launched.
But it could though couldn't it? Can't the slave guest just have full access to it's own address space (probably very similar to the masters address space) from machine init time? This seems more realistic than setting up the hardware based on guest level information.
Actually the address space for a slave is built at init time, the
thing that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
is not
completely configured is the memory region modeling the RAM. Such
region is
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
configured
in terms of size, but there is no pointer to the actual memory. The
pointer
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
is mmap-ed later
before the slave boots.based on what information? Is the master guest controlling this? If so what is the real-hardware analogue for this concept where the address map of the slave can change (i.e. be configured) at runtime?
Hello Peter,
The memory map of a slave is not controlled by the master guest, since
it is
Quoted 1 line
Quoted 1 line
dependent from the machine model used for the slave. The only thing the
master
Quoted 1 line
Quoted 1 line
controls is the subset of the main memory that is assigned to a slave.
By
Quoted 1 line
Quoted 1 line
saying that the memory pointer is sent to the slave later, before the
boot, it is like setting the
Quoted 2 lines
Quoted 2 lines
boot address for that specific slave within the whole platform memory. So essentially the offset passed for the mmap is from beginning of master
memory up to the
Quoted 1 line
Quoted 1 line
beginning of the memory carved out for the specific slave. I see this as
a way to
Quoted 1 line
Quoted 1 line
protect the master memory from malicious accesses from the slave side,
so this
Quoted 1 line
Quoted 1 line
way the slave will only "see" the part of the memory that it got
assigned.
Quoted 3 lines
Quoted 1 line
That does sound like memory map control though. Is it simpler to just
give the slave full access and implement such protections as a specific feature (probably some sort of IOMMU)?
Quoted 1 line
Yes it is a kind of memory map control. An IOMMU like component would do the job, but as said already most of the focus of this project was on the IDM and the inter-QEMU communication. We could consider the implementation of an IOMMU like component in future extensions.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The information about memory (fd + offset for mmap) is sent only
later
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
when
the boot is triggered. This is also
safe since the slave will be waiting in the incoming state, and thus
no
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
corruption or errors can happen before the boot is triggered.
I was thinking more about your comment about slave-to-slave interrupts. This would just trivially be a local software-generated interrupts of some form within the slave cluster.
Sorry, I did not catch your comment at first time. You are right, if
cores
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
are in the same cluster
a software generated interrupt is going to be enough. Of course the
eventfd
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
based interrupts
make sense for a remote QEMU.Is eventfd a better implementation of remote-port GPIOs as in the
Xilinx work?
Quoted 3 lines
Quoted 3 lines
Functionally I think they provide the same behavior. We went for eventfd
since
Quoted 1 line
Quoted 1 line
when designing the code of the IDM we based it on what available on
upstream QEMU
Quoted 9 lines
Quoted 9 lines
to signal events between processes (e.g., eventfd).
Quoted 4 lines
Re the terminology, I don't like the idea of thinking of inter-qemu "interrupts" as whatever system we decide on should be able to support arbitrary signals going from one QEMU to another. I think the Xilinx work already has reset signals going between the QEMU peers.
We used the inter-qemu interrupt term, since such signal was triggered
from the IDM
Quoted 1 line
Quoted 1 line
and is an interrupt. But I see your point and agree that such interrupt
could be a generic
Quoted 1 line
Quoted 1 line
inter-qemu signaling mechanism, that can be used as interrupt for this
specific purpose.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The multi client-socket is used for the master to trigger
the boot of a slave, and also for each master-slave couple
to
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
exchancge the
eventd file descriptors. The IDM device can be instantiated
either
as a
PCI or sysbus device.So if everything is is one QEMU, IPIs can be implemented with just a
of registers makes the master in
"control" each of the slaves. The IDM device is already seen as a
regular
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
device by each of the QEMU instances involved.
I'm starting to think this series is two things that should be decoupled. One is the abstract device(s) to facilitate your AMP, the other is the inter-qemu communication. For the abstract device, I guess this would be a new virtio-idm device. We should try and involve virtio people perhaps. I can see the value in it quite separate from modelling the real sysctrl hardware.
Interesting, which other value/usage do you see in it? For me the IDM
was
Quoted 15 lines
Quoted 15 lines
Quoted 13 lines
Quoted 1 line
meant to
It has value in prototyping with your abstract toolkit even with homogeneous hardware. E.g. I should be able to just use single-QEMU ARM virt machine -smp 2 and create one of these virtio-AMP setups. Homogeneous hardware with heterogenous software using your new pieces of abstract hardware.
It is also more practical for getting a merge of your work as you are targetting two different audiences with the work. People intersted in virtio can handle the new devices you create, while the core maintainers can handle your multi-QEMU work. It is two rather big new features.
Quoted 48 lines
Quoted 5 lines
This is true, too much meat on the fire for the same patch makes it difficult to get merged. Thanks.
We could split in multi-client socket work, the inter-qemu communication and virtio-idm.OK.
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
work as an abstract system controller to centralize the management of the slaves (
boot_regsand interrupts).Quoted 27 lines
But I think the implementation
should be free of any inter-QEMU awareness. E.g. from P4 of this series:(quoted diff) +15
-0+static void send_shmem_fd(IDMState *s, MSClient *c) +{ + int fd, len; + uint32_t *message; + HostMemoryBackend *backend = MEMORY_BACKEND(s->hostmem); + + len = strlen(SEND_MEM_FD_CMD)/4 + 3; + message = malloc(len * sizeof(uint32_t)); + strcpy((char *) message, SEND_MEM_FD_CMD); + message[len - 2] = s->pboot_size; + message[len - 1] = s->pboot_offset; + + fd = memory_region_get_fd(&backend->mr); + + multi_socket_send_fds_to(c, &fd, 1, (char *) message, len *
sizeof(uint32_t));The device itself is aware of shared-memory and multi-sockets. Using the device for single-QEMU AMP would require neither - can the IDM device be used in a homogeneous AMP flow in one of our existing SMP machine models (eg on a dual core A9 with one core being master and the other slave)?
Quoted 3 lines
Can this be architected in two phases for greater utility, with the AMP devices as just normal devices, and the inter-qemu communication as a separate feature?
I see your point, and it is an interesting proposal.
What I can think here to remove the awareness of how the IDM
communicates
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with
the slaves, is to define a kind of AMP Slave interface. So there will
be an
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
instance of the interface for each of the slaves, encapsulating the communication part (being either local or based on sockets). The AMP Slave interfaces would be what you called the AMP devices,
with one
Quoted 18 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
device per slave.
Do we need this hard definition of master and slave in the hardware? Can the virtio-device be more peer-peer and the master-slave relationship is purely implemented by the guest?
Quoted 10 lines
I think we can architect it in a way that the virtio-idm simply connects two or more peers, and depending from the usage done by the software, behaving as master from one side and slave on the other. I used the term slave AMP interface, I should have used AMP client interface, to indicate the cores/procesors the IDM has inter-connect (being local or on another QEMU instance).
So there would be an implementation of the AMP client interface that is based on the assumption that all the processors are on the same instance, and one based on sockets for the remote instances.
Quoted 1 line
Do you need this dual mode? Can the IDM just have GPIOs which are then
either directly connected to the local CPUs, or sent out over inter-qemu connectivity mechanism? Then the inter-qemu can be used for any GPIO communication.
Quoted 1 line
Yes, at a first glance it could be done as you propose. The only doubt I have is related to the boot triggering feature, that is used by master OSes and rely on the IDM, and might need to have a specific interface when slave is remote (so the dual mode).
I guess the best is now to continue working on the code to produce a new patchset, taking your suggestions into account when designing the new interface in order to abstract the inter-QEMU communication part as much as possible from the IDM code.
Thanks,
Christian
Quoted 16 lines
Quoted 16 lines
to make an example, for a single qemu instance with -smp 2 you would add something like :
(quoted diff) +0
-4-smp 2 -device amp-local-client, core_id=0, id=client0 -device amp-local-client, core_id=1, id=client1 -device virtio-idm, clients=2, id=idmwhile for remote qemu instances something like
(the opposite to be instantiated on the other remote instance):(quoted diff) +0
-3-device amp-local-client, id=client0 -device amp-remote-client, chardev=chdev_id, id=client1 -device virtio-idm, clients=2, id=idm-dev
Quoted 8 lines
Quoted 8 lines
This way the idm only knows about clients (all clients are the same for the IDM). The software running on the processors will enable the interaction between the clients by writing into the IDM device registers.
At a first glance, and according to my current proposal, I see such AMP client interfaces exporting the following methods:
raise_interrupt()function: called by the IDM to trigger an interrupt
towards the destination client
Quoted 1 line
Quoted 1 line
boot_trigger()function: called by the IDM to trigger the boot of the
client
Quoted 2 lines
Quoted 2 lines
If the clients are remote, socket communication will be used and hidden
in the AMP client interface implementation
Quoted 10 lines
Quoted 8 lines
Do you foresee a different type of interface for the use-case you have in mind? I ask because if for example the clients are cores of the same cluster (and same instance), interrupts could simply be software generated from the linux-kernel/firmware running on top of the processors and theoretically no need to go through the IDM, same I guess for the boot.
True. But if you are developing code for IDM, you can do a
crawl-before-walk test with an SMP test case.
Quoted 22 lines
Regards,
PeterQuoted 19 lines
Another thing that needs to be defined clearly is the interface between the IDM and the software running on the cores.
At the moment I am using a set of registers, namely the boot and the interrupt registers. By writing the ID of a client in such registers it is possible to forward an interrupt or trigger its boot.Thanks,
Christian
Quoted 5 lines
Regards,
PeterQuoted 2 lines
At master side, besides the IDM, one would instantiate
as many interface devices as slaves. During the initialization the IDM
will
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
link
with all those interfaces, and only call functions like:
send_interrupt() or
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
boot_slave()to interact with the slaves. The interface will be the
same for
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
both local or remote slaves, while the implementation of the methods
will
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
differ and reside in the specific AMP Slave Interface device. On the slave side, if the slave is remote, another instance of the interface is instantiated so to connect to socket/eventfd.
So as an example the
send_shmem_fdfunction you pointed could be
hidden in
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
slave interface, and invoked only when the IDM will invoke the
slave_boot()
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
function of a remote slave interface.
This would higher the level of abstraction and open the door to
potentially
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
any
communication mechanism between master and slave, without the need to
adapt
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the
IDM device to the specific case. Or, eventually, to mix between local
and
Quoted 12 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
remote instances.
Thanks,
Christian
Quoted 2 lines
Regards,
Peter