Loading new machines and devices from external modules
PATCH v2 1/1 Loading new machines and devices from external modules
From: "Drap, Anton" <[email protected]>
There is no mechanism to load external machines and classes from modules at the moment. This patch is to add two parameters add_machine and add_modinfo for it.add_machine is to add machines from external modules. add_modinfo is to add devices from external modules, needed for a new machine, for example.
Main aim is to have possibility to develop independent models and be able to use it with mainline QEMU. It will help to make develop new models of proprietary boards, simplify to use Qemu by hardware developers and extend number of supporting boards and devices in QEMU. It will be easier for small hardware manufacturers to use QEMU to develop their own board models and use them to shift left of FW/SW development.
---
v2:
- subject is fixed
--- include/qapi/qmp/qjson.h | 2 + include/qemu/module.h | 4 +- qemu-options.hx | 38 ++++++ qobject/qjson.c | 14 ++ scripts/modinfo-generate.py | 7 +- softmmu/vl.c | 14 ++ util/module.c | 249 +++++++++++++++++++++++++++++++++++- 7 files changed, 320 insertions(+), 8 deletions(-)
include/qapi/qmp/qjson.h +2 -0
@@ -25,6 +25,8 @@ QDict *qdict_from_vjsonf_nofail(const char *string, va_list ap)QDict *qdict_from_jsonf_nofail (const char *string, ...)G_GNUC_PRINTF (1 ,2 ); +QDict *qdict_from_json_nofail_nofmt (const char *string); +GString *qobject_to_json (const QObject *obj);GString *qobject_to_json_pretty (const QObject *obj,bool pretty);
include/qemu/module.h +3 -1
@@ -73,6 +73,8 @@ bool module_load_one(const char *prefix, const char *lib_name, bool mayfail);void module_load_qom_one (const char *type);void module_load_qom_all (void );void module_allow_arch (const char *arch); +bool load_external_modules (const char *mods_list); +bool add_modinfo (const char *filename);/** * DOC: module info annotation macros
@@ -154,7 +156,7 @@ void module_allow_arch(const char *arch);typedef struct QemuModinfo QemuModinfo ;struct QemuModinfo {const char *name; -const char *arch; +const char **arch;const char **objs;const char **deps;const char **opts;
qemu-options.hx +38 -0
@@ -5476,6 +5476,44 @@ SRST (qemu) qom-set /objects/iothread1 poll-max-ns 100000 ERST +DEF("add_machines", HAS_ARG, QEMU_OPTION_add_machines, \ + "-add_machines libname[,...]\n", + QEMU_ARCH_ALL) +SRST +``-add_machines libname[,...]`` + Add machines from external modules. + For example: + + :: + + -add_machines custom-arm-machine,custom-arm-machine2 +ERST + +DEF("add_modinfo", HAS_ARG, QEMU_OPTION_add_modinfo, \ + "-add_modinfo filename\n", + QEMU_ARCH_ALL) +SRST +``-add_modinfo filename`` + Extend modinfo from file. Used to add devices from external modules. + Modinfo extention file is a JSON file with dictionary of modules: + { + "short name of module": {"name": "module-name", + "arch": ["supported_arch_1", "supported_arch_2",], + "objs": ["object1_description", "object2_description",], + "deps": ["depend_of_module_name1", "depend_of_module_name2",], + "opts": ["option1", "option2",] + } + } + + Architectures should be designated as they are printed by ./configure --help in target list + without "-softmmu" or "-linux-user" suffixes. e.g.: "arm", "x86_64", "riscv32", etc. + + For example: + + :: + + -add_modinfo modinfo_extention.json +ERST HXCOMM This is the last statement. Insert new options before this line!
qobject/qjson.c +14 -0
@@ -148,6 +148,20 @@ QDict *qdict_from_jsonf_nofail(const char *string, ...)return qdict; } +/* +* Parse @string as JSON object without %-escapes interpolated. +* Abort on error. Do not use with untrusted @string. +* Return the resulting QDict. It is never null. +*/ +QDict *qdict_from_json_nofail_nofmt (const char *string) +{ +QDict *qdict; + qdict =qobject_to (QDict,qobject_from_json (string, &error_abort)); +assert (qdict); +return qdict; +} + +static void to_json (JSONWriter *writer,const char *name,const QObject *obj) {
scripts/modinfo-generate.py +3 -4
@@ -33,7 +33,7 @@ def parse_line(line): return (kind, data) def generate(name, lines, enabled): - arch = "" + arch = [] objs = [] deps = [] opts = []
@@ -47,7 +47,7 @@ def generate(name, lines, enabled): elif kind == 'opts': opts.append(data) elif kind == 'arch': - arch = data; + arch.append(data) elif kind == 'kconfig': # don't add a module which dependency is not enabled # in kconfig
@@ -61,8 +61,7 @@ def generate(name, lines, enabled): exit(1) print(" .name = \"%s\"," % name) - if arch != "": - print(" .arch = %s," % arch) + print_array("arch", arch) print_array("objs", objs) print_array("deps", deps) print_array("opts", opts)
softmmu/vl.c +14 -0
@@ -3465,6 +3465,20 @@ void qemu_init(int argc, char **argv, char **envp)case QEMU_OPTION_enable_sync_profile:qsp_enable ();break ; +case QEMU_OPTION_add_machines: +info_report ("External machines loading: %s" , optarg); +if (!load_external_modules (optarg)) { +error_report ("Modules loading error. Modules %s" , optarg); +exit (1 ); + } +break ; +case QEMU_OPTION_add_modinfo: +info_report ("Modinfo parsing: %s" , optarg); +if (!add_modinfo (optarg)) { +error_report ("Modinfo (%s) adding error" , optarg); +exit (1 ); + } +break ;case QEMU_OPTION_nouserconfig:/* Nothing to be parsed here. Especially, do not error out below. */ break ;
util/module.c +246 -3
285 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 11 lines · Drap Anton
From: "Drap, Anton" <[email protected]>
There is no mechanism to load external machines and classes from modules at the moment. This patch is to add two parameters
add_machineandadd_modinfofor it.add_machineis to add machines from external modules.add_modinfois to add devices from external modules, needed for a new machine, for example.
Main aim is to have possibility to develop independent models and be able to use it with mainline QEMU.
This is deliberate, as a policy decision. We don't really want to encourage people to develop device and machine models externally -- we'd rather they contributed them upstream. Code to support out-of-tree devices and boards is pure burden for upstream, with no benefit to us.
thanks
-- PMM
Quoted 14 lines · Drap Anton
From: "Drap, Anton" <[email protected]>
There is no mechanism to load external machines and classes from modules at the moment. This patch is to add two parameters
add_machineandadd_modinfofor it.add_machineis to add machines from external modules.add_modinfois to add devices from external modules, needed for a new machine, for example.
Main aim is to have possibility to develop independent models and be able to use it with mainline QEMU. It will help to make develop new models of proprietary boards, simplify to use Qemu by hardware developers and extend number of supporting boards and devices in QEMU. It will be easier for small hardware manufacturers to use QEMU to develop their own board models and use them to shift left of FW/SW development.
IIUC, this is suggesting QEMU load pre-built .so files created from non-upstream code, to arbitrarily extend QEMU's functionality. Such .so files will inherantly have to be GPLd as they'll derive from QEMU's internal APIs which are GPL. Given the proposed use case is to emulate non-released proprietary hardware, I struggle to see how you'll fullfill the requirements for GPL licensing of the loaded .so, without revealing your proprietary hardware design to any who receive the .so files.
More generally, QEMU's existing loadable module usage is explicitly designed to try to *prevent* loading of non-upstream code. It aims to only load code that was built as part of the integrated QEMU build process. ie, QEMU's loadable module system is about making it possible to build many QEMU features, but then selectively load them at runtime to reduce footprint/attack surface. It is *not* intended to allow non-upstream code to be loaded.
Aside from our goal to prevent/discourage GPL violation through closed source loadable modules, QEMU also has a strong desire to not lock ourselves into supporting a public API for loadable modules. Maintainers wish to retain flexibility to change the internal APIs at any time.
Partially related to this topic, there is some work taking place with the goal of making it possible to define new machine types in QEMU from a QAPI based JSON description. The actual hardware devices and CPUs would still need code to be built into QEMU and upstream, but the way the hardware devices & CPUs are wired together would be customizable via the JSON config. That could get some, but not all, of the benefits you seek without the downsides the QEMU maintainers wish to avoid. This isn't ready to consume yet and we don't have any firm ETA either I'm afraid.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi Guys,
Let me clarify my position about out of tree devices. Yes, I understand that current QEMU politics is to have all the supported platforms inside QEMU source tree, but actually simulator core development, development of the devices standard library and development of virtual platforms are three different tasks. Moreover different people interested in different parts of QEMU. QEMU core developers not interested in supporting and maintaining tons of platforms available on the market. Virtual platform developers not interested and usually don’t have resources to merge their changes upstream. So we have a lots of abandoned QEMU forks for different platforms. For example we’re now working on Raspberry Pi 4b implementation for our internal needs and we’re planning to merge it upstream. It’s based on some QEMU fork author of which wasn’t able to complete it and commit upstream. And it can’t be used with later QEMU without some efforts to port it to newer QEMU version. Nobody supports and maintaining it since constant efforts necessary to be in sync with QEMU mainline. So my opinion is that core development, core device library and virtual platform development should be divided to make life easier for everybody. And this changes is first step to it.
About legal reasons and GPL violations. Possibility to make .so with machine separately and load it without providing sources is a legal risk and can’t be completely solved with technical actions. Ban on external modules just makes it more difficult for everybody to use not upstream code (including GPL violators, but not only for them) and doesn’t block ability to distribute full QEMU fork with closed models without providing sources. So I don’t see any reason to make technical limitations which actually can’t solve legal problem.
Best Regards,
Anton
Software engineer from Auriga LLC<http://www.auriga.com/>
________________________________
От: Daniel P. Berrangé <[email protected]>
Отправлено: 19 июля 2022 г. 19:25
Кому: Drap Anton
Копия: [email protected]; Drap, Anton
Тема: Re: [PATCH v2] Loading new machines and devices from external modules
Quoted 14 lines · Drap Anton
From: "Drap, Anton" <[email protected]>
There is no mechanism to load external machines and classes from modules at the moment. This patch is to add two parameters
add_machineandadd_modinfofor it.add_machineis to add machines from external modules.add_modinfois to add devices from external modules, needed for a new machine, for example.
Main aim is to have possibility to develop independent models and be able to use it with mainline QEMU. It will help to make develop new models of proprietary boards, simplify to use Qemu by hardware developers and extend number of supporting boards and devices in QEMU. It will be easier for small hardware manufacturers to use QEMU to develop their own board models and use them to shift left of FW/SW development.
IIUC, this is suggesting QEMU load pre-built .so files created from non-upstream code, to arbitrarily extend QEMU's functionality. Such .so files will inherantly have to be GPLd as they'll derive from QEMU's internal APIs which are GPL. Given the proposed use case is to emulate non-released proprietary hardware, I struggle to see how you'll fullfill the requirements for GPL licensing of the loaded .so, without revealing your proprietary hardware design to any who receive the .so files.
More generally, QEMU's existing loadable module usage is explicitly designed to try to *prevent* loading of non-upstream code. It aims to only load code that was built as part of the integrated QEMU build process. ie, QEMU's loadable module system is about making it possible to build many QEMU features, but then selectively load them at runtime to reduce footprint/attack surface. It is *not* intended to allow non-upstream code to be loaded.
Aside from our goal to prevent/discourage GPL violation through closed source loadable modules, QEMU also has a strong desire to not lock ourselves into supporting a public API for loadable modules. Maintainers wish to retain flexibility to change the internal APIs at any time.
Partially related to this topic, there is some work taking place with the goal of making it possible to define new machine types in QEMU from a QAPI based JSON description. The actual hardware devices and CPUs would still need code to be built into QEMU and upstream, but the way the hardware devices & CPUs are wired together would be customizable via the JSON config. That could get some, but not all, of the benefits you seek without the downsides the QEMU maintainers wish to avoid. This isn't ready to consume yet and we don't have any firm ETA either I'm afraid.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :|
libvirt: The virtualization API<https://libvirt.org/> libvirt.org
libvirt, virtualization, virtualization API
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|