xen: add hypfs per-domain abi-features
cover xen: add hypfs per-domain abi-features
This small series is meant as an example how to add further dynamical directories to hypfs. It can be used to replace Paul's current approach to specify ABI-features via domain create flags and replace those by hypfs nodes.
The related libxl part could just use libxenhypfs to set the values.
This series is meant to be applied on top of my series:
xen: support per-cpupool scheduling granularity
Juergen Gross (3):
xen/hypfs: add support for bool leafs in dynamic directories xen/domain: add domain hypfs directories xen/domain: add per-domain hypfs directory abi-features
docs/misc/hypfs-paths.pandoc | 10 ++ xen/common/Makefile | 1 + xen/common/hypfs.c | 53 +++++++++-- xen/common/hypfs_dom.c | 176 +++++++++++++++++++++++++++++++++++ xen/include/xen/hypfs.h | 7 ++ xen/include/xen/sched.h | 4 + 6 files changed, 244 insertions(+), 7 deletions(-) create mode 100644 xen/common/hypfs_dom.c
Signature · 1 line
2.26.2
RFC PATCH 1/3 xen/hypfs: add support for bool leafs in dynamic directories
Add support for reading fixed sized leafs and writing bool leafs in dynamic directories with the backing variable being a member of the structure anchored in struct hypfs_dyndir->data.
This adds the related leaf read and write functions and a helper macro HYPFS_STRUCT_ELEM() for referencing the element.
--- xen/common/hypfs.c | 53 +++++++++++++++++++++++++++++++++++------ xen/include/xen/hypfs.h | 7 ++++++ 2 files changed, 53 insertions(+), 7 deletions(-)
xen/common/hypfs.c +46 -7
@@ -501,17 +501,26 @@ int hypfs_read_dir(const struct hypfs_entry *entry,return 0 ; } -int hypfs_read_leaf (const struct hypfs_entry *entry, -XEN_GUEST_HANDLE_PARAM (void ) uaddr) +static int hypfs_read_leaf_off (const struct hypfs_entry *entry, +XEN_GUEST_HANDLE_PARAM (void ) uaddr, +void *off) {const struct hypfs_entry_leaf *l;unsigned int size = entry->funcs->getsize (entry); +const void *ptr;ASSERT (this_cpu (hypfs_locked) != hypfs_unlocked); l = container_of(entry, const struct hypfs_entry_leaf, e); + ptr = off ? off + (unsigned long )l->u.content : l->u.content; + +return copy_to_guest (uaddr, ptr, size) ? -EFAULT :0 ; +} -return copy_to_guest (uaddr, l->u.content, size) ? -EFAULT :0 ; +int hypfs_read_leaf (const struct hypfs_entry *entry, +XEN_GUEST_HANDLE_PARAM (void ) uaddr) +{ +return hypfs_read_leaf_off (entry, uaddr,NULL ); } static int hypfs_read(const struct hypfs_entry *entry,
@@ -587,11 +596,12 @@ int hypfs_write_leaf(struct hypfs_entry_leaf *leaf,return ret; } -int hypfs_write_bool (struct hypfs_entry_leaf *leaf, -XEN_GUEST_HANDLE_PARAM (const_void) uaddr, -unsigned int ulen) +static int hypfs_write_bool_off (struct hypfs_entry_leaf *leaf, +XEN_GUEST_HANDLE_PARAM (const_void) uaddr, +unsigned int ulen,void *off) {bool buf; +bool *ptr;ASSERT (this_cpu (hypfs_locked) == hypfs_write_locked); ASSERT(leaf->e.type == XEN_HYPFS_TYPE_BOOL &&
@@ -604,11 +614,19 @@ int hypfs_write_bool(struct hypfs_entry_leaf *leaf,if (copy_from_guest (&buf, uaddr, ulen) )return -EFAULT ; - *(bool *)leaf->u.write_ptr = buf; + ptr = off ? off + (unsigned long )leaf->u.write_ptr : leaf->u.write_ptr; + *ptr = buf;return 0 ; } +int hypfs_write_bool (struct hypfs_entry_leaf *leaf, +XEN_GUEST_HANDLE_PARAM (const_void) uaddr, +unsigned int ulen) +{ +return hypfs_write_bool_off (leaf, uaddr, ulen,NULL ); +} +int hypfs_write_custom (struct hypfs_entry_leaf *leaf,XEN_GUEST_HANDLE_PARAM (const_void) uaddr,unsigned int ulen)
@@ -644,6 +662,27 @@ int hypfs_write_custom(struct hypfs_entry_leaf *leaf,return ret; } +int hypfs_dyndir_id_read_leaf (const struct hypfs_entry *entry, +XEN_GUEST_HANDLE_PARAM (void ) uaddr) +{ +struct hypfs_dyndir_id *dyndata; + + dyndata =hypfs_get_dyndata (); + +return hypfs_read_leaf_off (entry, uaddr, dyndata->data); +} + +int hypfs_dyndir_id_write_bool (struct hypfs_entry_leaf *leaf, +XEN_GUEST_HANDLE_PARAM (const_void) uaddr, +unsigned int ulen) +{ +struct hypfs_dyndir_id *dyndata; + + dyndata =hypfs_get_dyndata (); + +return hypfs_write_bool_off (leaf, uaddr, ulen, dyndata->data); +} +int hypfs_write_deny (struct hypfs_entry_leaf *leaf,XEN_GUEST_HANDLE_PARAM (const_void) uaddr,unsigned int ulen)
xen/include/xen/hypfs.h +7 -0
@@ -160,6 +160,8 @@ static inline void hypfs_string_set_reference(struct hypfs_entry_leaf *leaf,HYPFS_FIXEDSIZE_INIT (var,XEN_HYPFS_TYPE_BOOL , nam, contvar, \ &hypfs_bool_wr_funcs, 1) +#define HYPFS_STRUCT_ELEM (type, elem) (((type *)NULL)->elem) +extern struct hypfs_entry_dir hypfs_root; int hypfs_add_dir(struct hypfs_entry_dir *parent,
@@ -204,6 +206,11 @@ struct hypfs_entry *hypfs_gen_dyndir_id_entry( const struct hypfs_entry_dir *template, unsigned int id, void *data);unsigned int hypfs_dynid_entry_size (const struct hypfs_entry *template,unsigned int id); +int hypfs_dyndir_id_read_leaf (const struct hypfs_entry *entry, +XEN_GUEST_HANDLE_PARAM (void ) uaddr); +int hypfs_dyndir_id_write_bool (struct hypfs_entry_leaf *leaf, +XEN_GUEST_HANDLE_PARAM (const_void) uaddr, +unsigned int ulen);#endif #endif /* __XEN_HYPFS_H__ */
Signature · 1 line
2.26.2
Re: [PATCH RFC 1/3] xen/hypfs: add support for bool leafs in dynamic directories
Quoted 10 lines · Juergen Gross
xen/common/hypfs.c +2
-2@@ -501,17 +501,26 @@ int hypfs_read_dir(const struct hypfs_entry *entry,return 0 ; } -int hypfs_read_leaf (const struct hypfs_entry *entry, -XEN_GUEST_HANDLE_PARAM (void ) uaddr) +static int hypfs_read_leaf_off(const struct hypfs_entry *entry, + XEN_GUEST_HANDLE_PARAM(void) uaddr,
xen/common/hypfs.c int hypfs_read_dir(const struct hypfs_entry *entry, · line 515
Quoted 10 lines
xen/common/hypfs.c +3
-0+ void *off) {const struct hypfs_entry_leaf *l;unsigned int size = entry->funcs->getsize (entry); +const void *ptr;ASSERT (this_cpu (hypfs_locked) != hypfs_unlocked); l = container_of(entry, const struct hypfs_entry_leaf, e); + ptr = off ? off + (unsigned long )l->u.content : l->u.content;
This is very irritating - you effectively add together two pointers. And even if this was correct for some reason, it would seem better readable as
ptr = l->u.content + (unsigned long)off;
to me.
xen/include/xen/hypfs.h static inline void hypfs_string_set_reference(struct hypfs_entry_leaf *leaf, · line 163
Quoted 7 lines
xen/include/xen/hypfs.h +1
-0@@ -160,6 +160,8 @@ static inline void hypfs_string_set_reference(struct hypfs_entry_leaf *leaf,HYPFS_FIXEDSIZE_INIT (var,XEN_HYPFS_TYPE_BOOL , nam, contvar, \ &hypfs_bool_wr_funcs, 1) +#define HYPFS_STRUCT_ELEM (type, elem) (((type *)NULL)->elem)
Kind of similar here - this very much looks like a NULL deref, avoidable by a user only if it takes the address of the construct. If there's really some non-pointer value to be encoded here, it would be better to avoid misuse by making the construct safe in a self-contained way.
Jan
RFC PATCH 2/3 xen/domain: add domain hypfs directories
In reply to Juergen GrossAdd /domain/<domid> directories to hypfs. Those are completely dynamic, so the related hypfs access functions need to be implemented.
---
V3:
- new patch
--- docs/misc/hypfs-paths.pandoc | 10 +++ xen/common/Makefile | 1 + xen/common/hypfs_dom.c | 137 +++++++++++++++++++++++++++++++++++ 3 files changed, 148 insertions(+) create mode 100644 xen/common/hypfs_dom.c
docs/misc/hypfs-paths.pandoc +10 -0
@@ -34,6 +34,7 @@ not containing any '/' character. The names "." and ".." are reserved for file system internal use. VALUES are strings and can take the following forms (note that this represents +>>>>>>> patched only the syntax used in this document): * STRING -- an arbitrary 0-delimited byte string.
@@ -191,6 +192,15 @@ The scheduling granularity of a cpupool. Writing a value is allowed only for cpupools with no cpu assigned and if the architecture is supporting different scheduling granularities. +#### /domain/ + +A directory of all current domains. + +#### /domain/*/ + +The individual domains. Each entry is a directory with the name being the +domain-id (e.g. /domain/0/). + #### /params/ A directory of runtime parameters.
xen/common/Makefile +1 -0
@@ -15,6 +15,7 @@ obj-$(CONFIG_GRANT_TABLE) += grant_table.o obj-y += guestcopy.o obj-bin-y += gunzip.init.o obj-$(CONFIG_HYPFS) += hypfs.o +obj-$(CONFIG_HYPFS) += hypfs_dom.o obj-y += irq.o obj-y += kernel.o obj-y += keyhandler.o
xen/common/hypfs_dom.c new file +137 -0
137 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH RFC 2/3] xen/domain: add domain hypfs directories
Hi Juergen,
Quoted 19 lines · Juergen Gross
Add /domain/<domid> directories to hypfs. Those are completely dynamic, so the related hypfs access functions need to be implemented.
Signed-off-by Juergen Gross <[email protected]>---
V3:
- new patch--- docs/misc/hypfs-paths.pandoc | 10 +++ xen/common/Makefile | 1 + xen/common/hypfs_dom.c | 137 +++++++++++++++++++++++++++++++++++ 3 files changed, 148 insertions(+)create mode 100644
xen/common/hypfs_dom.cdocs/misc/hypfs-paths.pandoc +0
-0
docs/misc/hypfs-paths.pandoc not containing any '/' character. The names "." and ".." are reserved · line 37
Quoted 4 lines
docs/misc/hypfs-paths.pandoc +1
-0@@ -34,6 +34,7 @@ not containing any '/' character. The names "." and ".." are reserved for file system internal use. VALUES are strings and can take the following forms (note that this represents +>>>>>>> patched
This seems to be a left-over of a merge.
Quoted 7 lines
docs/misc/hypfs-paths.pandoc +0
-0only the syntax used in this document): * STRING -- an arbitrary 0-delimited byte string.@@ -191,6 +192,15 @@ The scheduling granularity of a cpupool. Writing a value is allowed only for cpupools with no cpu assigned and if the architecture is supporting different scheduling granularities.
[...]
docs/misc/hypfs-paths.pandoc
Quoted 8 lines
docs/misc/hypfs-paths.pandoc +8
-0+ +static int domain_dir_read(const struct hypfs_entry *entry, + XEN_GUEST_HANDLE_PARAM(void) uaddr) +{ + int ret = 0; + const struct domain *d; + + for_each_domain ( d )
This is definitely going to be an issue if you have a lot of domain running as Xen is not preemptible.
I think the first step is to make sure that HYPFS can scale without hogging a pCPU for a long time.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH RFC 2/3] xen/domain: add domain hypfs directories
Quoted 17 lines · Julien Grall
Hi Juergen,
Quoted 14 lines · Juergen Gross
Add /domain/<domid> directories to hypfs. Those are completely dynamic, so the related hypfs access functions need to be implemented.
Signed-off-by Juergen Gross <[email protected]>---
V3:
- new patch
---
docs/misc/hypfs-paths.pandoc | 10 +++
xen/common/Makefile | 1 + xen/common/hypfs_dom.c | 137 +++++++++++++++++++++++++++++++++++3 files changed, 148 insertions(+)create mode 100644
xen/common/hypfs_dom.c
Quoted 11 lines
Quoted 9 lines
docs/misc/hypfs-paths.pandoc +0
-0@@ -34,6 +34,7 @@ not containing any '/' character. The names "." and".." are reserved
for file system internal use.
VALUES are strings and can take the following forms (note that this represents +>>>>>>> patchedThis seems to be a left-over of a merge.
Oh, interesting that I wasn't warned about that.
Quoted 9 lines
Quoted 6 lines
only the syntax used in this document):
* STRING -- an arbitrary 0-delimited byte string.docs/misc/hypfs-paths.pandoc +0
-0@@ -191,6 +192,15 @@ The scheduling granularity of a cpupool.Writing a value is allowed only for cpupools with no cpu assigned and if the architecture is supporting different scheduling granularities.
[...]
Quoted 10 lines
docs/misc/hypfs-paths.pandoc
Quoted 8 lines
docs/misc/hypfs-paths.pandoc +8
-0+ +static int domain_dir_read(const struct hypfs_entry *entry, + XEN_GUEST_HANDLE_PARAM(void) uaddr) +{ + int ret = 0; + const struct domain *d; + + for_each_domain ( d )This is definitely going to be an issue if you have a lot of domain running as Xen is not preemptible.
In general this is correct, but in this case I don't think it will be a problem. The execution time for each loop iteration should be rather short (in the microsecond range?), so even with 32000 guests we would stay way below one second. And on rather slow cpus I don't think we'd have thousands of guests anyway.
Quoted 1 line
I think the first step is to make sure that HYPFS can scale without hogging a pCPU for a long time.
I agree this would be best.
Juergen
Re: [PATCH RFC 2/3] xen/domain: add domain hypfs directories
Quoted 20 lines · Jürgen Groß
Quoted 8 lines · Julien Grall
Quoted 5 lines
only the syntax used in this document):
* STRING -- an arbitrary 0-delimited byte string.(quoted diff) +0
-0@@ -191,6 +192,15 @@ The scheduling granularity of a cpupool.Writing a value is allowed only for cpupools with no cpu assigned and if the architecture is supporting different scheduling granularities.
[...]
Quoted 10 lines
xen/common/hypfs_dom.c line 55
Quoted 8 lines
xen/common/hypfs_dom.c new file +8
-0+ +static int domain_dir_read (const struct hypfs_entry *entry, + XEN_GUEST_HANDLE_PARAM(void) uaddr) +{ + int ret = 0; + const struct domain *d; + + for_each_domain ( d )This is definitely going to be an issue if you have a lot of domain running as Xen is not preemptible.
Quoted 4 lines
In general this is correct, but in this case I don't think it will be a problem. The execution time for each loop iteration should be rather short (in the microsecond range?), so even with 32000 guests we would stay way below one second.
The scheduling slice are usually in ms and not second (yet this will depend on your scheduler). It would be unacceptable to me if another vCPU cannot run for a second because dom0 is trying to list the domain via HYPFS.
Cheers,
Signature · 1 line
Julien Grall
Re: [PATCH RFC 2/3] xen/domain: add domain hypfs directories
Quoted 23 lines · Julien Grall
Quoted 20 lines · Jürgen Groß
Quoted 18 lines · Julien Grall
Quoted 5 lines
only the syntax used in this document):
* STRING -- an arbitrary 0-delimited byte string.(quoted diff) +0
-0@@ -191,6 +192,15 @@ The scheduling granularity of a cpupool.Writing a value is allowed only for cpupools with no cpu assigned and if the architecture is supporting different scheduling granularities.
[...]
xen/common/hypfs_dom.c line 55
Quoted 8 lines
xen/common/hypfs_dom.c new file +8
-0+ +static int domain_dir_read (const struct hypfs_entry *entry, + XEN_GUEST_HANDLE_PARAM(void) uaddr) +{ + int ret = 0; + const struct domain *d; + + for_each_domain ( d )This is definitely going to be an issue if you have a lot of domain running as Xen is not preemptible.
Quoted 6 lines
Quoted 4 lines
In general this is correct, but in this case I don't think it will be a problem. The execution time for each loop iteration should be rather short (in the microsecond range?), so even with 32000 guests we would stay way below one second.
The scheduling slice are usually in ms and not second (yet this will depend on your scheduler). It would be unacceptable to me if another vCPU cannot run for a second because dom0 is trying to list the domain via HYPFS.
Okay, I did a test.
The worrying operation is the reading of /domain/ with lots of domains.
"xenhypfs ls /domain" with 500 domains running needed 231 us of real time for the library call, while "xenhypfs ls /" needed about 70 us. This makes 3 domains per usec, resulting in about 10 msecs with 30000 domains.
Juergen
RFC PATCH 3/3 xen/domain: add per-domain hypfs directory abi-features
In reply to Juergen GrossAdd a new per-domain hypfs directory "abi-features" used to control some feature availability. Changing the availability of a feature is allowed only before the first activation of the domain.
The related leafs right now are "event-channel-upcall" and "fifo-event-channels". For those bool elements are added to struct domain, but for now without any further handling.
--- xen/common/hypfs_dom.c | 39 +++++++++++++++++++++++++++++++++++++++ xen/include/xen/sched.h | 4 ++++ 2 files changed, 43 insertions(+)
xen/common/hypfs_dom.c +39 -0
@@ -10,6 +10,42 @@#include <xen/lib.h> #include <xen/sched.h> +static int domain_abifeat_write (struct hypfs_entry_leaf *leaf, +XEN_GUEST_HANDLE_PARAM (const_void) uaddr, +unsigned int ulen) +{ +struct hypfs_dyndir_id *data; +struct domain *d; + + data =hypfs_get_dyndata (); + d = data->data; + +if ( d->creation_finished ) +return -EBUSY ; + +return hypfs_dyndir_id_write_bool (leaf, uaddr, ulen); +} + +static const struct hypfs_funcs abifeat_funcs = { + .enter = hypfs_node_enter, + .exit = hypfs_node_exit, + .read = hypfs_dyndir_id_read_leaf, + .write = domain_abifeat_write, + .getsize = hypfs_getsize, + .findentry = hypfs_leaf_findentry, +}; + +static HYPFS_FIXEDSIZE_INIT(abifeat_evtupcall, XEN_HYPFS_TYPE_BOOL, +"event-channel-upcall" , + HYPFS_STRUCT_ELEM(struct domain, abi_evt_upcall), + &abifeat_funcs, 1); +static HYPFS_FIXEDSIZE_INIT(abifeat_fifoevnt, XEN_HYPFS_TYPE_BOOL, +"fifo-event-channels" , + HYPFS_STRUCT_ELEM(struct domain, abi_fifo_evt), + &abifeat_funcs, 1); + +static HYPFS_DIR_INIT(domain_abifeatdir, "abi-features"); +static const struct hypfs_entry *domain_domdir_enter (const struct hypfs_entry *entry) {
@@ -131,6 +167,9 @@ static int __init domhypfs_init(void) {hypfs_add_dir (&hypfs_root, &domain_dir, true);hypfs_add_dyndir (&domain_dir, &domain_domdir); +hypfs_add_dir (&domain_domdir, &domain_abifeatdir, true); +hypfs_add_leaf (&domain_abifeatdir, &abifeat_evtupcall, true); +hypfs_add_leaf (&domain_abifeatdir, &abifeat_fifoevnt, true);return 0 ; }
xen/include/xen/sched.h +4 -0
@@ -424,6 +424,10 @@ struct domain*/ bool creation_finished; +/* ABI-features (can be set via hypfs before first unpause).*/ +bool abi_fifo_evt; +bool abi_evt_upcall; +/* Which guest this guest has privileges on */ struct domain *target;
Signature · 1 line
2.26.2
Hi,
Quoted 4 lines · Juergen Gross
This small series is meant as an example how to add further dynamical directories to hypfs. It can be used to replace Paul's current approach to specify ABI-features via domain create flags and replace those by hypfs nodes.
This can only work if all the ABI-features are not required at the time of creating the domain.
Those features should also be set only once. Furthermore, HYPFS is so far meant to be optional.
So it feels to me Paul's approach is leaner and better for the ABI-features purpose.
Cheers,
Signature · 1 line
Julien Grall
Quoted 9 lines · Julien Grall
Hi,
Quoted 4 lines · Juergen Gross
This small series is meant as an example how to add further dynamical directories to hypfs. It can be used to replace Paul's current approach to specify ABI-features via domain create flags and replace those by hypfs nodes.
This can only work if all the ABI-features are not required at the time of creating the domain.
Yes. In case there is some further initialization needed this has to be done later depending on the setting.
Quoted 1 line
Those features should also be set only once. Furthermore, HYPFS is so far meant to be optional.
"set once" isn't the point. They should not be able to change after the domain has been started, and that is covered.
Quoted 2 lines
So it feels to me Paul's approach is leaner and better for the ABI-features purpose.
Depends.
My approach doesn't need any tools side changes after first implementation when adding new abi-features. And it isn't expanding an unstable interface.
In the end this is the reason I marked this series as RFC. If using flags is preferred, fine.
Juergen
Quoted 9 lines · Jürgen Groß
Quoted 8 lines · Julien Grall
Hi,
Quoted 4 lines · Juergen Gross
This small series is meant as an example how to add further dynamical directories to hypfs. It can be used to replace Paul's current approach to specify ABI-features via domain create flags and replace those by hypfs nodes.
Quoted 4 lines
Quoted 1 line
This can only work if all the ABI-features are not required at the time of creating the domain.
Yes. In case there is some further initialization needed this has to be done later depending on the setting.
We used to allocate vCPUs after the domain has been created. But this ended up in a can of worms because this requires a careful ordering of the hypercalls.
So I would rather like to avoid introducing similar operation again...
Quoted 5 lines
Quoted 2 lines
Those features should also be set only once. Furthermore, HYPFS is so far meant to be optional.
"set once" isn't the point. They should not be able to change after the domain has been started, and that is covered.
That really depends on the flag. Imagine there are dependencies between flags or memory needs to be allocated.
Quoted 4 lines
Quoted 3 lines
So it feels to me Paul's approach is leaner and better for the ABI-features purpose.
Quoted 8 lines
Depends.
My approach doesn't need any tools side changes after first implementation when adding new abi-features. And it isn't expanding an unstable interface.
In the end this is the reason I marked this series as RFC. If using flags is preferred, fine.
I prefer the flag version. Let see what other thinks.
Cheers,
---
Julien Grall
Quoted 4 lines · Juergen Gross
This small series is meant as an example how to add further dynamical directories to hypfs. It can be used to replace Paul's current approach to specify ABI-features via domain create flags and replace those by hypfs nodes.
Actually, before I look at further of the patches here: I seem to vaguely recall agreement was reached to stick to Paul's approach for the time being? Or am I misremembering?
Jan
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Juergen Gross
This small series is meant as an example how to add further dynamical directories to hypfs. It can be used to replace Paul's current approach to specify ABI-features via domain create flags and replace those by hypfs nodes.
Actually, before I look at further of the patches here: I seem to vaguely recall agreement was reached to stick to Paul's approach for the time being? Or am I misremembering?
This is my understanding, too.
Juergen