xenguest optimisations
Subsystems (3)
v6 cover xenguest optimisations
Reduce some number of parts passed to writev. Avoid possible allocation sending data with writev. Reduce number of allocations sending memory state.
Implement and use new Xen and Linux kernel ABI to copy foreign memory. This new ABI allows to replace theexpensive map/copy/unmap sequence with a single call.
Changes since v1:
- add commit to cache up to 4 pages in hypercall;
- add other 2 commits reducing chunks passed to write/writev.
Changes since v2:
- update patches commit prefixes;
- add other 2 optisations.
Changes since v3:
- address some comments;
- add patches for foreign copy optimisation.
Changes since v4:
- added Reviewed-by;
- improved commit messages;
- other minor fixes, see individual commits.
Changes since v5:
- avoids potential buffer underflow if nr_pages is 0 calling cache_alloc;
- do not overwrite errno if xenforeignmemory_map fails;
- lot of changes to "implement new foreign copy hypercall", see specific commit.
Edwin Török (3):
libs/guest: allocate various migration arrays just once libs/call: cache up to 4 pages in hypercall bounce buffers
PoC: libs/guest: use foreign copy during migration
Frediano Ziglio (12):
libs/guest: Reduce number of parts in write_split_record libs/guest: Reduce number of I/O vectors in write_batch libs/guest: Reduce number of I/O vectors in write_batch libs/guest: Use a single write_exact in write_headers libs/guest: avoids using 2 indexes
libs/guest: fill directly iov structure
libs/ctrl: Allows writev_exact to change iov array libs/guest: add xg_foreignmemory_copy_{from,to}
xen: implement new foreign copy hypercall
privcmd: Add definition for new Linux privcmd to access new Xen
hypercall
libs/guest: use new hypercall if available libs/guest: finalize PoC
tools/include/xen-sys/Linux/privcmd.h | 10 ++ tools/libs/call/buffer.c | 31 ++-- tools/libs/call/core.c | 3 +- tools/libs/call/private.h | 8 +- tools/libs/ctrl/xc_private.c | 26 +-- tools/libs/ctrl/xc_private.h | 2 +- tools/libs/guest/xg_sr_common.c | 92 ++++++++++- tools/libs/guest/xg_sr_common.h | 21 +++ tools/libs/guest/xg_sr_restore.c | 100 ++++++------ tools/libs/guest/xg_sr_save.c | 224 +++++++++++--------------- xen/arch/x86/traps-setup.c | 2 +- xen/common/memory.c | 145 +++++++++++++++++ xen/include/public/memory.h | 44 ++++- 13 files changed, 489 insertions(+), 219 deletions(-)
Signature · 1 line
2.43.0
PATCH v6 1/16 libs/guest: Reduce number of parts in write_split_record
Small optimization.
There's no much sense to split the header in 2 pieces, it will just take more time and space to reassemble them in the final buffer.
This also avoids truncating combined_length to 32 bit in case of 64 bit machines potentially avoiding following record_length check (it could still be truncated writing it in xc_sr_rhdr structure but the following check will catch it).
The function become more coherent with following read_record function.
--
Changes since v2:
- change prefix in subject.
Changes since v3:
- clarify commit message.
Changes since v4:
- added Reviewed-by;
- improved commit message.
--- tools/libs/guest/xg_sr_common.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/libs/guest/xg_sr_common.c +3 -3
@@ -59,11 +59,11 @@ int write_split_record(struct xc_sr_context *ctx, struct xc_sr_record *rec,static const char zeroes[REC_ALIGN ] = {};xc_interface *xch = ctx->xch; - typeof(rec->length) combined_length = rec->length + sz; +size_t combined_length = rec->length + sz;size_t record_length =ROUNDUP (combined_length,REC_ALIGN ); +struct xc_sr_rhdr rhdr = { rec->type, combined_length };struct iovec parts[] = { - { &rec->type,sizeof (rec->type) }, - { &combined_length,sizeof (combined_length) }, + { &rhdr,sizeof (rhdr) }, { rec->data, rec->length }, { buf, sz }, { (void *)zeroes, record_length - combined_length },
Signature · 1 line
2.43.0
Re: [PATCH v6 01/16] libs/guest: Reduce number of parts in write_split_record
Quoted 11 lines · Frediano Ziglio
Small optimization.
There's no much sense to split the header in 2 pieces, it will just take more time and space to reassemble them in the final buffer.
This also avoids truncatingcombined_lengthto 32 bit in case of 64 bit machines potentially avoiding followingrecord_lengthcheck (it could still be truncated writing it inxc_sr_rhdrstructure but the following check will catch it).
The function become more coherent with followingread_recordfunction.
Quoted 3 lines
Signed-off-by Frediano Ziglio <[email protected]>Reviewed-by Roger Pau Monné <[email protected]>--
You need to use 3 dashes here.
~Andrew
Re: [PATCH v6 01/16] libs/guest: Reduce number of parts in write_split_record
In reply to Frediano ZiglioQuoted 11 lines · Frediano Ziglio
Small optimization.
There's no much sense to split the header in 2 pieces, it will just take more time and space to reassemble them in the final buffer.
This also avoids truncatingcombined_lengthto 32 bit in case of 64 bit machines potentially avoiding followingrecord_lengthcheck (it could still be truncated writing it inxc_sr_rhdrstructure but the following check will catch it).
The function become more coherent with followingread_recordfunction.
Quoted 2 lines
Signed-off-by Frediano Ziglio <[email protected]>Reviewed-by Roger Pau Monné <[email protected]>
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
PATCH v6 2/16 libs/guest: Reduce number of I/O vectors in write_batch
In reply to Frediano ZiglioSmall optimization.
Reduce number of pieces passed to writev.
--
Changes since v2:
- change prefix in subject.
Changes since v4:
- added Reviewed-by.
--- tools/libs/guest/xg_sr_save.c | 34 +++++++++++++++------------------- 1 file changed, 15 insertions(+), 19 deletions(-)
tools/libs/guest/xg_sr_save.c +15 -19
@@ -97,9 +97,11 @@ static int write_batch(struct xc_sr_context *ctx)void *page, *orig_page;uint64_t *rec_pfns =NULL ;struct iovec *iov =NULL ;int iovcnt =0 ; -struct xc_sr_rec_page_data_header hdr = {0 }; -struct xc_sr_record rec = { - .type =REC_TYPE_PAGE_DATA , +struct { +struct xc_sr_rhdr rec; +struct xc_sr_rec_page_data_header page_data; + } hdrs = { + { .type =REC_TYPE_PAGE_DATA }, };assert (nr_pfns !=0 );
@@ -115,7 +117,7 @@ static int write_batch(struct xc_sr_context *ctx)/* Pointers to locally allocated pages. Need freeing. */ local_pages =calloc (nr_pfns,sizeof (*local_pages));/* iovec[] for writev(). */ - iov =malloc ((nr_pfns +4 ) *sizeof (*iov)); + iov =malloc ((nr_pfns +2 ) *sizeof (*iov));if ( !mfns || !types || !errors || !guest_data || !local_pages || !iov ) {
@@ -216,28 +218,22 @@ static int write_batch(struct xc_sr_context *ctx) goto err; } - hdr.count = nr_pfns; + hdrs.rec.length =sizeof (hdrs.page_data); + hdrs.rec.length += nr_pfns *sizeof (*rec_pfns); + hdrs.rec.length += nr_pages *PAGE_SIZE ; - rec.length =sizeof (hdr); - rec.length += nr_pfns *sizeof (*rec_pfns); - rec.length += nr_pages *PAGE_SIZE ; + hdrs.page_data.count = nr_pfns;for ( i =0 ; i < nr_pfns; ++i ) rec_pfns[i] = ((uint64_t )(types[i]) <<32 ) | ctx->save.batch_pfns[i]; - iov[0 ].iov_base = &rec.type; - iov[0 ].iov_len =sizeof (rec.type); + iov[0 ].iov_base = &hdrs; + iov[0 ].iov_len =sizeof (hdrs); - iov[1 ].iov_base = &rec.length; - iov[1 ].iov_len =sizeof (rec.length); + iov[1 ].iov_base = rec_pfns; + iov[1 ].iov_len = nr_pfns *sizeof (*rec_pfns); - iov[2 ].iov_base = &hdr; - iov[2 ].iov_len =sizeof (hdr); - - iov[3 ].iov_base = rec_pfns; - iov[3 ].iov_len = nr_pfns *sizeof (*rec_pfns); - - iovcnt =4 ; + iovcnt =2 ;if ( nr_pages ) {
Signature · 1 line
2.43.0
Re: [PATCH v6 02/16] libs/guest: Reduce number of I/O vectors in write_batch
Quoted 6 lines · Frediano Ziglio
tools/libs/guest/xg_sr_save.c +0
-0@@ -97,9 +97,11 @@ static int write_batch(struct xc_sr_context *ctx)void *page, *orig_page;
tools/libs/guest/xg_sr_save.c static int write_batch(struct xc_sr_context *ctx) · line 104
Quoted 10 lines
tools/libs/guest/xg_sr_save.c +5
-3uint64_t *rec_pfns =NULL ;struct iovec *iov =NULL ;int iovcnt =0 ; -struct xc_sr_rec_page_data_header hdr = {0 }; -struct xc_sr_record rec = { - .type =REC_TYPE_PAGE_DATA , +struct { +struct xc_sr_rhdr rec; +struct xc_sr_rec_page_data_header page_data; + } hdrs = { + { .type =REC_TYPE_PAGE_DATA },
.rec = { .type = ... },
Otherwise this is fragile to reordering.
~Andrew
Re: [PATCH v6 02/16] libs/guest: Reduce number of I/O vectors in write_batch
Quoted 21 lines · Andrew Cooper
Quoted 6 lines · Frediano Ziglio
tools/libs/guest/xg_sr_save.c +0
-0@@ -97,9 +97,11 @@ static int write_batch(struct xc_sr_context *ctx)void *page, *orig_page;tools/libs/guest/xg_sr_save.c static int write_batch(struct xc_sr_context *ctx) · line 104
Quoted 10 lines
tools/libs/guest/xg_sr_save.c +5
-3uint64_t *rec_pfns =NULL ;struct iovec *iov =NULL ;int iovcnt =0 ; -struct xc_sr_rec_page_data_header hdr = {0 }; -struct xc_sr_record rec = { - .type =REC_TYPE_PAGE_DATA , +struct { +struct xc_sr_rhdr rec; +struct xc_sr_rec_page_data_header page_data; + } hdrs = { + { .type =REC_TYPE_PAGE_DATA },.rec = { .type = ... },
Quoted 2 lines
Otherwise this is fragile to reordering.
Changed.
Here we are implementing a network protocol, reordering should not be done, unless you want to break the code.
Quoted 1 line
~Andrew
Frediano
PATCH v6 1/1 libs/guest: Allocate rec_pfns earlier in write_batch()
In reply to Frediano ZiglioFor reasons which escape me, rec_pfns are allocated separately to the rest of the batch allocations.
Allocate them all together. This will allow for future simplifications to be performed in an incremental mannor.
No functional change.
---
CC: Anthony PERARD <[email protected]>
CC: Frediano Ziglio <[email protected]>
--- tools/libs/guest/xg_sr_save.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
tools/libs/guest/xg_sr_save.c +4 -10
@@ -116,8 +116,10 @@ static int write_batch(struct xc_sr_context *ctx) local_pages =calloc (nr_pfns,sizeof (*local_pages));/* iovec[] for writev(). */ iov =malloc ((nr_pfns +4 ) *sizeof (*iov)); +/* page_data record PFNs list */ + rec_pfns =malloc (nr_pfns *sizeof (*rec_pfns)); -if ( !mfns || !types || !errors || !guest_data || !local_pages || !iov ) +if ( !mfns || !types || !errors || !guest_data || !local_pages || !iov || !rec_pfns ) {ERROR ("Unable to allocate arrays for a batch of %u pages" , nr_pfns);
@@ -208,14 +210,6 @@ static int write_batch(struct xc_sr_context *ctx) } } - rec_pfns =malloc (nr_pfns *sizeof (*rec_pfns)); -if ( !rec_pfns ) - { -ERROR ("Unable to allocate %zu bytes of memory for page data pfn list" , - nr_pfns *sizeof (*rec_pfns)); - goto err; - } - hdr.count = nr_pfns; rec.length =sizeof (hdr);
@@ -264,11 +258,11 @@ static int write_batch(struct xc_sr_context *ctx) rc = ctx->save.nr_batch_pfns =0 ; err: -free (rec_pfns);if ( guest_mapping )xenforeignmemory_unmap (xch->fmem, guest_mapping, nr_pages_mapped);for ( i =0 ; local_pages && i < nr_pfns; ++i )free (local_pages[i]); +free (rec_pfns);free (iov);free (local_pages);free (guest_data);
Signature · 1 line
2.39.5
Re: [PATCH v6 1.9/16] libs/guest: Allocate rec_pfns earlier in write_batch()
Quoted 9 lines · Andrew Cooper
For reasons which escape me,
rec_pfnsare allocated separately to the rest of the batch allocations.Allocate them all together. This will allow for future simplifications to be performed in an incremental mannor.
No functional change.
Signed-off-by Andrew Cooper <[email protected]>
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
PATCH 2/16 libs/guest: Reduce number of iovecs in write_batch()
In reply to Frediano ZiglioFrom: Frediano Ziglio <[email protected]>
Construct all of the headers together in one block, rather than a field at a time. Initialise as many of the fields as possible at declaration time.
Start filling in iov[] earlier, to allow for future simplifications.
No practical change.
---
CC: Anthony PERARD <[email protected]>
CC: Frediano Ziglio <[email protected]>
--- tools/libs/guest/xg_sr_save.c | 45 +++++++++++++++++------------------ 1 file changed, 22 insertions(+), 23 deletions(-)
tools/libs/guest/xg_sr_save.c +22 -23
@@ -97,9 +97,18 @@ static int write_batch(struct xc_sr_context *ctx)void *page, *orig_page;uint64_t *rec_pfns =NULL ;struct iovec *iov =NULL ;int iovcnt =0 ; -struct xc_sr_rec_page_data_header hdr = {0 }; -struct xc_sr_record rec = { - .type =REC_TYPE_PAGE_DATA , +struct { +struct xc_sr_rhdr rec; +struct xc_sr_rec_page_data_header page_data; + } hdrs = { + .rec = { + .type =REC_TYPE_PAGE_DATA , + .length = offsetof(struct xc_sr_rec_page_data_header , + pfn[nr_pfns]), /* + the pages to send */ + }, + .page_data = { + .count = nr_pfns, + }, };assert (nr_pfns !=0 );
@@ -115,7 +124,7 @@ static int write_batch(struct xc_sr_context *ctx)/* Pointers to locally allocated pages. Need freeing. */ local_pages =calloc (nr_pfns,sizeof (*local_pages));/* iovec[] for writev(). */ - iov =malloc ((nr_pfns +4 ) *sizeof (*iov)); + iov =malloc ((nr_pfns +2 ) *sizeof (*iov));/* page_data record PFNs list */ rec_pfns =malloc (nr_pfns *sizeof (*rec_pfns));
@@ -126,6 +135,14 @@ static int write_batch(struct xc_sr_context *ctx) goto err; } + iov[0 ].iov_base = &hdrs; + iov[0 ].iov_len =sizeof (hdrs); + + iov[1 ].iov_base = rec_pfns; + iov[1 ].iov_len = nr_pfns *sizeof (*rec_pfns); + + iovcnt =2 ; +for ( i =0 ; i < nr_pfns; ++i ) { types[i] = mfns[i] = ctx->save.ops.pfn_to_gfn(ctx,
@@ -210,29 +227,11 @@ static int write_batch(struct xc_sr_context *ctx) } } - hdr.count = nr_pfns; - - rec.length =sizeof (hdr); - rec.length += nr_pfns *sizeof (*rec_pfns); - rec.length += nr_pages *PAGE_SIZE ; + hdrs.rec.length += nr_pages *PAGE_SIZE ;for ( i =0 ; i < nr_pfns; ++i ) rec_pfns[i] = ((uint64_t )(types[i]) <<32 ) | ctx->save.batch_pfns[i]; - iov[0 ].iov_base = &rec.type; - iov[0 ].iov_len =sizeof (rec.type); - - iov[1 ].iov_base = &rec.length; - iov[1 ].iov_len =sizeof (rec.length); - - iov[2 ].iov_base = &hdr; - iov[2 ].iov_len =sizeof (hdr); - - iov[3 ].iov_base = rec_pfns; - iov[3 ].iov_len = nr_pfns *sizeof (*rec_pfns); - - iovcnt =4 ; -if ( nr_pages ) {for ( i =0 ; i < nr_pfns; ++i )
Signature · 1 line
2.39.5
Re: [PATCH v6.1 02/16] libs/guest: Reduce number of iovecs in write_batch()
Quoted 11 lines · Andrew Cooper
From: Frediano Ziglio <[email protected]>
Construct all of the headers together in one block, rather than a field at a time. Initialise as many of the fields as possible at declaration time.
Start filling in iov[] earlier, to allow for future simplifications.
No practical change.
Signed-off-by Frediano Ziglio <[email protected]>Signed-off-by Andrew Cooper <[email protected]>
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
PATCH v6 3/16 libs/guest: Reduce number of I/O vectors in write_batch
In reply to Frediano ZiglioEach page was sent using a different iovec item. This potentially exceed Linux maximum (1024).
Coalesce adjacent IO vector elements to attempt to reduce the number of overall IO vectors for each operation.
Also some implementation (MiniOS) emulate writev with multiple write calls.
--
Changes since v2:
- change prefix in subject.
Changes since v4:
- added Reviewed-by;
- improved commit message;
- minor style fix.
--- tools/libs/guest/xg_sr_save.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
tools/libs/guest/xg_sr_save.c +10 -2
@@ -239,13 +239,21 @@ static int write_batch(struct xc_sr_context *ctx) {for ( i =0 ; i < nr_pfns; ++i ) { -if ( guest_data[i] ) +if ( !guest_data[i] ) +continue ; + +if ( iov[iovcnt -1 ].iov_base + iov[iovcnt -1 ].iov_len != + guest_data[i] ) { iov[iovcnt].iov_base = guest_data[i]; iov[iovcnt].iov_len =PAGE_SIZE ; iovcnt++; - --nr_pages; } +else + { + iov[iovcnt -1 ].iov_len +=PAGE_SIZE ; + } + --nr_pages; } }
Signature · 1 line
2.43.0
Re: [PATCH v6 03/16] libs/guest: Reduce number of I/O vectors in write_batch
This has the same exact subject as the prior patch.
Either it wants merging, as they're both in the same function, or the subject wants to be different. Even a "Further ..." prefix would help.
Quoted 2 lines · Frediano Ziglio
Each page was sent using a different iovec item. This potentially exceed Linux maximum (1024).
Linux cannot have a maximum of 1024 because this has been working fine for a decade using 1028 in the common case.
Quoted 6 lines
Coalesce adjacent IO vector elements to attempt to reduce the number of overall IO vectors for each operation.
Also some implementation (MiniOS) emulate writev with multiple write calls.Signed-off-by Frediano Ziglio <[email protected]>Reviewed-by Roger Pau Monné <[email protected]>
Re: [PATCH v6 03/16] libs/guest: Reduce number of I/O vectors in write_batch
Quoted 7 lines · Andrew Cooper
This has the same exact subject as the prior patch.
Either it wants merging, as they're both in the same function, or the subject wants to be different. Even a "Further ..." prefix would help.
Quoted 7 lines
Quoted 2 lines · Frediano Ziglio
Each page was sent using a different iovec item. This potentially exceed Linux maximum (1024).
Linux cannot have a maximum of 1024 because this has been working fine for a decade using 1028 in the common case.
But the code does not call writev or similars directly, so there's no limit besides the sky.
The result with 1028 is simply that you do 2 system calls instead of one.
Quoted 7 lines
Quoted 6 lines
Coalesce adjacent IO vector elements to attempt to reduce the number of overall IO vectors for each operation.
Also some implementation (MiniOS) emulate writev with multiple write calls.Signed-off-by Frediano Ziglio <[email protected]>Reviewed-by Roger Pau Monné <[email protected]>
Frediano
Re: [PATCH v6 03/16] libs/guest: Reduce number of I/O vectors in write_batch
Quoted 12 lines · Frediano Ziglio
Quoted 11 lines · Andrew Cooper
This has the same exact subject as the prior patch.
Either it wants merging, as they're both in the same function, or the subject wants to be different. Even a "Further ..." prefix would help.
Quoted 2 lines · Frediano Ziglio
Each page was sent using a different iovec item. This potentially exceed Linux maximum (1024).
Quoted 7 lines
Quoted 3 lines
Linux cannot have a maximum of 1024 because this has been working fine for a decade using 1028 in the common case.
But the code does not call writev or similars directly, so there's no limit besides the sky.
The result with 1028 is simply that you do 2 system calls instead of one.
Could you add something along those lines to the description? And turn a sentence saying there's a bug into a description saying it's suboptimal.
With that: Acked-by: Anthony PERARD <[email protected]>
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
PATCH v6 4/16 libs/guest: Use a single write_exact in write_headers
In reply to Frediano ZiglioReduce number of syscalls by coalescing the image and the domain headers into a single I/O vector array.
--
Changes since v2:
- change prefix in subject.
Changes since v4:
- added Reviewed-by;
- improved commit message.
--- tools/libs/guest/xg_sr_save.c | 37 +++++++++++++++++------------------ 1 file changed, 18 insertions(+), 19 deletions(-)
tools/libs/guest/xg_sr_save.c +18 -19
@@ -10,17 +10,22 @@ static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) {xc_interface *xch = ctx->xch;int32_t xen_version =xc_version (xch, XENVER_version,NULL ); -struct xc_sr_ihdr ihdr = { - .marker =IHDR_MARKER , - .id =htonl (IHDR_ID ), - .version =htonl (3 ), - .options =htons (IHDR_OPT_LITTLE_ENDIAN ), - }; -struct xc_sr_dhdr dhdr = { - .type = guest_type, - .page_shift =XC_PAGE_SHIFT , - .xen_major = (xen_version >>16 ) &0xffff , - .xen_minor = (xen_version) &0xffff , +struct { +struct xc_sr_ihdr ihdr; +struct xc_sr_dhdr dhdr; + } hdrs = { + { + .marker =IHDR_MARKER , + .id =htonl (IHDR_ID ), + .version =htonl (3 ), + .options =htons (IHDR_OPT_LITTLE_ENDIAN ), + }, + { + .type = guest_type, + .page_shift =XC_PAGE_SHIFT , + .xen_major = (xen_version >>16 ) &0xffff , + .xen_minor = (xen_version) &0xffff , + }, };if ( xen_version <0 )
@@ -29,15 +34,9 @@ static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type)return -1 ; } -if (write_exact (ctx->fd, &ihdr,sizeof (ihdr)) ) - { -PERROR ("Unable to write Image Header to stream" ); -return -1 ; - } - -if (write_exact (ctx->fd, &dhdr,sizeof (dhdr)) ) +if (write_exact (ctx->fd, &hdrs,sizeof (hdrs)) ) { -PERROR ("Unable to write Domain Header to stream" ); +PERROR ("Unable to write Headers to stream" );return -1 ; }
Signature · 1 line
2.43.0
Re: [PATCH v6 04/16] libs/guest: Use a single write_exact in write_headers
Quoted 14 lines · Frediano Ziglio
tools/libs/guest/xg_sr_save.c +0
-6@@ -10,17 +10,22 @@ static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) {xc_interface *xch = ctx->xch;int32_t xen_version =xc_version (xch, XENVER_version,NULL ); -struct xc_sr_ihdr ihdr = { - .marker =IHDR_MARKER , - .id =htonl (IHDR_ID ), - .version =htonl (3 ), - .options =htons (IHDR_OPT_LITTLE_ENDIAN ), - };
tools/libs/guest/xg_sr_save.c static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) · line 17
Quoted 10 lines
tools/libs/guest/xg_sr_save.c +5
-5-struct xc_sr_dhdr dhdr = { - .type = guest_type, - .page_shift =XC_PAGE_SHIFT , - .xen_major = (xen_version >>16 ) &0xffff , - .xen_minor = (xen_version) &0xffff , +struct { +struct xc_sr_ihdr ihdr; +struct xc_sr_dhdr dhdr; + } hdrs = { + {
.ihdr = {
tools/libs/guest/xg_sr_save.c static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) · line 23
Quoted 6 lines
tools/libs/guest/xg_sr_save.c +6
-0+ .marker = IHDR_MARKER, + .id = htonl(IHDR_ID), + .version = htonl(3), + .options = htons(IHDR_OPT_LITTLE_ENDIAN), + }, + {
.dhdr = {
~Andrew
Re: [PATCH v6 04/16] libs/guest: Use a single write_exact in write_headers
Quoted 10 lines · Andrew Cooper
tools/libs/guest/xg_sr_save.c
Quoted 6 lines · Frediano Ziglio
tools/libs/guest/xg_sr_save.c +2
-0+ } hdrs = { + {.ihdr = {
Quoted 8 lines
tools/libs/guest/xg_sr_save.c static int write_headers(struct xc_sr_context *ctx, uint16_t guest_type) · line 23
Quoted 6 lines
tools/libs/guest/xg_sr_save.c +6
-0+ .marker = IHDR_MARKER, + .id = htonl(IHDR_ID), + .version = htonl(3), + .options = htons(IHDR_OPT_LITTLE_ENDIAN), + }, + {.dhdr = {
With that: Acked-by: Anthony PERARD <[email protected]>
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
PATCH v6 5/16 libs/guest: allocate various migration arrays just once
In reply to Frediano ZiglioFrom: Edwin Török <[email protected]>
Allocate these array just once at the start of migration, using the maximum batch size, and free them at the end.
--
Changes since v2:
- change prefix in subject.
Changes since v3:
- fix comment style
Changes since v4:
- change order of fields in structure.
--- tools/libs/guest/xg_sr_common.h | 13 +++++++ tools/libs/guest/xg_sr_save.c | 66 +++++++++++++-------------------- 2 files changed, 39 insertions(+), 40 deletions(-)
tools/libs/guest/xg_sr_common.h +13 -0
@@ -209,6 +209,18 @@ static inline int update_blob(struct xc_sr_blob *blob,return 0 ; } +struct xc_sr_context_save_buffers +{ +xen_pfn_t batch_pfns[MAX_BATCH_SIZE ]; +xen_pfn_t mfns[MAX_BATCH_SIZE ]; +xen_pfn_t types[MAX_BATCH_SIZE ]; +void *guest_data[MAX_BATCH_SIZE ]; +void *local_pages[MAX_BATCH_SIZE ]; +struct iovec iov[MAX_BATCH_SIZE +2 ];/* Headers + data. */ +uint64_t rec_pfns[MAX_BATCH_SIZE ]; +int errors[MAX_BATCH_SIZE ]; +}; +struct xc_sr_context {xc_interface *xch;
@@ -244,6 +256,7 @@ struct xc_sr_contextunsigned long *deferred_pages;unsigned long nr_deferred_pages;xc_hypercall_buffer_t dirty_bitmap_hbuf; +struct xc_sr_context_save_buffers *buffers; } save; struct /* Restore data. */
tools/libs/guest/xg_sr_save.c +26 -40
124 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v6 05/16] libs/guest: allocate various migration arrays just once
Quoted 7 lines · Frediano Ziglio
From: Edwin Török <[email protected]>
Allocate these array just once at the start of migration, using the maximum batch size, and free them at the end.
Signed-off-by Edwin Török <[email protected]>Signed-off-by Frediano Ziglio <[email protected]>
The reason why these are allocated and freed on every iteration is so they show up as uninitialised to valgrind or ASAN.
Maybe that's overly cautious, and maybe we can relax it, but it's also not as if these allocations/frees are anywhere but in the noise on this path.
Quoted 38 lines
--
Changes since v2:
- change prefix in subject.Changes since v3:
- fix comment styleChanges since v4:
- change order of fields in structure.--- tools/libs/guest/xg_sr_common.h | 13 +++++++ tools/libs/guest/xg_sr_save.c | 66 +++++++++++++-------------------- 2 files changed, 39 insertions(+), 40 deletions(-)tools/libs/guest/xg_sr_common.h +12
-0@@ -209,6 +209,18 @@ static inline int update_blob(struct xc_sr_blob *blob,return 0 ; } +struct xc_sr_context_save_buffers +{ +xen_pfn_t batch_pfns[MAX_BATCH_SIZE ]; +xen_pfn_t mfns[MAX_BATCH_SIZE ]; +xen_pfn_t types[MAX_BATCH_SIZE ]; +void *guest_data[MAX_BATCH_SIZE ]; +void *local_pages[MAX_BATCH_SIZE ]; +struct iovec iov[MAX_BATCH_SIZE +2 ];/* Headers + data. */ +uint64_t rec_pfns[MAX_BATCH_SIZE ]; +int errors[MAX_BATCH_SIZE ]; +}; +struct xc_sr_context {xc_interface *xch;
tools/libs/guest/xg_sr_common.h struct xc_sr_context · line 259
Quoted 4 lines
tools/libs/guest/xg_sr_common.h +1
-0@@ -244,6 +256,7 @@ struct xc_sr_contextunsigned long *deferred_pages;unsigned long nr_deferred_pages;xc_hypercall_buffer_t dirty_bitmap_hbuf; +struct xc_sr_context_save_buffers *buffers;
Please move the higher hunk down here, as:
struct xc_sr_context_safe_buffers {
...
} *buffers;
This helps keep related content together.
(I'm half tempted to say they don't even need a second memory allocation, but right now xc_sr_context is 538 bytes, and this buffer object is nearly 16k which we don't really want to be adding as overhead to the restore side.)
Quoted 55 lines
tools/libs/guest/xg_sr_common.h +0
-0} save; struct /* Restore data. */tools/libs/guest/xg_sr_save.c +15
-11@@ -86,16 +86,16 @@ static int write_checkpoint_record(struct xc_sr_context *ctx)static int write_batch (struct xc_sr_context *ctx) {xc_interface *xch = ctx->xch; -xen_pfn_t *mfns =NULL , *types =NULL ; +xen_pfn_t *mfns, *types;void *guest_mapping =NULL ; -void **guest_data =NULL ; -void **local_pages =NULL ; -int *errors =NULL , rc =-1 ; +void **guest_data; +void **local_pages; +int *errors, rc =-1 ;unsigned int i, p, nr_pages =0 , nr_pages_mapped =0 ;unsigned int nr_pfns = ctx->save.nr_batch_pfns;void *page, *orig_page; -uint64_t *rec_pfns =NULL ; -struct iovec *iov =NULL ;int iovcnt =0 ; +uint64_t *rec_pfns; +struct iovec *iov;int iovcnt =0 ;struct {struct xc_sr_rhdr rec;struct xc_sr_rec_page_data_header page_data;@@ -104,26 +104,24 @@ static int write_batch(struct xc_sr_context *ctx) };assert (nr_pfns !=0 ); +assert (nr_pfns <=MAX_BATCH_SIZE ); +assert (ctx->save.buffers);/* Mfns of the batch pfns. */ - mfns =malloc (nr_pfns *sizeof (*mfns)); + mfns = ctx->save.buffers->mfns;/* Types of the batch pfns. */ - types =malloc (nr_pfns *sizeof (*types)); + types = ctx->save.buffers->types;/* Errors from attempting to map the gfns. */ - errors =malloc (nr_pfns *sizeof (*errors)); + errors = ctx->save.buffers->errors;/* Pointers to page data to send. Mapped gfns or local allocations. */ - guest_data =calloc (nr_pfns,sizeof (*guest_data)); + guest_data = ctx->save.buffers->guest_data; +memset (guest_data,0 ,sizeof (*guest_data) * nr_pfns);/* Pointers to locally allocated pages. Need freeing. */ - local_pages =calloc (nr_pfns,sizeof (*local_pages)); + local_pages = ctx->save.buffers->local_pages; +memset (local_pages,0 ,sizeof (*local_pages) * nr_pfns);/* iovec[] for writev(). */
tools/libs/guest/xg_sr_save.c static int write_batch(struct xc_sr_context *ctx) · line 124
Quoted 10 lines
tools/libs/guest/xg_sr_save.c +2
-8- iov =malloc ((nr_pfns +2 ) *sizeof (*iov)); - -if ( !mfns || !types || !errors || !guest_data || !local_pages || !iov ) - { -ERROR ("Unable to allocate arrays for a batch of %u pages" , - nr_pfns); - goto err; - } + iov = ctx->save.buffers->iov; + rec_pfns = ctx->save.buffers->rec_pfns;
These two hunks are rather messy. You don't actually need the first hunk at all; the pointers can all start initialised to NULL.
Alternatively, if you want to avoid the redundant assignments, then split the variable block in half and list the second half as /* shorthand names for the buffers */ or somesuch. This will need to come ahead of the asserts().
But if you're going to try cleaning this up, please do it in a separate patch.
Quoted 26 lines
tools/libs/guest/xg_sr_save.c +0
-9for ( i =0 ; i < nr_pfns; ++i ) {@@ -209,14 +207,6 @@ static int write_batch(struct xc_sr_context *ctx) } } - rec_pfns =malloc (nr_pfns *sizeof (*rec_pfns)); -if ( !rec_pfns ) - { -ERROR ("Unable to allocate %zu bytes of memory for page data pfn list" , - nr_pfns *sizeof (*rec_pfns)); - goto err; - } - hdrs.rec.length =sizeof (hdrs.page_data); hdrs.rec.length += nr_pfns *sizeof (*rec_pfns); hdrs.rec.length += nr_pages *PAGE_SIZE ;@@ -267,17 +257,13 @@ static int write_batch(struct xc_sr_context *ctx) rc = ctx->save.nr_batch_pfns =0 ; err: -free (rec_pfns);if ( guest_mapping )xenforeignmemory_unmap (xch->fmem, guest_mapping, nr_pages_mapped);for ( i =0 ; local_pages && i < nr_pfns; ++i )
tools/libs/guest/xg_sr_save.c static int write_batch(struct xc_sr_context *ctx) · line 266
Quoted 10 lines
tools/libs/guest/xg_sr_save.c +3
-6+ {free (local_pages[i]); -free (iov); -free (local_pages); -free (guest_data); -free (errors); -free (types); -free (mfns); + local_pages[i] =NULL ; + }
Given this NULL-ing, the memset earlier shouldn't be needed.
Along with a memset() over guest_mapping, that gets rid of all the early memset()'s I think.
Quoted 14 lines
tools/libs/guest/xg_sr_save.c +2
-3return rc; }@@ -806,18 +792,18 @@ static int setup(struct xc_sr_context *ctx) dirty_bitmap =xc_hypercall_buffer_alloc_pages ( xch, dirty_bitmap,NRPAGES (bitmap_size (ctx->save.p2m_size))); - ctx->save.batch_pfns =malloc (MAX_BATCH_SIZE * -sizeof (*ctx->save.batch_pfns)); ctx->save.deferred_pages =bitmap_alloc (ctx->save.p2m_size); + ctx->save.buffers =calloc (1 ,sizeof (*ctx->save.buffers)); -if ( !ctx->save.batch_pfns || !dirty_bitmap || !ctx->save.deferred_pages ) +if ( !dirty_bitmap || !ctx->save.deferred_pages || !ctx->save.buffers)
tools/libs/guest/xg_sr_save.c static int setup(struct xc_sr_context *ctx) · line 806
Quoted 10 lines
tools/libs/guest/xg_sr_save.c +3
-2{ -ERROR ("Unable to allocate memory for dirty bitmaps, batch pfns and" -" deferred pages" ); +ERROR ("Unable to allocate memory for dirty bitmaps, deferred pages" +" and various batch buffers" ); rc =-1 ; errno =ENOMEM ; goto err; } + ctx->save.batch_pfns = ctx->save.buffers->batch_pfns;
This is wonky. As far as I can tell, you've included batch_pfns in the buffers struct, but left it's old pointer in place, meaning it becomes dangling when the allocation is freed.
This wants splitting into two patches. First introduce the buffers struct with batch_pfns moved only, and sort out the allocation here. Then in the subsequent patch, move the contents of write_batch() into the buffers struct.
~Andrew
PATCH v6 6/16 libs/call: cache up to 4 pages in hypercall bounce buffers
In reply to Frediano ZiglioFrom: Edwin Török <[email protected]>
During migration there are a lot of mmap/munmap calls, because xc_get_pfn_type_batch exceeds the default hypercall bounce buffer cache size, and needs to allocate every time it is called.
munmap is slow, especially in a PV Dom0 (takes an emulation fault), so is best avoided.
Eventually it'd be good if the memory pool from xmalloc_tlsf.c was reused here, but for now make it handle the commonly encountered sizes (so far up to 4 pages).
--
Changes since v2:
- change prefix in subject.
Changes since v4:
- fix off-by-one bug.
Changes since v5:
- avoids potential buffer underflow if nr_pages is 0 calling cache_alloc.
--- tools/libs/call/buffer.c | 31 ++++++++++++++++++++----------- tools/libs/call/core.c | 3 ++- tools/libs/call/private.h | 8 +++++--- 3 files changed, 27 insertions(+), 15 deletions(-)
tools/libs/call/buffer.c +20 -11
@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) {void *p =NULL ; +if ( nr_pages ==0 ) +return NULL ; +cache_lock (xcall); xcall->buffer_total_allocations++;
@@ -56,13 +59,13 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages)if ( xcall->buffer_current_allocations > xcall->buffer_maximum_allocations ) xcall->buffer_maximum_allocations = xcall->buffer_current_allocations; -if ( nr_pages >1 ) +if ( nr_pages >ARRAY_SIZE (xcall->buffer_cache) ) { xcall->buffer_cache_toobig++; } -else if ( xcall->buffer_cache_nr >0 ) +else if ( xcall->buffer_cache_nr[nr_pages-1 ] >0 ) { - p = xcall->buffer_cache[--xcall->buffer_cache_nr]; + p = xcall->buffer_cache[nr_pages-1 ][--xcall->buffer_cache_nr[nr_pages-1 ]]; xcall->buffer_cache_hits++; }else
@@ -84,10 +87,10 @@ static int cache_free(xencall_handle *xcall, void *p, size_t nr_pages) xcall->buffer_total_releases++; xcall->buffer_current_allocations--; -if ( nr_pages ==1 && - xcall->buffer_cache_nr <BUFFER_CACHE_SIZE ) +if ( nr_pages && nr_pages <=ARRAY_SIZE (xcall->buffer_cache) && + xcall->buffer_cache_nr[nr_pages-1 ] <BUFFER_CACHE_SIZE ) { - xcall->buffer_cache[xcall->buffer_cache_nr++] = p; + xcall->buffer_cache[nr_pages-1 ][xcall->buffer_cache_nr[nr_pages-1 ]++] = p; rc =1 ; }
@@ -108,17 +111,23 @@ void buffer_release_cache(xencall_handle *xcall)DBGPRINTF ("current allocations:%d maximum allocations:%d" , xcall->buffer_current_allocations, xcall->buffer_maximum_allocations); -DBGPRINTF ("cache current size:%d" , - xcall->buffer_cache_nr); +for (unsigned i =0 ; i <ARRAY_SIZE (xcall->buffer_cache_nr); ++i ) + { +DBGPRINTF ("cache current size[%u pages]:%d" , i+1 , + xcall->buffer_cache_nr[i]); + }DBGPRINTF ("cache hits:%d misses:%d toobig:%d" , xcall->buffer_cache_hits, xcall->buffer_cache_misses, xcall->buffer_cache_toobig); -while ( xcall->buffer_cache_nr >0 ) +for (unsigned i =0 ; i <ARRAY_SIZE (xcall->buffer_cache_nr); ++i ) { - p = xcall->buffer_cache[--xcall->buffer_cache_nr]; -osdep_free_pages (xcall, p,1 ); +while ( xcall->buffer_cache_nr[i] >0 ) + { + p = xcall->buffer_cache[i][--xcall->buffer_cache_nr[i]]; +osdep_free_pages (xcall, p, i +1 ); + } }cache_unlock (xcall);
tools/libs/call/core.c +2 -1
@@ -14,6 +14,7 @@*/ #include <stdlib.h> +#include <string.h> #include "private.h"
@@ -44,7 +45,7 @@ xencall_handle *xencall_open(xentoollog_logger *logger, unsigned open_flags)xentoolcore__register_active_handle (&xcall->tc_ah); xcall->flags = open_flags; - xcall->buffer_cache_nr =0 ; +memset (xcall->buffer_cache_nr,0 ,sizeof (xcall->buffer_cache_nr)); xcall->buffer_total_allocations =0 ; xcall->buffer_total_releases =0 ;
tools/libs/call/private.h +5 -3
@@ -31,13 +31,15 @@ struct xencall_handle {Xentoolcore__Active_Handle tc_ah;/* -* A simple cache of unused, single page, hypercall buffers +* A simple cache of unused, small, hypercall buffers +* buffer_cache[i]'s size is (i+1) pages * * Protected by a global lock. */ #define BUFFER_CACHE_SIZE 4 -int buffer_cache_nr; -void *buffer_cache[BUFFER_CACHE_SIZE ]; +#define BUFFER_CACHE_NRPAGES 4 +int buffer_cache_nr[BUFFER_CACHE_NRPAGES ]; +void *buffer_cache[BUFFER_CACHE_NRPAGES ][BUFFER_CACHE_SIZE ];/* * Hypercall buffer statistics. All protected by the global
Signature · 1 line
2.43.0
Re: [PATCH v6 06/16] libs/call: cache up to 4 pages in hypercall bounce buffers
Quoted 5 lines · Frediano Ziglio
From: Edwin Török <[email protected]>
During migration there are a lot of mmap/munmap calls, because
xc_get_pfn_type_batchexceeds the default hypercall bounce buffer cache size, and needs to allocate every time it is called.
I think xc_get_pfn_type_batch() would allocate a buffer of 2 page top, in write_batch(), right ?
Quoted 30 lines
munmapis slow, especially in a PV Dom0 (takes an emulation fault), so is best avoided.Eventually it'd be good if the memory pool from
xmalloc_tlsf.cwas reused here, but for now make it handle the commonly encountered sizes (so far up to 4 pages).Signed-off-by Edwin Török <[email protected]>Signed-off-by Frediano Ziglio <[email protected]>--
Changes since v2:
- change prefix in subject.Changes since v4:
- fix off-by-one bug.Changes since v5:
- avoids potential buffer underflow ifnr_pagesis 0 callingcache_alloc.--- tools/libs/call/buffer.c | 31 ++++++++++++++++++++----------- tools/libs/call/core.c | 3 ++- tools/libs/call/private.h | 8 +++++--- 3 files changed, 27 insertions(+), 15 deletions(-)tools/libs/call/buffer.c +0
-0
tools/libs/call/buffer.c static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) · line 53
Quoted 5 lines
tools/libs/call/buffer.c +2
-0@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) {void *p =NULL ; +if ( nr_pages ==0 ) +return NULL ;
By doing that check here, we don't update the stat anymore. And it's getting out-of-sync with the updates done in cache_free().
Before, we where returning a cache entry for that, and cache_hit++. I think it's ok to return cache_miss++ instead.
Quoted 111 lines
tools/libs/call/buffer.c +17
-11cache_lock (xcall); xcall->buffer_total_allocations++;@@ -56,13 +59,13 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages)if ( xcall->buffer_current_allocations > xcall->buffer_maximum_allocations ) xcall->buffer_maximum_allocations = xcall->buffer_current_allocations; -if ( nr_pages >1 ) +if ( nr_pages >ARRAY_SIZE (xcall->buffer_cache) ) { xcall->buffer_cache_toobig++; } -else if ( xcall->buffer_cache_nr >0 ) +else if ( xcall->buffer_cache_nr[nr_pages-1 ] >0 ) { - p = xcall->buffer_cache[--xcall->buffer_cache_nr]; + p = xcall->buffer_cache[nr_pages-1 ][--xcall->buffer_cache_nr[nr_pages-1 ]]; xcall->buffer_cache_hits++; }else @@ -84,10 +87,10 @@ static int cache_free(xencall_handle *xcall, void *p, size_t nr_pages) xcall->buffer_total_releases++; xcall->buffer_current_allocations--; -if ( nr_pages ==1 && - xcall->buffer_cache_nr <BUFFER_CACHE_SIZE ) +if ( nr_pages && nr_pages <=ARRAY_SIZE (xcall->buffer_cache) && + xcall->buffer_cache_nr[nr_pages-1 ] <BUFFER_CACHE_SIZE ) { - xcall->buffer_cache[xcall->buffer_cache_nr++] = p; + xcall->buffer_cache[nr_pages-1 ][xcall->buffer_cache_nr[nr_pages-1 ]++] = p; rc =1 ; }@@ -108,17 +111,23 @@ void buffer_release_cache(xencall_handle *xcall)DBGPRINTF ("current allocations:%d maximum allocations:%d" , xcall->buffer_current_allocations, xcall->buffer_maximum_allocations); -DBGPRINTF ("cache current size:%d" , - xcall->buffer_cache_nr); +for (unsigned i =0 ; i <ARRAY_SIZE (xcall->buffer_cache_nr); ++i ) + { +DBGPRINTF ("cache current size[%u pages]:%d" , i+1 , + xcall->buffer_cache_nr[i]); + }DBGPRINTF ("cache hits:%d misses:%d toobig:%d" , xcall->buffer_cache_hits, xcall->buffer_cache_misses, xcall->buffer_cache_toobig); -while ( xcall->buffer_cache_nr >0 ) +for (unsigned i =0 ; i <ARRAY_SIZE (xcall->buffer_cache_nr); ++i ) { - p = xcall->buffer_cache[--xcall->buffer_cache_nr]; -osdep_free_pages (xcall, p,1 ); +while ( xcall->buffer_cache_nr[i] >0 ) + { + p = xcall->buffer_cache[i][--xcall->buffer_cache_nr[i]]; +osdep_free_pages (xcall, p, i +1 ); + } }cache_unlock (xcall);tools/libs/call/core.c +2
-1@@ -14,6 +14,7 @@*/ #include <stdlib.h> +#include <string.h> #include "private.h" @@ -44,7 +45,7 @@ xencall_handle *xencall_open(xentoollog_logger *logger, unsigned open_flags)xentoolcore__register_active_handle (&xcall->tc_ah); xcall->flags = open_flags; - xcall->buffer_cache_nr =0 ; +memset (xcall->buffer_cache_nr,0 ,sizeof (xcall->buffer_cache_nr)); xcall->buffer_total_allocations =0 ; xcall->buffer_total_releases =0 ;tools/libs/call/private.h +5
-3@@ -31,13 +31,15 @@ struct xencall_handle {Xentoolcore__Active_Handle tc_ah;/* -* A simple cache of unused, single page, hypercall buffers +* A simple cache of unused, small, hypercall buffers +* buffer_cache[i]'s size is (i+1) pages * * Protected by a global lock. */ #define BUFFER_CACHE_SIZE 4 -int buffer_cache_nr; -void *buffer_cache[BUFFER_CACHE_SIZE ]; +#define BUFFER_CACHE_NRPAGES 4 +int buffer_cache_nr[BUFFER_CACHE_NRPAGES ]; +void *buffer_cache[BUFFER_CACHE_NRPAGES ][BUFFER_CACHE_SIZE ];/* * Hypercall buffer statistics. All protected by the global Signature · 1 line
2.43.0
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH v6 06/16] libs/call: cache up to 4 pages in hypercall bounce buffers
Quoted 9 lines · Anthony PERARD
Quoted 5 lines · Frediano Ziglio
From: Edwin Török <[email protected]>
During migration there are a lot of mmap/munmap calls, because
xc_get_pfn_type_batchexceeds the default hypercall bounce buffer cache size, and needs to allocate every time it is called.I think
xc_get_pfn_type_batch()would allocate a buffer of 2 page top, inwrite_batch(), right ?
(because nr_pfns <= MAX_BATCH_SIZE, and we allocate nr_pfns*sizeof(unsigned long)
Quoted 6 lines
Quoted 6 lines
munmapis slow, especially in a PV Dom0 (takes an emulation fault), so is best avoided.Eventually it'd be good if the memory pool from
xmalloc_tlsf.cwas reused here, but for now make it handle the commonly encountered sizes (so far up to 4 pages).
So do you know what would allocate 4 pages?
In anycase, I guess it's ok to keep an allocation of 160kb for a short while.
Quoted 8 lines
Quoted 8 lines
tools/libs/call/buffer.c +0
-0@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) {void *p =NULL ;
Quoted 9 lines
tools/libs/call/buffer.c static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) · line 53
Quoted 2 lines
tools/libs/call/buffer.c +2
-0+if ( nr_pages ==0 ) +return NULL ;By doing that check here, we don't update the stat anymore. And it's getting out-of-sync with the updates done in
cache_free().Before, we where returning a cache entry for that, and cache_hit++. I think it's ok to return cache_miss++ instead.
The rest of the patch looks fine to me, and I guess is ok.
(and I send the previous mail a bit too soon)
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH v6 06/16] libs/call: cache up to 4 pages in hypercall bounce buffers
In reply to Anthony PERARDQuoted 11 lines · Anthony PERARD
Quoted 5 lines · Frediano Ziglio
From: Edwin Török <[email protected]>
During migration there are a lot of mmap/munmap calls, because
xc_get_pfn_type_batchexceeds the default hypercall bounce buffer cache size, and needs to allocate every time it is called.I think
xc_get_pfn_type_batch()would allocate a buffer of 2 page top, inwrite_batch(), right ?
Yes. That however does not contradict the sense of the sentence (or even the commit message).
Quoted 8 lines
Quoted 8 lines
munmapis slow, especially in a PV Dom0 (takes an emulation fault), so is best avoided.Eventually it'd be good if the memory pool from
xmalloc_tlsf.cwas reused here, but for now make it handle the commonly encountered sizes (so far up to 4 pages).
If a program uses 3/4 pages it will use the additional cache, if not there's no much difference.
Quoted 25 lines
Quoted 25 lines
Signed-off-by Edwin Török <[email protected]>Signed-off-by Frediano Ziglio <[email protected]>--
Changes since v2:
- change prefix in subject.Changes since v4:
- fix off-by-one bug.Changes since v5:
- avoids potential buffer underflow ifnr_pagesis 0 callingcache_alloc.--- tools/libs/call/buffer.c | 31 ++++++++++++++++++++----------- tools/libs/call/core.c | 3 ++- tools/libs/call/private.h | 8 +++++--- 3 files changed, 27 insertions(+), 15 deletions(-)tools/libs/call/buffer.c +0
-0@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) {void *p =NULL ;
Quoted 9 lines
tools/libs/call/buffer.c static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) · line 53
Quoted 2 lines
tools/libs/call/buffer.c +2
-0+if ( nr_pages ==0 ) +return NULL ;By doing that check here, we don't update the stat anymore. And it's getting out-of-sync with the updates done in
cache_free().Before, we where returning a cache entry for that, and cache_hit++. I think it's ok to return cache_miss++ instead.
Well... requesting 0 pages is weird by definition, even malloc(0) is not well defined.
In theory in this case returning NULL would cause cache_free to not be called as filtered by xencall_free_buffer_pages.
I think the most symmetric think would be adding a similar test in cache_free, like
static int cache_free(xencall_handle *xcall, void *p, size_t nr_pages)
{
int rc = 0;
if ( nr_pages == 0 )
return 1;
cache_lock(xcall);(the return 1 is needed to prevent the attempt to munmap the pointer which does not make sense).
Quoted 105 lines
Quoted 105 lines
tools/libs/call/buffer.c +17
-11cache_lock (xcall); xcall->buffer_total_allocations++;@@ -56,13 +59,13 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages)if ( xcall->buffer_current_allocations > xcall->buffer_maximum_allocations ) xcall->buffer_maximum_allocations = xcall->buffer_current_allocations; -if ( nr_pages >1 ) +if ( nr_pages >ARRAY_SIZE (xcall->buffer_cache) ) { xcall->buffer_cache_toobig++; } -else if ( xcall->buffer_cache_nr >0 ) +else if ( xcall->buffer_cache_nr[nr_pages-1 ] >0 ) { - p = xcall->buffer_cache[--xcall->buffer_cache_nr]; + p = xcall->buffer_cache[nr_pages-1 ][--xcall->buffer_cache_nr[nr_pages-1 ]]; xcall->buffer_cache_hits++; }else @@ -84,10 +87,10 @@ static int cache_free(xencall_handle *xcall, void *p, size_t nr_pages) xcall->buffer_total_releases++; xcall->buffer_current_allocations--; -if ( nr_pages ==1 && - xcall->buffer_cache_nr <BUFFER_CACHE_SIZE ) +if ( nr_pages && nr_pages <=ARRAY_SIZE (xcall->buffer_cache) && + xcall->buffer_cache_nr[nr_pages-1 ] <BUFFER_CACHE_SIZE ) { - xcall->buffer_cache[xcall->buffer_cache_nr++] = p; + xcall->buffer_cache[nr_pages-1 ][xcall->buffer_cache_nr[nr_pages-1 ]++] = p; rc =1 ; }@@ -108,17 +111,23 @@ void buffer_release_cache(xencall_handle *xcall)DBGPRINTF ("current allocations:%d maximum allocations:%d" , xcall->buffer_current_allocations, xcall->buffer_maximum_allocations); -DBGPRINTF ("cache current size:%d" , - xcall->buffer_cache_nr); +for (unsigned i =0 ; i <ARRAY_SIZE (xcall->buffer_cache_nr); ++i ) + { +DBGPRINTF ("cache current size[%u pages]:%d" , i+1 , + xcall->buffer_cache_nr[i]); + }DBGPRINTF ("cache hits:%d misses:%d toobig:%d" , xcall->buffer_cache_hits, xcall->buffer_cache_misses, xcall->buffer_cache_toobig); -while ( xcall->buffer_cache_nr >0 ) +for (unsigned i =0 ; i <ARRAY_SIZE (xcall->buffer_cache_nr); ++i ) { - p = xcall->buffer_cache[--xcall->buffer_cache_nr]; -osdep_free_pages (xcall, p,1 ); +while ( xcall->buffer_cache_nr[i] >0 ) + { + p = xcall->buffer_cache[i][--xcall->buffer_cache_nr[i]]; +osdep_free_pages (xcall, p, i +1 ); + } }cache_unlock (xcall);tools/libs/call/core.c +2
-1@@ -14,6 +14,7 @@*/ #include <stdlib.h> +#include <string.h> #include "private.h" @@ -44,7 +45,7 @@ xencall_handle *xencall_open(xentoollog_logger *logger, unsigned open_flags)xentoolcore__register_active_handle (&xcall->tc_ah); xcall->flags = open_flags; - xcall->buffer_cache_nr =0 ; +memset (xcall->buffer_cache_nr,0 ,sizeof (xcall->buffer_cache_nr)); xcall->buffer_total_allocations =0 ; xcall->buffer_total_releases =0 ;tools/libs/call/private.h +5
-3@@ -31,13 +31,15 @@ struct xencall_handle {Xentoolcore__Active_Handle tc_ah;/* -* A simple cache of unused, single page, hypercall buffers +* A simple cache of unused, small, hypercall buffers +* buffer_cache[i]'s size is (i+1) pages * * Protected by a global lock. */ #define BUFFER_CACHE_SIZE 4 -int buffer_cache_nr; -void *buffer_cache[BUFFER_CACHE_SIZE ]; +#define BUFFER_CACHE_NRPAGES 4 +int buffer_cache_nr[BUFFER_CACHE_NRPAGES ]; +void *buffer_cache[BUFFER_CACHE_NRPAGES ][BUFFER_CACHE_SIZE ];
Quoted 2 lines
tools/libs/call/private.h struct xencall_handle { · line 45
Quoted 2 lines
tools/libs/call/private.h +0
-0/* * Hypercall buffer statistics. All protected by the global
Frediano
Re: [PATCH v6 06/16] libs/call: cache up to 4 pages in hypercall bounce buffers
Quoted 16 lines · Frediano Ziglio
Quoted 15 lines · Anthony PERARD
tools/libs/call/buffer.c static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) · line 53
Quoted 10 lines · Frediano Ziglio
tools/libs/call/buffer.c +2
-0@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) {void *p =NULL ; +if ( nr_pages ==0 ) +return NULL ;By doing that check here, we don't update the stat anymore. And it's getting out-of-sync with the updates done in
cache_free().
Quoted 6 lines
Quoted 3 lines
Before, we where returning a cache entry for that, and cache_hit++. I think it's ok to return cache_miss++ instead.
Well... requesting 0 pages is weird by definition, even malloc(0) is not well defined.
malloc(0) isn't defined as weird, it is defined as "implementation-defined" ;-). But the pointer that the cache function handle isn't from malloc().
Quoted 2 lines
In theory in this case returning NULL would cause
cache_freeto not be called as filtered byxencall_free_buffer_pages.
Yes, for cases where the allocator returned NULL. But I can't find any guaranty of this. So I would prefer to have both cache_alloc() and cache_free() behave the same way when faced with nr_pages==0, without hindsight into the value of the pointer.
Quoted 14 lines
I think the most symmetric think would be adding a similar test in
cache_free, likestatic int
cache_free(xencall_handle *xcall, void *p, size_t nr_pages){ int rc = 0; if ( nr_pages == 0 ) return 1; cache_lock(xcall);
Quoted 2 lines
(the return 1 is needed to prevent the attempt to munmap the pointer which does not make sense).
If we have a pointer that is not NULL, we must free it. Even if you think it doesn't make sense. Also, there's no way to know, here, whether munmap() or an other function is going to be used. So, cache_free() must not say that it cached the pointer, and let the caller free it.
Cheers,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH v6 06/16] libs/call: cache up to 4 pages in hypercall bounce buffers
Quoted 51 lines · Anthony PERARD
Quoted 16 lines · Frediano Ziglio
Quoted 15 lines · Anthony PERARD
tools/libs/call/buffer.c static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) · line 53
Quoted 10 lines · Frediano Ziglio
tools/libs/call/buffer.c +2
-0@@ -49,6 +49,9 @@ static void *cache_alloc(xencall_handle *xcall, size_t nr_pages) {void *p =NULL ; +if ( nr_pages ==0 ) +return NULL ;By doing that check here, we don't update the stat anymore. And it's getting out-of-sync with the updates done in
cache_free().Quoted 6 lines
Quoted 3 lines
Before, we where returning a cache entry for that, and cache_hit++. I think it's ok to return cache_miss++ instead.
Well... requesting 0 pages is weird by definition, even malloc(0) is not well defined.
malloc(0) isn't defined as weird, it is defined as "implementation-defined" ;-). But the pointer that the cache function handle isn't from
malloc().Quoted 2 lines
In theory in this case returning NULL would cause
cache_freeto not be called as filtered byxencall_free_buffer_pages.Yes, for cases where the allocator returned NULL. But I can't find any guaranty of this. So I would prefer to have both
cache_alloc()andcache_free()behave the same way when faced with nr_pages==0, without hindsight into the value of the pointer.Quoted 14 lines
I think the most symmetric think would be adding a similar test in
cache_free, likestatic int
cache_free(xencall_handle *xcall, void *p, size_t nr_pages){ int rc = 0; if ( nr_pages == 0 ) return 1; cache_lock(xcall);
Quoted 8 lines
Quoted 2 lines
(the return 1 is needed to prevent the attempt to munmap the pointer which does not make sense).
If we have a pointer that is not NULL, we must free it. Even if you think it doesn't make sense. Also, there's no way to know, here, whether
munmap()or an other function is going to be used. So,cache_free()must not say that it cached the pointer, and let the caller free it.
Changed to return 0, NULL pointer is handled by the caller anyway.
Quoted 9 lines
Cheers,
--
Anthony Perard | Vates XCP-ng DeveloperXCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
PATCH v6 7/16 libs/guest: avoids using 2 indexes
In reply to Frediano ZiglioSimplify code, after the first scan of the various arrays we don't need to keep original types and PFNs but only the ones having data.
--
Changes since v4:
- added Reviewed-by.
--- tools/libs/guest/xg_sr_restore.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
tools/libs/guest/xg_sr_restore.c +13 -13
@@ -260,9 +260,7 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count,int *map_errs =malloc (count *sizeof (*map_errs));int rc;void *mapping =NULL , *guest_page =NULL ; -unsigned int i,/* i indexes the pfns from the record. */ - j,/* j indexes the subset of pfns we decide to map. */ - nr_pages =0 ; +unsigned nr_pages;if ( !mfns || !map_errs ) {
@@ -279,12 +277,18 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count, goto err; } -for ( i =0 ; i < count; ++i ) + nr_pages =0 ; +for (unsigned i =0 ; i < count; ++i ) { ctx->restore.ops.set_page_type (ctx, pfns[i], types[i]); -if (page_type_has_stream_data (types[i]) ) - mfns[nr_pages++] = ctx->restore.ops.pfn_to_gfn (ctx, pfns[i]); +if ( !page_type_has_stream_data (types[i]) ) +continue ; + + mfns[nr_pages] = ctx->restore.ops.pfn_to_gfn (ctx, pfns[i]); + pfns[nr_pages] = pfns[i]; + types[nr_pages] = types[i]; + nr_pages++; }/* Nothing to do? */
@@ -302,16 +306,13 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count, goto err; } -for ( i =0 , j =0 ; i < count; ++i ) +for (unsigned i =0 ; i < nr_pages; ++i ) { -if ( !page_type_has_stream_data (types[i]) ) -continue ; - -if ( map_errs[j] ) +if ( map_errs[i] ) { rc =-1 ;ERROR ("Mapping pfn %#" PRIpfn" (mfn %#" PRIpfn", type %#" PRIx32") failed with %d" , - pfns[i], mfns[j], types[i], map_errs[j]); + pfns[i], mfns[i], types[i], map_errs[i]); goto err; }
@@ -337,7 +338,6 @@ static int process_page_data(struct xc_sr_context *ctx, unsigned int count,memcpy (guest_page, page_data,PAGE_SIZE ); } - ++j; guest_page +=PAGE_SIZE ; page_data +=PAGE_SIZE ; }
Signature · 1 line
2.43.0
Re: [PATCH v6 07/16] libs/guest: avoids using 2 indexes
Quoted 5 lines · Frediano Ziglio
Simplify code, after the first scan of the various arrays we don't need to keep original types and PFNs but only the ones having data.
Signed-off-by Frediano Ziglio <[email protected]>Reviewed-by Andrew Cooper <[email protected]>
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
PATCH v6 8/16 libs/guest: fill directly iov structure
In reply to Frediano ZiglioInstead of storing page pointers into an array and lately adding to iov vector add the pages directly to iov to avoid "guest_data" array.
--- tools/libs/guest/xg_sr_common.h | 1 - tools/libs/guest/xg_sr_save.c | 64 ++++++++++++--------------------- 2 files changed, 23 insertions(+), 42 deletions(-)
tools/libs/guest/xg_sr_common.h +0 -1
@@ -214,7 +214,6 @@ struct xc_sr_context_save_buffersxen_pfn_t batch_pfns[MAX_BATCH_SIZE ];xen_pfn_t mfns[MAX_BATCH_SIZE ];xen_pfn_t types[MAX_BATCH_SIZE ]; -void *guest_data[MAX_BATCH_SIZE ];void *local_pages[MAX_BATCH_SIZE ];struct iovec iov[MAX_BATCH_SIZE +2 ];/* Headers + data. */ uint64_t rec_pfns[MAX_BATCH_SIZE ];
tools/libs/guest/xg_sr_save.c +23 -41
106 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v6 08/16] libs/guest: fill directly iov structure
Quoted 126 lines · Frediano Ziglio
Instead of storing page pointers into an array and lately adding to iov vector add the pages directly to iov to avoid "
guest_data" array.Signed-off-by Frediano Ziglio <[email protected]>--- tools/libs/guest/xg_sr_common.h | 1 - tools/libs/guest/xg_sr_save.c | 64 ++++++++++++--------------------- 2 files changed, 23 insertions(+), 42 deletions(-)tools/libs/guest/xg_sr_common.h +0
-1@@ -214,7 +214,6 @@ struct xc_sr_context_save_buffersxen_pfn_t batch_pfns[MAX_BATCH_SIZE ];xen_pfn_t mfns[MAX_BATCH_SIZE ];xen_pfn_t types[MAX_BATCH_SIZE ]; -void *guest_data[MAX_BATCH_SIZE ];void *local_pages[MAX_BATCH_SIZE ];struct iovec iov[MAX_BATCH_SIZE +2 ];/* Headers + data. */ uint64_t rec_pfns[MAX_BATCH_SIZE ];tools/libs/guest/xg_sr_save.c +23
-39@@ -88,7 +88,6 @@ static int write_batch(struct xc_sr_context *ctx)xc_interface *xch = ctx->xch;xen_pfn_t *mfns, *types;void *guest_mapping =NULL ; -void **guest_data;void **local_pages;int *errors, rc =-1 ;unsigned int i, p, nr_pages =0 , nr_pages_mapped =0 ;@@ -113,9 +112,6 @@ static int write_batch(struct xc_sr_context *ctx) types = ctx->save.buffers->types;/* Errors from attempting to map the gfns. */ errors = ctx->save.buffers->errors; -/* Pointers to page data to send. Mapped gfns or local allocations. */ - guest_data = ctx->save.buffers->guest_data; -memset (guest_data,0 ,sizeof (*guest_data) * nr_pfns);/* Pointers to locally allocated pages. Need freeing. */ local_pages = ctx->save.buffers->local_pages;memset (local_pages,0 ,sizeof (*local_pages) * nr_pfns);@@ -158,6 +154,19 @@ static int write_batch(struct xc_sr_context *ctx) mfns[nr_pages++] = mfns[i]; } + hdrs.rec.length =sizeof (hdrs.page_data); + hdrs.rec.length += nr_pfns *sizeof (*rec_pfns); + + hdrs.page_data.count = nr_pfns; + + iov[0 ].iov_base = &hdrs; + iov[0 ].iov_len =sizeof (hdrs); + + iov[1 ].iov_base = rec_pfns; + iov[1 ].iov_len = nr_pfns *sizeof (*rec_pfns); + + iovcnt =2 ; +if ( nr_pages >0 ) { guest_mapping = xenforeignmemory_map(@@ -199,61 +208,34 @@ static int write_batch(struct xc_sr_context *ctx)else goto err; } +else if ( iov[iovcnt -1 ].iov_base + iov[iovcnt -1 ].iov_len != + page ) + { + iov[iovcnt].iov_base = page; + iov[iovcnt].iov_len =PAGE_SIZE ; + iovcnt++; + }else - guest_data[i] = page; + { + iov[iovcnt -1 ].iov_len +=PAGE_SIZE ; + } rc =-1 ; ++p; } } - hdrs.rec.length =sizeof (hdrs.page_data); - hdrs.rec.length += nr_pfns *sizeof (*rec_pfns); hdrs.rec.length += nr_pages *PAGE_SIZE ; - hdrs.page_data.count = nr_pfns; -for ( i =0 ; i < nr_pfns; ++i ) rec_pfns[i] = ((uint64_t )(types[i]) <<32 ) | ctx->save.batch_pfns[i]; - iov[0 ].iov_base = &hdrs; - iov[0 ].iov_len =sizeof (hdrs); - - iov[1 ].iov_base = rec_pfns; - iov[1 ].iov_len = nr_pfns *sizeof (*rec_pfns); - - iovcnt =2 ; - -if ( nr_pages ) - { -for ( i =0 ; i < nr_pfns; ++i ) - { -if ( !guest_data[i] ) -continue ; - -if ( iov[iovcnt -1 ].iov_base + iov[iovcnt -1 ].iov_len != - guest_data[i] ) - { - iov[iovcnt].iov_base = guest_data[i]; - iov[iovcnt].iov_len =PAGE_SIZE ; - iovcnt++; - } -else - { - iov[iovcnt -1 ].iov_len +=PAGE_SIZE ; - } - --nr_pages; - } - } -if (writev_exact (ctx->fd, iov, iovcnt) )
tools/libs/guest/xg_sr_save.c static int write_batch(struct xc_sr_context *ctx) · line 241
Quoted 10 lines
tools/libs/guest/xg_sr_save.c +0
-2{PERROR ("Failed to write page data to stream" ); goto err; } -/* Sanity check we have sent all the pages we expected to. */ -assert (nr_pages ==0 ); rc = ctx->save.nr_batch_pfns =0 ; err:
Looking at this patch, I think it wants merging with patch 3 and bringing ahead of patch 5.
You're undoing/redoing work in both of those patches, where I think it would be simpler to drop guest_data rather than convert it then drop.
Moving hdrs.rec.* can be done in patch 1 (which will probably simplify it's diff too).
~Andrew
PATCH v6 9/16 libs/ctrl: Allows writev_exact to change iov array
In reply to Frediano ZiglioAvoid having to allocate and copy the array if a partial write happens.
The implementation in tools/libs/store/xs.c already use this signature and method.
--
Changes since v2:
- change prefix in subject.
--- tools/libs/ctrl/xc_private.c | 26 +++++--------------------- tools/libs/ctrl/xc_private.h | 2 +- 2 files changed, 6 insertions(+), 22 deletions(-)
tools/libs/ctrl/xc_private.c +5 -21
@@ -635,7 +635,7 @@ int write_exact(int fd, const void *data, size_t size)/* * MiniOS's libc doesn't know about writev(). Implement it as multiple write()s. */ -int writev_exact (int fd,const struct iovec *iov,int iovcnt) +int writev_exact (int fd,struct iovec *iov,int iovcnt) {int rc, i;
@@ -649,9 +649,8 @@ int writev_exact(int fd, const struct iovec *iov, int iovcnt) return 0; }#else -int writev_exact (int fd,const struct iovec *iov,int iovcnt) +int writev_exact (int fd,struct iovec *iov,int iovcnt) { -struct iovec *local_iov =NULL ;int rc =0 , iov_idx =0 , saved_errno =0 ;ssize_t len;
@@ -686,23 +685,9 @@ int writev_exact(int fd, const struct iovec *iov, int iovcnt) len -= iov[iov_idx++].iov_len; else { -/* Partial write of iov[iov_idx]. Copy iov so we can adjust -* element iov_idx and resubmit the rest. */ -if ( !local_iov ) - { - local_iov =malloc (iovcnt *sizeof (*iov)); -if ( !local_iov ) - { - saved_errno =ENOMEM ; - rc =-1 ; - goto out; - } - - iov =memcpy (local_iov, iov, iovcnt *sizeof (*iov)); - } - - local_iov[iov_idx].iov_base += len; - local_iov[iov_idx].iov_len -= len; +/* Partial write of iov[iov_idx]. */ + iov[iov_idx].iov_base += len; + iov[iov_idx].iov_len -= len;break ; } }
@@ -711,7 +696,6 @@ int writev_exact(int fd, const struct iovec *iov, int iovcnt) saved_errno =0 ; out: -free (local_iov); errno = saved_errno;return rc; }
tools/libs/ctrl/xc_private.h +1 -1
@@ -383,7 +383,7 @@ int xc_flush_mmu_updates(xc_interface *xch, struct xc_mmu *mmu);/* Return 0 on success; -1 on error setting errno. */ int read_exact (int fd,void *data,size_t size);/* EOF => -1, errno=0 */ int write_exact (int fd,const void *data,size_t size); -int writev_exact (int fd,const struct iovec *iov,int iovcnt); +int writev_exact (int fd,struct iovec *iov,int iovcnt);int xc_ffs8 (uint8_t x);int xc_ffs16 (uint16_t x);
Signature · 1 line
2.43.0
Re: [PATCH v6 09/16] libs/ctrl: Allows writev_exact to change iov array
tools/libs/ctrl/xc_private.h int xc_flush_mmu_updates(xc_interface *xch, struct xc_mmu *mmu); · line 386
Quoted 10 lines · Frediano Ziglio
tools/libs/ctrl/xc_private.h +1
-1@@ -383,7 +383,7 @@ int xc_flush_mmu_updates(xc_interface *xch, struct xc_mmu *mmu);/* Return 0 on success; -1 on error setting errno. */ int read_exact (int fd,void *data,size_t size);/* EOF => -1, errno=0 */ int write_exact (int fd,const void *data,size_t size); -int writev_exact (int fd,const struct iovec *iov,int iovcnt); +int writev_exact (int fd,struct iovec *iov,int iovcnt);
No callers care, but this is written with a const pointer to match writev().
If we really do want to take this change, then it needs to come with a comment saying /* May edit iov to cope with partial writes. */
~Andrew
PATCH v6 10/16 libs/guest: add xg_foreignmemory_copy_{from,to}
In reply to Frediano ZiglioThis change prepare code to use a new "foreign copy" hypercall. The new hypercall will copy memory from/to a foreign domain. The new hypercall can be emulated with a sequence of:
- map foreign memory;
- copy memory;
- unmap foreign memory.
--
Changes since v5:
- Do not overwrite errno if xenforeignmemory_map fails.
--- tools/libs/guest/xg_sr_common.c | 57 +++++++++++++++++++++++++++++++++ tools/libs/guest/xg_sr_common.h | 8 +++++ 2 files changed, 65 insertions(+)
tools/libs/guest/xg_sr_common.c +57 -0
@@ -156,6 +156,63 @@ static void __attribute__((unused)) build_assertions(void)BUILD_BUG_ON (sizeof (struct xc_sr_rec_hvm_params ) !=8 ); } +enum { + foreigncopy_from, + foreigncopy_to +}; + +static int xg_foreignmemory_copy (xc_interface *xch,domid_t domid, +int dir,size_t nr_pages,void *buffer, +const xen_pfn_t foreign_pfns[nr_pages]) +{ +if ( nr_pages ==0 ) +return 0 ; + +if ( !buffer || !foreign_pfns ) + { + errno =EINVAL ; +return -1 ; + } + +int err[nr_pages]; +const int prot = (dir == foreigncopy_from) ?PROT_READ :PROT_READ |PROT_WRITE ; + +void *p =xenforeignmemory_map (xch->fmem, domid, prot, nr_pages, foreign_pfns, err); +if ( !p ) +return -1 ; + +for (size_t n =0 ; n < nr_pages; ++n ) +if ( err[n] ) + { +xenforeignmemory_unmap (xch->fmem, p, nr_pages); + errno = -err[n]; +return -1 ; + } + +if ( dir == foreigncopy_from ) +memcpy (buffer, p, nr_pages *XC_PAGE_SIZE ); +else +memcpy (p, buffer, nr_pages *XC_PAGE_SIZE ); + +return xenforeignmemory_unmap (xch->fmem, p, nr_pages); +} + +int xg_foreignmemory_copy_from (xc_interface *xch,domid_t dom, +size_t nr_pages,void *dest, +const xen_pfn_t source[nr_pages]) +{ +return xg_foreignmemory_copy (xch, dom, foreigncopy_from, + nr_pages, dest, source); +} + +int xg_foreignmemory_copy_to (xc_interface *xch,domid_t dom, +size_t nr_pages,const xen_pfn_t dest[nr_pages], +const void *source) +{ +return xg_foreignmemory_copy (xch, dom, foreigncopy_to, + nr_pages, (void *) source, dest); +} +/* * Local variables: * mode: C
tools/libs/guest/xg_sr_common.h +8 -0
@@ -553,6 +553,14 @@ static inline bool page_type_has_stream_data(uint32_t type) } } +int xg_foreignmemory_copy_from (xc_interface *xch,domid_t dom, +size_t nr_pages,void *dest, +const xen_pfn_t source[nr_pages]); + +int xg_foreignmemory_copy_to (xc_interface *xch,domid_t dom, +size_t nr_pages,const xen_pfn_t dest[nr_pages], +const void *source); +#endif /* * Local variables:
Signature · 1 line
2.43.0
Re: [PATCH v6 10/16] libs/guest: add xg_foreignmemory_copy_{from,to}
Quoted 6 lines · Frediano Ziglio
This change prepare code to use a new "foreign copy" hypercall. The new hypercall will copy memory from/to a foreign domain. The new hypercall can be emulated with a sequence of:
- map foreign memory;
- copy memory;
- unmap foreign memory.
I don't understand the point of this patch. The hypercall doesn't exist so there's nothing to emulate.
I've notice there's a patch later in the series which introduce a new hypercall, but the changes to the library should come after the hypercall is been introduced, only then can we check if the emulation is correct, or even needed.
Cheers,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH v6 10/16] libs/guest: add xg_foreignmemory_copy_{from,to}
Quoted 11 lines · Anthony PERARD
Quoted 6 lines · Frediano Ziglio
This change prepare code to use a new "foreign copy" hypercall. The new hypercall will copy memory from/to a foreign domain. The new hypercall can be emulated with a sequence of:
- map foreign memory;
- copy memory;
- unmap foreign memory.I don't understand the point of this patch. The hypercall doesn't exist so there's nothing to emulate.
I think that imitating something else is the definition of "emulate", we know what the hypercall should do so we emulate the wanted behavior.
Quoted 4 lines
I've notice there's a patch later in the series which introduce a new hypercall, but the changes to the library should come after the hypercall is been introduced, only then can we check if the emulation is correct, or even needed.
There are other changes after the hypercall.
In this case the new hypercall is to replace something that is already there. The base idea is that the new hypercall is able to do 3 steps together.
The reason to introduce the emulation first is that you can refactor on the emulation without having to introduce the new hypercall. Introducing the hypercall first would make testing more complicated as bugs on the hypercall have to be taken into account and considered. Also it is easier that way to enable or disable new code. For instance you want to test for performance regression (in this case the code emulated should not perform worse).
In the beginning the PoC was much more hacky and it was more similar to the idea you have probably in mind. But a big part of the "hack" was removing code, in particular the entire support for PV and the verification code. Obviously that hacks could not be accepted so instead I decided to change the code in a more incremental way not removing things but instead changing to make it easier to use the future hypercall. So I was able to test all cases (like PV) incrementally and keep it working.
You are however the second person (after Andrew) to ask this, so a big comment (probably in the commit message) is due.
Quoted 3 lines
Cheers,
Frediano
PATCH v6 11/16 PoC: libs/guest: use foreign copy during migration
In reply to Frediano ZiglioFrom: Edwin Török <[email protected]>
ministat confirms the improvement:
```
x baseline
+ foreigncopy
N Min Max Median Avg Stddev
x 20 1.1306997 1.1447931 1.1356569 1.1365742 0.003242175
+ 20 0.4311504 0.44180303 0.43616705 0.43600089 0.0031094689
Difference at 95.0% confidence
-0.700573 +/- 0.00203311
-61.639% +/- 0.133355%
(Student's t, pooled s = 0.00317652)
```The tests pass too, which means that it has correctly migrated all guest memory.
Frediano: This PoC was adapted to be included in a final series.
--- tools/libs/guest/xg_sr_common.h | 1 + tools/libs/guest/xg_sr_restore.c | 43 +++-------------- tools/libs/guest/xg_sr_save.c | 82 +++++++++----------------------- 3 files changed, 31 insertions(+), 95 deletions(-)
tools/libs/guest/xg_sr_common.h +1 -0
@@ -256,6 +256,7 @@ struct xc_sr_contextunsigned long nr_deferred_pages;xc_hypercall_buffer_t dirty_bitmap_hbuf;struct xc_sr_context_save_buffers *buffers; +void *dest_buf; } save; struct /* Restore data. */
tools/libs/guest/xg_sr_restore.c +7 -36
71 lines not on this page · raw message
tools/libs/guest/xg_sr_save.c +23 -59
123 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v6 11/16] PoC: libs/guest: use foreign copy during migration
A note about the subject, a "PoC" or Proof-of-concept to me isn't a patch that can be accepted, especialy if is a patch to an existing library.
Quoted 4 lines · Frediano Ziglio
From: Edwin Török <[email protected]>
ministat confirms the improvement:
Quoted 11 lines
```
x baseline
+ foreigncopy
N Min Max Median Avg Stddev
x 20 1.1306997 1.1447931 1.1356569 1.1365742 0.003242175
+ 20 0.4311504 0.44180303 0.43616705 0.43600089 0.0031094689
Difference at 95.0% confidence
-0.700573 +/- 0.00203311
-61.639% +/- 0.133355%
(Student's t, pooled s = 0.00317652)
```There's been some comment about this stat in previous version of the series, and the description is still the same. Could you describe how the stat have been generated, and what the number mean?
Also, what's the different between "baseline" and "foreigncopy". I've only had a glimpse at this patch, and it just looks like the code have been moved to a different part of the library, with somehow less lines of code.
Quoted 3 lines
The tests pass too, which means that it has correctly migrated all guest memory.
Which tests?
Thanks,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH v6 11/16] PoC: libs/guest: use foreign copy during migration
Quoted 6 lines · Anthony PERARD
A note about the subject, a "PoC" or Proof-of-concept to me isn't a patch that can be accepted, especialy if is a patch to an existing library.
Yes, mainly it should be merged with the final "finalize PoC". Still nice for review at the moment.
Quoted 17 lines · Frediano Ziglio
Quoted 4 lines
From: Edwin Török <[email protected]>
ministat confirms the improvement:
Quoted 11 lines
``` x baseline + foreigncopy N Min Max Median Avg Stddev x 20 1.1306997 1.1447931 1.1356569 1.1365742 0.003242175 + 20 0.4311504 0.44180303 0.43616705 0.43600089 0.0031094689 Difference at 95.0% confidence -0.700573 +/- 0.00203311 -61.639% +/- 0.133355% (Student's t, pooled s = 0.00317652) ```
Quoted 4 lines
There's been some comment about this stat in previous version of the series, and the description is still the same. Could you describe how the stat have been generated, and what the number mean?
We managed to ask the author but got not much clue. Mainly timing. Probably better to do new statistics.
Quoted 5 lines
Also, what's the different between "baseline" and "foreigncopy". I've only had a glimpse at this patch, and it just looks like the code have been moved to a different part of the library, with somehow less lines of code.
"baseline" I suppose without these changes, "foreigncopy" will the changes.
Quoted 6 lines
Quoted 3 lines
The tests pass too, which means that it has correctly migrated all guest memory.
Which tests?
Different migrations with multiple OSes and configuration (for instance PV and not PV, these from me).
Quoted 3 lines
Thanks,
Frediano
PATCH v6 12/16 xen: implement new foreign copy hypercall
In reply to Frediano ZiglioAdd a sub hypercall to __HYPERVISOR_memory_op to allow to read/write memory from/to a foreign domain.
Extending MMUEXT_COPY_PAGE seems better on first sight but considering that MMUEXT is meant for PV only and trying to change that sub-op this solution is better.
--
Changes since v4:
- Fix typo in comment.
Changes since v5:
- update xen_foreigncopy structure comments;
- move check for no frames after checking the domain;
- use mnemonic instead of 1U;
- fix page type checks;
- do not overwrite error copying back structure;
- latch MFN value;
- improved commit message.
--- xen/common/memory.c | 145 ++++++++++++++++++++++++++++++++++++ xen/include/public/memory.h | 44 ++++++++++- 2 files changed, 188 insertions(+), 1 deletion(-)
xen/common/memory.c +145 -0
@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; + +if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d); +if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + } + +/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out; + +do { +/* +* Arbitrary size. Not too much stack space, and a reasonable stride +* for continuation checks. +*/ +xen_pfn_t gfn_list[32 ]; +unsigned int todo =MIN (ARRAY_SIZE (gfn_list), copy.nr_frames); + + rc = -EFAULT ; +if (copy_from_guest (gfn_list, copy.frame_list, todo) ) + goto out; + +for (unsigned int i =0 ; i < todo; i++ ) + { +struct page_info *foreign_page; +mfn_t foreign_mfn; +void *foreign; +p2m_type_t p2mt; + const unsigned long valid_mask = +#ifdef CONFIG_X86 +p2m_to_mask (p2m_ram_rw) |p2m_to_mask (p2m_ram_logdirty); +#else +p2m_to_mask (p2m_ram_rw); +#endif + + foreign_page =get_page_from_gfn (d, gfn_list[i], &p2mt,P2M_ALLOC ); + +if (unlikely (!(p2m_to_mask (p2mt) & valid_mask)) && foreign_page ) + { +put_page (foreign_page); + foreign_page =NULL ; + } +if (unlikely (!foreign_page) ) + { +gdprintk (XENLOG_WARNING , +"Error accessing foreign gfn %" PRI_gfn"\n" , + gfn_list[i]); + rc = -EINVAL ; + copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + } + + foreign_mfn =page_to_mfn (foreign_page); + +/* A page is dirtied when it's being copied to. */ +if ( direction == XENMEM_foreigncopy_to ) +paging_mark_dirty (d, foreign_mfn); + + foreign =map_domain_page (foreign_mfn); +if ( direction == XENMEM_foreigncopy_from ) + rc =copy_to_guest (copy.buffer, foreign,PAGE_SIZE ); +else + rc =copy_from_guest (foreign, copy.buffer,PAGE_SIZE ); +unmap_domain_page (foreign); +put_page (foreign_page); + +if (unlikely (rc) ) + { +gdprintk (XENLOG_WARNING , +"Error %d copying gfn %" PRI_gfn"\n" , + -rc, gfn_list[i]); + copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + } + +guest_handle_add_offset (copy.buffer,PAGE_SIZE ); + } + + copy.nr_frames -= todo; +guest_handle_add_offset (copy.frame_list, todo); + +if ( copy.nr_frames &&hypercall_preempt_check () ) + { + rc =hypercall_create_continuation ( + __HYPERVISOR_memory_op,"lh" , XENMEM_foreigncopy, arg); + goto out; + } + }while ( copy.nr_frames ); + + rc =0 ; + + out: +rcu_unlock_domain (d); + +/* +* Update in all cases, it allows the caller to know how many +* frames were successfully copied and the continuation to +* continue correctly. +*/ +if (__copy_to_guest (arg, ©,1 ) && rc >=0 ) + rc = -EFAULT ; + +return rc; +} +long do_memory_op (unsigned long cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) {struct domain *d, *curr_d = current->domain;
@@ -2012,6 +2145,18 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ; +case XENMEM_foreigncopy: +/* +* Instead of using "start_extent" we update the structure back, +* we update it back in anyway to tell caller were the copy +* stopped. +*/ +if (unlikely (start_extent) ) +return -EINVAL ; + + rc =mem_foreigncopy (guest_handle_cast (arg, xen_foreigncopy_t)); +break ; +default : rc =arch_memory_op (cmd, arg);break ;
xen/include/public/memory.h +43 -1
50 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 16 lines · Frediano Ziglio
xen/common/memory.c +10
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction;
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 10 lines
xen/common/memory.c +10
-0+ +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; + +if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);
Iirc I did ask before why this isn't ..._by_any_id().
xen/common/memory.c static int acquire_resource( · line 1575
Quoted 8 lines
xen/common/memory.c +8
-0+if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + }
Any reason this cannot also be "goto out"? The more that now that you have moved this past the domid validity check, imo it should further move to ...
xen/common/memory.c static int acquire_resource( · line 1582
Quoted 6 lines
xen/common/memory.c +6
-0+/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out;
... below here. Perhaps simply as
if ( rc || !copy.nr_frames )
goto out;Quoted 14 lines
xen/common/memory.c +14
-0+ do { + /* + * Arbitrary size. Not too much stack space, and a reasonable stride + * for continuation checks. + */ + xen_pfn_t gfn_list[32]; + unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames); + + rc = -EFAULT; + if ( copy_from_guest(gfn_list, copy.frame_list, todo) ) + goto out; + + for ( unsigned int i = 0; i < todo; i++ ) + {
xen/common/memory.c static int acquire_resource( · line 1607
Quoted 10 lines
xen/common/memory.c +10
-0+struct page_info *foreign_page; +mfn_t foreign_mfn; +void *foreign; +p2m_type_t p2mt; + const unsigned long valid_mask = +#ifdef CONFIG_X86 +p2m_to_mask (p2m_ram_rw) |p2m_to_mask (p2m_ram_logdirty); +#else +p2m_to_mask (p2m_ram_rw); +#endif
The set of permitted types didn't change, yet a justification for the resulting limitation also didn't appear.
Quoted 19 lines
xen/common/memory.c +19
-0+ foreign_page =get_page_from_gfn (d, gfn_list[i], &p2mt,P2M_ALLOC ); + +if (unlikely (!(p2m_to_mask (p2mt) & valid_mask)) && foreign_page ) + { +put_page (foreign_page); + foreign_page =NULL ; + } +if (unlikely (!foreign_page) ) + { +gdprintk (XENLOG_WARNING , +"Error accessing foreign gfn %" PRI_gfn"\n" , + gfn_list[i]); + rc = -EINVAL ; + copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + } + + foreign_mfn =page_to_mfn (foreign_page);
xen/common/memory.c static int acquire_resource( · line 1637
Quoted 10 lines
xen/common/memory.c +10
-0+ +/* A page is dirtied when it's being copied to. */ +if ( direction == XENMEM_foreigncopy_to ) +paging_mark_dirty (d, foreign_mfn); + + foreign =map_domain_page (foreign_mfn); +if ( direction == XENMEM_foreigncopy_from ) + rc =copy_to_guest (copy.buffer, foreign,PAGE_SIZE ); +else + rc =copy_from_guest (foreign, copy.buffer,PAGE_SIZE );
You cannot validly write to the page without holding a PGT_writable ref. Else you might overwrite a page table or a descriptor table in a PV guest.
Once again - can you please make sure you have addressed earlier review comments, before sending a new version? I did point this out before.
xen/common/memory.c static int acquire_resource( · line 1645
Quoted 8 lines
xen/common/memory.c +8
-0+unmap_domain_page (foreign); +put_page (foreign_page); + +if (unlikely (rc) ) + { +gdprintk (XENLOG_WARNING , +"Error %d copying gfn %" PRI_gfn"\n" , + -rc, gfn_list[i]);
Why "-rc"? (See other log messages including error codes.)
xen/common/memory.c static int acquire_resource( · line 1655
Quoted 10 lines
xen/common/memory.c +10
-0+ copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + } + +guest_handle_add_offset (copy.buffer,PAGE_SIZE ); + } + + copy.nr_frames -= todo; +guest_handle_add_offset (copy.frame_list, todo);
Don't you need to also update copy.buffer?
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2155
Quoted 12 lines
xen/common/memory.c +8
-0@@ -2012,6 +2145,18 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ; +case XENMEM_foreigncopy: +/* +* Instead of using "start_extent" we update the structure back, +* we update it back in anyway to tell caller were the copy +* stopped. +*/ +if (unlikely (start_extent) ) +return -EINVAL ;
As before - please be precise with comments like this. We update it back also when encoding a continuation. Perhaps instead "..., to indicate the point of failure to the caller as well as to encode continuations without being constrained by MEMOP_EXTENT_SHIFT".
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746
Quoted 11 lines
xen/include/public/memory.h +4
-1@@ -740,7 +740,49 @@ struct xen_vnuma_topology_info {typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t ;DEFINE_XEN_GUEST_HANDLE (xen_vnuma_topology_info_t); -/* Next available subop number is 29 */ +/* +* Copy memory from/to a given domain. +* As this call requires target access and guest with target access won't be +* compat guests supported for compat guests this is not implemented.
As before - I question this. You simply can't know. (I'm also struggling with wording / grammar.)
Quoted 15 lines
xen/include/public/memory.h +15
-0+*/ +#define XENMEM_foreigncopy 29 +struct xen_foreigncopy { +/* IN - The domain whose memory is to be copied. */ +domid_t domid; + +/* IN - Flags. */ +#define XENMEM_foreigncopy_from 0 +#define XENMEM_foreigncopy_to 1 +#define XENMEM_foreigncopy_direction 1 +uint16_t flags; + +/* +* IN/OUT +*
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 771
Quoted 10 lines
xen/include/public/memory.h +10
-0+* As an IN parameter number of frames of the domain to be copied. +* On output on error updated number of frames left. +*/ +uint32_t nr_frames; + +/* +* IN/OUT +* +* Frames to be copied. +* On output on error updated to point to first frame unhandled.
Is "on error" really correct / meaningful? The field can be updated at any intermediate point, when a continuation is scheduled. Perhaps:
* On output:
* - on error updated to point to first frame which couldn't be handled,
* - on success undefined.
Along these lines for nr_frames then as well (if needed at all, seeing that it could as well be undefined in both cases, as the information is redundant with the frame_list update).
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 778
Quoted 7 lines
xen/include/public/memory.h +7
-0+*/ +XEN_GUEST_HANDLE (xen_pfn_t) frame_list; + +/* +* IN/OUT +* +* Userspace buffer to read/write from.
s/Userspace/Guest/ ?
Also still no mention of when / how this field is updated.
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 781
Quoted 3 lines
xen/include/public/memory.h +3
-0+*/ +XEN_GUEST_HANDLE (uint8) buffer; +};
What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd domid_t one then.
Jan
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Frediano Ziglio
xen/common/memory.c +14
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; +
Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 6 lines
xen/common/memory.c +6
-0+if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);Iirc I did ask before why this isn't ..._by_any_id().
I probably was confused by the question about MMUEXT and the 2 domains. There are different similar hypercalls (like the mentioned MMUEXT but also hypercalls to map foreign domain memory) that have this check (not the same domain). Any domain has, obviously, access to its own memory, so it should not have to use hypercall to access its own memory. If it does it looks like a mistake causing performance issues or an attempt to circumvent security; in either case you would like to avoid it.
Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1575
Quoted 8 lines
xen/common/memory.c +8
-0+if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + }
Quoted 3 lines
Any reason this cannot also be "goto out"? The more that now that you have moved this past the domid validity check, imo it should further move to ...
The only reason was style and to avoid a memory copy, but it's not a hot case so I'll change to "goto out" (no strong about it).
Quoted 7 lines
xen/common/memory.c static int acquire_resource( · line 1582
Quoted 6 lines
xen/common/memory.c +6
-0+/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out;
Quoted 5 lines
... below here. Perhaps simply as
if ( rc || !copy.nr_frames ) goto out;
I think this would be confusing with the above "Check we are allowed to map and access these foreign pages" comment.
Are you okay with just the change above to "goto out" ?
Also moving here would potentially change the result and do a useless check.
Quoted 19 lines
Quoted 19 lines
xen/common/memory.c +19
-0+ do { + /* + * Arbitrary size. Not too much stack space, and a reasonable stride + * for continuation checks. + */ + xen_pfn_t gfn_list[32]; + unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames); + + rc = -EFAULT; + if ( copy_from_guest(gfn_list, copy.frame_list, todo) ) + goto out; + + for ( unsigned int i = 0; i < todo; i++ ) + { + struct page_info *foreign_page; + mfn_t foreign_mfn; + void *foreign; + p2m_type_t p2mt; + const unsigned long valid_mask =
Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1607
Quoted 5 lines
xen/common/memory.c +5
-0+#ifdef CONFIG_X86 +p2m_to_mask (p2m_ram_rw) |p2m_to_mask (p2m_ram_logdirty); +#else +p2m_to_mask (p2m_ram_rw); +#endif The set of permitted types didn't change, yet a justification for the resulting limitation also didn't appear.
Yes, that's missing, indeed.
Should the set of types be different for reading and writing? For instance do not allow writing to read-only memory?
Given that it looks like different architectures have different meanings and definitions for these constants, should it not be better to define some new constants for this specific usage? For instance P2M_READ_TYPES and P2M_WRITE_TYPES?
Quoted 24 lines
Quoted 24 lines
xen/common/memory.c +24
-0+ foreign_page =get_page_from_gfn (d, gfn_list[i], &p2mt,P2M_ALLOC ); + +if (unlikely (!(p2m_to_mask (p2mt) & valid_mask)) && foreign_page ) + { +put_page (foreign_page); + foreign_page =NULL ; + } +if (unlikely (!foreign_page) ) + { +gdprintk (XENLOG_WARNING , +"Error accessing foreign gfn %" PRI_gfn"\n" , + gfn_list[i]); + rc = -EINVAL ; + copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + } + + foreign_mfn =page_to_mfn (foreign_page); + +/* A page is dirtied when it's being copied to. */ +if ( direction == XENMEM_foreigncopy_to ) +paging_mark_dirty (d, foreign_mfn); +
Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1637
Quoted 5 lines
xen/common/memory.c +5
-0+ foreign =map_domain_page (foreign_mfn); +if ( direction == XENMEM_foreigncopy_from ) + rc =copy_to_guest (copy.buffer, foreign,PAGE_SIZE ); +else + rc =copy_from_guest (foreign, copy.buffer,PAGE_SIZE );You cannot validly write to the page without holding a
PGT_writableref. Else you might overwrite a page table or a descriptor table in a PV guest.
Given that this code was "inspired" by other hypercalls I'll also check the other code.
Quoted 3 lines
Once again - can you please make sure you have addressed earlier review comments, before sending a new version? I did point this out before.
Apparently not.
Quoted 11 lines
xen/common/memory.c static int acquire_resource( · line 1645
Quoted 8 lines
xen/common/memory.c +8
-0+unmap_domain_page (foreign); +put_page (foreign_page); + +if (unlikely (rc) ) + { +gdprintk (XENLOG_WARNING , +"Error %d copying gfn %" PRI_gfn"\n" , + -rc, gfn_list[i]);Why "-rc"? (See other log messages including error codes.)
Because the errors are positive but for ABI we return them as negative. But I suppose if for other messages we use the negated value this should be just "rc".
I'll change.
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +4
-0+ copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + }
Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1655
Quoted 6 lines
xen/common/memory.c +6
-0+ +guest_handle_add_offset (copy.buffer,PAGE_SIZE ); + } + + copy.nr_frames -= todo; +guest_handle_add_offset (copy.frame_list, todo);Don't you need to also update copy.buffer?
It's updated some lines above inside the loop.
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +0
-0@@ -2012,6 +2145,18 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ;
Quoted 14 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2155
Quoted 8 lines
xen/common/memory.c +8
-0+case XENMEM_foreigncopy: +/* +* Instead of using "start_extent" we update the structure back, +* we update it back in anyway to tell caller were the copy +* stopped. +*/ +if (unlikely (start_extent) ) +return -EINVAL ;As before - please be precise with comments like this. We update it back also when encoding a continuation. Perhaps instead "..., to indicate the point of failure to the caller as well as to encode continuations without being constrained by
MEMOP_EXTENT_SHIFT".
What about (trying to include your suggestion, to be fixed for line length):
/*
* Instead of using "start_extent" for the continuation, we
update the structure back,
* we update the xen_foreigncopy structure back, so we are not
constrained
* by MEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped. */
Quoted 6 lines
Quoted 6 lines
xen/include/public/memory.h +0
-0@@ -740,7 +740,49 @@ struct xen_vnuma_topology_info {typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t ;DEFINE_XEN_GUEST_HANDLE (xen_vnuma_topology_info_t);
Quoted 9 lines
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746
Quoted 5 lines
xen/include/public/memory.h +4
-1-/* Next available subop number is 29 */ +/* +* Copy memory from/to a given domain. +* As this call requires target access and guest with target access won't be +* compat guests supported for compat guests this is not implemented. As before - I question this. You simply can't know. (I'm also struggling with wording / grammar.)
I was trying to code the compatibility layer. Is there a way to have 64 bit PFN even for compatibility guests instead of having to limit and convert PFN numbers?
Quoted 45 lines
Quoted 15 lines
xen/include/public/memory.h +15
-0+*/ +#define XENMEM_foreigncopy 29 +struct xen_foreigncopy { +/* IN - The domain whose memory is to be copied. */ +domid_t domid; + +/* IN - Flags. */ +#define XENMEM_foreigncopy_from 0 +#define XENMEM_foreigncopy_to 1 +#define XENMEM_foreigncopy_direction 1 +uint16_t flags; + +/* +* IN/OUT +* xen/include/public/memory.h struct xen_vnuma_topology_info { · line 771
Quoted 10 lines
xen/include/public/memory.h +10
-0+* As an IN parameter number of frames of the domain to be copied. +* On output on error updated number of frames left. +*/ +uint32_t nr_frames; + +/* +* IN/OUT +* +* Frames to be copied. +* On output on error updated to point to first frame unhandled. Is "on error" really correct / meaningful? The field can be updated at any intermediate point, when a continuation is scheduled. Perhaps:
* On output:
* - on error updated to point to first frame which couldn't be handled,
* - on success undefined.Along these lines for
nr_framesthen as well (if needed at all, seeing that it could as well be undefined in both cases, as the information is redundant with theframe_listupdate).xen/include/public/memory.h struct xen_vnuma_topology_info { · line 778
Quoted 7 lines
xen/include/public/memory.h +7
-0+*/ +XEN_GUEST_HANDLE (xen_pfn_t) frame_list; + +/* +* IN/OUT +* +* Userspace buffer to read/write from.
Quoted 4 lines
s/Userspace/Guest/ ?
Also still no mention of when / how this field is updated.
What about:
/*
* Copy memory from/to a given domain.
*/
#define XENMEM_foreigncopy 29
struct xen_foreigncopy {
/* IN - The domain whose memory is to be copied. */
domid_t domid;
/* IN - Flags. */
#define XENMEM_foreigncopy_from 0
#define XENMEM_foreigncopy_to 1
#define XENMEM_foreigncopy_direction 1
uint16_t flags;
/*
* IN/OUT
*
* As an IN parameter number of frames of the domain to be copied.
* On output updated number of frames left (0 if success).
*/
uint32_t nr_frames;
/*
* IN/OUT
*
* Frames to be copied.
* On output updated to point to the first frame unhandled.
*/
XEN_GUEST_HANDLE(xen_pfn_t) frame_list;
/*
* IN/OUT
*
* Guest buffer to read/write from.
* On output updated to point to the first frame unhandled.
*/
XEN_GUEST_HANDLE(uint8) buffer;
};typedef struct xen_foreigncopy xen_foreigncopy_t;DEFINE_XEN_GUEST_HANDLE(xen_foreigncopy_t);
Quoted 8 lines
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 781
Quoted 3 lines
xen/include/public/memory.h +3
-0+*/ +XEN_GUEST_HANDLE (uint8) buffer; +};What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd
domid_tone then.
This was addressed in
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00567.html
and in minor way by
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00847.html.
It was considered but more complicated and worse from a performance perspective.
Quoted 1 line
Jan
Frediano
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 31 lines · Frediano Ziglio
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Frediano Ziglio
xen/common/memory.c +14
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; +Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 6 lines
xen/common/memory.c +6
-0+if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);Iirc I did ask before why this isn't ..._by_any_id().
Quoted 8 lines
I probably was confused by the question about MMUEXT and the 2 domains. There are different similar hypercalls (like the mentioned MMUEXT but also hypercalls to map foreign domain memory) that have this check (not the same domain). Any domain has, obviously, access to its own memory, so it should not have to use hypercall to access its own memory. If it does it looks like a mistake causing performance issues or an attempt to circumvent security; in either case you would like to avoid it.
No. Self-grants are possible as well, for example, and for a good reason. Allowing normally-remote operations on oneself helps with testing, for example. It may also help avoid needing to special-case "self" in code which needs to cover both cases.
Quoted 24 lines
Quoted 11 lines
xen/common/memory.c static int acquire_resource( · line 1575
Quoted 8 lines
xen/common/memory.c +8
-0+if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + }Any reason this cannot also be "goto out"? The more that now that you have moved this past the domid validity check, imo it should further move to ...
The only reason was style and to avoid a memory copy, but it's not a hot case so I'll change to "goto out" (no strong about it).
Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1582
Quoted 6 lines
xen/common/memory.c +6
-0+/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out;... below here. Perhaps simply as
Quoted 7 lines
Quoted 3 lines
if ( rc || !copy.nr_frames ) goto out;I think this would be confusing with the above "Check we are allowed to map and access these foreign pages" comment.
Are you okay with just the change above to "goto out" ?
I do want the order adjusted as indicated. I won't insist on (but I would prefer) folding both if()-s.
Quoted 1 line
Also moving here would potentially change the result and do a useless check.
Affecting the result is the goal of the re-ordering.
Quoted 29 lines
Quoted 19 lines
Quoted 19 lines
xen/common/memory.c +19
-0+ do { + /* + * Arbitrary size. Not too much stack space, and a reasonable stride + * for continuation checks. + */ + xen_pfn_t gfn_list[32]; + unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames); + + rc = -EFAULT; + if ( copy_from_guest(gfn_list, copy.frame_list, todo) ) + goto out; + + for ( unsigned int i = 0; i < todo; i++ ) + { + struct page_info *foreign_page; + mfn_t foreign_mfn; + void *foreign; + p2m_type_t p2mt; + const unsigned long valid_mask =Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1607
Quoted 5 lines
xen/common/memory.c +5
-0+#ifdef CONFIG_X86 +p2m_to_mask (p2m_ram_rw) |p2m_to_mask (p2m_ram_logdirty); +#else +p2m_to_mask (p2m_ram_rw); +#endif The set of permitted types didn't change, yet a justification for the resulting limitation also didn't appear.
Quoted 3 lines
Yes, that's missing, indeed.
Should the set of types be different for reading and writing? For instance do not allow writing to read-only memory?
Of course.
Quoted 4 lines
Given that it looks like different architectures have different meanings and definitions for these constants, should it not be better to define some new constants for this specific usage? For instance
P2M_READ_TYPESandP2M_WRITE_TYPES?
Perhaps, yes. The suggested names look overly generic to me, though.
Quoted 34 lines
Quoted 24 lines
Quoted 24 lines
xen/common/memory.c +24
-0+ foreign_page =get_page_from_gfn (d, gfn_list[i], &p2mt,P2M_ALLOC ); + +if (unlikely (!(p2m_to_mask (p2mt) & valid_mask)) && foreign_page ) + { +put_page (foreign_page); + foreign_page =NULL ; + } +if (unlikely (!foreign_page) ) + { +gdprintk (XENLOG_WARNING , +"Error accessing foreign gfn %" PRI_gfn"\n" , + gfn_list[i]); + rc = -EINVAL ; + copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + } + + foreign_mfn =page_to_mfn (foreign_page); + +/* A page is dirtied when it's being copied to. */ +if ( direction == XENMEM_foreigncopy_to ) +paging_mark_dirty (d, foreign_mfn); +Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1637
Quoted 5 lines
xen/common/memory.c +5
-0+ foreign =map_domain_page (foreign_mfn); +if ( direction == XENMEM_foreigncopy_from ) + rc =copy_to_guest (copy.buffer, foreign,PAGE_SIZE ); +else + rc =copy_from_guest (foreign, copy.buffer,PAGE_SIZE );You cannot validly write to the page without holding a
PGT_writableref. Else you might overwrite a page table or a descriptor table in a PV guest.
Quoted 7 lines
Given that this code was "inspired" by other hypercalls I'll also check the other code.
Quoted 2 lines
Once again - can you please make sure you have addressed earlier review comments, before sending a new version? I did point this out before.
Apparently not.
https://lists.xen.org/archives/html/xen-devel/2026-06/msg00850.html
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +4
-0+ copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + }Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1655
Quoted 6 lines
xen/common/memory.c +6
-0+ +guest_handle_add_offset (copy.buffer,PAGE_SIZE ); + } + + copy.nr_frames -= todo; +guest_handle_add_offset (copy.frame_list, todo);Don't you need to also update copy.buffer?
Quoted 1 line
It's updated some lines above inside the loop.
Oh, sorry. Yet then - not doing all updates together is, as you can see, potentially confusing.
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +0
-0@@ -2012,6 +2145,18 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ;Quoted 14 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2155
Quoted 8 lines
xen/common/memory.c +8
-0+case XENMEM_foreigncopy: +/* +* Instead of using "start_extent" we update the structure back, +* we update it back in anyway to tell caller were the copy +* stopped. +*/ +if (unlikely (start_extent) ) +return -EINVAL ;As before - please be precise with comments like this. We update it back also when encoding a continuation. Perhaps instead "..., to indicate the point of failure to the caller as well as to encode continuations without being constrained by
MEMOP_EXTENT_SHIFT".What about (trying to include your suggestion, to be fixed for line length):
Quoted 8 lines
/*
* Instead of using "start_extent" for the continuation, we
update the structure back,
* we update thexen_foreigncopystructure back, so we are not
constrained
* byMEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped. */
One of the things I take issue with (because it's hard to read that way, at least for me) is the repeated use of "update ... back", effectively saying the same things twice. The last sentence also wants disambiguating towards the "stopped" possibly being a non-error situation as well.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
xen/include/public/memory.h +0
-0@@ -740,7 +740,49 @@ struct xen_vnuma_topology_info {typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t ;DEFINE_XEN_GUEST_HANDLE (xen_vnuma_topology_info_t);Quoted 8 lines
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746
Quoted 5 lines
xen/include/public/memory.h +4
-1-/* Next available subop number is 29 */ +/* +* Copy memory from/to a given domain. +* As this call requires target access and guest with target access won't be +* compat guests supported for compat guests this is not implemented. As before - I question this. You simply can't know. (I'm also struggling with wording / grammar.)
Quoted 3 lines
I was trying to code the compatibility layer. Is there a way to have 64 bit PFN even for compatibility guests instead of having to limit and convert PFN numbers?
compat_pfn_t is a typedef of unsigned int (since a 32-bit guest seeing "typedef unsigned long xen_pfn_t;" results in xen_pfn_t being a 32-bit quantity for it), so 32-bit guests can only supply 32-bit frame numbers. There's also no value in trying to be clever and using uint64_t instead for the frame_list handle, as 32-bit guests won't ever own pages with MFNs wider than 32 bits.
Quoted 71 lines
Quoted 45 lines
Quoted 15 lines
xen/include/public/memory.h +15
-0+*/ +#define XENMEM_foreigncopy 29 +struct xen_foreigncopy { +/* IN - The domain whose memory is to be copied. */ +domid_t domid; + +/* IN - Flags. */ +#define XENMEM_foreigncopy_from 0 +#define XENMEM_foreigncopy_to 1 +#define XENMEM_foreigncopy_direction 1 +uint16_t flags; + +/* +* IN/OUT +* xen/include/public/memory.h struct xen_vnuma_topology_info { · line 771
Quoted 10 lines
xen/include/public/memory.h +10
-0+* As an IN parameter number of frames of the domain to be copied. +* On output on error updated number of frames left. +*/ +uint32_t nr_frames; + +/* +* IN/OUT +* +* Frames to be copied. +* On output on error updated to point to first frame unhandled. Is "on error" really correct / meaningful? The field can be updated at any intermediate point, when a continuation is scheduled. Perhaps:
* On output:
* - on error updated to point to first frame which couldn't be handled,
* - on success undefined.Along these lines for
nr_framesthen as well (if needed at all, seeing that it could as well be undefined in both cases, as the information is redundant with theframe_listupdate).xen/include/public/memory.h struct xen_vnuma_topology_info { · line 778
Quoted 7 lines
xen/include/public/memory.h +7
-0+*/ +XEN_GUEST_HANDLE (xen_pfn_t) frame_list; + +/* +* IN/OUT +* +* Userspace buffer to read/write from. Quoted 4 lines
s/Userspace/Guest/ ?
Also still no mention of when / how this field is updated.
What about:
/*
* Copy memory from/to a given domain.
*/
#defineXENMEM_foreigncopy29struct xen_foreigncopy { /* IN - The domain whose memory is to be copied. */ domid_t domid; /* IN - Flags. */ #define XENMEM_foreigncopy_from 0 #define XENMEM_foreigncopy_to 1 #define XENMEM_foreigncopy_direction 1 uint16_t flags; /* * IN/OUT * * As an IN parameter number of frames of the domain to be copied. * On output updated number of frames left (0 if success).
Quoted 8 lines
*/
uint32_t nr_frames;
/*
* IN/OUT
*
* Frames to be copied.
* On output updated to point to the first frame unhandled.There may be no such frame, so at the very least add "..., if any"?
Quoted 8 lines
*/
XEN_GUEST_HANDLE(xen_pfn_t) frame_list;
/*
* IN/OUT
*
* Guest buffer to read/write from.
* On output updated to point to the first frame unhandled.There's no frame here, as long as you don't switch to using two frame lists (for source and destination).
Quoted 10 lines
*/ XEN_GUEST_HANDLE(uint8) buffer; };typedef struct
xen_foreigncopyxen_foreigncopy_t;DEFINE_XEN_GUEST_HANDLE(xen_foreigncopy_t);Quoted 4 lines
Quoted 3 lines
xen/include/public/memory.h +3
-0+*/ +XEN_GUEST_HANDLE (uint8) buffer; +};
Quoted 7 lines
Quoted 4 lines
What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd
domid_tone then.This was addressed in
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00567.html
Well, yes, but not in a satisfactory way. Back channels tell me that you actually got the same feedback already on internal review. Which makes it all the more puzzling that you insist on doing it differently. Multiple maintainers asking for the same thing may be an indication of something.
Quoted 3 lines
and in minor way by
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00847.html.
It was considered but more complicated and worse from a performance perspective.
Okay, performance-wise worse would of course be relevant. But that would need supporting by numbers (for both PV and PVH Dom0, as the latter incurs extra overhead for virtual-address-based hypercall buffer operands).
Jan
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 42 lines · Jan Beulich
Quoted 31 lines · Frediano Ziglio
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Frediano Ziglio
xen/common/memory.c +14
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; +Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 6 lines
xen/common/memory.c +6
-0+if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);Iirc I did ask before why this isn't ..._by_any_id().
Quoted 8 lines
I probably was confused by the question about MMUEXT and the 2 domains. There are different similar hypercalls (like the mentioned MMUEXT but also hypercalls to map foreign domain memory) that have this check (not the same domain). Any domain has, obviously, access to its own memory, so it should not have to use hypercall to access its own memory. If it does it looks like a mistake causing performance issues or an attempt to circumvent security; in either case you would like to avoid it.
Quoted 5 lines
No. Self-grants are possible as well, for example, and for a good reason. Allowing normally-remote operations on oneself helps with testing, for example. It may also help avoid needing to special-case "self" in code which needs to cover both cases.
But this is not a grant, it's a copy.
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
xen/common/memory.c static int acquire_resource( · line 1575
Quoted 8 lines
xen/common/memory.c +8
-0+if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + }Any reason this cannot also be "goto out"? The more that now that you have moved this past the domid validity check, imo it should further move to ...
The only reason was style and to avoid a memory copy, but it's not a hot case so I'll change to "goto out" (no strong about it).
Quoted 12 lines
xen/common/memory.c static int acquire_resource( · line 1582
Quoted 6 lines
xen/common/memory.c +6
-0+/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out;... below here. Perhaps simply as
if ( rc || !copy.nr_frames ) goto out;
Quoted 7 lines
Quoted 3 lines
I think this would be confusing with the above "Check we are allowed to map and access these foreign pages" comment.
Are you okay with just the change above to "goto out" ?I do want the order adjusted as indicated. I won't insist on (but I would prefer) folding both if()-s.
What about
/*
* Check we are allowed to map and access these foreign pages.
*/
rc = xsm_map_gmfn_foreign(XSM_TARGET, currd, d);
if ( rc )
goto out;
while ( copy.nr_frames )
{
/*
* Arbitrary size. Not too much stack space, and a reasonable stride
* for continuation checks.
*/Quoted 4 lines
Quoted 1 line
Also moving here would potentially change the result and do a useless check.
Affecting the result is the goal of the re-ordering.
Changed.
Quoted 33 lines
Quoted 29 lines
Quoted 19 lines
Quoted 19 lines
xen/common/memory.c +19
-0+ do { + /* + * Arbitrary size. Not too much stack space, and a reasonable stride + * for continuation checks. + */ + xen_pfn_t gfn_list[32]; + unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames); + + rc = -EFAULT; + if ( copy_from_guest(gfn_list, copy.frame_list, todo) ) + goto out; + + for ( unsigned int i = 0; i < todo; i++ ) + { + struct page_info *foreign_page; + mfn_t foreign_mfn; + void *foreign; + p2m_type_t p2mt; + const unsigned long valid_mask =Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1607
Quoted 5 lines
xen/common/memory.c +5
-0+#ifdef CONFIG_X86 +p2m_to_mask (p2m_ram_rw) |p2m_to_mask (p2m_ram_logdirty); +#else +p2m_to_mask (p2m_ram_rw); +#endif The set of permitted types didn't change, yet a justification for the resulting limitation also didn't appear.
Quoted 3 lines
Yes, that's missing, indeed.
Should the set of types be different for reading and writing? For instance do not allow writing to read-only memory?
Quoted 9 lines
Of course.
Quoted 4 lines
Given that it looks like different architectures have different meanings and definitions for these constants, should it not be better to define some new constants for this specific usage? For instance
P2M_READ_TYPESandP2M_WRITE_TYPES?Perhaps, yes. The suggested names look overly generic to me, though.
I suppose P2M_READABLE_TYPES and P2M_WRITABLE_TYPES are more correct but still too generic.P2M_EXPORTABLE_TYPES and P2M_IMPORTABLE_TYPES ?
Quoted 37 lines
Quoted 37 lines
Quoted 24 lines
Quoted 24 lines
xen/common/memory.c +24
-0+ foreign_page =get_page_from_gfn (d, gfn_list[i], &p2mt,P2M_ALLOC ); + +if (unlikely (!(p2m_to_mask (p2mt) & valid_mask)) && foreign_page ) + { +put_page (foreign_page); + foreign_page =NULL ; + } +if (unlikely (!foreign_page) ) + { +gdprintk (XENLOG_WARNING , +"Error accessing foreign gfn %" PRI_gfn"\n" , + gfn_list[i]); + rc = -EINVAL ; + copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + } + + foreign_mfn =page_to_mfn (foreign_page); + +/* A page is dirtied when it's being copied to. */ +if ( direction == XENMEM_foreigncopy_to ) +paging_mark_dirty (d, foreign_mfn); +Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1637
Quoted 5 lines
xen/common/memory.c +5
-0+ foreign =map_domain_page (foreign_mfn); +if ( direction == XENMEM_foreigncopy_from ) + rc =copy_to_guest (copy.buffer, foreign,PAGE_SIZE ); +else + rc =copy_from_guest (foreign, copy.buffer,PAGE_SIZE );You cannot validly write to the page without holding a
PGT_writableref. Else you might overwrite a page table or a descriptor table in a PV guest.Given that this code was "inspired" by other hypercalls I'll also check the other code.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Once again - can you please make sure you have addressed earlier review comments, before sending a new version? I did point this out before.
Apparently not.
https://lists.xen.org/archives/html/xen-devel/2026-06/msg00850.html
The "apparently not" is the reply to "can you please make sure you have addressed earlier review comments, before sending a new version".
Quoted 48 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +4
-0+ copy.nr_frames -= i; +guest_handle_add_offset (copy.frame_list, i); + goto out; + }Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1655
Quoted 6 lines
xen/common/memory.c +6
-0+ +guest_handle_add_offset (copy.buffer,PAGE_SIZE ); + } + + copy.nr_frames -= todo; +guest_handle_add_offset (copy.frame_list, todo);Don't you need to also update copy.buffer?
Quoted 1 line
It's updated some lines above inside the loop.
Oh, sorry. Yet then - not doing all updates together is, as you can see, potentially confusing.
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +0
-0@@ -2012,6 +2145,18 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ;Quoted 14 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2155
Quoted 8 lines
xen/common/memory.c +8
-0+case XENMEM_foreigncopy: +/* +* Instead of using "start_extent" we update the structure back, +* we update it back in anyway to tell caller were the copy +* stopped. +*/ +if (unlikely (start_extent) ) +return -EINVAL ;As before - please be precise with comments like this. We update it back also when encoding a continuation. Perhaps instead "..., to indicate the point of failure to the caller as well as to encode continuations without being constrained by
MEMOP_EXTENT_SHIFT".What about (trying to include your suggestion, to be fixed for line length):
Quoted 8 lines
/*
* Instead of using "start_extent" for the continuation, we
update the structure back,
* we update thexen_foreigncopystructure back, so we are not
constrained
* byMEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped. */
Quoted 5 lines
One of the things I take issue with (because it's hard to read that way, at least for me) is the repeated use of "update ... back", effectively saying the same things twice. The last sentence also wants disambiguating towards the "stopped" possibly being a non-error situation as well.
Changed to
/*
* Instead of using "start_extent" for the continuation, we update
* the xen_foreigncopy structure back, so we are not constrained by
* MEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped
* (either for error or because all frames were copied). */
Quoted 19 lines
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
xen/include/public/memory.h +0
-0@@ -740,7 +740,49 @@ struct xen_vnuma_topology_info {typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t ;DEFINE_XEN_GUEST_HANDLE (xen_vnuma_topology_info_t);Quoted 8 lines
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 746
Quoted 5 lines
xen/include/public/memory.h +4
-1-/* Next available subop number is 29 */ +/* +* Copy memory from/to a given domain. +* As this call requires target access and guest with target access won't be +* compat guests supported for compat guests this is not implemented. As before - I question this. You simply can't know. (I'm also struggling with wording / grammar.)
Quoted 3 lines
I was trying to code the compatibility layer. Is there a way to have 64 bit PFN even for compatibility guests instead of having to limit and convert PFN numbers?
Quoted 7 lines
compat_pfn_tis a typedef of unsigned int (since a 32-bit guest seeing "typedef unsigned longxen_pfn_t;" results inxen_pfn_tbeing a 32-bit quantity for it), so 32-bit guests can only supply 32-bit frame numbers. There's also no value in trying to be clever and usinguint64_tinstead for theframe_listhandle, as 32-bit guests won't ever own pages with MFNs wider than 32 bits.
They don't need to own such pages. But probably they would also need other hypercalls to support larger frame numbers. To be honest you just point out a reason to not support this for compatible guests.
Quoted 74 lines
Quoted 74 lines
Quoted 45 lines
Quoted 15 lines
xen/include/public/memory.h +15
-0+*/ +#define XENMEM_foreigncopy 29 +struct xen_foreigncopy { +/* IN - The domain whose memory is to be copied. */ +domid_t domid; + +/* IN - Flags. */ +#define XENMEM_foreigncopy_from 0 +#define XENMEM_foreigncopy_to 1 +#define XENMEM_foreigncopy_direction 1 +uint16_t flags; + +/* +* IN/OUT +* xen/include/public/memory.h struct xen_vnuma_topology_info { · line 771
Quoted 10 lines
xen/include/public/memory.h +10
-0+* As an IN parameter number of frames of the domain to be copied. +* On output on error updated number of frames left. +*/ +uint32_t nr_frames; + +/* +* IN/OUT +* +* Frames to be copied. +* On output on error updated to point to first frame unhandled. Is "on error" really correct / meaningful? The field can be updated at any intermediate point, when a continuation is scheduled. Perhaps:
* On output:
* - on error updated to point to first frame which couldn't be handled,
* - on success undefined.Along these lines for
nr_framesthen as well (if needed at all, seeing that it could as well be undefined in both cases, as the information is redundant with theframe_listupdate).xen/include/public/memory.h struct xen_vnuma_topology_info { · line 778
Quoted 7 lines
xen/include/public/memory.h +7
-0+*/ +XEN_GUEST_HANDLE (xen_pfn_t) frame_list; + +/* +* IN/OUT +* +* Userspace buffer to read/write from. Quoted 4 lines
s/Userspace/Guest/ ?
Also still no mention of when / how this field is updated.
What about:
/*
* Copy memory from/to a given domain.
*/
#defineXENMEM_foreigncopy29struct xen_foreigncopy { /* IN - The domain whose memory is to be copied. */ domid_t domid; /* IN - Flags. */ #define XENMEM_foreigncopy_from 0 #define XENMEM_foreigncopy_to 1 #define XENMEM_foreigncopy_direction 1 uint16_t flags; /* * IN/OUT * * As an IN parameter number of frames of the domain to be copied. * On output updated number of frames left (0 if success). */ uint32_t nr_frames;
Quoted 8 lines
Quoted 5 lines
/*
* IN/OUT
*
* Frames to be copied.
* On output updated to point to the first frame unhandled.There may be no such frame, so at the very least add "..., if any"?
Added.
Quoted 12 lines
Quoted 8 lines
*/ XEN_GUEST_HANDLE(xen_pfn_t) frame_list; /* * IN/OUT * * Guest buffer to read/write from. * On output updated to point to the first frame unhandled.There's no frame here, as long as you don't switch to using two frame lists (for source and destination).
Changed to
/*
* IN/OUT
*
* Guest buffer to read/write from.
* On output updated to point to the first page pointer unhandled. */
Quoted 15 lines
Quoted 15 lines
*/ XEN_GUEST_HANDLE(uint8) buffer; };typedef struct
xen_foreigncopyxen_foreigncopy_t;DEFINE_XEN_GUEST_HANDLE(xen_foreigncopy_t);Quoted 8 lines
xen/include/public/memory.h struct xen_vnuma_topology_info { · line 781
Quoted 3 lines
xen/include/public/memory.h +3
-0+*/ +XEN_GUEST_HANDLE (uint8) buffer; +};What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd
domid_tone then.
Quoted 8 lines
Quoted 2 lines
This was addressed in
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00567.htmlWell, yes, but not in a satisfactory way. Back channels tell me that you actually got the same feedback already on internal review. Which makes it all the more puzzling that you insist on doing it differently. Multiple maintainers asking for the same thing may be an indication of something.
Not needing to have backchannel feedback, I already wrote that a similar approach was tried and made the code more complicated. Both maintainers didn't comment on my replies so I assume they were fine with it.
And you are failing to provide positive feedback.
I asked (that one internally) for examples of guest buffers provided as frame numbers but I got no answer (or better the answer was more "currently there are not").
Also note that the location of xen_foreigncopy_t structure is also provided using a guest pointer.
I remember there were some discussions about ABI changes (2/3 years ago) to address this and other issues but I cannot see much progress. That's why I say this is out of scope.
Quoted 8 lines
Quoted 3 lines
and in minor way by
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00847.html.
It was considered but more complicated and worse from a performance perspective.Okay, performance-wise worse would of course be relevant. But that would need supporting by numbers (for both PV and PVH Dom0, as the latter incurs extra overhead for virtual-address-based hypercall buffer operands).
I'm more concerned about the PV case than PVH to be honest.
I was having an idea about solving the pointer/frames issue but, as I said, it's out of scope here.
Quoted 1 line
Jan
Frediano
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 42 lines · Frediano Ziglio
Quoted 41 lines · Jan Beulich
Quoted 39 lines · Frediano Ziglio
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Frediano Ziglio
xen/common/memory.c +14
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; +Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 6 lines
xen/common/memory.c +6
-0+if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);Iirc I did ask before why this isn't ..._by_any_id().
I probably was confused by the question about MMUEXT and the 2 domains. There are different similar hypercalls (like the mentioned MMUEXT but also hypercalls to map foreign domain memory) that have this check (not the same domain). Any domain has, obviously, access to its own memory, so it should not have to use hypercall to access its own memory. If it does it looks like a mistake causing performance issues or an attempt to circumvent security; in either case you would like to avoid it.
Quoted 6 lines
Quoted 4 lines
No. Self-grants are possible as well, for example, and for a good reason. Allowing normally-remote operations on oneself helps with testing, for example. It may also help avoid needing to special-case "self" in code which needs to cover both cases.
But this is not a grant, it's a copy.
Sure, but the underlying principle is what matters. Plus you don't prevent self-copy by using ..._by_id(), you only preclude the use of DOMID_SELF.
Quoted 38 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
xen/common/memory.c static int acquire_resource( · line 1575
Quoted 8 lines
xen/common/memory.c +8
-0+if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + }Any reason this cannot also be "goto out"? The more that now that you have moved this past the domid validity check, imo it should further move to ...
The only reason was style and to avoid a memory copy, but it's not a hot case so I'll change to "goto out" (no strong about it).
Quoted 12 lines
xen/common/memory.c static int acquire_resource( · line 1582
Quoted 6 lines
xen/common/memory.c +6
-0+/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out;... below here. Perhaps simply as
if ( rc || !copy.nr_frames ) goto out;Quoted 7 lines
Quoted 3 lines
I think this would be confusing with the above "Check we are allowed to map and access these foreign pages" comment.
Are you okay with just the change above to "goto out" ?I do want the order adjusted as indicated. I won't insist on (but I would prefer) folding both if()-s.
What about
Quoted 13 lines
/*
* Check we are allowed to map and access these foreign pages.
*/
rc = xsm_map_gmfn_foreign(XSM_TARGET, currd, d);
if ( rc )
goto out;
while ( copy.nr_frames )
{
/*
* Arbitrary size. Not too much stack space, and a reasonable stride
* for continuation checks.
*/That's fine.
Quoted 40 lines
Quoted 40 lines
Quoted 29 lines
Quoted 19 lines
Quoted 19 lines
xen/common/memory.c +19
-0+ do { + /* + * Arbitrary size. Not too much stack space, and a reasonable stride + * for continuation checks. + */ + xen_pfn_t gfn_list[32]; + unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames); + + rc = -EFAULT; + if ( copy_from_guest(gfn_list, copy.frame_list, todo) ) + goto out; + + for ( unsigned int i = 0; i < todo; i++ ) + { + struct page_info *foreign_page; + mfn_t foreign_mfn; + void *foreign; + p2m_type_t p2mt; + const unsigned long valid_mask =Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1607
Quoted 5 lines
xen/common/memory.c +5
-0+#ifdef CONFIG_X86 +p2m_to_mask (p2m_ram_rw) |p2m_to_mask (p2m_ram_logdirty); +#else +p2m_to_mask (p2m_ram_rw); +#endif The set of permitted types didn't change, yet a justification for the resulting limitation also didn't appear.
Quoted 3 lines
Yes, that's missing, indeed.
Should the set of types be different for reading and writing? For instance do not allow writing to read-only memory?Of course.
Quoted 4 lines
Given that it looks like different architectures have different meanings and definitions for these constants, should it not be better to define some new constants for this specific usage? For instance
P2M_READ_TYPESandP2M_WRITE_TYPES?
Quoted 5 lines
Quoted 1 line
Perhaps, yes. The suggested names look overly generic to me, though.
I suppose
P2M_READABLE_TYPESandP2M_WRITABLE_TYPESare more correct but still too generic.P2M_EXPORTABLE_TYPESandP2M_IMPORTABLE_TYPES?
First: Do you foresee uses of those constants anywhere else? If not (I don't), tie the names to this particular operation. That'll make them entirely non-generic.
Quoted 38 lines
Quoted 30 lines
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +0
-0@@ -2012,6 +2145,18 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ;Quoted 14 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2155
Quoted 8 lines
xen/common/memory.c +8
-0+case XENMEM_foreigncopy: +/* +* Instead of using "start_extent" we update the structure back, +* we update it back in anyway to tell caller were the copy +* stopped. +*/ +if (unlikely (start_extent) ) +return -EINVAL ;As before - please be precise with comments like this. We update it back also when encoding a continuation. Perhaps instead "..., to indicate the point of failure to the caller as well as to encode continuations without being constrained by
MEMOP_EXTENT_SHIFT".What about (trying to include your suggestion, to be fixed for line length):
Quoted 8 lines
/*
* Instead of using "start_extent" for the continuation, we
update the structure back,
* we update thexen_foreigncopystructure back, so we are not
constrained
* byMEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped. */Quoted 5 lines
One of the things I take issue with (because it's hard to read that way, at least for me) is the repeated use of "update ... back", effectively saying the same things twice. The last sentence also wants disambiguating towards the "stopped" possibly being a non-error situation as well.
Changed to
Quoted 7 lines
/*
* Instead of using "start_extent" for the continuation, we update
* thexen_foreigncopystructure back, so we are not constrained by
*MEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped
* (either for error or because all frames were copied). */
Thanks.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2157
Quoted 2 lines
xen/common/memory.c +2
-0+XEN_GUEST_HANDLE (uint8) buffer; +};What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd
domid_tone then.This was addressed in
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00567.html
Quoted 7 lines
Quoted 4 lines
Well, yes, but not in a satisfactory way. Back channels tell me that you actually got the same feedback already on internal review. Which makes it all the more puzzling that you insist on doing it differently. Multiple maintainers asking for the same thing may be an indication of something.
Not needing to have backchannel feedback, I already wrote that a similar approach was tried and made the code more complicated.
Even if indeed so: Yet at the same time more flexible.
Quoted 10 lines
Both maintainers didn't comment on my replies so I assume they were fine with it.
And you are failing to provide positive feedback.
I asked (that one internally) for examples of guest buffers provided as frame numbers but I got no answer (or better the answer was more "currently there are not").
Also note that the location ofxen_foreigncopy_tstructure is also provided using a guest pointer.
I remember there were some discussions about ABI changes (2/3 years ago) to address this and other issues but I cannot see much progress.
And it's that (very slowly progressing effort) which made me ask. The fewer virtual addresses we bake into new sub-ops, the better for that effort. And no, that doesn't go as far as completely eliminating handles (presently representing virtual addresses) - that needs to be part of the new ABI.
To preempt the argument towards "fewer virtual addresses" not really being true when changing from handle-to-uint8 to handle-to-pfn: The former won't be able to express a buffer mapped contiguously in VA space, but discontiguous in PA space. The latter will, simply be avoiding buffer VAs in the first place (the array of frame numbers can e.g. be placed in a dedicated hypercall argument area known to be physically contiguous).
Quoted 1 line
That's why I say this is out of scope.
There's nothing scope related here. We're discussing how to shape the new sub-op interface.
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
and in minor way by
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00847.html.
It was considered but more complicated and worse from a performance perspective.Okay, performance-wise worse would of course be relevant. But that would need supporting by numbers (for both PV and PVH Dom0, as the latter incurs extra overhead for virtual-address-based hypercall buffer operands).
I'm more concerned about the PV case than PVH to be honest.
For your (immediate) internal purposes that may be fine, but PVH Dom0 more likely being the future, for upstream both need considering equally.
Jan
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 49 lines · Jan Beulich
Quoted 47 lines · Frediano Ziglio
Quoted 45 lines · Jan Beulich
Quoted 31 lines · Frediano Ziglio
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Frediano Ziglio
xen/common/memory.c +14
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; +Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 6 lines
xen/common/memory.c +6
-0+if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);Iirc I did ask before why this isn't ..._by_any_id().
Quoted 8 lines
I probably was confused by the question about MMUEXT and the 2 domains. There are different similar hypercalls (like the mentioned MMUEXT but also hypercalls to map foreign domain memory) that have this check (not the same domain). Any domain has, obviously, access to its own memory, so it should not have to use hypercall to access its own memory. If it does it looks like a mistake causing performance issues or an attempt to circumvent security; in either case you would like to avoid it.
No. Self-grants are possible as well, for example, and for a good reason. Allowing normally-remote operations on oneself helps with testing, for example. It may also help avoid needing to special-case "self" in code which needs to cover both cases.
Quoted 5 lines
Quoted 1 line
But this is not a grant, it's a copy.
Sure, but the underlying principle is what matters. Plus you don't prevent self-copy by using ..._by_id(), you only preclude the use of
DOMID_SELF.
Sure about this? The current implementation is
int rcu_lock_remote_domain_by_id(domid_t dom, struct domain **d)
{
if ( (*d = rcu_lock_domain_by_id(dom)) == NULL )
return -ESRCH;
if ( *d == current->domain )
{
rcu_unlock_domain(*d);
return -EPERM;
}
return 0;
}rcu_lock_domain_by_id returns NULL if the domain is DOMID_SELF so it would be a -ESRCH, if it's the current domain it would be excluded by *d == current->domain returning -EPERM.
Quoted 45 lines
Quoted 45 lines
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
xen/common/memory.c static int acquire_resource( · line 1575
Quoted 8 lines
xen/common/memory.c +8
-0+if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + }Any reason this cannot also be "goto out"? The more that now that you have moved this past the domid validity check, imo it should further move to ...
The only reason was style and to avoid a memory copy, but it's not a hot case so I'll change to "goto out" (no strong about it).
Quoted 12 lines
xen/common/memory.c static int acquire_resource( · line 1582
Quoted 6 lines
xen/common/memory.c +6
-0+/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out;... below here. Perhaps simply as
if ( rc || !copy.nr_frames ) goto out;Quoted 7 lines
Quoted 3 lines
I think this would be confusing with the above "Check we are allowed to map and access these foreign pages" comment.
Are you okay with just the change above to "goto out" ?I do want the order adjusted as indicated. I won't insist on (but I would prefer) folding both if()-s.
What about
/* * Check we are allowed to map and access these foreign pages. */ rc = xsm_map_gmfn_foreign(XSM_TARGET, currd, d); if ( rc ) goto out;
Quoted 9 lines
Quoted 6 lines
while ( copy.nr_frames )
{
/*
* Arbitrary size. Not too much stack space, and a reasonable stride
* for continuation checks.
*/That's fine.
Changed
Quoted 130 lines
Quoted 40 lines
Quoted 40 lines
Quoted 29 lines
Quoted 19 lines
Quoted 19 lines
xen/common/memory.c +19
-0+ do { + /* + * Arbitrary size. Not too much stack space, and a reasonable stride + * for continuation checks. + */ + xen_pfn_t gfn_list[32]; + unsigned int todo = MIN(ARRAY_SIZE(gfn_list), copy.nr_frames); + + rc = -EFAULT; + if ( copy_from_guest(gfn_list, copy.frame_list, todo) ) + goto out; + + for ( unsigned int i = 0; i < todo; i++ ) + { + struct page_info *foreign_page; + mfn_t foreign_mfn; + void *foreign; + p2m_type_t p2mt; + const unsigned long valid_mask =Quoted 9 lines
xen/common/memory.c static int acquire_resource( · line 1607
Quoted 5 lines
xen/common/memory.c +5
-0+#ifdef CONFIG_X86 +p2m_to_mask (p2m_ram_rw) |p2m_to_mask (p2m_ram_logdirty); +#else +p2m_to_mask (p2m_ram_rw); +#endif The set of permitted types didn't change, yet a justification for the resulting limitation also didn't appear.
Quoted 3 lines
Yes, that's missing, indeed.
Should the set of types be different for reading and writing? For instance do not allow writing to read-only memory?Of course.
Quoted 4 lines
Given that it looks like different architectures have different meanings and definitions for these constants, should it not be better to define some new constants for this specific usage? For instance
P2M_READ_TYPESandP2M_WRITE_TYPES?Quoted 5 lines
Quoted 1 line
Perhaps, yes. The suggested names look overly generic to me, though.
I suppose
P2M_READABLE_TYPESandP2M_WRITABLE_TYPESare more correct but still too generic.P2M_EXPORTABLE_TYPESandP2M_IMPORTABLE_TYPES?First: Do you foresee uses of those constants anywhere else? If not (I don't), tie the names to this particular operation. That'll make them entirely non-generic.
Quoted 38 lines
Quoted 30 lines
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
xen/common/memory.c +0
-0@@ -2012,6 +2145,18 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ;Quoted 14 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2155
Quoted 8 lines
xen/common/memory.c +8
-0+case XENMEM_foreigncopy: +/* +* Instead of using "start_extent" we update the structure back, +* we update it back in anyway to tell caller were the copy +* stopped. +*/ +if (unlikely (start_extent) ) +return -EINVAL ;As before - please be precise with comments like this. We update it back also when encoding a continuation. Perhaps instead "..., to indicate the point of failure to the caller as well as to encode continuations without being constrained by
MEMOP_EXTENT_SHIFT".What about (trying to include your suggestion, to be fixed for line length):
Quoted 8 lines
/*
* Instead of using "start_extent" for the continuation, we
update the structure back,
* we update thexen_foreigncopystructure back, so we are not
constrained
* byMEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped. */Quoted 5 lines
One of the things I take issue with (because it's hard to read that way, at least for me) is the repeated use of "update ... back", effectively saying the same things twice. The last sentence also wants disambiguating towards the "stopped" possibly being a non-error situation as well.
Changed to
Quoted 7 lines
/*
* Instead of using "start_extent" for the continuation, we update
* thexen_foreigncopystructure back, so we are not constrained by
*MEMOP_EXTENT_SHIFT.
* We copy it back also to tell the caller where the copy stopped
* (either for error or because all frames were copied). */Thanks.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 2157
Quoted 2 lines
xen/common/memory.c +2
-0+XEN_GUEST_HANDLE (uint8) buffer; +};What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd
domid_tone then.This was addressed in
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00567.htmlQuoted 7 lines
Quoted 4 lines
Well, yes, but not in a satisfactory way. Back channels tell me that you actually got the same feedback already on internal review. Which makes it all the more puzzling that you insist on doing it differently. Multiple maintainers asking for the same thing may be an indication of something.
Not needing to have backchannel feedback, I already wrote that a similar approach was tried and made the code more complicated.
Even if indeed so: Yet at the same time more flexible.
Quoted 10 lines
Both maintainers didn't comment on my replies so I assume they were fine with it.
And you are failing to provide positive feedback.
I asked (that one internally) for examples of guest buffers provided as frame numbers but I got no answer (or better the answer was more "currently there are not").
Also note that the location ofxen_foreigncopy_tstructure is also provided using a guest pointer.
I remember there were some discussions about ABI changes (2/3 years ago) to address this and other issues but I cannot see much progress.
Quoted 6 lines
And it's that (very slowly progressing effort) which made me ask. The fewer virtual addresses we bake into new sub-ops, the better for that effort. And no, that doesn't go as far as completely eliminating handles (presently representing virtual addresses) - that needs to be part of the new ABI.
In other words, you want me to code something temporary that you already know that needs to be changed.
Quoted 8 lines
To preempt the argument towards "fewer virtual addresses" not really being true when changing from handle-to-uint8 to handle-to-pfn: The former won't be able to express a buffer mapped contiguously in VA space, but discontiguous in PA space. The latter will, simply be avoiding buffer VAs in the first place (the array of frame numbers can e.g. be placed in a dedicated hypercall argument area known to be physically contiguous).
If it's mapped continuously in VA and you pass the VA I don't understand the problem. From the way I see it's more the latter that's the problem.
Quoted 13 lines
Quoted 1 line
That's why I say this is out of scope.
There's nothing scope related here. We're discussing how to shape the new sub-op interface.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
and in minor way by
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00847.html.
It was considered but more complicated and worse from a performance perspective.Okay, performance-wise worse would of course be relevant. But that would need supporting by numbers (for both PV and PVH Dom0, as the latter incurs extra overhead for virtual-address-based hypercall buffer operands).
Quoted 6 lines
Quoted 1 line
I'm more concerned about the PV case than PVH to be honest.
For your (immediate) internal purposes that may be fine, but PVH Dom0 more likely being the future, for upstream both need considering equally.
From an internal discussion:
--------------------
About HVM and VAs/PFNs, I was thinking. We pass a single PFN for a page containing either
- a single list of PFNs fitting into the page (plus number of entries) for small hypercalls (most of them)
- a page containing a list of PFNs pointing to arrays of PFNs for large hypercalls (like kexec load and few others)
Now the GUEST_HANDLEs are treated differently, instead of VAs they contain offset into array above plus page offset.
Okay, let's do an example. I want to call a DOMCTL which have a handle to a small array, so I need to pass the domctl structure and the array, I suppose I need 2 PFNs (unless domct or the array span multiple pages) so kernel would need to build and pass an array of 2 PFNs, let's say 0xabcabc01 and 0xabcabc02. I build a page with 1- 2 (number of PFNs)
2- 0xabcabc01
3- 0xabcabc02
I pass the PFN of the above page (how it's a detail, probably an additional register), number of hypercall and... a GUEST_HANDLE. Here the guest handle would be something like 0x0000ppp where ppp is the page offset inside 0xabcabc01 page. Why 0xabcabc01 ? Because the upper part of the guest handle is 0 (so the first entry in the PFNs array above). Inside the domctl structure the guest handle to the array will be something like 0x0001ppp where the 0x0001 means 0xabcabc02 while ppp here is the page offset into 0xabcabc02.
What if the array is bigger than a single page? Let's say it spans into 3 pages, you would have something like
1- 4 (number of PFNs)
2- 0xabcabc01
3- 0xabcabc02
4- 0xabcabc03
5- 0xabcabc04
(okay, in all example pages are contiguous but not important) The guest handle for the array would still be 0x0001ppp but the code will continue on array entry 2 (0xabcabc03) and 3 (0xabcabc04)
--------------------
The distinction for the above could be a flag added to the hypercall number or automatic on type of VM (PV/HVM) but for compatibility I would go for the first. But these are details.
This would work without much API changes and minor hypervisor changes (mainly copy_from_guest and similar macros).
Okay, the above is a bit OT here but the point is that the change you are asking me won't help with this in the future, basically you are asking me an implementation based on an implementation that is not currently even on paper.
Quoted 1 line
Jan
Frediano
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 49 lines · Frediano Ziglio
Quoted 48 lines · Jan Beulich
Quoted 47 lines · Frediano Ziglio
Quoted 45 lines · Jan Beulich
Quoted 31 lines · Frediano Ziglio
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Frediano Ziglio
xen/common/memory.c +14
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; +Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 6 lines
xen/common/memory.c +6
-0+if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);Iirc I did ask before why this isn't ..._by_any_id().
Quoted 8 lines
I probably was confused by the question about MMUEXT and the 2 domains. There are different similar hypercalls (like the mentioned MMUEXT but also hypercalls to map foreign domain memory) that have this check (not the same domain). Any domain has, obviously, access to its own memory, so it should not have to use hypercall to access its own memory. If it does it looks like a mistake causing performance issues or an attempt to circumvent security; in either case you would like to avoid it.
No. Self-grants are possible as well, for example, and for a good reason. Allowing normally-remote operations on oneself helps with testing, for example. It may also help avoid needing to special-case "self" in code which needs to cover both cases.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
But this is not a grant, it's a copy.
Sure, but the underlying principle is what matters. Plus you don't prevent self-copy by using ..._by_id(), you only preclude the use of
DOMID_SELF.Sure about this?
No, I'm sorry: I (repeatedly) managed to ignore the "remote" in the function called. That said, my request stands: No arbitrary restrictions please. If you can properly justify a restriction, that's a different thing.
Quoted 32 lines
Quoted 32 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
xen/common/memory.c static int acquire_resource( · line 1569
Quoted 2 lines
xen/common/memory.c +2
-0+XEN_GUEST_HANDLE (uint8) buffer; +};What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd
domid_tone then.This was addressed in
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00567.htmlQuoted 7 lines
Quoted 4 lines
Well, yes, but not in a satisfactory way. Back channels tell me that you actually got the same feedback already on internal review. Which makes it all the more puzzling that you insist on doing it differently. Multiple maintainers asking for the same thing may be an indication of something.
Not needing to have backchannel feedback, I already wrote that a similar approach was tried and made the code more complicated.
Even if indeed so: Yet at the same time more flexible.
Quoted 10 lines
Both maintainers didn't comment on my replies so I assume they were fine with it.
And you are failing to provide positive feedback.
I asked (that one internally) for examples of guest buffers provided as frame numbers but I got no answer (or better the answer was more "currently there are not").
Also note that the location ofxen_foreigncopy_tstructure is also provided using a guest pointer.
I remember there were some discussions about ABI changes (2/3 years ago) to address this and other issues but I cannot see much progress.
Quoted 8 lines
Quoted 5 lines
And it's that (very slowly progressing effort) which made me ask. The fewer virtual addresses we bake into new sub-ops, the better for that effort. And no, that doesn't go as far as completely eliminating handles (presently representing virtual addresses) - that needs to be part of the new ABI.
In other words, you want me to code something temporary that you already know that needs to be changed.
What do you mean by "temporary"? We will need to live with the present ABI for the foreseeable future. The new ABI's requirements haven't even been spelled out yet. Patches to allow use of physical addresses in place of virtual ones were actually turned down on the grounds of there not having been a write-down of all requirements.
Quoted 11 lines
Quoted 7 lines
To preempt the argument towards "fewer virtual addresses" not really being true when changing from handle-to-uint8 to handle-to-pfn: The former won't be able to express a buffer mapped contiguously in VA space, but discontiguous in PA space. The latter will, simply be avoiding buffer VAs in the first place (the array of frame numbers can e.g. be placed in a dedicated hypercall argument area known to be physically contiguous).
If it's mapped continuously in VA and you pass the VA I don't understand the problem. From the way I see it's more the latter that's the problem.
I'm talking of the future, where VAs wouldn't be used anymore. The buffer you use couldn't be described by a single PA, unless the caller took specific measures up front.
Jan
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
Quoted 53 lines · Jan Beulich
Quoted 51 lines · Frediano Ziglio
Quoted 50 lines · Jan Beulich
Quoted 48 lines · Frediano Ziglio
Quoted 41 lines · Jan Beulich
Quoted 31 lines · Frediano Ziglio
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Frediano Ziglio
xen/common/memory.c +14
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; +Quoted 8 lines
xen/common/memory.c static int acquire_resource( · line 1567
Quoted 6 lines
xen/common/memory.c +6
-0+if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d);Iirc I did ask before why this isn't ..._by_any_id().
Quoted 8 lines
I probably was confused by the question about MMUEXT and the 2 domains. There are different similar hypercalls (like the mentioned MMUEXT but also hypercalls to map foreign domain memory) that have this check (not the same domain). Any domain has, obviously, access to its own memory, so it should not have to use hypercall to access its own memory. If it does it looks like a mistake causing performance issues or an attempt to circumvent security; in either case you would like to avoid it.
Quoted 4 lines
No. Self-grants are possible as well, for example, and for a good reason. Allowing normally-remote operations on oneself helps with testing, for example. It may also help avoid needing to special-case "self" in code which needs to cover both cases.
But this is not a grant, it's a copy.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Sure, but the underlying principle is what matters. Plus you don't prevent self-copy by using ..._by_id(), you only preclude the use of
DOMID_SELF.Sure about this?
No, I'm sorry: I (repeatedly) managed to ignore the "remote" in the function called. That said, my request stands: No arbitrary restrictions please. If you can properly justify a restriction, that's a different thing.
Not strong about it.
I'll change to rcu_lock_domain_by_any_id.
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
xen/common/memory.c static int acquire_resource( · line 1569
Quoted 2 lines
xen/common/memory.c +2
-0+XEN_GUEST_HANDLE (uint8) buffer; +};What was (again) left unaddressed is the question towards using GFNs on both sides of the copy. This would eliminate the need for the flags field, taken by a 2nd
domid_tone then.This was addressed in
https://lists.xenproject.org/archives/html/xen-devel/2026-06/msg00567.htmlQuoted 7 lines
Quoted 4 lines
Well, yes, but not in a satisfactory way. Back channels tell me that you actually got the same feedback already on internal review. Which makes it all the more puzzling that you insist on doing it differently. Multiple maintainers asking for the same thing may be an indication of something.
Not needing to have backchannel feedback, I already wrote that a similar approach was tried and made the code more complicated.
Even if indeed so: Yet at the same time more flexible.
Quoted 10 lines
Both maintainers didn't comment on my replies so I assume they were fine with it.
And you are failing to provide positive feedback.
I asked (that one internally) for examples of guest buffers provided as frame numbers but I got no answer (or better the answer was more "currently there are not").
Also note that the location ofxen_foreigncopy_tstructure is also provided using a guest pointer.
I remember there were some discussions about ABI changes (2/3 years ago) to address this and other issues but I cannot see much progress.And it's that (very slowly progressing effort) which made me ask. The fewer virtual addresses we bake into new sub-ops, the better for that effort. And no, that doesn't go as far as completely eliminating handles (presently representing virtual addresses) - that needs to be part of the new ABI.
Quoted 9 lines
Quoted 2 lines
In other words, you want me to code something temporary that you already know that needs to be changed.
What do you mean by "temporary"? We will need to live with the present ABI for the foreseeable future. The new ABI's requirements haven't even been spelled out yet. Patches to allow use of physical addresses in place of virtual ones were actually turned down on the grounds of there not having been a write-down of all requirements.
Temporary in the sense that there will be new ABIs to deal with not using virtual addresses.
The second sentence is a bit contradictory. You want me to address the virtual address complaint but you are telling me that the change will be turned down if I don't address everything. And this is why this is out of scope here.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
To preempt the argument towards "fewer virtual addresses" not really being true when changing from handle-to-uint8 to handle-to-pfn: The former won't be able to express a buffer mapped contiguously in VA space, but discontiguous in PA space. The latter will, simply be avoiding buffer VAs in the first place (the array of frame numbers can e.g. be placed in a dedicated hypercall argument area known to be physically contiguous).
Quoted 8 lines
Quoted 3 lines
If it's mapped continuously in VA and you pass the VA I don't understand the problem. From the way I see it's more the latter that's the problem.
I'm talking of the future, where VAs wouldn't be used anymore. The buffer you use couldn't be described by a single PA, unless the caller took specific measures up front.
If you read my reply I suggested a way to avoid virtual addresses completely.
Quoted 1 line
Jan
About the P2M type check it turned out that I was wrong with the checking. The MMAP way use MMU_UPDATE calls which do not care about P2M type at all. Changing the code to
...
for ( unsigned int i = 0; i < todo; i++ )
{
struct page_info *foreign_page;
mfn_t foreign_mfn;
void *foreign;
p2m_type_t p2mt;
p2m_query_t q = (direction == XENMEM_foreigncopy_to) ?
P2M_ALLOC | P2M_UNSHARE : P2M_ALLOC;
foreign_page = get_page_from_gfn(d, gfn_list[i], &p2mt, q);
if ( unlikely(p2m_is_paged(p2mt)) )
{
if ( foreign_page )
put_page(foreign_page);
p2m_mem_paging_populate(d, _gfn(gfn_list[i]));
p2mt = p2m_ram_paging_in;
foreign_page = NULL;
}
if ( unlikely(!foreign_page) )
{
rc = -ENOENT;
if ( p2mt != p2m_ram_paging_in )
{
gdprintk(XENLOG_WARNING,
"Error accessing foreign gfn %" PRI_gfn "\n",
gfn_list[i]);
rc = -EINVAL;
}
copy.nr_frames -= i;
guest_handle_add_offset(copy.frame_list, i);
goto out;
}...
About the XSM part I have now
...
/*
* Check we are allowed to map and access these foreign pages.
*/
if ( direction == XENMEM_foreigncopy_from )
rc = xsm_foreigncopy_from(XSM_TARGET, currd, d);
else
rc = xsm_foreigncopy_to(XSM_TARGET, currd, d);
if ( rc )
goto out;...
I wrote some code for the compat mode but I need to test it. Still I think that adding it it's a mistake, it's just a new, probably unused, ABI that must be maintained till a probable "no virtual address" ABI will replace it.
Frediano
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
In reply to Frediano ZiglioDaniel,
Quoted 29 lines · Frediano Ziglio
xen/common/memory.c +25
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; + +if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d); +if ( rc ) +return rc; + +if ( copy.nr_frames ==0 ) + {
xen/common/memory.c static int acquire_resource( · line 1582
Quoted 10 lines
xen/common/memory.c +10
-0+rcu_unlock_domain (d); +return 0 ; + } + +/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d); +if ( rc ) + goto out;
can you please clarify whether such a re-use of an existing predicate is acceptable?
Jan
Re: [PATCH v6 12/16] xen: implement new foreign copy hypercall
In reply to Frediano ZiglioQuoted 56 lines · Frediano Ziglio
Add a sub hypercall to
__HYPERVISOR_memory_opto allow to read/write memory from/to a foreign domain.Extending
MMUEXT_COPY_PAGEseems better on first sight but considering that MMUEXT is meant for PV only and trying to change that sub-op this solution is better.Signed-off-by Frediano Ziglio <[email protected]>--
Changes since v4:
- Fix typo in comment.Changes since v5:
- updatexen_foreigncopystructure comments;
- move check for no frames after checking the domain;
- use mnemonic instead of 1U;
- fix page type checks;
- do not overwrite error copying back structure;
- latch MFN value;
- improved commit message.--- xen/common/memory.c | 145 ++++++++++++++++++++++++++++++++++++ xen/include/public/memory.h | 44 ++++++++++- 2 files changed, 188 insertions(+), 1 deletion(-)xen/common/memory.c +23
-0@@ -1545,6 +1545,139 @@ static int acquire_resource(return rc; } +/* +* The "noinline" qualifier avoids the compiler to create a large function +* consuming quite a lot of stack. +*/ +static int noinline mem_foreigncopy( +XEN_GUEST_HANDLE_PARAM (xen_foreigncopy_t) arg) +{ +struct domain *d, *const currd = current->domain; +xen_foreigncopy_t copy; +int rc, direction; + +if (copy_from_guest (©, arg,1 ) ) +return -EFAULT ; + +if ( copy.flags & ~XENMEM_foreigncopy_direction ) +return -EINVAL ; + + direction = copy.flags & XENMEM_foreigncopy_direction; + + rc =rcu_lock_remote_domain_by_id (copy.domid, &d); +if ( rc ) +return rc; +
xen/common/memory.c static int acquire_resource( · line 1580
Quoted 10 lines
xen/common/memory.c +10
-0+if ( copy.nr_frames ==0 ) + { +rcu_unlock_domain (d); +return 0 ; + } + +/* +* Check we are allowed to map and access these foreign pages. +*/ + rc =xsm_map_gmfn_foreign (XSM_TARGET , currd, d);
While on the result is the same, this is a different action. I didn't immediately answer because I am split on what the new hook should be. In particular if it should be only one like map_gmfn, if it should instead also take a direction (in/out), of should we have two hooks (xxx_read, xxx_write). Myself, I am leaning toward the last option but I am open to hearing other's opinions. If you need any assistance with writing the hooks, feel free to reach out.
v/r,
dps
PATCH v6 13/16 privcmd: Add definition for new Linux privcmd to access new Xen hypercall
In reply to Frediano ZiglioUserspace should use new ioctl to access new hypercall.
--
Changes since v4:
- update comment.
--- tools/include/xen-sys/Linux/privcmd.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
tools/include/xen-sys/Linux/privcmd.h +10 -0
@@ -100,6 +100,14 @@ typedef struct privcmd_pcidev_get_gsi {__u32 gsi; } privcmd_pcidev_get_gsi_t; +typedef struct privcmd_foreigncopy { +domid_t dom;/* Foreign domain. */ +__u16 dir;/* Direction, 0 from, 1 to. */ +__u32 num;/* Number of pages to copy. */ +const xen_pfn_t __user *pfns;/* Array of pfns. */ +void __user *buffer;/* Buffer to copy to/from. */ +}privcmd_foreigncopy_t ; +/* * @cmd: IOCTL_PRIVCMD_HYPERCALL * @arg: &privcmd_hypercall_t
@@ -121,6 +129,8 @@ typedef struct privcmd_pcidev_get_gsi { _IOC(_IOC_NONE, 'P', 7, sizeof(privcmd_mmap_resource_t))#define IOCTL_PRIVCMD_PCIDEV_GET_GSI \ _IOC(_IOC_NONE, 'P', 10, sizeof(privcmd_pcidev_get_gsi_t)) +#define IOCTL_PRIVCMD_FOREIGNCOPY \ + _IOWR('P', 11, privcmd_foreigncopy_t)#define IOCTL_PRIVCMD_UNIMPLEMENTED \ _IOC(_IOC_NONE, 'P', 0xFF, 0)
Signature · 1 line
2.43.0
Re: [PATCH v6 13/16] privcmd: Add definition for new Linux privcmd to access new Xen hypercall
Quoted 20 lines · Frediano Ziglio
tools/include/xen-sys/Linux/privcmd.h +8
-0@@ -100,6 +100,14 @@ typedef struct privcmd_pcidev_get_gsi {__u32 gsi; } privcmd_pcidev_get_gsi_t; +typedef struct privcmd_foreigncopy { +domid_t dom;/* Foreign domain. */ +__u16 dir;/* Direction, 0 from, 1 to. */ +__u32 num;/* Number of pages to copy. */ +const xen_pfn_t __user *pfns;/* Array of pfns. */ +void __user *buffer;/* Buffer to copy to/from. */ +}privcmd_foreigncopy_t ; +/* * @cmd: IOCTL_PRIVCMD_HYPERCALL * @arg: &privcmd_hypercall_t
tools/include/xen-sys/Linux/privcmd.h typedef struct privcmd_pcidev_get_gsi { · line 136
Quoted 8 lines
tools/include/xen-sys/Linux/privcmd.h +2
-0@@ -121,6 +129,8 @@ typedef struct privcmd_pcidev_get_gsi { _IOC(_IOC_NONE, 'P', 7, sizeof(privcmd_mmap_resource_t))#define IOCTL_PRIVCMD_PCIDEV_GET_GSI \ _IOC(_IOC_NONE, 'P', 10, sizeof(privcmd_pcidev_get_gsi_t)) +#define IOCTL_PRIVCMD_FOREIGNCOPY \ + _IOWR('P', 11, privcmd_foreigncopy_t)#define IOCTL_PRIVCMD_UNIMPLEMENTED \ _IOC(_IOC_NONE, 'P', 0xFF, 0)
I don't think I can accept this patch until the changes have been added to Linux. Also, I think it would be fine to squash this changes into the patch that start using this new hypercall.
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH v6 13/16] privcmd: Add definition for new Linux privcmd to access new Xen hypercall
Quoted 26 lines · Anthony PERARD
Quoted 24 lines · Frediano Ziglio
tools/include/xen-sys/Linux/privcmd.h +9
-0@@ -100,6 +100,14 @@ typedef struct privcmd_pcidev_get_gsi {__u32 gsi; } privcmd_pcidev_get_gsi_t; +typedef struct privcmd_foreigncopy { +domid_t dom;/* Foreign domain. */ +__u16 dir;/* Direction, 0 from, 1 to. */ +__u32 num;/* Number of pages to copy. */ +const xen_pfn_t __user *pfns;/* Array of pfns. */ +void __user *buffer;/* Buffer to copy to/from. */ +}privcmd_foreigncopy_t ; +/* * @cmd: IOCTL_PRIVCMD_HYPERCALL * @arg: &privcmd_hypercall_t @@ -121,6 +129,8 @@ typedef struct privcmd_pcidev_get_gsi { _IOC(_IOC_NONE, 'P', 7, sizeof(privcmd_mmap_resource_t))#define IOCTL_PRIVCMD_PCIDEV_GET_GSI \ _IOC(_IOC_NONE, 'P', 10, sizeof(privcmd_pcidev_get_gsi_t)) +#define IOCTL_PRIVCMD_FOREIGNCOPY \
Quoted 9 lines
tools/include/xen-sys/Linux/privcmd.h typedef struct privcmd_pcidev_get_gsi { · line 136
Quoted 4 lines
tools/include/xen-sys/Linux/privcmd.h +1
-0+ _IOWR('P', 11, privcmd_foreigncopy_t)#define IOCTL_PRIVCMD_UNIMPLEMENTED \ _IOC(_IOC_NONE, 'P', 0xFF, 0)I don't think I can accept this patch until the changes have been added to Linux. Also, I think it would be fine to squash this changes into the patch that start using this new hypercall.
The last patch is for Linux, but I saw no comments on it. For the merge, yes, it can be done, not a big deal.
Frediano
PATCH v6 14/16 libs/guest: use new hypercall if available
In reply to Frediano ZiglioUse new hypercall if available, otherwise fall back to map+copy+unmap sequence.
--
Changes since v4:
- use int8_t instead of char for signed type.
--- tools/libs/guest/xg_sr_common.c | 47 ++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 9 deletions(-)
tools/libs/guest/xg_sr_common.c +38 -9
@@ -156,11 +156,6 @@ static void __attribute__((unused)) build_assertions(void)BUILD_BUG_ON (sizeof (struct xc_sr_rec_hvm_params ) !=8 ); } -enum { - foreigncopy_from, - foreigncopy_to -}; -static int xg_foreignmemory_copy (xc_interface *xch,domid_t domid,int dir,size_t nr_pages,void *buffer,const xen_pfn_t foreign_pfns[nr_pages])
@@ -174,8 +169,42 @@ static int xg_foreignmemory_copy(xc_interface *xch, domid_t domid,return -1 ; } +/* +* If foreign copy is supported, -1 not initialized, 0 not supported, +* 1 supported. +*/ +static int8_t foreign_copy_supported =-1 ; + +if ( foreign_copy_supported ) + { +int rc; +privcmd_foreigncopy_t copy = { + .dom = domid, + .dir = dir, + .num = nr_pages, + .buffer = buffer, + }; +DECLARE_HYPERCALL_BOUNCE_IN (foreign_pfns, nr_pages *sizeof (xen_pfn_t)); + +if (xc_hypercall_bounce_pre (xch, foreign_pfns) ) +return -1 ; + + copy.pfns = foreign_pfns; + + rc =ioctl (xencall_fd (xch->xcall),IOCTL_PRIVCMD_FOREIGNCOPY , ©); +if ( foreign_copy_supported <0 ) + foreign_copy_supported = + (!rc || (errno !=ENOTTY && errno !=ENOSYS )); + +xc_hypercall_bounce_post (xch, foreign_pfns); + +if ( foreign_copy_supported ) +return rc; + } + +/* Fallback, emulate. */ int err[nr_pages]; -const int prot = (dir == foreigncopy_from) ?PROT_READ :PROT_READ |PROT_WRITE ; +const int prot = (dir == XENMEM_foreigncopy_from) ?PROT_READ :PROT_READ |PROT_WRITE ;void *p =xenforeignmemory_map (xch->fmem, domid, prot, nr_pages, foreign_pfns, err);if ( !p )
@@ -189,7 +218,7 @@ static int xg_foreignmemory_copy(xc_interface *xch, domid_t domid,return -1 ; } -if ( dir == foreigncopy_from ) +if ( dir == XENMEM_foreigncopy_from )memcpy (buffer, p, nr_pages *XC_PAGE_SIZE );else memcpy (p, buffer, nr_pages *XC_PAGE_SIZE );
@@ -201,7 +230,7 @@ int xg_foreignmemory_copy_from(xc_interface *xch, domid_t dom, size_t nr_pages, void *dest, const xen_pfn_t source[nr_pages]) { - return xg_foreignmemory_copy(xch, dom, foreigncopy_from, + return xg_foreignmemory_copy(xch, dom, XENMEM_foreigncopy_from,nr_pages ,dest ,source ); }
@@ -209,7 +238,7 @@ int xg_foreignmemory_copy_to(xc_interface *xch, domid_t dom, size_t nr_pages, const xen_pfn_t dest[nr_pages], const void *source) { - return xg_foreignmemory_copy(xch, dom, foreigncopy_to, + return xg_foreignmemory_copy(xch, dom, XENMEM_foreigncopy_to, nr_pages, (void *) source, dest); }
Signature · 1 line
2.43.0
PATCH v6 15/16 libs/guest: finalize PoC
In reply to Frediano ZiglioRemove now unused map_errs array.
Test and restore verification code.
Report correctly errors from writev_exact.
Allocate verification buffer using hypercall buffer to avoid errors using hypercall.
--- tools/libs/guest/xg_sr_common.h | 4 +- tools/libs/guest/xg_sr_restore.c | 45 +++++++++++++++-- tools/libs/guest/xg_sr_save.c | 83 +++++++++++++++++++++----------- 3 files changed, 98 insertions(+), 34 deletions(-)
tools/libs/guest/xg_sr_common.h +2 -2
@@ -217,7 +217,6 @@ struct xc_sr_context_save_buffersvoid *local_pages[MAX_BATCH_SIZE ];struct iovec iov[MAX_BATCH_SIZE +2 ];/* Headers + data. */ uint64_t rec_pfns[MAX_BATCH_SIZE ]; -int errors[MAX_BATCH_SIZE ]; };struct xc_sr_context
@@ -255,8 +254,8 @@ struct xc_sr_contextunsigned long *deferred_pages;unsigned long nr_deferred_pages;xc_hypercall_buffer_t dirty_bitmap_hbuf; +xc_hypercall_buffer_t dest_buf;struct xc_sr_context_save_buffers *buffers; -void *dest_buf; } save; struct /* Restore data. */
@@ -267,6 +266,7 @@ struct xc_sr_contextint send_back_fd;unsigned long p2m_size;xc_hypercall_buffer_t dirty_bitmap_hbuf; +xc_hypercall_buffer_t verify_buf;/* From Image Header. */ uint32_t format_version;
tools/libs/guest/xg_sr_restore.c +40 -5
87 lines not on this page · raw message
tools/libs/guest/xg_sr_save.c +56 -27
141 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v6 15/16] libs/guest: finalize PoC
A note about the subject: When taken out of the context of this patch series (so once commited), we don't know what "PoC" is refering to. In "libs/guest: finalize PoC", it looks like "libxenguest" was the PoC, but it isn't.
It feels like this patch wants to be merged into that other PoC patch, and have a patch description completely rewritten to have something that doesn't looks like the description of an experiment.
Cheers,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: [PATCH v6 15/16] libs/guest: finalize PoC
Quoted 5 lines · Anthony PERARD
A note about the subject: When taken out of the context of this patch series (so once commited), we don't know what "PoC" is refering to. In "libs/guest: finalize PoC", it looks like "libxenguest" was the PoC, but it isn't.
It's referring to the previous "PoC: libs/guest: use foreign copy during migration" commit.
Quoted 4 lines
It feels like this patch wants to be merged into that other PoC patch, and have a patch description completely rewritten to have something that doesn't looks like the description of an experiment.
Definitively.
Quoted 3 lines
Cheers,
Frediano
PATCH v6 16/16 xen/privcmd: Add new ABI to allow copying foreign memory
In reply to Frediano ZiglioThis new ABI allows to copy foreign domain memory to/from a buffer. This avoids having to map/copy/unmap foreign memory which is expensive.
This operation is done particularly when migrating VMs.
--
Changes since v4:
- fix wrong assign;
- use set_xen_guest_handle to set handle;
- wrap slow hypercall with xen_preemptible_hcall_{begin,end};
- use _IOWR for ioctl code to be more specific;
- use __copy_to_user if buffer already checked.
--- arch/x86/include/asm/xen/interface.h | 1 + drivers/xen/privcmd.c | 49 ++++++++++++++++++++++++++++ include/uapi/xen/privcmd.h | 10 ++++++ include/xen/interface/memory.h | 37 +++++++++++++++++++++ 4 files changed, 97 insertions(+)
arch/x86/include/asm/xen/interface.h +1 -0
@@ -91,6 +91,7 @@ DEFINE_GUEST_HANDLE(int);DEFINE_GUEST_HANDLE (void);DEFINE_GUEST_HANDLE (uint64_t);DEFINE_GUEST_HANDLE (uint32_t); +DEFINE_GUEST_HANDLE (uint8_t);DEFINE_GUEST_HANDLE (xen_pfn_t);DEFINE_GUEST_HANDLE (xen_ulong_t);#endif
drivers/xen/privcmd.c +49 -0
61 lines not on this page · raw message
include/uapi/xen/privcmd.h +10 -0
@@ -131,6 +131,14 @@ struct privcmd_pcidev_get_gsi {__u32 gsi; }; +struct privcmd_foreigncopy { +domid_t dom;/* foreign domain */ +__u16 dir;/* direction, 0 from, 1 to */ +__u32 num;/* number of pages to copy */ +const xen_pfn_t __user *pfns;/* array of pfns */ +void __user *buffer;/* buffer to copy to/from */ +}; +/* * @cmd: IOCTL_PRIVCMD_HYPERCALL * @arg: &privcmd_hypercall_t
@@ -164,5 +172,7 @@ struct privcmd_pcidev_get_gsi { _IOW('P', 9, struct privcmd_ioeventfd)#define IOCTL_PRIVCMD_PCIDEV_GET_GSI \ _IOC(_IOC_NONE, 'P', 10, sizeof(struct privcmd_pcidev_get_gsi)) +#define IOCTL_PRIVCMD_FOREIGNCOPY \ + _IOWR('P', 11, struct privcmd_foreigncopy)#endif /* __LINUX_PUBLIC_PRIVCMD_H__ */
include/xen/interface/memory.h +37 -0
@@ -325,4 +325,41 @@ struct xen_mem_acquire_resource { };DEFINE_GUEST_HANDLE_STRUCT (xen_mem_acquire_resource); +/* +* Copy memory from/to a given domain. +*/ +#define XENMEM_foreigncopy 29 +struct xen_foreigncopy { +/* IN - The domain whose resource is to be copied */ +domid_t domid; + +/* IN - Flags */ +#define XENMEM_foreigncopy_from 0 +#define XENMEM_foreigncopy_to 1 +#define XENMEM_foreigncopy_direction 1 +uint16_t flags; + +/* +* IN +* +* As an IN parameter number of frames of the domain to be copied. +*/ +uint32_t nr_frames; + +/* +* IN +* +* Frames to be copied. +*/ +GUEST_HANDLE (xen_pfn_t) frame_list; + +/* +* IN/OUT +* +* Userspace buffer to read/write from. +*/ +GUEST_HANDLE (uint8_t ) buffer; +}; +DEFINE_GUEST_HANDLE_STRUCT (xen_foreigncopy); +#endif /* __XEN_PUBLIC_MEMORY_H__ */
Signature · 1 line
2.54.0
Re: [PATCH Linux v6 16/16] xen/privcmd: Add new ABI to allow copying foreign memory
Quoted 6 lines · Frediano Ziglio
This new ABI allows to copy foreign domain memory to/from a buffer. This avoids having to map/copy/unmap foreign memory which is expensive.
This operation is done particularly when migrating VMs.Signed-off-by Frediano Ziglio <[email protected]>
Juergen
Juergen Gross
maintainerRe: [PATCH Linux v6 16/16] xen/privcmd: Add new ABI to allow copying foreign memory
In reply to Frediano ZiglioQuoted 6 lines · Frediano Ziglio
This new ABI allows to copy foreign domain memory to/from a buffer. This avoids having to map/copy/unmap foreign memory which is expensive.
This operation is done particularly when migrating VMs.Signed-off-by Frediano Ziglio <[email protected]>
While doing a test build I got the following build failures for 32-bit arm:
CC arch/arm/xen/enlighten.o
In file included from /home/gross/korg/src/arch/arm/include/asm/xen/interface.h:1,
from /home/gross/korg/src/include/xen/interface/xen.h:13,
from /home/gross/korg/src/include/xen/xen.h:52,
from /home/gross/korg/src/arch/arm/xen/enlighten.c:2:
/home/gross/korg/src/include/xen/arm/interface.h:22:35: error: unknown type name '__guest_handle_uint8_t'
22 | #define GUEST_HANDLE(name) __guest_handle_ ## name
| ^~~~~~~~~~~~~~~
/home/gross/korg/src/include/xen/interface/memory.h:361:5: note: in expansion of macro 'GUEST_HANDLE'
361 | GUEST_HANDLE(uint8_t) buffer;
| ^~~~~~~~~~~~make[5]: *** [/home/gross/korg/src/scripts/Makefile.build:289: arch/arm/xen/enlighten.o] Error 1 make[4]: *** [/home/gross/korg/src/scripts/Makefile.build:549: arch/arm/xen] Error 2 make[3]: *** [/home/gross/korg/src/scripts/Makefile.build:549: arch/arm] Error 2
Please fix those.
Juergen
Re: [PATCH Linux v6 16/16] xen/privcmd: Add new ABI to allow copying foreign memory
Quoted 30 lines · Juergen Gross
Quoted 6 lines · Frediano Ziglio
This new ABI allows to copy foreign domain memory to/from a buffer. This avoids having to map/copy/unmap foreign memory which is expensive.
This operation is done particularly when migrating VMs.Signed-off-by Frediano Ziglio <[email protected]>While doing a test build I got the following build failures for 32-bit arm:
CC
arch/arm/xen/enlighten.o
In file included from/home/gross/korg/src/arch/arm/include/asm/xen/interface.h:1,from /home/gross/korg/src/include/xen/interface/xen.h:13, from /home/gross/korg/src/include/xen/xen.h:52, from /home/gross/korg/src/arch/arm/xen/enlighten.c:2: /home/gross/korg/src/include/xen/arm/interface.h:22:35: error: unknown type name '__guest_handle_uint8_t' 22 | #define GUEST_HANDLE(name) __guest_handle_ ## name | ^~~~~~~~~~~~~~~ /home/gross/korg/src/include/xen/interface/memory.h:361:5: note: in expansion of macro 'GUEST_HANDLE' 361 | GUEST_HANDLE(uint8_t) buffer; | ^~~~~~~~~~~~make[5]: *** [
/home/gross/korg/src/scripts/Makefile.build:289:arch/arm/xen/enlighten.o] Error 1
make[4]: *** [/home/gross/korg/src/scripts/Makefile.build:549:arch/arm/xen] Error 2 make[3]: *** [/home/gross/korg/src/scripts/Makefile.build:549:arch/arm] Error 2
Quoted 4 lines
Please fix those.
Juergen
Mumble...
I suppose this would fix it
include/xen/arm/interface.h +1 -0
@@ -53,6 +53,7 @@ DEFINE_GUEST_HANDLE(int);DEFINE_GUEST_HANDLE (void);DEFINE_GUEST_HANDLE (uint64_t);DEFINE_GUEST_HANDLE (uint32_t); +DEFINE_GUEST_HANDLE (uint8_t);DEFINE_GUEST_HANDLE (xen_pfn_t);DEFINE_GUEST_HANDLE (xen_ulong_t);
Frediano
Juergen Gross
maintainerRe: [PATCH Linux v6 16/16] xen/privcmd: Add new ABI to allow copying foreign memory
Quoted 52 lines · Frediano Ziglio
Quoted 30 lines · Juergen Gross
Quoted 6 lines · Frediano Ziglio
This new ABI allows to copy foreign domain memory to/from a buffer. This avoids having to map/copy/unmap foreign memory which is expensive.
This operation is done particularly when migrating VMs.Signed-off-by Frediano Ziglio <[email protected]>While doing a test build I got the following build failures for 32-bit arm:
CC
arch/arm/xen/enlighten.o
In file included from/home/gross/korg/src/arch/arm/include/asm/xen/interface.h:1,from /home/gross/korg/src/include/xen/interface/xen.h:13, from /home/gross/korg/src/include/xen/xen.h:52, from /home/gross/korg/src/arch/arm/xen/enlighten.c:2: /home/gross/korg/src/include/xen/arm/interface.h:22:35: error: unknown type name '__guest_handle_uint8_t' 22 | #define GUEST_HANDLE(name) __guest_handle_ ## name | ^~~~~~~~~~~~~~~ /home/gross/korg/src/include/xen/interface/memory.h:361:5: note: in expansion of macro 'GUEST_HANDLE' 361 | GUEST_HANDLE(uint8_t) buffer; | ^~~~~~~~~~~~make[5]: *** [
/home/gross/korg/src/scripts/Makefile.build:289:arch/arm/xen/enlighten.o] Error 1
make[4]: *** [/home/gross/korg/src/scripts/Makefile.build:549:arch/arm/xen] Error 2 make[3]: *** [/home/gross/korg/src/scripts/Makefile.build:549:arch/arm] Error 2Quoted 4 lines
Please fix those.
Juergen
Mumble...
I suppose this would fix it
include/xen/arm/interface.h +1
-0@@ -53,6 +53,7 @@ DEFINE_GUEST_HANDLE(int);DEFINE_GUEST_HANDLE (void);DEFINE_GUEST_HANDLE (uint64_t);DEFINE_GUEST_HANDLE (uint32_t); +DEFINE_GUEST_HANDLE (uint8_t);DEFINE_GUEST_HANDLE (xen_pfn_t);DEFINE_GUEST_HANDLE (xen_ulong_t);
Quoted 2 lines
Frediano
And now another one:
/home/gross/korg/src/drivers/xen/privcmd.c: In function 'privcmd_ioctl_foreigncopy':/home/gross/korg/src/drivers/xen/privcmd.c:1591:29: error: incompatible types when assigning to type 'const xen_pfn_t *' {aka 'const long long unsigned int *'} from type '__guest_handle_xen_pfn_t'
1591 | copy.pfns = xcopy.frame_list;
| ^~~~~/home/gross/korg/src/drivers/xen/privcmd.c:1592:31: error: incompatible types when assigning to type 'void *' from type '__guest_handle_uint8_t'
1592 | copy.buffer = xcopy.buffer;
| ^~~~~
Juergen