Skip to contentSkip to conversation

mm: fix public declaration of struct xen_mem_acquire_resource

PATCH Superseded Roger Pau Monne 4 participants 26 messages Jun 23, 2020 – Jun 26, 2020

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: fix public declaration of struct xen_mem_acquire_resource

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

No functional change expected.

Fixes 3f8f12281dd20 ('x86/mm: add HYPERVISOR_memory_op to acquire guest resources')

---
Would be good to get this fixed before the release in order to avoid shipping bogus headers. Should also be backported.

---
 xen/include/public/memory.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
xen/include/public/memory.h +3 -3
index dbd35305df..1767d7d5f5 100644
@@ -607,6 +607,8 @@ struct xen_reserved_device_memory_map {
 typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
 DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
 
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 /*
  * Get the pages for a particular guest resource, so that they can be
  * mapped directly by a tools domain.
@@ -645,7 +647,7 @@ struct xen_mem_acquire_resource {
      * IN - the index of the initial frame to be mapped. This parameter
      *      is ignored if nr_frames is 0.
      */
-    uint64_aligned_t frame;
+    uint64_t frame;
 
 #define XENMEM_resource_ioreq_server_frame_bufioreq 0
 #define XENMEM_resource_ioreq_server_frame_ioreq(n) (1 + (n))
@@ -666,8 +668,6 @@ struct xen_mem_acquire_resource {
 typedef struct xen_mem_acquire_resource xen_mem_acquire_resource_t;
 DEFINE_XEN_GUEST_HANDLE(xen_mem_acquire_resource_t);
 
-#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
-
 /*
  * XENMEM_get_vnumainfo used by guest to get
  * vNUMA topology from hypervisor.
Signature · 1 line
2.26.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 21 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 23 June 2020 14:53
To: [email protected]
Cc: [email protected]; Roger Pau Monne <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH for-4.14] mm: fix public declaration of struct xen_mem_acquire_resource

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

Quoted 7 lines

No functional change expected.

Fixes 3f8f12281dd20 ('x86/mm: add HYPERVISOR_memory_op to acquire guest resources')

---
Would be good to get this fixed before the release in order to avoid shipping bogus headers. Should also be backported.

Quoted 37 lines
---
 xen/include/public/memory.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
xen/include/public/memory.h +3 -4
index dbd35305df..1767d7d5f5 100644
@@ -607,6 +607,8 @@ struct xen_reserved_device_memory_map {
 typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
 DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
 
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 /*
  * Get the pages for a particular guest resource, so that they can be
  * mapped directly by a tools domain.
@@ -645,7 +647,7 @@ struct xen_mem_acquire_resource {
      * IN - the index of the initial frame to be mapped. This parameter
      *      is ignored if nr_frames is 0.
      */
-    uint64_aligned_t frame;
+    uint64_t frame;
 
 #define XENMEM_resource_ioreq_server_frame_bufioreq 0
 #define XENMEM_resource_ioreq_server_frame_ioreq(n) (1 + (n))
@@ -666,8 +668,6 @@ struct xen_mem_acquire_resource {
 typedef struct xen_mem_acquire_resource xen_mem_acquire_resource_t;
 DEFINE_XEN_GUEST_HANDLE(xen_mem_acquire_resource_t);
 
-#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
-
 /*
  * XENMEM_get_vnumainfo used by guest to get
  * vNUMA topology from hypervisor.
--

2.26.2

Reviewed-by on v1 1/1
#2 lore raw
In reply to Roger Pau Monne
Quoted 11 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

Quoted 4 lines

No functional change expected.

Fixes 3f8f12281dd20 ('x86/mm: add HYPERVISOR_memory_op to acquire guest resources')
Quoted 39 lines

---
Would be good to get this fixed before the release in order to avoid shipping bogus headers. Should also be backported.

---
  xen/include/public/memory.h | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
xen/include/public/memory.h +3 -3
index dbd35305df..1767d7d5f5 100644
@@ -607,6 +607,8 @@ struct xen_reserved_device_memory_map {
  typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
  DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
 
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
  /*
   * Get the pages for a particular guest resource, so that they can be
   * mapped directly by a tools domain.
@@ -645,7 +647,7 @@ struct xen_mem_acquire_resource {
       * IN - the index of the initial frame to be mapped. This parameter
       *      is ignored if nr_frames is 0.
       */
-    uint64_aligned_t frame;
+    uint64_t frame;
 
  #define XENMEM_resource_ioreq_server_frame_bufioreq 0
  #define XENMEM_resource_ioreq_server_frame_ioreq(n) (1 + (n))
@@ -666,8 +668,6 @@ struct xen_mem_acquire_resource {
  typedef struct xen_mem_acquire_resource xen_mem_acquire_resource_t;
  DEFINE_XEN_GUEST_HANDLE(xen_mem_acquire_resource_t);
 
-#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
-
  /*
   * XENMEM_get_vnumainfo used by guest to get
   * vNUMA topology from hypervisor.
Signature · 1 line
Julien Grall
#3 lore raw
In reply to Roger Pau Monne
Quoted 10 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

I'm afraid it's more complicated, and hence ...

Quoted 1 line

No functional change expected.

... this doesn't hold: As I notice only now the struct also wrongly (if it was meant to be a tools-only interface) failed to use XEN_GUEST_HANDLE_64(), which would in principle have been a problem for 32-bit callers (passing garbage in the upper half of a handle). It's not an actual problem because, unlike would typically be the case for tools-only interfaces, there is compat translation for this struct.

With this, however, the problem of your change becomes noticeable:
The structure's alignment for 32-bit x86, and with it the structure's sizeof(), both change. You should be able to see this by comparing xen/common/compat/memory.c's disassembly before and after your change - the number of bytes to copy by the respective copy_from_guest() ought to have changed.

The question now is whether we're willing to accept such a (marginal) incompatibility. The chance of a 32-bit guest actually running into it shouldn't be very high, but with the right placement of an instance of the struct at the end of a page it would be possible to observe afaict.

Or whether we go the alternative route and pad the struct suitably for 32-bit.

Quoted 5 lines
Fixes 3f8f12281dd20 ('x86/mm: add HYPERVISOR_memory_op to acquire guest resources')

---
Would be good to get this fixed before the release in order to avoid shipping bogus headers. Should also be backported.

This was already part of 4.13, as you imply by requesting a backport. Hence the bogus header had already been shipped.

xen/include/public/memory.h struct xen_reserved_device_memory_map { · line 614

Quoted 11 lines
xen/include/public/memory.h +2 -0
@@ -607,6 +607,8 @@ struct xen_reserved_device_memory_map {
 typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
 DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
 
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 /*
  * Get the pages for a particular guest resource, so that they can be
  * mapped directly by a tools domain.

This comment is now stale.

Jan

#4 lore raw
Quoted 16 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

I'm afraid it's more complicated, and hence ...

Quoted 1 line

No functional change expected.

Quoted 7 lines

... this doesn't hold: As I notice only now the struct also wrongly (if it was meant to be a tools-only interface) failed to use XEN_GUEST_HANDLE_64(), which would in principle have been a problem for 32-bit callers (passing garbage in the upper half of a handle). It's not an actual problem because, unlike would typically be the case for tools-only interfaces, there is compat translation for this struct.

Yes, there's already compat translation for the frame_list array.

Quoted 6 lines

With this, however, the problem of your change becomes noticeable:
The structure's alignment for 32-bit x86, and with it the structure's sizeof(), both change. You should be able to see this by comparing xen/common/compat/memory.c's disassembly before and after your change - the number of bytes to copy by the respective copy_from_guest() ought to have changed.

Right, this is the layout with frame aligned to 8 bytes:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 32, cachelines: 1, members: 7 */
	/* padding: 4 */
	/* last cacheline: 32 bytes */
};

And this is without:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 28, cachelines: 1, members: 7 */
	/* last cacheline: 28 bytes */
};

Note Xen has already been copying those extra 4 padding bytes from 32bit Linux kernels AFAICT, as the struct declaration in Linux has dropped the aligned attribute.

Quoted 8 lines

The question now is whether we're willing to accept such a (marginal) incompatibility. The chance of a 32-bit guest actually running into it shouldn't be very high, but with the right placement of an instance of the struct at the end of a page it would be possible to observe afaict.

Or whether we go the alternative route and pad the struct suitably for 32-bit.

I guess we are trapped with what Linux has on it's headers now, so we should handle the struct size difference in Xen?

I'm very tempted to just say switch the XEN_GUEST_HANDLE to XEN_GUEST_HANDLE_64, but I guess it's risky. I'm not sure how much people still run a toolstack from a 32bit domain however.

Quoted 8 lines
Quoted 5 lines
Fixes 3f8f12281dd20 ('x86/mm: add HYPERVISOR_memory_op to acquire guest resources')

---
Would be good to get this fixed before the release in order to avoid shipping bogus headers. Should also be backported.

This was already part of 4.13, as you imply by requesting a backport. Hence the bogus header had already been shipped.

Indeed. I would however try to prevent from shipping it in 4.14.

Quoted 6 lines
Quoted 6 lines
xen/include/public/memory.h +0 -0
@@ -607,6 +607,8 @@ struct xen_reserved_device_memory_map {
 typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
 DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
 
Quoted 7 lines

xen/include/public/memory.h struct xen_reserved_device_memory_map { · line 614

Quoted 5 lines
xen/include/public/memory.h +2 -0
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 /*
  * Get the pages for a particular guest resource, so that they can be
  * mapped directly by a tools domain.

This comment is now stale.

Hm, I'm not so sure. This is indeed used by Xen related tools (or emulators) and then those domains running such tools would also be tools domains?

Anyway, I don't mind removing it, but I also don't think it's so off.

Roger.

#5 lore raw
Quoted 75 lines · Roger Pau Monné
Quoted 16 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

I'm afraid it's more complicated, and hence ...

Quoted 1 line

No functional change expected.

Quoted 7 lines

... this doesn't hold: As I notice only now the struct also wrongly (if it was meant to be a tools-only interface) failed to use XEN_GUEST_HANDLE_64(), which would in principle have been a problem for 32-bit callers (passing garbage in the upper half of a handle). It's not an actual problem because, unlike would typically be the case for tools-only interfaces, there is compat translation for this struct.

Yes, there's already compat translation for the frame_list array.

Quoted 6 lines

With this, however, the problem of your change becomes noticeable:
The structure's alignment for 32-bit x86, and with it the structure's sizeof(), both change. You should be able to see this by comparing xen/common/compat/memory.c's disassembly before and after your change - the number of bytes to copy by the respective copy_from_guest() ought to have changed.

Right, this is the layout with frame aligned to 8 bytes:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 32, cachelines: 1, members: 7 */
	/* padding: 4 */
	/* last cacheline: 32 bytes */
};

And this is without:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 28, cachelines: 1, members: 7 */
	/* last cacheline: 28 bytes */
};

Note Xen has already been copying those extra 4 padding bytes from 32bit Linux kernels AFAICT, as the struct declaration in Linux has dropped the aligned attribute.

Quoted 6 lines

The question now is whether we're willing to accept such a (marginal) incompatibility. The chance of a 32-bit guest actually running into it shouldn't be very high, but with the right placement of an instance of the struct at the end of a page it would be possible to observe afaict.

Quoted 5 lines
Quoted 2 lines

Or whether we go the alternative route and pad the struct suitably for 32-bit.

I guess we are trapped with what Linux has on it's headers now, so we should handle the struct size difference in Xen?

How do you mean to notice the difference in Xen? You don't know what the caller's environment's sizeof() would yield.

Quoted 2 lines

I'm very tempted to just say switch the XEN_GUEST_HANDLE to XEN_GUEST_HANDLE_64, but I guess it's risky.

Plus, like uint64_aligned_t, iirc it's a construct exposed through tool-stack-only interfaces, but not generally.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
xen/include/public/memory.h +0 -0
@@ -607,6 +607,8 @@ struct xen_reserved_device_memory_map {
 typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
 DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
 
Quoted 7 lines

xen/include/public/memory.h struct xen_reserved_device_memory_map { · line 614

Quoted 5 lines
xen/include/public/memory.h +2 -0
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 /*
  * Get the pages for a particular guest resource, so that they can be
  * mapped directly by a tools domain.

This comment is now stale.

Quoted 6 lines

Hm, I'm not so sure. This is indeed used by Xen related tools (or emulators) and then those domains running such tools would also be tools domains?

Anyway, I don't mind removing it, but I also don't think it's so off.

Well - if this isn't a tool stack interface (see also my 2nd reply to your original patch), then the comment shouldn't suggest it is.

Jan

#6 lore raw
Quoted 76 lines · Jan Beulich
Quoted 75 lines · Roger Pau Monné
Quoted 16 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

I'm afraid it's more complicated, and hence ...

Quoted 1 line

No functional change expected.

Quoted 7 lines

... this doesn't hold: As I notice only now the struct also wrongly (if it was meant to be a tools-only interface) failed to use XEN_GUEST_HANDLE_64(), which would in principle have been a problem for 32-bit callers (passing garbage in the upper half of a handle). It's not an actual problem because, unlike would typically be the case for tools-only interfaces, there is compat translation for this struct.

Yes, there's already compat translation for the frame_list array.

Quoted 6 lines

With this, however, the problem of your change becomes noticeable:
The structure's alignment for 32-bit x86, and with it the structure's sizeof(), both change. You should be able to see this by comparing xen/common/compat/memory.c's disassembly before and after your change - the number of bytes to copy by the respective copy_from_guest() ought to have changed.

Right, this is the layout with frame aligned to 8 bytes:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 32, cachelines: 1, members: 7 */
	/* padding: 4 */
	/* last cacheline: 32 bytes */
};

And this is without:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 28, cachelines: 1, members: 7 */
	/* last cacheline: 28 bytes */
};

Note Xen has already been copying those extra 4 padding bytes from 32bit Linux kernels AFAICT, as the struct declaration in Linux has dropped the aligned attribute.

Quoted 6 lines

The question now is whether we're willing to accept such a (marginal) incompatibility. The chance of a 32-bit guest actually running into it shouldn't be very high, but with the right placement of an instance of the struct at the end of a page it would be possible to observe afaict.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Or whether we go the alternative route and pad the struct suitably for 32-bit.

I guess we are trapped with what Linux has on it's headers now, so we should handle the struct size difference in Xen?

How do you mean to notice the difference in Xen? You don't know what the caller's environment's sizeof() would yield.

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

This would of course assume that no toolstack has implemented direct calls using this interface, which seems likely because it either returns mfns to be mapped in the PV case or require gfns to be provided for HVM.

Quoted 5 lines
Quoted 2 lines

I'm very tempted to just say switch the XEN_GUEST_HANDLE to XEN_GUEST_HANDLE_64, but I guess it's risky.

Plus, like uint64_aligned_t, iirc it's a construct exposed through tool-stack-only interfaces, but not generally.

Ack.

Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
xen/include/public/memory.h +0 -0
@@ -607,6 +607,8 @@ struct xen_reserved_device_memory_map {
 typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
 DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
 
Quoted 7 lines

xen/include/public/memory.h struct xen_reserved_device_memory_map { · line 614

Quoted 5 lines
xen/include/public/memory.h +2 -0
+#endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
+
 /*
  * Get the pages for a particular guest resource, so that they can be
  * mapped directly by a tools domain.

This comment is now stale.

Hm, I'm not so sure. This is indeed used by Xen related tools (or emulators) and then those domains running such tools would also be tools domains?

Quoted 5 lines
Quoted 2 lines

Anyway, I don't mind removing it, but I also don't think it's so off.

Well - if this isn't a tool stack interface (see also my 2nd reply to your original patch), then the comment shouldn't suggest it is.

OK, let me take a look.

Roger.

#7 lore raw
Quoted 83 lines · Roger Pau Monné
Quoted 82 lines · Jan Beulich
Quoted 80 lines · Roger Pau Monné
Quoted 16 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

I'm afraid it's more complicated, and hence ...

Quoted 1 line

No functional change expected.

Quoted 7 lines

... this doesn't hold: As I notice only now the struct also wrongly (if it was meant to be a tools-only interface) failed to use XEN_GUEST_HANDLE_64(), which would in principle have been a problem for 32-bit callers (passing garbage in the upper half of a handle). It's not an actual problem because, unlike would typically be the case for tools-only interfaces, there is compat translation for this struct.

Yes, there's already compat translation for the frame_list array.

Quoted 6 lines

With this, however, the problem of your change becomes noticeable:
The structure's alignment for 32-bit x86, and with it the structure's sizeof(), both change. You should be able to see this by comparing xen/common/compat/memory.c's disassembly before and after your change - the number of bytes to copy by the respective copy_from_guest() ought to have changed.

Right, this is the layout with frame aligned to 8 bytes:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 32, cachelines: 1, members: 7 */
	/* padding: 4 */
	/* last cacheline: 32 bytes */
};

And this is without:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 28, cachelines: 1, members: 7 */
	/* last cacheline: 28 bytes */
};

Note Xen has already been copying those extra 4 padding bytes from 32bit Linux kernels AFAICT, as the struct declaration in Linux has dropped the aligned attribute.

Quoted 8 lines

The question now is whether we're willing to accept such a (marginal) incompatibility. The chance of a 32-bit guest actually running into it shouldn't be very high, but with the right placement of an instance of the struct at the end of a page it would be possible to observe afaict.

Or whether we go the alternative route and pad the struct suitably for 32-bit.

I guess we are trapped with what Linux has on it's headers now, so we should handle the struct size difference in Xen?

Quoted 7 lines
Quoted 2 lines

How do you mean to notice the difference in Xen? You don't know what the caller's environment's sizeof() would yield.

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question), then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 4 lines

This would of course assume that no toolstack has implemented direct calls using this interface, which seems likely because it either returns mfns to be mapped in the PV case or require gfns to be provided for HVM.

What are "direct calls" in the case here? We aren't talking about a dmop interface, but a memop one here, are we? It's merely an interface that an ioreq server implementation ought to use to get hold of the ring pages (i.e. there's only a weak connection to dmop).

Jan

#8 lore raw

Hi Jan,

Quoted 87 lines · Jan Beulich
Quoted 86 lines · Roger Pau Monné
Quoted 84 lines · Jan Beulich
Quoted 75 lines · Roger Pau Monné
Quoted 16 lines · Jan Beulich
Quoted 10 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Also switch the usage of uint64_aligned_t to plain uint64_t, as uint64_aligned_t is only to be used by the toolstack. Note that the layout of the structure will be the same, as the field is already naturally aligned to a 8 byte boundary.

I'm afraid it's more complicated, and hence ...

Quoted 1 line

No functional change expected.

Quoted 7 lines

... this doesn't hold: As I notice only now the struct also wrongly (if it was meant to be a tools-only interface) failed to use XEN_GUEST_HANDLE_64(), which would in principle have been a problem for 32-bit callers (passing garbage in the upper half of a handle). It's not an actual problem because, unlike would typically be the case for tools-only interfaces, there is compat translation for this struct.

Yes, there's already compat translation for the frame_list array.

Quoted 6 lines

With this, however, the problem of your change becomes noticeable:
The structure's alignment for 32-bit x86, and with it the structure's sizeof(), both change. You should be able to see this by comparing xen/common/compat/memory.c's disassembly before and after your change - the number of bytes to copy by the respective copy_from_guest() ought to have changed.

Right, this is the layout with frame aligned to 8 bytes:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 32, cachelines: 1, members: 7 */
	/* padding: 4 */
	/* last cacheline: 32 bytes */
};

And this is without:

struct xen_mem_acquire_resource {

	uint16_t                   domid;                /*     0     2 */
	uint16_t                   type;                 /*     2     2 */
	uint32_t                   id;                   /*     4     4 */
	uint32_t                   nr_frames;            /*     8     4 */
	uint32_t                   pad;                  /*    12     4 */
	uint64_t                   frame;                /*    16     8 */
	long unsigned int *        frame_list;           /*    24     4 */
	/* size: 28, cachelines: 1, members: 7 */
	/* last cacheline: 28 bytes */
};

Note Xen has already been copying those extra 4 padding bytes from 32bit Linux kernels AFAICT, as the struct declaration in Linux has dropped the aligned attribute.

Quoted 6 lines

The question now is whether we're willing to accept such a (marginal) incompatibility. The chance of a 32-bit guest actually running into it shouldn't be very high, but with the right placement of an instance of the struct at the end of a page it would be possible to observe afaict.

Quoted 5 lines
Quoted 2 lines

Or whether we go the alternative route and pad the struct suitably for 32-bit.

I guess we are trapped with what Linux has on it's headers now, so we should handle the struct size difference in Xen?

How do you mean to notice the difference in Xen? You don't know what the caller's environment's sizeof() would yield.

Quoted 8 lines
Quoted 4 lines

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

Cheers,

Signature · 1 line
Julien Grall
#9 lore raw
Quoted 11 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

Quoted 8 lines

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

Jan

#10 lore raw
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 8 lines
Quoted 2 lines

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

Adding the padding is cleaner because we don't need any compat stuff in order to access the structure from the caller, and we also keep the original layout currently present on Xen headers.

I can prepare a fix for the Linux kernel, if this approach is fine.

Roger.

#11 lore raw
Quoted 28 lines · Roger Pau Monné
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 8 lines
Quoted 2 lines

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

Quoted 5 lines

Adding the padding is cleaner because we don't need any compat stuff in order to access the structure from the caller, and we also keep the original layout currently present on Xen headers.

I can prepare a fix for the Linux kernel, if this approach is fine.

So I went over this, and I'm not sure the point of adding the padding field at the end of the structure for 32bit x86.

The current situation is the following:

- Linux will use a struct on 32bit x86 that doesn't have the 4byte padding at the end.
- Xen will copy 4bytes of garbage in that case, since the struct on Linux is allocated on the stack.

So I suggest we take the approach found on this patch, that is remove the 8byte alignment from the frame field, which will in turn remove 4bytes of padding from the tail of the structure on 32bit x86.

That would leave the following scenario:

- The struct layout in Linux headers would be correct.
- Xen already handles the struct size difference on x86 32bit vs 64bit, as the compat layer is currently doing the copy in compat_memory_op taking into account the size of the compat structure.
- Removing the padding will work for all use cases: Linux will already be using the correct layout on x86 32bits, so no change will be required there. Any consumers using the tail padded structure will continue to work without issues, as Xen simply won't copy the tailing 4bytes.

So I think the solution proposed in this patch is the correct one:
switch uint64_aligned_t to uint64_t, no tail padding added on x86 32bits. I will adjust the commit message and resubmit if that's fine.

Roger.

#12 lore raw
Quoted 49 lines · Roger Pau Monné
Quoted 28 lines · Roger Pau Monné
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 8 lines
Quoted 2 lines

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

Quoted 5 lines

Adding the padding is cleaner because we don't need any compat stuff in order to access the structure from the caller, and we also keep the original layout currently present on Xen headers.

I can prepare a fix for the Linux kernel, if this approach is fine.

So I went over this, and I'm not sure the point of adding the padding field at the end of the structure for 32bit x86.

The current situation is the following:

- Linux will use a struct on 32bit x86 that doesn't have the 4byte padding at the end.
- Xen will copy 4bytes of garbage in that case, since the struct on Linux is allocated on the stack.

So I suggest we take the approach found on this patch, that is remove the 8byte alignment from the frame field, which will in turn remove 4bytes of padding from the tail of the structure on 32bit x86.

Quoted 7 lines

That would leave the following scenario:

- The struct layout in Linux headers would be correct.
- Xen already handles the struct size difference on x86 32bit vs 64bit, as the compat layer is currently doing the copy in compat_memory_op taking into account the size of the compat structure.

Hmm, I didn't even notice this until now - it looks to do so indeed, but apparently because of a bug: The original uint64_aligned_t gets translated to mere uint64_t in the compat header, whereas it should have been retained. This means that my concern of ...

Quoted 5 lines

- Removing the padding will work for all use cases: Linux will already be using the correct layout on x86 32bits, so no change will be required there. Any consumers using the tail padded structure will continue to work without issues, as Xen simply won't copy the tailing 4bytes.

... code using the new definition then potentially not working correctly on 4.13, at least on versions not having this backported, was not actually true.

I'll try to sort out this other bug then ...

Quoted 3 lines

So I think the solution proposed in this patch is the correct one:
switch uint64_aligned_t to uint64_t, no tail padding added on x86 32bits. I will adjust the commit message and resubmit if that's fine.

I think it is indeed.

Jan

#13 lore raw
Quoted 70 lines · Jan Beulich
Quoted 49 lines · Roger Pau Monné
Quoted 28 lines · Roger Pau Monné
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 8 lines
Quoted 2 lines

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

Quoted 5 lines

Adding the padding is cleaner because we don't need any compat stuff in order to access the structure from the caller, and we also keep the original layout currently present on Xen headers.

I can prepare a fix for the Linux kernel, if this approach is fine.

So I went over this, and I'm not sure the point of adding the padding field at the end of the structure for 32bit x86.

The current situation is the following:

- Linux will use a struct on 32bit x86 that doesn't have the 4byte padding at the end.
- Xen will copy 4bytes of garbage in that case, since the struct on Linux is allocated on the stack.

So I suggest we take the approach found on this patch, that is remove the 8byte alignment from the frame field, which will in turn remove 4bytes of padding from the tail of the structure on 32bit x86.

Quoted 7 lines

That would leave the following scenario:

- The struct layout in Linux headers would be correct.
- Xen already handles the struct size difference on x86 32bit vs 64bit, as the compat layer is currently doing the copy in compat_memory_op taking into account the size of the compat structure.

Hmm, I didn't even notice this until now - it looks to do so indeed, but apparently because of a bug: The original uint64_aligned_t gets translated to mere uint64_t in the compat header, whereas it should have been retained. This means that my concern of ...

Quoted 5 lines

- Removing the padding will work for all use cases: Linux will already be using the correct layout on x86 32bits, so no change will be required there. Any consumers using the tail padded structure will continue to work without issues, as Xen simply won't copy the tailing 4bytes.

Quoted 5 lines

... code using the new definition then potentially not working correctly on 4.13, at least on versions not having this backported, was not actually true.

I'll try to sort out this other bug then ...

I was wrong, there is no bug - translating uint64_aligned_t to uint64_t is fine, as these are seen only by 64-bit code, where both are identical anyway. Hence there still is the concern that code working fine on the supposed 4.14 might then not work on unfixed 4.13, due to 4.13 copying 4 extra bytes.

Jan

#14 lore raw
Quoted 75 lines · Jan Beulich
Quoted 74 lines · Jan Beulich
Quoted 49 lines · Roger Pau Monné
Quoted 28 lines · Roger Pau Monné
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 8 lines
Quoted 2 lines

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

Quoted 5 lines

Adding the padding is cleaner because we don't need any compat stuff in order to access the structure from the caller, and we also keep the original layout currently present on Xen headers.

I can prepare a fix for the Linux kernel, if this approach is fine.

So I went over this, and I'm not sure the point of adding the padding field at the end of the structure for 32bit x86.

The current situation is the following:

- Linux will use a struct on 32bit x86 that doesn't have the 4byte padding at the end.
- Xen will copy 4bytes of garbage in that case, since the struct on Linux is allocated on the stack.

So I suggest we take the approach found on this patch, that is remove the 8byte alignment from the frame field, which will in turn remove 4bytes of padding from the tail of the structure on 32bit x86.

Quoted 7 lines

That would leave the following scenario:

- The struct layout in Linux headers would be correct.
- Xen already handles the struct size difference on x86 32bit vs 64bit, as the compat layer is currently doing the copy in compat_memory_op taking into account the size of the compat structure.

Hmm, I didn't even notice this until now - it looks to do so indeed, but apparently because of a bug: The original uint64_aligned_t gets translated to mere uint64_t in the compat header, whereas it should have been retained. This means that my concern of ...

Quoted 5 lines

- Removing the padding will work for all use cases: Linux will already be using the correct layout on x86 32bits, so no change will be required there. Any consumers using the tail padded structure will continue to work without issues, as Xen simply won't copy the tailing 4bytes.

... code using the new definition then potentially not working correctly on 4.13, at least on versions not having this backported, was not actually true.

Quoted 7 lines
Quoted 1 line

I'll try to sort out this other bug then ...

I was wrong, there is no bug - translating uint64_aligned_t to uint64_t is fine, as these are seen only by 64-bit code, where both are identical anyway. Hence there still is the concern that code working fine on the supposed 4.14 might then not work on unfixed 4.13, due to 4.13 copying 4 extra bytes.

So here are the structures on 64bit x86 according to pahole against xen-syms:

struct xen_mem_acquire_resource {

        domid_t                    domid;                /*     0     2 */
        uint16_t                   type;                 /*     2     2 */
        uint32_t                   id;                   /*     4     4 */
        uint32_t                   nr_frames;            /*     8     4 */
        uint32_t                   pad;                  /*    12     4 */
        uint64_t                   frame;                /*    16     8 */
        __guest_handle_xen_pfn_t   frame_list;           /*    24     8 */
        /* size: 32, cachelines: 1, members: 7 */
        /* last cacheline: 32 bytes */
};

struct compat_mem_acquire_resource {

        domid_compat_t             domid;                /*     0     2 */
        uint16_t                   type;                 /*     2     2 */
        uint32_t                   id;                   /*     4     4 */
        uint32_t                   nr_frames;            /*     8     4 */
        uint32_t                   pad;                  /*    12     4 */
        uint64_t                   frame;                /*    16     8 */
        __compat_handle_compat_pfn_t frame_list;         /*    24     4 */
        /* size: 28, cachelines: 1, members: 7 */
        /* last cacheline: 28 bytes */
};

There's no tailing padding on the compat struct ATM, and hence the current code will behave correctly when used against a compat structure without the tailing padding (as it's already ignored).

There's a #pragma pack(4) at the top of compat/memory.h which forces this AFAICT. So I think the suggested approach is fine and will avoid any breakage.

Roger.

#15 lore raw
Quoted 113 lines · Roger Pau Monné
Quoted 75 lines · Jan Beulich
Quoted 74 lines · Jan Beulich
Quoted 49 lines · Roger Pau Monné
Quoted 28 lines · Roger Pau Monné
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 8 lines
Quoted 2 lines

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

Quoted 5 lines

Adding the padding is cleaner because we don't need any compat stuff in order to access the structure from the caller, and we also keep the original layout currently present on Xen headers.

I can prepare a fix for the Linux kernel, if this approach is fine.

So I went over this, and I'm not sure the point of adding the padding field at the end of the structure for 32bit x86.

The current situation is the following:

- Linux will use a struct on 32bit x86 that doesn't have the 4byte padding at the end.
- Xen will copy 4bytes of garbage in that case, since the struct on Linux is allocated on the stack.

So I suggest we take the approach found on this patch, that is remove the 8byte alignment from the frame field, which will in turn remove 4bytes of padding from the tail of the structure on 32bit x86.

Quoted 7 lines

That would leave the following scenario:

- The struct layout in Linux headers would be correct.
- Xen already handles the struct size difference on x86 32bit vs 64bit, as the compat layer is currently doing the copy in compat_memory_op taking into account the size of the compat structure.

Hmm, I didn't even notice this until now - it looks to do so indeed, but apparently because of a bug: The original uint64_aligned_t gets translated to mere uint64_t in the compat header, whereas it should have been retained. This means that my concern of ...

Quoted 5 lines

- Removing the padding will work for all use cases: Linux will already be using the correct layout on x86 32bits, so no change will be required there. Any consumers using the tail padded structure will continue to work without issues, as Xen simply won't copy the tailing 4bytes.

... code using the new definition then potentially not working correctly on 4.13, at least on versions not having this backported, was not actually true.

Quoted 7 lines
Quoted 1 line

I'll try to sort out this other bug then ...

I was wrong, there is no bug - translating uint64_aligned_t to uint64_t is fine, as these are seen only by 64-bit code, where both are identical anyway. Hence there still is the concern that code working fine on the supposed 4.14 might then not work on unfixed 4.13, due to 4.13 copying 4 extra bytes.

So here are the structures on 64bit x86 according to pahole against xen-syms:

struct xen_mem_acquire_resource {

        domid_t                    domid;                /*     0     2 */
        uint16_t                   type;                 /*     2     2 */
        uint32_t                   id;                   /*     4     4 */
        uint32_t                   nr_frames;            /*     8     4 */
        uint32_t                   pad;                  /*    12     4 */
        uint64_t                   frame;                /*    16     8 */
        __guest_handle_xen_pfn_t   frame_list;           /*    24     8 */
        /* size: 32, cachelines: 1, members: 7 */
        /* last cacheline: 32 bytes */
};

struct compat_mem_acquire_resource {

        domid_compat_t             domid;                /*     0     2 */
        uint16_t                   type;                 /*     2     2 */
        uint32_t                   id;                   /*     4     4 */
        uint32_t                   nr_frames;            /*     8     4 */
        uint32_t                   pad;                  /*    12     4 */
        uint64_t                   frame;                /*    16     8 */
        __compat_handle_compat_pfn_t frame_list;         /*    24     4 */
        /* size: 28, cachelines: 1, members: 7 */
        /* last cacheline: 28 bytes */
};
Quoted 7 lines

There's no tailing padding on the compat struct ATM, and hence the current code will behave correctly when used against a compat structure without the tailing padding (as it's already ignored).

There's a #pragma pack(4) at the top of compat/memory.h which forces this AFAICT. So I think the suggested approach is fine and will avoid any breakage.

Oh, so I was mislead to believe there's no bug with the uint64_aligned_t handling because, after having it made survive the compat header generation, the generated code didn't change. But that's only because the aligned() attribute has no effect in a #pragma pack() region, not because things work as intended. So indeed, another 180° turn later, I again agree your change - with an extended description - ought to be fine. The bug we'll have to deal with has become more difficult now,
though: We can't use #pragma pack() then, but we also can't attach packed attributes to the structs and unions, as that would force 1-byte packing instead of 4-byte one.

Jan

#16 lore raw
In reply to Jan Beulich

Hi Jan,

Quoted 18 lines · Jan Beulich
Quoted 17 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Roger Pau Monné

I'm confused. Couldn't we switch from uint64_aligned_t to plain uint64_t (like it's currently on the Linux headers), and then use the compat layer in Xen to handle the size difference when called from 32bit environments?

And which size would we use there? The old or the new one (breaking future or existing callers respectively)? Meanwhile I think that if this indeed needs to not be tools-only (which I still question),

I think we now agreed on a subthread that the kernel needs to know the layout of the hypercall.

Quoted 2 lines

then our only possible route is to add explicit padding for the 32-bit case alongside the change you're already making.

Quoted 8 lines
Quoted 2 lines

AFAICT Linux 32-bit doesn't have this padding. So wouldn't it make incompatible the two incompatible?

In principle yes. But they're putting the structure instance on the stack, so there's not risk from Xen reading 4 bytes too many. I'd prefer keeping the interface as is (i.e. with the previously implicit padding made explicit) to avoid risking to break other possible callers. But that's just my view on it, anyway ...

It makes sense to me. As Roger said, we would also want to update the Linux kernel header for future release.

Cheers,

Signature · 1 line
Julien Grall
#17 lore raw
In reply to Roger Pau Monne
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

    rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
    if ( rc )
        return rc;

    rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
    if ( rc )
        goto out;

in the function?

Jan

#18 lore raw
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

    rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
    if ( rc )
        return rc;

    rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
    if ( rc )
        goto out;
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Roger.

#19 lore raw
Quoted 18 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

    rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
    if ( rc )
        return rc;

    rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
    if ( rc )
        goto out;
Quoted 7 lines
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Yes, but that's a tool stack operation, not something the kernel would do all by itself. The kernel would only ever pass DOMID_SELF (or the actual local domain ID), I would think.

Jan

#20 lore raw

Hi Jan,

Quoted 27 lines · Jan Beulich
Quoted 18 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

     rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
     if ( rc )
         return rc;

     rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
     if ( rc )
         goto out;
Quoted 7 lines
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Quoted 3 lines

Yes, but that's a tool stack operation, not something the kernel would do all by itself. The kernel would only ever pass DOMID_SELF (or the actual local domain ID), I would think.

You can't issue that hypercall directly from userspace because you need to map the page in the physical address space of the toolstack domain.

So the kernel has to act as the proxy for the hypercall. This is implemented as mmap() in Linux.

Cheers,

Signature · 1 line
Julien Grall
#21 lore raw
Quoted 34 lines · Julien Grall

Hi Jan,

Quoted 27 lines · Jan Beulich
Quoted 18 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

     rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
     if ( rc )
         return rc;

     rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
     if ( rc )
         goto out;
Quoted 7 lines
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Quoted 3 lines

Yes, but that's a tool stack operation, not something the kernel would do all by itself. The kernel would only ever pass DOMID_SELF (or the actual local domain ID), I would think.

Quoted 5 lines

You can't issue that hypercall directly from userspace because you need to map the page in the physical address space of the toolstack domain.

So the kernel has to act as the proxy for the hypercall. This is implemented as mmap() in Linux.

Oh, and there's no generic wrapping available here, unlike for dmop. Makes me wonder whether, for this purpose, there should be (have been) a new dmop with identical functionality, to allow such funneling.

Janan

#22 lore raw

Hi,

Quoted 35 lines · Jan Beulich
Quoted 34 lines · Julien Grall

Hi Jan,

Quoted 30 lines · Jan Beulich
Quoted 18 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

      rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
      if ( rc )
          return rc;

      rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
      if ( rc )
          goto out;
Quoted 7 lines
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Yes, but that's a tool stack operation, not something the kernel would do all by itself. The kernel would only ever pass DOMID_SELF (or the actual local domain ID), I would think.

Quoted 8 lines
Quoted 5 lines

You can't issue that hypercall directly from userspace because you need to map the page in the physical address space of the toolstack domain.

So the kernel has to act as the proxy for the hypercall. This is implemented as mmap() in Linux.

Oh, and there's no generic wrapping available here, unlike for dmop.

It is not clear to me the sort of generic wrapping you are referring to. Are you referring to a stable interface for an application?

Quoted 3 lines

Makes me wonder whether, for this purpose, there should be (have been) a new dmop with identical functionality, to allow such funneling.

I am not sure how using DMOP will allow us to implement it fully in userspace. Do you mind expanding it?

Cheers,

Signature · 1 line
Julien Grall
#23 lore raw
Quoted 50 lines · Julien Grall

Hi,

Quoted 35 lines · Jan Beulich
Quoted 34 lines · Julien Grall

Hi Jan,

Quoted 30 lines · Jan Beulich
Quoted 18 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

      rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
      if ( rc )
          return rc;

      rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
      if ( rc )
          goto out;
Quoted 7 lines
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Yes, but that's a tool stack operation, not something the kernel would do all by itself. The kernel would only ever pass DOMID_SELF (or the actual local domain ID), I would think.

Quoted 8 lines
Quoted 5 lines

You can't issue that hypercall directly from userspace because you need to map the page in the physical address space of the toolstack domain.

So the kernel has to act as the proxy for the hypercall. This is implemented as mmap() in Linux.

Oh, and there's no generic wrapping available here, unlike for dmop.

It is not clear to me the sort of generic wrapping you are referring to. Are you referring to a stable interface for an application?

Quoted 6 lines
Quoted 3 lines

Makes me wonder whether, for this purpose, there should be (have been) a new dmop with identical functionality, to allow such funneling.

I am not sure how using DMOP will allow us to implement it fully in userspace. Do you mind expanding it?

dmop was designed so that a kernel proxying requests wouldn't need updating for every new request added to the interface. If the request here was made through a new dmop, the kernel would never have had a need to know of an interface structure that's of no interest to it, but only to the tool stack.

Jan

#24 lore raw
Quoted 64 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 24 June 2020 13:52
To: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>; [email protected]; [email protected]; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH for-4.14] mm: fix public declaration of struct xen_mem_acquire_resource

Quoted 54 lines · Julien Grall

Hi,

Quoted 35 lines · Jan Beulich
Quoted 34 lines · Julien Grall

Hi Jan,

Quoted 30 lines · Jan Beulich
Quoted 18 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

      rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
      if ( rc )
          return rc;

      rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
      if ( rc )
          goto out;
Quoted 7 lines
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Yes, but that's a tool stack operation, not something the kernel would do all by itself. The kernel would only ever pass DOMID_SELF (or the actual local domain ID), I would think.

Quoted 8 lines
Quoted 5 lines

You can't issue that hypercall directly from userspace because you need to map the page in the physical address space of the toolstack domain.

So the kernel has to act as the proxy for the hypercall. This is implemented as mmap() in Linux.

Oh, and there's no generic wrapping available here, unlike for dmop.

It is not clear to me the sort of generic wrapping you are referring to. Are you referring to a stable interface for an application?

Quoted 3 lines

Makes me wonder whether, for this purpose, there should be (have been) a new dmop with identical functionality, to allow such funneling.

Quoted 8 lines
Quoted 2 lines

I am not sure how using DMOP will allow us to implement it fully in userspace. Do you mind expanding it?

dmop was designed so that a kernel proxying requests wouldn't need updating for every new request added to the interface. If the request here was made through a new dmop, the kernel would never have had a need to know of an interface structure that's of no interest to it, but only to the tool stack.

How would the pages get mapped into process address space if the kernel doesn't know what's being done?

Paul

Quoted 2 lines

Jan

#25 lore raw
Quoted 67 lines · Paul Durrant
Quoted 67 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 24 June 2020 13:52
To: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>; [email protected]; [email protected]; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH for-4.14] mm: fix public declaration of struct xen_mem_acquire_resource

Quoted 56 lines · Julien Grall

Hi,

Quoted 35 lines · Jan Beulich
Quoted 34 lines · Julien Grall

Hi Jan,

Quoted 30 lines · Jan Beulich
Quoted 18 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

XENMEM_acquire_resource and it's related structure is currently inside a __XEN__ or __XEN_TOOLS__ guarded section to limit it's scope to the hypervisor or the toolstack only. This is wrong as the hypercall is already being used by the Linux kernel at least, and as such needs to be public.

Actually - how does this work for the Linux kernel, seeing

      rc = rcu_lock_remote_domain_by_id(xmar.domid, &d);
      if ( rc )
          return rc;

      rc = xsm_domain_resource_map(XSM_DM_PRIV, d);
      if ( rc )
          goto out;
Quoted 7 lines
Quoted 1 line

in the function?

It's my understanding (I haven't tried to use that hypercall yet on FreeBSD, so I cannot say I've tested it), that xmar.domid is the remote domain, which the functions locks and then uses xsm_domain_resource_map to check whether the current domain has permissions to do privileged operations against it.

Yes, but that's a tool stack operation, not something the kernel would do all by itself. The kernel would only ever pass DOMID_SELF (or the actual local domain ID), I would think.

Quoted 8 lines
Quoted 5 lines

You can't issue that hypercall directly from userspace because you need to map the page in the physical address space of the toolstack domain.

So the kernel has to act as the proxy for the hypercall. This is implemented as mmap() in Linux.

Oh, and there's no generic wrapping available here, unlike for dmop.

It is not clear to me the sort of generic wrapping you are referring to. Are you referring to a stable interface for an application?

Quoted 3 lines

Makes me wonder whether, for this purpose, there should be (have been) a new dmop with identical functionality, to allow such funneling.

I am not sure how using DMOP will allow us to implement it fully in userspace. Do you mind expanding it?

Quoted 8 lines
Quoted 5 lines

dmop was designed so that a kernel proxying requests wouldn't need updating for every new request added to the interface. If the request here was made through a new dmop, the kernel would never have had a need to know of an interface structure that's of no interest to it, but only to the tool stack.

How would the pages get mapped into process address space if the kernel doesn't know what's being done?

Urgh, yes, silly me.

Jan