rxe: Fix structure layouts for 32/64 bit compat

The rxe driver structure layouts have implicit padding which differs depending on 32 bit or 64 bit mode, meaning rxe does not work if a 32 bit userspace is used on a 64 bit kernel.

They do work if the kernel and user space are the same bit width.

Since this is an ABI break change the ABI version. Unfortunately, the userspace driver does not handle the ABI version properly, so this will not stop any broken user spaces, but it does let us fix userspace to work properly in future.

---
 drivers/infiniband/sw/rxe/rxe.h   | 6 +++++-
 include/uapi/rdma/rdma_user_rxe.h | 6 +++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

This is marked RFC to pause and give some thought to the best solution, here is my first suggestion.

If we go this way, the userspace patch would be something like:

static const struct verbs_device_ops rxe_dev_ops = {
	.name = "rxe",

- .match_min_abi_version = 0,
- .match_max_abi_version = INT_MAX,
+ .match_min_abi_version = (sizeof(void *) == 8?1:2),
+ .match_max_abi_version = 2,

Then 32 builds of the kernel and rdma-core would demand to both be updated to work correctly.

I can't see any easy solution that lets existing 32 user/32 kernel users survive unchanged while still allowing compat for 32 user/64 kernel mode.

The few places I know of that are likely to be 32 bit, like ARM cores, already generally don't work with rxe because rxe has broken use of the cache APIs. So I don't actually think there is a 32 bit user of rxe..

index 7d232611303f40..d6cce6b6d88034 100644
@@ -59,7 +59,11 @@
 #include "rxe_verbs.h"
 #include "rxe_loc.h"
 
-#define RXE_UVERBS_ABI_VERSION		(1)
+/*
+ * Version 1 and Version 2 are identical on 64 bit machines, but on 32 bit
+ * machines Version 2 has a different struct layout.
+ */
+#define RXE_UVERBS_ABI_VERSION		(sizeof(void *) == 8?1:2)
 
 #define IB_PHYS_STATE_LINK_UP		(5)
 #define IB_PHYS_STATE_LINK_DOWN		(3)

index e3e6852b58eb45..b501deb18d8184 100644
@@ -58,6 +58,8 @@ struct rxe_global_route {
 struct rxe_av {
 	__u8			port_num;
 	__u8			network_type;
+	__u16			reserved1;
+	__u32			reserved2;
 	struct rxe_global_route	grh;
 	union {
 		struct sockaddr_in	_sockaddr_in;

@@ -84,6 +86,7 @@ struct rxe_send_wr {
 			__u64	compare_add;
 			__u64	swap;
 			__u32	rkey;
+			__u32	reserved;
 		} atomic;
 		struct {
 			__u32	remote_qpn;

@@ -93,7 +96,7 @@ struct rxe_send_wr {
 		struct {
 			struct ib_mr *mr;
 			__u32        key;
-			int          access;
+			__u32        access;
 		} reg;
 	} wr;
 };

@@ -116,6 +119,7 @@ struct rxe_dma_info {
 	__u32			cur_sge;
 	__u32			num_sge;
 	__u32			sge_offset;
+	__u32			reserved;
 	union {
 		__u8		inline_data[0];
 		struct rxe_sge	sge[0];

2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

T24gV2VkLCAyMDE4LTAyLTA3IGF0IDE2OjQxIC0wNzAwLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6 DQo+IGRpZmYgLS1naXQgYS9pbmNsdWRlL3VhcGkvcmRtYS9yZG1hX3VzZXJfcnhlLmggYi9pbmNs dWRlL3VhcGkvcmRtYS9yZG1hX3VzZXJfcnhlLmgNCj4gaW5kZXggZTNlNjg1MmI1OGViNDUuLmI1 MDFkZWIxOGQ4MTg0IDEwMDY0NA0KPiAtLS0gYS9pbmNsdWRlL3VhcGkvcmRtYS9yZG1hX3VzZXJf cnhlLmgNCj4gKysrIGIvaW5jbHVkZS91YXBpL3JkbWEvcmRtYV91c2VyX3J4ZS5oDQo+IEBAIC01 OCw2ICs1OCw4IEBAIHN0cnVjdCByeGVfZ2xvYmFsX3JvdXRlIHsNCj4gIHN0cnVjdCByeGVfYXYg ew0KPiAgCV9fdTgJCQlwb3J0X251bTsNCj4gIAlfX3U4CQkJbmV0d29ya190eXBlOw0KPiArCV9f dTE2CQkJcmVzZXJ2ZWQxOw0KPiArCV9fdTMyCQkJcmVzZXJ2ZWQyOw0KDQpIZWxsbyBKYXNvbiwN Cg0KU2VlaW5nIHR3byBjb25zZWN1dGl2ZSByZXNlcnZlZCBtZW1iZXJzIGlzIGEgYml0IHdlaXJk LiBIYXZlIHlvdSBjb25zaWRlcmVkDQp0byB1c2Ugc29tZXRoaW5nIGxpa2UgX191OCByZXNlcnZl ZFs2XSBpbnN0ZWFkPw0KDQpUaGFua3MsDQoNCkJhcnQuDQoNCg0KDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Ah, I had that originally but changed it to make pahole work properly.. Can switch it back.

Although, this does really firmly say what the alighment is, eg down the road someone might be tempted to do:

- __u8 reserved[6];
+ __u32 new_value;
+ __u16 reserved;

Which would be quite wrong..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

T24gV2VkLCAyMDE4LTAyLTA3IGF0IDE2OjU3IC0wNzAwLCBKYXNvbiBHdW50aG9ycGUgd3JvdGU6 DQo+IE9uIFdlZCwgRmViIDA3LCAyMDE4IGF0IDExOjU0OjI1UE0gKzAwMDAsIEJhcnQgVmFuIEFz c2NoZSB3cm90ZToNCj4gPiBPbiBXZWQsIDIwMTgtMDItMDcgYXQgMTY6NDEgLTA3MDAsIEphc29u IEd1bnRob3JwZSB3cm90ZToNCj4gPiA+IGRpZmYgLS1naXQgYS9pbmNsdWRlL3VhcGkvcmRtYS9y ZG1hX3VzZXJfcnhlLmggYi9pbmNsdWRlL3VhcGkvcmRtYS9yZG1hX3VzZXJfcnhlLmgNCj4gPiA+ IGluZGV4IGUzZTY4NTJiNThlYjQ1Li5iNTAxZGViMThkODE4NCAxMDA2NDQNCj4gPiA+ICsrKyBi L2luY2x1ZGUvdWFwaS9yZG1hL3JkbWFfdXNlcl9yeGUuaA0KPiA+ID4gQEAgLTU4LDYgKzU4LDgg QEAgc3RydWN0IHJ4ZV9nbG9iYWxfcm91dGUgew0KPiA+ID4gIHN0cnVjdCByeGVfYXYgew0KPiA+ ID4gIAlfX3U4CQkJcG9ydF9udW07DQo+ID4gPiAgCV9fdTgJCQluZXR3b3JrX3R5cGU7DQo+ID4g PiArCV9fdTE2CQkJcmVzZXJ2ZWQxOw0KPiA+ID4gKwlfX3UzMgkJCXJlc2VydmVkMjsNCj4gPiAN Cj4gPiBIZWxsbyBKYXNvbiwNCj4gPiANCj4gPiBTZWVpbmcgdHdvIGNvbnNlY3V0aXZlIHJlc2Vy dmVkIG1lbWJlcnMgaXMgYSBiaXQgd2VpcmQuIEhhdmUgeW91IGNvbnNpZGVyZWQNCj4gPiB0byB1 c2Ugc29tZXRoaW5nIGxpa2UgX191OCByZXNlcnZlZFs2XSBpbnN0ZWFkPw0KPiANCj4gQWgsIEkg aGFkIHRoYXQgb3JpZ2luYWxseSBidXQgY2hhbmdlZCBpdCB0byBtYWtlIHBhaG9sZSB3b3JrIHBy b3Blcmx5Li4NCj4gQ2FuIHN3aXRjaCBpdCBiYWNrLg0KPiANCj4gQWx0aG91Z2gsIHRoaXMgZG9l cyByZWFsbHkgZmlybWx5IHNheSB3aGF0IHRoZSBhbGlnaG1lbnQgaXMsIGVnDQo+IGRvd24gdGhl IHJvYWQgc29tZW9uZSBtaWdodCBiZSB0ZW1wdGVkIHRvIGRvOg0KPiANCj4gLSBfX3U4IHJlc2Vy dmVkWzZdOw0KPiArIF9fdTMyIG5ld192YWx1ZTsNCj4gKyBfX3UxNiByZXNlcnZlZDsNCj4gDQo+ IFdoaWNoIHdvdWxkIGJlIHF1aXRlIHdyb25nLi4NCg0KUGxlYXNlIGNvbnNpZGVyIHRvIGFkZCBC VUlMRF9CVUdfT04oKSBzdGF0ZW1lbnRzIGludG8gdGhlIHJ4ZSBkcml2ZXIgdG8gdmVyaWZ5DQpv ZmZzZXRzIG9mIG1lbWJlcnMgb2YgQUJJIHN0cnVjdHVyZXMuIFRoYXQgYXBwcm9hY2ggaXMgdXNl ZCBpbiB0aGUgcmRtYS1jb3JlDQpsaWJyYXJ5IGZvciBlLmcuIHRoZSB1bWFkIGxpYnJhcnkgYW5k IHNycF9kYWVtb24uDQoNClRoYW5rcywNCg0KQmFydC4NCg0KDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html