xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.
Subsystems (4)
PATCH 1/1 xprtrdma: The frmr iova_start values are truncated by the nfs rdma client.
A bad cast causes the iova_start, which in this case is a DMA bus address, to be truncated on 32b systems. No cast is needed.
---
net/sunrpc/xprtrdma/verbs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
net/sunrpc/xprtrdma/verbs.c +1 -1
@@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,memset (&frmr_wr,0 ,sizeof frmr_wr); frmr_wr.opcode =IB_WR_FAST_REG_MR ; frmr_wr.send_flags =0 ;/* unsignaled */ - frmr_wr.wr.fast_reg.iova_start = (unsigned long )seg1->mr_dma; + frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma; frmr_wr.wr.fast_reg.page_list = seg1->mr_chunk.rl_mw->r.frmr.fr_pgl; frmr_wr.wr.fast_reg.page_list_len = i; frmr_wr.wr.fast_reg.page_shift =PAGE_SHIFT ;
A message that is not in the archive (sent off-list or not yet received): 20090424190510.3134.90405.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org
Trond,
There is a similar bug in the server side too. I'll repost a single patch that fixes both.
Steve.
Quoted 28 lines · Steve Wise
A bad cast causes the
iova_start, which in this case is a DMA bus address, to be truncated on 32b systems. No cast is needed.Signed-off-by Steve Wise <[email protected]>---
net/sunrpc/xprtrdma/verbs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)net/sunrpc/xprtrdma/verbs.c +1
-2@@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,memset (&frmr_wr,0 ,sizeof frmr_wr); frmr_wr.opcode =IB_WR_FAST_REG_MR ; frmr_wr.send_flags =0 ;/* unsignaled */ - frmr_wr.wr.fast_reg.iova_start = (unsigned long )seg1->mr_dma; + frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma; frmr_wr.wr.fast_reg.page_list = seg1->mr_chunk.rl_mw->r.frmr.fr_pgl; frmr_wr.wr.fast_reg.page_list_len = i; frmr_wr.wr.fast_reg.page_shift =PAGE_SHIFT ; --To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hey Trond,
Turns out the server side is fine. So this patch is good to go.
Thanks,
Steve.
Quoted 42 lines · Steve Wise
Trond,
There is a similar bug in the server side too. I'll repost a single patch that fixes both.
Steve.
Quoted 30 lines · Steve Wise
A bad cast causes the
iova_start, which in this case is a DMA bus address, to be truncated on 32b systems. No cast is needed.Signed-off-by Steve Wise <[email protected]>---
net/sunrpc/xprtrdma/verbs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)net/sunrpc/xprtrdma/verbs.c +1
-1@@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,memset (&frmr_wr,0 ,sizeof frmr_wr); frmr_wr.opcode =IB_WR_FAST_REG_MR ; frmr_wr.send_flags =0 ;/* unsignaled */ - frmr_wr.wr.fast_reg.iova_start = (unsigned long )seg1->mr_dma; + frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma; frmr_wr.wr.fast_reg.page_list = seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;frmr_wr.wr.fast_reg.page_list_len = i; frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Steve Wise
Hey Trond,
Turns out the server side is fine. So this patch is good to go.
I'll ACK the patch, but I do wonder if the code will compile cleanly on all platforms. The iova_start is a u64, whereas the mr_dma is a dma_addr_t, which is variable sized, depending on platform. Would a (u64) cast be a safer patch, warning-wise?
Tom.
Quoted 52 lines
Thanks,
Steve.
Quoted 44 lines · Steve Wise
Trond,
There is a similar bug in the server side too. I'll repost a single patch that fixes both.
Steve.
Quoted 31 lines · Steve Wise
A bad cast causes the
iova_start, which in this case is a DMA bus address,
to be truncated on 32b systems. No cast is needed.Signed-off-by Steve Wise <[email protected]>---
net/sunrpc/xprtrdma/verbs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)net/sunrpc/xprtrdma/verbs.c +1
-1@@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,memset (&frmr_wr,0 ,sizeof frmr_wr); frmr_wr.opcode =IB_WR_FAST_REG_MR ; frmr_wr.send_flags =0 ;/* unsignaled */ - frmr_wr.wr.fast_reg.iova_start = (unsigned long )seg1->mr_dma; + frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma; frmr_wr.wr.fast_reg.page_list = seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;frmr_wr.wr.fast_reg.page_list_len = i; frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 49f515a5.1d1e640a.1c82.6677-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org
Quoted 4 lines · Tom Talpey
Quoted 2 lines · Steve Wise
Hey Trond,
Quoted 9 lines
Quoted 2 lines
Turns out the server side is fine. So this patch is good to go.
I'll ACK the patch, but I do wonder if the code will compile cleanly on all platforms. The
iova_startis a u64, whereas themr_dmais adma_addr_t, which is variable sized, depending on platform. Would a (u64) cast be a safer patch, warning-wise?
Well, if dma_addr_t is smaller, I don't think you'll get a warning, will you? And if its larger than 64b, then you _want_ the warning, because nothing will work. :)
Its up to you and/or Trond.
Steve.
Quoted 54 lines
Tom.
Quoted 51 lines
Thanks,
Steve.
Quoted 44 lines · Steve Wise
Trond,
There is a similar bug in the server side too. I'll repost a single patch that fixes both.
Steve.
Quoted 31 lines · Steve Wise
A bad cast causes the
iova_start, which in this case is a DMA bus address, to be truncated on 32b systems. No cast is needed.Signed-off-by Steve Wise <[email protected]>---
net/sunrpc/xprtrdma/verbs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)net/sunrpc/xprtrdma/verbs.c +1
-1@@ -1489,7 +1489,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg,memset (&frmr_wr,0 ,sizeof frmr_wr); frmr_wr.opcode =IB_WR_FAST_REG_MR ; frmr_wr.send_flags =0 ;/* unsignaled */ - frmr_wr.wr.fast_reg.iova_start = (unsigned long )seg1->mr_dma; + frmr_wr.wr.fast_reg.iova_start = seg1->mr_dma; frmr_wr.wr.fast_reg.page_list = seg1->mr_chunk.rl_mw->r.frmr.fr_pgl;frmr_wr.wr.fast_reg.page_list_len = i; frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 15 lines · Steve Wise
Quoted 4 lines · Tom Talpey
Quoted 2 lines · Steve Wise
Hey Trond,
Quoted 9 lines
Quoted 2 lines
Turns out the server side is fine. So this patch is good to go.
I'll ACK the patch, but I do wonder if the code will compile cleanly on all platforms. The
iova_startis a u64, whereas themr_dmais adma_addr_t, which is variable sized, depending on platform. Would a (u64) cast be a safer patch, warning-wise?
Quoted 5 lines
Well, if
dma_addr_tis smaller, I don't think you'll get a warning, will you? And if its larger than 64b, then you _want_ the warning, because nothing will work. :)Its up to you and/or Trond.
It looks looks as though the bug is really that the IB code is using a u64 to store dma handles. As an external user of the IB api, we really shouldn't have to perform this sort of transformation. If it is absolutely necessary, then it should be done by means of specialised accessor functions to initialise/read iova_start value when given a dma_addr_t.
I'd therefore prefer the no-cast version (with eventual compiler warnings), in the hope that eventually the IB folks will fix their interface.
Cheers
Trond
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
A message that is not in the archive (sent off-list or not yet received): 1240855510.8818.9.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 10 lines · Trond Myklebust
It looks looks as though the bug is really that the IB code is using a u64 to store dma handles. As an external user of the IB api, we really shouldn't have to perform this sort of transformation. If it is absolutely necessary, then it should be done by means of specialised accessor functions to initialise/read
iova_startvalue when given adma_addr_t.I'd therefore prefer the no-cast version (with eventual compiler warnings), in the hope that eventually the IB folks will fix their interface.
Translation: It looks to me as if the interface that we're using is a bit too corrupted with IB low level implementation grime. In the future, I'd like to see someone come up with a more high level interface for use by external code such as the sunrpc module.
Cheers
Trond
A message that is not in the archive (sent off-list or not yet received): 1240856613.8818.16.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 14 lines · Trond Myklebust
Quoted 11 lines · Trond Myklebust
It looks looks as though the bug is really that the IB code is using a u64 to store dma handles. As an external user of the IB api, we really shouldn't have to perform this sort of transformation. If it is absolutely necessary, then it should be done by means of specialised accessor functions to initialise/read
iova_startvalue when given adma_addr_t.I'd therefore prefer the no-cast version (with eventual compiler warnings), in the hope that eventually the IB folks will fix their interface.
Quoted 6 lines
Translation: It looks to me as if the interface that we're using is a bit too corrupted with IB low level implementation grime. In the future, I'd like to see someone come up with a more high level interface for use by external code such as the sunrpc module.
Clarification: The iova_start isn't used to store dma handles. The iova_start is the "address" base value that is advertised to a peer to describe the base address of a memory region. The contents of that can be more than just a dma handle...its up to the application. For instance, you could advertise a iova_start of zero or a kernel VA as the rdma server does. Also, the type is u64 because that is the size used on the wire as part of the rdma (IB and iWARP) protocols.
Steve.
Quoted 15 lines · Steve Wise
Quoted 14 lines · Trond Myklebust
Quoted 11 lines · Trond Myklebust
It looks looks as though the bug is really that the IB code is using a u64 to store dma handles. As an external user of the IB api, we really shouldn't have to perform this sort of transformation. If it is absolutely necessary, then it should be done by means of specialised accessor functions to initialise/read
iova_startvalue when given adma_addr_t.I'd therefore prefer the no-cast version (with eventual compiler warnings), in the hope that eventually the IB folks will fix their interface.
Quoted 8 lines
Quoted 6 lines
Translation: It looks to me as if the interface that we're using is a bit too corrupted with IB low level implementation grime. In the future, I'd like to see someone come up with a more high level interface for use by external code such as the sunrpc module.
Clarification: The
iova_startisn't used to store dma handles. The
Agreed, it's more of a hardware register, that ends up on the wire as well.
I think the net of this is that the mr_dma should have a more sensible up-cast that yields the right bits in the iova_start. Maybe a nice machine-dependent macro, defined in the RDMA layer, would be a good approach. Surely the other upper layers need it too.
While I have the floor, why doesn't the server have this issue? Looking at the code, it has the same (unsigned long) cast as the client when initializing its iova_start.
Tom.
Quoted 10 lines
iova_startis the "address" base value that is advertised to a peer to describe the base address of a memory region. The contents of that can be more than just a dma handle...its up to the application. For instance, you could advertise aiova_startof zero or a kernel VA as the rdma server does. Also, the type is u64 because that is the size used on the wire as part of the rdma (IB and iWARP) protocols.Steve.
A message that is not in the archive (sent off-list or not yet received): 49f60ac4.1c1d640a.2d0a.61a7-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org
Quoted 31 lines · Tom Talpey
Quoted 17 lines · Steve Wise
Quoted 15 lines · Trond Myklebust
Quoted 12 lines · Trond Myklebust
It looks looks as though the bug is really that the IB code is using a u64 to store dma handles. As an external user of the IB api, we really shouldn't have to perform this sort of transformation. If it is absolutely necessary, then it should be done by means of specialised accessor functions to initialise/read
iova_startvalue when given adma_addr_t.I'd therefore prefer the no-cast version (with eventual compiler warnings), in the hope that eventually the IB folks will fix their interface.
Quoted 9 lines
Quoted 7 lines
Translation: It looks to me as if the interface that we're using is a bit too corrupted with IB low level implementation grime. In the future, I'd like to see someone come up with a more high level interface for use by external code such as the sunrpc module.
Clarification: The
iova_startisn't used to store dma handles. TheAgreed, it's more of a hardware register, that ends up on the wire as well.
Quoted 10 lines
I think the net of this is that the
mr_dmashould have a more sensible up-cast that yields the right bits in theiova_start. Maybe a nice machine-dependent macro, defined in the RDMA layer, would be a good approach. Surely the other upper layers need it too.While I have the floor, why doesn't the server have this issue? Looking at the code, it has the same (unsigned long) cast as the client when initializing its
iova_start.
The server isn't using the dma address as the iova_start, but rather a kernel virtual address pointer, which is 32b on a i386 system. If you take the cast off, then the the signed bit gets extended into the u64. Apparently pointers are signed?
For instance, the server had a kva of 0xf5b75000. If you remove the (unsigned long) cast and stuff that into a u64, it ends up as 0xfffffffff5b75000.
here was a trace I took of the server doing the first rdma write using an frmr:
Apr 26 13:14:07 rac2 kernel: build_fastreg iova_start 0xfffffffff5b75000 rkey 0x500 len 4096 Apr 26 13:14:07 rac2 kernel: build_fastreg pbl[0] 0x35b75000 Apr 26 13:14:07 rac2 kernel: build_rdma_write sge[0] lkey 0x500 addr 0xf5b75000 len 24 Apr 26 13:14:07 rac2 kernel: post_qp_event - AE qpid 0x23 opcode 0 status 0x6 type 1 wrid.hi 0x1 wrid.lo 0x0
So the frmr registration ends up with 0xfffffffff5b75000 as the iova_start, yet the rdma write work request has 0xf5b75000 as the sge address entry. And the rnic fails this WR with a base/bounds violation (status 0x6 in the chelsio Async Event).
Steve.
Quoted 43 lines · Steve Wise
Quoted 31 lines · Tom Talpey
Quoted 17 lines · Steve Wise
Quoted 15 lines · Trond Myklebust
Quoted 12 lines · Trond Myklebust
It looks looks as though the bug is really that the IB code is using a u64 to store dma handles. As an external user of the IB api, we really shouldn't have to perform this sort of transformation. If it is absolutely necessary, then it should be done by means of specialised accessor functions to initialise/read
iova_startvalue when given adma_addr_t.I'd therefore prefer the no-cast version (with eventual compiler warnings), in the hope that eventually the IB folks will fix their interface.
Quoted 9 lines
Quoted 7 lines
Translation: It looks to me as if the interface that we're using is a bit too corrupted with IB low level implementation grime. In the future, I'd like to see someone come up with a more high level interface for use by external code such as the sunrpc module.
Clarification: The
iova_startisn't used to store dma handles. TheAgreed, it's more of a hardware register, that ends up on the wire as well.
Quoted 10 lines
I think the net of this is that the
mr_dmashould have a more sensible up-cast that yields the right bits in theiova_start. Maybe a nice machine-dependent macro, defined in the RDMA layer, would be a good approach. Surely the other upper layers need it too.While I have the floor, why doesn't the server have this issue? Looking at the code, it has the same (unsigned long) cast as the client when initializing its
iova_start.
Quoted 4 lines
The server isn't using the dma address as the
iova_start, but rather a kernel virtual address pointer, which is 32b on a i386 system. If you take the cast off, then the the signed bit gets extended into the u64. Apparently pointers are signed?
Why is the server using a u64 to store a naked pointer? That has to be a bug. Casting to (unsigned long) is just hiding it.
Does this address get handed to the RNIC to perform some sort of local DMA? If so, how does it work if there's an IOMMU in the system? The kva isn't necessarily the same as the dma_addr, right?
BTW, pointers are unsigned, but the assignment to u64 causes the compiler to convert the pointer into a ptrdiff_t, in effect evaluating ((pointer) - NULL). Then, since the ptrdiff_t is a signed 32 bits, the promotion results in the sign extension. I think! IOW, bug.
Tom.
Quoted 25 lines
For instance, the server had a kva of
0xf5b75000. If you remove the (unsigned long) cast and stuff that into a u64, it ends up as0xfffffffff5b75000.here was a trace I took of the server doing the first rdma write using an frmr:
Apr 26 13:14:07 rac2 kernel: build_fastreg iova_start 0xfffffffff5b75000 rkey 0x500 len 4096 Apr 26 13:14:07 rac2 kernel: build_fastreg pbl[0] 0x35b75000 Apr 26 13:14:07 rac2 kernel: build_rdma_write sge[0] lkey 0x500 addr 0xf5b75000 len 24 Apr 26 13:14:07 rac2 kernel: post_qp_event - AE qpid 0x23 opcode 0status
0x6type 1 wrid.hi0x1wrid.lo0x0So the frmr registration ends up with
0xfffffffff5b75000as theiova_start, yet the rdma write work request has0xf5b75000as the sge
address entry. And the rnic fails this WR with a base/bounds violation
(status0x6in the chelsio Async Event).Steve.
A message that is not in the archive (sent off-list or not yet received): 49f61067.181e640a.3cb9.0e6c-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org
Quoted 51 lines · Tom Talpey
Quoted 49 lines · Steve Wise
Quoted 47 lines · Tom Talpey
Quoted 28 lines · Steve Wise
Quoted 17 lines · Trond Myklebust
Quoted 7 lines · Trond Myklebust
It looks looks as though the bug is really that the IB code is using a u64 to store dma handles. As an external user of the IB api, we really shouldn't have to perform this sort of transformation. If it is absolutely necessary, then it should be done by means of specialised accessor functions to initialise/read
iova_startvalue when given adma_addr_t.Quoted 6 lines
I'd therefore prefer the no-cast version (with eventual compiler warnings), in the hope that eventually the IB folks will fix their interface.
Quoted 8 lines
Translation: It looks to me as if the interface that we're using is a bit too corrupted with IB low level implementation grime. In the future, I'd like to see someone come up with a more high level interface for use by external code such as the sunrpc module.
Quoted 3 lines
Clarification: The
iova_startisn't used to store dma handles. TheAgreed, it's more of a hardware register, that ends up on the wire as well.
I think the net of this is that the
mr_dmashould have a more sensible up-cast that yields the right bits in theiova_start. Maybe a nice machine-dependent macro, defined in the RDMA layer, would be a good approach. Surely the other upper layers need it too.While I have the floor, why doesn't the server have this issue? Looking at the code, it has the same (unsigned long) cast as the client when initializing its
iova_start.
Quoted 6 lines
Quoted 2 lines
The server isn't using the dma address as the
iova_start, but rather a kernel virtual address pointer, which is 32b on a i386 system. If you take the cast off, then the the signed bit gets extended into the u64. Apparently pointers are signed?Why is the server using a u64 to store a naked pointer? That has to be a bug. Casting to (unsigned long) is just hiding it.
That is what it wants to use as the registration for its frmr, which in this case is used as the source of an RDMA Write.
Quoted 2 lines
Does this address get handed to the RNIC to perform some sort of local DMA?
No.
Quoted 4 lines
If so, how does it work if there's an IOMMU in the system? The kva isn't necessarily the same as the
dma_addr, right?
Correct.
This kva is used as the iova_start for the fast-registered memory region. All it is used for is to mark the base value for "addresses" passed in via sge entries in the work requests, and also for incoming "addresses" in rdma packets. So you can use the kva when you fastreg the mr, and then also use the kva + any offset in the sge entries of your work requests that utilize it. Additionally, you can advertise the fastreg rkey, iova_start, and length to the peer for doing rdma into that region. The HW will validate any SGE entry in and any incoming rdma packets to ensure that the rkey/addr/len in the sge/packet is within the bounds of the fastregmr. Namely that the sge/packet address and length fall within the iova_start and iova_start+fastreg_len.
Quoted 6 lines
BTW, pointers are unsigned, but the assignment to u64 causes the compiler to convert the pointer into a
ptrdiff_t, in effect evaluating ((pointer) - NULL). Then, since theptrdiff_tis a signed 32 bits, the promotion results in the sign extension. I think! IOW, bug.
I see. And that's why the cast is needed for the server side.
Steve.
Quoted 7 lines · Steve Wise
Clarification: The
iova_startisn't used to store dma handles. Theiova_startis the "address" base value that is advertised to a peer to describe the base address of a memory region. The contents of that can be more than just a dma handle...its up to the application. For instance, you could advertise aiova_startof zero or a kernel VA as the rdma server does. Also, the type is u64 because that is the size used on the wire as part of the rdma (IB and iWARP) protocols.
OK, but my point is we shouldn't be having this discussion at all. I shouldn't be required to know that the wire protocol uses a 64-bit unsigned little-endian/big endian integer in order to use the rdma api.
All I should need to know is that I can advertise either dma handles or kernel VAs, and know that I can choose between two functions, say, ib_send_wr_fastreg_dma_init() and ib_send_wr_fastreg_kva_init() to initialise the ib_send_wr structure correctly.
Trond
A message that is not in the archive (sent off-list or not yet received): 1240865214.8818.73.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 5 lines · Trond Myklebust
Quoted 2 lines · Steve Wise
Clarification: The
iova_startisn't used to store dma handles. Theiova_startis the "address" base value that is advertised to a peer to describe the base address of a memory region. The contents of that can be more than just a dma handle...its up to the application. For instance, you could advertise aiova_startof zero or a kernel VA as the rdma server does. Also, the type is u64 because that is the size used on the wire as part of the rdma (IB and iWARP) protocols.
Quoted 9 lines
OK, but my point is we shouldn't be having this discussion at all. I shouldn't be required to know that the wire protocol uses a 64-bit unsigned little-endian/big endian integer in order to use the rdma api.
All I should need to know is that I can advertise either dma handles or kernel VAs, and know that I can choose between two functions, say,
ib_send_wr_fastreg_dma_init()andib_send_wr_fastreg_kva_init()to initialise theib_send_wrstructure correctly.
I hear ya.
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Quoted 7 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Cheers
Trond
A message that is not in the archive (sent off-list or not yet received): 1242082203.1743.11.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 11 lines · Trond Myklebust
Quoted 8 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Quoted 6 lines
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Cheers
Trond
How is it circumventing? It's currently incorrectly casting a pointer into a u64. That seems just broken to me. Also, its really the sunrpc rdma transport layer. It deals specifically with rdma. It _should_ know about rdma interfaces and types.
But I'll whip up get/put type accessor methods for this field if that's what you require. In the meantime, customers will just crash I guess.
Steve.
Quoted 12 lines · Steve Wise
Quoted 11 lines · Trond Myklebust
Quoted 8 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Quoted 8 lines
Quoted 6 lines
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Cheers
TrondHow is it circumventing? It's currently incorrectly casting a pointer into a u64. That seems just broken to me.
The cast is definitely broken, and since your patch removes it I agree with the change, for the record. Especially since with the change, the code doesn't work on 32-bit CPU / 64-bit IOMMU platforms.
Quoted 3 lines
Also, its really the sunrpc
rdma transport layer. It deals specifically with rdma. It _should_ know about rdma interfaces and types.
IOW, this is an issue for the OFA API to address? I agree with Trond that there are some hardware-specific warts in there. It might be a good idea for us to go through both the client and server code overall - you up for doing that? I'm in, if so.
Tom.
Quoted 10 lines
But I'll whip up get/put type accessor methods for this field if that's what you require. In the meantime, customers will just crash I guess.
Steve.
A message that is not in the archive (sent off-list or not yet received): 4a08c1b5.151e640a.0a99.fffff868-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org
Quoted 23 lines · Tom Talpey
Quoted 21 lines · Steve Wise
Quoted 19 lines · Trond Myklebust
Quoted 9 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Cheers
Trond
Quoted 8 lines
Quoted 2 lines
How is it circumventing? It's currently incorrectly casting a pointer into a u64. That seems just broken to me.
The cast is definitely broken, and since your patch removes it I agree with the change, for the record. Especially since with the change, the code doesn't work on 32-bit CPU / 64-bit IOMMU platforms.
That was my short-term concern. Its just broke.
Quoted 10 lines
Quoted 3 lines
Also, its really the sunrpc
rdma transport layer. It deals specifically with rdma. It _should_ know about rdma interfaces and types.IOW, this is an issue for the OFA API to address? I agree with Trond that there are some hardware-specific warts in there. It might be a good idea for us to go through both the client and server code overall - you up for doing that? I'm in, if so.
Ok.
Quoted 16 lines · Steve Wise
Quoted 15 lines · Trond Myklebust
Quoted 8 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Quoted 6 lines
Quoted 2 lines
Cheers
TrondHow is it circumventing? It's currently incorrectly casting a pointer into a u64. That seems just broken to me. Also, its really the sunrpc rdma transport layer. It deals specifically with rdma. It _should_ know about rdma interfaces and types.
The fact is that I'm simply not interested enough in rdma to tolerate hacks. If it isn't done cleanly, in a manner that I can maintain, then the whole transport layer comes out...
Trond
A message that is not in the archive (sent off-list or not yet received): 1242089066.1743.19.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 19 lines · Trond Myklebust
Quoted 18 lines · Steve Wise
Quoted 17 lines · Trond Myklebust
Quoted 8 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Cheers
Trond
Quoted 8 lines
Quoted 4 lines
How is it circumventing? It's currently incorrectly casting a pointer into a u64. That seems just broken to me. Also, its really the sunrpc rdma transport layer. It deals specifically with rdma. It _should_ know about rdma interfaces and types.
The fact is that I'm simply not interested enough in rdma to tolerate hacks. If it isn't done cleanly, in a manner that I can maintain, then the whole transport layer comes out...
I know exactly what you want - it's not what the code does now and it's not an accessor function to set the hardware's u64 field. What's needed is a new function to manage the entire RDMA triplet, and the memory registration behind it, in the OFA code side. Put the hardware goop below the line, IOW. I'll dust up Steve on this.
Tom.
A message that is not in the archive (sent off-list or not yet received): 4a08cd7b.48c3f10a.6bb1.fffff6d3-ATjtLOhZ0NVl57MIdRCFDg@public.gmane.org
Quoted 29 lines · Tom Talpey
Quoted 19 lines · Trond Myklebust
Quoted 18 lines · Steve Wise
Quoted 17 lines · Trond Myklebust
Quoted 8 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Cheers
TrondQuoted 8 lines
Quoted 4 lines
How is it circumventing? It's currently incorrectly casting a pointer into a u64. That seems just broken to me. Also, its really the sunrpc rdma transport layer. It deals specifically with rdma. It _should_ know about rdma interfaces and types.
The fact is that I'm simply not interested enough in rdma to tolerate hacks. If it isn't done cleanly, in a manner that I can maintain, then the whole transport layer comes out...
Quoted 5 lines
I know exactly what you want - it's not what the code does now and it's not an accessor function to set the hardware's u64 field. What's needed is a new function to manage the entire RDMA triplet, and the memory registration behind it, in the OFA code side. Put the hardware goop below the line, IOW. I'll dust up Steve on this.
This does indeed sound like what I'd looking for.
There is a huge difference between having code that depends on well defined rdma interfaces, and code that depends on rdma hacks. A piece of code that requires casts from a non-local opaque type into another protocol-dependent non-local type will definitely fall in the latter category. I really don't care what the current code does, but a fix for that code is something that does it _correctly_; it is not yet another hack, whether or not it fixes a bug in the short term.
Trond
A message that is not in the archive (sent off-list or not yet received): 1242092150.16618.15.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 50 lines · Trond Myklebust
Quoted 31 lines · Tom Talpey
Quoted 20 lines · Trond Myklebust
Quoted 18 lines · Steve Wise
Quoted 16 lines · Trond Myklebust
Quoted 9 lines · Steve Wise
Hey Trond,
Will this bug fix make 2.6.30?
Thanks,
Steve.
Not in the form it is in now. As I've said earlier, I'm not happy about the sunrpc layer having to circumvent ordinary type checking on non-sunrpc structures.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Cheers
TrondHow is it circumventing? It's currently incorrectly casting a pointer into a u64. That seems just broken to me. Also, its really the sunrpc rdma transport layer. It deals specifically with rdma. It _should_ know about rdma interfaces and types.
The fact is that I'm simply not interested enough in rdma to tolerate hacks. If it isn't done cleanly, in a manner that I can maintain, then the whole transport layer comes out...
Quoted 6 lines
I know exactly what you want - it's not what the code does now and it's not an accessor function to set the hardware's u64 field. What's needed is a new function to manage the entire RDMA triplet, and the memory registration behind it, in the OFA code side. Put the hardware goop below the line, IOW. I'll dust up Steve on this.
This does indeed sound like what I'd looking for.
There is a huge difference between having code that depends on well defined rdma interfaces, and code that depends on rdma hacks. A piece of code that requires casts from a non-local opaque type into another protocol-dependent non-local type will definitely fall in the latter category. I really don't care what the current code does, but a fix for that code is something that does it _correctly_; it is not yet another hack, whether or not it fixes a bug in the short term.
Quoted 3 lines
Trond
Trond, I get your point, and we can certainly work on improving this with the rdma developer community. But removing the one-line-broken cast will resolve a current crash situation for 2.6.30. Can't we get this fix in 2.6.30 and work on the API improvements for 2.6.31? I've CCed Roland and the ofa general list to get everyone involved in this thread so we can get this API design change going.
I agree we can clean this up moving forward, but lets fix the broken 2.6.30 code.
Will this work?
Steve.
Quoted 6 lines · Steve Wise
Trond Myklebust wrote (earlier in this thread):
All I should need to know is that I can advertise either dma handles or kernel VAs, and know that I can choose between two functions, say,
ib_send_wr_fastreg_dma_init()andib_send_wr_fastreg_kva_init()to initialise theib_send_wrstructure correctly.
To align more with the rest of the fast_reg API in ib_verbs.h, I propose:
static inline void ib_init_fast_reg_iova_start_dma(struct ib_send_wr *send_wr, dma_addr_t dma); static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr *send_wr, void *kva);
Thoughts?
Quoted 10 lines · Steve Wise
Quoted 6 lines · Steve Wise
Trond Myklebust wrote (earlier in this thread):
All I should need to know is that I can advertise either dma handles or kernel VAs, and know that I can choose between two functions, say,
ib_send_wr_fastreg_dma_init()andib_send_wr_fastreg_kva_init()to initialise theib_send_wrstructure correctly.
Quoted 8 lines
To align more with the rest of the
fast_regAPI inib_verbs.h, I propose:static inline void
ib_init_fast_reg_iova_start_dma(struct ib_send_wr *send_wr, dma_addr_t dma); static inline voidib_init_fast_reg_iova_start_kva(struct ib_send_wr *send_wr, void *kva);Thoughts?
uncompiled patch:
include/rdma/ib_verbs.h +0 -0
@@ -1940,6 +1940,30 @@ static inline void ib_update_fast_reg_key(struct ib_mr *mr, u8 newkey)
}
/**
+ * ib_init_fast_reg_iova_start_dma - initializes the iova_start field
+ * based on a dma address supplied by the user.
+ * @wr - struct ib_send_wr pointer to be initialized
+ * @addr - dma_addr_t value to be used as the iova_start
+ */
+static inline void ib_init_fast_reg_iova_start_dma(struct ib_send_wr *wr,
+ dma_addr_t addr)
+{
+ wr->wr.fast_reg.iova_start = addr;
+}
+
+/**
+ * ib_init_fast_reg_iova_start_kva - initializes the iova_start field
+ * based on a kernel virtual address supplied by the user.
+ * @wr - struct ib_send_wr pointer to be initialized
+ * @addr - void * address to be used as the iova_start
+ */
+static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr *wr,
+ void *addr)
+{
+ wr->wr.fast_reg.iova_start = (unsigned long)addr;
+}
+
+/**
* ib_alloc_mw - Allocates a memory window.
* @pd: The protection domain associated with the memory window. */
Quoted 6 lines
Trond Myklebust wrote (earlier in this thread):
Quoted 5 lines
All I should need to know is that I can advertise either dma handles or kernel VAs, and know that I can choose between two functions, say,
ib_send_wr_fastreg_dma_init()andib_send_wr_fastreg_kva_init()to initialise theib_send_wrstructure correctly.
I skimmed the earlier thread, and I have to say that I don't quite see what the problem with assigning things to a u64 directly is. You can use any address you want, and I don't quite understand why using the correct cast to avoid sign extension or truncation problems is such a big maintenance burden?
The code below really just looks like obfuscation to me -- are we going to want to add something like
/**
* ib_init_fast_reg_iova_start_u64 - initializes the iova_start field
* based on a 64-bit address supplied by the user.
* @wr - struct ib_send_wr pointer to be initialized
* @addr - void * address to be used as the iova_start
*/
static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr *wr,
u64 addr)
{
wr->wr.fast_reg.iova_start = addr;
}next, to make sure we don't get confused about assigning a u64 to a u64? It all looks a bit overcomplicated to me.
- R.
Quoted 28 lines
/**
+ *ib_init_fast_reg_iova_start_dma- initializes theiova_startfield
+ * based on a dma address supplied by the user.
+ * @wr - structib_send_wrpointer to be initialized
+ * @addr -dma_addr_tvalue to be used as theiova_start
+ */(quoted diff) +19
-0+static inline void ib_init_fast_reg_iova_start_dma(struct ib_send_wr *wr, + dma_addr_t addr) +{ + wr->wr.fast_reg.iova_start = addr; +} + +/** + * ib_init_fast_reg_iova_start_kva - initializes the iova_start field + * based on a kernel virtual address supplied by the user. + * @wr - struct ib_send_wr pointer to be initialized + * @addr - void * address to be used as the iova_start + */ +static inline void ib_init_fast_reg_iova_start_kva(struct ib_send_wr *wr, + void *addr) +{ + wr->wr.fast_reg.iova_start = (unsigned long)addr; +} + +/** * ib_alloc_mw - Allocates a memory window. * @pd: The protection domain associated with the memory window. */
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines
Trond Myklebust wrote
Quoted 1 line
All I should need to know is that I can advertise either dma handles or kernel VAs
Maybe its obvious to some people here, but may I ask why there's a need to post either dma address or kernel virtual address? is it application need? hardware (e.g IB vs iWARP vs vendor implementation) specific? or something else?
Or.
A message that is not in the archive (sent off-list or not yet received): 4A0BC6A6.1070002-smomgflXvOZWk0Htik3J/[email protected]
Quoted 8 lines · Or Gerlitz
Quoted 2 lines
Trond Myklebust wrote
Quoted 1 line
All I should need to know is that I can advertise either dma handles or kernel VAs
Maybe its obvious to some people here, but may I ask why there's a need to post either dma address or kernel virtual address? is it application need? hardware (e.g IB vs iWARP vs vendor implementation) specific? or something else?
Or.
The NFSRDMA transport uses Fast Register Memory Regions. In this particular section of code, the NFSRDMA client is building a fastreg work request to bind a page list to a fastreg mr. You can read about this in the IBTA spec on memory management extensions, or in the RDMA Verbs draft.
Steve.
Quoted 4 lines · Steve Wise
The NFSRDMA transport uses Fast Register Memory Regions. In this particular section of code, the NFSRDMA client is building a fastreg work request to bind a page list to a fastreg mr. You can read about this in the IBTA spec on memory management extensions, or in the RDMA Verbs draft.
Hi Steve,
I was aware for the context being fastreg work request. I was thinking that the spec mandates either dma addr or kva on the iova but from your reply I assume to be wrong, thanks.
Or.