Skip to contentSkip to conversation

siw: fix for 'is_kva' flag issue in siw_tx_hdt()

PATCH In review Krishnamraju Eraparaju 3 participants 5 messages Aug 19, 2019 – Aug 22, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 siw: fix for 'is_kva' flag issue in siw_tx_hdt()

"is_kva" variable in siw_tx_hdt() is not currently being updated for each while loop iteration(the loop which walks the list of SGE's).

Usecase:

say a WQE has two SGE's :
- first with "assigned kernel buffer", so not MR allocated.
- second with PBL memory region, so mem_obj == PBL.

Now while processing first SGE, in iteration 1, "is_kva" is set to "1" because there is no MR allocated.
And while processing the second SGE in iteration 2, since mem_obj is PBL, the statement "if (!mem->mem_obj)" becomes false but is_kva is still "1"(previous state). Thus, the PBL memory is handled as "direct assigned kernel virtual memory", which eventually results in PAGE FAULTS, MPA CRC issues.

                if (!(tx_flags(wqe) & SIW_WQE_INLINE)) {
                        mem = wqe->mem[sge_idx];
                        if (!mem->mem_obj)
                                is_kva = 1;
                } else {
                        is_kva = 1;
                }

Note that you need to set MTU size more than the PAGESIZE to recreate this issue(as the address of "PBL index 0" and "assigned kernel virtual memory" are always same for SIW). In my case, I used SIW as ISER initiator with MTU 9000, issue occurs with
SCSI WRITE operation.

---
 drivers/infiniband/sw/siw/siw_qp_tx.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/infiniband/sw/siw/siw_qp_tx.c +2 -0
index 43020d2040fc..e2175a08269a 100644
@@ -465,6 +465,8 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx, struct socket *s)
 			mem = wqe->mem[sge_idx];
 			if (!mem->mem_obj)
 				is_kva = 1;
+			else
+				is_kva = 0;
 		} else {
 			is_kva = 1;
 		}
Signature · 1 line
2.23.0.rc0
#1 lore raw

-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----

Quoted 27 lines

To: [email protected], [email protected]
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 08/19/2019 01:14PM
Cc: [email protected], [email protected], [email protected], "Krishnamraju Eraparaju" <[email protected]>
Subject: [EXTERNAL] [PATCH for-rc] siw: fix for 'is_kva' flag issue in siw_tx_hdt()

"is_kva" variable in siw_tx_hdt() is not currently being updated for each while loop iteration(the loop which walks the list of SGE's).

Usecase:

say a WQE has two SGE's :
- first with "assigned kernel buffer", so not MR allocated.
- second with PBL memory region, so mem_obj == PBL.

Now while processing first SGE, in iteration 1, "is_kva" is set to "1"
because there is no MR allocated.
And while processing the second SGE in iteration 2, since mem_obj is PBL, the statement "if (!mem->mem_obj)" becomes false but is_kva is still "1"(previous state). Thus, the PBL memory is handled as "direct assigned kernel virtual memory", which eventually results in PAGE FAULTS, MPA CRC issues.

Quoted 8 lines
               if (!(tx_flags(wqe) & SIW_WQE_INLINE)) {
                       mem = wqe->mem[sge_idx];
                       if (!mem->mem_obj)
                               is_kva = 1;
               } else {
                       is_kva = 1;
               }

Hi Krishna,
That is a good catch. I was not aware of the possibility of mixed PBL and kernel buffer addresses in one SQE.

A correct fix must also handle the un-mapping of any kmap()'d buffers. The current TX code expects all buffers be either kmap()'d or all not kmap()'d. So the fix is a little more complex, if we must handle mixed SGL's during un-mapping. I think I can provide it by tomorrow. It's almost midnight ;)

Thanks!
Bernard.

Quoted 30 lines

Note that you need to set MTU size more than the PAGESIZE to recreate this issue(as the address of "PBL index 0" and "assigned kernel virtual memory" are always same for SIW). In my case, I used SIW as ISER initiator with MTU 9000, issue occurs with
SCSI WRITE operation.

---
drivers/infiniband/sw/siw/siw_qp_tx.c | 2 ++

1 file changed, 2 insertions(+)
drivers/infiniband/sw/siw/siw_qp_tx.c +2 -0
index 43020d2040fc..e2175a08269a 100644
@@ -465,6 +465,8 @@ static int siw_tx_hdt(struct siw_iwarp_tx *c_tx,
 struct socket *s)
 			mem = wqe->mem[sge_idx];
 			if (!mem->mem_obj)
 				is_kva = 1;
+			else
+				is_kva = 0;
 		} else {
 			is_kva = 1;
		}
Signature · 1 line
2.23.0.rc0
#2 lore raw
Quoted 9 lines · Bernard Metzler

Hi Krishna,
That is a good catch. I was not aware of the possibility of mixed PBL and kernel buffer addresses in one SQE.

A correct fix must also handle the un-mapping of any kmap()'d buffers. The current TX code expects all buffers be either kmap()'d or all not kmap()'d. So the fix is a little more complex, if we must handle mixed SGL's during un-mapping. I think I can provide it by tomorrow. It's almost midnight ;)

I'll wait for a proper fix. Dropping this patch. Thanks.

Signature · 3 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
#3 lore raw

-----"Doug Ledford" <[email protected]> wrote: -----

Quoted 14 lines

To: "Bernard Metzler" <[email protected]>, "Krishnamraju Eraparaju" <[email protected]>
From: "Doug Ledford" <[email protected]>
Date: 08/20/2019 07:56PM
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: [EXTERNAL] Re: [PATCH for-rc] siw: fix for 'is_kva' flag issue in siw_tx_hdt()

Quoted 4 lines · Bernard Metzler

Hi Krishna,
That is a good catch. I was not aware of the possibility of mixed PBL and kernel buffer addresses in one SQE.

Quoted 9 lines
Quoted 2 lines

A correct fix must also handle the un-mapping of any kmap()'d buffers. The current TX code expects all buffers be either kmap()'d

or

Quoted 3 lines

all not kmap()'d. So the fix is a little more complex, if we must handle mixed SGL's during un-mapping. I think I can provide it by tomorrow. It's almost midnight ;)

I'll wait for a proper fix. Dropping this patch. Thanks.

Thanks Doug!

I have a fix ready but still have to test it with iSER. Unfortunately I have a hard time to test iSER with siw, since both iSCSI-TCP target and iSER want to bind to the same TCP port. While this may be considered a bug in that code, siw is the first RDMA provider to take notice (since using kernel sockets and not offloaded, hitting a TCP port already bound).

I sent the patch to Chelsio folks and hope for
the best. They know the trick to make it working.

Thanks
Bernard.

#4 lore raw
Quoted 27 lines · Bernard Metzler

-----"Doug Ledford" <[email protected]> wrote: -----

Quoted 14 lines

To: "Bernard Metzler" <[email protected]>, "Krishnamraju Eraparaju" <[email protected]>
From: "Doug Ledford" <[email protected]>
Date: 08/20/2019 07:56PM
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: [EXTERNAL] Re: [PATCH for-rc] siw: fix for 'is_kva' flag issue in siw_tx_hdt()

Quoted 4 lines · Bernard Metzler

Hi Krishna,
That is a good catch. I was not aware of the possibility of mixed PBL and kernel buffer addresses in one SQE.

Quoted 9 lines
Quoted 2 lines

A correct fix must also handle the un-mapping of any kmap()'d buffers. The current TX code expects all buffers be either kmap()'d

or

Quoted 3 lines

all not kmap()'d. So the fix is a little more complex, if we must handle mixed SGL's during un-mapping. I think I can provide it by tomorrow. It's almost midnight ;)

I'll wait for a proper fix. Dropping this patch. Thanks.

Thanks Doug!

Quoted 7 lines

I have a fix ready but still have to test it with iSER. Unfortunately I have a hard time to test iSER with siw, since both iSCSI-TCP target and iSER want to bind to the same TCP port. While this may be considered a bug in that code, siw is the first RDMA provider to take notice (since using kernel sockets and not offloaded, hitting a TCP port already bound).

Not sure if this will become a serious problem when a iSCSI target is configured to serve both iSCSI-TCP & iSER connections simultaniously. Because, offloaded iSER CM handles all the TCP SYN packets that were destined to iSCSI-TCP.

Quoted 3 lines

I sent the patch to Chelsio folks and hope for
the best. They know the trick to make it working.

I have tested your patch, it's working fine.

Quoted 4 lines

Thanks
Bernard.