IB/srp bug fixes
v2 cover IB/srp bug fixes
Hello Doug,
The patches in this series are the initiator patches I came up with while testing the SRP initiator and target drivers. Please consider these patches for inclusion in the upstream kernel.
The changes compared to the previous version of this series are:
- Left out the "Use the IB_DEVICE_SG_GAPS_REG HCA feature if supported" patch.
- Added a patch to stop using memory regions with gaps since Laurence ran into an issue with these memory regions that cannot be explained by analzying the SRP initiator code only.
- Addressed Leon's comments about patch "Avoid that duplicate responses trigger a kernel bug" and "Add support for draining IB_POLL_DIRECT completion queues".
The patches in this series are:
Bart Van Assche (8):
IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
IB/srp: Avoid that duplicate responses trigger a kernel bug IB/srp: Fix race conditions related to task management IB/srp: Document locking conventions
IB/srp: Make a diagnostic message more informative IB/srp: Improve an error path
IB/core: Add support for draining IB_POLL_DIRECT completion queues IB/srp: Drain the send queue before destroying a QP
drivers/infiniband/core/cq.c | 4 +- drivers/infiniband/core/verbs.c | 35 ++++++-------- drivers/infiniband/ulp/srp/ib_srp.c | 92 ++++++++++++++++++++++++------------- drivers/infiniband/ulp/srp/ib_srp.h | 1 + 4 files changed, 77 insertions(+), 55 deletions(-)
Thanks,
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170214185636.29250-1-bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org
PATCH v2 1/8 IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 2500002a ad9fafd1
scsi host1: ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 25000032 00105dd0
scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the blk_queue_virt_boundary() to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).
--- drivers/infiniband/ulp/srp/ib_srp.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +3 -8
@@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,struct srp_fr_desc *d;struct ib_mr *mr;int i, ret = -EINVAL ; -enum ib_mr_type mr_type;if (pool_size <=0 ) goto err;
@@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,spin_lock_init (&pool->lock);INIT_LIST_HEAD (&pool->free_list); -if (device->attrs.device_cap_flags &IB_DEVICE_SG_GAPS_REG ) - mr_type =IB_MR_TYPE_SG_GAPS ; -else - mr_type =IB_MR_TYPE_MEM_REG ; -for (i =0 , d = &pool->desc[0 ]; i < pool->size; i++, d++) { - mr =ib_alloc_mr (pd, mr_type, max_page_list_len); + mr =ib_alloc_mr (pd,IB_MR_TYPE_MEM_REG , + max_page_list_len);if (IS_ERR (mr)) { ret =PTR_ERR (mr);if (ret == -ENOMEM )
@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)struct srp_device *srp_dev = target->srp_host->srp_dev;struct ib_device *ibdev = srp_dev->dev; -if (!(ibdev->attrs.device_cap_flags &IB_DEVICE_SG_GAPS_REG )) +if (true)blk_queue_virt_boundary (sdev->request_queue, ~srp_dev->mr_page_mask);
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170214185636.29250-2-bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
drivers/infiniband/ulp/srp/ib_srp.c static int srp_slave_alloc(struct scsi_device *sdev) · line 2666
Quoted 8 lines · Bart Van Assche
drivers/infiniband/ulp/srp/ib_srp.c +1
-1@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)struct srp_device *srp_dev = target->srp_host->srp_dev;struct ib_device *ibdev = srp_dev->dev; -if (!(ibdev->attrs.device_cap_flags &IB_DEVICE_SG_GAPS_REG )) +if (true)blk_queue_virt_boundary (sdev->request_queue, ~srp_dev->mr_page_mask);
Hello Doug,
This patch introduces a new warning when building with W=1, namely that variable ibdev is not used. The build bot figured this out. Do you want me to address this?
Thanks,
Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
In reply to a message not in the archiveQuoted 16 lines · Bart Van Assche
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).
According to the error dump, we have an issue with max_page_list_len supplied and/or internal calculations from that value to the UMR byte count.
I assume that iser works with SG_GAPS.
Quoted 59 lines
Fixes commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")Fixes commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")Reported-by Laurence Oberman <[email protected]>Signed-off-by Bart Van Assche <[email protected]>Cc Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/[email protected]>Cc Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>Cc Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>Cc Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/[email protected]>Cc Yuval Shaia <[email protected]>Cc <[email protected]> # 4.7+--- drivers/infiniband/ulp/srp/ib_srp.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)drivers/infiniband/ulp/srp/ib_srp.c +3
-9@@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,struct srp_fr_desc *d;struct ib_mr *mr;int i, ret = -EINVAL ; -enum ib_mr_type mr_type;if (pool_size <=0 ) goto err;@@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,spin_lock_init (&pool->lock);INIT_LIST_HEAD (&pool->free_list); -if (device->attrs.device_cap_flags &IB_DEVICE_SG_GAPS_REG ) - mr_type =IB_MR_TYPE_SG_GAPS ; -else - mr_type =IB_MR_TYPE_MEM_REG ; -for (i =0 , d = &pool->desc[0 ]; i < pool->size; i++, d++) { - mr =ib_alloc_mr (pd, mr_type, max_page_list_len); + mr =ib_alloc_mr (pd,IB_MR_TYPE_MEM_REG , + max_page_list_len);if (IS_ERR (mr)) { ret =PTR_ERR (mr);if (ret == -ENOMEM )@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)struct srp_device *srp_dev = target->srp_host->srp_dev;struct ib_device *ibdev = srp_dev->dev; -if (!(ibdev->attrs.device_cap_flags &IB_DEVICE_SG_GAPS_REG )) +if (true)blk_queue_virt_boundary (sdev->request_queue, ~srp_dev->mr_page_mask); --2.11.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 17 lines · Leon Romanovsky
Quoted 16 lines · Bart Van Assche
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138
Quoted 7 lines
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).According to the error dump, we have an issue with
max_page_list_lensupplied and/or internal calculations from that value to the UMR byte count.
Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following fixup?
drivers/infiniband/hw/mlx5/mr.c +2 -1
@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages,for (pages_mapped =0 ; pages_mapped < pages_to_map && !err; pages_mapped += pages_iter, idx += pages_iter) { + npages =min_t (int, pages_iter, pages_to_map - pages_mapped);dma_sync_single_for_cpu (ddev, dma, size,DMA_TO_DEVICE ); - npages =populate_xlt (mr, idx, pages_iter, xlt, + npages =populate_xlt (mr, idx, npages, xlt, page_shift, size, flags);dma_sync_single_for_device (ddev, dma, size,DMA_TO_DEVICE );
[1] https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
Thanks
A message that is not in the archive (sent off-list or not yet received): 20170215081945.GP6989-U/[email protected]
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
----- Original Message -----
Quoted 60 lines
From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>
To: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc: "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 3:19:45 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 19 lines · Leon Romanovsky
Quoted 18 lines · Bart Van Assche
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138Quoted 8 lines
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).According to the error dump, we have an issue with
max_page_list_lensupplied and/or
internal calculations from that value to the UMR byte count.Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following fixup?
drivers/infiniband/hw/mlx5/mr.c +2
-1@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx,int npages, for (pages_mapped = 0; pages_mapped < pages_to_map && !err; pages_mapped += pages_iter, idx += pages_iter) { + npages =min_t (int, pages_iter, pages_to_map - pages_mapped);dma_sync_single_for_cpu (ddev, dma, size,DMA_TO_DEVICE ); - npages =populate_xlt (mr, idx, pages_iter, xlt, + npages =populate_xlt (mr, idx, npages, xlt, page_shift, size, flags);
dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
Quoted 5 lines
[1]
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-nextThanks
Hello Leon
Replied earlier but I dont know if my reply made it. I will have to test this.
is this repo https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next already patched with the change you want. If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
----- Original Message -----
Quoted 92 lines
From: "Laurence Oberman" <[email protected]>
To: "Leon Romanovsky" <[email protected]>
Cc: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>, "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 8:33:35 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPS----- Original Message -----
Quoted 62 lines
From: "Leon Romanovsky" <[email protected]>
To: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>
Cc: "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark
Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4
. 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 3:19:45 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 19 lines · Leon Romanovsky
Quoted 18 lines · Bart Van Assche
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138Quoted 8 lines
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).According to the error dump, we have an issue with
max_page_list_lensupplied and/or
internal calculations from that value to the UMR byte count.Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following fixup?
drivers/infiniband/hw/mlx5/mr.c +2
-1@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages, for (pages_mapped = 0; pages_mapped < pages_to_map && !err; pages_mapped += pages_iter, idx += pages_iter) { + npages =min_t (int, pages_iter, pages_to_map - pages_mapped);dma_sync_single_for_cpu (ddev, dma, size,DMA_TO_DEVICE ); - npages =populate_xlt (mr, idx, pages_iter, xlt, + npages =populate_xlt (mr, idx, npages, xlt, page_shift, size, flags);Quoted 7 lines
dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);[1]
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-nextThanks
Hello Leon
Replied earlier but I dont know if my reply made it. I will have to test this.is this repo
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.
Quoted 7 lines
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
I guess not, I looked at applying this patch to Linus's tree but the function mlx5_ib_update_xlt() is not in there yet.
I will see if I can get your tree staged and test this for you.
Thanks
Laurence
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 103 lines · Laurence Oberman
----- Original Message -----
Quoted 92 lines
From: "Laurence Oberman" <[email protected]>
To: "Leon Romanovsky" <[email protected]>
Cc: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>, "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 8:33:35 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPS----- Original Message -----
Quoted 62 lines
From: "Leon Romanovsky" <[email protected]>
To: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>
Cc: "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark
Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4
. 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 3:19:45 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 19 lines · Leon Romanovsky
Quoted 18 lines · Bart Van Assche
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138Quoted 8 lines
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).According to the error dump, we have an issue with
max_page_list_lensupplied and/or
internal calculations from that value to the UMR byte count.Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following fixup?
drivers/infiniband/hw/mlx5/mr.c +2
-1@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages, for (pages_mapped = 0; pages_mapped < pages_to_map && !err; pages_mapped += pages_iter, idx += pages_iter) { + npages =min_t (int, pages_iter, pages_to_map - pages_mapped);dma_sync_single_for_cpu (ddev, dma, size,DMA_TO_DEVICE ); - npages =populate_xlt (mr, idx, pages_iter, xlt, + npages =populate_xlt (mr, idx, npages, xlt, page_shift, size, flags);Quoted 7 lines
dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);[1]
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-nextThanks
Hello Leon
Replied earlier but I dont know if my reply made it. I will have to test this.is this repo
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.Quoted 7 lines
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
I guess not, I looked at applying this patch to Linus's tree but the function
mlx5_ib_update_xlt()is not in there yet.
This is why I asked to use my tree, there is a chance that wrong calculation was before we introduced mlx5_ib_update_xlt() function.
Quoted 2 lines
I will see if I can get your tree staged and test this for you.
Thanks a lot.
Quoted 7 lines
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
----- Original Message -----
Quoted 123 lines
From: "Leon Romanovsky" <[email protected]>
To: "Laurence Oberman" <[email protected]>
Cc: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>, "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 8:47:07 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 109 lines · Laurence Oberman
----- Original Message -----
Quoted 98 lines
From: "Laurence Oberman" <[email protected]>
To: "Leon Romanovsky" <[email protected]>
Cc: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>, "Doug Ledford"
<[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark Bloch"
<[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 8:33:35 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPS----- Original Message -----
Quoted 64 lines
From: "Leon Romanovsky" <[email protected]>
To: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>
Cc: "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark
Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4
. 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 3:19:45 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 20 lines · Leon Romanovsky
Quoted 19 lines · Bart Van Assche
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation
error (6) for CQEffff880b92860138Quoted 8 lines
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).According to the error dump, we have an issue with
max_page_list_lensupplied and/or
internal calculations from that value to the UMR byte count.Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following
fixup?drivers/infiniband/hw/mlx5/mr.c +2
-1@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages, for (pages_mapped = 0; pages_mapped < pages_to_map && !err; pages_mapped += pages_iter, idx += pages_iter) { + npages =min_t (int, pages_iter, pages_to_map - pages_mapped);dma_sync_single_for_cpu (ddev, dma, size,DMA_TO_DEVICE ); - npages =populate_xlt (mr, idx, pages_iter, xlt, + npages =populate_xlt (mr, idx, npages, xlt, page_shift, size, flags);Quoted 7 lines
dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);[1]
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-nextThanks
Hello Leon
Replied earlier but I dont know if my reply made it. I will have to test this.is this repo
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on
Linus's tree where I reverted the patch.Quoted 7 lines
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 3 lines
I guess not, I looked at applying this patch to Linus's tree but the function
mlx5_ib_update_xlt()is not in there yet.
Quoted 16 lines
This is why I asked to use my tree, there is a chance that wrong calculation was before we introduced
mlx5_ib_update_xlt()function.Quoted 2 lines
I will see if I can get your tree staged and test this for you.
Thanks a lot.
Quoted 7 lines
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello Leon
I pulled your tree, ran git checkout rdma-next
I applied the patch manually, built the kernel and started the tests.
Pretty soon I ran into
ibclient login: [ 132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
[ 132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0 [ 132.718106] 00000000 00000000 00000000 00000000 [ 132.743616] 00000000 00000000 00000000 00000000 [ 132.767790] 00000000 00000000 00000000 00000000 [ 132.793626] 00000000 0f007806 2500002a 8ad015d1 [ 136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links [ 136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links [ 147.791028] scsi host2: ib_srp: reconnect succeeded [ 147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330 [ 162.908764] scsi host2: ib_srp: reconnect succeeded [ 162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330 [ 166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570 [ 177.292496] scsi host2: ib_srp: reconnect succeeded [ 177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 177.363793] 00000000 00000000 00000000 00000000 [ 177.388434] 00000000 00000000 00000000 00000000 [ 177.413918] 00000000 00000000 00000000 00000000 [ 177.438911] 00000000 0f007806 25000042 00102dd0 [ 177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138 [ 181.386124] scsi host1: ib_srp: reconnect succeeded [ 181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 193.230036] scsi host2: ib_srp: reconnect succeeded [ 193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 193.356245] 00000000 00000000 00000000 00000000 [ 193.379488] 00000000 00000000 00000000 00000000 [ 193.404853] 00000000 00000000 00000000 00000000 [ 193.429633] 00000000 0f007806 2500004a 006c46d0 [ 193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8 [ 196.178111] scsi host1: ib_srp: reconnect succeeded [ 196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 208.277396] scsi host2: ib_srp: reconnect succeeded [ 208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 208.349012] 00000000 00000000 00000000 00000000 [ 208.374873] 00000000 00000000 00000000 00000000 [ 208.401070] 00000000 00000000 00000000 00000000 [ 208.426873] 00000000 0f007806 25000052 00103dd0 [ 208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138 [ 211.066945] scsi host1: ib_srp: reconnect succeeded [ 211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 223.296126] scsi host2: ib_srp: reconnect succeeded [ 223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 223.423310] 00000000 00000000 00000000 00000000 [ 223.448590] 00000000 00000000 00000000 00000000 [ 223.473520] 00000000 00000000 00000000 00000000 [ 223.499008] 00000000 0f007806 2500005a 006c56d0 [ 223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8 [ 225.463312] scsi host1: ib_srp: reconnect succeeded [ 225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 239.254388] scsi host2: ib_srp: reconnect succeeded [ 239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0 [ 240.619914] scsi host1: ib_srp: reconnect succeeded [ 240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 254.620679] scsi host2: ib_srp: reconnect succeeded [ 254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0 [ 255.798616] scsi host1: ib_srp: reconnect succeeded [ 255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 269.228783] scsi host2: ib_srp: reconnect succeeded [ 269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 269.300655] 00000000 00000000 00000000 00000000 [ 269.325885] 00000000 00000000 00000000 00000000 [ 269.350457] 00000000 00000000 00000000 00000000 [ 269.375757] 00000000 0f007806 25000072 005847d0 [ 269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8 [ 269.856464] scsi host1: ib_srp: reconnect succeeded [ 269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
Thanks
Laurence
A message that is not in the archive (sent off-list or not yet received): 242820990.31706010.1487170436012.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 209 lines · Laurence Oberman
----- Original Message -----
Quoted 123 lines
From: "Leon Romanovsky" <[email protected]>
To: "Laurence Oberman" <[email protected]>
Cc: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>, "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 8:47:07 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 109 lines · Laurence Oberman
----- Original Message -----
Quoted 98 lines
From: "Laurence Oberman" <[email protected]>
To: "Leon Romanovsky" <[email protected]>
Cc: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>, "Doug Ledford"
<[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark Bloch"
<[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 8:33:35 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPS----- Original Message -----
Quoted 64 lines
From: "Leon Romanovsky" <[email protected]>
To: "Bart Van Assche" <[email protected]>, "Max Gurtovoy" <[email protected]>
Cc: "Doug Ledford" <[email protected]>, [email protected], "Israel Rukshin" <[email protected]>, "Mark
Bloch" <[email protected]>, "Yuval Shaia" <[email protected]>, "Artemy Kovalyov" <[email protected]>, "# 4
. 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 3:19:45 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 20 lines · Leon Romanovsky
Quoted 19 lines · Bart Van Assche
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation
error (6) for CQEffff880b92860138Quoted 8 lines
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).According to the error dump, we have an issue with
max_page_list_lensupplied and/or
internal calculations from that value to the UMR byte count.Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following
fixup?drivers/infiniband/hw/mlx5/mr.c +2
-1@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages, for (pages_mapped = 0; pages_mapped < pages_to_map && !err; pages_mapped += pages_iter, idx += pages_iter) { + npages =min_t (int, pages_iter, pages_to_map - pages_mapped);dma_sync_single_for_cpu (ddev, dma, size,DMA_TO_DEVICE ); - npages =populate_xlt (mr, idx, pages_iter, xlt, + npages =populate_xlt (mr, idx, npages, xlt, page_shift, size, flags);Quoted 7 lines
dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);[1]
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-nextThanks
Hello Leon
Replied earlier but I dont know if my reply made it. I will have to test this.is this repo
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on
Linus's tree where I reverted the patch.Quoted 7 lines
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 3 lines
I guess not, I looked at applying this patch to Linus's tree but the function
mlx5_ib_update_xlt()is not in there yet.Quoted 16 lines
This is why I asked to use my tree, there is a chance that wrong calculation was before we introduced
mlx5_ib_update_xlt()function.Quoted 2 lines
I will see if I can get your tree staged and test this for you.
Thanks a lot.
Quoted 7 lines
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlHello Leon
I pulled your tree, ran git checkout rdma-next
I applied the patch manually, built the kernel and started the tests.Pretty soon I ran into
ibclient login: [ 132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
[ 132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0 [ 132.718106] 00000000 00000000 00000000 00000000 [ 132.743616] 00000000 00000000 00000000 00000000 [ 132.767790] 00000000 00000000 00000000 00000000 [ 132.793626] 00000000 0f007806 2500002a 8ad015d1 [ 136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links [ 136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links [ 147.791028] scsi host2: ib_srp: reconnect succeeded [ 147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330 [ 162.908764] scsi host2: ib_srp: reconnect succeeded [ 162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330 [ 166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570 [ 177.292496] scsi host2: ib_srp: reconnect succeeded [ 177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 177.363793] 00000000 00000000 00000000 00000000 [ 177.388434] 00000000 00000000 00000000 00000000 [ 177.413918] 00000000 00000000 00000000 00000000 [ 177.438911] 00000000 0f007806 25000042 00102dd0 [ 177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138 [ 181.386124] scsi host1: ib_srp: reconnect succeeded [ 181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 193.230036] scsi host2: ib_srp: reconnect succeeded [ 193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 193.356245] 00000000 00000000 00000000 00000000 [ 193.379488] 00000000 00000000 00000000 00000000 [ 193.404853] 00000000 00000000 00000000 00000000 [ 193.429633] 00000000 0f007806 2500004a 006c46d0 [ 193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8 [ 196.178111] scsi host1: ib_srp: reconnect succeeded [ 196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 208.277396] scsi host2: ib_srp: reconnect succeeded [ 208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 208.349012] 00000000 00000000 00000000 00000000 [ 208.374873] 00000000 00000000 00000000 00000000 [ 208.401070] 00000000 00000000 00000000 00000000 [ 208.426873] 00000000 0f007806 25000052 00103dd0 [ 208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138 [ 211.066945] scsi host1: ib_srp: reconnect succeeded [ 211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 223.296126] scsi host2: ib_srp: reconnect succeeded [ 223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 223.423310] 00000000 00000000 00000000 00000000 [ 223.448590] 00000000 00000000 00000000 00000000 [ 223.473520] 00000000 00000000 00000000 00000000 [ 223.499008] 00000000 0f007806 2500005a 006c56d0 [ 223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8 [ 225.463312] scsi host1: ib_srp: reconnect succeeded [ 225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 239.254388] scsi host2: ib_srp: reconnect succeeded [ 239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0 [ 240.619914] scsi host1: ib_srp: reconnect succeeded [ 240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 254.620679] scsi host2: ib_srp: reconnect succeeded [ 254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0 [ 255.798616] scsi host1: ib_srp: reconnect succeeded [ 255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0 [ 269.228783] scsi host2: ib_srp: reconnect succeeded
Quoted 9 lines
[ 269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe [ 269.300655] 00000000 00000000 00000000 00000000 [ 269.325885] 00000000 00000000 00000000 00000000 [ 269.350457] 00000000 00000000 00000000 00000000 [ 269.375757] 00000000 0f007806 25000072 005847d0 [ 269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8 [ 269.856464] scsi host1: ib_srp: reconnect succeeded [ 269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
Thanks a lot for your effort, we tried fast path to debug, but it didn't work :(.
Quoted 7 lines
Thanks
Laurence--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
In reply to Laurence ObermanQuoted 5 lines
Hello Leon
I pulled your tree, ran git checkout rdma-next
I applied the patch manually, built the kernel and started the tests.
Quoted 8 lines
Pretty soon I ran into
ibclient login: [ 132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
[ 132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0 [ 132.718106] 00000000 00000000 00000000 00000000 [ 132.743616] 00000000 00000000 00000000 00000000 [ 132.767790] 00000000 00000000 00000000 00000000 [ 132.793626] 00000000 0f007806 2500002a 8ad015d1
Not surprising given that the ib_map_mr_sg code never visits there...
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
In reply to Bart Van AsscheQuoted 17 lines
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138
Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).
Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly incremented. Does the following change fix the problem? --
drivers/infiniband/hw/mlx5/mr.c +1 -1
@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr, klms[i].va =cpu_to_be64 (sg_dma_address (sg) + sg_offset); klms[i].bcount =cpu_to_be32 (sg_dma_len (sg) - sg_offset); klms[i].key =cpu_to_be32 (lkey); - mr->ibmr.length +=sg_dma_len (sg); + mr->ibmr.length +=sg_dma_len (sg) - sg_offset; sg_offset =0 ; }
A message that is not in the archive (sent off-list or not yet received): cebcaeae-94a6-de82-cfc8-ce055b273836-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
----- Original Message -----
Quoted 57 lines
From: "Sagi Grimberg" <[email protected]>
To: "Bart Van Assche" <[email protected]>, "Doug Ledford" <[email protected]>
Cc: [email protected], "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Yuval Shaia" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 10:38:06 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 18 lines
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Looking at
mlx5_ib_sg_to_klms, I think themr->lengthis incorrectly incremented. Does the following change fix the problem? --drivers/infiniband/hw/mlx5/mr.c +1
-3@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr, klms[i].va =cpu_to_be64 (sg_dma_address (sg) + sg_offset); klms[i].bcount =cpu_to_be32 (sg_dma_len (sg) - sg_offset); klms[i].key =cpu_to_be32 (lkey); - mr->ibmr.length +=sg_dma_len (sg); + mr->ibmr.length +=sg_dma_len (sg) - sg_offset; sg_offset =0 ; } -- --
Quoted 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello Sagi,
I will get this tested.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
In reply to a message not in the archiveQuoted 24 lines · Sagi Grimberg
Quoted 19 lines
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).
Quoted 5 lines
Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Sagi,
this syndrome says that number of klms to write is bigger than number of mtts.
Artemy started investigating it and proposed solution that were tested by Laurence. Let's see if your fix will help.
Quoted 10 lines
Looking at
mlx5_ib_sg_to_klms, I think themr->lengthis incorrectly incremented. Does the following change fix the problem? --drivers/infiniband/hw/mlx5/mr.c +0
-0
drivers/infiniband/hw/mlx5/mr.c mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
Quoted 9 lines
drivers/infiniband/hw/mlx5/mr.c +1
-2@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr, klms[i].va =cpu_to_be64 (sg_dma_address (sg) + sg_offset); klms[i].bcount =cpu_to_be32 (sg_dma_len (sg) - sg_offset); klms[i].key =cpu_to_be32 (lkey); - mr->ibmr.length +=sg_dma_len (sg); + mr->ibmr.length +=sg_dma_len (sg) - sg_offset; sg_offset =0 ; } --
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 10 lines
Quoted 6 lines
Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Sagi,
this syndrome says that number of klms to write is bigger than number of mtts.
That is strange, given that we check for it explicitly...
if this is indeed the case, then mlx5 reports wrong max_fast_reg_page_list_len.
Quoted 3 lines
Artemy started investigating it and proposed solution that were tested by Laurence.
Let's see if your fix will help.
Not sure it will help if the syndrome is what you say it is...
A message that is not in the archive (sent off-list or not yet received): 0514bb01-95cf-c10a-b883-494f149845f3-VPRAkNaXOzVWk0Htik3J/[email protected]
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 33 lines · Max Gurtovoy
Quoted 24 lines · Sagi Grimberg
Quoted 19 lines
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).Quoted 5 lines
Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Quoted 7 lines
Sagi,
this syndrome says that number of klms to write is bigger than number of mtts.Artemy started investigating it and proposed solution that were tested by Laurence.
Let's see if your fix will help.
No, Artemy's change doesn't fix it.
Quoted 24 lines
Quoted 11 lines
Looking at
mlx5_ib_sg_to_klms, I think themr->lengthis incorrectly incremented. Does the following change fix the problem? --drivers/infiniband/hw/mlx5/mr.c +0
-0drivers/infiniband/hw/mlx5/mr.c mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
Quoted 9 lines
drivers/infiniband/hw/mlx5/mr.c +1
-2@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr, klms[i].va =cpu_to_be64 (sg_dma_address (sg) + sg_offset); klms[i].bcount =cpu_to_be32 (sg_dma_len (sg) - sg_offset); klms[i].key =cpu_to_be32 (lkey); - mr->ibmr.length +=sg_dma_len (sg); + mr->ibmr.length +=sg_dma_len (sg) - sg_offset; sg_offset =0 ; } ----
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
In reply to a message not in the archive----- Original Message -----
Quoted 57 lines
From: "Sagi Grimberg" <[email protected]>
To: "Bart Van Assche" <[email protected]>, "Doug Ledford" <[email protected]>
Cc: [email protected], "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/[email protected]>, "Yuval Shaia" <[email protected]>, "# 4 . 7+" <[email protected]>
Sent: Wednesday, February 15, 2017 10:38:06 AM
Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPSQuoted 18 lines
Tests have shown that the following error message is reported when using SG-GAPS registration with an mlx5 adapter:
scsi host1:
ib_srp: failed RECV status WR flushed (5) for CQEffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500002aad9fafd1
scsi host1:ib_srp: reconnect succeeded mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
000000000f0078062500003200105dd0
scsi host1:ib_srp: failed FAST REG status memory management operation error (6) for CQEffff880b92860138Quoted 4 lines
Hence avoid using SG-GAPS memory registrations. Additionally, always configure the
blk_queue_virt_boundary()to avoid to trigger a mapping failure when using adapters that support SG-GAPS (e.g. mlx5).Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Looking at
mlx5_ib_sg_to_klms, I think themr->lengthis incorrectly incremented. Does the following change fix the problem? --drivers/infiniband/hw/mlx5/mr.c +1
-3@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr, klms[i].va =cpu_to_be64 (sg_dma_address (sg) + sg_offset); klms[i].bcount =cpu_to_be32 (sg_dma_len (sg) - sg_offset); klms[i].key =cpu_to_be32 (lkey); - mr->ibmr.length +=sg_dma_len (sg); + mr->ibmr.length +=sg_dma_len (sg) - sg_offset; sg_offset =0 ; } -- --
Quoted 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Very quickly get to this
[ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe [ 181.016899] 00000000 00000000 00000000 00000000 [ 181.040949] 00000000 00000000 00000000 00000000 [ 181.066960] 00000000 00000000 00000000 00000000 [ 181.092030] 00000000 0f007806 2500002a bf1913d0 [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778 [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 197.090886] scsi host2: ib_srp: reconnect succeeded [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 17 lines
Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017
x86_64x86_64x86_64GNU/LinuxVery quickly get to this
[ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe [ 181.016899] 00000000 00000000 00000000 00000000 [ 181.040949] 00000000 00000000 00000000 00000000 [ 181.066960] 00000000 00000000 00000000 00000000 [ 181.092030] 00000000 0f007806 2500002a bf1913d0 [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778 [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 197.090886] scsi host2: ib_srp: reconnect succeeded [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
Quoted 3 lines
So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
Yea, probably this is the best way forward.
Bart, I think the change I suggested is still needed regardless, do you agree?
Max, Leon, is it possible that the max number of klms pr mr is less than what reported in device capabilities for page_list_len?
If so, this means that either:
1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
2. we need yet another capability for SG_GAPS (sucks^2 because the whole point was to make it transparent to the user)
3. mlx5 does not support SG_GAPS (sucks^3 because we now have something thats not supported by any device).
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 2 lines · Sagi Grimberg
Bart, I think the change I suggested is still needed regardless, do you agree?
I'm not sure. I have not yet had a close look at that part of the mlx5 driver.
Bart.
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
In reply to Sagi GrimbergQuoted 21 lines · Sagi Grimberg
Quoted 17 lines
Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017
x86_64x86_64x86_64GNU/LinuxVery quickly get to this
[ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe [ 181.016899] 00000000 00000000 00000000 00000000 [ 181.040949] 00000000 00000000 00000000 00000000 [ 181.066960] 00000000 00000000 00000000 00000000 [ 181.092030] 00000000 0f007806 2500002a bf1913d0 [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778 [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 197.090886] scsi host2: ib_srp: reconnect succeeded [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30Quoted 3 lines
So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
Quoted 7 lines
Yea, probably this is the best way forward.
Bart, I think the change I suggested is still needed regardless, do you agree?
Max, Leon, is it possible that the max number of klms pr mr is less than what reported in device capabilities for
page_list_len?
I hope no and we will check.
I already asked it, but didn't get any response, and I'll repeat it again. ISER has similar code with SG_GAPS, does it work?
Quoted 11 lines
If so, this means that either:
1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
2. we need yet another capability forSG_GAPS(sucks^2 because the whole point was to make it transparent to the user)
3. mlx5 does not supportSG_GAPS(sucks^3 because we now have something thats not supported by any device).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
Quoted 27 lines · Leon Romanovsky
Quoted 26 lines · Sagi Grimberg
Quoted 17 lines
Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017
x86_64x86_64x86_64GNU/LinuxVery quickly get to this
[ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe [ 181.016899] 00000000 00000000 00000000 00000000 [ 181.040949] 00000000 00000000 00000000 00000000 [ 181.066960] 00000000 00000000 00000000 00000000 [ 181.092030] 00000000 0f007806 2500002a bf1913d0 [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778 [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 197.090886] scsi host2: ib_srp: reconnect succeeded [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30Quoted 3 lines
So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.Yea, probably this is the best way forward.
Bart, I think the change I suggested is still needed regardless, do you agree?
Quoted 6 lines
Quoted 2 lines
Max, Leon, is it possible that the max number of klms pr mr is less than what reported in device capabilities for
page_list_len?I hope no and we will check.
I already asked it, but didn't get any response, and I'll repeat it again. ISER has similar code withSG_GAPS, does it work?
Yes, I haven't seen issues with that in iSER.
We need to continue with the debug.
Quoted 12 lines
Quoted 12 lines
If so, this means that either:
1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
2. we need yet another capability forSG_GAPS(sucks^2 because the whole point was to make it transparent to the user)
3. mlx5 does not supportSG_GAPS(sucks^3 because we now have something thats not supported by any device).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 2/8 IB/srp: Avoid that duplicate responses trigger a kernel bug
In reply to a message not in the archiveAfter srp_process_rsp() returns there is a short time during which the scsi_host_find_tag() call will return a pointer to the SCSI command that is being completed. If during that time a duplicate response is received, avoid that the following call stack appears:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: srp_recv_done+0x450/0x6b0 [ib_srp] Oops: 0000 [#1] SMP CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.10.0-rc7-dbg+ #1 Call Trace: <IRQ> __ib_process_cq+0x4b/0xd0 [ib_core] ib_poll_handler+0x1d/0x70 [ib_core] irq_poll_softirq+0xba/0x120 __do_softirq+0xba/0x4c0 irq_exit+0xbe/0xd0 smp_apic_timer_interrupt+0x38/0x50 apic_timer_interrupt+0x90/0xa0 </IRQ> RIP: srp_recv_done+0x450/0x6b0 [ib_srp] RSP: ffff88046f483e20
--- drivers/infiniband/ulp/srp/ib_srp.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/srp/ib_srp.c +3 -1
@@ -1892,9 +1892,11 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp)complete (&ch->tsk_mgmt_done); }else { scmnd =scsi_host_find_tag (target->scsi_host, rsp->tag); -if (scmnd) { +if (scmnd && scmnd->host_scribble) { req = (void *)scmnd->host_scribble; scmnd =srp_claim_req (ch, req,NULL , scmnd); + }else { + scmnd =NULL ; }if (!scmnd) { shost_printk(KERN_ERR, target->scsi_host,
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 2/8] IB/srp: Avoid that duplicate responses trigger a kernel bug
Quoted 27 lines · Bart Van Assche
After
srp_process_rsp()returns there is a short time during which thescsi_host_find_tag()call will return a pointer to the SCSI command that is being completed. If during that time a duplicate response is received, avoid that the following call stack appears:BUG: unable to handle kernel NULL pointer dereference at (null) IP: srp_recv_done+0x450/0x6b0 [ib_srp] Oops: 0000 [#1] SMP CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.10.0-rc7-dbg+ #1 Call Trace: <IRQ> __ib_process_cq+0x4b/0xd0 [ib_core] ib_poll_handler+0x1d/0x70 [ib_core] irq_poll_softirq+0xba/0x120 __do_softirq+0xba/0x4c0 irq_exit+0xbe/0xd0 smp_apic_timer_interrupt+0x38/0x50 apic_timer_interrupt+0x90/0xa0 </IRQ> RIP: srp_recv_done+0x450/0x6b0 [ib_srp] RSP: ffff88046f483e20Signed-off-by Bart Van Assche <[email protected]>Cc Israel Rukshin <[email protected]>Cc Max Gurtovoy <[email protected]>Cc Laurence Oberman <[email protected]>Cc Steve Feeley <[email protected]>Cc <[email protected]>
Quoted 4 lines
--- drivers/infiniband/ulp/srp/ib_srp.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Thanks,
PATCH v2 3/8 IB/srp: Fix race conditions related to task management
In reply to a message not in the archiveAvoid that srp_process_rsp() overwrites the status information in ch if the SRP target response timed out and processing of another task management function has already started. Avoid that issuing multiple task management functions concurrently triggers list corruption. This patch prevents that the following stack trace appears in the system log:
WARNING: CPU: 8 PID: 9269 at lib/list_debug.c:52 __list_del_entry_valid+0xbc/0xc0 list_del corruption. prev->next should be ffffc90004bb7b00, but was ffff8804052ecc68 CPU: 8 PID: 9269 Comm: sg_reset Tainted: G W 4.10.0-rc7-dbg+ #3 Call Trace: dump_stack+0x68/0x93 __warn+0xc6/0xe0 warn_slowpath_fmt+0x4a/0x50 __list_del_entry_valid+0xbc/0xc0 wait_for_completion_timeout+0x12e/0x170 srp_send_tsk_mgmt+0x1ef/0x2d0 [ib_srp] srp_reset_device+0x5b/0x110 [ib_srp] scsi_ioctl_reset+0x1c7/0x290 scsi_ioctl+0x12a/0x420 sd_ioctl+0x9d/0x100 blkdev_ioctl+0x51e/0x9f0 block_ioctl+0x38/0x40 do_vfs_ioctl+0x8f/0x700 SyS_ioctl+0x3c/0x70 entry_SYSCALL_64_fastpath+0x18/0xad
--- drivers/infiniband/ulp/srp/ib_srp.c | 45 ++++++++++++++++++++++++------------- drivers/infiniband/ulp/srp/ib_srp.h | 1 + 2 files changed, 30 insertions(+), 16 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +29 -16
@@ -1884,12 +1884,17 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp)if (unlikely (rsp->tag &SRP_TAG_TSK_MGMT )) {spin_lock_irqsave (&ch->lock, flags); ch->req_lim +=be32_to_cpu (rsp->req_lim_delta); +if (rsp->tag == ch->tsk_mgmt_tag) { + ch->tsk_mgmt_status =-1 ; +if (be32_to_cpu (rsp->resp_data_len) >=4 ) + ch->tsk_mgmt_status = rsp->data[3 ]; +complete (&ch->tsk_mgmt_done); + }else { +shost_printk (KERN_ERR , target->scsi_host, +"Received tsk mgmt response too late for tag %#llx\n" , + rsp->tag); + }spin_unlock_irqrestore (&ch->lock, flags); - - ch->tsk_mgmt_status =-1 ; -if (be32_to_cpu (rsp->resp_data_len) >=4 ) - ch->tsk_mgmt_status = rsp->data[3 ]; -complete (&ch->tsk_mgmt_done); }else { scmnd =scsi_host_find_tag (target->scsi_host, rsp->tag);if (scmnd && scmnd->host_scribble) {
@@ -2528,19 +2533,18 @@ srp_change_queue_depth(struct scsi_device *sdev, int qdepth) }static int srp_send_tsk_mgmt (struct srp_rdma_ch *ch,u64 req_tag,u64 lun, -u8 func) +u8 func,u8 *status) {struct srp_target_port *target = ch->target;struct srp_rport *rport = target->rport;struct ib_device *dev = target->srp_host->srp_dev->dev;struct srp_iu *iu;struct srp_tsk_mgmt *tsk_mgmt; +int res;if (!ch->connected || target->qp_in_error)return -1 ; -init_completion (&ch->tsk_mgmt_done); -/* * Lock the rport mutex to avoid that srp_create_ch_ib() is * invoked while a task management function is being sent.
@@ -2563,10 +2567,16 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun, tsk_mgmt->opcode =SRP_TSK_MGMT ;int_to_scsilun (lun, &tsk_mgmt->lun); - tsk_mgmt->tag = req_tag |SRP_TAG_TSK_MGMT ; tsk_mgmt->tsk_mgmt_func = func; tsk_mgmt->task_tag = req_tag; +spin_lock_irq (&ch->lock); + ch->tsk_mgmt_tag = (ch->tsk_mgmt_tag +1 ) |SRP_TAG_TSK_MGMT ; + tsk_mgmt->tag = ch->tsk_mgmt_tag; +spin_unlock_irq (&ch->lock); + +init_completion (&ch->tsk_mgmt_done); +ib_dma_sync_single_for_device (dev, iu->dma,sizeof *tsk_mgmt,DMA_TO_DEVICE );if (srp_post_send (ch, iu,sizeof (*tsk_mgmt))) {
@@ -2575,13 +2585,15 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun,return -1 ; } + res =wait_for_completion_timeout (&ch->tsk_mgmt_done, +msecs_to_jiffies (SRP_ABORT_TIMEOUT_MS )); +if (res >0 && status) + *status = ch->tsk_mgmt_status;mutex_unlock (&rport->mutex); -if (!wait_for_completion_timeout (&ch->tsk_mgmt_done, -msecs_to_jiffies (SRP_ABORT_TIMEOUT_MS ))) -return -1 ; +WARN_ON_ONCE (res <0 ); -return 0 ; +return res >0 ?0 :-1 ; }static int srp_abort (struct scsi_cmnd *scmnd)
@@ -2607,7 +2619,7 @@ static int srp_abort(struct scsi_cmnd *scmnd)shost_printk (KERN_ERR , target->scsi_host,"Sending SRP abort for tag %#x\n" , tag);if (srp_send_tsk_mgmt (ch, tag, scmnd->device->lun, -SRP_TSK_ABORT_TASK ) ==0 ) +SRP_TSK_ABORT_TASK ,NULL ) ==0 ) ret =SUCCESS ;else if (target->rport->state ==SRP_RPORT_LOST ) ret =FAST_IO_FAIL ;
@@ -2625,14 +2637,15 @@ static int srp_reset_device(struct scsi_cmnd *scmnd)struct srp_target_port *target =host_to_target (scmnd->device->host);struct srp_rdma_ch *ch;int i; +u8 status;shost_printk (KERN_ERR , target->scsi_host,"SRP reset_device called\n" ); ch = &target->ch[0 ];if (srp_send_tsk_mgmt (ch,SRP_TAG_NO_REQ , scmnd->device->lun, -SRP_TSK_LUN_RESET )) +SRP_TSK_LUN_RESET , &status))return FAILED ; -if (ch->tsk_mgmt_status) +if (status)return FAILED ;for (i =0 ; i < target->ch_count; i++) {
drivers/infiniband/ulp/srp/ib_srp.h +1 -0
@@ -163,6 +163,7 @@ struct srp_rdma_ch {int max_ti_iu_len;int comp_vector; +u64 tsk_mgmt_tag;struct completion tsk_mgmt_done;u8 tsk_mgmt_status;bool connected;
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 4/8 IB/srp: Document locking conventions
In reply to a message not in the archiveUse lockdep_assert_held() statements to verify at run-time whether the proper locks are held.
--- drivers/infiniband/ulp/srp/ib_srp.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/infiniband/ulp/srp/ib_srp.c +5 -0
@@ -40,6 +40,7 @@#include <linux/parser.h> #include <linux/random.h> #include <linux/jiffies.h> +#include <linux/lockdep.h> #include <rdma/ib_cache.h> #include <linux/atomic.h>
@@ -1799,6 +1800,8 @@ static struct srp_iu *__srp_get_tx_iu(struct srp_rdma_ch *ch,s32 rsv = (iu_type ==SRP_IU_TSK_MGMT ) ?0 :SRP_TSK_MGMT_SQ_SIZE ;struct srp_iu *iu; +lockdep_assert_held (&ch->lock); +ib_process_cq_direct (ch->send_cq,-1 );if (list_empty (&ch->free_tx))
@@ -1829,6 +1832,8 @@ static void srp_send_done(struct ib_cq *cq, struct ib_wc *wc)return ; } +lockdep_assert_held (&ch->lock); +list_add (&iu->list, &ch->free_tx); }
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 5/8 IB/srp: Make a diagnostic message more informative
In reply to a message not in the archiveReport the destination port GID if connecting fails.
--- drivers/infiniband/ulp/srp/ib_srp.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +3 -2
@@ -3436,9 +3436,10 @@ static ssize_t srp_create_target(struct device *dev, ret =srp_connect_ch (ch, multich);if (ret) {shost_printk (KERN_ERR , target->scsi_host, -PFX "Connection %d/%d failed\n" , +PFX "Connection %d/%d to %pI6 failed\n" , ch_start + cpu_idx, - target->ch_count); + target->ch_count, + ch->target->orig_dgid.raw);if (node_idx ==0 && cpu_idx ==0 ) { goto err_disconnect; }else {
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 6/8 IB/srp: Improve an error path
In reply to a message not in the archiveAvoid that the following message is printed if login fails:
scsi host0: ib_srp: Sending CM DREQ failed
--- drivers/infiniband/ulp/srp/ib_srp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/srp/ib_srp.c +2 -1
@@ -3441,7 +3441,7 @@ static ssize_t srp_create_target(struct device *dev, target->ch_count, ch->target->orig_dgid.raw);if (node_idx ==0 && cpu_idx ==0 ) { - goto err_disconnect; + goto free_ch; }else {srp_free_ch_ib (target, ch);srp_free_req_data (target, ch);
@@ -3488,6 +3488,7 @@ static ssize_t srp_create_target(struct device *dev, err_disconnect:srp_disconnect_target (target); +free_ch:for (i =0 ; i < target->ch_count; i++) { ch = &target->ch[i];srp_free_ch_ib (target, ch);
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 7/8 IB/core: Add support for draining IB_POLL_DIRECT completion queues
In reply to a message not in the archive--- drivers/infiniband/core/cq.c | 4 ++-- drivers/infiniband/core/verbs.c | 35 +++++++++++++++-------------------- 2 files changed, 17 insertions(+), 22 deletions(-)
drivers/infiniband/core/cq.c +2 -2
@@ -58,8 +58,8 @@ static int __ib_process_cq(struct ib_cq *cq, int budget)* %IB_POLL_DIRECT CQ. It does not offload CQ processing to a different * context and does not ask for completion interrupts from the HCA. * -* Note: for compatibility reasons -1 can be passed in %budget for unlimited -* polling. Do not use this feature in new code, it will be removed soon. +* Note: do not pass -1 as %budget unless it is guaranteed that the number +* of completions that will be processed is small. */ int ib_process_cq_direct (struct ib_cq *cq,int budget) {
drivers/infiniband/core/verbs.c +15 -20
@@ -1949,17 +1949,12 @@ static void ib_drain_qp_done(struct ib_cq *cq, struct ib_wc *wc)*/ static void __ib_drain_sq (struct ib_qp *qp) { +struct ib_cq *cq = qp->send_cq;struct ib_qp_attr attr = { .qp_state =IB_QPS_ERR };struct ib_drain_cqe sdrain;struct ib_send_wr swr = {}, *bad_swr;int ret; -if (qp->send_cq->poll_ctx ==IB_POLL_DIRECT ) { -WARN_ONCE (qp->send_cq->poll_ctx ==IB_POLL_DIRECT , -"IB_POLL_DIRECT poll_ctx not supported for drain\n" ); -return ; - } - swr.wr_cqe = &sdrain.cqe; sdrain.cqe.done = ib_drain_qp_done;init_completion (&sdrain.done);
@@ -1976,7 +1971,11 @@ static void __ib_drain_sq(struct ib_qp *qp)return ; } -wait_for_completion (&sdrain.done); +if (cq->poll_ctx ==IB_POLL_DIRECT ) +while (wait_for_completion_timeout (&sdrain.done,HZ /10 ) <=0 ) +ib_process_cq_direct (cq,-1 ); +else +wait_for_completion (&sdrain.done); }/*
@@ -1984,17 +1983,12 @@ static void __ib_drain_sq(struct ib_qp *qp)*/ static void __ib_drain_rq (struct ib_qp *qp) { +struct ib_cq *cq = qp->recv_cq;struct ib_qp_attr attr = { .qp_state =IB_QPS_ERR };struct ib_drain_cqe rdrain;struct ib_recv_wr rwr = {}, *bad_rwr;int ret; -if (qp->recv_cq->poll_ctx ==IB_POLL_DIRECT ) { -WARN_ONCE (qp->recv_cq->poll_ctx ==IB_POLL_DIRECT , -"IB_POLL_DIRECT poll_ctx not supported for drain\n" ); -return ; - } - rwr.wr_cqe = &rdrain.cqe; rdrain.cqe.done = ib_drain_qp_done;init_completion (&rdrain.done);
@@ -2011,7 +2005,11 @@ static void __ib_drain_rq(struct ib_qp *qp)return ; } -wait_for_completion (&rdrain.done); +if (cq->poll_ctx ==IB_POLL_DIRECT ) +while (wait_for_completion_timeout (&rdrain.done,HZ /10 ) <=0 ) +ib_process_cq_direct (cq,-1 ); +else +wait_for_completion (&rdrain.done); }/**
@@ -2028,8 +2026,7 @@ static void __ib_drain_rq(struct ib_qp *qp)* ensure there is room in the CQ and SQ for the drain work request and * completion. * -* allocate the CQ using ib_alloc_cq() and the CQ poll context cannot be -* IB_POLL_DIRECT. +* allocate the CQ using ib_alloc_cq(). * * ensure that there are no other contexts that are posting WRs concurrently. * Otherwise the drain is not guaranteed.
@@ -2057,8 +2054,7 @@ EXPORT_SYMBOL(ib_drain_sq);* ensure there is room in the CQ and RQ for the drain work request and * completion. * -* allocate the CQ using ib_alloc_cq() and the CQ poll context cannot be -* IB_POLL_DIRECT. +* allocate the CQ using ib_alloc_cq(). * * ensure that there are no other contexts that are posting WRs concurrently. * Otherwise the drain is not guaranteed.
@@ -2082,8 +2078,7 @@ EXPORT_SYMBOL(ib_drain_rq);* ensure there is room in the CQ(s), SQ, and RQ for drain work requests * and completions. * -* allocate the CQs using ib_alloc_cq() and the CQ poll context cannot be -* IB_POLL_DIRECT. +* allocate the CQs using ib_alloc_cq(). * * ensure that there are no other contexts that are posting WRs concurrently. * Otherwise the drain is not guaranteed.
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170214185636.29250-8-bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org
Steve Wise
Reviewed-byRE: [PATCH v2 7/8] IB/core: Add support for draining IB_POLL_DIRECT completion queues
Quoted 8 lines
Subject: [PATCH v2 7/8] IB/core: Add support for draining
IB_POLL_DIRECTcompletion queuesSigned-off-by Bart Van Assche <[email protected]>Cc Steve Wise <[email protected]>Cc Chuck Lever <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>
Quoted 5 lines
--- drivers/infiniband/core/cq.c | 4 ++-- drivers/infiniband/core/verbs.c | 35 +++++++++++++++-------------------- 2 files changed, 17 insertions(+), 22 deletions(-)
Looks ok to me.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Leon Romanovsky
Reviewed-byRe: [PATCH v2 7/8] IB/core: Add support for draining IB_POLL_DIRECT completion queues
In reply to a message not in the archiveQuoted 15 lines · Bart Van Assche
Signed-off-by Bart Van Assche <[email protected]>Cc Steve Wise <[email protected]>Cc Chuck Lever <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>--- drivers/infiniband/core/cq.c | 4 ++-- drivers/infiniband/core/verbs.c | 35 +++++++++++++++-------------------- 2 files changed, 17 insertions(+), 22 deletions(-)drivers/infiniband/core/cq.c +0
-0
drivers/infiniband/core/cq.c static int __ib_process_cq(struct ib_cq *cq, int budget) · line 63
Quoted 8 lines
drivers/infiniband/core/cq.c +2
-2@@ -58,8 +58,8 @@ static int __ib_process_cq(struct ib_cq *cq, int budget)* %IB_POLL_DIRECT CQ. It does not offload CQ processing to a different * context and does not ask for completion interrupts from the HCA. * -* Note: for compatibility reasons -1 can be passed in %budget for unlimited -* polling. Do not use this feature in new code, it will be removed soon. +* Note: do not pass -1 as %budget unless it is guaranteed that the number +* of completions that will be processed is small. */
Interesting solution.
Thanks,
PATCH v2 8/8 IB/srp: Drain the send queue before destroying a QP
In reply to a message not in the archiveA quote from the IB spec:
However, if the Consumer does not wait for the Affiliated Asynchronous
Last WQE Reached Event, then WQE and Data Segment leakage may occur. Therefore, it is good programming practice to tear down a QP that is associated with an SRQ by using the following process:
* Put the QP in the Error State;
* wait for the Affiliated Asynchronous Last WQE Reached Event;
* either:
* drain the CQ by invoking the Poll CQ verb and either wait for CQ
to be empty or the number of Poll CQ operations has exceeded CQ capacity size; or
* post another WR that completes on the same CQ and wait for this WR to return as a WC;
* and then invoke a Destroy QP or Reset QP.
--- drivers/infiniband/ulp/srp/ib_srp.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +14 -5
@@ -466,9 +466,13 @@ static struct srp_fr_pool *srp_alloc_fr_pool(struct srp_target_port *target)* completion handler can access the queue pair while it is * being destroyed. */ -static void srp_destroy_qp (struct ib_qp *qp) +static void srp_destroy_qp (struct srp_rdma_ch *ch,struct ib_qp *qp) { -ib_drain_rq (qp); +spin_lock_irq (&ch->lock); +ib_process_cq_direct (ch->send_cq,-1 ); +spin_unlock_irq (&ch->lock); + +ib_drain_qp (qp);ib_destroy_qp (qp); }
@@ -542,7 +546,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) }if (ch->qp) -srp_destroy_qp (ch->qp); +srp_destroy_qp (ch, ch->qp);if (ch->recv_cq)ib_free_cq (ch->recv_cq);if (ch->send_cq)
@@ -566,7 +570,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)return 0 ; err_qp: -srp_destroy_qp (qp); +srp_destroy_qp (ch, qp); err_send_cq:ib_free_cq (send_cq);
@@ -609,7 +613,7 @@ static void srp_free_ch_ib(struct srp_target_port *target,ib_destroy_fmr_pool (ch->fmr_pool); } -srp_destroy_qp (ch->qp); +srp_destroy_qp (ch, ch->qp);ib_free_cq (ch->send_cq);ib_free_cq (ch->recv_cq);
@@ -1822,6 +1826,11 @@ static struct srp_iu *__srp_get_tx_iu(struct srp_rdma_ch *ch,return iu; } +/* +* Note: if this function is called from inside ib_drain_sq() then it will +* be called without ch->lock being held. If ib_drain_sq() dequeues a WQE +* with status IB_WC_SUCCESS then that's a bug. +*/ static void srp_send_done (struct ib_cq *cq,struct ib_wc *wc) { struct srp_iu *iu = container_of(wc->wr_cqe, struct srp_iu, cqe);
Signature · 6 lines
2.11.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 37 lines · Bart Van Assche
Hello Doug,
The patches in this series are the initiator patches I came up with while
testing the SRP initiator and target drivers. Please consider these patches
for inclusion in the upstream kernel.The changes compared to the previous version of this series are:
- Left out the "Use theIB_DEVICE_SG_GAPS_REGHCA feature if supported" patch.
- Added a patch to stop using memory regions with gaps since Laurence ran
into an issue with these memory regions that cannot be explained by analzying the SRP initiator code only.
- Addressed Leon's comments about patch "Avoid that duplicate responses
trigger a kernel bug" and "Add support for drainingIB_POLL_DIRECTcompletion queues".The patches in this series are:
Bart Van Assche (8):
IB/SRP: Avoid usingIB_MR_TYPE_SG_GAPS
IB/srp: Avoid that duplicate responses trigger a kernel bug IB/srp: Fix race conditions related to task management IB/srp: Document locking conventions
IB/srp: Make a diagnostic message more informative IB/srp: Improve an error path
IB/core: Add support for drainingIB_POLL_DIRECTcompletion queues IB/srp: Drain the send queue before destroying a QPdrivers/infiniband/core/cq.c | 4 +- drivers/infiniband/core/verbs.c | 35 ++++++--------
drivers/infiniband/ulp/srp/ib_srp.c| 92 ++++++++++++++++++++++++---
----------
drivers/infiniband/ulp/srp/ib_srp.h | 1 +
Quoted 5 lines
4 files changed, 77 insertions(+), 55 deletions(-)Thanks,
Bart.
Thanks Bart, series applied.
Signature · 4 lines
Doug Ledford <[email protected]> GPG KeyID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD