RDMA/rxe: Fix blktests srp lead kernel panic with 64k page size
Subsystems (2)
PATCH 1/1 RDMA/rxe: Fix blktests srp lead kernel panic with 64k page size
From: Zhu Yanjun <[email protected]>
The page_size of mr is set in infiniband core originally. In the commit 325a7eb85199 ("RDMA/rxe: Cleanup page variables in rxe_mr.c"), the page_size is also set. Sometime this will cause conflict.
--- drivers/infiniband/sw/rxe/rxe_mr.c | 11 ++++------- drivers/infiniband/sw/rxe/rxe_verbs.h | 5 ----- 2 files changed, 4 insertions(+), 12 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +4 -7
@@ -58,7 +58,6 @@ static void rxe_mr_init(int access, struct rxe_mr *mr) mr->rkey = mr->ibmr.rkey = key; mr->access = access; - mr->ibmr.page_size =PAGE_SIZE ; mr->page_mask =PAGE_MASK ; mr->page_shift =PAGE_SHIFT ; mr->state =RXE_MR_STATE_INVALID ;
@@ -79,7 +78,7 @@ static unsigned long rxe_mr_iova_to_index(struct rxe_mr *mr, u64 iova)static unsigned long rxe_mr_iova_to_page_offset (struct rxe_mr *mr,u64 iova) { -return iova & (mr_page_size (mr) -1 ); +return iova & (PAGE_SIZE -1 ); }static bool is_pmem_page (struct page *pg)
@@ -232,7 +231,7 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, int sg_nents, unsigned int *sg_offset) { struct rxe_mr *mr = to_rmr(ibmr); -unsigned int page_size =mr_page_size (mr); +unsigned int page_size = ibmr->page_size; mr->nbuf =0 ; mr->page_shift =ilog2 (page_size);
@@ -256,8 +255,7 @@ static int rxe_mr_copy_xarray(struct rxe_mr *mr, u64 iova, void *addr,if (!page)return -EFAULT ; - bytes = min_t(unsigned int, length, -mr_page_size (mr) - page_offset); + bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset); va =kmap_local_page (page);if (dir ==RXE_FROM_MR_OBJ )memcpy (addr, va + page_offset, bytes);
@@ -450,8 +448,7 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) page_offset =rxe_mr_iova_to_page_offset (mr, iova);if (!page)return -EFAULT ; - bytes = min_t(unsigned int, length, -mr_page_size (mr) - page_offset); + bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset); va =kmap_local_page (page);arch_wb_cache_pmem (va + page_offset, bytes);
drivers/infiniband/sw/rxe/rxe_verbs.h +0 -5
@@ -319,11 +319,6 @@ struct rxe_mr {struct xarray page_list; }; -static inline unsigned int mr_page_size (struct rxe_mr *mr) -{ -return mr ? mr->ibmr.page_size :PAGE_SIZE ; -} -enum rxe_mw_state {RXE_MW_STATE_INVALID =RXE_MR_STATE_INVALID ,RXE_MW_STATE_FREE =RXE_MR_STATE_FREE ,
Signature · 1 line
2.40.1
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.
I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the page_size.
=====
static int srp_add_one(struct ib_device *device)
{
struct srp_device *srp_dev;
struct ib_device_attr *attr = &device->attrs;
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift = max(12, ffs(attr->page_size_cap) - 1);
srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver, mr_page_size is calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specified CONFIG_ARM64_64K_PAGES, the system naturally met the condition.
=====
static int srp_map_finish_fr(struct srp_map_state *state,
struct srp_request *req, struct srp_rdma_ch *ch, int sg_nents, unsigned int *sg_offset_p)
{
struct srp_target_port *target = ch->target;
struct srp_device *dev = target->srp_host->srp_dev;
<...>
n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, sg_offset_p,
dev->mr_page_size);=====
After that, mr_page_size is presumably passed to ib_core layer.
=====
int ib_map_mr_sg(struct ib_mr *mr, struct scatterlist *sg, int sg_nents,
unsigned int *sg_offset, unsigned int page_size)
{
if (unlikely(!mr->device->ops.map_mr_sg))
return -EOPNOTSUPP;
mr->page_size = page_size;
return mr->device->ops.map_mr_sg(mr, sg, sg_nents, sg_offset);
}EXPORT_SYMBOL(ib_map_mr_sg);
=====
Consequently, the page size calculated in srp driver is set to ibmr.page_size.
Coming back to rxe, the device attribute is set here:
=====rxe.c
<...>
/* initialize rxe device parameters */
static void rxe_init_device_param(struct rxe_dev *rxe)
{
rxe->max_inline_data = RXE_MAX_INLINE_DATA;
rxe->attr.vendor_id = RXE_VENDOR_ID;
rxe->attr.max_mr_size = RXE_MAX_MR_SIZE;
rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP;
rxe->attr.max_qp = RXE_MAX_QP;---rxe_param.h
<...>
/* default/initial rxe device parameter settings */ enum rxe_device_param {
RXE_MAX_MR_SIZE = -1ull,
RXE_PAGE_SIZE_CAP = 0xfffff000,
RXE_MAX_QP_WR = DEFAULT_MAX_VALUE,=====rxe_init_device_param() sets the attributes to rxe_dev->attr, and it is later copied to ib_device->attrs in setup_device()@core/device.c.
See that the page size cap is hardcoded to 4096 bytes. I suspect this led to incorrect page_size being set to ibmr.page_size, resulting in the kernel crash.
I think rxe driver is made to be able to handle arbitrary page sizes. Probably, we can just modify RXE_PAGE_SIZE_CAP to fix the issue. What do you guys think?
Thanks,
Daisuke Matsuda
Quoted 10 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
Please check the source code. ibmr.page_size is assigned in infiniband/core. And then it is assigned in rxe. When the 2 are different, the problem will occur.
Please add logs to infiniband/core and rxe to check them.
Zhu Yanjun
Quoted 85 lines
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs; <...> /* * Use the smallest page size supported by the HCA, down to a * minimum of 4096 bytes. We're unlikely to build large sglists * out of smaller entries. */ mr_page_shift = max(12, ffs(attr->page_size_cap) - 1); srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver,mr_page_sizeis calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specifiedCONFIG_ARM64_64K_PAGES, the system naturally met the condition.=====
static int srp_map_finish_fr(structsrp_map_state*state,
structsrp_request*req, structsrp_rdma_ch*ch, intsg_nents, unsigned int*sg_offset_p){ struct srp_target_port *target = ch->target; struct srp_device *dev = target->srp_host->srp_dev; <...> n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, sg_offset_p, dev->mr_page_size);=====
After that,mr_page_sizeis presumably passed toib_corelayer.=====
int ib_map_mr_sg(structib_mr*mr, struct scatterlist *sg, intsg_nents,
unsigned int*sg_offset, unsigned intpage_size){ if (unlikely(!mr->device->ops.map_mr_sg)) return -EOPNOTSUPP; mr->page_size = page_size; return mr->device->ops.map_mr_sg(mr, sg, sg_nents, sg_offset); }
EXPORT_SYMBOL(ib_map_mr_sg);
=====
Consequently, the page size calculated in srp driver is set to ibmr.page_size.Coming back to rxe, the device attribute is set here:
=====rxe.c
<...>
/* initialize rxe device parameters */
static voidrxe_init_device_param(struct rxe_dev *rxe){ rxe->max_inline_data = RXE_MAX_INLINE_DATA; rxe->attr.vendor_id = RXE_VENDOR_ID; rxe->attr.max_mr_size = RXE_MAX_MR_SIZE; rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP; rxe->attr.max_qp = RXE_MAX_QP;---
rxe_param.h
<...>
/* default/initial rxe device parameter settings */ enumrxe_device_param{RXE_MAX_MR_SIZE = -1ull, RXE_PAGE_SIZE_CAP = 0xfffff000, RXE_MAX_QP_WR = DEFAULT_MAX_VALUE,=====
rxe_init_device_param()sets the attributes torxe_dev->attr, and it is later copied toib_device->attrsin setup_device()@core/device.c.
See that the page size cap is hardcoded to 4096 bytes. I suspect this led to incorrectpage_sizebeing set to ibmr.page_size, resulting in the kernel crash.I think rxe driver is made to be able to handle arbitrary page sizes. Probably, we can just modify
RXE_PAGE_SIZE_CAPto fix the issue. What do you guys think?Thanks,
Daisuke Matsuda
On Friday, October 13, 2023 9:29 PM Zhu Yanjun:
Quoted 8 lines · Daisuke Matsuda (Fujitsu)
Quoted 7 lines · Zhu Yanjun
Quoted 5 lines
From: Zhu Yanjun <[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.
Quoted 7 lines
Quoted 3 lines
I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
Please check the source code. ibmr.page_size is assigned in infiniband/core. And then it is assigned in rxe. When the 2 are different, the problem will occur.
In the first place, the two must always be equal.
Is there any situation there are two different page sizes for a MR? I think I have explained the value assigned in the core layer is wrong when the PAGE_SIZE is bigger than 4k, and that is why they are inconsistent.
As I have no environment to test the kernel panic, it remains speculative, but I have provided enough information for everyone to rebut my argument. It is possible I am wrong. I hope someone will tell me specifically where my guess is wrong, or Yi would kindly take the trouble to verify it.
Thanks,
Daisuke Matsuda
Quoted 89 lines
Please add logs to infiniband/core and rxe to check them.
Zhu Yanjun
Quoted 85 lines
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs; <...> /* * Use the smallest page size supported by the HCA, down to a * minimum of 4096 bytes. We're unlikely to build large sglists * out of smaller entries. */ mr_page_shift = max(12, ffs(attr->page_size_cap) - 1); srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver,mr_page_sizeis calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specifiedCONFIG_ARM64_64K_PAGES, the system naturally met the condition.=====
static int srp_map_finish_fr(structsrp_map_state*state,
structsrp_request*req, structsrp_rdma_ch*ch, intsg_nents, unsigned int*sg_offset_p){ struct srp_target_port *target = ch->target; struct srp_device *dev = target->srp_host->srp_dev; <...> n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, sg_offset_p, dev->mr_page_size);=====
After that,mr_page_sizeis presumably passed toib_corelayer.=====
int ib_map_mr_sg(structib_mr*mr, struct scatterlist *sg, intsg_nents,
unsigned int*sg_offset, unsigned intpage_size){ if (unlikely(!mr->device->ops.map_mr_sg)) return -EOPNOTSUPP; mr->page_size = page_size; return mr->device->ops.map_mr_sg(mr, sg, sg_nents, sg_offset); }
EXPORT_SYMBOL(ib_map_mr_sg);
=====
Consequently, the page size calculated in srp driver is set to ibmr.page_size.Coming back to rxe, the device attribute is set here:
=====rxe.c
<...>
/* initialize rxe device parameters */
static voidrxe_init_device_param(struct rxe_dev *rxe){ rxe->max_inline_data = RXE_MAX_INLINE_DATA; rxe->attr.vendor_id = RXE_VENDOR_ID; rxe->attr.max_mr_size = RXE_MAX_MR_SIZE; rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP; rxe->attr.max_qp = RXE_MAX_QP;---
rxe_param.h
<...>
/* default/initial rxe device parameter settings */ enumrxe_device_param{RXE_MAX_MR_SIZE = -1ull, RXE_PAGE_SIZE_CAP = 0xfffff000, RXE_MAX_QP_WR = DEFAULT_MAX_VALUE,=====
rxe_init_device_param()sets the attributes torxe_dev->attr, and it is later copied toib_device->attrsin setup_device()@core/device.c.
See that the page size cap is hardcoded to 4096 bytes. I suspect this led to incorrectpage_sizebeing set to ibmr.page_size, resulting in the kernel crash.I think rxe driver is made to be able to handle arbitrary page sizes. Probably, we can just modify
RXE_PAGE_SIZE_CAPto fix the issue. What do you guys think?Thanks,
Daisuke Matsuda
Quoted 23 lines · Daisuke Matsuda (Fujitsu)
On Friday, October 13, 2023 9:29 PM Zhu Yanjun:
Quoted 8 lines · Daisuke Matsuda (Fujitsu)
Quoted 7 lines · Zhu Yanjun
Quoted 5 lines
From: Zhu Yanjun <[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.Quoted 7 lines
Quoted 3 lines
I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
Please check the source code. ibmr.page_size is assigned in infiniband/core. And then it is assigned in rxe. When the 2 are different, the problem will occur.
In the first place, the two must always be equal.
Is there any situation there are two different page sizes for a MR? I think I have explained the value assigned in the core layer is wrong when thePAGE_SIZEis bigger than 4k, and that is why they are inconsistent.
Quoted 4 lines
As I have no environment to test the kernel panic, it remains speculative, but I have provided enough information for everyone to rebut my argument. It is possible I am wrong. I hope someone will tell me specifically where my guess is wrong, or Yi would kindly take the trouble to verify it.
I made a quick test.
With Zhu's patch, the problem fixed.
With your patch, this problem exists. And other problems occur. I do not know why.
Will continue to make tests.
Quoted 94 lines
Thanks,
Daisuke MatsudaQuoted 89 lines
Please add logs to infiniband/core and rxe to check them.
Zhu Yanjun
Quoted 85 lines
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs; <...> /* * Use the smallest page size supported by the HCA, down to a * minimum of 4096 bytes. We're unlikely to build large sglists * out of smaller entries. */ mr_page_shift = max(12, ffs(attr->page_size_cap) - 1); srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver,mr_page_sizeis calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specifiedCONFIG_ARM64_64K_PAGES, the system naturally met the condition.=====
static int srp_map_finish_fr(structsrp_map_state*state,
structsrp_request*req, structsrp_rdma_ch*ch, intsg_nents, unsigned int*sg_offset_p){ struct srp_target_port *target = ch->target; struct srp_device *dev = target->srp_host->srp_dev; <...> n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, sg_offset_p, dev->mr_page_size);=====
After that,mr_page_sizeis presumably passed toib_corelayer.=====
int ib_map_mr_sg(structib_mr*mr, struct scatterlist *sg, intsg_nents,
unsigned int*sg_offset, unsigned intpage_size){ if (unlikely(!mr->device->ops.map_mr_sg)) return -EOPNOTSUPP; mr->page_size = page_size; return mr->device->ops.map_mr_sg(mr, sg, sg_nents, sg_offset); }
EXPORT_SYMBOL(ib_map_mr_sg);
=====
Consequently, the page size calculated in srp driver is set to ibmr.page_size.Coming back to rxe, the device attribute is set here:
=====rxe.c
<...>
/* initialize rxe device parameters */
static voidrxe_init_device_param(struct rxe_dev *rxe){ rxe->max_inline_data = RXE_MAX_INLINE_DATA; rxe->attr.vendor_id = RXE_VENDOR_ID; rxe->attr.max_mr_size = RXE_MAX_MR_SIZE; rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP; rxe->attr.max_qp = RXE_MAX_QP;---
rxe_param.h
<...>
/* default/initial rxe device parameter settings */ enumrxe_device_param{RXE_MAX_MR_SIZE = -1ull, RXE_PAGE_SIZE_CAP = 0xfffff000, RXE_MAX_QP_WR = DEFAULT_MAX_VALUE,=====
rxe_init_device_param()sets the attributes torxe_dev->attr, and it is later copied toib_device->attrsin setup_device()@core/device.c.
See that the page size cap is hardcoded to 4096 bytes. I suspect this led to incorrectpage_sizebeing set to ibmr.page_size, resulting in the kernel crash.I think rxe driver is made to be able to handle arbitrary page sizes. Probably, we can just modify
RXE_PAGE_SIZE_CAPto fix the issue. What do you guys think?Thanks,
Daisuke Matsuda
Quoted 27 lines · Rain River
Quoted 22 lines
On Friday, October 13, 2023 9:29 PM Zhu Yanjun:
Quoted 8 lines · Daisuke Matsuda (Fujitsu)
Quoted 7 lines · Zhu Yanjun
Quoted 5 lines
From: Zhu Yanjun <[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.Quoted 7 lines
Quoted 3 lines
I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
Please check the source code. ibmr.page_size is assigned in infiniband/core. And then it is assigned in rxe. When the 2 are different, the problem will occur.
In the first place, the two must always be equal.
Is there any situation there are two different page sizes for a MR? I think I have explained the value assigned in the core layer is wrong when thePAGE_SIZEis bigger than 4k, and that is why they are inconsistent.Quoted 4 lines
As I have no environment to test the kernel panic, it remains speculative, but I have provided enough information for everyone to rebut my argument. It is possible I am wrong. I hope someone will tell me specifically where my guess is wrong, or Yi would kindly take the trouble to verify it.
Quoted 6 lines
I made a quick test.
With Zhu's patch, the problem fixed.
With your patch, this problem exists. And other problems occur. I do not know why.
Will continue to make tests.
Thank you for your time and work.
I will try to find out why there were two different page sizes from different perspectives. This may take a while because I am busy with other projects now. If anybody need the driver without the crash issue right now, I do not object to partially undoing "RDMA/rxe: Cleanup page variables in rxe_mr.c" as Zhu suggests, but that should be interim and not a final solution.
Thanks,
Daisuke
Quoted 95 lines
Quoted 95 lines
Thanks,
Daisuke MatsudaQuoted 89 lines
Please add logs to infiniband/core and rxe to check them.
Zhu Yanjun
Quoted 85 lines
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs; <...> /* * Use the smallest page size supported by the HCA, down to a * minimum of 4096 bytes. We're unlikely to build large sglists * out of smaller entries. */ mr_page_shift = max(12, ffs(attr->page_size_cap) - 1); srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver,mr_page_sizeis calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specifiedCONFIG_ARM64_64K_PAGES, the system naturally met the condition.=====
static int srp_map_finish_fr(structsrp_map_state*state,
structsrp_request*req, structsrp_rdma_ch*ch, intsg_nents, unsigned int*sg_offset_p){ struct srp_target_port *target = ch->target; struct srp_device *dev = target->srp_host->srp_dev; <...> n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, sg_offset_p, dev->mr_page_size);=====
After that,mr_page_sizeis presumably passed toib_corelayer.=====
int ib_map_mr_sg(structib_mr*mr, struct scatterlist *sg, intsg_nents,
unsigned int*sg_offset, unsigned intpage_size){ if (unlikely(!mr->device->ops.map_mr_sg)) return -EOPNOTSUPP; mr->page_size = page_size; return mr->device->ops.map_mr_sg(mr, sg, sg_nents, sg_offset); }
EXPORT_SYMBOL(ib_map_mr_sg);
=====
Consequently, the page size calculated in srp driver is set to ibmr.page_size.Coming back to rxe, the device attribute is set here:
=====rxe.c
<...>
/* initialize rxe device parameters */
static voidrxe_init_device_param(struct rxe_dev *rxe){ rxe->max_inline_data = RXE_MAX_INLINE_DATA; rxe->attr.vendor_id = RXE_VENDOR_ID; rxe->attr.max_mr_size = RXE_MAX_MR_SIZE; rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP; rxe->attr.max_qp = RXE_MAX_QP;---
rxe_param.h
<...>
/* default/initial rxe device parameter settings */ enumrxe_device_param{RXE_MAX_MR_SIZE = -1ull, RXE_PAGE_SIZE_CAP = 0xfffff000, RXE_MAX_QP_WR = DEFAULT_MAX_VALUE,=====
rxe_init_device_param()sets the attributes torxe_dev->attr, and it is later copied toib_device->attrsin setup_device()@core/device.c.
See that the page size cap is hardcoded to 4096 bytes. I suspect this led to incorrectpage_sizebeing set to ibmr.page_size, resulting in the kernel crash.I think rxe driver is made to be able to handle arbitrary page sizes. Probably, we can just modify
RXE_PAGE_SIZE_CAPto fix the issue. What do you guys think?Thanks,
Daisuke Matsuda
Quoted 35 lines · Daisuke Matsuda (Fujitsu)
Quoted 27 lines · Rain River
Quoted 22 lines
On Friday, October 13, 2023 9:29 PM Zhu Yanjun:
Quoted 8 lines · Daisuke Matsuda (Fujitsu)
Quoted 7 lines · Zhu Yanjun
Quoted 5 lines
From: Zhu Yanjun <[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.Quoted 7 lines
Quoted 3 lines
I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
Please check the source code. ibmr.page_size is assigned in infiniband/core. And then it is assigned in rxe. When the 2 are different, the problem will occur.
In the first place, the two must always be equal.
Is there any situation there are two different page sizes for a MR? I think I have explained the value assigned in the core layer is wrong when thePAGE_SIZEis bigger than 4k, and that is why they are inconsistent.Quoted 4 lines
As I have no environment to test the kernel panic, it remains speculative, but I have provided enough information for everyone to rebut my argument. It is possible I am wrong. I hope someone will tell me specifically where my guess is wrong, or Yi would kindly take the trouble to verify it.
Quoted 6 lines
I made a quick test.
With Zhu's patch, the problem fixed.
With your patch, this problem exists. And other problems occur. I do not know why.
Will continue to make tests.
Quoted 7 lines
Thank you for your time and work.
I will try to find out why there were two different page sizes from different perspectives. This may take a while because I am busy with other projects now. If anybody need the driver without the crash issue right now, I do not object to partially undoing "RDMA/rxe: Cleanup page variables inrxe_mr.c" as Zhu suggests, but that should be interim and not a final solution.
Sorry, it is late to reply.
This is a difficult and complicated problem because it involves rdma and block/blktests. This commit is based on the fact that the commit 325a7eb85199 ("RDMA/rxe: Cleanup page variables in rxe_mr.c") causes this problem.
So I delved into this commit. In this commit, the core part is to change the size of ibmr.page_size.
From this, I checked the source code. I found that this page_size is also set in infiniband/core. So I try to remove the change of ibmr.page_size.
I will continue to delve into this problem and the source code to find the root cause that you also agree with.
Zhu Yanjun
Quoted 99 lines
Thanks,
DaisukeQuoted 95 lines
Quoted 95 lines
Thanks,
Daisuke MatsudaQuoted 89 lines
Please add logs to infiniband/core and rxe to check them.
Zhu Yanjun
Quoted 85 lines
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs; <...> /* * Use the smallest page size supported by the HCA, down to a * minimum of 4096 bytes. We're unlikely to build large sglists * out of smaller entries. */ mr_page_shift = max(12, ffs(attr->page_size_cap) - 1); srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver,mr_page_sizeis calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specifiedCONFIG_ARM64_64K_PAGES, the system naturally met the condition.=====
static int srp_map_finish_fr(structsrp_map_state*state,
structsrp_request*req, structsrp_rdma_ch*ch, intsg_nents, unsigned int*sg_offset_p){ struct srp_target_port *target = ch->target; struct srp_device *dev = target->srp_host->srp_dev; <...> n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, sg_offset_p, dev->mr_page_size);=====
After that,mr_page_sizeis presumably passed toib_corelayer.=====
int ib_map_mr_sg(structib_mr*mr, struct scatterlist *sg, intsg_nents,
unsigned int*sg_offset, unsigned intpage_size){ if (unlikely(!mr->device->ops.map_mr_sg)) return -EOPNOTSUPP; mr->page_size = page_size; return mr->device->ops.map_mr_sg(mr, sg, sg_nents, sg_offset); }
EXPORT_SYMBOL(ib_map_mr_sg);
=====
Consequently, the page size calculated in srp driver is set to ibmr.page_size.Coming back to rxe, the device attribute is set here:
=====rxe.c
<...>
/* initialize rxe device parameters */
static voidrxe_init_device_param(struct rxe_dev *rxe){ rxe->max_inline_data = RXE_MAX_INLINE_DATA; rxe->attr.vendor_id = RXE_VENDOR_ID; rxe->attr.max_mr_size = RXE_MAX_MR_SIZE; rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP; rxe->attr.max_qp = RXE_MAX_QP;---
rxe_param.h
<...>
/* default/initial rxe device parameter settings */ enumrxe_device_param{RXE_MAX_MR_SIZE = -1ull, RXE_PAGE_SIZE_CAP = 0xfffff000, RXE_MAX_QP_WR = DEFAULT_MAX_VALUE,=====
rxe_init_device_param()sets the attributes torxe_dev->attr, and it is later copied toib_device->attrsin setup_device()@core/device.c.
See that the page size cap is hardcoded to 4096 bytes. I suspect this led to incorrectpage_sizebeing set to ibmr.page_size, resulting in the kernel crash.I think rxe driver is made to be able to handle arbitrary page sizes. Probably, we can just modify
RXE_PAGE_SIZE_CAPto fix the issue. What do you guys think?Thanks,
Daisuke Matsuda
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;
Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);
You light me up.
RXE provides attr.page_size_cap(RXE_PAGE_SIZE_CAP) which means it can support 4K-2G page size
so i think more accurate logic should be:
if (device supports PAGE_SIZE)
use PAGE_SIZE
else if (device support 4096 page_size) // fallback to 4096
use 4096 etc...
else
...Quoted 6 lines
srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver,mr_page_sizeis calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specifiedCONFIG_ARM64_64K_PAGES, the system naturally met the condition.
Add and Hi Bart,
Yi reported a crash[1] when PAGE_SIZE is 64K [1]https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@mail.gmail.com/T/
The root cause is unknown so far, but I notice that SRP over RXE always uses MR with page_size 4K = MAX(4K, min(device_support_page_size)) even though the device supports 64K page size.
* RXE device support 4K ~ 2G page size
4024 /*
4025 * Use the smallest page size supported by the HCA, down to a 4026 * minimum of 4096 bytes. We're unlikely to build large sglists 4027 * out of smaller entries. 4028 */ 4029 mr_page_shift = max(12, ffs(attr->page_size_cap) - 1); 4030 srp_dev->mr_page_size = 1 << mr_page_shift; 4031 srp_dev->mr_page_mask = ~((u64) srp_dev->mr_page_size - 1); 4032 max_pages_per_mr = attr->max_mr_size;
I doubt if we can use PAGE_SIZE MR if the device supports it.
BTW, rtrs seems have the same code. @rtrs
Thanks
Zhijian
Quoted 50 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeso i think more accurate logic should be:
if (device supports
PAGE_SIZE)
usePAGE_SIZE
else if (device support 4096page_size) // fallback to 4096use 4096 etc... else ...Quoted 6 lines
srp_dev->mr_page_size = 1 << mr_page_shift;=====
On initialization of srp driver,mr_page_sizeis calculated here. Note that the device attribute is used to calculate the value of page shift when the device is trying to use a page size larger than 4096. Since Yi specifiedCONFIG_ARM64_64K_PAGES, the system naturally met the condition.
Quoted 21 lines · Zhijian Li (Fujitsu)
Add and Hi Bart,
Yi reported a crash[1] when
PAGE_SIZEis 64K [1]https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@mail.gmail.com/T/The root cause is unknown so far, but I notice that SRP over RXE always uses MR with
page_size4K =MAX(4K, min(device_support_page_size))even though the device supports 64K page size.
* RXE device support 4K ~ 2G page size4024 /*
4025 * Use the smallest page size supported by the HCA, down to a 4026 * minimum of 4096 bytes. We're unlikely to build large sglists 4027 * out of smaller entries. 4028 */ 4029 mr_page_shift = max(12, ffs(attr->page_size_cap) - 1); 4030 srp_dev->mr_page_size = 1 << mr_page_shift; 4031 srp_dev->mr_page_mask = ~((u64) srp_dev->mr_page_size - 1); 4032 max_pages_per_mr = attr->max_mr_size;
Quoted 1 line
I doubt if we can use
PAGE_SIZEMR if the device supports it.
Hi Zhijian,
Thank you for having Cc-ed me. How about changing "12" in the above code into PAGE_SHIFT?
Thanks,
Bart.
Quoted 22 lines · Bart Van Assche
Quoted 21 lines · Zhijian Li (Fujitsu)
Add and Hi Bart,
Yi reported a crash[1] when
PAGE_SIZEis 64K [1]https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@mail.gmail.com/T/The root cause is unknown so far, but I notice that SRP over RXE always uses MR with
page_size4K =MAX(4K, min(device_support_page_size))even though the device supports 64K page size.
* RXE device support 4K ~ 2G page size4024 /*
4025 * Use the smallest page size supported by the HCA, down to a 4026 * minimum of 4096 bytes. We're unlikely to build large sglists 4027 * out of smaller entries.
4028 */
4029 mr_page_shift =max(12, ffs(attr->page_size_cap) - 1); 4030 srp_dev->mr_page_size = 1 <<mr_page_shift;
4031 srp_dev->mr_page_mask = ~((u64)srp_dev->mr_page_size- 1); 4032 max_pages_per_mr =attr->max_mr_size;
Quoted 6 lines
Quoted 1 line
I doubt if we can use
PAGE_SIZEMR if the device supports it.Hi Zhijian,
Thank you for having Cc-ed me. How about changing "12" in the above code into
PAGE_SHIFT?
Yeah, after changed to PAGE_SHIFT, it works fine and the crash is gone in my aarch64 environment.
Thanks
Zhijian
Quoted 4 lines
Thanks,
Bart.
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);
Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page size
That doesn't seem right even in concept.
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less than PAGE_SIZE but more than PAGE_SIZE will explode because kmap_local_page() does only 4K.
If RXE_PAGE_SIZE_CAP == PAGE_SIZE will everything work?
Jason
Quoted 33 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page size
Quoted 8 lines
That doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.If
RXE_PAGE_SIZE_CAP==PAGE_SIZEwill everything work?
Yeah, this should work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)
'0x10000'
Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 1
16
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
somr_page_shift = max(12, ffs(attr->page_size_cap) - 1) = max(12, 16) = 16;
So I think Daisuke's patch should work as well.
Quoted 1 line
Jason
Quoted 56 lines · Zhijian Li (Fujitsu)
Quoted 33 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeQuoted 8 lines
That doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.If
RXE_PAGE_SIZE_CAP==PAGE_SIZEwill everything work?Yeah, this should
work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)'
0x10000'Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 116
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
Quoted 7 lines
so
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1)=max(12, 16)= 16;So I think Daisuke's patch should work as well.
About this patch, please make full tests. Please ensure that this will not introduce some potential problems. In the discussion, this patch possibly has side effect.
Zhu Yanjun
Quoted 3 lines
Quoted 3 lines
Jason
Quoted 43 lines · Zhijian Li (Fujitsu)
Quoted 39 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeThat doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.
Quoted 4 lines
Quoted 2 lines
If
RXE_PAGE_SIZE_CAP==PAGE_SIZEwill everything work?Yeah, this should
work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).
Hi Zhijian
Did you try blktests nvme/rdma use_rxe on your environment, it still failed on my side.
# use_rxe=1 nvme_trtype=rdma ./check nvme/003
nvme/003 (test if we're sending keep-alives to a discovery controller) [failed]
runtime 12.179s ... 11.941s
--- tests/nvme/003.out 2023-10-22 10:54:43.041749537 -0400
+++ /root/blktests/results/nodev/nvme/003.out.bad 2023-10-2305:52:27.882759168 -0400
@@ -1,3 +1,3 @@
Running nvme/003
-NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s) +NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 0 controller(s)
Test complete
[ 7033.431910] rdma_rxe: loaded [ 7033.456341] run blktests nvme/003 at 2023-10-23 05:52:15 [ 7033.502306] (null): rxe_set_mtu: Set mtu to 1024 [ 7033.510969] infiniband enP2p1s0v0_rxe: set active [ 7033.510980] infiniband enP2p1s0v0_rxe: added enP2p1s0v0 [ 7033.549301] loop0: detected capacity change from 0 to 2097152 [ 7033.556966] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 7033.566711] nvmet_rdma: enabling port 0 (10.19.240.81:4420) [ 7033.588605] nvmet: connect attempt for invalid controller ID 0x808 [ 7033.594909] nvme nvme0: Connect Invalid Data Parameter, cntlid: 65535 [ 7033.601504] nvme nvme0: failed to connect queue: 0 ret=16770 [ 7046.317861] rdma_rxe: unloaded
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)'
0x10000'Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 116
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
Quoted 10 lines
so
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1)=max(12, 16)= 16;So I think Daisuke's patch should work as well.
Quoted 1 line
Jason
--
Best Regards,
Yi Zhang
Quoted 48 lines · Yi Zhang
Quoted 46 lines · Zhijian Li (Fujitsu)
Quoted 41 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeThat doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.If
RXE_PAGE_SIZE_CAP==PAGE_SIZEwill everything work?
Quoted 7 lines
Quoted 1 line
Yeah, this should
work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).Hi Zhijian
Did you try blktests nvme/rdma
use_rxeon your environment, it still failed on my side.
Thanks for your testing.
I just did that, it also failed in my environment. After a glance debug, I found there are other places still registered 4K MR. I will dig into it later.
Thanks
Zhijian
Quoted 53 lines
# use_rxe=1 nvme_trtype=rdma ./check nvme/003
nvme/003 (test if we're sending keep-alives to a discovery controller) [failed]runtime 12.179s ... 11.941s --- tests/nvme/003.out 2023-10-22 10:54:43.041749537 -0400 +++ /root/blktests/results/nodev/nvme/003.out.bad 2023-10-2305:52:27.882759168 -0400
@@ -1,3 +1,3 @@
Running nvme/003
-NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s) +NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 0 controller(s)
Test complete[ 7033.431910] rdma_rxe: loaded [ 7033.456341] run blktests nvme/003 at 2023-10-23 05:52:15 [ 7033.502306] (null): rxe_set_mtu: Set mtu to 1024 [ 7033.510969] infiniband enP2p1s0v0_rxe: set active [ 7033.510980] infiniband enP2p1s0v0_rxe: added enP2p1s0v0 [ 7033.549301] loop0: detected capacity change from 0 to 2097152 [ 7033.556966] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 7033.566711] nvmet_rdma: enabling port 0 (10.19.240.81:4420) [ 7033.588605] nvmet: connect attempt for invalid controller ID 0x808 [ 7033.594909] nvme nvme0: Connect Invalid Data Parameter, cntlid: 65535 [ 7033.601504] nvme nvme0: failed to connect queue: 0 ret=16770 [ 7046.317861] rdma_rxe: unloadedQuoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)'
0x10000'Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 116
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
Quoted 10 lines
so
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1)=max(12, 16)= 16;So I think Daisuke's patch should work as well.
Quoted 1 line
Jason
--
Best Regards,
Yi Zhang
Quoted 55 lines · Zhijian Li (Fujitsu)
Quoted 52 lines · Yi Zhang
Quoted 43 lines · Zhijian Li (Fujitsu)
Quoted 39 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeThat doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.Quoted 4 lines
Quoted 2 lines
If
RXE_PAGE_SIZE_CAP==PAGE_SIZEwill everything work?Yeah, this should
work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).Hi Zhijian
Quoted 7 lines
Quoted 3 lines
Did you try blktests nvme/rdma
use_rxeon your environment, it still failed on my side.Thanks for your testing.
I just did that, it also failed in my environment. After a glance debug, I found there are other places still registered 4K MR. I will dig into it later.
nvme intend to register 4K mr, but it should work IMO, at least the RXE have handled it correctly.
1293 static int nvme_rdma_map_sg_fr(struct nvme_rdma_queue *queue,
1294 struct nvme_rdma_request *req, struct nvme_command *c,
1295 int count)
1296 {
1297 struct nvme_keyed_sgl_desc *sg = &c->common.dptr.ksgl;
1298 int nr;
1299
1300 req->mr = ib_mr_pool_get(queue->qp, &queue->qp->rdma_mrs);
1301 if (WARN_ON_ONCE(!req->mr))
1302 return -EAGAIN;
1303
1304 /*
1305 * Align the MR to a 4K page size to match the ctrl page size and 1306 * the block virtual boundary. 1307 */
1308 nr = ib_map_mr_sg(req->mr, req->data_sgl.sg_table.sgl, count, NULL,
1309 SZ_4K);
Anyway, i will go ahead. if you have any thought, please let me know.
Quoted 59 lines
Thanks
ZhijianQuoted 52 lines
# use_rxe=1 nvme_trtype=rdma ./check nvme/003
nvme/003 (test if we're sending keep-alives to a discovery controller) [failed]runtime 12.179s ... 11.941s --- tests/nvme/003.out 2023-10-22 10:54:43.041749537 -0400 +++ /root/blktests/results/nodev/nvme/003.out.bad 2023-10-2305:52:27.882759168 -0400
@@ -1,3 +1,3 @@
Running nvme/003
-NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s) +NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 0 controller(s)
Test complete[ 7033.431910] rdma_rxe: loaded [ 7033.456341] run blktests nvme/003 at 2023-10-23 05:52:15 [ 7033.502306] (null): rxe_set_mtu: Set mtu to 1024 [ 7033.510969] infiniband enP2p1s0v0_rxe: set active [ 7033.510980] infiniband enP2p1s0v0_rxe: added enP2p1s0v0 [ 7033.549301] loop0: detected capacity change from 0 to 2097152 [ 7033.556966] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 7033.566711] nvmet_rdma: enabling port 0 (10.19.240.81:4420) [ 7033.588605] nvmet: connect attempt for invalid controller ID 0x808 [ 7033.594909] nvme nvme0: Connect Invalid Data Parameter, cntlid: 65535 [ 7033.601504] nvme nvme0: failed to connect queue: 0 ret=16770 [ 7046.317861] rdma_rxe: unloadedQuoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)'
0x10000'Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 116
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
Quoted 10 lines
so
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1)=max(12, 16)= 16;So I think Daisuke's patch should work as well.
Quoted 1 line
Jason
--
Best Regards,
Yi Zhang
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size != PAGE_SIZE where it only stores the *page to xarray. So the offset will get lost.
For example,
store process:page_size = 0x1000;PAGE_SIZE = 0x10000;
va0 = 0xffff000020651000;page_offset = 0 = va & (page_size - 1);
page = va_to_page(va); xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page = xa_load(&mr->page_list, index);page_va = kmap_local_page(page) --> it must be a PAGE_SIZE align value, assume it as 0xffff000020650000 va1 = page_va + page_offset = 0xffff000020650000 + 0 = 0xffff000020650000;
Obviously, *va0 != va1*, page_offset get lost.
How to fix:
- revert 325a7eb85199 ("RDMA/rxe: Cleanup page variables in rxe_mr.c")
- don't allow ulp registering mr.page_size != PAGE_SIZE ?
Thanks
Zhijian
Quoted 152 lines · Li Zhijian
Quoted 55 lines · Zhijian Li (Fujitsu)
Quoted 52 lines · Yi Zhang
Quoted 43 lines · Zhijian Li (Fujitsu)
Quoted 39 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 10 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
Quoted 14 lines
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device)
{
structsrp_device*srp_dev;
structib_device_attr*attr =&device->attrs;
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift =max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeThat doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.Quoted 4 lines
Quoted 2 lines
If
RXE_PAGE_SIZE_CAP== PAGE_SIZE will everything work?Yeah, this should
work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).Hi Zhijian
Quoted 7 lines
Quoted 3 lines
Did you try blktests nvme/rdma
use_rxeon your environment, it still failed on my side.Thanks for your testing.
I just did that, it also failed in my environment. After a glance debug, I found there are other places still registered 4K MR. I will dig into it later.nvme intend to register 4K mr, but it should work IMO, at least the RXE have handled it correctly.
1293 static int nvme_rdma_map_sg_fr(struct
nvme_rdma_queue*queue, 1294 structnvme_rdma_request*req, structnvme_command*c, 1295 int count)
1296 {
1297 structnvme_keyed_sgl_desc*sg =&c->common.dptr.ksgl; 1298 int nr;
1299
1300req->mr=ib_mr_pool_get(queue->qp, &queue->qp->rdma_mrs); 1301 if (WARN_ON_ONCE(!req->mr))
1302 return-EAGAIN;
1303
1304 /*
1305 * Align the MR to a 4K page size to match the ctrl page size and 1306 * the block virtual boundary.
1307 */
1308 nr = ib_map_mr_sg(req->mr,req->data_sgl.sg_table.sgl, count, NULL, 1309SZ_4K);Anyway, i will go ahead. if you have any thought, please let me know.
Quoted 59 lines
Thanks
ZhijianQuoted 52 lines
# use_rxe=1 nvme_trtype=rdma ./check nvme/003
nvme/003 (test if we're sending keep-alives to a discovery controller) [failed] runtime 12.179s ... 11.941s
--- tests/nvme/003.out 2023-10-22 10:54:43.041749537 -0400 +++/root/blktests/results/nodev/nvme/003.out.bad2023-10-23 05:52:27.882759168 -0400
@@ -1,3 +1,3 @@
Running nvme/003
-NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s) +NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 0 controller(s) Test complete[ 7033.431910] rdma_rxe: loaded [ 7033.456341] run blktests nvme/003 at 2023-10-23 05:52:15 [ 7033.502306] (null): rxe_set_mtu: Set mtu to 1024 [ 7033.510969] infiniband enP2p1s0v0_rxe: set active [ 7033.510980] infiniband enP2p1s0v0_rxe: added enP2p1s0v0 [ 7033.549301] loop0: detected capacity change from 0 to 2097152 [ 7033.556966] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 7033.566711] nvmet_rdma: enabling port 0 (10.19.240.81:4420) [ 7033.588605] nvmet: connect attempt for invalid controller ID 0x808 [ 7033.594909] nvme nvme0: Connect Invalid Data Parameter, cntlid: 65535 [ 7033.601504] nvme nvme0: failed to connect queue: 0 ret=16770 [ 7046.317861] rdma_rxe: unloadedQuoted 20 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)'
0x10000'Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 116
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
so
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1)=max(12, 16)= 16;So I think Daisuke's patch should work as well.
Quoted 1 line
Jason
Signature · 2 lines
Best Regards, Yi Zhang
Quoted 19 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;
Quoted 6 lines
Obviously, *va0 != va1*,
page_offsetget lost.How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?
Lets do the second one please. Most devices only support PAGE_SIZE anyhow.
Jason
Quoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;
Quoted 8 lines
Quoted 6 lines
Obviously, *va0 != va1*,
page_offsetget lost.How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?Lets do the second one please. Most devices only support
PAGE_SIZEanyhow.
Normally page_size is PAGE_SIZE or the size of the whole compound page (in the latest kernel version, it is the size of folio). When compound page or folio is taken into account, the page_size is not equal to PAGE_SIZE.
If the ULP uses the compound page or folio, the similar problem will occur again. Since this problem is involved with RDMA and block, it will cost a lot of time and efforts to debug. If we fix it now, it will save a lot of time and efforts for the future ULP with folio.
Because folio has a lot of benefits, sooner or later, folio or other similar MMU technology will be popular in the kernel. At that time, this problem will appear again. We can do something for the future folio ^_^
Zhu Yanjun
Quoted 2 lines
Jason
Quoted 28 lines · Zhu Yanjun
Quoted 27 lines · Jason Gunthorpe
Quoted 25 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;Obviously, *va0 != va1*,
page_offsetget lost.How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?
Quoted 6 lines
Quoted 1 line
Lets do the second one please. Most devices only support
PAGE_SIZEanyhow.Normally
page_sizeisPAGE_SIZEor the size of the whole compound page (in the latest kernel version, it is the size of folio). When compound page or folio is taken into account, thepage_sizeis not equal toPAGE_SIZE.
folios are always multiples of PAGE_SIZE. rxe splits everything into PAGE_SIZE units in the xarray.
Quoted 2 lines
If the ULP uses the compound page or folio, the similar problem will occur again.
No, it won't. We never store folios in the xarray.
Jason
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Zhu Yanjun
Quoted 26 lines · Jason Gunthorpe
Quoted 25 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;Obviously, *va0 != va1*,
page_offsetget lost.How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Lets do the second one please. Most devices only support
PAGE_SIZEanyhow.Normally
page_sizeisPAGE_SIZEor the size of the whole compound page (in the latest kernel version, it is the size of folio). When compound page or folio is taken into account, thepage_sizeis not equal toPAGE_SIZE.folios are always multiples of
PAGE_SIZE. rxe splits everything intoPAGE_SIZEunits in the xarray.
Sure. Thanks. Folio is multiple base pages. So the page size should be multiple PAGE_SIZE.
This page size is set in infiniband core and rxe.
Hope no problem will occur when folio or compound page is used in ULP.
Zhu Yanjun
Quoted 6 lines
Quoted 3 lines
If the ULP uses the compound page or folio, the similar problem will occur again.
No, it won't. We never store folios in the xarray.
Jason
Quoted 33 lines · Jason Gunthorpe
Quoted 24 lines · Zhu Yanjun
Quoted 23 lines · Jason Gunthorpe
Quoted 22 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;Obviously, *va0 != va1*,
page_offsetget lost.Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?Lets do the second one please. Most devices only support
PAGE_SIZEanyhow.Normally
page_sizeisPAGE_SIZEor the size of the whole compound page (in the latest kernel version, it is the size of folio). When compound page or folio is taken into account, thepage_sizeis not equal toPAGE_SIZE.
Quoted 6 lines
folios are always multiples of
PAGE_SIZE. rxe splits everything intoPAGE_SIZEunits in the xarray.Quoted 2 lines
If the ULP uses the compound page or folio, the similar problem will occur again.
No, it won't. We never store folios in the xarray.
Sure.
I mean, in ULP, if folio is used, the page size is set to multiple PAGE_SIZE, but in RXE, the page size is set to PAGE_SIZE.
So the page size in ULP is different with the page size in RXE.
I am not sure whether this similar problem still occur or not.
I hope this problem will not occur even with folio.
Zhu Yanjun
Quoted 2 lines
Jason
Quoted 40 lines · Zhu Yanjun
Quoted 38 lines · Jason Gunthorpe
Quoted 24 lines · Zhu Yanjun
Quoted 23 lines · Jason Gunthorpe
Quoted 22 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;Obviously, *va0 != va1*,
page_offsetget lost.Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?Lets do the second one please. Most devices only support
PAGE_SIZEanyhow.Normally
page_sizeisPAGE_SIZEor the size of the whole compound page (in the latest kernel version, it is the size of folio). When compound page or folio is taken into account, thepage_sizeis not equal toPAGE_SIZE.folios are always multiples of
PAGE_SIZE. rxe splits everything intoPAGE_SIZEunits in the xarray.Quoted 2 lines
If the ULP uses the compound page or folio, the similar problem will occur again.
Quoted 8 lines
Quoted 1 line
No, it won't. We never store folios in the xarray.
Sure.
I mean, in ULP, if folio is used, the page size is set to multiple
PAGE_SIZE, but in RXE, the page size is set toPAGE_SIZE.So the page size in ULP is different with the page size in RXE.
There is no such thing as a "page size" in the ULP. rxe is the thing that keeps things in PAGE_SIZE units, and it should be fragmenting whatever the ulp gives into that. The ULP must simply give virtually contiguous runs of memory that are PAGE_SIZE aligned
Jason
Quoted 19 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;
Quoted 6 lines
Obviously, *va0 != va1*,
page_offsetget lost.How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?
Thank you Zhijian for this root-cause analysis.
Hardware RDMA adapters may not support the host page size (PAGE_SIZE) so disallowing mr.page_size != PAGE_SIZE would be wrong.
If the rxe driver only supports mr.page_size == PAGE_SIZE, does this mean that RXE_PAGE_SIZE_CAP should be changed from 0xfffff000 into PAGE_SHIFT?
Thanks,
Bart.
Quoted 23 lines · Bart Van Assche
Quoted 22 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;Obviously, *va0 != va1*,
page_offsetget lost.
Quoted 8 lines
Quoted 3 lines
How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?Thank you Zhijian for this root-cause analysis.
Hardware RDMA adapters may not support the host page size (
PAGE_SIZE) so disallowing mr.page_size !=PAGE_SIZEwould be wrong.
PAGE_SIZE must always be a valid way to construct a MR. We have code in all the DMA drivers to break PAGE_SIZE chunks to something smaller when building MRs.
rxe is missing this now with the xarray stuff since we can't encode the struct page offset and a struct page in a single xarray entry. It would have to go back to encoding pfns and doing pfn to page.
Quoted 3 lines
If the rxe driver only supports mr.page_size ==
PAGE_SIZE, does this mean thatRXE_PAGE_SIZE_CAPshould be changed from0xfffff000intoPAGE_SHIFT?
Yes
Jason
Quoted 6 lines · Jason Gunthorpe
Quoted 3 lines · Bart Van Assche
If the rxe driver only supports mr.page_size ==
PAGE_SIZE, does this mean thatRXE_PAGE_SIZE_CAPshould be changed from0xfffff000intoPAGE_SHIFT?Yes
Bob, do you plan to convert the above change into a patch or do you perhaps expect me to do that?
Thanks,
Bart.
Quoted 5 lines · Bart Van Assche
Quoted 4 lines · Jason Gunthorpe
Quoted 3 lines · Bart Van Assche
If the rxe driver only supports mr.page_size ==
PAGE_SIZE, does this mean thatRXE_PAGE_SIZE_CAPshould be changed from0xfffff000intoPAGE_SHIFT?
Some drivers pad upper bits with 1, and others with 0. Is there any rule? I think both are practically allowed, but we should try to eliminate inconsistency.
Quoted 5 lines
Quoted 2 lines
Yes
Bob, do you plan to convert the above change into a patch or do you perhaps expect me to do that?
It looks Bob has not been involved in this thread.
Thanks,
Daisuke Matsuda
Quoted 4 lines
Thanks,
Bart.
Quoted 10 lines · Bart Van Assche
Quoted 6 lines · Jason Gunthorpe
Quoted 3 lines · Bart Van Assche
If the rxe driver only supports mr.page_size ==
PAGE_SIZE, does this mean thatRXE_PAGE_SIZE_CAPshould be changed from0xfffff000intoPAGE_SHIFT?Yes
Bob, do you plan to convert the above change into a patch or do you perhaps expect me to do that?
Zhijian has done a lot of work on this problem. And he found out the root cause.
Perhaps Zhijian should file a patch for this problem?
Zhu Yanjun
Quoted 5 lines
Thanks,
Bart.
Quoted 10 lines · Bart Van Assche
Quoted 6 lines · Jason Gunthorpe
Quoted 3 lines · Bart Van Assche
If the rxe driver only supports mr.page_size ==
PAGE_SIZE, does this mean thatRXE_PAGE_SIZE_CAPshould be changed from0xfffff000intoPAGE_SHIFT?Yes
Bob, do you plan to convert the above change into a patch or do you perhaps expect me to do that?
I just cooked the patches, it can fix SRP problem. But nvme still fails.
Quoted 5 lines
Thanks,
Bart.
Quoted 4 lines · Zhijian Li (Fujitsu)
The root cause is that
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.
It is interesting root cause.page_size is 0x1000, PAGE_SIZE 0x10000.
Zhu Yanjun
Quoted 181 lines
For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
va0 =0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;Obviously, *va0 != va1*,
page_offsetget lost.How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?Thanks
ZhijianQuoted 152 lines · Li Zhijian
Quoted 55 lines · Zhijian Li (Fujitsu)
Quoted 52 lines · Yi Zhang
Quoted 43 lines · Zhijian Li (Fujitsu)
Quoted 39 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 10 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
Quoted 14 lines
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device)
{
structsrp_device*srp_dev;
structib_device_attr*attr =&device->attrs;
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift =max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeThat doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.Quoted 4 lines
Quoted 2 lines
If
RXE_PAGE_SIZE_CAP== PAGE_SIZE will everything work?Yeah, this should
work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).Hi Zhijian
Quoted 7 lines
Quoted 3 lines
Did you try blktests nvme/rdma
use_rxeon your environment, it still failed on my side.Thanks for your testing.
I just did that, it also failed in my environment. After a glance debug, I found there are other places still registered 4K MR. I will dig into it later.nvme intend to register 4K mr, but it should work IMO, at least the RXE have handled it correctly.
1293 static int nvme_rdma_map_sg_fr(struct
nvme_rdma_queue*queue, 1294 structnvme_rdma_request*req, structnvme_command*c, 1295 int count)
1296 {
1297 structnvme_keyed_sgl_desc*sg =&c->common.dptr.ksgl; 1298 int nr;
1299
1300req->mr=ib_mr_pool_get(queue->qp, &queue->qp->rdma_mrs); 1301 if (WARN_ON_ONCE(!req->mr))
1302 return-EAGAIN;
1303
1304 /*
1305 * Align the MR to a 4K page size to match the ctrl page size and 1306 * the block virtual boundary.
1307 */
1308 nr = ib_map_mr_sg(req->mr,req->data_sgl.sg_table.sgl, count, NULL, 1309SZ_4K);Anyway, i will go ahead. if you have any thought, please let me know.
Quoted 59 lines
Thanks
ZhijianQuoted 52 lines
# use_rxe=1 nvme_trtype=rdma ./check nvme/003
nvme/003 (test if we're sending keep-alives to a discovery controller) [failed]
runtime 12.179s ... 11.941s
--- tests/nvme/003.out 2023-10-22 10:54:43.041749537 -0400 +++/root/blktests/results/nodev/nvme/003.out.bad2023-10-23
05:52:27.882759168 -0400
@@ -1,3 +1,3 @@
Running nvme/003
-NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s) +NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 0 controller(s) Test complete[ 7033.431910] rdma_rxe: loaded [ 7033.456341] run blktests nvme/003 at 2023-10-23 05:52:15 [ 7033.502306] (null): rxe_set_mtu: Set mtu to 1024 [ 7033.510969] infiniband enP2p1s0v0_rxe: set active [ 7033.510980] infiniband enP2p1s0v0_rxe: added enP2p1s0v0 [ 7033.549301] loop0: detected capacity change from 0 to 2097152 [ 7033.556966] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 7033.566711] nvmet_rdma: enabling port 0 (10.19.240.81:4420) [ 7033.588605] nvmet: connect attempt for invalid controller ID 0x808 [ 7033.594909] nvme nvme0: Connect Invalid Data Parameter, cntlid: 65535 [ 7033.601504] nvme nvme0: failed to connect queue: 0 ret=16770 [ 7046.317861] rdma_rxe: unloadedQuoted 20 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)'
0x10000'Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 116
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
so
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1)=max(12, 16)= 16;So I think Daisuke's patch should work as well.
Quoted 1 line
Jason
Signature · 2 lines
Best Regards, Yi Zhang
A message that is not in the archive (sent off-list or not yet received): CAEz=LcuLCe7bhUohh6BcHdJ1_ocJdZq=eu07vWb3Md5_ZOGDBg@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): CAEz=LcuQ6fFpHqBPT1oTUgKABAHFJqYDC-AHidE-+n6OtzmCPQ@mail.gmail.com
Quoted 30 lines · Greg Sword
Quoted 26 lines · Greg Sword
Quoted 4 lines · Zhu Yanjun
Quoted 2 lines · Zhijian Li (Fujitsu)
The root cause is that
Quoted 13 lines
Quoted 2 lines
rxe:rxe_set_page() gets wrong when mr.page_size !=
PAGE_SIZEwhere it only stores the *page to xarray. So the offset will get lost.It is interesting root cause.
page_sizeis0x1000,PAGE_SIZE0x10000.Zhu Yanjun
Quoted 5 lines
For example,
store process:page_size=0x1000;PAGE_SIZE=0x10000;
page_sizeis0x1000,PAGE_SIZEis0x10000.
Quoted 10 lines
Quoted 8 lines
Is it correct?
page_sizeshould bePAGE_SIZE. That is,page_sizeshould be greater thanPAGE_SIZE. It is not reasonable now.The root cause is to find out why
page_sizeis set 4K (0x1000). On ARM64 hostwithCONFIG_ARM64_64K_PAGESenabled, thepage_sizeshould not be 4K, it should be 64K.So Zhijian's commit seems not to be correct.
linux-rdma not reached. Resend it again.
Quoted 215 lines
Because rxe splits everything into
PAGE_SIZEunits in the xarray, so it should only supportPAGE_SIZEpage size.WithCONFIG_ARM64_64K_PAGES,PAGE_SIZEis 64K.But from NVME side, the page size is 4K bytes, it is not 64K.
In the function
ib_map_mr_sgis fixedSZ_4K. So the root cause is not in rxe. Before rxe is called, the capability of rxe should be tested.static int nvme_rdma_map_sg_fr(struct
nvme_rdma_queue*queue,
structnvme_rdma_request*req, structnvme_command*c, int count){ struct nvme_keyed_sgl_desc *sg = &c->common.dptr.ksgl; int nr; req->mr = ib_mr_pool_get(queue->qp, &queue->qp->rdma_mrs); if (WARN_ON_ONCE(!req->mr)) return -EAGAIN; /* * Align the MR to a 4K page size to match the ctrl page size and * the block virtual boundary. */ nr = ib_map_mr_sg(req->mr, req->data_sgl.sg_table.sgl, count, NULL, SZ_4K); if (unlikely(nr < count)) { ib_mr_pool_put(queue->qp, &queue->qp->rdma_mrs, req->mr); req->mr = NULL; if (nr < 0) return nr; return -EINVAL; } ib_update_fast_reg_key(req->mr, ib_inc_rkey(req->mr->rkey));Quoted 181 lines
Quoted 181 lines
Quoted 180 lines
va0 =
0xffff000020651000;page_offset= 0 = va & (page_size- 1);
page =va_to_page(va);xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
load_process:
page =xa_load(&mr->page_list, index);page_va=kmap_local_page(page)--> it must be aPAGE_SIZEalign value, assume it as0xffff000020650000va1 =page_va+page_offset=0xffff000020650000+ 0 =0xffff000020650000;Obviously, *va0 != va1*,
page_offsetget lost.How to fix:
- revert325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c")
- don't allow ulp registering mr.page_size !=PAGE_SIZE?Thanks
ZhijianQuoted 152 lines · Li Zhijian
Quoted 55 lines · Zhijian Li (Fujitsu)
Quoted 52 lines · Yi Zhang
Quoted 43 lines · Zhijian Li (Fujitsu)
Quoted 39 lines · Jason Gunthorpe
Quoted 29 lines · Zhijian Li (Fujitsu)
CC Bart
Quoted 17 lines · Daisuke Matsuda (Fujitsu)
Quoted 5 lines · Zhu Yanjun
From: Zhu Yanjun<[email protected]>
The
page_sizeof mr is set in infiniband core originally. In the commit325a7eb85199("RDMA/rxe: Cleanup page variables inrxe_mr.c"), thepage_sizeis also set. Sometime this will cause conflict.I appreciate your prompt action, but I do not think this commit deals with the root cause. I agree that the problem lies in rxe driver, but what is wrong with assigning actual page size to ibmr.page_size?
IMO, the problem comes from the device attribute of rxe driver, which is used in ulp/srp layer to calculate the
page_size.
=====
static intsrp_add_one(struct ib_device *device){ struct srp_device *srp_dev; struct ib_device_attr *attr = &device->attrs;Quoted 7 lines
<...>
/*
* Use the smallest page size supported by the HCA, down to a
* minimum of 4096 bytes. We're unlikely to build large sglists
* out of smaller entries.
*/
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1);Quoted 2 lines
You light me up.
RXE providesattr.page_size_cap(RXE_PAGE_SIZE_CAP)which means it can support 4K-2G page sizeThat doesn't seem right even in concept.>
I think the multi-size support in the new xarray code does not work right, just looking at it makes me think it does not work right. It looks like it can do less thanPAGE_SIZEbut more thanPAGE_SIZEwill explode becausekmap_local_page()does only 4K.Quoted 4 lines
Quoted 2 lines
If
RXE_PAGE_SIZE_CAP==PAGE_SIZEwill everything work?Yeah, this should
work(even though i only verified hardcoding mr_page_shift to PAGE_SHIFT).Hi Zhijian
Quoted 7 lines
Quoted 3 lines
Did you try blktests nvme/rdma
use_rxeon your environment, it still failed on my side.Thanks for your testing.
I just did that, it also failed in my environment. After a glance debug, I found there are other places still registered 4K MR. I will dig into it later.nvme intend to register 4K mr, but it should work IMO, at least the RXE have handled it correctly.
1293 static int nvme_rdma_map_sg_fr(struct
nvme_rdma_queue*queue,
1294 structnvme_rdma_request*req, structnvme_command*c,
1295 int count)
1296 {
1297 structnvme_keyed_sgl_desc*sg =&c->common.dptr.ksgl;
1298 int nr;
1299
1300req->mr=ib_mr_pool_get(queue->qp, &queue->qp->rdma_mrs);
1301 if (WARN_ON_ONCE(!req->mr))
1302 return-EAGAIN;
1303
1304 /*1305 * Align the MR to a 4K page size to match the ctrl page size and 1306 * the block virtual boundary. 1307 */1308 nr = ib_map_mr_sg(req->mr,
req->data_sgl.sg_table.sgl, count, NULL,
1309SZ_4K);Anyway, i will go ahead. if you have any thought, please let me know.
Quoted 59 lines
Thanks
ZhijianQuoted 52 lines
# use_rxe=1 nvme_trtype=rdma ./check nvme/003
nvme/003 (test if we're sending keep-alives to a discovery controller) [failed]runtime 12.179s ... 11.941s --- tests/nvme/003.out 2023-10-22 10:54:43.041749537 -0400 +++ /root/blktests/results/nodev/nvme/003.out.bad 2023-10-2305:52:27.882759168 -0400
@@ -1,3 +1,3 @@
Running nvme/003
-NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 1 controller(s) +NQN:nqn.2014-08.org.nvmexpress.discovery disconnected 0 controller(s)
Test complete[ 7033.431910] rdma_rxe: loaded [ 7033.456341] run blktests nvme/003 at 2023-10-23 05:52:15 [ 7033.502306] (null): rxe_set_mtu: Set mtu to 1024 [ 7033.510969] infiniband enP2p1s0v0_rxe: set active [ 7033.510980] infiniband enP2p1s0v0_rxe: added enP2p1s0v0 [ 7033.549301] loop0: detected capacity change from 0 to 2097152 [ 7033.556966] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 7033.566711] nvmet_rdma: enabling port 0 (10.19.240.81:4420) [ 7033.588605] nvmet: connect attempt for invalid controller ID 0x808 [ 7033.594909] nvme nvme0: Connect Invalid Data Parameter, cntlid: 65535 [ 7033.601504] nvme nvme0: failed to connect queue: 0 ret=16770 [ 7046.317861] rdma_rxe: unloadedQuoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
import ctypes
libc =ctypes.cdll.LoadLibrary('libc.so.6')
hex(65536)'
0x10000'Quoted 1 line
Quoted 1 line
Quoted 1 line
libc.ffs(0x10000)- 116
Quoted 1 line
Quoted 1 line
Quoted 1 line
1 << 16
65536
Quoted 10 lines
so
mr_page_shift=max(12, ffs(attr->page_size_cap) - 1)=max(12, 16)= 16;So I think Daisuke's patch should work as well.
Quoted 1 line
Jason
--
Best Regards,
Yi Zhang