Skip to contentSkip to conversation

dynamic-sg patch has broken rdma_rxe

DISCUSSION Bob Pearson 6 participants 20 messages Oct 13, 2020 – Oct 20, 2020 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

Bob

[  209.562096] WARNING: CPU: 12 PID: 5343 at lib/scatterlist.c:438 __sg_alloc_table_from_pages+0x21/0x440
[  209.562097] Modules linked in: rdma_rxe ip6_udp_tunnel udp_tunnel rfcomm xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc cmac algif_hash algif_skcipher af_alg bnep binfmt_misc iwlmvm btusb btrtl btbcm input_leds mac80211 btintel bluetooth libarc4 iwlwifi ib_uverbs snd_hda_codec_realtek snd_hda_codec_generic ecdh_generic ib_core cfg80211 igb ledtrig_audio ecc snd_hda_codec_hdmi hid_generic snd_hda_intel snd_intel_dspcfg dca snd_hda_codec snd_hda_core usbhid snd_hwdep nls_iso8859_1 amdgpu mlx5_core edac_mce_amd hid snd_pcm kvm_amd kvm iommu_v2 snd_seq_midi snd_seq_midi_event gpu_sched ttm wmi_bmof mxm_wmi snd_rawmidi drm_kms_helper snd_seq tls crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_device crypto_simd snd_timer mlxfw cryptd cec
[  209.562135]  glue_helper ccp snd rapl pci_hyperv_intf i2c_algo_bit ahci i2c_piix4 efi_pstore soundcore fb_sys_fops libahci syscopyarea k10temp sysfillrect sysimgblt wmi drm sunrpc sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 nvme nvme_core mac_hid
[  209.562149] CPU: 12 PID: 5343 Comm: python3 Not tainted 5.9.0-rc8+ #20
[  209.562150] Hardware name: Gigabyte Technology Co., Ltd. X570 AORUS PRO WIFI/X570 AORUS PRO WIFI, BIOS F11 12/06/2019
[  209.562153] RIP: 0010:__sg_alloc_table_from_pages+0x21/0x440
[  209.562155] Code: c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 89 d5 41 54 53 48 83 ec 48 48 89 7d a0 48 8b 55 10 45 85 c9 75 1b <0f> 0b 49 c7 c2 ea ff ff ff 48 83 c4 48 4c 89 d0 5b 41 5c 41 5d 41
[  209.562156] RSP: 0018:ffffb86648b879a8 EFLAGS: 00010206
[  209.562157] RAX: 0000000000001000 RBX: 000000000238c000 RCX: 0000000000000000
[  209.562158] RDX: 0000000000000000 RSI: ffff8ab0c2667000 RDI: ffff8ab0d7b0c1d0
[  209.562159] RBP: ffffb86648b87a18 R08: 0000000000001000 R09: 00000000ffffffff
[  209.562160] R10: 0000000000000022 R11: 0000000000000001 R12: ffff8ab0faace000
[  209.562161] R13: 0000000000000001 R14: 00000000ffffffff R15: ffff8ab0c2667000
[  209.562162] FS:  00007f1846f00740(0000) GS:ffff8ab0feb00000(0000) knlGS:0000000000000000
[  209.562163] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  209.562164] CR2: 00000000022c5098 CR3: 000000074ab48000 CR4: 0000000000350ee0
[  209.562165] Call Trace:
[  209.562176]  ib_umem_get+0x1b1/0x390 [ib_uverbs]
[  209.562182]  rxe_mem_init_user+0x48/0x1e0 [rdma_rxe]
[  209.562187]  rxe_reg_user_mr+0x96/0x160 [rdma_rxe]
[  209.562193]  ib_uverbs_reg_mr+0x141/0x270 [ib_uverbs]
[  209.562199]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0xd2/0x140 [ib_uverbs]
[  209.562202]  ? __check_object_size+0x4d/0x150
[  209.562207]  ib_uverbs_cmd_verbs+0xb37/0xc70 [ib_uverbs]
#1 lore raw
Quoted 9 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

Bob

[  209.562096] WARNING: CPU: 12 PID: 5343 at lib/scatterlist.c:438 __sg_alloc_table_from_pages+0x21/0x440

SNIP

Quoted 1 line

I found it. The rxe driver had set the max_segment_size to UINT_MAX (0xffffffff) which triggered the warning since it has an 'offset into page' in __sg_alloc_table_from_pages. Can you tell me what this parameter is supposed to do and what is a reasonable value. What scares me is that the default used in ib_umem_get is 64K. Does this have anything to do with the largest SGL size or is it something else.

Bob

maintainer
#2 lore raw
In reply to Bob Pearson
Quoted 5 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

The offset_in_page is wrong, but it is protecting some other logic..

Maor? Leon? Can you sort it out tomorrow?

Jason

#3 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 5 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

The offset_in_page is wrong, but it is protecting some other logic..

Maor? Leon? Can you sort it out tomorrow?

Leon and I investigated it. This check existed before my series to protect the alloc_table_from_pages logic. It's still relevant.
This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the dma_device to ib_register_device"), and according to below link it was expected.  The safest approach is to set the max_segment_size back the 2GB in all drivers. What do you think?

https://lore.kernel.org/linux-rdma/[email protected]/

Quoted 2 lines

Jason

#4 lore raw
Quoted 12 lines · Maor Gottlieb
Quoted 9 lines · Jason Gunthorpe
Quoted 5 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

The offset_in_page is wrong, but it is protecting some other logic..

Maor? Leon? Can you sort it out tomorrow?

Quoted 8 lines

Leon and I investigated it. This check existed before my series to protect the alloc_table_from_pages logic. It's still relevant.
This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the dma_device to ib_register_device"), and according to below link it was expected.  The safest approach is to set the max_segment_size back the 2GB in all drivers. What do you think?

https://lore.kernel.org/linux-rdma/[email protected]/

FWIW, EFA is broken as well (same call trace) so it's not just software drivers.

#5 lore raw
Quoted 18 lines · Gal Pressman
Quoted 17 lines · Maor Gottlieb
Quoted 9 lines · Jason Gunthorpe
Quoted 5 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

The offset_in_page is wrong, but it is protecting some other logic..

Maor? Leon? Can you sort it out tomorrow?

Leon and I investigated it. This check existed before my series to protect the alloc_table_from_pages logic. It's still relevant.
This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the dma_device to ib_register_device"), and according to below link it was expected.  The safest approach is to set the max_segment_size back the 2GB in all drivers. What do you think?

Quoted 2 lines
Quoted 1 line

https://lore.kernel.org/linux-rdma/[email protected]/

FWIW, EFA is broken as well (same call trace) so it's not just software drivers.

This is true to all drivers that call to ib_umem_get and set UINT_MAX  as max_segment_size.
Jason,  maybe instead of set UINT_MAX as max_segment_size, need to set SCATTERLIST_MAX_SEGMENT which does the required alignment.

maintainer
#6 lore raw
Quoted 20 lines · Maor Gottlieb
Quoted 18 lines · Gal Pressman
Quoted 17 lines · Maor Gottlieb
Quoted 9 lines · Jason Gunthorpe
Quoted 5 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

The offset_in_page is wrong, but it is protecting some other logic..

Maor? Leon? Can you sort it out tomorrow?

Leon and I investigated it. This check existed before my series to protect the alloc_table_from_pages logic. It's still relevant.
This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the dma_device to ib_register_device"), and according to below link it was expected.  The safest approach is to set the max_segment_size back the 2GB in all drivers. What do you think?

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

https://lore.kernel.org/linux-rdma/[email protected]/

FWIW, EFA is broken as well (same call trace) so it's not just software drivers.

This is true to all drivers that call to ib_umem_get and set UINT_MAX  as max_segment_size.
Jason,  maybe instead of set UINT_MAX as max_segment_size, need to set SCATTERLIST_MAX_SEGMENT which does the required alignment.

SCATTERLIST_MAX_SEGMENT is almost never used, however there are lots of places passing UINT_MAX or similar as the max_segsize for DMA.

The only place that does use it looks goofy to me:

	dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
					     SCATTERLIST_MAX_SEGMENT));

The seg_size should reflect the HW capability, not be mixed in with knowledge about SGL internals. If the SGL can't build up to the HW limit then it is fine to internally silently reduce it.

So I think we need to fix the scatterlist code, like below, and just remove SCATTERLIST_MAX_SEGMENT completely.

It fixes things? Are you OK with this Christoph?

I need to get this fixed for the merge window PR I want to send on Friday.

lib/scatterlist.c +3 -1
index e102fdfaa75be7..d158033834cdbc 100644
@@ -435,7 +435,9 @@ struct scatterlist *__sg_alloc_table_from_pages(struct sg_table *sgt,
 	unsigned int added_nents = 0;
 	struct scatterlist *s = prv;
 
-	if (WARN_ON(!max_segment || offset_in_page(max_segment)))
+	/* Avoid overflow when computing sg_len + PAGE_SIZE */
+	max_segment = max_segment & PAGE_MASK;
+	if (WARN_ON(max_segment < PAGE_SIZE))
 		return ERR_PTR(-EINVAL);
 
 	if (IS_ENABLED(CONFIG_ARCH_NO_SG_CHAIN) && prv)
#7 lore raw
Quoted 41 lines · Jason Gunthorpe
Quoted 20 lines · Maor Gottlieb
Quoted 18 lines · Gal Pressman
Quoted 17 lines · Maor Gottlieb
Quoted 9 lines · Jason Gunthorpe
Quoted 5 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

The offset_in_page is wrong, but it is protecting some other logic..

Maor? Leon? Can you sort it out tomorrow?

Leon and I investigated it. This check existed before my series to protect the alloc_table_from_pages logic. It's still relevant.
This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the dma_device to ib_register_device"), and according to below link it was expected.  The safest approach is to set the max_segment_size back the 2GB in all drivers. What do you think?

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

https://lore.kernel.org/linux-rdma/[email protected]/

FWIW, EFA is broken as well (same call trace) so it's not just software drivers.

This is true to all drivers that call to ib_umem_get and set UINT_MAX  as max_segment_size.
Jason,  maybe instead of set UINT_MAX as max_segment_size, need to set SCATTERLIST_MAX_SEGMENT which does the required alignment.

SCATTERLIST_MAX_SEGMENT is almost never used, however there are lots of places passing UINT_MAX or similar as the max_segsize for DMA.

The only place that does use it looks goofy to me:

	dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
					     SCATTERLIST_MAX_SEGMENT));

The seg_size should reflect the HW capability, not be mixed in with knowledge about SGL internals. If the SGL can't build up to the HW limit then it is fine to internally silently reduce it.

Quoted 7 lines

So I think we need to fix the scatterlist code, like below, and just remove SCATTERLIST_MAX_SEGMENT completely.

It fixes things? Are you OK with this Christoph?

I need to get this fixed for the merge window PR I want to send on Friday.

This looks ok. Not critical, but I think we should follow up and kill off the confusing SCATTERLIST_MAX_SEGMENT as well soon

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#9 lore raw
Quoted 6 lines · Tvrtko

Hi guys,

[I removed the mailing list from cc since from this email address) I can't reply properly inline. ([email protected] works better.)]

I put it back

Quoted 11 lines

However:

(quoted diff) +3 -0
+	/* Avoid overflow when computing sg_len + PAGE_SIZE */
+	max_segment = max_segment & PAGE_MASK;
+	if (WARN_ON(max_segment < PAGE_SIZE))
 		return ERR_PTR(-EINVAL);

Maybe it's too early for me but I don't get this. It appears the condition can only be true if the max_segment is smaller than page size as passed in to the function to _start with_. Don't see what does filtering out low bits achieves on top.

The entire problem is the algorithm in __sg_alloc_table_from_pages() only limits sg_len to

   sg_len == N * PAGE_SIZE <= ALIGN_UP(max_segment, PAGE_SIZE);

ie it overshoots max_segment if it is unaligned.

It also badly malfunctions if the ALIGN_UP() overflows, eg for ALIGN_UP(UINT_MAX).

This is all internal problems inside __sg_alloc_table_from_pages() and has nothing to do with the scatter lists themselves.

Adding an ALIGN_DOWN guarentees this algorithm produces sg_len <= max_segment in all cases.

Quoted 2 lines

If the intent is to allow unaligned max_segment then also please change kerneldoc.

Sure

Quoted 3 lines

Although TBH I don't get how unaligned max segment makes sense. List can end on an unaligned segment but surely shouldn't have then in the middle.

The max_segment should either be UINT_MAX because the caller doesn't care, or come from the DMA max_segment_size which is a HW limitation usually derived from the # of bits available to express a length.

Conflating the HW limitation with the system PAGE_SIZE is nonsense. This is further confused because the only reason we have an alignment restriction is due to this algorithm design, the SGL rules don't prevent the use of unaligned lengths, or length smaller than PAGE_SIZE, even in the interior.

Jason

From b03302028893ce7465ba7e8736abba1922469bc1 Mon Sep 17 00:00:00 2001
From: Jason Gunthorpe <[email protected]>
Date: Fri, 16 Oct 2020 08:46:01 -0300
Subject: [PATCH] lib/scatterlist: Do not limit max_segment to PAGE_ALIGNED
values

The main intention of the max_segment argument to __sg_alloc_table_from_pages() is to match the DMA layer segment size set by dma_set_max_seg_size().

Restricting the input to be page aligned makes it impossible to just connect the DMA layer to this API.

The only reason for a page alignment here is because the algorithm will overshoot the max_segment if it is not a multiple of PAGE_SIZE. Simply fix the alignment before starting and don't expose this implementation detail to the callers.

A future patch will completely remove SCATTERLIST_MAX_SEGMENT.

---
 lib/scatterlist.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
lib/scatterlist.c +8 -4
index e102fdfaa75be7..ed2497c79a216b 100644
@@ -404,7 +404,7 @@ static struct scatterlist *get_next_sg(struct sg_table *table,
  * @n_pages:	 Number of pages in the pages array
  * @offset:      Offset from start of the first page to the start of a buffer
  * @size:        Number of valid bytes in the buffer (after offset)
- * @max_segment: Maximum size of a scatterlist node in bytes (page aligned)
+ * @max_segment: Maximum size of a scatterlist element in bytes
  * @prv:	 Last populated sge in sgt
  * @left_pages:  Left pages caller have to set after this call
  * @gfp_mask:	 GFP allocation mask
@@ -435,7 +435,12 @@ struct scatterlist *__sg_alloc_table_from_pages(struct sg_table *sgt,
 	unsigned int added_nents = 0;
 	struct scatterlist *s = prv;
 
-	if (WARN_ON(!max_segment || offset_in_page(max_segment)))
+	/*
+	 * The algorithm below requires max_segment to be aligned to PAGE_SIZE
+	 * otherwise it can overshoot.
+	 */
+	max_segment = ALIGN_DOWN(max_segment, PAGE_SIZE);
+	if (WARN_ON(max_segment < PAGE_SIZE))
 		return ERR_PTR(-EINVAL);
 
 	if (IS_ENABLED(CONFIG_ARCH_NO_SG_CHAIN) && prv)
@@ -542,8 +547,7 @@ int sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages,
 			      unsigned long size, gfp_t gfp_mask)
 {
 	return PTR_ERR_OR_ZERO(__sg_alloc_table_from_pages(sgt, pages, n_pages,
-			offset, size, SCATTERLIST_MAX_SEGMENT,
-			NULL, 0, gfp_mask));
+			offset, size, UINT_MAX, NULL, 0, gfp_mask));
 }
 EXPORT_SYMBOL(sg_alloc_table_from_pages);
 
Signature · 1 line
2.28.0
#10 lore raw
Quoted 29 lines · Jason Gunthorpe
Quoted 6 lines · Tvrtko

Hi guys,

[I removed the mailing list from cc since from this email address) I can't reply properly inline. ([email protected] works better.)]

I put it back

Quoted 11 lines

However:

lib/scatterlist.c +3 -0
+	/* Avoid overflow when computing sg_len + PAGE_SIZE */
+	max_segment = max_segment & PAGE_MASK;
+	if (WARN_ON(max_segment < PAGE_SIZE))
  		return ERR_PTR(-EINVAL);

Maybe it's too early for me but I don't get this. It appears the condition can only be true if the max_segment is smaller than page size as passed in to the function to _start with_. Don't see what does filtering out low bits achieves on top.

The entire problem is the algorithm in __sg_alloc_table_from_pages() only limits sg_len to

    sg_len == N * PAGE_SIZE <= ALIGN_UP(max_segment, PAGE_SIZE);

ie it overshoots max_segment if it is unaligned.

Quoted 8 lines

It also badly malfunctions if the ALIGN_UP() overflows, eg for ALIGN_UP(UINT_MAX).

This is all internal problems inside __sg_alloc_table_from_pages() and has nothing to do with the scatter lists themselves.

Adding an ALIGN_DOWN guarentees this algorithm produces sg_len <= max_segment in all cases.

Right, I can follow the story now that ALIGN_DOWN is in the picture.

Quoted 31 lines
Quoted 2 lines

If the intent is to allow unaligned max_segment then also please change kerneldoc.

Sure

Quoted 3 lines

Although TBH I don't get how unaligned max segment makes sense. List can end on an unaligned segment but surely shouldn't have then in the middle.

The max_segment should either be UINT_MAX because the caller doesn't care, or come from the DMA max_segment_size which is a HW limitation usually derived from the # of bits available to express a length.

Conflating the HW limitation with the system PAGE_SIZE is nonsense. This is further confused because the only reason we have an alignment restriction is due to this algorithm design, the SGL rules don't prevent the use of unaligned lengths, or length smaller than PAGE_SIZE, even in the interior.

Jason

Quoted 1 line

From b03302028893ce7465ba7e8736abba1922469bc1 Mon Sep 17 00:00:00 2001

From: Jason Gunthorpe <[email protected]>
Date: Fri, 16 Oct 2020 08:46:01 -0300
Subject: [PATCH] lib/scatterlist: Do not limit max_segment to PAGE_ALIGNED
values

The main intention of the max_segment argument to __sg_alloc_table_from_pages() is to match the DMA layer segment size set by dma_set_max_seg_size().

Quoted 7 lines

Restricting the input to be page aligned makes it impossible to just connect the DMA layer to this API.

The only reason for a page alignment here is because the algorithm will overshoot the max_segment if it is not a multiple of PAGE_SIZE. Simply fix the alignment before starting and don't expose this implementation detail to the callers.

What does not make complete sense to me is the statement that input alignment requirement makes it impossible to connect to DMA layer, but then the patch goes to align behind the covers anyway.

At minimum the kerneldoc should explain that max_segment will still be rounded down. But wouldn't it be better for the API to be more explicit and just require aligned anyway?

I mean I have no idea what is the problem for connecting to the DMA layer. Is it a matter of too many call sites which would need to align? Or there is more to it?

Quoted 21 lines

A future patch will completely remove SCATTERLIST_MAX_SEGMENT.

---
  lib/scatterlist.c | 12 ++++++++----
  1 file changed, 8 insertions(+), 4 deletions(-)
lib/scatterlist.c +1 -1
index e102fdfaa75be7..ed2497c79a216b 100644
@@ -404,7 +404,7 @@ static struct scatterlist *get_next_sg(struct sg_table *table,
   * @n_pages:	 Number of pages in the pages array
   * @offset:      Offset from start of the first page to the start of a buffer
   * @size:        Number of valid bytes in the buffer (after offset)
- * @max_segment: Maximum size of a scatterlist node in bytes (page aligned)
+ * @max_segment: Maximum size of a scatterlist element in bytes
   * @prv:	 Last populated sge in sgt
   * @left_pages:  Left pages caller have to set after this call
   * @gfp_mask:	 GFP allocation mask

lib/scatterlist.c struct scatterlist *__sg_alloc_table_from_pages(struct sg_table *sgt, · line 443

Quoted 10 lines
lib/scatterlist.c +6 -1
@@ -435,7 +435,12 @@ struct scatterlist *__sg_alloc_table_from_pages(struct sg_table *sgt,
  	unsigned int added_nents = 0;
  	struct scatterlist *s = prv;
 
-	if (WARN_ON(!max_segment || offset_in_page(max_segment)))
+	/*
+	 * The algorithm below requires max_segment to be aligned to PAGE_SIZE
+	 * otherwise it can overshoot.
+	 */
+	max_segment = ALIGN_DOWN(max_segment, PAGE_SIZE);
+	if (WARN_ON(max_segment < PAGE_SIZE))

Equivalent to !max_segment or max_segment == 0 now.

And it's a bit weird API - "you can pass in any unaligned size apart from unaligned sizes less than a PAGE_SIZE". Makes me think more that explicit requirement to pass in page aligned was better.

Quoted 5 lines
lib/scatterlist.c +0 -0
  		return ERR_PTR(-EINVAL);
 
  	if (IS_ENABLED(CONFIG_ARCH_NO_SG_CHAIN) && prv)

lib/scatterlist.c int sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages, · line 553

Quoted 9 lines
lib/scatterlist.c +1 -2
@@ -542,8 +547,7 @@ int sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages,
  			      unsigned long size, gfp_t gfp_mask)
  {
  	return PTR_ERR_OR_ZERO(__sg_alloc_table_from_pages(sgt, pages, n_pages,
-			offset, size, SCATTERLIST_MAX_SEGMENT,
-			NULL, 0, gfp_mask));
+			offset, size, UINT_MAX, NULL, 0, gfp_mask));
  }
  EXPORT_SYMBOL(sg_alloc_table_from_pages);
 

Regards,

Tvrtko

maintainer
#11 lore raw
Quoted 11 lines · Tvrtko Ursulin
Quoted 3 lines

overshoot the max_segment if it is not a multiple of PAGE_SIZE. Simply fix the alignment before starting and don't expose this implementation detail to the callers.

What does not make complete sense to me is the statement that input alignment requirement makes it impossible to connect to DMA layer, but then the patch goes to align behind the covers anyway.

At minimum the kerneldoc should explain that max_segment will still be rounded down. But wouldn't it be better for the API to be more explicit and just require aligned anyway?

Why?

The API is to not produce sge's with a length longer than max_segment, it isn't to produce sge's of exactly max_segment.

Everything else is an internal detail

Jason

#12 lore raw
Quoted 13 lines · Jason Gunthorpe
Quoted 11 lines · Tvrtko Ursulin
Quoted 3 lines

overshoot the max_segment if it is not a multiple of PAGE_SIZE. Simply fix the alignment before starting and don't expose this implementation detail to the callers.

What does not make complete sense to me is the statement that input alignment requirement makes it impossible to connect to DMA layer, but then the patch goes to align behind the covers anyway.

At minimum the kerneldoc should explain that max_segment will still be rounded down. But wouldn't it be better for the API to be more explicit and just require aligned anyway?

Quoted 6 lines

Why?

The API is to not produce sge's with a length longer than max_segment, it isn't to produce sge's of exactly max_segment.

Everything else is an internal detail

(Half-)pretending it is disconnected from PAGE_SIZE, when it really isn't, isn't the most obvious API design in my view.

In other words, if you let users pass in 4097 and it just works by rounding down to 4096, but you don't let them pass 4095, I just find this odd.

My question was why not have callers pass in page aligned max segment like today which makes it completely transparent.

Regards,

Tvrtko

maintainer
#13 lore raw
Quoted 17 lines · Tvrtko Ursulin
Quoted 15 lines · Jason Gunthorpe
Quoted 11 lines · Tvrtko Ursulin
Quoted 3 lines

overshoot the max_segment if it is not a multiple of PAGE_SIZE. Simply fix the alignment before starting and don't expose this implementation detail to the callers.

What does not make complete sense to me is the statement that input alignment requirement makes it impossible to connect to DMA layer, but then the patch goes to align behind the covers anyway.

At minimum the kerneldoc should explain that max_segment will still be rounded down. But wouldn't it be better for the API to be more explicit and just require aligned anyway?

Why?

Quoted 7 lines
Quoted 4 lines

The API is to not produce sge's with a length longer than max_segment, it isn't to produce sge's of exactly max_segment.

Everything else is an internal detail

(Half-)pretending it is disconnected from PAGE_SIZE, when it really isn't, isn't the most obvious API design in my view.

It is not information the callers need to care about

Quoted 2 lines

In other words, if you let users pass in 4097 and it just works by rounding down to 4096, but you don't let them pass 4095, I just find this odd.

That is also an implementation detail, there is nothing preventing smaller than page size other than complexity of implementing the algorithm. If something ever needed it, it could be done.

Quoted 2 lines

My question was why not have callers pass in page aligned max segment like today which makes it completely transparent.

Why put this confusing code in every caller? Especially for something a driver is supposed to call. Will just make bugs

Jason

#14 lore raw
Quoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Tvrtko Ursulin
Quoted 18 lines · Jason Gunthorpe
Quoted 11 lines · Tvrtko Ursulin
Quoted 3 lines

overshoot the max_segment if it is not a multiple of PAGE_SIZE. Simply fix the alignment before starting and don't expose this implementation detail to the callers.

What does not make complete sense to me is the statement that input alignment requirement makes it impossible to connect to DMA layer, but then the patch goes to align behind the covers anyway.

At minimum the kerneldoc should explain that max_segment will still be rounded down. But wouldn't it be better for the API to be more explicit and just require aligned anyway?

Why?

The API is to not produce sge's with a length longer than max_segment, it isn't to produce sge's of exactly max_segment.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Everything else is an internal detail

(Half-)pretending it is disconnected from PAGE_SIZE, when it really isn't, isn't the most obvious API design in my view.

It is not information the callers need to care about

Unless they want a 1024 byte segment - then they do need to care. The patch just changes the rules from "any non-zero aligned" to "any greater or equal than PAGE_SIZE". I am simply asking why in more specific terms.

Quoted 6 lines
Quoted 2 lines

In other words, if you let users pass in 4097 and it just works by rounding down to 4096, but you don't let them pass 4095, I just find this odd.

That is also an implementation detail, there is nothing preventing smaller than page size other than complexity of implementing the algorithm. If something ever needed it, it could be done.

But does non aligned segment size make sense? Is there any current or future platform which will need to to warrant changing the API in this way?

Quoted 5 lines
Quoted 2 lines

My question was why not have callers pass in page aligned max segment like today which makes it completely transparent.

Why put this confusing code in every caller? Especially for something a driver is supposed to call. Will just make bugs

For max_segment to be aligned is a requirement today so callers are ready. I still don't understand why change that, what will it help with? Is it just about the desire to be able to pass in UINT_MAX?

Regards,

Tvrtko

maintainer
#15 lore raw
Quoted 5 lines · Tvrtko Ursulin
Quoted 2 lines

Why put this confusing code in every caller? Especially for something a driver is supposed to call. Will just make bugs

For max_segment to be aligned is a requirement today so callers are ready.

No, it turns out all the RDMA drivers were became broken when they converted to use the proper U32_MAX for their DMA max_segment size, then they couldn't form SGLs anymore.

I don't want to see nonsense code like this:

        dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
                                             SCATTERLIST_MAX_SEGMENT));

In drivers.

dma_set_max_seg_size is the *hardware* capability, and mixing in things like PAG_MASK here is just nonsense.

Jason

#16 lore raw
Quoted 14 lines · Jason Gunthorpe
Quoted 5 lines · Tvrtko Ursulin
Quoted 2 lines

Why put this confusing code in every caller? Especially for something a driver is supposed to call. Will just make bugs

For max_segment to be aligned is a requirement today so callers are ready.

No, it turns out all the RDMA drivers were became broken when they converted to use the proper U32_MAX for their DMA max_segment size, then they couldn't form SGLs anymore.

I don't want to see nonsense code like this:

Quoted 7 lines
         dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
                                              SCATTERLIST_MAX_SEGMENT));

In drivers.

dma_set_max_seg_size is the *hardware* capability, and mixing in things like PAG_MASK here is just nonsense.

Code was obviously a no-op non-sense.

So the crux of the argument is that U32_MAX is conceptually the right thing which defines the DMA max_segment size? Not some DMA define or anything, but really U32_MAX? And that all/some DMA hardware does not think in pages but really in bytes?

Regards,

Tvrtko

maintainer
#17 lore raw
Quoted 24 lines · Tvrtko Ursulin
Quoted 14 lines · Jason Gunthorpe
Quoted 5 lines · Tvrtko Ursulin
Quoted 2 lines

Why put this confusing code in every caller? Especially for something a driver is supposed to call. Will just make bugs

For max_segment to be aligned is a requirement today so callers are ready.

No, it turns out all the RDMA drivers were became broken when they converted to use the proper U32_MAX for their DMA max_segment size, then they couldn't form SGLs anymore.

I don't want to see nonsense code like this:

Quoted 7 lines
         dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
                                              SCATTERLIST_MAX_SEGMENT));

In drivers.

dma_set_max_seg_size is the *hardware* capability, and mixing in things like PAG_MASK here is just nonsense.

Quoted 6 lines

Code was obviously a no-op non-sense.

So the crux of the argument is that U32_MAX is conceptually the right thing which defines the DMA max_segment size? Not some DMA define or anything, but really U32_MAX? And that all/some DMA hardware does not think in pages but really in bytes?

Yes.

The HW has 32 bits for a length field, so U32_MAX accurately defines the HW DMA capability

Jason

#18 lore raw
Quoted 32 lines · Jason Gunthorpe
Quoted 24 lines · Tvrtko Ursulin
Quoted 14 lines · Jason Gunthorpe
Quoted 5 lines · Tvrtko Ursulin
Quoted 2 lines

Why put this confusing code in every caller? Especially for something a driver is supposed to call. Will just make bugs

For max_segment to be aligned is a requirement today so callers are ready.

No, it turns out all the RDMA drivers were became broken when they converted to use the proper U32_MAX for their DMA max_segment size, then they couldn't form SGLs anymore.

I don't want to see nonsense code like this:

Quoted 7 lines
          dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
                                               SCATTERLIST_MAX_SEGMENT));

In drivers.

dma_set_max_seg_size is the *hardware* capability, and mixing in things like PAG_MASK here is just nonsense.

Quoted 6 lines

Code was obviously a no-op non-sense.

So the crux of the argument is that U32_MAX is conceptually the right thing which defines the DMA max_segment size? Not some DMA define or anything, but really U32_MAX? And that all/some DMA hardware does not think in pages but really in bytes?

Quoted 4 lines

Yes.

The HW has 32 bits for a length field, so U32_MAX accurately defines the HW DMA capability

Not just the max, but the granularity as well. If byte granularity (well less than PAGE_SIZE/4k) exists in some hw then I agree the API change makes sense.

Regards,

Tvrtko

maintainer
#19 lore raw
Quoted 3 lines · Tvrtko Ursulin

Not just the max, but the granularity as well. If byte granularity (well less than PAGE_SIZE/4k) exists in some hw then I agree the API change makes sense.

scatter/gather lists are byte granular in HW, this is basically the norm.

Page lists are something a little different.

At least in RDMA we use scatterlist for both types of objects, but their treatement is quite different. For page lists the max_segment_size is just something that gets in the way, we really want the SGLs to be as large, and as highly aligned as possible.

Fewer SGE's means less work and less memory everywhere that processes them.

When it comes times to progam the HW the SGL is analyzed and the HW specific page size selected, then the SGL is broken up into a page list. Each SGE may be split into many HW DMA blocks.

Again, this has nothing to do with PAGE_SIZE, the HW DMA block size selection is HW specific and handled when splitting the SGL.

Jason

#20 lore raw
In reply to Maor Gottlieb
Quoted 8 lines · Maor Gottlieb
Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · Bob Pearson

Jason,

Just pulled for-next and now hit the following warning.
Register user space memory is not longer working.
I am trying to debug this but if you have any idea where to look let me know.

Quoted 11 lines
Quoted 3 lines

The offset_in_page is wrong, but it is protecting some other logic..

Maor? Leon? Can you sort it out tomorrow?

Leon and I investigated it. This check existed before my series to protect the alloc_table_from_pages logic. It's still relevant.
This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the dma_device to ib_register_device"), and according to below link it was expected.  The safest approach is to set the max_segment_size back the 2GB in all drivers. What do you think?

https://lore.kernel.org/linux-rdma/[email protected]/

Quoted 2 lines

Jason

That's what I did to fix it.