Skip to contentSkip to conversation

vmxnet3: Adjust maximum Rx ring buffer size

RFC PATCH In review Aaron Tomlin 4 participants 11 messages Jan 5, 2025 – Jan 15, 2025 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover vmxnet3: Adjust maximum Rx ring buffer size

Hi Ronak, Paolo,

I managed to trigger the MAX_PAGE_ORDER warning in the context of function __alloc_pages_noprof() with /usr/sbin/ethtool --set-ring rx 4096 rx-mini 2048 [devname]' using the maximum supported Ring 0 and Rx ring buffer size. Admittedly this was under the stock Linux kernel-4.18.0-477.27.1.el8_8 whereby CONFIG_CMA is not enabled. I think it does not make sense to attempt a large memory allocation request for physically contiguous memory, to hold the Rx Data ring that could exceed the maximum page-order supported by the system.

I am not familiar with drivers/net/vmxnet3 related code. Please let me know your thoughts. Thank you.

Aaron Tomlin (1):
vmxnet3: Adjust maximum Rx ring buffer size

 drivers/net/vmxnet3/vmxnet3_defs.h | 4 ++++
 1 file changed, 4 insertions(+)
Signature · 1 line
2.47.1
#1 lore raw

RFC PATCH 1/1 vmxnet3: Adjust maximum Rx ring buffer size

In the context of vmxnet3_rq_create(), the Rx Data ring's size is calculated by multiplying the size of Ring 0 by the size of the Rx ring buffer. See __dma_direct_alloc_pages(). Now if CMA (Contiguous Memory Allocator) is not available or the allocation attempt failed, the zone buddy allocator is used to try to allocate physically contiguous memory.

The problem is, when the maximum supported Ring 0 and Rx ring buffer size is selected, the page-order required to accommodate the new size of the Rx Data ring is greater than the default MAX_PAGE_ORDER (10) i.e. __get_order(4096 * 2048) == 11. Consequently, this request can trigger the following warning condition in __alloc_pages_noprof():

	if (WARN_ON_ONCE_GFP(order > MAX_PAGE_ORDER, gfp))
		return NULL;

This patch ensures that the maximum Rx ring buffer size is reduced under a Linux kernel without CMA (Contiguous Memory Allocator) support. There is no point attempting a large memory allocation request that could exceed the maximum page-order supported by the system.

---
 drivers/net/vmxnet3/vmxnet3_defs.h | 4 ++++
 1 file changed, 4 insertions(+)
drivers/net/vmxnet3/vmxnet3_defs.h +4 -0
index 5c5148768039..cc71e697a5f3 100644
@@ -466,7 +466,11 @@ union Vmxnet3_GenericDesc {
 #define VMXNET3_TXDATA_DESC_MIN_SIZE 128
 #define VMXNET3_TXDATA_DESC_MAX_SIZE 2048
 
+#if defined(CONFIG_DMA_CMA)
 #define VMXNET3_RXDATA_DESC_MAX_SIZE 2048
+#else
+#define VMXNET3_RXDATA_DESC_MAX_SIZE 1024
+#endif
 
 #define VMXNET3_TXTS_DESC_MAX_SIZE   256
 #define VMXNET3_RXTS_DESC_MAX_SIZE   256
Signature · 1 line
2.47.1
maintainer
#2 lore raw
In reply to Aaron Tomlin
Quoted 8 lines · On Sun, 5 Jan 2025 21:30:35 +0000 Aaron Tomlin wrote:

I managed to trigger the MAX_PAGE_ORDER warning in the context of function __alloc_pages_noprof() with /usr/sbin/ethtool --set-ring rx 4096 rx-mini 2048 [devname]' using the maximum supported Ring 0 and Rx ring buffer size. Admittedly this was under the stock Linux kernel-4.18.0-477.27.1.el8_8 whereby CONFIG_CMA is not enabled. I think it does not make sense to attempt a large memory allocation request for physically contiguous memory, to hold the Rx Data ring that could exceed the maximum page-order supported by the system.

I think CMA should be a bit orthogonal to the warning.

Off the top of my head the usual way to solve the warning is to add __GFP_NOWARN to the allocations which trigger it. And then handle the error gracefully.

#3 lore raw
Quoted 10 lines · Jakub Kicinski' via BCM-KERNEL-FEEDBACK-LIST,PDL
Quoted 8 lines · On Sun, 5 Jan 2025 21:30:35 +0000 Aaron Tomlin wrote:

I managed to trigger the MAX_PAGE_ORDER warning in the context of function __alloc_pages_noprof() with /usr/sbin/ethtool --set-ring rx 4096 rx-mini 2048 [devname]' using the maximum supported Ring 0 and Rx ring buffer size. Admittedly this was under the stock Linux kernel-4.18.0-477.27.1.el8_8 whereby CONFIG_CMA is not enabled. I think it does not make sense to attempt a large memory allocation request for physically contiguous memory, to hold the Rx Data ring that could exceed the maximum page-order supported by the system.

Quoted 5 lines

I think CMA should be a bit orthogonal to the warning.

Off the top of my head the usual way to solve the warning is to add __GFP_NOWARN to the allocations which trigger it. And then handle the error gracefully.

That IMHO should really be the default for any driver that calls __netdev_alloc_skb() under the hood, we should not really have to specify __GFP_NOWARN, rather if people want it, they should specify it.

Signature · 1 line
Florian
maintainer
#4 lore raw
Quoted 13 lines · On Mon, 6 Jan 2025 15:51:10 -0800 Florian Fainelli wrote:
Quoted 12 lines · Jakub Kicinski' via BCM-KERNEL-FEEDBACK-LIST,PDL
Quoted 8 lines · On Sun, 5 Jan 2025 21:30:35 +0000 Aaron Tomlin wrote:

I managed to trigger the MAX_PAGE_ORDER warning in the context of function __alloc_pages_noprof() with /usr/sbin/ethtool --set-ring rx 4096 rx-mini 2048 [devname]' using the maximum supported Ring 0 and Rx ring buffer size. Admittedly this was under the stock Linux kernel-4.18.0-477.27.1.el8_8 whereby CONFIG_CMA is not enabled. I think it does not make sense to attempt a large memory allocation request for physically contiguous memory, to hold the Rx Data ring that could exceed the maximum page-order supported by the system.

I think CMA should be a bit orthogonal to the warning.

Quoted 7 lines
Quoted 3 lines

Off the top of my head the usual way to solve the warning is to add __GFP_NOWARN to the allocations which trigger it. And then handle the error gracefully.

That IMHO should really be the default for any driver that calls __netdev_alloc_skb() under the hood, we should not really have to specify __GFP_NOWARN, rather if people want it, they should specify it.

True, although TBH I don't fully understand why this flag exists in the first place. Is it just supposed to be catching programming errors, or is it due to potential DoS implications of users triggering large allocations?

#5 lore raw
Quoted 4 lines · Jakub Kicinski

True, although TBH I don't fully understand why this flag exists in the first place. Is it just supposed to be catching programming errors, or is it due to potential DoS implications of users triggering large allocations?

Jakub,

I suspect that introducing __GFP_NOWARN would mask the issue, no? I think the warning was useful. Otherwise it would be rather difficult to establish precisely why the Rx Data ring was disable. In this particular case, if I understand correctly, the intended size of the Rx Data ring was simply too large due to the size of the maximum supported Rx Data buffer.

Signature · 1 line
Aaron Tomlin
maintainer
#6 lore raw
Quoted 6 lines · On Tue, 7 Jan 2025 22:55:38 +0000 (GMT) Aaron Tomlin wrote:
Quoted 4 lines · Jakub Kicinski

True, although TBH I don't fully understand why this flag exists in the first place. Is it just supposed to be catching programming errors, or is it due to potential DoS implications of users triggering large allocations?

Quoted 7 lines

Jakub,

I suspect that introducing __GFP_NOWARN would mask the issue, no? I think the warning was useful. Otherwise it would be rather difficult to establish precisely why the Rx Data ring was disable. In this particular case, if I understand correctly, the intended size of the Rx Data ring was simply too large due to the size of the maximum supported Rx Data buffer.

This is a bit of a weird driver. But we should distinguish the default ring size, which yes, should not be too large, and max ring size which can be large but user setting a large size risks the fact the allocations will fail and device will not open.

This driver seems to read the default size from the hypervisor, is that the value that is too large in your case? Maybe we should min() it with something reasonable? The max allowed to be set via ethtool can remain high IMO

A message that is not in the archive (sent off-list or not yet received): CAP1Q3XQ_Fubke4=SYrFkaiJj0RHB99ehdMedMVDTFtRS6R_RCw@mail.gmail.com

maintainer
#8 lore raw
Quoted 2 lines · Jakub Kicinski

This driver seems to read the default size from the hypervisor, is that the value that is too large in your case?

The default should be 128 which is way less than max value.

Thanks,
Ronak

Signature · 11 lines
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.
#9 lore raw
In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski

This is a bit of a weird driver. But we should distinguish the default ring size, which yes, should not be too large, and max ring size which can be large but user setting a large size risks the fact the allocations will fail and device will not open.

This driver seems to read the default size from the hypervisor, is that the value that is too large in your case? Maybe we should min() it with something reasonable? The max allowed to be set via ethtool can remain high IMO

See vmxnet3_get_ringparam(). If I understand correctly, since commit 50a5ce3e7116a ("vmxnet3: add receive data ring support"), if the specified VMXNET3 adapter has support for the Rx Data ring feature then the maximum Rx Data buffer size is reported as VMXNET3_RXDATA_DESC_MAX_SIZE (i.e. 2048) by 'ethtool'. Furthermore, See vmxnet3_set_ringparam(). A user specified Rx mini value cannot be more than VMXNET3_RXDATA_DESC_MAX_SIZE. Indeed the Rx mini value in the context of VMXNET3 would be the size of the Rx Data ring buffer. See the following excerpt from vmxnet3_set_ringparam(). As far as I can tell, the Rx Data buffer cannot be more than VMXNET3_RXDATA_DESC_MAX_SIZE:

686 static int
687 vmxnet3_set_ringparam(struct net_device *netdev,

 688                       struct ethtool_ringparam *param,
 689                       struct kernel_ethtool_ringparam *kernel_param,
 690                       struct netlink_ext_ack *extack)

691 {
:
760 new_rxdata_desc_size =
761 (param->rx_mini_pending + VMXNET3_RXDATA_DESC_SIZE_MASK) &
762 ~VMXNET3_RXDATA_DESC_SIZE_MASK;
763 new_rxdata_desc_size = min_t(u16, new_rxdata_desc_size,
764 VMXNET3_RXDATA_DESC_MAX_SIZE);

Have I missed something?

Signature · 1 line
Aaron Tomlin
#10 lore raw
Quoted 37 lines · Aaron Tomlin
Quoted 10 lines · Jakub Kicinski

This is a bit of a weird driver. But we should distinguish the default ring size, which yes, should not be too large, and max ring size which can be large but user setting a large size risks the fact the allocations will fail and device will not open.

This driver seems to read the default size from the hypervisor, is that the value that is too large in your case? Maybe we should min() it with something reasonable? The max allowed to be set via ethtool can remain high IMO

See vmxnet3_get_ringparam(). If I understand correctly, since commit 50a5ce3e7116a ("vmxnet3: add receive data ring support"), if the specified VMXNET3 adapter has support for the Rx Data ring feature then the maximum Rx Data buffer size is reported as VMXNET3_RXDATA_DESC_MAX_SIZE (i.e. 2048) by 'ethtool'. Furthermore, See vmxnet3_set_ringparam(). A user specified Rx mini value cannot be more than VMXNET3_RXDATA_DESC_MAX_SIZE. Indeed the Rx mini value in the context of VMXNET3 would be the size of the Rx Data ring buffer. See the following excerpt from vmxnet3_set_ringparam(). As far as I can tell, the Rx Data buffer cannot be more than VMXNET3_RXDATA_DESC_MAX_SIZE:

686 static int
687 vmxnet3_set_ringparam(struct net_device *netdev,

 688                       struct ethtool_ringparam *param,
 689                       struct kernel_ethtool_ringparam *kernel_param,
 690                       struct netlink_ext_ack *extack)

691 {
:
760 new_rxdata_desc_size =
761 (param->rx_mini_pending + VMXNET3_RXDATA_DESC_SIZE_MASK) &
762 ~VMXNET3_RXDATA_DESC_SIZE_MASK;
763 new_rxdata_desc_size = min_t(u16, new_rxdata_desc_size,
764 VMXNET3_RXDATA_DESC_MAX_SIZE);

Quoted 1 line

Have I missed something?

Any thoughts?

Kind regards,

Signature · 1 line
Aaron Tomlin
#11 lore raw
In reply to Jakub Kicinski
Quoted 18 lines · Jakub Kicinski
Quoted 17 lines · On Mon, 6 Jan 2025 15:51:10 -0800 Florian Fainelli wrote:
Quoted 15 lines · Jakub Kicinski' via BCM-KERNEL-FEEDBACK-LIST,PDL
Quoted 8 lines · On Sun, 5 Jan 2025 21:30:35 +0000 Aaron Tomlin wrote:

I managed to trigger the MAX_PAGE_ORDER warning in the context of function __alloc_pages_noprof() with /usr/sbin/ethtool --set-ring rx 4096 rx-mini 2048 [devname]' using the maximum supported Ring 0 and Rx ring buffer size. Admittedly this was under the stock Linux kernel-4.18.0-477.27.1.el8_8 whereby CONFIG_CMA is not enabled. I think it does not make sense to attempt a large memory allocation request for physically contiguous memory, to hold the Rx Data ring that could exceed the maximum page-order supported by the system.

I think CMA should be a bit orthogonal to the warning.

Off the top of my head the usual way to solve the warning is to add __GFP_NOWARN to the allocations which trigger it. And then handle the error gracefully.

Quoted 9 lines
Quoted 3 lines

That IMHO should really be the default for any driver that calls __netdev_alloc_skb() under the hood, we should not really have to specify __GFP_NOWARN, rather if people want it, they should specify it.

True, although TBH I don't fully understand why this flag exists in the first place. Is it just supposed to be catching programming errors, or is it due to potential DoS implications of users triggering large allocations?

There is some value IMHO in printing when allocations fail, where they came from, their gfp_t flags and page order so you can track high order offenders in hot paths (one of our Wi-Fi driver was notorious for doing that and having verbose out of memory dumps by default definitively helped). Once you fix those however, hogging the system while dumping lines and lines of information onto a slow console tends to be worse than the recovery from out of memory itself. One could argue that triggering an OOM plus dumping information can result in a DoS, so that should be frowned upon...

Signature · 1 line
Florian