Skip to contentSkip to conversation

net/mlx5: Use unique names for software steering caches

PATCH RESEND In review Tariq Toukan 4 participants 8 messages Jul 30 – Oct 27

Subsystems (4)
lore
Share & tools
lore

Tariq Toukan

author maintainer
#0 lore raw

PATCH RESEND 1/1 net/mlx5: Use unique names for software steering caches

From: Leon Romanovsky <[email protected]>

Each software steering domain creates its own slab caches, but all domains use the same names. When domains for two devices are alive at once, the second kmem_cache_create() triggers the following splat:

WARNING: mm/slab_common.c:111 at __kmem_cache_create_args+0xca/0x480, CPU#18: devlink/331372
Modules linked in: act_mirred act_skbedit cls_matchall act_gact cls_flower sch_ingress

vhost_vdpa veth nfnetlink_cttimeout openvswitch macvtap macvlan vfio_ap kvm nf_nat_tftp nf_conntrack_tftp nsh nf_conncount vfio_pci_core irqbypass scsi_debug vhost_net tap tun
vhost_vsock vmw_vsock_virtio_transport_common vsock vhost nft_masq nft_reject_ipv4 act_csum
cls_u32 sch_htb smc_diag smc ppp_deflate bsd_comp ppp_async crc_ccitt ppp_generic slhc loop algif_hash af_alg nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables mlx5_vdpa vdpa mlx5_ib dm_service_time ib_uverbs_support vringh ib_core vhost_iotlb mlx5_core s390_trng eadm_sch vfio_ccw mdev vfio_iommu_type1 vfio sch_fq_codel drm i2c_core dm_multipath drm_panel_orientation_quirks uvdevice diag288_wdt watchdog hmac_s390 prng aes_s390 zfcp scsi_transport_fc pkey_pckmo pkey_cca pkey_ep11 zcrypt paes_s390 phmac_s390 rng_core scsi_dh_alua pkey scsi_dh_rdac scsi_dh_emc crypto_engine autofs4 ecdsa_generic ecc sha512 [last unloaded: openvswitch]

CPU: 18 UID: 0 PID: 331372 Comm: devlink Tainted: G        W           7.2.0-20260712.rc2.git0.e3321fa3034d.300.fc44.s390x+debug #1 PREEMPT
Tainted: [W]=WARN
Hardware name: IBM 9175 ME1 701 (LPAR)
Krnl PSW : 0704c00180000000 0000038139a6771a (__kmem_cache_create_args+0xda/0x480)
            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
Krnl GPRS: 0000000000000000 0000000000000000 000003813b7de578 00000380b9eb8974
            00000276c417f690 00000380b9eb8974 0000030144623348 00000277234a1660
            0000000000000020 00000380b9eb8974 00000277234a1600 000003813b686d30
            0000000000000000 00000380b9e9a810 0000038139a6771a 0000030144623238
Krnl Code: 0000038139a6770a: c02000ebb737       larl    %r2,000003813b7de578
            0000038139a67710: b9040039           lgr     %r3,%r9
           *0000038139a67714: c0e5006e2eb2       brasl   %r14,000003813a82d478
           >0000038139a6771a: a7390020           lghi    %r3,32
            0000038139a6771e: b9040029           lgr     %r2,%r9
            0000038139a67722: c0e5006cf98f       brasl   %r14,000003813a806a40
            0000038139a67728: ec26018e007c       cgij    %r2,0,6,0000038139a67a44
            0000038139a6772e: 58d0f0a4           l       %r13,164(%r15)
Call Trace:
  [<0000038139a6771a>] __kmem_cache_create_args+0xda/0x480
([<0000038139a676a2>] __kmem_cache_create_args+0x62/0x480)
  [<00000380b9da1c70>] dr_domain_init_mem_resources+0x80/0x240 [mlx5_core]
  [<00000380b9da226e>] dr_domain_init_resources.constprop.0+0x7e/0x2c0 [mlx5_core]
  [<00000380b9da28b2>] mlx5dr_domain_create+0x132/0x250 [mlx5_core]
  [<00000380b9dc1e20>] mlx5_cmd_dr_create_ns+0x30/0x90 [mlx5_core]
  [<00000380b9cd8dbe>] mlx5_flow_namespace_set_mode+0x6e/0x130 [mlx5_core]
  [<00000380b9d846ec>] esw_create_offloads_fdb_tables+0xac/0x5a0 [mlx5_core]
  [<00000380b9d865b6>] esw_offloads_steering_init+0x1c6/0x480 [mlx5_core]
  [<00000380b9d86e8e>] esw_offloads_enable+0x13e/0x410 [mlx5_core]
  [<00000380b9d7b04a>] mlx5_eswitch_enable_locked+0x36a/0x540 [mlx5_core]
  [<00000380b9d84ff0>] esw_offloads_start+0x50/0x1d0 [mlx5_core]
  [<00000380b9d8774a>] mlx5_devlink_eswitch_mode_set+0x35a/0x3f0 [mlx5_core]
  [<000003813a791e68>] devlink_nl_eswitch_set_doit+0x88/0x120
  [<000003813a5b93ea>] genl_family_rcv_msg_doit+0xea/0x150
  [<000003813a5b95c2>] genl_family_rcv_msg+0x172/0x210
  [<000003813a5b96c2>] genl_rcv_msg+0x62/0xc0
  [<000003813a5b7cac>] netlink_rcv_skb+0x5c/0x120
  [<000003813a5b8f0c>] genl_rcv+0x3c/0x50
  [<000003813a5b74a4>] netlink_unicast+0x1f4/0x2b0
  [<000003813a5b783c>] netlink_sendmsg+0x2dc/0x460
  [<000003813a4c9764>] __sock_sendmsg+0x64/0xd0
  [<000003813a4cc878>] __sys_sendto+0x108/0x160
  [<000003813a4cdf50>] __do_sys_socketcall+0x350/0x460
  [<000003813a8184d2>] __do_syscall+0x172/0x750
  [<000003813a82d5d2>] system_call+0x72/0x90

Prefix each cache name with the device name to make it unique.

Fixes fd785e5213f0 ("net/mlx5: DR, Allocate icm_chunks from their own slab allocator")
Fixes fb628b71fb2a ("net/mlx5: DR, Allocate htbl from its own slab allocator")
---
 .../ethernet/mellanox/mlx5/core/steering/sws/dr_domain.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/steering/sws/dr_domain.c +7 -2
index fedefb565a21..c9f20a9033eb 100644
@@ -98,9 +98,12 @@ int mlx5dr_domain_get_recalc_cs_ft_addr(struct mlx5dr_domain *dmn,
 
 static int dr_domain_init_mem_resources(struct mlx5dr_domain *dmn)
 {
+	char name[80];
 	int ret;
 
-	dmn->chunks_kmem_cache = kmem_cache_create("mlx5_dr_chunks",
+	snprintf(name, sizeof(name), "%s-mlx5_dr_chunks",
+		 dev_name(dmn->mdev->device));
+	dmn->chunks_kmem_cache = kmem_cache_create(name,
 						   sizeof(struct mlx5dr_icm_chunk), 0,
 						   SLAB_HWCACHE_ALIGN, NULL);
 	if (!dmn->chunks_kmem_cache) {
@@ -108,7 +111,9 @@ static int dr_domain_init_mem_resources(struct mlx5dr_domain *dmn)
 		return -ENOMEM;
 	}
 
-	dmn->htbls_kmem_cache = kmem_cache_create("mlx5_dr_htbls",
+	snprintf(name, sizeof(name), "%s-mlx5_dr_htbls",
+		 dev_name(dmn->mdev->device));
+	dmn->htbls_kmem_cache = kmem_cache_create(name,
 						  sizeof(struct mlx5dr_ste_htbl), 0,
 						  SLAB_HWCACHE_ALIGN, NULL);
 	if (!dmn->htbls_kmem_cache) {

base-commit: 3bd438a58e910db5dc369aa25dfed1fc95f1b596

Signature · 1 line
2.44.0
maintainer
#1 lore raw
Quoted 1 line · On Fri, 31 Jul 2026 00:02:00 +0300 Tariq Toukan wrote:

Prefix each cache name with the device name to make it unique.

Why do we need a per-device kmem_cache? Isn't the right fix to make this driver-level state?

#2 lore raw
Quoted 5 lines · Jakub Kicinski
Quoted 1 line · On Fri, 31 Jul 2026 00:02:00 +0300 Tariq Toukan wrote:

Prefix each cache name with the device name to make it unique.

Why do we need a per-device kmem_cache? Isn't the right fix to make this driver-level state?

This kmem_cache is required only for software managed steering mode, which is configured per device by mlx5_fs_mode_set(). In general, resources should be allocated only when they are actually needed.

I personally run everything in FW-managed steering mode, so this kmem_cache is useless to me.

Thanks

maintainer
#3 lore raw
Quoted 10 lines · On Tue, 4 Aug 2026 11:04:55 +0300 Leon Romanovsky wrote:
Quoted 5 lines · Jakub Kicinski
Quoted 1 line · On Fri, 31 Jul 2026 00:02:00 +0300 Tariq Toukan wrote:

Prefix each cache name with the device name to make it unique.

Why do we need a per-device kmem_cache? Isn't the right fix to make this driver-level state?

This kmem_cache is required only for software managed steering mode, which is configured per device by mlx5_fs_mode_set(). In general, resources should be allocated only when they are actually needed.

Of course you have to wrap the init in a lock and count users. Trivial stuff.

#4 lore raw
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 4 Aug 2026 11:04:55 +0300 Leon Romanovsky wrote:
Quoted 5 lines · Jakub Kicinski
Quoted 1 line · On Fri, 31 Jul 2026 00:02:00 +0300 Tariq Toukan wrote:

Prefix each cache name with the device name to make it unique.

Why do we need a per-device kmem_cache? Isn't the right fix to make this driver-level state?

Quoted 6 lines
Quoted 3 lines

This kmem_cache is required only for software managed steering mode, which is configured per device by mlx5_fs_mode_set(). In general, resources should be allocated only when they are actually needed.

Of course you have to wrap the init in a lock and count users. Trivial stuff.

I don't see why we should abandon the simple, architecturally correct approach of allocating and managing resources where they belong in favor of a more complex global scheme involving locking and reference counting.

Thanks

#5 lore raw

Am 06.08.26 um 13:17 schrieb Leon Romanovsky:

Quoted 13 lines · Jakub Kicinski
Quoted 12 lines · On Tue, 4 Aug 2026 11:04:55 +0300 Leon Romanovsky wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 31 Jul 2026 00:02:00 +0300 Tariq Toukan wrote:
Quoted 1 line

Prefix each cache name with the device name to make it unique.

Why do we need a per-device kmem_cache? Isn't the right fix to make this driver-level state?

This kmem_cache is required only for software managed steering mode, which is configured per device by mlx5_fs_mode_set(). In general, resources should be allocated only when they are actually needed.

Quoted 8 lines
Quoted 2 lines

Of course you have to wrap the init in a lock and count users. Trivial stuff.

I don't see why we should abandon the simple, architecturally correct approach of allocating and managing resources where they belong in favor of a more complex global scheme involving locking and reference counting.

Any updates on this?

#6 lore raw
Quoted 17 lines · Christian Borntraeger

Am 06.08.26 um 13:17 schrieb Leon Romanovsky:

Quoted 16 lines · Jakub Kicinski
Quoted 14 lines · On Tue, 4 Aug 2026 11:04:55 +0300 Leon Romanovsky wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 31 Jul 2026 00:02:00 +0300 Tariq Toukan wrote:
Quoted 1 line

Prefix each cache name with the device name to make it unique.

Why do we need a per-device kmem_cache? Isn't the right fix to make this driver-level state?

This kmem_cache is required only for software managed steering mode, which is configured per device by mlx5_fs_mode_set(). In general, resources should be allocated only when they are actually needed.

Of course you have to wrap the init in a lock and count users. Trivial stuff.

Quoted 6 lines
Quoted 5 lines

I don't see why we should abandon the simple, architecturally correct approach of allocating and managing resources where they belong in favor of a more complex global scheme involving locking and reference counting.

Any updates on this?

No, we are waiting for Jakub to explain why should we complicate simple and working solution.

Thanks

#7 lore raw

Am 15.09.26 um 13:13 schrieb Leon Romanovsky:

Quoted 23 lines · Christian Borntraeger
Quoted 22 lines

Am 06.08.26 um 13:17 schrieb Leon Romanovsky:

Quoted 21 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 4 Aug 2026 11:04:55 +0300 Leon Romanovsky wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 31 Jul 2026 00:02:00 +0300 Tariq Toukan wrote:
Quoted 1 line

Prefix each cache name with the device name to make it unique.

Why do we need a per-device kmem_cache? Isn't the right fix to make this driver-level state?

Quoted 6 lines
Quoted 3 lines

This kmem_cache is required only for software managed steering mode, which is configured per device by mlx5_fs_mode_set(). In general, resources should be allocated only when they are actually needed.

Of course you have to wrap the init in a lock and count users. Trivial stuff.

I don't see why we should abandon the simple, architecturally correct approach of allocating and managing resources where they belong in favor of a more complex global scheme involving locking and reference counting.

Quoted 4 lines
Quoted 1 line

Any updates on this?

No, we are waiting for Jakub to explain why should we complicate simple and working solution.

Ping.
So in the end this is something for you to sort out with Jakub. I can just say that we see this issue in our s390 CI without the patch so the patch as is is

I can offer to test alternative patches if any.