static_call: Fix a wild-memory-access bug in static_call_del_module()

Inject fault while probing btrfs.ko, if the first kzalloc() fails in __static_call_init(), key->mods will no be initialized. And then in static_call_del_module() the site_mod->mod will cause wild-memory-access as below:

So assign key->mods to NULL in __static_call_init() if it fails to fix the issue. And if kzalloc fails, it will just return in init func, so it should break if it the key->mods is NULL in exit func.

 general protection fault, probably for non-canonical address 0xeb800159c89f94a0: 0000 [#1] PREEMPT SMP KASAN
 KASAN: maybe wild-memory-access in range [0x5c002ace44fca500-0x5c002ace44fca507]
 CPU: 2 PID: 1843 Comm: modprobe Tainted: G        W        N 6.6.0-rc1+ #60
 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
 RIP: 0010:static_call_del_module+0x113/0x280
 Code: 3c 20 00 0f 85 ef 00 00 00 49 8b 6e 08 48 85 ed 75 0d eb 75 48 85 db 74 70 49 89 ef 48 89 dd 48 8d 7d 08 48 89 f8 48 c1 e8 03 <42> 80 3c 20 00 75 78 48 89 e8 4c 8b 6d 08 48 c1 e8 03 42 80 3c 20
 RSP: 0018:ffff888101d3f860 EFLAGS: 00010206
 RAX: 0b800559c89f94a0 RBX: 5c002ace44fca4f8 RCX: ffffffffa0210f00
 RDX: ffffffffa0210ed4 RSI: ffffffffa0210edc RDI: 5c002ace44fca500
 RBP: 5c002ace44fca4f8 R08: 0000000000000000 R09: ffffed10233e4eea
 R10: ffffed10233e4ee9 R11: ffff888119f2774b R12: dffffc0000000000
 R13: 80002ace3cfca4f8 R14: ffffffff85196de0 R15: ffffffff84ee9f99
 FS:  00007f4ff6faa540(0000) GS:ffff888119f00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007ffc3d1f19e8 CR3: 0000000109fa6001 CR4: 0000000000170ee0
 DR0: ffffffff8faefce8 DR1: ffffffff8faefce9 DR2: ffffffff8faefcea
 DR3: ffffffff8faefceb DR6: 00000000ffff0ff0 DR7: 0000000000000600
 Call Trace:
  <TASK>
  ? __die_body+0x1b/0x60
  ? die_addr+0x43/0x70
  ? exc_general_protection+0x121/0x210
  ? asm_exc_general_protection+0x22/0x30
  ? static_call_del_module+0x113/0x280
  ? __SCT__tp_func_ipi_exit+0x8/0x8
  static_call_module_notify+0x27f/0x390
  ? rcu_segcblist_inc_len+0x17/0x20
  notifier_call_chain+0xbf/0x280
  notifier_call_chain_robust+0x7f/0xe0
  ? notifier_call_chain+0x280/0x280
  ? kasan_quarantine_put+0x46/0x160
  blocking_notifier_call_chain_robust+0x5b/0x80
  load_module+0x4d1d/0x69f0
  ? module_frob_arch_sections+0x20/0x20
  ? update_cfs_group+0x10c/0x2a0
  ? __wake_up_common+0x10b/0x5d0
  ? kernel_read_file+0x3ca/0x510
  ? __x64_sys_fsconfig+0x650/0x650
  ? __schedule+0xa0b/0x2a60
  ? init_module_from_file+0xd2/0x130
  init_module_from_file+0xd2/0x130
  ? __ia32_sys_init_module+0xa0/0xa0
  ? _raw_spin_lock_irqsave+0xe0/0xe0
  ? ptrace_stop+0x487/0x790
  idempotent_init_module+0x32d/0x6a0
  ? init_module_from_file+0x130/0x130
  ? __fget_light+0x57/0x500
  __x64_sys_finit_module+0xbb/0x130
  do_syscall_64+0x35/0x80
  entry_SYSCALL_64_after_hwframe+0x46/0xb0
 RIP: 0033:0x7f4ff691b839
 Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1f f6 2c 00 f7 d8 64 89 01 48
 RSP: 002b:00007ffc07b09718 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
 RAX: ffffffffffffffda RBX: 000055978f13e070 RCX: 00007f4ff691b839
 RDX: 0000000000000000 RSI: 000055978da1bc2e RDI: 0000000000000003
 RBP: 000055978da1bc2e R08: 0000000000000000 R09: 000055978f13ddb0
 R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000
 R13: 000055978f13e020 R14: 0000000000040000 R15: 000055978f13ddb0
  </TASK>
 Modules linked in: tifm_core(+)
 Dumping ftrace buffer:
    (ftrace buffer empty)
 ---[ end trace 0000000000000000 ]---
 RIP: 0010:static_call_del_module+0x113/0x280
 Code: 3c 20 00 0f 85 ef 00 00 00 49 8b 6e 08 48 85 ed 75 0d eb 75 48 85 db 74 70 49 89 ef 48 89 dd 48 8d 7d 08 48 89 f8 48 c1 e8 03 <42> 80 3c 20 00 75 78 48 89 e8 4c 8b 6d 08 48 c1 e8 03 42 80 3c 20
 RSP: 0018:ffff888101d3f860 EFLAGS: 00010206
 RAX: 0b800559c89f94a0 RBX: 5c002ace44fca4f8 RCX: ffffffffa0210f00
 RDX: ffffffffa0210ed4 RSI: ffffffffa0210edc RDI: 5c002ace44fca500
 RBP: 5c002ace44fca4f8 R08: 0000000000000000 R09: ffffed10233e4eea
 R10: ffffed10233e4ee9 R11: ffff888119f2774b R12: dffffc0000000000
 R13: 80002ace3cfca4f8 R14: ffffffff85196de0 R15: ffffffff84ee9f99
 FS:  00007f4ff6faa540(0000) GS:ffff888119f00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007ffc3d1f19e8 CR3: 0000000109fa6001 CR4: 0000000000170ee0
 DR0: ffffffff8faefce8 DR1: ffffffff8faefce9 DR2: ffffffff8faefcea
 DR3: ffffffff8faefceb DR6: 00000000ffff0ff0 DR7: 0000000000000600
 Kernel panic - not syncing: Fatal exception
 Dumping ftrace buffer:
    (ftrace buffer empty)
 Kernel Offset: disabled
 Rebooting in 1 seconds..

---
 kernel/static_call_inline.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

index 639397b5491c..e7aa70d33530 100644
@@ -256,8 +256,10 @@ static int __static_call_init(struct module *mod,
 			}
 
 			site_mod = kzalloc(sizeof(*site_mod), GFP_KERNEL);
-			if (!site_mod)
+			if (!site_mod) {
+				key->mods = NULL;
 				return -ENOMEM;
+			}
 
 			/*
 			 * When the key has a direct sites pointer, extract

@@ -422,7 +424,7 @@ static void static_call_del_module(struct module *mod)
 			;
 
 		if (!site_mod)
-			continue;
+			break;
 
 		*prev = site_mod->next;
 		kfree(site_mod);

2.34.1

I don't think we need that full splat.

And that looks wrong, that just moved code around.

Hi, it seems that this bug still exit.

Hi,

Le 02/09/2024 à 14:01, Jinjie Ruan a écrit :

Can you provide more details ?

In particular, what makes you think it is the exact same bug ?

Christophe

I apply this patch, and the problem not occurs.

Le 02/09/2024 à 14:07, Jinjie Ruan a écrit :

Ah, you mean the patch hasn't been applied yet ?

I thought you were telling that despite the patch the problem was still there.

Christophe

You fail to explain what setting key->mods to NULL prevents and why the same problem does not happen when the kzalloc() five lines further down fails.

Actually if you read the comment below carefully then you might find the reason why the current code explodes and you might figure out why you can't set the pointer to NULL.

Now why does this "fix" things?

The unmodified kernel crashes dereferencing key->mods. That means there is garbage in key->mods. But that garbage is not garbage:

struct static_call_key {
	void *func;
	union {
		/* bit 0: 0 = mods, 1 = sites */
		unsigned long type;
		struct static_call_mod *mods;
		struct static_call_site *sites;
	};
};

key->mods is actually key->sites, which holds the static call sites of the key. It has bit 0 set.

As that code does not expect anything else than a valid static_call_mod pointer it dereferences it unconditionally and crashes. See?

So why can't you set the pointer to NULL?

If that key is builtin and has actual builtin usage sites, then you destroy the key. If the memory allocation fail was temporary then a subsequent update of that static call will find no call site and the kernel happily invokes the previous function.

Now that the root cause is properly analyzed, the proper fix is obvious. See uncomplied below.

It does not need to do anything vs. the other kzalloc() fail because that is harmless. The key has key->mods set, but key->mods->mod and key->mods->next are both NULL in that case. So the inner loop breaks out and continues to the next site which has the uninitialized one. If all keys were already converted to mods _before_ this fail then the loop simply iterates further, but will nowhere find a site_mod->mod == mod.

Thanks,

tglx
---

@@ -411,6 +411,17 @@ static void static_call_del_module(struc
 
 	for (site = start; site < stop; site++) {
 		key = static_call_key(site);
+
+		/*
+		 * If the key was not updated due to a memory allocation
+		 * failure in __static_call_init() then treating key::sites
+		 * as key::mods in the code below would cause random memory
+		 * access and #GP. In that case all subsequent sites have
+		 * not been touched either, so stop iterating.
+		 */
+		if (static_call_key_sites(key))
+			break;
+
 		if (key == prev_key)
 			continue;
 

Module insertion invokes static_call_add_module() to initialize the static calls in a module. static_call_add_module() invokes __static_call_init(), which allocates a struct static_call_mod to either encapsulate the built-in static call sites of the associated key into it so further modules can be added or to append the module to the module chain.

If that allocation fails the function returns with an error code and the module core invokes static_call_del_module() to clean up eventually added static_call_mod entries.

This works correctly, when all keys used by the module were converted over to a module chain before the failure. If not then static_call_del_module() causes a #GP as it blindly assumes that key::mods points to a valid struct static_call_mod.

The problem is that key::mods is not a individual struct member of struct static_call_key, it's part of a union to save space:

        union {
                /* bit 0: 0 = mods, 1 = sites */
                unsigned long type;
                struct static_call_mod *mods;
                struct static_call_site *sites;
	};

key::sites is a pointer to the list of built-in usage sites of the static call. The type of the pointer is differentiated by bit 0. A mods pointer has the bit clear, the sites pointer has the bit set.

As static_call_del_module() blindly assumes that the pointer is a valid static_call_mod type, it fails to check for this failure case and dereferences the pointer to the list of built-in call sites, which is obviously bogus.

Cure it by checking whether the key has a sites or a mods pointer.

If it is a sites pointer then the key is not to be touched. As the sites are walked in the same order as in __static_call_init() the site walk can be terminated because all subsequent sites have not been touched by the init code due to the error exit.

If it was converted before the allocation fail, then the inner loop which searches for a module match will find nothing.

A fail in the second allocation in __static_call_init() is harmless and does not require special treatment. The first allocation succeeded and converted the key to a module chain. That first entry has mod::mod == NULL and mod::next == NULL, so the inner loop of static_call_del_module() will neither find a module match nor a module chain. The next site in the walk was either already converted, but can't match the module, or it will exit the outer loop because it has a static_call_site pointer and not a static_call_mod pointer.

---
 kernel/static_call_inline.c |   11 +++++++++++
 1 file changed, 11 insertions(+)

@@ -411,6 +411,17 @@ static void static_call_del_module(struc
 
 	for (site = start; site < stop; site++) {
 		key = static_call_key(site);
+
+		/*
+		 * If the key was not updated due to a memory allocation
+		 * failure in __static_call_init() then treating key::sites
+		 * as key::mods in the code below would cause random memory
+		 * access and #GP. In that case all subsequent sites have
+		 * not been touched either, so stop iterating.
+		 */
+		if (static_call_key_sites(key))
+			break;
+
 		if (key == prev_key)
 			continue;
 

That's a different problem.

 Oops: general protection fault, probably for
 non-canonical address 0xdffffc0000000001: 0000 [#1] PREEMPT SMP KASAN NOPTI

It's dereferencing a NULL pointer at 0x1. That's odd because bit 0 is set, which looks like a sites pointer. So static_call_key_sites() should return true, but obviously does not. So how does that happen?

It can't be a built-in key, so it's a module local one with key::sites == 0x1. So static_call_key_sites() sees bit 0 set, and then returns key::sites & ~0x01, which is obviously NULL. So the condition is false and the code below uses key::mods == 0x1....

So the check must be:

	if (!static_call_key_has_mods(key))
        	break;

I missed the module local case completely in my analysis. Can you please modify the condition and retest?

Thanks,

tglx

That said. This code is pointlessly noisy for the failure case.

Allocation fails are not a reason to warn about. -ENOMEM is propagated all the way to the caller, so it's sufficient to emit a pr_warn().

Peter?

Thanks,

tglx
---

@@ -453,7 +453,7 @@ static int static_call_module_notify(str
 	case MODULE_STATE_COMING:
 		ret = static_call_add_module(mod);
 		if (ret) {
-			WARN(1, "Failed to allocate memory for static calls");
+			pr_warn("Failed to allocate memory for static calls\n");
 			static_call_del_module(mod);
 		}
 		break;

Yeah, I think that should do.

static_call_module_notify() triggers a WARN_ON(), when memory allocation fails in __static_call_add_module().

That's not really justified, because the failure case must be correctly handled by the well known call chain and the error code is passed through to the initiating userspace application.

A memory allocation fail is not a fatal problem, but the WARN_ON() takes the machine out when panic_on_warn is set.

Replace it with a pr_warn().

---
 kernel/static_call_inline.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

@@ -453,7 +453,7 @@ static int static_call_module_notify(str
 	case MODULE_STATE_COMING:
 		ret = static_call_add_module(mod);
 		if (ret) {
-			WARN(1, "Failed to allocate memory for static calls");
+			pr_warn("Failed to allocate memory for static calls\n");
 			static_call_del_module(mod);
 		}
 		break;

Hi, Thomas,

with this patch, the issue not occurs again，

but there are some memory leak here same to the following problem:

Link:
https://lore.kernel.org/all/[email protected]/

unreferenced object 0xff11000104078480 (size 32):
comm "modprobe", pid 16978, jiffies 4295928553
hex dump (first 32 bytes):

    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 07 04 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 86f0f2f0):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000432b600 (size 32):
  comm "modprobe", pid 16994, jiffies 4295929985
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 32 04 00 00 11 ff 00 00 00 00 00 00 00 00  y.2.............
  backtrace (crc 32209446):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000115b24ac0 (size 32):
  comm "modprobe", pid 17010, jiffies 4295931422
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 b2 15 01 00 11 ff 02 00 00 00 01 00 00 00  y...............
  backtrace (crc b9fe288c):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100010196bc40 (size 32):
  comm "modprobe", pid 17032, jiffies 4295933559
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 96 01 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc e311731b):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000009b4ee40 (size 32):
  comm "modprobe", pid 17038, jiffies 4295934300
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 b4 09 00 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc ebd506c):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000a3fe300 (size 32):
  comm "modprobe", pid 17044, jiffies 4295935011
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 3f 0a 00 00 11 ff 00 00 00 fc ff df 41 57  y.?...........AW
  backtrace (crc b6459b1e):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000007ef52c0 (size 32):
  comm "modprobe", pid 17050, jiffies 4295935726
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 ef 07 00 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 9c68acac):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000798b980 (size 32):
  comm "modprobe", pid 17066, jiffies 4295937149
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 98 07 00 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 13cab529):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000006d79100 (size 32):
  comm "modprobe", pid 17072, jiffies 4295937871
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 d7 06 00 00 11 ff 34 0a 0e 00 00 00 00 00  y.......4.......
  backtrace (crc 468c4c12):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000eb6d300 (size 32):
  comm "modprobe", pid 17088, jiffies 4295939305
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 b6 0e 00 00 11 ff 44 24 08 89 0c 24 e8 9d  y.......D$...$..
  backtrace (crc 39abd416):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000001029680 (size 32):
  comm "modprobe", pid 17114, jiffies 4295941403
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 02 01 00 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 84d963e1):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000101919f80 (size 32):
  comm "modprobe", pid 17130, jiffies 4295942829
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 91 01 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 86dcd9db):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000108634800 (size 32):
  comm "modprobe", pid 17136, jiffies 4295943552
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 63 08 01 00 11 ff 00 00 00 00 00 00 00 00  y.c.............
  backtrace (crc 893b4b39):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000051d5640 (size 32):
  comm "modprobe", pid 17142, jiffies 4295944268
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 1d 05 00 00 11 ff 00 00 00 00 00 fc ff df  y...............
  backtrace (crc 4dd0396e):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110001013def40 (size 32):
  comm "modprobe", pid 17148, jiffies 4295944996
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 3d 01 01 00 11 ff 40 2b 0d 00 00 00 d4 ff  y.=.....@+......
  backtrace (crc 72d18c0f):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000102f95280 (size 32):
  comm "modprobe", pid 17154, jiffies 4295945705
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 f9 02 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 8492240a):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000a957fc0 (size 32):
  comm "modprobe", pid 17170, jiffies 4295947128
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 95 0a 00 00 11 ff 65 63 6b 00 00 00 00 00  y.......eck.....
  backtrace (crc a37aa232):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000ec1d300 (size 32):
  comm "modprobe", pid 17186, jiffies 4295948570
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 c1 0e 00 00 11 ff 69 6e 69 73 68 00 00 00  y.......inish...
  backtrace (crc 66341c55):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000108066ac0 (size 32):
  comm "modprobe", pid 17212, jiffies 4295950745
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 06 08 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 8145aa7d):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000125e2980 (size 32):
  comm "modprobe", pid 17218, jiffies 4295951462
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 5e 12 00 00 11 ff 00 00 00 00 00 00 00 00  y.^.............
  backtrace (crc 80eaf345):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000002460380 (size 32):
  comm "modprobe", pid 17264, jiffies 4295954895
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 00 00 00 00 00 00 00 ed 1e 00 00 00 d4 ff  y...............
  backtrace (crc aaffd3fe):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000009b4e180 (size 32):
  comm "modprobe", pid 17290, jiffies 4295957017
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 b4 09 00 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc ebd506c):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000040cf140 (size 32):
  comm "modprobe", pid 17306, jiffies 4295958400
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 0c 04 00 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 526d8572):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000006d77f80 (size 32):
  comm "modprobe", pid 17322, jiffies 4295959804
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 d7 06 00 00 11 ff f4 11 0e 00 00 00 00 00  y...............
  backtrace (crc 59f3c311):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110001013d3180 (size 32):
  comm "modprobe", pid 17398, jiffies 4295965375
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 3d 01 01 00 11 ff 69 6e 69 73 68 00 00 00  y.=.....inish...
  backtrace (crc 99084885):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000f1826c0 (size 32):
  comm "modprobe", pid 17414, jiffies 4295966816
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 18 0f 00 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc caba89a6):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000100d2a240 (size 32):
  comm "modprobe", pid 17450, jiffies 4295969601
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 d2 00 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 6692d274):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000524ea40 (size 32):
  comm "modprobe", pid 17466, jiffies 4295971059
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 24 05 00 00 11 ff 2b 00 00 00 00 00 00 00  y.$.....+.......
  backtrace (crc d5da4419):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000102f957c0 (size 32):
  comm "modprobe", pid 17482, jiffies 4295972468
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 f9 02 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 8492240a):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110001072e3d00 (size 32):
  comm "modprobe", pid 17526, jiffies 4295976765
  hex dump (first 32 bytes):
    61 6d 64 67 70 75 5f 73 79 6e 63 5f 65 6e 74 72  amdgpu_sync_entr
    79 00 2e 07 01 00 11 ff 00 00 00 00 00 00 00 00  y...............
  backtrace (crc 580819a6):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<00000000c53a6d7f>] 0xffffffffc01503a2
    [<00000000c68a3256>] 0xffffffffc0058026
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000a3fdb60 (size 16):
  comm "modprobe", pid 17578, jiffies 4295981003
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000005e51280 (size 16):
  comm "modprobe", pid 17594, jiffies 4295982450
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000071a1b60 (size 16):
  comm "modprobe", pid 17609, jiffies 4295983945
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100010f4a2ae0 (size 16):
  comm "modprobe", pid 17646, jiffies 4295986890
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000f20bd80 (size 16):
  comm "modprobe", pid 17662, jiffies 4295988248
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000100d26a00 (size 16):
  comm "modprobe", pid 17718, jiffies 4295992419
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100010152d240 (size 16):
  comm "modprobe", pid 17734, jiffies 4295993810
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 00 00  amdgpu_fence....
  backtrace (crc 9f81b1cd):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110001013ce0a0 (size 16):
  comm "modprobe", pid 17750, jiffies 4295995280
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000005e51dc0 (size 16):
  comm "modprobe", pid 17756, jiffies 4295996002
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000008b1bf80 (size 16):
  comm "modprobe", pid 17782, jiffies 4295998115
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000114116820 (size 16):
  comm "modprobe", pid 17798, jiffies 4295999501
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100011101d5e0 (size 16):
  comm "modprobe", pid 17825, jiffies 4296001629
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000009718320 (size 16):
  comm "modprobe", pid 17831, jiffies 4296002346
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100011515d660 (size 16):
  comm "modprobe", pid 17847, jiffies 4296003774
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000114a85080 (size 16):
  comm "modprobe", pid 17854, jiffies 4296004484
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000eb139a0 (size 16):
  comm "modprobe", pid 17860, jiffies 4296005214
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000009718b60 (size 16):
  comm "modprobe", pid 17876, jiffies 4296006637
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000005b81fc0 (size 16):
  comm "modprobe", pid 17892, jiffies 4296008053
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000071a1b40 (size 16):
  comm "modprobe", pid 17898, jiffies 4296008778
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000f20bb00 (size 16):
  comm "modprobe", pid 17904, jiffies 4296009509
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000df76200 (size 16):
  comm "modprobe", pid 17930, jiffies 4296011592
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000064b1ba0 (size 16):
  comm "modprobe", pid 17965, jiffies 4296014396
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110001011cbe80 (size 16):
  comm "modprobe", pid 17971, jiffies 4296015146
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000f073bc0 (size 16):
  comm "modprobe", pid 17987, jiffies 4296016566
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000107214fe0 (size 16):
  comm "modprobe", pid 18003, jiffies 4296017979
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000df769c0 (size 16):
  comm "modprobe", pid 18019, jiffies 4296019396
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100010f4a28a0 (size 16):
  comm "modprobe", pid 18035, jiffies 4296020834
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100000383b980 (size 16):
  comm "modprobe", pid 18041, jiffies 4296021552
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000023b4e60 (size 16):
  comm "modprobe", pid 18047, jiffies 4296022306
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff11000008b1b940 (size 16):
  comm "modprobe", pid 18053, jiffies 4296023061
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff1100010b8d64c0 (size 16):
  comm "modprobe", pid 18059, jiffies 4296023783
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110001151a1ac0 (size 16):
  comm "modprobe", pid 18075, jiffies 4296025098
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150
unreferenced object 0xff110000050fcea0 (size 16):
  comm "modprobe", pid 18101, jiffies 4296027230
  hex dump (first 16 bytes):
    61 6d 64 67 70 75 5f 66 65 6e 63 65 00 00 11 ff  amdgpu_fence....
  backtrace (crc e15a7d50):
    [<000000001b48c137>] __kmalloc_node_track_caller_noprof+0x33c/0x410
    [<000000003ab002c7>] kstrdup+0x3f/0x80
    [<00000000479d3195>] kstrdup_const+0x39/0x60
    [<000000005f96320e>] kvasprintf_const+0xf9/0x190
    [<00000000725db2e2>] kobject_set_name_vargs+0x5a/0x150
    [<00000000745aee9e>] kobject_init_and_add+0xcd/0x170
    [<00000000c0d3244b>] sysfs_slab_add+0x172/0x210
    [<00000000a6c0a88d>] __kmem_cache_create+0x215/0x5c0
    [<0000000081d00250>] kmem_cache_create_usercopy+0x213/0x340
    [<000000006d9fd37f>] kmem_cache_create+0x11/0x20
    [<000000009827d27d>] 0xffffffffc00c8052
    [<00000000d74769dd>] 0xffffffffc0058038
    [<00000000b58b0e32>] do_one_initcall+0xdc/0x550
    [<0000000040a69501>] do_init_module+0x241/0x630
    [<00000000bca76796>] load_module+0x52be/0x63e0
    [<000000006d613fc0>] init_module_from_file+0xe6/0x150

That has absolutely nothing to do with static calls and the memory allocation failure case there.

The module passed all preparatory steps, otherwise it would not be able to create a kmem_cache from the module init() function:

     kmem_cache_create+0x11/0x20
     do_one_initcall+0xdc/0x550
     do_init_module+0x241/0x630

amdgpu_init()

	r = amdgpu_sync_init();
	if (r)
		goto error_sync;

	r = amdgpu_fence_slab_init();
	if (r)
		goto error_fence;

        <SNIP>
        
	return pci_register_driver(&amdgpu_kms_pci_driver);

error_fence:

	amdgpu_sync_fini();
error_sync:
        return r;

Can you spot the problem?

Thanks,

tglx

I see, let me test it.

Hi, Thomas,

It seems that it is not the memory leak source,

I test with the following patch, the memory leak also occurs,

but with the Link patch, the problem solved and the memory leak missed.

 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 094498a0964b..3589e4768bd6 100644

@@ -3032,11 +3032,11 @@ static int __init amdgpu_init(void)
 
        r = amdgpu_sync_init();
        if (r)
-               goto error_sync;
+               return r;
 
        r = amdgpu_fence_slab_init();
        if (r)
-               goto error_fence;
+               goto error_sync;
 
        DRM_INFO("amdgpu kernel modesetting enabled.\n");
        amdgpu_register_atpx_handler();

@@ -3046,12 +3046,18 @@ static int __init amdgpu_init(void)
        amdgpu_amdkfd_init();
 
        /* let modprobe override vga console setting */
-       return pci_register_driver(&amdgpu_kms_pci_driver);
+       r = pci_register_driver(&amdgpu_kms_pci_driver);
+       if (r)
+               goto error_fence;
+
+       return 0;
 
 error_fence:
-       amdgpu_sync_fini();
+       amdgpu_fence_slab_fini();
 
 error_sync:
+       amdgpu_sync_fini();
+
        return r;
 }
 

That's possible, but the code in there is definitely incorrect when pci_register_driver() fails. Simply because module->exit() is not invoked when module->init() fails.

Sure, but the change log of this patch is not really helpful at all and you sent me a gazillion of leak dumps, but failed to explain what the actual failure scenario is.

So without a coherent description how to reproduce this issue there is not much I can do than looking at the code. The amdgpu init() function was an obvious candidate, but there seems to be some other way to cause that problem.

Now you at least provided the information that the missing cleanup in the init() function is not the problem. So the obvious place to look is in the module core code whether there is a failure path _after_ module->init() returned success.

do_init_module()

        ret = do_one_initcall(mod->init);
        ...
	ret = module_enable_rodata_ro(mod, true);
	if (ret)
		goto fail_mutex_unlock;

and that error path does _not_ invoke module->exit(), which is obviously not correct. Luis?

I assume that's not the problem when looking at the change log of that:

https://lore.kernel.org/all/[email protected]/

Why? The rules are exactly the same, no?

Which function and which problem?

Let me try to decode this. I assume that sysfs_slab_add() fails after kobject_init_and_add() or kobject_init_and_add) itself fails.

So the problem is that there are two ways how this can be invoked:

1) from slab_sysfs_init() for the kmem_cache instances which have been allocated during early boot before sysfs was available.

2) from kmem_cache_create() after early boot

So what Liu tries to avoid is to invoke kobject_put(), because kobject_put() would not only free the name (which is what the leak complains about), but also invoke the release callback, which would try to destroy the kmem_cache itself.

That's a problem for the mm people to solve, but that does not make my observations about the amdgpu init() function and the error path in do_init_module() moot :)

Thanks,

tglx

You're spot on this needs fixing.

Luis

Christophe, this is a regression caused by the second hunk of your commit d1909c0221739 ("module: Don't ignore errors from set_memory_XX()") on v6.9. Sadly there are a few issues with trying to get to call mod->exit():

- We should try try_stop_module() and that can fail
- source_list may not be empty and that would block removal
- mod->exit may not exist

I'm wondering if instead we should try to do the module_enable_rodata_ro() before the init, but that requires a bit more careful evaluation...

Luis

There is ro_after_init section, we can't really make it RO before ->init()

Sincerely yours,
Mike.

Hi Luis,

Le 24/09/2024 à 09:22, Mike Rapoport a écrit :

Surprisingly I never received Luis's email allthough I got this answer from Mike that I overlooked.

So coming back here from https://lore.kernel.org/all/[email protected]/

As far as I understand, indeed once init is called it is too late to fail, right ? Especially when the module has no exit() path or when CONFIG_MODULE_UNLOAD is not built in.

So the only thing we can do then is a big fat warning telling set_memory_ro() on ro_after_init memory has failed ?

Maybe we should try and change it to RO then back to RW before calling init, to be on a safer side hopping that if change to RO works once it will work twice ?

Christophe

+ Other new module maintainers

So odd..

Partly yes, party no. Party yes in that its a can of worms we have not had to deal with before, and also I worry about deadlocks, and the code to address this seems complex. right ?

That's exactly the other extreme case I fear for.

I suspect this is more sensible to do.

That's another approach wich could work, if we proove that this does work, it's a nice best effort and I think less or a mess to the codebase then special-casing the error handling of trying to deal with the driver's exit.

Daniel Gomez has been looking at this, so his feedback here would be valuable.

Luis

I have a RFC ready with this, I'll send this now so we can discuss on with a proposal.

I came to the same conclusion while trying to fix this path. + I added an alternative for discussion.

What if we detect ro_after_init first, and block any module initialization depending on this ro_after_init to actually start loading it? That way we can stop and unload the module successfully.

In case I'm missing someone, I've just sent the RFC here:

https://lore.kernel.org/linux-modules/[email protected]/T/#t

Please ignore "v3" prefix. That was a mistake. Not sure why b4 added that.

Module insertion invokes static_call_add_module() to initialize the static calls in a module. static_call_add_module() invokes __static_call_init(), which allocates a struct static_call_mod to either encapsulate the built-in static call sites of the associated key into it so further modules can be added or to append the module to the module chain.

If that allocation fails the function returns with an error code and the module core invokes static_call_del_module() to clean up eventually added static_call_mod entries.

This works correctly, when all keys used by the module were converted over to a module chain before the failure. If not then static_call_del_module() causes a #GP as it blindly assumes that key::mods points to a valid struct static_call_mod.

The problem is that key::mods is not a individual struct member of struct static_call_key, it's part of a union to save space:

        union {
                /* bit 0: 0 = mods, 1 = sites */
                unsigned long type;
                struct static_call_mod *mods;
                struct static_call_site *sites;
	};

key::sites is a pointer to the list of built-in usage sites of the static call. The type of the pointer is differentiated by bit 0. A mods pointer has the bit clear, the sites pointer has the bit set.

As static_call_del_module() blidly assumes that the pointer is a valid static_call_mod type, it fails to check for this failure case and dereferences the pointer to the list of built-in call sites, which is obviously bogus.

Cure it by checking whether the key has a sites or a mods pointer.

If it's a sites pointer then the key is not to be touched. As the sites are walked in the same order as in __static_call_init() the site walk can be terminated because all subsequent sites have not been touched by the init code due to the error exit.

If it was converted before the allocation fail, then the inner loop which searches for a module match will find nothing.

A fail in the second allocation in __static_call_init() is harmless and does not require special treatment. The first allocation succeeded and converted the key to a module chain. That first entry has mod::mod == NULL and mod::next == NULL, so the inner loop of static_call_del_module() will neither find a module match nor a module chain. The next site in the walk was either already converted, but can't match the module, or it will exit the outer loop because it has a static_call_site pointer and not a static_call_mod pointer.

---
V2: Use static_call_key_has_mods() instead

---
 kernel/static_call_inline.c |   11 +++++++++++
 1 file changed, 11 insertions(+)

@@ -411,6 +411,17 @@ static void static_call_del_module(struc
 
 	for (site = start; site < stop; site++) {
 		key = static_call_key(site);
+
+		/*
+		 * If the key was not updated due to a memory allocation
+		 * failure in __static_call_init() then treating key::sites
+		 * as key::mods in the code below would cause random memory
+		 * access and #GP. In that case all subsequent sites have
+		 * not been touched either, so stop iterating.
+		 */
+		if (!static_call_key_has_mods(key))
+			break;
+
 		if (key == prev_key)
 			continue;