pmu_dev_alloc; warning at at kernel/locking/lockdep.c:3002 lockdep_init_map
Hi,
4.1.0-rc2 spews the below warning and disables lockdep for me at boot.
Any ideas?
May 7 11:58:20 skl kernel: [ 6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes) May 7 11:58:20 skl kernel: [ 6.075774] BUG: key ffff88014973b850 not in .data! May 7 11:58:20 skl kernel: [ 6.081369] ------------[ cut here ]------------ May 7 11:58:20 skl kernel: [ 6.086695] WARNING: CPU: 0 PID: 1 at kernel/locking/lockdep.c:3002 lockdep_init_map+0x14b/0x5e0() May 7 11:58:20 skl kernel: [ 6.096928] DEBUG_LOCKS_WARN_ON(1) May 7 11:58:20 skl kernel: [ 6.100659] Modules linked in: May 7 11:58:20 skl kernel: [ 6.104433] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.0-rc2-150507+ #3 May 7 11:58:20 skl kernel: [ 6.112284] Hardware name: Intel Corporation Skylake Client platform/Skylake Y LPDDR3 RVP3, BIOS SKLSE2R1.86C.X070.R01.1501282110 01/28/2015 May 7 11:58:20 skl kernel: [ 6.126689] 0000000000000bba ffff880149e1fb88 ffffffff816213f8 ffffffff8109fcc6 May 7 11:58:20 skl kernel: [ 6.135381] ffff880149e1fbd8 ffff880149e1fbc8 ffffffff810528e2 0000000000000000 May 7 11:58:20 skl kernel: [ 6.143992] ffff880146428858 0000000000000000 0000000000000000 ffff88014973b838 May 7 11:58:20 skl kernel: [ 6.152682] Call Trace: May 7 11:58:20 skl kernel: [ 6.155513] [<ffffffff816213f8>] dump_stack+0x4c/0x6e May 7 11:58:20 skl kernel: [ 6.161399] [<ffffffff8109fcc6>] ? console_unlock+0x406/0x440 May 7 11:58:20 skl kernel: [ 6.168075] [<ffffffff810528e2>] warn_slowpath_common+0xb2/0xe0 May 7 11:58:20 skl kernel: [ 6.174946] [<ffffffff810529c6>] warn_slowpath_fmt+0x46/0x50 May 7 11:58:20 skl kernel: [ 6.181525] [<ffffffff81093b6b>] lockdep_init_map+0x14b/0x5e0 May 7 11:58:20 skl kernel: [ 6.188202] [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0 May 7 11:58:20 skl kernel: [ 6.195069] [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0 May 7 11:58:20 skl kernel: [ 6.202346] [<ffffffff811dd92a>] internal_create_group+0x19a/0x280 May 7 11:58:20 skl kernel: [ 6.209515] [<ffffffff811dda43>] sysfs_create_group+0x13/0x20 May 7 11:58:20 skl kernel: [ 6.216185] [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90 May 7 11:58:20 skl kernel: [ 6.222958] [<ffffffff8145c3d6>] device_add+0x276/0x5d0 May 7 11:58:20 skl kernel: [ 6.229040] [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0 May 7 11:58:20 skl kernel: [ 6.235215] [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61 May 7 11:58:20 skl kernel: [ 6.242184] [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95 May 7 11:58:20 skl kernel: [ 6.249144] [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61 May 7 11:58:20 skl kernel: [ 6.256118] [<ffffffff81000352>] do_one_initcall+0x122/0x1f0 May 7 11:58:20 skl kernel: [ 6.262700] [<ffffffff81072e00>] ? parse_args+0x240/0x450 May 7 11:58:20 skl kernel: [ 6.268980] [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a May 7 11:58:20 skl kernel: [ 6.276045] [<ffffffff81d26878>] ? loglevel+0x31/0x31 May 7 11:58:20 skl kernel: [ 6.281939] [<ffffffff8161a380>] ? rest_init+0x140/0x140 May 7 11:58:20 skl kernel: [ 6.288120] [<ffffffff8161a38e>] kernel_init+0xe/0xf0 May 7 11:58:20 skl kernel: [ 6.294004] [<ffffffff81629e92>] ret_from_fork+0x42/0x70 May 7 11:58:20 skl kernel: [ 6.300186] [<ffffffff8161a380>] ? rest_init+0x140/0x140 May 7 11:58:20 skl kernel: [ 6.306367] ---[ end trace 98f2b03275198a56 ]--- May 7 11:58:20 skl kernel: [ 6.311703] BUG: key ffff88014973b888 not in .data! May 7 11:58:20 skl kernel: [ 6.317300] BUG: key ffff88014973b8c0 not in .data! May 7 11:58:20 skl kernel: [ 6.322895] BUG: key ffff88014973b8f8 not in .data! May 7 11:58:20 skl kernel: [ 6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages
Regards,
Tvrtko
P.S. Apologies if my attempt to find suitable people to copy wasn't that great!
Quoted 39 lines · Tvrtko Ursulin
Hi,
4.1.0-rc2 spews the below warning and disables lockdep for me at boot.
Any ideas?
May 7 11:58:20 skl kernel: [ 6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes) May 7 11:58:20 skl kernel: [ 6.075774] BUG: key ffff88014973b850 not in .data! May 7 11:58:20 skl kernel: [ 6.081369] ------------[ cut here ]------------ May 7 11:58:20 skl kernel: [ 6.086695] WARNING: CPU: 0 PID: 1 at kernel/locking/lockdep.c:3002 lockdep_init_map+0x14b/0x5e0() May 7 11:58:20 skl kernel: [ 6.096928] DEBUG_LOCKS_WARN_ON(1) May 7 11:58:20 skl kernel: [ 6.100659] Modules linked in: May 7 11:58:20 skl kernel: [ 6.104433] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.0-rc2-150507+ #3 May 7 11:58:20 skl kernel: [ 6.112284] Hardware name: Intel Corporation Skylake Client platform/Skylake Y LPDDR3 RVP3, BIOS SKLSE2R1.86C.X070.R01.1501282110 01/28/2015 May 7 11:58:20 skl kernel: [ 6.126689] 0000000000000bba ffff880149e1fb88 ffffffff816213f8 ffffffff8109fcc6 May 7 11:58:20 skl kernel: [ 6.135381] ffff880149e1fbd8 ffff880149e1fbc8 ffffffff810528e2 0000000000000000 May 7 11:58:20 skl kernel: [ 6.143992] ffff880146428858 0000000000000000 0000000000000000 ffff88014973b838 May 7 11:58:20 skl kernel: [ 6.152682] Call Trace: May 7 11:58:20 skl kernel: [ 6.155513] [<ffffffff816213f8>] dump_stack+0x4c/0x6e May 7 11:58:20 skl kernel: [ 6.161399] [<ffffffff8109fcc6>] ? console_unlock+0x406/0x440 May 7 11:58:20 skl kernel: [ 6.168075] [<ffffffff810528e2>] warn_slowpath_common+0xb2/0xe0 May 7 11:58:20 skl kernel: [ 6.174946] [<ffffffff810529c6>] warn_slowpath_fmt+0x46/0x50 May 7 11:58:20 skl kernel: [ 6.181525] [<ffffffff81093b6b>] lockdep_init_map+0x14b/0x5e0 May 7 11:58:20 skl kernel: [ 6.188202] [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0 May 7 11:58:20 skl kernel: [ 6.195069] [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0 May 7 11:58:20 skl kernel: [ 6.202346] [<ffffffff811dd92a>] internal_create_group+0x19a/0x280 May 7 11:58:20 skl kernel: [ 6.209515] [<ffffffff811dda43>] sysfs_create_group+0x13/0x20 May 7 11:58:20 skl kernel: [ 6.216185] [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90 May 7 11:58:20 skl kernel: [ 6.222958] [<ffffffff8145c3d6>] device_add+0x276/0x5d0 May 7 11:58:20 skl kernel: [ 6.229040] [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0 May 7 11:58:20 skl kernel: [ 6.235215] [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61 May 7 11:58:20 skl kernel: [ 6.242184] [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95 May 7 11:58:20 skl kernel: [ 6.249144] [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61 May 7 11:58:20 skl kernel: [ 6.256118] [<ffffffff81000352>] do_one_initcall+0x122/0x1f0 May 7 11:58:20 skl kernel: [ 6.262700] [<ffffffff81072e00>] ? parse_args+0x240/0x450 May 7 11:58:20 skl kernel: [ 6.268980] [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a May 7 11:58:20 skl kernel: [ 6.276045] [<ffffffff81d26878>] ? loglevel+0x31/0x31 May 7 11:58:20 skl kernel: [ 6.281939] [<ffffffff8161a380>] ? rest_init+0x140/0x140
Quoted 8 lines
May 7 11:58:20 skl kernel: [ 6.288120] [<ffffffff8161a38e>] kernel_init+0xe/0xf0 May 7 11:58:20 skl kernel: [ 6.294004] [<ffffffff81629e92>] ret_from_fork+0x42/0x70 May 7 11:58:20 skl kernel: [ 6.300186] [<ffffffff8161a380>] ? rest_init+0x140/0x140 May 7 11:58:20 skl kernel: [ 6.306367] ---[ end trace 98f2b03275198a56 ]--- May 7 11:58:20 skl kernel: [ 6.311703] BUG: key ffff88014973b888 not in .data! May 7 11:58:20 skl kernel: [ 6.317300] BUG: key ffff88014973b8c0 not in .data! May 7 11:58:20 skl kernel: [ 6.322895] BUG: key ffff88014973b8f8 not in .data! May 7 11:58:20 skl kernel: [ 6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages
That's in sysfs magic mushroom land, lets Cc people who know about this.
Hello,
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Tvrtko Ursulin
Hi,
4.1.0-rc2 spews the below warning and disables lockdep for me at boot.
Any ideas?
May 7 11:58:20 skl kernel: [ 6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes) May 7 11:58:20 skl kernel: [ 6.075774] BUG: key ffff88014973b850 not in .data!
...
Quoted 18 lines
Quoted 9 lines
May 7 11:58:20 skl kernel: [ 6.188202] [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0 May 7 11:58:20 skl kernel: [ 6.195069] [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0 May 7 11:58:20 skl kernel: [ 6.202346] [<ffffffff811dd92a>] internal_create_group+0x19a/0x280 May 7 11:58:20 skl kernel: [ 6.209515] [<ffffffff811dda43>] sysfs_create_group+0x13/0x20 May 7 11:58:20 skl kernel: [ 6.216185] [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90 May 7 11:58:20 skl kernel: [ 6.222958] [<ffffffff8145c3d6>] device_add+0x276/0x5d0 May 7 11:58:20 skl kernel: [ 6.229040] [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0 May 7 11:58:20 skl kernel: [ 6.242184] [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95 May 7 11:58:20 skl kernel: [ 6.256118] [<ffffffff81000352>] do_one_initcall+0x122/0x1f0Quoted 8 lines
May 7 11:58:20 skl kernel: [ 6.268980] [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a May 7 11:58:20 skl kernel: [ 6.288120] [<ffffffff8161a38e>] kernel_init+0xe/0xf0 May 7 11:58:20 skl kernel: [ 6.294004] [<ffffffff81629e92>] ret_from_fork+0x42/0x70 May 7 11:58:20 skl kernel: [ 6.306367] ---[ end trace 98f2b03275198a56 ]--- May 7 11:58:20 skl kernel: [ 6.311703] BUG: key ffff88014973b888 not in .data! May 7 11:58:20 skl kernel: [ 6.317300] BUG: key ffff88014973b8c0 not in .data! May 7 11:58:20 skl kernel: [ 6.322895] BUG: key ffff88014973b8f8 not in .data! May 7 11:58:20 skl kernel: [ 6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages
Quoted 1 line
That's in sysfs magic mushroom land, lets Cc people who know about this.
sysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?
Thanks.
Signature · 1 line
tejun
Quoted 32 lines · Tejun Heo
Hello,
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Tvrtko Ursulin
Hi,
4.1.0-rc2 spews the below warning and disables lockdep for me at boot.
Any ideas?
May 7 11:58:20 skl kernel: [ 6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes) May 7 11:58:20 skl kernel: [ 6.075774] BUG: key ffff88014973b850 not in .data!...
Quoted 18 lines
Quoted 17 lines
May 7 11:58:20 skl kernel: [ 6.188202] [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0 May 7 11:58:20 skl kernel: [ 6.195069] [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0 May 7 11:58:20 skl kernel: [ 6.202346] [<ffffffff811dd92a>] internal_create_group+0x19a/0x280 May 7 11:58:20 skl kernel: [ 6.209515] [<ffffffff811dda43>] sysfs_create_group+0x13/0x20 May 7 11:58:20 skl kernel: [ 6.216185] [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90 May 7 11:58:20 skl kernel: [ 6.222958] [<ffffffff8145c3d6>] device_add+0x276/0x5d0 May 7 11:58:20 skl kernel: [ 6.229040] [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0 May 7 11:58:20 skl kernel: [ 6.242184] [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95 May 7 11:58:20 skl kernel: [ 6.256118] [<ffffffff81000352>] do_one_initcall+0x122/0x1f0 May 7 11:58:20 skl kernel: [ 6.268980] [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a May 7 11:58:20 skl kernel: [ 6.288120] [<ffffffff8161a38e>] kernel_init+0xe/0xf0 May 7 11:58:20 skl kernel: [ 6.294004] [<ffffffff81629e92>] ret_from_fork+0x42/0x70 May 7 11:58:20 skl kernel: [ 6.306367] ---[ end trace 98f2b03275198a56 ]--- May 7 11:58:20 skl kernel: [ 6.311703] BUG: key ffff88014973b888 not in .data! May 7 11:58:20 skl kernel: [ 6.317300] BUG: key ffff88014973b8c0 not in .data! May 7 11:58:20 skl kernel: [ 6.322895] BUG: key ffff88014973b8f8 not in .data! May 7 11:58:20 skl kernel: [ 6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages
Quoted 5 lines
Quoted 1 line
That's in sysfs magic mushroom land, lets Cc people who know about this.
sysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?
I am not familiar with this area, is this the answer?
May 7 11:58:20 skl kernel: [ 0.849877] Performance Events: no PEBS fmt3+, generic architected perfmon, full-width counters, Intel PMU driver. May 7 11:58:20 skl kernel: [ 0.861827] ... version: 4 May 7 11:58:20 skl kernel: [ 0.866406] ... bit width: 48 May 7 11:58:20 skl kernel: [ 0.871083] ... generic registers: 4 May 7 11:58:20 skl kernel: [ 0.875664] ... value mask: 0000ffffffffffff May 7 11:58:20 skl kernel: [ 0.881724] ... max period: 0000ffffffffffff May 7 11:58:20 skl kernel: [ 0.887778] ... fixed-purpose events: 3 May 7 11:58:20 skl kernel: [ 0.892352] ... event mask: 000000070000000f May 7 11:58:20 skl kernel: [ 0.909665] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
Regards,
Tvrtko
Quoted 10 lines · Tvrtko Ursulin
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
May 7 11:58:20 skl kernel: [ 6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)May 7 11:58:20 skl kernel: [ 6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pagessysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?
I am not familiar with this area, is this the answer?
Nope, that's way before you get the fail. initcall_debug might get you the right answer, but I suspect its the intel_uncore driver.
I see arch/x86/kernel/cpu/perf_event_intel_uncore.c:uncore_type_init() do exactly what TJ says, its dynamically allocating struct attribute.
Now, let me try and trigger that locally, that should not be skylake specific at all.
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Tvrtko Ursulin
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
May 7 11:58:20 skl kernel: [ 6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)May 7 11:58:20 skl kernel: [ 6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pagessysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?
I am not familiar with this area, is this the answer?
Quoted 8 lines
Nope, that's way before you get the fail.
initcall_debugmight get you the right answer, but I suspect its theintel_uncoredriver.I see arch/x86/kernel/cpu/perf_event_intel_uncore.c:uncore_type_init() do exactly what TJ says, its dynamically allocating struct attribute.
Now, let me try and trigger that locally, that should not be skylake specific at all.
4.1-rc2+ with lockdep enabled boots without issue on my ivb-ep, no idea what's going on on your skylake thingy.
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 10 lines · Tvrtko Ursulin
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
May 7 11:58:20 skl kernel: [ 6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)May 7 11:58:20 skl kernel: [ 6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pagessysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?
I am not familiar with this area, is this the answer?
Nope, that's way before you get the fail.
initcall_debugmight get you the right answer, but I suspect its theintel_uncoredriver.
Quoted 8 lines
Quoted 5 lines
I see arch/x86/kernel/cpu/perf_event_intel_uncore.c:uncore_type_init() do exactly what TJ says, its dynamically allocating struct attribute.
Now, let me try and trigger that locally, that should not be skylake specific at all.
4.1-rc2+ with lockdep enabled boots without issue on my ivb-ep, no idea what's going on on your skylake thingy.
Managed to debug it is the intel_pt driver which fails.
Which if I see correctly is passing in attr_groups from the data section which should be OK. So I don't know either.. reported offending address is somewhere, terabytes away, before start of kernel.
Regards,
Tvrtko