Skip to contentSkip to conversation

pmu_dev_alloc; warning at at kernel/locking/lockdep.c:3002 lockdep_init_map

DISCUSSION Tvrtko Ursulin 3 participants 7 messages May 7, 2015 – May 8, 2015

lore
Share & tools
lore
#0 lore raw

Hi,

4.1.0-rc2 spews the below warning and disables lockdep for me at boot.

Any ideas?

May  7 11:58:20 skl kernel: [    6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)
May  7 11:58:20 skl kernel: [    6.075774] BUG: key ffff88014973b850 not in .data!
May  7 11:58:20 skl kernel: [    6.081369] ------------[ cut here ]------------
May  7 11:58:20 skl kernel: [    6.086695] WARNING: CPU: 0 PID: 1 at kernel/locking/lockdep.c:3002 lockdep_init_map+0x14b/0x5e0()
May  7 11:58:20 skl kernel: [    6.096928] DEBUG_LOCKS_WARN_ON(1)
May  7 11:58:20 skl kernel: [    6.100659] Modules linked in:
May  7 11:58:20 skl kernel: [    6.104433] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.0-rc2-150507+ #3
May  7 11:58:20 skl kernel: [    6.112284] Hardware name: Intel Corporation Skylake Client platform/Skylake Y LPDDR3 RVP3, BIOS SKLSE2R1.86C.X070.R01.1501282110 01/28/2015
May  7 11:58:20 skl kernel: [    6.126689]  0000000000000bba ffff880149e1fb88 ffffffff816213f8 ffffffff8109fcc6
May  7 11:58:20 skl kernel: [    6.135381]  ffff880149e1fbd8 ffff880149e1fbc8 ffffffff810528e2 0000000000000000
May  7 11:58:20 skl kernel: [    6.143992]  ffff880146428858 0000000000000000 0000000000000000 ffff88014973b838
May  7 11:58:20 skl kernel: [    6.152682] Call Trace:
May  7 11:58:20 skl kernel: [    6.155513]  [<ffffffff816213f8>] dump_stack+0x4c/0x6e
May  7 11:58:20 skl kernel: [    6.161399]  [<ffffffff8109fcc6>] ? console_unlock+0x406/0x440
May  7 11:58:20 skl kernel: [    6.168075]  [<ffffffff810528e2>] warn_slowpath_common+0xb2/0xe0
May  7 11:58:20 skl kernel: [    6.174946]  [<ffffffff810529c6>] warn_slowpath_fmt+0x46/0x50
May  7 11:58:20 skl kernel: [    6.181525]  [<ffffffff81093b6b>] lockdep_init_map+0x14b/0x5e0
May  7 11:58:20 skl kernel: [    6.188202]  [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0
May  7 11:58:20 skl kernel: [    6.195069]  [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0
May  7 11:58:20 skl kernel: [    6.202346]  [<ffffffff811dd92a>] internal_create_group+0x19a/0x280
May  7 11:58:20 skl kernel: [    6.209515]  [<ffffffff811dda43>] sysfs_create_group+0x13/0x20
May  7 11:58:20 skl kernel: [    6.216185]  [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90
May  7 11:58:20 skl kernel: [    6.222958]  [<ffffffff8145c3d6>] device_add+0x276/0x5d0
May  7 11:58:20 skl kernel: [    6.229040]  [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0
May  7 11:58:20 skl kernel: [    6.235215]  [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61
May  7 11:58:20 skl kernel: [    6.242184]  [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95
May  7 11:58:20 skl kernel: [    6.249144]  [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61
May  7 11:58:20 skl kernel: [    6.256118]  [<ffffffff81000352>] do_one_initcall+0x122/0x1f0
May  7 11:58:20 skl kernel: [    6.262700]  [<ffffffff81072e00>] ? parse_args+0x240/0x450
May  7 11:58:20 skl kernel: [    6.268980]  [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a
May  7 11:58:20 skl kernel: [    6.276045]  [<ffffffff81d26878>] ? loglevel+0x31/0x31
May  7 11:58:20 skl kernel: [    6.281939]  [<ffffffff8161a380>] ? rest_init+0x140/0x140
May  7 11:58:20 skl kernel: [    6.288120]  [<ffffffff8161a38e>] kernel_init+0xe/0xf0
May  7 11:58:20 skl kernel: [    6.294004]  [<ffffffff81629e92>] ret_from_fork+0x42/0x70
May  7 11:58:20 skl kernel: [    6.300186]  [<ffffffff8161a380>] ? rest_init+0x140/0x140
May  7 11:58:20 skl kernel: [    6.306367] ---[ end trace 98f2b03275198a56 ]---
May  7 11:58:20 skl kernel: [    6.311703] BUG: key ffff88014973b888 not in .data!
May  7 11:58:20 skl kernel: [    6.317300] BUG: key ffff88014973b8c0 not in .data!
May  7 11:58:20 skl kernel: [    6.322895] BUG: key ffff88014973b8f8 not in .data!
May  7 11:58:20 skl kernel: [    6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages

Regards,

Tvrtko

P.S. Apologies if my attempt to find suitable people to copy wasn't that great!

#1 lore raw
Quoted 39 lines · Tvrtko Ursulin

Hi,

4.1.0-rc2 spews the below warning and disables lockdep for me at boot.

Any ideas?

May  7 11:58:20 skl kernel: [    6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)
May  7 11:58:20 skl kernel: [    6.075774] BUG: key ffff88014973b850 not in .data!
May  7 11:58:20 skl kernel: [    6.081369] ------------[ cut here ]------------
May  7 11:58:20 skl kernel: [    6.086695] WARNING: CPU: 0 PID: 1 at kernel/locking/lockdep.c:3002 lockdep_init_map+0x14b/0x5e0()
May  7 11:58:20 skl kernel: [    6.096928] DEBUG_LOCKS_WARN_ON(1)
May  7 11:58:20 skl kernel: [    6.100659] Modules linked in:
May  7 11:58:20 skl kernel: [    6.104433] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.0-rc2-150507+ #3
May  7 11:58:20 skl kernel: [    6.112284] Hardware name: Intel Corporation Skylake Client platform/Skylake Y LPDDR3 RVP3, BIOS SKLSE2R1.86C.X070.R01.1501282110 01/28/2015
May  7 11:58:20 skl kernel: [    6.126689]  0000000000000bba ffff880149e1fb88 ffffffff816213f8 ffffffff8109fcc6
May  7 11:58:20 skl kernel: [    6.135381]  ffff880149e1fbd8 ffff880149e1fbc8 ffffffff810528e2 0000000000000000
May  7 11:58:20 skl kernel: [    6.143992]  ffff880146428858 0000000000000000 0000000000000000 ffff88014973b838
May  7 11:58:20 skl kernel: [    6.152682] Call Trace:
May  7 11:58:20 skl kernel: [    6.155513]  [<ffffffff816213f8>] dump_stack+0x4c/0x6e
May  7 11:58:20 skl kernel: [    6.161399]  [<ffffffff8109fcc6>] ? console_unlock+0x406/0x440
May  7 11:58:20 skl kernel: [    6.168075]  [<ffffffff810528e2>] warn_slowpath_common+0xb2/0xe0
May  7 11:58:20 skl kernel: [    6.174946]  [<ffffffff810529c6>] warn_slowpath_fmt+0x46/0x50
May  7 11:58:20 skl kernel: [    6.181525]  [<ffffffff81093b6b>] lockdep_init_map+0x14b/0x5e0
May  7 11:58:20 skl kernel: [    6.188202]  [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0
May  7 11:58:20 skl kernel: [    6.195069]  [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0
May  7 11:58:20 skl kernel: [    6.202346]  [<ffffffff811dd92a>] internal_create_group+0x19a/0x280
May  7 11:58:20 skl kernel: [    6.209515]  [<ffffffff811dda43>] sysfs_create_group+0x13/0x20
May  7 11:58:20 skl kernel: [    6.216185]  [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90
May  7 11:58:20 skl kernel: [    6.222958]  [<ffffffff8145c3d6>] device_add+0x276/0x5d0
May  7 11:58:20 skl kernel: [    6.229040]  [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0
May  7 11:58:20 skl kernel: [    6.235215]  [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61
May  7 11:58:20 skl kernel: [    6.242184]  [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95
May  7 11:58:20 skl kernel: [    6.249144]  [<ffffffff81d44c93>] ? perf_workqueue_init+0x61/0x61
May  7 11:58:20 skl kernel: [    6.256118]  [<ffffffff81000352>] do_one_initcall+0x122/0x1f0
May  7 11:58:20 skl kernel: [    6.262700]  [<ffffffff81072e00>] ? parse_args+0x240/0x450
May  7 11:58:20 skl kernel: [    6.268980]  [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a
May  7 11:58:20 skl kernel: [    6.276045]  [<ffffffff81d26878>] ? loglevel+0x31/0x31
May  7 11:58:20 skl kernel: [    6.281939]  [<ffffffff8161a380>] ? rest_init+0x140/0x140
Quoted 8 lines
May  7 11:58:20 skl kernel: [    6.288120]  [<ffffffff8161a38e>] kernel_init+0xe/0xf0
May  7 11:58:20 skl kernel: [    6.294004]  [<ffffffff81629e92>] ret_from_fork+0x42/0x70
May  7 11:58:20 skl kernel: [    6.300186]  [<ffffffff8161a380>] ? rest_init+0x140/0x140
May  7 11:58:20 skl kernel: [    6.306367] ---[ end trace 98f2b03275198a56 ]---
May  7 11:58:20 skl kernel: [    6.311703] BUG: key ffff88014973b888 not in .data!
May  7 11:58:20 skl kernel: [    6.317300] BUG: key ffff88014973b8c0 not in .data!
May  7 11:58:20 skl kernel: [    6.322895] BUG: key ffff88014973b8f8 not in .data!
May  7 11:58:20 skl kernel: [    6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages

That's in sysfs magic mushroom land, lets Cc people who know about this.

#2 lore raw

Hello,

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Tvrtko Ursulin

Hi,

4.1.0-rc2 spews the below warning and disables lockdep for me at boot.

Any ideas?

May  7 11:58:20 skl kernel: [    6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)
May  7 11:58:20 skl kernel: [    6.075774] BUG: key ffff88014973b850 not in .data!

...

Quoted 18 lines
Quoted 9 lines
May  7 11:58:20 skl kernel: [    6.188202]  [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0
May  7 11:58:20 skl kernel: [    6.195069]  [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0
May  7 11:58:20 skl kernel: [    6.202346]  [<ffffffff811dd92a>] internal_create_group+0x19a/0x280
May  7 11:58:20 skl kernel: [    6.209515]  [<ffffffff811dda43>] sysfs_create_group+0x13/0x20
May  7 11:58:20 skl kernel: [    6.216185]  [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90
May  7 11:58:20 skl kernel: [    6.222958]  [<ffffffff8145c3d6>] device_add+0x276/0x5d0
May  7 11:58:20 skl kernel: [    6.229040]  [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0
May  7 11:58:20 skl kernel: [    6.242184]  [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95
May  7 11:58:20 skl kernel: [    6.256118]  [<ffffffff81000352>] do_one_initcall+0x122/0x1f0
Quoted 8 lines
May  7 11:58:20 skl kernel: [    6.268980]  [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a
May  7 11:58:20 skl kernel: [    6.288120]  [<ffffffff8161a38e>] kernel_init+0xe/0xf0
May  7 11:58:20 skl kernel: [    6.294004]  [<ffffffff81629e92>] ret_from_fork+0x42/0x70
May  7 11:58:20 skl kernel: [    6.306367] ---[ end trace 98f2b03275198a56 ]---
May  7 11:58:20 skl kernel: [    6.311703] BUG: key ffff88014973b888 not in .data!
May  7 11:58:20 skl kernel: [    6.317300] BUG: key ffff88014973b8c0 not in .data!
May  7 11:58:20 skl kernel: [    6.322895] BUG: key ffff88014973b8f8 not in .data!
May  7 11:58:20 skl kernel: [    6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages
Quoted 1 line

That's in sysfs magic mushroom land, lets Cc people who know about this.

sysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?

Thanks.

Signature · 1 line
tejun
#3 lore raw
Quoted 32 lines · Tejun Heo

Hello,

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Tvrtko Ursulin

Hi,

4.1.0-rc2 spews the below warning and disables lockdep for me at boot.

Any ideas?

May  7 11:58:20 skl kernel: [    6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)
May  7 11:58:20 skl kernel: [    6.075774] BUG: key ffff88014973b850 not in .data!

...

Quoted 18 lines
Quoted 17 lines
May  7 11:58:20 skl kernel: [    6.188202]  [<ffffffff811dc06a>] __kernfs_create_file+0x7a/0xf0
May  7 11:58:20 skl kernel: [    6.195069]  [<ffffffff811dcb4e>] sysfs_add_file_mode_ns+0x16e/0x1b0
May  7 11:58:20 skl kernel: [    6.202346]  [<ffffffff811dd92a>] internal_create_group+0x19a/0x280
May  7 11:58:20 skl kernel: [    6.209515]  [<ffffffff811dda43>] sysfs_create_group+0x13/0x20
May  7 11:58:20 skl kernel: [    6.216185]  [<ffffffff811dda88>] sysfs_create_groups+0x38/0x90
May  7 11:58:20 skl kernel: [    6.222958]  [<ffffffff8145c3d6>] device_add+0x276/0x5d0
May  7 11:58:20 skl kernel: [    6.229040]  [<ffffffff8110a80b>] pmu_dev_alloc+0x8b/0xd0
May  7 11:58:20 skl kernel: [    6.242184]  [<ffffffff81d44cd9>] perf_event_sysfs_init+0x46/0x95
May  7 11:58:20 skl kernel: [    6.256118]  [<ffffffff81000352>] do_one_initcall+0x122/0x1f0
May  7 11:58:20 skl kernel: [    6.268980]  [<ffffffff81d27126>] kernel_init_freeable+0x106/0x19a
May  7 11:58:20 skl kernel: [    6.288120]  [<ffffffff8161a38e>] kernel_init+0xe/0xf0
May  7 11:58:20 skl kernel: [    6.294004]  [<ffffffff81629e92>] ret_from_fork+0x42/0x70
May  7 11:58:20 skl kernel: [    6.306367] ---[ end trace 98f2b03275198a56 ]---
May  7 11:58:20 skl kernel: [    6.311703] BUG: key ffff88014973b888 not in .data!
May  7 11:58:20 skl kernel: [    6.317300] BUG: key ffff88014973b8c0 not in .data!
May  7 11:58:20 skl kernel: [    6.322895] BUG: key ffff88014973b8f8 not in .data!
May  7 11:58:20 skl kernel: [    6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages
Quoted 5 lines
Quoted 1 line

That's in sysfs magic mushroom land, lets Cc people who know about this.

sysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?

I am not familiar with this area, is this the answer?

May  7 11:58:20 skl kernel: [    0.849877] Performance Events: no PEBS fmt3+, generic architected perfmon, full-width counters, Intel PMU driver.
May  7 11:58:20 skl kernel: [    0.861827] ... version:                4
May  7 11:58:20 skl kernel: [    0.866406] ... bit width:              48
May  7 11:58:20 skl kernel: [    0.871083] ... generic registers:      4
May  7 11:58:20 skl kernel: [    0.875664] ... value mask:             0000ffffffffffff
May  7 11:58:20 skl kernel: [    0.881724] ... max period:             0000ffffffffffff
May  7 11:58:20 skl kernel: [    0.887778] ... fixed-purpose events:   3
May  7 11:58:20 skl kernel: [    0.892352] ... event mask:             000000070000000f
May  7 11:58:20 skl kernel: [    0.909665] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.

Regards,

Tvrtko

#4 lore raw
Quoted 10 lines · Tvrtko Ursulin
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
May  7 11:58:20 skl kernel: [    6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)
May  7 11:58:20 skl kernel: [    6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages

sysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?

I am not familiar with this area, is this the answer?

Nope, that's way before you get the fail. initcall_debug might get you the right answer, but I suspect its the intel_uncore driver.

I see arch/x86/kernel/cpu/perf_event_intel_uncore.c:uncore_type_init() do exactly what TJ says, its dynamically allocating struct attribute.

Now, let me try and trigger that locally, that should not be skylake specific at all.

#5 lore raw
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Tvrtko Ursulin
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
May  7 11:58:20 skl kernel: [    6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)
May  7 11:58:20 skl kernel: [    6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages

sysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?

I am not familiar with this area, is this the answer?

Quoted 8 lines

Nope, that's way before you get the fail. initcall_debug might get you the right answer, but I suspect its the intel_uncore driver.

I see arch/x86/kernel/cpu/perf_event_intel_uncore.c:uncore_type_init() do exactly what TJ says, its dynamically allocating struct attribute.

Now, let me try and trigger that locally, that should not be skylake specific at all.

4.1-rc2+ with lockdep enabled boots without issue on my ivb-ep, no idea what's going on on your skylake thingy.

#6 lore raw
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 10 lines · Tvrtko Ursulin
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
May  7 11:58:20 skl kernel: [    6.066696] futex hash table entries: 1024 (order: 5, 131072 bytes)
May  7 11:58:20 skl kernel: [    6.329549] HugeTLB registered 2 MB page size, pre-allocated 0 pages

sysfs by default uses lockdep key embedded in attrs. It looks like somebody is making on-heap copies of attrs and using them as the group attrs. What's the offending pmu type?

I am not familiar with this area, is this the answer?

Nope, that's way before you get the fail. initcall_debug might get you the right answer, but I suspect its the intel_uncore driver.

Quoted 8 lines
Quoted 5 lines

I see arch/x86/kernel/cpu/perf_event_intel_uncore.c:uncore_type_init() do exactly what TJ says, its dynamically allocating struct attribute.

Now, let me try and trigger that locally, that should not be skylake specific at all.

4.1-rc2+ with lockdep enabled boots without issue on my ivb-ep, no idea what's going on on your skylake thingy.

Managed to debug it is the intel_pt driver which fails.

Which if I see correctly is passing in attr_groups from the data section which should be OK. So I don't know either.. reported offending address is somewhere, terabytes away, before start of kernel.

Regards,

Tvrtko