kernel oops and panic in acpi_atomic_read under 2.6.39.3. call trace included

Hi all,

I am getting a kernel oops/panic on a dual xeon system that is the master of a 60 node HPC cluster. This is happening while stress testing the system including significant network traffic. The OS is openSuse 11.4.

We are running a custom compiled 2.6.39.3 kernel on the systems due to a bug in the stock kernel 11.4 provided (igb driver related). After 1-3 days of heavy testing, the master node locks up with the caps lock and scroll lock keys on the keyboard blinking with the following output captured via a serial console:

[381920.681113] BUG: unable to handle kernel NULL pointer dereference at           
(null)
[381920.689067] IP: [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381920.695187] PGD 30c27a067 PUD 16efe6067 PMD 0 
[381920.699782] Oops: 0000 [#1] PREEMPT SMP 
[381920.703866] last sysfs file: 
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
[381920.711868] CPU 6 
[381920.713800] Modules linked in: md5 ipmi_devintf ipmi_si ipmi_msghandler 

nfsd lockd nfs_acl auth_rpcgss sunrpc ipt_MASQUERADE iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables af_packet edd cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf xfs dm_mod ioatdma i7core_edac edac_core sr_mod cdrom joydev igb i2c_i801 sg button ghes hed iTCO_wdt iTCO_vendor_support dca pcspkr ext4 jbd2 crc16 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 fan processor thermal thermal_sys ata_generic pata_atiixp arcmsr

[381920.771623] 
[381920.773210] Pid: 12701, comm: cluster Not tainted 2.6.39.3-microwaycustom 
#1 Supermicro X8DTH-i/6/iF/6F/X8DTH
[381920.783292] RIP: 0010:[<ffffffff812a7510>]  [<ffffffff812a7510>] 
acpi_atomic_read+0xe3/0x120
[381920.791853] RSP: 0000:ffff88063fc47d98  EFLAGS: 00010046
[381920.797260] RAX: 0000000000000000 RBX: 00000000bf7b5ff0 RCX: ffffffff81a3cdd0
[381920.804486] RDX: 00000000bf7b6010 RSI: 00000000bf7b6000 RDI: 
ffff88062d4b95c0
[381920.811712] RBP: ffff88063fc47dc8 R08: ffff88063fc47d98 R09: 0000000000000002
[381920.818940] R10: 0000000000000083 R11: 0000000000000010 R12: 
ffffc90003044c20
[381920.826168] R13: ffff88063fc47de0 R14: 0000000000000000 R15: 
0000000000000000
[381920.833392] FS:  0000000000000000(0000) GS:ffff88063fc40000(0000) 
knlGS:0000000000000000
[381920.841571] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[381920.847409] CR2: 0000000000000000 CR3: 00000001d2d0f000 CR4: 
00000000000006e0
[381920.854635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[381920.861861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[381920.869088] Process cluster (pid: 12701, threadinfo ffff880028284000, task 
ffff880550184380)
[381920.877608] Stack:
[381920.879717]  ffffffff81a3cdd0 00000000bf7b5ff0 ffff88062b2cc140 ffffc90003044ca8
[381920.887288]  ffff88062b2cc140 0000000000000001 ffff88063fc47e08 ffffffffa002b21f
[381920.894862]  0000000000000000 0000000000000000 ffff88062b2cc140 
0000000000000000
[381920.902435] Call Trace:
[381920.904969]  <NMI> 
[381920.907201]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170 [ghes]
[381920.913726]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0 [ghes]
[381920.920076]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381920.926080]  [<ffffffff8151a1d8>] __atomic_notifier_call_chain+0x48/0x70
[381920.932864]  [<ffffffff8151a211>] atomic_notifier_call_chain+0x11/0x20
[381920.939473]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381920.944691]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381920.949649]  [<ffffffff81517550>] nmi+0x20/0x30
[381920.954269]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10
[381920.960536]  <<EOE>> 
[381920.962725]  <IRQ> 
[381920.965132]  [<ffffffff8101131e>] intel_pmu_disable_all+0x3e/0x120
[381920.971399]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381920.977066]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40
[381920.982817]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381920.989082]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381920.994740]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.000832]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.006578]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.012758]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.018414]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.024343]  [<ffffffff810206d7>] smp_apic_timer_interrupt+0x67/0xa0
[381921.030793]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.036891]  <EOI> 
[381921.038913] Code: fc 10 74 1f 77 08 41 80 fc 08 75 48 eb 0e 41 80 fc 20 74 
17 41 80 fc 40 75 3a eb 15 8a 00 0f b6 c0 eb 11 66 8b 00 0f b7 c0 eb 09 <8b> 
00 89 c0 eb 03 48 8b 00 49 89 45 00 e8 8e 2b e2 ff eb 1b 0f 
[381921.059462] RIP  [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381921.065669]  RSP <ffff88063fc47d98>
[381921.069242] CR2: 0000000000000000
[381921.072645] ---[ end trace 52697bfc73a34a90 ]---
[381921.077343] Kernel panic - not syncing: Fatal exception in interrupt
[381921.083784] Pid: 12701, comm: cluster Tainted: G      D     2.6.39.3-
microwaycustom #1
[381921.091788] Call Trace:
[381921.094333]  <NMI>  [<ffffffff81513300>] panic+0x9f/0x1da
[381921.099881]  [<ffffffff81517f8c>] oops_end+0xdc/0xf0
[381921.104952]  [<ffffffff81032b91>] no_context+0xf1/0x260
[381921.110277]  [<ffffffff81032e55>] __bad_area_nosemaphore+0x155/0x200
[381921.116739]  [<ffffffff81032f0e>] bad_area_nosemaphore+0xe/0x10
[381921.122766]  [<ffffffff81519f46>] do_page_fault+0x366/0x530
[381921.128440]  [<ffffffff810ee929>] ? __perf_event_overflow+0xa9/0x220
[381921.134896]  [<ffffffff810ef99b>] ? perf_event_update_userpage+0x9b/0xe0
[381921.141698]  [<ffffffff81012249>] ? intel_pmu_enable_all+0xc9/0x1a0
[381921.148061]  [<ffffffff810123ff>] ? x86_perf_event_set_period+0xdf/0x170
[381921.154852]  [<ffffffff81517295>] page_fault+0x25/0x30
[381921.160081]  [<ffffffff812a7510>] ? acpi_atomic_read+0xe3/0x120
[381921.166094]  [<ffffffff812a7486>] ? acpi_atomic_read+0x59/0x120
[381921.172107]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170 [ghes]
[381921.178639]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0 [ghes]
[381921.185004]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381921.191003]  [<ffffffff8151a1d8>] __atomic_notifier_call_chain+0x48/0x70
[381921.197789]  [<ffffffff8151a211>] atomic_notifier_call_chain+0x11/0x20
[381921.204399]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381921.209624]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381921.214584]  [<ffffffff81517550>] nmi+0x20/0x30
[381921.219208]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10
[381921.225470]  <<EOE>>  <IRQ>  [<ffffffff8101131e>] 
intel_pmu_disable_all+0x3e/0x120
[381921.233174]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381921.238836]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40
[381921.244582]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381921.250855]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381921.256520]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.262618]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.268363]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.274540]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.280198]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.286125]  [<ffffffff810206d7>] smp_apic_timer_interrupt+0x67/0xa0
[381921.292578]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.298673]  <EOI>

I recompiled the kernel again, disabling the tickless feature as I saw "nohz" early in the call trace. After that, it reproduced again but the call trace only changed slightly having tick_init_highres in there instead of tick_nohz_handler. I have that call trace if it is desired as well. It is nearly identical though.

I am currently trying a custom 2.6.36.4 on the system, but will need up to 3 days before I know if the problem exists there as well.

Any ideas on this?

Hi,

It's better to report ACPI failues to linux-acpi.

Quoted 139 lines · Rick Warner

Hi all,

I am getting a kernel oops/panic on a dual xeon system that is the master of a 60 node HPC cluster. This is happening while stress testing the system including significant network traffic. The OS is openSuse 11.4.

We are running a custom compiled 2.6.39.3 kernel on the systems due to a bug in the stock kernel 11.4 provided (igb driver related). After 1-3 days of heavy testing, the master node locks up with the caps lock and scroll lock keys on the keyboard blinking with the following output captured via a serial console:

[381920.681113] BUG: unable to handle kernel NULL pointer dereference at           
(null)
[381920.689067] IP: [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381920.695187] PGD 30c27a067 PUD 16efe6067 PMD 0 
[381920.699782] Oops: 0000 [#1] PREEMPT SMP 
[381920.703866] last sysfs file: 
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
[381920.711868] CPU 6 
[381920.713800] Modules linked in: md5 ipmi_devintf ipmi_si ipmi_msghandler 

nfsd lockd nfs_acl auth_rpcgss sunrpc ipt_MASQUERADE iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables af_packet edd cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf xfs dm_mod ioatdma i7core_edac edac_core sr_mod cdrom joydev igb i2c_i801 sg button ghes hed iTCO_wdt iTCO_vendor_support dca pcspkr ext4 jbd2 crc16 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1 raid0 fan processor thermal thermal_sys ata_generic pata_atiixp arcmsr

[381920.771623] 
[381920.773210] Pid: 12701, comm: cluster Not tainted 2.6.39.3-microwaycustom 
#1 Supermicro X8DTH-i/6/iF/6F/X8DTH
[381920.783292] RIP: 0010:[<ffffffff812a7510>]  [<ffffffff812a7510>] 
acpi_atomic_read+0xe3/0x120
[381920.791853] RSP: 0000:ffff88063fc47d98  EFLAGS: 00010046
[381920.797260] RAX: 0000000000000000 RBX: 00000000bf7b5ff0 RCX: ffffffff81a3cdd0
[381920.804486] RDX: 00000000bf7b6010 RSI: 00000000bf7b6000 RDI: 
ffff88062d4b95c0
[381920.811712] RBP: ffff88063fc47dc8 R08: ffff88063fc47d98 R09: 0000000000000002
[381920.818940] R10: 0000000000000083 R11: 0000000000000010 R12: 
ffffc90003044c20
[381920.826168] R13: ffff88063fc47de0 R14: 0000000000000000 R15: 
0000000000000000
[381920.833392] FS:  0000000000000000(0000) GS:ffff88063fc40000(0000) 
knlGS:0000000000000000
[381920.841571] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[381920.847409] CR2: 0000000000000000 CR3: 00000001d2d0f000 CR4: 
00000000000006e0
[381920.854635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[381920.861861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[381920.869088] Process cluster (pid: 12701, threadinfo ffff880028284000, task 
ffff880550184380)
[381920.877608] Stack:
[381920.879717]  ffffffff81a3cdd0 00000000bf7b5ff0 ffff88062b2cc140 ffffc90003044ca8
[381920.887288]  ffff88062b2cc140 0000000000000001 ffff88063fc47e08 ffffffffa002b21f
[381920.894862]  0000000000000000 0000000000000000 ffff88062b2cc140 
0000000000000000
[381920.902435] Call Trace:
[381920.904969]  <NMI> 
[381920.907201]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170 [ghes]
[381920.913726]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0 [ghes]
[381920.920076]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381920.926080]  [<ffffffff8151a1d8>] __atomic_notifier_call_chain+0x48/0x70
[381920.932864]  [<ffffffff8151a211>] atomic_notifier_call_chain+0x11/0x20
[381920.939473]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381920.944691]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381920.949649]  [<ffffffff81517550>] nmi+0x20/0x30
[381920.954269]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10
[381920.960536]  <<EOE>> 
[381920.962725]  <IRQ> 
[381920.965132]  [<ffffffff8101131e>] intel_pmu_disable_all+0x3e/0x120
[381920.971399]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381920.977066]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40
[381920.982817]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381920.989082]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381920.994740]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.000832]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.006578]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.012758]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.018414]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.024343]  [<ffffffff810206d7>] smp_apic_timer_interrupt+0x67/0xa0
[381921.030793]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.036891]  <EOI> 
[381921.038913] Code: fc 10 74 1f 77 08 41 80 fc 08 75 48 eb 0e 41 80 fc 20 74 
17 41 80 fc 40 75 3a eb 15 8a 00 0f b6 c0 eb 11 66 8b 00 0f b7 c0 eb 09 <8b> 
00 89 c0 eb 03 48 8b 00 49 89 45 00 e8 8e 2b e2 ff eb 1b 0f 
[381921.059462] RIP  [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381921.065669]  RSP <ffff88063fc47d98>
[381921.069242] CR2: 0000000000000000
[381921.072645] ---[ end trace 52697bfc73a34a90 ]---
[381921.077343] Kernel panic - not syncing: Fatal exception in interrupt
[381921.083784] Pid: 12701, comm: cluster Tainted: G      D     2.6.39.3-
microwaycustom #1
[381921.091788] Call Trace:
[381921.094333]  <NMI>  [<ffffffff81513300>] panic+0x9f/0x1da
[381921.099881]  [<ffffffff81517f8c>] oops_end+0xdc/0xf0
[381921.104952]  [<ffffffff81032b91>] no_context+0xf1/0x260
[381921.110277]  [<ffffffff81032e55>] __bad_area_nosemaphore+0x155/0x200
[381921.116739]  [<ffffffff81032f0e>] bad_area_nosemaphore+0xe/0x10
[381921.122766]  [<ffffffff81519f46>] do_page_fault+0x366/0x530
[381921.128440]  [<ffffffff810ee929>] ? __perf_event_overflow+0xa9/0x220
[381921.134896]  [<ffffffff810ef99b>] ? perf_event_update_userpage+0x9b/0xe0
[381921.141698]  [<ffffffff81012249>] ? intel_pmu_enable_all+0xc9/0x1a0
[381921.148061]  [<ffffffff810123ff>] ? x86_perf_event_set_period+0xdf/0x170
[381921.154852]  [<ffffffff81517295>] page_fault+0x25/0x30
[381921.160081]  [<ffffffff812a7510>] ? acpi_atomic_read+0xe3/0x120
[381921.166094]  [<ffffffff812a7486>] ? acpi_atomic_read+0x59/0x120
[381921.172107]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170 [ghes]
[381921.178639]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0 [ghes]
[381921.185004]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381921.191003]  [<ffffffff8151a1d8>] __atomic_notifier_call_chain+0x48/0x70
[381921.197789]  [<ffffffff8151a211>] atomic_notifier_call_chain+0x11/0x20
[381921.204399]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381921.209624]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381921.214584]  [<ffffffff81517550>] nmi+0x20/0x30
[381921.219208]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10
[381921.225470]  <<EOE>>  <IRQ>  [<ffffffff8101131e>] 
intel_pmu_disable_all+0x3e/0x120
[381921.233174]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381921.238836]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40
[381921.244582]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381921.250855]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381921.256520]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.262618]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.268363]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.274540]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.280198]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.286125]  [<ffffffff810206d7>] smp_apic_timer_interrupt+0x67/0xa0
[381921.292578]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.298673]  <EOI>

I recompiled the kernel again, disabling the tickless feature as I saw "nohz" early in the call trace. After that, it reproduced again but the call trace only changed slightly having tick_init_highres in there instead of tick_nohz_handler. I have that call trace if it is desired as well. It is nearly identical though.

Quoted 4 lines

I am currently trying a custom 2.6.36.4 on the system, but will need up to 3 days before I know if the problem exists there as well.

Any ideas on this?

It looks like a wrong address in ghes_read_estatus(). I'll have a look at it later today.

Quoted 187 lines

Hi,

It's better to report ACPI failues to linux-acpi.

Quoted 3 lines · Rick Warner

Hi all,

I am getting a kernel oops/panic on a dual xeon system that is the

master of a

Quoted 1 line

60 node HPC cluster. This is happening while stress testing the

system

Quoted 3 lines

including significant network traffic. The OS is openSuse 11.4.

We are running a custom compiled 2.6.39.3 kernel on the systems due to

a bug

Quoted 1 line

in the stock kernel 11.4 provided (igb driver related). After 1-3

days of

Quoted 1 line

heavy testing, the master node locks up with the caps lock and scroll

lock

Quoted 1 line

keys on the keyboard blinking with the following output captured via a

serial

Quoted 3 lines

console:

[381920.681113] BUG: unable to handle kernel NULL pointer dereference

at

Quoted 8 lines

(null)

[381920.689067] IP: [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381920.695187] PGD 30c27a067 PUD 16efe6067 PMD 0
[381920.699782] Oops: 0000 [#1] PREEMPT SMP
[381920.703866] last sysfs file:
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
[381920.711868] CPU 6
[381920.713800] Modules linked in: md5 ipmi_devintf ipmi_si

ipmi_msghandler

Quoted 2 lines

nfsd lockd nfs_acl auth_rpcgss sunrpc ipt_MASQUERADE iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4

iptable_filter

Quoted 1 line

ip_tables x_tables af_packet edd cpufreq_conservative

cpufreq_userspace

Quoted 1 line

cpufreq_powersave acpi_cpufreq mperf xfs dm_mod ioatdma i7core_edac

edac_core

Quoted 2 lines

sr_mod cdrom joydev igb i2c_i801 sg button ghes hed iTCO_wdt iTCO_vendor_support dca pcspkr ext4 jbd2 crc16 raid456

async_raid6_recov

Quoted 1 line

async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1

raid0 fan

Quoted 3 lines

processor thermal thermal_sys ata_generic pata_atiixp arcmsr

[381920.771623]
[381920.773210] Pid: 12701, comm: cluster Not tainted

2.6.39.3-microwaycustom

Quoted 5 lines

#1 Supermicro X8DTH-i/6/iF/6F/X8DTH

[381920.783292] RIP: 0010:[<ffffffff812a7510>]  [<ffffffff812a7510>]
acpi_atomic_read+0xe3/0x120
[381920.791853] RSP: 0000:ffff88063fc47d98  EFLAGS: 00010046
[381920.797260] RAX: 0000000000000000 RBX: 00000000bf7b5ff0 RCX:

ffffffff81a3cdd0

Quoted 3 lines

[381920.804486] RDX: 00000000bf7b6010 RSI: 00000000bf7b6000 RDI:
ffff88062d4b95c0
[381920.811712] RBP: ffff88063fc47dc8 R08: ffff88063fc47d98 R09:

0000000000000002

Quoted 4 lines

[381920.818940] R10: 0000000000000083 R11: 0000000000000010 R12:
ffffc90003044c20
[381920.826168] R13: ffff88063fc47de0 R14: 0000000000000000 R15:

0000000000000000

Quoted 8 lines

[381920.833392] FS:  0000000000000000(0000) GS:ffff88063fc40000(0000)
knlGS:0000000000000000
[381920.841571] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[381920.847409] CR2: 0000000000000000 CR3: 00000001d2d0f000 CR4:
00000000000006e0
[381920.854635] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[381920.861861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:

0000000000000400

Quoted 1 line

[381920.869088] Process cluster (pid: 12701, threadinfo

ffff880028284000, task

Quoted 3 lines

ffff880550184380)

[381920.877608] Stack:
[381920.879717]  ffffffff81a3cdd0 00000000bf7b5ff0 ffff88062b2cc140

ffffc90003044ca8

Quoted 1 line

[381920.887288]  ffff88062b2cc140 0000000000000001 ffff88063fc47e08

ffffffffa002b21f

Quoted 5 lines

[381920.894862]  0000000000000000 0000000000000000 ffff88062b2cc140
0000000000000000
[381920.902435] Call Trace:
[381920.904969]  <NMI>
[381920.907201]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381920.913726]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381920.920076]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381920.926080]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381920.932864]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20
> [381920.939473]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
> [381920.944691]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
> [381920.949649]  [<ffffffff81517550>] nmi+0x20/0x30
> [381920.954269]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10

Quoted 5 lines

[381920.960536]  <<EOE>>
[381920.962725]  <IRQ>
[381920.965132]  [<ffffffff8101131e>] intel_pmu_disable_all+0x3e/0x120
[381920.971399]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381920.977066]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40

Quoted 8 lines

[381920.982817]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381920.989082]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381920.994740]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.000832]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.006578]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.012758]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.018414]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.024343]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0
> [381921.030793]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20

Quoted 2 lines

[381921.036891]  <EOI>
[381921.038913] Code: fc 10 74 1f 77 08 41 80 fc 08 75 48 eb 0e 41 80

fc 20 74

Quoted 1 line

17 41 80 fc 40 75 3a eb 15 8a 00 0f b6 c0 eb 11 66 8b 00 0f b7 c0 eb

09 <8b>

Quoted 6 lines

00 89 c0 eb 03 48 8b 00 49 89 45 00 e8 8e 2b e2 ff eb 1b 0f

[381921.059462] RIP  [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381921.065669]  RSP <ffff88063fc47d98>
[381921.069242] CR2: 0000000000000000
[381921.072645] ---[ end trace 52697bfc73a34a90 ]---
[381921.077343] Kernel panic - not syncing: Fatal exception in

interrupt

Quoted 1 line

[381921.083784] Pid: 12701, comm: cluster Tainted: G      D

2.6.39.3-

Quoted 6 lines

microwaycustom #1

[381921.091788] Call Trace:
[381921.094333]  <NMI>  [<ffffffff81513300>] panic+0x9f/0x1da
[381921.099881]  [<ffffffff81517f8c>] oops_end+0xdc/0xf0
[381921.104952]  [<ffffffff81032b91>] no_context+0xf1/0x260
[381921.110277]  [<ffffffff81032e55>]

__bad_area_nosemaphore+0x155/0x200
> [381921.116739]  [<ffffffff81032f0e>] bad_area_nosemaphore+0xe/0x10
> [381921.122766]  [<ffffffff81519f46>] do_page_fault+0x366/0x530

Quoted 1 line

[381921.128440]  [<ffffffff810ee929>] ?

__perf_event_overflow+0xa9/0x220

Quoted 1 line

[381921.134896]  [<ffffffff810ef99b>] ?

perf_event_update_userpage+0x9b/0xe0

Quoted 1 line

[381921.141698]  [<ffffffff81012249>] ?

intel_pmu_enable_all+0xc9/0x1a0

Quoted 1 line

[381921.148061]  [<ffffffff810123ff>] ?

x86_perf_event_set_period+0xdf/0x170
> [381921.154852]  [<ffffffff81517295>] page_fault+0x25/0x30
> [381921.160081]  [<ffffffff812a7510>] ? acpi_atomic_read+0xe3/0x120
> [381921.166094]  [<ffffffff812a7486>] ? acpi_atomic_read+0x59/0x120
> [381921.172107]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381921.178639]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381921.185004]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381921.191003]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381921.197789]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20
> [381921.204399]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
> [381921.209624]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
> [381921.214584]  [<ffffffff81517550>] nmi+0x20/0x30
> [381921.219208]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10

Quoted 4 lines

[381921.225470]  <<EOE>>  <IRQ>  [<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120
[381921.233174]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381921.238836]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40

Quoted 8 lines

[381921.244582]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381921.250855]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381921.256520]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.262618]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.268363]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.274540]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.280198]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.286125]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0
> [381921.292578]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20

Quoted 4 lines

[381921.298673]  <EOI>

I recompiled the kernel again, disabling the tickless feature as I saw

"nohz"

Quoted 1 line

early in the call trace. After that, it reproduced again but the call

trace

Quoted 2 lines

only changed slightly having tick_init_highres in there instead of tick_nohz_handler. I have that call trace if it is desired as well.

It is

Quoted 3 lines

nearly identical though.

I am currently trying a custom 2.6.36.4 on the system, but will need

Quoted 8 lines

up to 3

Quoted 3 lines

days before I know if the problem exists there as well.

Any ideas on this?

It looks like a wrong address in ghes_read_estatus(). I'll have a look at it
later today.

Hmm. Please apply the appended patch and see if the error is still reproducible (it's on top of the current mainline, so you may need to adjust it by hand).

Thanks,
Rafael

---
 drivers/acpi/apei/ghes.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/drivers/acpi/apei/ghes.c
===================================================================

Quoted 216 lines · Rafael J. Wysocki

Quoted 187 lines

Hi,

It's better to report ACPI failues to linux-acpi.

Quoted 3 lines · Rick Warner

Hi all,

I am getting a kernel oops/panic on a dual xeon system that is the

master of a

Quoted 1 line

60 node HPC cluster. This is happening while stress testing the

system

Quoted 3 lines

including significant network traffic. The OS is openSuse 11.4.

We are running a custom compiled 2.6.39.3 kernel on the systems due to

a bug

Quoted 1 line

in the stock kernel 11.4 provided (igb driver related). After 1-3

days of

Quoted 1 line

heavy testing, the master node locks up with the caps lock and scroll

lock

Quoted 1 line

keys on the keyboard blinking with the following output captured via a

serial

Quoted 3 lines

console:

[381920.681113] BUG: unable to handle kernel NULL pointer dereference

at

Quoted 8 lines

(null)

[381920.689067] IP: [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381920.695187] PGD 30c27a067 PUD 16efe6067 PMD 0
[381920.699782] Oops: 0000 [#1] PREEMPT SMP
[381920.703866] last sysfs file:
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
[381920.711868] CPU 6
[381920.713800] Modules linked in: md5 ipmi_devintf ipmi_si

ipmi_msghandler

Quoted 2 lines

nfsd lockd nfs_acl auth_rpcgss sunrpc ipt_MASQUERADE iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4

iptable_filter

Quoted 1 line

ip_tables x_tables af_packet edd cpufreq_conservative

cpufreq_userspace

Quoted 1 line

cpufreq_powersave acpi_cpufreq mperf xfs dm_mod ioatdma i7core_edac

edac_core

Quoted 2 lines

sr_mod cdrom joydev igb i2c_i801 sg button ghes hed iTCO_wdt iTCO_vendor_support dca pcspkr ext4 jbd2 crc16 raid456

async_raid6_recov

Quoted 1 line

async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1

raid0 fan

Quoted 3 lines

processor thermal thermal_sys ata_generic pata_atiixp arcmsr

[381920.771623]
[381920.773210] Pid: 12701, comm: cluster Not tainted

2.6.39.3-microwaycustom

Quoted 5 lines

#1 Supermicro X8DTH-i/6/iF/6F/X8DTH

[381920.783292] RIP: 0010:[<ffffffff812a7510>]  [<ffffffff812a7510>]
acpi_atomic_read+0xe3/0x120
[381920.791853] RSP: 0000:ffff88063fc47d98  EFLAGS: 00010046
[381920.797260] RAX: 0000000000000000 RBX: 00000000bf7b5ff0 RCX:

ffffffff81a3cdd0

Quoted 3 lines

[381920.804486] RDX: 00000000bf7b6010 RSI: 00000000bf7b6000 RDI:
ffff88062d4b95c0
[381920.811712] RBP: ffff88063fc47dc8 R08: ffff88063fc47d98 R09:

0000000000000002

Quoted 4 lines

[381920.818940] R10: 0000000000000083 R11: 0000000000000010 R12:
ffffc90003044c20
[381920.826168] R13: ffff88063fc47de0 R14: 0000000000000000 R15:

0000000000000000

Quoted 8 lines

[381920.833392] FS:  0000000000000000(0000) GS:ffff88063fc40000(0000)
knlGS:0000000000000000
[381920.841571] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[381920.847409] CR2: 0000000000000000 CR3: 00000001d2d0f000 CR4:
00000000000006e0
[381920.854635] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[381920.861861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:

0000000000000400

Quoted 1 line

[381920.869088] Process cluster (pid: 12701, threadinfo

ffff880028284000, task

Quoted 3 lines

ffff880550184380)

[381920.877608] Stack:
[381920.879717]  ffffffff81a3cdd0 00000000bf7b5ff0 ffff88062b2cc140

ffffc90003044ca8

Quoted 1 line

[381920.887288]  ffff88062b2cc140 0000000000000001 ffff88063fc47e08

ffffffffa002b21f

Quoted 5 lines

[381920.894862]  0000000000000000 0000000000000000 ffff88062b2cc140
0000000000000000
[381920.902435] Call Trace:
[381920.904969]  <NMI>
[381920.907201]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381920.913726]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381920.920076]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381920.926080]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381920.932864]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20
> [381920.939473]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
> [381920.944691]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
> [381920.949649]  [<ffffffff81517550>] nmi+0x20/0x30
> [381920.954269]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10

Quoted 5 lines

[381920.960536]  <<EOE>>
[381920.962725]  <IRQ>
[381920.965132]  [<ffffffff8101131e>] intel_pmu_disable_all+0x3e/0x120
[381920.971399]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381920.977066]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40

Quoted 8 lines

[381920.982817]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381920.989082]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381920.994740]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.000832]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.006578]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.012758]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.018414]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.024343]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0
> [381921.030793]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20

Quoted 2 lines

[381921.036891]  <EOI>
[381921.038913] Code: fc 10 74 1f 77 08 41 80 fc 08 75 48 eb 0e 41 80

fc 20 74

Quoted 1 line

17 41 80 fc 40 75 3a eb 15 8a 00 0f b6 c0 eb 11 66 8b 00 0f b7 c0 eb

09 <8b>

Quoted 6 lines

00 89 c0 eb 03 48 8b 00 49 89 45 00 e8 8e 2b e2 ff eb 1b 0f

[381921.059462] RIP  [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381921.065669]  RSP <ffff88063fc47d98>
[381921.069242] CR2: 0000000000000000
[381921.072645] ---[ end trace 52697bfc73a34a90 ]---
[381921.077343] Kernel panic - not syncing: Fatal exception in

interrupt

Quoted 1 line

[381921.083784] Pid: 12701, comm: cluster Tainted: G      D

2.6.39.3-

Quoted 6 lines

microwaycustom #1

[381921.091788] Call Trace:
[381921.094333]  <NMI>  [<ffffffff81513300>] panic+0x9f/0x1da
[381921.099881]  [<ffffffff81517f8c>] oops_end+0xdc/0xf0
[381921.104952]  [<ffffffff81032b91>] no_context+0xf1/0x260
[381921.110277]  [<ffffffff81032e55>]

__bad_area_nosemaphore+0x155/0x200
> [381921.116739]  [<ffffffff81032f0e>] bad_area_nosemaphore+0xe/0x10
> [381921.122766]  [<ffffffff81519f46>] do_page_fault+0x366/0x530

Quoted 1 line

[381921.128440]  [<ffffffff810ee929>] ?

__perf_event_overflow+0xa9/0x220

Quoted 1 line

[381921.134896]  [<ffffffff810ef99b>] ?

perf_event_update_userpage+0x9b/0xe0

Quoted 1 line

[381921.141698]  [<ffffffff81012249>] ?

intel_pmu_enable_all+0xc9/0x1a0

Quoted 1 line

[381921.148061]  [<ffffffff810123ff>] ?

x86_perf_event_set_period+0xdf/0x170
> [381921.154852]  [<ffffffff81517295>] page_fault+0x25/0x30
> [381921.160081]  [<ffffffff812a7510>] ? acpi_atomic_read+0xe3/0x120
> [381921.166094]  [<ffffffff812a7486>] ? acpi_atomic_read+0x59/0x120
> [381921.172107]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381921.178639]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381921.185004]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381921.191003]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381921.197789]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20
> [381921.204399]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
> [381921.209624]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
> [381921.214584]  [<ffffffff81517550>] nmi+0x20/0x30
> [381921.219208]  [<ffffffff8102ae8a>] ? native_write_msr_safe+0xa/0x10

Quoted 4 lines

[381921.225470]  <<EOE>>  <IRQ>  [<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120
[381921.233174]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381921.238836]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40

Quoted 8 lines

[381921.244582]  [<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381921.250855]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381921.256520]  [<ffffffff81067097>] update_process_times+0x67/0x80
[381921.262618]  [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0
[381921.268363]  [<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.274540]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.280198]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.286125]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0
> [381921.292578]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20

Quoted 4 lines

[381921.298673]  <EOI>

I recompiled the kernel again, disabling the tickless feature as I saw

"nohz"

Quoted 1 line

early in the call trace. After that, it reproduced again but the call

trace

Quoted 2 lines

only changed slightly having tick_init_highres in there instead of tick_nohz_handler. I have that call trace if it is desired as well.

It is

Quoted 3 lines

nearly identical though.

I am currently trying a custom 2.6.36.4 on the system, but will need

Quoted 8 lines

up to 3

Quoted 3 lines

days before I know if the problem exists there as well.

Any ideas on this?

It looks like a wrong address in ghes_read_estatus(). I'll have a look at it
later today.

Hmm. Please apply the appended patch and see if the error is still reproducible (it's on top of the current mainline, so you may need to adjust it by hand).

Thanks,
Rafael

---
 drivers/acpi/apei/ghes.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/drivers/acpi/apei/ghes.c
===================================================================

drivers/acpi/apei/ghes.c +0 -0

@@ -398,6 +398,9 @@ static int ghes_read_estatus(struct ghes
 	u32 len;
 	int rc;
 

drivers/acpi/apei/ghes.c static int ghes_read_estatus(struct ghes · line 406

Quoted 7 lines

drivers/acpi/apei/ghes.c +3 -0

+	if (!g)
+		return -EINVAL;
+
 	rc = acpi_atomic_read(&buf_paddr, &g->error_status_address);
 	if (rc) {
 		if (!silent && printk_ratelimit())

Thanks for the quick patch! Unfortunately, the system crashed again with the same call trace. I'll list it below in case there are any differences I missed. I have currently added a test for (!&g->error_status_address) just like your test on (!g) to verify that isn't a null pointer either, and will start tests back up with that change. Please let me know if you have any other suggestions or patches for me to try.

Quoted 321 lines

Quoted 321 lines

Quoted 285 lines

Hi,

It's better to report ACPI failues to linux-acpi.

Quoted 3 lines · Rick Warner

Hi all,

I am getting a kernel oops/panic on a dual xeon system that is the

master of a

Quoted 1 line

60 node HPC cluster. This is happening while stress testing the

system

Quoted 4 lines

including significant network traffic. The OS is openSuse 11.4.

We are running a custom compiled 2.6.39.3 kernel on the systems due to

a bug

Quoted 1 line

in the stock kernel 11.4 provided (igb driver related). After 1-3

days of

Quoted 2 lines

heavy testing, the master node locks up with the caps lock and scroll

lock

Quoted 2 lines

keys on the keyboard blinking with the following output captured via a

serial

Quoted 4 lines

console:

[381920.681113] BUG: unable to handle kernel NULL pointer

dereference

at

Quoted 8 lines

(null)

[381920.689067] IP: [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381920.695187] PGD 30c27a067 PUD 16efe6067 PMD 0
[381920.699782] Oops: 0000 [#1] PREEMPT SMP
[381920.703866] last sysfs file:
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
[381920.711868] CPU 6
[381920.713800] Modules linked in: md5 ipmi_devintf ipmi_si

ipmi_msghandler

Quoted 2 lines

nfsd lockd nfs_acl auth_rpcgss sunrpc ipt_MASQUERADE iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4

iptable_filter

Quoted 1 line

ip_tables x_tables af_packet edd cpufreq_conservative

cpufreq_userspace

Quoted 1 line

cpufreq_powersave acpi_cpufreq mperf xfs dm_mod ioatdma i7core_edac

edac_core

Quoted 2 lines

sr_mod cdrom joydev igb i2c_i801 sg button ghes hed iTCO_wdt iTCO_vendor_support dca pcspkr ext4 jbd2 crc16 raid456

async_raid6_recov

Quoted 1 line

async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1

raid0 fan

Quoted 3 lines

processor thermal thermal_sys ata_generic pata_atiixp arcmsr

[381920.771623]
[381920.773210] Pid: 12701, comm: cluster Not tainted

2.6.39.3-microwaycustom

Quoted 6 lines

#1 Supermicro X8DTH-i/6/iF/6F/X8DTH

[381920.783292] RIP: 0010:[<ffffffff812a7510>]  [<ffffffff812a7510>]
acpi_atomic_read+0xe3/0x120
[381920.791853] RSP: 0000:ffff88063fc47d98  EFLAGS: 00010046

[381920.797260] RAX: 0000000000000000 RBX: 00000000bf7b5ff0 RCX:

ffffffff81a3cdd0

Quoted 4 lines

[381920.804486] RDX: 00000000bf7b6010 RSI: 00000000bf7b6000 RDI:

ffff88062d4b95c0

[381920.811712] RBP: ffff88063fc47dc8 R08: ffff88063fc47d98 R09:

0000000000000002

Quoted 10 lines

[381920.818940] R10: 0000000000000083 R11: 0000000000000010 R12:
ffffc90003044c20
[381920.826168] R13: ffff88063fc47de0 R14: 0000000000000000 R15:
0000000000000000
[381920.833392] FS:  0000000000000000(0000)
GS:ffff88063fc40000(0000) knlGS:0000000000000000
[381920.841571] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[381920.847409] CR2: 0000000000000000 CR3: 00000001d2d0f000 CR4:
00000000000006e0
[381920.854635] DR0: 0000000000000000 DR1: 0000000000000000 DR2:

Quoted 3 lines

0000000000000000

[381920.861861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:

0000000000000400

Quoted 1 line

[381920.869088] Process cluster (pid: 12701, threadinfo

ffff880028284000, task

Quoted 3 lines

ffff880550184380)

[381920.877608] Stack:
[381920.879717]  ffffffff81a3cdd0 00000000bf7b5ff0 ffff88062b2cc140

ffffc90003044ca8

Quoted 1 line

[381920.887288]  ffff88062b2cc140 0000000000000001 ffff88063fc47e08

ffffffffa002b21f

Quoted 5 lines

[381920.894862]  0000000000000000 0000000000000000 ffff88062b2cc140
0000000000000000
[381920.902435] Call Trace:
[381920.904969]  <NMI>
[381920.907201]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381920.913726]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381920.920076]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381920.926080]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381920.932864]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20

Quoted 11 lines

[381920.939473]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381920.944691]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381920.949649]  [<ffffffff81517550>] nmi+0x20/0x30
[381920.954269]  [<ffffffff8102ae8a>] ?
native_write_msr_safe+0xa/0x10 [381920.960536]  <<EOE>>
[381920.962725]  <IRQ>
[381920.965132]  [<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120 [381920.971399] 
[<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50 [381920.977066] 
[<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40 [381920.982817] 
[<ffffffff810ee828>] perf_event_task_tick+0x218/0x270

Quoted 8 lines

[381920.989082]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381920.994740]  [<ffffffff81067097>]
update_process_times+0x67/0x80 [381921.000832] 
[<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0 [381921.006578] 
[<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.012758]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.018414]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.024343]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0

Quoted 4 lines

[381921.030793]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.036891]  <EOI>
[381921.038913] Code: fc 10 74 1f 77 08 41 80 fc 08 75 48 eb 0e 41

80

fc 20 74

Quoted 1 line

17 41 80 fc 40 75 3a eb 15 8a 00 0f b6 c0 eb 11 66 8b 00 0f b7 c0 eb

09 <8b>

Quoted 6 lines

00 89 c0 eb 03 48 8b 00 49 89 45 00 e8 8e 2b e2 ff eb 1b 0f

[381921.059462] RIP  [<ffffffff812a7510>]
acpi_atomic_read+0xe3/0x120 [381921.065669]  RSP <ffff88063fc47d98>
[381921.069242] CR2: 0000000000000000
[381921.072645] ---[ end trace 52697bfc73a34a90 ]---
[381921.077343] Kernel panic - not syncing: Fatal exception in

interrupt

Quoted 1 line

[381921.083784] Pid: 12701, comm: cluster Tainted: G      D

2.6.39.3-

Quoted 6 lines

microwaycustom #1

[381921.091788] Call Trace:
[381921.094333]  <NMI>  [<ffffffff81513300>] panic+0x9f/0x1da
[381921.099881]  [<ffffffff81517f8c>] oops_end+0xdc/0xf0
[381921.104952]  [<ffffffff81032b91>] no_context+0xf1/0x260
[381921.110277]  [<ffffffff81032e55>]

__bad_area_nosemaphore+0x155/0x200

Quoted 3 lines

[381921.116739]  [<ffffffff81032f0e>] bad_area_nosemaphore+0xe/0x10
[381921.122766]  [<ffffffff81519f46>] do_page_fault+0x366/0x530
[381921.128440]  [<ffffffff810ee929>] ?

__perf_event_overflow+0xa9/0x220

Quoted 1 line

[381921.134896]  [<ffffffff810ef99b>] ?

perf_event_update_userpage+0x9b/0xe0

Quoted 1 line

[381921.141698]  [<ffffffff81012249>] ?

intel_pmu_enable_all+0xc9/0x1a0

Quoted 1 line

[381921.148061]  [<ffffffff810123ff>] ?

x86_perf_event_set_period+0xdf/0x170

Quoted 4 lines

[381921.154852]  [<ffffffff81517295>] page_fault+0x25/0x30
[381921.160081]  [<ffffffff812a7510>] ? acpi_atomic_read+0xe3/0x120
[381921.166094]  [<ffffffff812a7486>] ? acpi_atomic_read+0x59/0x120
[381921.172107]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381921.178639]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381921.185004]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381921.191003]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381921.197789]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20

Quoted 10 lines

[381921.204399]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381921.209624]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381921.214584]  [<ffffffff81517550>] nmi+0x20/0x30
[381921.219208]  [<ffffffff8102ae8a>] ?
native_write_msr_safe+0xa/0x10 [381921.225470]  <<EOE>>  <IRQ> 
[<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120
[381921.233174]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381921.238836]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40
[381921.244582]  [<ffffffff810ee828>]

Quoted 8 lines

perf_event_task_tick+0x218/0x270 [381921.250855] 
[<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0 [381921.256520] 
[<ffffffff81067097>] update_process_times+0x67/0x80 [381921.262618]
 [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0 [381921.268363] 
[<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.274540]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.280198]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.286125]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0

Quoted 6 lines

[381921.292578]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.298673]  <EOI>

I recompiled the kernel again, disabling the tickless feature as I saw

"nohz"

Quoted 2 lines

early in the call trace. After that, it reproduced again but the call

trace

Quoted 2 lines

only changed slightly having tick_init_highres in there instead of tick_nohz_handler. I have that call trace if it is desired as well.

It is

Quoted 3 lines

nearly identical though.

I am currently trying a custom 2.6.36.4 on the system, but will need

up to 3

Quoted 7 lines

Quoted 3 lines

days before I know if the problem exists there as well.

Any ideas on this?

It looks like a wrong address in ghes_read_estatus(). I'll have a look at it
later today.

Hmm. Please apply the appended patch and see if the error is still reproducible (it's on top of the current mainline, so you may need to adjust it by hand).

Thanks,
Rafael

---

 drivers/acpi/apei/ghes.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/drivers/acpi/apei/ghes.c
===================================================================

drivers/acpi/apei/ghes.c +3 -0

@@ -398,6 +398,9 @@ static int ghes_read_estatus(struct ghes
 
 	u32 len;
 	int rc;
 
+	if (!g)
+		return -EINVAL;
+
 
 	rc = acpi_atomic_read(&buf_paddr, &g->error_status_address);

if (rc) {

Quoted 8 lines

Quoted 1 line

if (!silent && printk_ratelimit())

Thanks for the quick patch! Unfortunately, the system crashed again with the same call trace. I'll list it below in case there are any differences I missed. I have currently added a test for (!&g->error_status_address) just like your test on (!g) to verify that isn't a null pointer either, and will start tests back up with that change. Please let me know if you have any other suggestions or patches for me to try.

That was a long shot, but perhaps you can check what source code corresponds to ghes_read_estatus+0x38 in your kernel?

Rafael

Hi Rafael,

Thanks for the off-list help in getting you this info.

I had already rebuilt the kernel using the change I mentioned earlier (test on !&g->error_status_address) since the call trace I got.

I luckily still had a copy of the kernel and modules I built previously using just your patch, so I undid my change to the ghes.c source, leaving just your patch but not mine so it would match the ghes.ko module I ran on. This is the output of gdb on that ghes.ko now:

(gdb) l *ghes_read_estatus+0x38
0x258 is in ghes_read_estatus (drivers/acpi/apei/ghes.c:296).
warning: Source file is more recent than executable.
291 int rc;
292 if (!g)
293 return -EINVAL;
294
295 rc = acpi_atomic_read(&buf_paddr, &g->error_status_address);
296 if (rc) {
297 if (!silent && printk_ratelimit())
298 pr_warning(FW_WARN GHES_PFX
299 "Failed to read error status block address for hardware error source:
%d.\n",
300 g->header.source_id);

The warning about the source being newer is because of the reverted change in the ghes.c source mentioned above.

Hi,

On Monday, August 22, 2011, Rick Warner wrote:
...

Quoted 33 lines

Hi Rafael,

Thanks for the off-list help in getting you this info.

I had already rebuilt the kernel using the change I mentioned earlier (test on
!&g->error_status_address) since the call trace I got.

I luckily still had a copy of the kernel and modules I built previously using
just your patch, so I undid my change to the ghes.c source, leaving just your
patch but not mine so it would match the ghes.ko module I ran on. This is the
output of gdb on that ghes.ko now:

(gdb) l *ghes_read_estatus+0x38
0x258 is in ghes_read_estatus (drivers/acpi/apei/ghes.c:296).
warning: Source file is more recent than executable.
291 int rc;
292 if (!g)
293 return -EINVAL;
294
295 rc = acpi_atomic_read(&buf_paddr,
&g->error_status_address);
296 if (rc) {
297 if (!silent && printk_ratelimit())
298 pr_warning(FW_WARN GHES_PFX
299 "Failed to read error status block address for hardware error
source:
%d.\n",
300 g->header.source_id);

Quoted 3 lines

The warning about the source being newer is because of the reverted change in
the ghes.c source mentioned above.

OK, since &buf_addr cannot be NULL, perhaps ghes is. Please check if the appended patch makes a difference.

Thanks,
Rafael

---
 drivers/acpi/apei/ghes.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

Index: linux/drivers/acpi/apei/ghes.c
===================================================================

drivers/acpi/apei/ghes.c +1 -1

@@ -393,11 +393,16 @@ static void ghes_copy_tofrom_phys(void *
 
 static int ghes_read_estatus(struct ghes *ghes, int silent)
 {
-	struct acpi_hest_generic *g = ghes->generic;
+	struct acpi_hest_generic *g;
 	u64 buf_paddr;
 	u32 len;

Quoted 330 lines

Quoted 321 lines

Quoted 321 lines

Quoted 285 lines

Hi,

It's better to report ACPI failues to linux-acpi.

Quoted 3 lines · Rick Warner

Hi all,

I am getting a kernel oops/panic on a dual xeon system that is the

master of a

Quoted 1 line

60 node HPC cluster. This is happening while stress testing the

system

Quoted 4 lines

including significant network traffic. The OS is openSuse 11.4.

We are running a custom compiled 2.6.39.3 kernel on the systems due to

a bug

Quoted 1 line

in the stock kernel 11.4 provided (igb driver related). After 1-3

days of

Quoted 2 lines

heavy testing, the master node locks up with the caps lock and scroll

lock

Quoted 2 lines

keys on the keyboard blinking with the following output captured via a

serial

Quoted 4 lines

console:

[381920.681113] BUG: unable to handle kernel NULL pointer

dereference

at

Quoted 8 lines

(null)

[381920.689067] IP: [<ffffffff812a7510>] acpi_atomic_read+0xe3/0x120
[381920.695187] PGD 30c27a067 PUD 16efe6067 PMD 0
[381920.699782] Oops: 0000 [#1] PREEMPT SMP
[381920.703866] last sysfs file:
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
[381920.711868] CPU 6
[381920.713800] Modules linked in: md5 ipmi_devintf ipmi_si

ipmi_msghandler

Quoted 2 lines

nfsd lockd nfs_acl auth_rpcgss sunrpc ipt_MASQUERADE iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4

iptable_filter

Quoted 1 line

ip_tables x_tables af_packet edd cpufreq_conservative

cpufreq_userspace

Quoted 1 line

cpufreq_powersave acpi_cpufreq mperf xfs dm_mod ioatdma i7core_edac

edac_core

Quoted 2 lines

sr_mod cdrom joydev igb i2c_i801 sg button ghes hed iTCO_wdt iTCO_vendor_support dca pcspkr ext4 jbd2 crc16 raid456

async_raid6_recov

Quoted 1 line

async_pq raid6_pq async_xor xor async_memcpy async_tx raid10 raid1

raid0 fan

Quoted 3 lines

processor thermal thermal_sys ata_generic pata_atiixp arcmsr

[381920.771623]
[381920.773210] Pid: 12701, comm: cluster Not tainted

2.6.39.3-microwaycustom

Quoted 6 lines

#1 Supermicro X8DTH-i/6/iF/6F/X8DTH

[381920.783292] RIP: 0010:[<ffffffff812a7510>]  [<ffffffff812a7510>]
acpi_atomic_read+0xe3/0x120
[381920.791853] RSP: 0000:ffff88063fc47d98  EFLAGS: 00010046

[381920.797260] RAX: 0000000000000000 RBX: 00000000bf7b5ff0 RCX:

ffffffff81a3cdd0

Quoted 4 lines

[381920.804486] RDX: 00000000bf7b6010 RSI: 00000000bf7b6000 RDI:

ffff88062d4b95c0

[381920.811712] RBP: ffff88063fc47dc8 R08: ffff88063fc47d98 R09:

0000000000000002

Quoted 10 lines

[381920.818940] R10: 0000000000000083 R11: 0000000000000010 R12:
ffffc90003044c20
[381920.826168] R13: ffff88063fc47de0 R14: 0000000000000000 R15:
0000000000000000
[381920.833392] FS:  0000000000000000(0000)
GS:ffff88063fc40000(0000) knlGS:0000000000000000
[381920.841571] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[381920.847409] CR2: 0000000000000000 CR3: 00000001d2d0f000 CR4:
00000000000006e0
[381920.854635] DR0: 0000000000000000 DR1: 0000000000000000 DR2:

Quoted 3 lines

0000000000000000

[381920.861861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:

0000000000000400

Quoted 1 line

[381920.869088] Process cluster (pid: 12701, threadinfo

ffff880028284000, task

Quoted 3 lines

ffff880550184380)

[381920.877608] Stack:
[381920.879717]  ffffffff81a3cdd0 00000000bf7b5ff0 ffff88062b2cc140

ffffc90003044ca8

Quoted 1 line

[381920.887288]  ffff88062b2cc140 0000000000000001 ffff88063fc47e08

ffffffffa002b21f

Quoted 5 lines

[381920.894862]  0000000000000000 0000000000000000 ffff88062b2cc140
0000000000000000
[381920.902435] Call Trace:
[381920.904969]  <NMI>
[381920.907201]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381920.913726]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381920.920076]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381920.926080]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381920.932864]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20

Quoted 11 lines

[381920.939473]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381920.944691]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381920.949649]  [<ffffffff81517550>] nmi+0x20/0x30
[381920.954269]  [<ffffffff8102ae8a>] ?
native_write_msr_safe+0xa/0x10 [381920.960536]  <<EOE>>
[381920.962725]  <IRQ>
[381920.965132]  [<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120 [381920.971399] 
[<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50 [381920.977066] 
[<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40 [381920.982817] 
[<ffffffff810ee828>] perf_event_task_tick+0x218/0x270

Quoted 8 lines

[381920.989082]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381920.994740]  [<ffffffff81067097>]
update_process_times+0x67/0x80 [381921.000832] 
[<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0 [381921.006578] 
[<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.012758]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.018414]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.024343]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0

Quoted 4 lines

[381921.030793]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.036891]  <EOI>
[381921.038913] Code: fc 10 74 1f 77 08 41 80 fc 08 75 48 eb 0e 41

80

fc 20 74

Quoted 1 line

17 41 80 fc 40 75 3a eb 15 8a 00 0f b6 c0 eb 11 66 8b 00 0f b7 c0 eb

09 <8b>

Quoted 6 lines

00 89 c0 eb 03 48 8b 00 49 89 45 00 e8 8e 2b e2 ff eb 1b 0f

[381921.059462] RIP  [<ffffffff812a7510>]
acpi_atomic_read+0xe3/0x120 [381921.065669]  RSP <ffff88063fc47d98>
[381921.069242] CR2: 0000000000000000
[381921.072645] ---[ end trace 52697bfc73a34a90 ]---
[381921.077343] Kernel panic - not syncing: Fatal exception in

interrupt

Quoted 1 line

[381921.083784] Pid: 12701, comm: cluster Tainted: G      D

2.6.39.3-

Quoted 6 lines

microwaycustom #1

[381921.091788] Call Trace:
[381921.094333]  <NMI>  [<ffffffff81513300>] panic+0x9f/0x1da
[381921.099881]  [<ffffffff81517f8c>] oops_end+0xdc/0xf0
[381921.104952]  [<ffffffff81032b91>] no_context+0xf1/0x260
[381921.110277]  [<ffffffff81032e55>]

__bad_area_nosemaphore+0x155/0x200

Quoted 3 lines

[381921.116739]  [<ffffffff81032f0e>] bad_area_nosemaphore+0xe/0x10
[381921.122766]  [<ffffffff81519f46>] do_page_fault+0x366/0x530
[381921.128440]  [<ffffffff810ee929>] ?

__perf_event_overflow+0xa9/0x220

Quoted 1 line

[381921.134896]  [<ffffffff810ef99b>] ?

perf_event_update_userpage+0x9b/0xe0

Quoted 1 line

[381921.141698]  [<ffffffff81012249>] ?

intel_pmu_enable_all+0xc9/0x1a0

Quoted 1 line

[381921.148061]  [<ffffffff810123ff>] ?

x86_perf_event_set_period+0xdf/0x170

Quoted 4 lines

[381921.154852]  [<ffffffff81517295>] page_fault+0x25/0x30
[381921.160081]  [<ffffffff812a7510>] ? acpi_atomic_read+0xe3/0x120
[381921.166094]  [<ffffffff812a7486>] ? acpi_atomic_read+0x59/0x120
[381921.172107]  [<ffffffffa002b21f>] ghes_read_estatus+0x2f/0x170

[ghes]

Quoted 1 line

[381921.178639]  [<ffffffffa002b618>] ghes_notify_nmi+0xd8/0x1b0

[ghes]

Quoted 2 lines

[381921.185004]  [<ffffffff8151a14f>] notifier_call_chain+0x3f/0x80
[381921.191003]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381921.197789]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20

Quoted 10 lines

[381921.204399]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381921.209624]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381921.214584]  [<ffffffff81517550>] nmi+0x20/0x30
[381921.219208]  [<ffffffff8102ae8a>] ?
native_write_msr_safe+0xa/0x10 [381921.225470]  <<EOE>>  <IRQ> 
[<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120
[381921.233174]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381921.238836]  [<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40
[381921.244582]  [<ffffffff810ee828>]

Quoted 8 lines

perf_event_task_tick+0x218/0x270 [381921.250855] 
[<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0 [381921.256520] 
[<ffffffff81067097>] update_process_times+0x67/0x80 [381921.262618]
 [<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0 [381921.268363] 
[<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.274540]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.280198]  [<ffffffff8107dea7>] hrtimer_interrupt+0xb7/0x1e0
[381921.286125]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0

Quoted 6 lines

[381921.292578]  [<ffffffff8151e3f3>] apic_timer_interrupt+0x13/0x20
[381921.298673]  <EOI>

I recompiled the kernel again, disabling the tickless feature as I saw

"nohz"

Quoted 2 lines

early in the call trace. After that, it reproduced again but the call

trace

Quoted 2 lines

only changed slightly having tick_init_highres in there instead of tick_nohz_handler. I have that call trace if it is desired as well.

It is

Quoted 3 lines

nearly identical though.

I am currently trying a custom 2.6.36.4 on the system, but will need

up to 3

Quoted 7 lines

Quoted 3 lines

days before I know if the problem exists there as well.

Any ideas on this?

It looks like a wrong address in ghes_read_estatus(). I'll have a look at it
later today.

Hmm. Please apply the appended patch and see if the error is still reproducible (it's on top of the current mainline, so you may need to adjust it by hand).

Thanks,
Rafael

---

 drivers/acpi/apei/ghes.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/drivers/acpi/apei/ghes.c
===================================================================

drivers/acpi/apei/ghes.c +3 -0

@@ -398,6 +398,9 @@ static int ghes_read_estatus(struct ghes
 
 	u32 len;
 	int rc;
 
+	if (!g)
+		return -EINVAL;
+
 
 	rc = acpi_atomic_read(&buf_paddr, &g->error_status_address);

if (rc) {

Quoted 8 lines

Quoted 1 line

if (!silent && printk_ratelimit())

Thanks for the quick patch! Unfortunately, the system crashed again with the same call trace. I'll list it below in case there are any differences I missed. I have currently added a test for (!&g->error_status_address) just like your test on (!g) to verify that isn't a null pointer either, and will start tests back up with that change. Please let me know if you have any other suggestions or patches for me to try.

Quoted 4 lines

That was a long shot, but perhaps you can check what source code corresponds to ghes_read_estatus+0x38 in your kernel?

Rafael

Hi Rafael,

Thanks for the off-list help in getting you this info.

I had already rebuilt the kernel using the change I mentioned earlier (test on !&g->error_status_address) since the call trace I got.

I luckily still had a copy of the kernel and modules I built previously using just your patch, so I undid my change to the ghes.c source, leaving just your patch but not mine so it would match the ghes.ko module I ran on. This is the output of gdb on that ghes.ko now:

(gdb) l *ghes_read_estatus+0x38
0x258 is in ghes_read_estatus (drivers/acpi/apei/ghes.c:296).
warning: Source file is more recent than executable.
291 int rc;
292 if (!g)
293 return -EINVAL;
294
295 rc = acpi_atomic_read(&buf_paddr, &g->error_status_address);
296 if (rc) {
297 if (!silent && printk_ratelimit())
298 pr_warning(FW_WARN GHES_PFX
299 "Failed to read error status block address for hardware error source:
%d.\n",
300 g->header.source_id);

The warning about the source being newer is because of the reverted change in the ghes.c source mentioned above.

Thanks,
Rick
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

adding Ying. He wrote the ghes code. But if you using 2.6.39, then the ghes code probably still has some bugs in it that were fixed in 3.0 and 3.1.
And unless you are using a Nehalem machine or later (Westmere, Sandy Bridge), ghes won't do much for you. Can you just disable it in the config?

Cheers,
Don

Quoted 435 lines · Rick Warner

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Hi,

It's better to report ACPI failues to linux-acpi.

Quoted 3 lines · Rick Warner

Hi all,

I am getting a kernel oops/panic on a dual xeon system that is

the

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

master of a

Quoted 1 line

60 node HPC cluster. This is happening while stress testing

the

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

system

Quoted 1 line

including significant network traffic. The OS is openSuse

11.4.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

We are running a custom compiled 2.6.39.3 kernel on the systems

due

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

to

a bug

Quoted 1 line

in the stock kernel 11.4 provided (igb driver related). After

1-3

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

days of

Quoted 2 lines

heavy testing, the master node locks up with the caps lock and scroll

lock

Quoted 1 line

keys on the keyboard blinking with the following output

captured via

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

a

serial

Quoted 2 lines

console:

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

[381920.681113] BUG: unable to handle kernel NULL pointer

dereference

at

Quoted 2 lines

(null)

[381920.689067] IP: [<ffffffff812a7510>]

acpi_atomic_read+0xe3/0x120

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

[381920.695187] PGD 30c27a067 PUD 16efe6067 PMD 0
[381920.699782] Oops: 0000 [#1] PREEMPT SMP
[381920.703866] last sysfs file:
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
[381920.711868] CPU 6
[381920.713800] Modules linked in: md5 ipmi_devintf ipmi_si

ipmi_msghandler

Quoted 1 line

nfsd lockd nfs_acl auth_rpcgss sunrpc ipt_MASQUERADE

iptable_mangle

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack

nf_defrag_ipv4

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

iptable_filter

Quoted 1 line

ip_tables x_tables af_packet edd cpufreq_conservative

cpufreq_userspace

Quoted 1 line

cpufreq_powersave acpi_cpufreq mperf xfs dm_mod ioatdma

i7core_edac

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

edac_core

Quoted 2 lines

sr_mod cdrom joydev igb i2c_i801 sg button ghes hed iTCO_wdt iTCO_vendor_support dca pcspkr ext4 jbd2 crc16 raid456

async_raid6_recov

Quoted 1 line

async_pq raid6_pq async_xor xor async_memcpy async_tx raid10

raid1

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

raid0 fan

Quoted 3 lines

processor thermal thermal_sys ata_generic pata_atiixp arcmsr

[381920.771623]
[381920.773210] Pid: 12701, comm: cluster Not tainted

2.6.39.3-microwaycustom

Quoted 2 lines

#1 Supermicro X8DTH-i/6/iF/6F/X8DTH

[381920.783292] RIP: 0010:[<ffffffff812a7510>]

[<ffffffff812a7510>]

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

acpi_atomic_read+0xe3/0x120
[381920.791853] RSP: 0000:ffff88063fc47d98  EFLAGS: 00010046

[381920.797260] RAX: 0000000000000000 RBX: 00000000bf7b5ff0

RCX:

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

ffffffff81a3cdd0

Quoted 1 line

[381920.804486] RDX: 00000000bf7b6010 RSI: 00000000bf7b6000

RDI:

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

ffff88062d4b95c0

[381920.811712] RBP: ffff88063fc47dc8 R08: ffff88063fc47d98

R09:

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

0000000000000002

Quoted 1 line

[381920.818940] R10: 0000000000000083 R11: 0000000000000010

R12:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

ffffc90003044c20

[381920.826168] R13: ffff88063fc47de0 R14: 0000000000000000

R15:

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

0000000000000000

[381920.833392] FS:  0000000000000000(0000)
GS:ffff88063fc40000(0000) knlGS:0000000000000000
[381920.841571] CS:  0010 DS: 002b ES: 002b CR0:

0000000080050033

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

[381920.847409] CR2: 0000000000000000 CR3: 00000001d2d0f000

CR4:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

00000000000006e0

[381920.854635] DR0: 0000000000000000 DR1: 0000000000000000

DR2:

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

0000000000000000

[381920.861861] DR3: 0000000000000000 DR6: 00000000ffff0ff0

DR7:

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

0000000000000400

Quoted 1 line

[381920.869088] Process cluster (pid: 12701, threadinfo

ffff880028284000, task

Quoted 3 lines

ffff880550184380)

[381920.877608] Stack:
[381920.879717]  ffffffff81a3cdd0 00000000bf7b5ff0

ffff88062b2cc140

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

ffffc90003044ca8

Quoted 1 line

[381920.887288]  ffff88062b2cc140 0000000000000001

ffff88063fc47e08

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

ffffffffa002b21f

Quoted 1 line

[381920.894862]  0000000000000000 0000000000000000

ffff88062b2cc140

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

0000000000000000

[381920.902435] Call Trace:
[381920.904969]  <NMI>
[381920.907201]  [<ffffffffa002b21f>]

ghes_read_estatus+0x2f/0x170

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

[ghes]

Quoted 1 line

[381920.913726]  [<ffffffffa002b618>]

ghes_notify_nmi+0xd8/0x1b0

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

[ghes]

Quoted 1 line

[381920.920076]  [<ffffffff8151a14f>]

notifier_call_chain+0x3f/0x80

Quoted 18 lines

Quoted 18 lines

Quoted 18 lines

Quoted 18 lines

Quoted 1 line

[381920.926080]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381920.932864]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20

Quoted 10 lines

[381920.939473]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381920.944691]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381920.949649]  [<ffffffff81517550>] nmi+0x20/0x30
[381920.954269]  [<ffffffff8102ae8a>] ?
native_write_msr_safe+0xa/0x10 [381920.960536]  <<EOE>>
[381920.962725]  <IRQ>
[381920.965132]  [<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120 [381920.971399]
[<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50 [381920.977066]
[<ffffffff810e7f2b>] perf_pmu_disable+0x2b/0x40 [381920.982817]

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

[<ffffffff810ee828>] perf_event_task_tick+0x218/0x270
[381920.989082]  [<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0
[381920.994740]  [<ffffffff81067097>]
update_process_times+0x67/0x80 [381921.000832]
[<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0 [381921.006578]
[<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100
[381921.012758]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.018414]  [<ffffffff8107dea7>]

hrtimer_interrupt+0xb7/0x1e0

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

[381921.024343]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0

Quoted 1 line

[381921.030793]  [<ffffffff8151e3f3>]

apic_timer_interrupt+0x13/0x20

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

[381921.036891]  <EOI>
[381921.038913] Code: fc 10 74 1f 77 08 41 80 fc 08 75 48 eb 0e

41

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

80

fc 20 74

Quoted 1 line

17 41 80 fc 40 75 3a eb 15 8a 00 0f b6 c0 eb 11 66 8b 00 0f b7

c0 eb

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

09 <8b>

Quoted 3 lines

00 89 c0 eb 03 48 8b 00 49 89 45 00 e8 8e 2b e2 ff eb 1b 0f

[381921.059462] RIP  [<ffffffff812a7510>]

acpi_atomic_read+0xe3/0x120 [381921.065669] RSP

<ffff88063fc47d98>

Quoted 17 lines

Quoted 17 lines

Quoted 17 lines

Quoted 17 lines

Quoted 3 lines

[381921.069242] CR2: 0000000000000000
[381921.072645] ---[ end trace 52697bfc73a34a90 ]---
[381921.077343] Kernel panic - not syncing: Fatal exception in

interrupt

Quoted 1 line

[381921.083784] Pid: 12701, comm: cluster Tainted: G      D

2.6.39.3-

Quoted 6 lines

microwaycustom #1

[381921.091788] Call Trace:
[381921.094333]  <NMI>  [<ffffffff81513300>] panic+0x9f/0x1da
[381921.099881]  [<ffffffff81517f8c>] oops_end+0xdc/0xf0
[381921.104952]  [<ffffffff81032b91>] no_context+0xf1/0x260
[381921.110277]  [<ffffffff81032e55>]

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

__bad_area_nosemaphore+0x155/0x200

Quoted 1 line

[381921.116739]  [<ffffffff81032f0e>]

bad_area_nosemaphore+0xe/0x10
> > >> > [381921.122766]  [<ffffffff81519f46>] do_page_fault+0x366/0x530

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 1 line

[381921.128440]  [<ffffffff810ee929>] ?

__perf_event_overflow+0xa9/0x220

Quoted 1 line

[381921.134896]  [<ffffffff810ef99b>] ?

perf_event_update_userpage+0x9b/0xe0

Quoted 1 line

[381921.141698]  [<ffffffff81012249>] ?

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

intel_pmu_enable_all+0xc9/0x1a0

Quoted 1 line

[381921.148061]  [<ffffffff810123ff>] ?

x86_perf_event_set_period+0xdf/0x170

Quoted 2 lines

[381921.154852]  [<ffffffff81517295>] page_fault+0x25/0x30
[381921.160081]  [<ffffffff812a7510>] ?

acpi_atomic_read+0xe3/0x120

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

[381921.166094]  [<ffffffff812a7486>] ?

acpi_atomic_read+0x59/0x120

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

[381921.172107]  [<ffffffffa002b21f>]

ghes_read_estatus+0x2f/0x170

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

[ghes]

Quoted 1 line

[381921.178639]  [<ffffffffa002b618>]

ghes_notify_nmi+0xd8/0x1b0

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

[ghes]

Quoted 1 line

[381921.185004]  [<ffffffff8151a14f>]

notifier_call_chain+0x3f/0x80

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

[381921.191003]  [<ffffffff8151a1d8>]

__atomic_notifier_call_chain+0x48/0x70

Quoted 1 line

[381921.197789]  [<ffffffff8151a211>]

atomic_notifier_call_chain+0x11/0x20

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

[381921.204399]  [<ffffffff8151a24e>] notify_die+0x2e/0x30
[381921.209624]  [<ffffffff81517bd2>] do_nmi+0xa2/0x270
[381921.214584]  [<ffffffff81517550>] nmi+0x20/0x30
[381921.219208]  [<ffffffff8102ae8a>] ?
native_write_msr_safe+0xa/0x10 [381921.225470]  <<EOE>>  <IRQ>
[<ffffffff8101131e>]
intel_pmu_disable_all+0x3e/0x120
[381921.233174]  [<ffffffff81010d5a>] x86_pmu_disable+0x4a/0x50
[381921.238836]  [<ffffffff810e7f2b>]

perf_pmu_disable+0x2b/0x40

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

[381921.244582]  [<ffffffff810ee828>]
perf_event_task_tick+0x218/0x270 [381921.250855]
[<ffffffff81046f4d>] scheduler_tick+0xdd/0x2c0 [381921.256520]
[<ffffffff81067097>] update_process_times+0x67/0x80

[381921.262618]

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

[<ffffffff81089eef>] tick_sched_timer+0x5f/0xc0

[381921.268363]

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

[<ffffffff81089e90>] ? tick_nohz_handler+0x100/0x100

[381921.274540]  [<ffffffff8107db3d>] __run_hrtimer+0x12d/0x280
[381921.280198]  [<ffffffff8107dea7>]

hrtimer_interrupt+0xb7/0x1e0

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

[381921.286125]  [<ffffffff810206d7>]

smp_apic_timer_interrupt+0x67/0xa0

Quoted 1 line

[381921.292578]  [<ffffffff8151e3f3>]

apic_timer_interrupt+0x13/0x20

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

[381921.298673]  <EOI>

I recompiled the kernel again, disabling the tickless feature

as I

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

saw

"nohz"

Quoted 1 line

early in the call trace. After that, it reproduced again but

the

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

call

trace

Quoted 1 line

only changed slightly having tick_init_highres in there instead

of

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

tick_nohz_handler. I have that call trace if it is desired as

well.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

It is

Quoted 3 lines

nearly identical though.

I am currently trying a custom 2.6.36.4 on the system, but will

need

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

up to 3

Quoted 3 lines

days before I know if the problem exists there as well.

Any ideas on this?

It looks like a wrong address in ghes_read_estatus(). I'll have

a

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

look at it
later today.

Hmm. Please apply the appended patch and see if the error is

still

Quoted 1 line

Quoted 1 line

Quoted 1 line

reproducible (it's on top of the current mainline, so you may need

to

Quoted 23 lines

Quoted 23 lines

Quoted 23 lines

adjust it by hand).

Thanks,
Rafael

---

 drivers/acpi/apei/ghes.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux/drivers/acpi/apei/ghes.c
===================================================================

drivers/acpi/apei/ghes.c +3 -0

@@ -398,6 +398,9 @@ static int ghes_read_estatus(struct ghes
 
 	u32 len;
 	int rc;
 
+	if (!g)
+		return -EINVAL;
+
 

Quoted 6 lines

Quoted 6 lines

drivers/acpi/apei/ghes.c static int ghes_read_estatus(struct ghes · line 406

Quoted 4 lines

drivers/acpi/apei/ghes.c +0 -0

 	rc = acpi_atomic_read(&buf_paddr, &g->error_status_address);

drivers/acpi/apei/ghes.c +0 -0

 	if (rc) {
 
 		if (!silent && printk_ratelimit())

Thanks for the quick patch! Unfortunately, the system crashed again

with

Quoted 1 line

Quoted 1 line

the same call trace. I'll list it below in case there are any

differences

Quoted 1 line

Quoted 1 line

I missed. I have currently added a test for

(!&g->error_status_address)

Quoted 1 line

Quoted 1 line

just like your test on (!g) to verify that isn't a null pointer

either,

Quoted 1 line

Quoted 1 line

and will start tests back up with that change. Please let me know

if you

Quoted 6 lines

Quoted 1 line

have any other suggestions or patches for me to try.

That was a long shot, but perhaps you can check what source code corresponds to ghes_read_estatus+0x38 in your kernel?

Rafael

Hi Rafael,

Thanks for the off-list help in getting you this info.

I had already rebuilt the kernel using the change I mentioned earlier (test on
!&g->error_status_address) since the call trace I got.

I luckily still had a copy of the kernel and modules I built previously using
just your patch, so I undid my change to the ghes.c source, leaving just your
patch but not mine so it would match the ghes.ko module I ran on. This is the
output of gdb on that ghes.ko now:

(gdb) l *ghes_read_estatus+0x38
0x258 is in ghes_read_estatus (drivers/acpi/apei/ghes.c:296).
warning: Source file is more recent than executable.
291 int rc;
292 if (!g)
293 return -EINVAL;
294
295 rc = acpi_atomic_read(&buf_paddr,
&g->error_status_address);
296 if (rc) {
297 if (!silent && printk_ratelimit())
298 pr_warning(FW_WARN GHES_PFX
299 "Failed to read error status block address for hardware error
source:
%d.\n",
300 g->header.source_id);

The warning about the source being newer is because of the reverted change in
the ghes.c source mentioned above.

Thanks,
Rick
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi, Rick,

It appears that panic occurs in acpi_atomic_read. I think the most likely cause is that the acpi_generic_address is not pre-mapped. Can you try the patch attached?

It will print registers mapped and accessed. To use it, run the following command line before workload.

dmesg | grep GHES

Then try to find something like

GHES: gar accessed: x, xxxx

in kernel log when panic occurs.

Best Regards,
Huang Ying

Hi Huang,

The original system needs to ship to our customer ASAP. Disabling ghes is sufficient for the time being for that. As such, I have set up an identical system as a temporary master for another cluster to continue this testing.

I have applied your patch. Here is the output of dmesg | grep GHES so far:

[    9.272198] GHES: gar mapped: 0, 0xbf7b5ff0
[    9.280782] GHES: gar mapped: 0, 0xbf7b6200
[    9.285102] [Firmware Warn]: GHES: Poll interval is 0 for generic

hardware error source: 1, disabled.

I have the serial console activated and stress tests started back up. I'll reply with the output once I get another panic.

Thanks!
Rick

Quoted 21 lines

Hi, Rick,

It appears that panic occurs in acpi_atomic_read. I think the most likely cause is that the acpi_generic_address is not pre-mapped. Can you try the patch attached?

It will print registers mapped and accessed. To use it, run the following command line before workload.

dmesg | grep GHES

Then try to find something like

GHES: gar accessed: x, xxxx

in kernel log when panic occurs.

Best Regards,
Huang Ying

Hi Huang,

My new setup reproduced the panic. However I do not have any gar accessed messages on it. The gar mapped messages are in my previous email. Here is the latest call trace. There is no GHES output prior to it:

Quoted 7 lines

Hi Huang,

My new setup reproduced the panic. However I do not have any gar accessed
messages on it. The gar mapped messages are in my previous email. Here is the latest call trace. There is no GHES output prior to it:

That is wired. Can you try the patch attached? If my guessing is correct, there will be no panic, but something as follow will be in dmesg:

ACPI atomic read mem: addr 0xxxxx mapped to 0

Best Regards,
Huang Ying

Hi Huang,

Sorry for the delay in my response. Hurricane Irene delayed our testing a bit.

I had to switch the 5620 CPUS I had for 5670s. After 4 days of running (it was usually about 2 before) I finally got this output in dmesg:

[337296.365930] GHES: gar accessed: 0, 0xbf7b9370
[337296.365936] ACPI atomic read mem: addr 0xbf7b9370 mapped to

ffffc90013ee8370

It is not mapped to 0 as expected, but it didn't crash now!

Quoted 6 lines

Hi Huang,

Sorry for the delay in my response. Hurricane Irene delayed our testing a
bit.

Quoted 8 lines

I had to switch the 5620 CPUS I had for 5670s. After 4 days of running (it was usually about 2 before) I finally got this output in dmesg:

[337296.365930] GHES: gar accessed: 0, 0xbf7b9370
[337296.365936] ACPI atomic read mem: addr 0xbf7b9370 mapped to

ffffc90013ee8370

It is not mapped to 0 as expected, but it didn't crash now!

But I don't think this patch fixed the issue. Maybe just hided the issue. Do you have time to try the new patch attached?

Best Regards,
Huang Ying

Hi Huang,

I've been running for a while now with this new patch. I have not gotten any panic output though. This is the only output I have from GHES:

[22256.616069] GHES: gar accessed: 0, 0xbf7b9370
[102115.873369] GHES: gar accessed: 0, 0xbf7b9370
[175460.888809] GHES: gar accessed: 0, 0xbf7b9370
[223652.976925] GHES: gar accessed: 0, 0xbf7b9370
[284678.016631] GHES: gar accessed: 0, 0xbf7b9370
[321563.363578] GHES: gar accessed: 0, 0xbf7b9370
[500789.731559] GHES: gar accessed: 0, 0xbf7b9370
[539808.495653] GHES: gar accessed: 0, 0xbf7b9370
[545116.419274] GHES: gar accessed: 0, 0xbf7b9370
[686744.665292] GHES: gar accessed: 0, 0xbf7b9370

Do you think changing from 5620-5670 cpus changed the behavior?