Skip to contentSkip to conversation

Oops running oprofile in 2.5.62

DISCUSSION Dave Hansen 3 participants 4 messages Feb 27, 2003 – Mar 5, 2003

lore
Share & tools
lore
#0 lore raw

This happened while running dbench on 2.5.62. I haven't seen it before, but I thought I'd report it anyway. I'm using the 0.5 version of the userspace tools.

I'm pretty sure it happened on this line in oprofile_add_sample():

	cpu_buf->buffer[cpu_buf->pos].eip = eip;
Unable to handle kernel paging request at virtual address f8c3c000
c0212022
*pde = 00000000
Oops: 0002
CPU:    13
EIP:    0060:[<c0212022>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010046
eax: 40082d94 ebx: 00000340 ecx: 00002000 edx: f8c2c000
esi: 00000000 edi: c0310f00 ebp: 00000000 esp: e2fbdf60
ds: 007b es: 007b ss: 0068

Stack: 00000000 00000000 c03a22c8 c02139f0 40082d94 00000000 00000000
       0000000d e2fbc000 e2fbdfc4 bffff7a0 bffffbb8 40082d94 c0213291
       0000000d c03a22c8 e2fbdfc4 c010a1eb e2fbdfc4 0000000d 40122020
       00000000 c010962e e2fbdfc4
Call Trace:
 [<c02139f0>] ppro_check_ctrs+0x4c/0x80
 [<c0213291>] nmi_callback+0x21/0x28
 [<c010a1eb>] do_nmi+0x2b/0x48
 [<c010962e>] nmi+0x1e/0x30
Code: 89 04 ca 8b 8b c4 0b 31 c0 8b 93 d0 0b 31 c0 8b 44 24 18 89

p

Quoted 3 lines
Quoted 3 lines

EIP; c0212022 <oprofile_add_sample+102/128> <=====

edi; c0310f00 <cpu_buffer+340/800>

Trace; c02139f0 <ppro_check_ctrs+4c/80>
Trace; c0213291 <nmi_callback+21/28>
Trace; c010a1eb <do_nmi+2b/48>
Trace; c010962e <nmi+1e/30>

Code; c0212022 <oprofile_add_sample+102/128>
00000000 <_EIP>:
Code; c0212022 <oprofile_add_sample+102/128> <=====
0: 89 04 ca mov %eax,(%edx,%ecx,8) <=====
Code; c0212025 <oprofile_add_sample+105/128>
3: 8b 8b c4 0b 31 c0 mov 0xc0310bc4(%ebx),%ecx
Code; c021202b <oprofile_add_sample+10b/128>
9: 8b 93 d0 0b 31 c0 mov 0xc0310bd0(%ebx),%edx
Code; c0212031 <oprofile_add_sample+111/128>
f: 8b 44 24 18 mov 0x18(%esp,1),%eax
Code; c0212035 <oprofile_add_sample+115/128>
13: 89 00 mov %eax,(%eax)

Signature · 2 lines
Dave Hansen
[email protected]
#1 lore raw
Quoted 4 lines · Dave Hansen

I'm pretty sure it happened on this line in oprofile_add_sample():

	cpu_buf->buffer[cpu_buf->pos].eip = eip;
Unable to handle kernel paging request at virtual address f8c3c000

Odd. UP or SMP ? Were you shutting down oprofile at the time (or did the daemon crash ? check /var/lib/oprofile/oprofiled.log)

The only thing I can think is that the buffer got freed then we got an NMI afterwards (which obviously isn't supposed to happen...)

UP oprofile in current linus is still totally broken, but I can't imagine it causing the above.

regards
john

#2 lore raw
Quoted 9 lines · John Levon
Quoted 4 lines · Dave Hansen

I'm pretty sure it happened on this line in oprofile_add_sample():

	cpu_buf->buffer[cpu_buf->pos].eip = eip;
Unable to handle kernel paging request at virtual address f8c3c000

Odd. UP or SMP ? Were you shutting down oprofile at the time (or did the daemon crash ? check /var/lib/oprofile/oprofiled.log)

16-way NUMAQ :) It was in the middle of a dbench run, so nothing critical was happening with oprofile.

Quoted 2 lines

The only thing I can think is that the buffer got freed then we got an NMI afterwards (which obviously isn't supposed to happen...)

The irqbalance code is causing some funniness on these machines, so it might be related. I'll speak up if I see it again.

Signature · 2 lines
Dave Hansen
[email protected]
#3 lore raw
In reply to Dave Hansen
Quoted 6 lines · Dave Hansen

This happened while running dbench on 2.5.62. I haven't seen it before, but I thought I'd report it anyway. I'm using the 0.5 version of the userspace tools.

I'm pretty sure it happened on this line in oprofile_add_sample():

	cpu_buf->buffer[cpu_buf->pos].eip = eip;

yes, in the last chunk of code in oprofile_add_sample()

Quoted 9 lines
Unable to handle kernel paging request at virtual address f8c3c000
c0212022
*pde = 00000000
Oops: 0002
CPU:    13
EIP:    0060:[<c0212022>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010046
eax: 40082d94 ebx: 00000340 ecx: 00002000 edx: f8c2c000

                                       ^^^^^^^^

buffer overrrun by one entry (8192 entry by default of 8 bytes each on x86), potentially oprofile_add_sample() add 3 events in buffer but the protection at begin of code protect against two addition not three

The bug is rare because add_sample use three entry in rare case, and thing are wrong only when cpu_buf->pos == buffer_size - 2 the code is not fixed in 2.5.64, John if you have not already a patch pending for this can you push it in your tree ?

void oprofile_add_sample(unsigned long eip, unsigned int is_kernel,
unsigned long event, int cpu)
.....
- if (cpu_buf->pos > buffer_size - 2) {

+ if (cpu_buf->pos > buffer_size - 3) {
		cpu_buf->sample_lost_overflow++;
		goto out;
	}
Quoted 11 lines
Quoted 5 lines
Quoted 4 lines

EIP; c0212022 <oprofile_add_sample+102/128> <=====

edi; c0310f00 <cpu_buffer+340/800>

Trace; c02139f0 <ppro_check_ctrs+4c/80>
Trace; c0213291 <nmi_callback+21/28>
Trace; c010a1eb <do_nmi+2b/48>
Trace; c010962e <nmi+1e/30>

Quoted 4 lines

Code; c0212022 <oprofile_add_sample+102/128>
00000000 <_EIP>:
Code; c0212022 <oprofile_add_sample+102/128> <=====
0: 89 04 ca mov %eax,(%edx,%ecx,8) <=====

ecx == cpu_bufffer->pos == buffer_size ... boom ...

regards,
Philippe Elie