Skip to contentSkip to conversation

mmotm 2008-12-01-19-41: early exception (page fault -- deref of 0x20)

DISCUSSION Jiri Slaby 5 participants 10 messages Dec 2, 2008 – Dec 22, 2008

lore
Share & tools
lore

Jiri Slaby

author
#0 lore raw

Hi,

while writing out warning about empty mtrr (or any other in the early stage), I'm getting this:
PANIC: early exception 0e rip 10:ffffffff8025be26 error 0 cr2 20
Pid: 0, comm: swapper Not tainted 2.6.28-rc6-mm1_64 #484

Call Trace:
 [<ffffffff80770195>] early_idt_handler+0x55/0x69
 [<ffffffff8025be26>] ? getnstimeofday+0x46/0xc0
 [<ffffffff80258a81>] ktime_get_real+0x11/0x50
 [<ffffffff803aa8d1>] seed_std_data+0x11/0x30
 [<ffffffff8020ff60>] ? show_trace+0x10/0x20
 [<ffffffff803aa900>] seed_random_pools+0x10/0x30
 [<ffffffff8023dda9>] init_oops_id+0x9/0x40
 [<ffffffff8023dde9>] print_oops_end_marker+0x9/0x20
 [<ffffffff8023dfed>] warn_slowpath+0x9d/0xd0
 [<ffffffff802595e4>] ? up+0x34/0x50
 [<ffffffff8023e77d>] ? release_console_sem+0x1bd/0x210
 [<ffffffff80592c38>] ? printk+0x3c/0x44
 [<ffffffff80776e67>] mtrr_trim_uncached_memory+0x166/0x390
 [<ffffffff8077fca0>] ? early_gart_iommu_check+0xaf/0x2a7
 [<ffffffff80772574>] setup_arch+0x3ec/0x6a8
 [<ffffffff80770adf>] start_kernel+0x65/0x3c6
 [<ffffffff8077027d>] x86_64_start_reservations+0x7d/0x89
 [<ffffffff80770384>] x86_64_start_kernel+0xd8/0xdf

RIP 0x10

The clock is still null in kernel/time/timekeeping.c.

This one is to blame:
random-add-a-way-to-get-some-random-bits-into-the-entropy-pools-early-on.patch

#1 lore raw
Quoted 25 lines · On Wed, 03 Dec 2008 00:09:46 +0100 Jiri Slaby <[email protected]> wrote:

Hi,

while writing out warning about empty mtrr (or any other in the early stage), I'm getting this:
PANIC: early exception 0e rip 10:ffffffff8025be26 error 0 cr2 20
Pid: 0, comm: swapper Not tainted 2.6.28-rc6-mm1_64 #484

Call Trace:
 [<ffffffff80770195>] early_idt_handler+0x55/0x69
 [<ffffffff8025be26>] ? getnstimeofday+0x46/0xc0
 [<ffffffff80258a81>] ktime_get_real+0x11/0x50
 [<ffffffff803aa8d1>] seed_std_data+0x11/0x30
 [<ffffffff8020ff60>] ? show_trace+0x10/0x20
 [<ffffffff803aa900>] seed_random_pools+0x10/0x30
 [<ffffffff8023dda9>] init_oops_id+0x9/0x40
 [<ffffffff8023dde9>] print_oops_end_marker+0x9/0x20
 [<ffffffff8023dfed>] warn_slowpath+0x9d/0xd0
 [<ffffffff802595e4>] ? up+0x34/0x50
 [<ffffffff8023e77d>] ? release_console_sem+0x1bd/0x210
 [<ffffffff80592c38>] ? printk+0x3c/0x44
 [<ffffffff80776e67>] mtrr_trim_uncached_memory+0x166/0x390
 [<ffffffff8077fca0>] ? early_gart_iommu_check+0xaf/0x2a7
 [<ffffffff80772574>] setup_arch+0x3ec/0x6a8
 [<ffffffff80770adf>] start_kernel+0x65/0x3c6
 [<ffffffff8077027d>] x86_64_start_reservations+0x7d/0x89
 [<ffffffff80770384>] x86_64_start_kernel+0xd8/0xdf
Quoted 6 lines

RIP 0x10

The clock is still null in kernel/time/timekeeping.c.

This one is to blame:
random-add-a-way-to-get-some-random-bits-into-the-entropy-pools-early-on.patch

urgh, OK, thanks, I'll drop it.

Presumably your machine picked a different clocksource from Arjan's, mine and others. Now, how do we work out what clocksource you're using?

dmesg -s 1000000 | grep -i clock
nope
dmesg -s 1000000 | grep -i using
nope

hrm.

#9 lore raw
In reply to Jiri Slaby
Quoted 25 lines · Jiri Slaby

Hi,

while writing out warning about empty mtrr (or any other in the early stage), I'm getting this:
PANIC: early exception 0e rip 10:ffffffff8025be26 error 0 cr2 20
Pid: 0, comm: swapper Not tainted 2.6.28-rc6-mm1_64 #484

Call Trace:
 [<ffffffff80770195>] early_idt_handler+0x55/0x69
 [<ffffffff8025be26>] ? getnstimeofday+0x46/0xc0
 [<ffffffff80258a81>] ktime_get_real+0x11/0x50
 [<ffffffff803aa8d1>] seed_std_data+0x11/0x30
 [<ffffffff8020ff60>] ? show_trace+0x10/0x20
 [<ffffffff803aa900>] seed_random_pools+0x10/0x30
 [<ffffffff8023dda9>] init_oops_id+0x9/0x40
 [<ffffffff8023dde9>] print_oops_end_marker+0x9/0x20
 [<ffffffff8023dfed>] warn_slowpath+0x9d/0xd0
 [<ffffffff802595e4>] ? up+0x34/0x50
 [<ffffffff8023e77d>] ? release_console_sem+0x1bd/0x210
 [<ffffffff80592c38>] ? printk+0x3c/0x44
 [<ffffffff80776e67>] mtrr_trim_uncached_memory+0x166/0x390
 [<ffffffff8077fca0>] ? early_gart_iommu_check+0xaf/0x2a7
 [<ffffffff80772574>] setup_arch+0x3ec/0x6a8
 [<ffffffff80770adf>] start_kernel+0x65/0x3c6
 [<ffffffff8077027d>] x86_64_start_reservations+0x7d/0x89
 [<ffffffff80770384>] x86_64_start_kernel+0xd8/0xdf
Quoted 6 lines

RIP 0x10

The clock is still null in kernel/time/timekeeping.c.

This one is to blame:
random-add-a-way-to-get-some-random-bits-into-the-entropy-pools-early-on.patch

Nice. I didn't like that patch anyway.

Seems to me we really need to simplify all our startup dependencies. Some things should -always- work. Time is one of them, random number generation may be another, memory allocation ought to be one as well.

So I think:

a) we need to come up with a solution/rule for early ktime faulting
b) the RNG should initialize itself on demand
c) seeding the RNG should include a (void *, len)

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#2 lore raw
In reply to Andrew Morton
Quoted 37 lines · Andrew Morton
Quoted 25 lines · On Wed, 03 Dec 2008 00:09:46 +0100 Jiri Slaby <[email protected]> wrote:

Hi,

while writing out warning about empty mtrr (or any other in the early stage), I'm getting this:
PANIC: early exception 0e rip 10:ffffffff8025be26 error 0 cr2 20
Pid: 0, comm: swapper Not tainted 2.6.28-rc6-mm1_64 #484

Call Trace:
 [<ffffffff80770195>] early_idt_handler+0x55/0x69
 [<ffffffff8025be26>] ? getnstimeofday+0x46/0xc0
 [<ffffffff80258a81>] ktime_get_real+0x11/0x50
 [<ffffffff803aa8d1>] seed_std_data+0x11/0x30
 [<ffffffff8020ff60>] ? show_trace+0x10/0x20
 [<ffffffff803aa900>] seed_random_pools+0x10/0x30
 [<ffffffff8023dda9>] init_oops_id+0x9/0x40
 [<ffffffff8023dde9>] print_oops_end_marker+0x9/0x20
 [<ffffffff8023dfed>] warn_slowpath+0x9d/0xd0
 [<ffffffff802595e4>] ? up+0x34/0x50
 [<ffffffff8023e77d>] ? release_console_sem+0x1bd/0x210
 [<ffffffff80592c38>] ? printk+0x3c/0x44
 [<ffffffff80776e67>] mtrr_trim_uncached_memory+0x166/0x390
 [<ffffffff8077fca0>] ? early_gart_iommu_check+0xaf/0x2a7
 [<ffffffff80772574>] setup_arch+0x3ec/0x6a8
 [<ffffffff80770adf>] start_kernel+0x65/0x3c6
 [<ffffffff8077027d>] x86_64_start_reservations+0x7d/0x89
 [<ffffffff80770384>] x86_64_start_kernel+0xd8/0xdf
Quoted 6 lines

RIP 0x10

The clock is still null in kernel/time/timekeeping.c.

This one is to blame:
random-add-a-way-to-get-some-random-bits-into-the-entropy-pools-early-on.patch

urgh, OK, thanks, I'll drop it.

Quoted 8 lines

Presumably your machine picked a different clocksource from Arjan's, mine and others. Now, how do we work out what clocksource you're using?

dmesg -s 1000000 | grep -i clock
nope
dmesg -s 1000000 | grep -i using
nope

If we oops or warn while picking a timesource, we'll have lots of fun?

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#3 lore raw
Quoted 3 lines · Matt Mackall

If we oops or warn while picking a timesource, we'll have lots of fun?

we really only need to mix in the tsc; ktime_get() is just an arch friendly way to get that I supposed (wrongly).

but yes we need to do a few things
1) seed on demand with a platform time source
2) have a way where arch init can just hand semi random data during the boot process to increase the randomness (even if it doesn't count as entropy)
I got pulled in some big project at work so I might not get to it this week, but if nobody beats me to it I will get to it ;-(

#6 lore raw
In reply to Andrew Morton
Quoted 7 lines · Andrew Morton

Presumably your machine picked a different clocksource from Arjan's, mine and others. Now, how do we work out what clocksource you're using? dmesg -s 1000000 | grep -i clock
nope
dmesg -s 1000000 | grep -i using
nope
hrm.

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

Signature · 2 lines
Jiri Kosina
SUSE Labs
#4 lore raw
In reply to Arjan van de Ven
Quoted 10 lines · Arjan van de Ven
Quoted 3 lines · Matt Mackall

If we oops or warn while picking a timesource, we'll have lots of fun?

we really only need to mix in the tsc; ktime_get() is just an arch friendly way to get that I supposed (wrongly).

but yes we need to do a few things
1) seed on demand with a platform time source

Currently we use jiffies + get_cycles(). That's going to have somewhere between, oh, 3 bits of entropy (very stable boot with only jiffies) and 25 bits of entropy (TSC with lots of waiting for hardware) at boot. Ideally, we'd have access to a wall clock of some sort as well. But that's also a fairly limited source - wall clocks are both low resolution and predictable/collision-prone.

Quoted 2 lines

2) have a way where arch init can just hand semi random data during the boot process to increase the randomness (even if it doesn't count as entropy)

A simple wrapper around mix_pool_bytes probably fits the ticket.

But I don't think this will solve the general problem of 'large numbers of practically identical machines booting up with the same pre-init random number pools'. Beyond things like MAC addresses and serial numbers (predictable/observable but at least not collision-prone), we have no way to differentiate some boxes. We may need to forcibly generate some timing entropy. Perhaps something like this:

http://markmail.org/message/xwsbywr6ziil2qu2

(which is way too slow in its current form)

There's a related problem of systems with no way to store a seed across boots.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#5 lore raw
Quoted 7 lines · Matt Mackall
Quoted 6 lines · Arjan van de Ven
Quoted 2 lines · Matt Mackall

If we oops or warn while picking a timesource, we'll have lots of fun?

we really only need to mix in the tsc; ktime_get() is just an arch friendly way to get that I supposed (wrongly).

Quoted 8 lines
Quoted 2 lines

but yes we need to do a few things
1) seed on demand with a platform time source

Currently we use jiffies + get_cycles(). That's going to have somewhere
between, oh, 3 bits of entropy (very stable boot with only jiffies) and
25 bits of entropy (TSC with lots of waiting for hardware) at boot. Ideally, we'd have access to a wall clock of some sort as well. But that's also a fairly limited source - wall clocks are both low
resolution and predictable/collision-prone.

tsc taken quite some time appart in the boot, we will get multiple times a bunch of entropy. This comes from variations in time various hardware things take (due to spread spectrum clocks) and because of cpu speculation not being deterministic to this level of multi-milliseconds.

author
#7 lore raw
In reply to Jiri Kosina
Quoted 11 lines · Jiri Kosina
Quoted 7 lines · Andrew Morton

Presumably your machine picked a different clocksource from Arjan's, mine and others. Now, how do we work out what clocksource you're using? dmesg -s 1000000 | grep -i clock
nope
dmesg -s 1000000 | grep -i using
nope
hrm.

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

tsc

It ran from qemu, but it oopsed even on my desktop (tsc too).

#8 lore raw
Quoted 10 lines · Jiri Slaby
Quoted 8 lines
Quoted 7 lines

Presumably your machine picked a different clocksource from Arjan's, mine and others. Now, how do we work out what clocksource you're using? dmesg -s 1000000 | grep -i clock
nope
dmesg -s 1000000 | grep -i using
nope
hrm.

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

tsc
It ran from qemu, but it oopsed even on my desktop (tsc too).

Could you please send the full oops to Thomas and Ingo?

Signature · 2 lines
Jiri Kosina
SUSE Labs