2.6.25-rc3 hangs in early boot on Sun Ultra5
Booting 2.6.25-rc3 on my Ultra5 causes a hang before or as the console is switched over to the framebuffer. The console output is (extrapolated from dmesg in -rc2 and handwritten notes, as I don't have a serial cable to my U5):
PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
PROMLIB: Root node compatible:
*** the following line can't be seen in dmesg after rc2 has booted
console [earlyprom0] enabled
Linux version 2.6.25-rc3 (mikpe@sparge) (gcc version 4.2.3) #1 Mon Feb 25 18:49:41 CET 2008
ARCH: SUN4U
Ethernet address: 08:00:20:fd:ec:1f
[0000000200000000-fffff80000400000] page_structs&2144 node=0 entry=0/0 [0000000200000000-fffff80000800000] page_structs&2144 node=0 entry=1/0 [0000000200000000-fffff80000c00000] page_structs&2144 node=0 entry=2/0 [0000000200000000-fffff80001000000] page_structs&2144 node=0 entry=3/0
OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
PROM: Built device tree with 46617 bytes of memory.
On node 0 totalpages: 32299
Normal zone: 335 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 31964 pages, LIFO batch:7
Movable zone: 0 pages used for memmap
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 31964 Kernel command line: ro root=/dev/sda5
PID hash table entries: 1024 (order: 10, 8192 bytes)
clocksource: mult[28000] shift[16]
clockevent: mult[66666666] shift[32]
Console: colour dummy device 80x25
*** the following line can't be seen in dmesg after rc2 has booted
console handover: boot [earlyprom0] -> real [tty0]
At this point rc3 hangs hard and won't even respond to sysrq.
Another difference is that with rc2 the first few lines of kernel output while the console is still in OF mode either aren't shown or disappear quickly since the switch to the framebuffer occurs within a fraction of a second after the kernel has been loaded. With rc3 the kernel output (the text shown above) in the OF-mode console is very very slow.
(I should have quoted my .config here but I forgot to bring it. It's exactly the same in rc2 and rc3, however.)
I'll try some rc2->rc3 bisecting later.
/Mikael
Quoted 41 lines · Mikael Pettersson
Booting 2.6.25-rc3 on my Ultra5 causes a hang before or as the console is switched over to the framebuffer. The console output is (extrapolated from dmesg in -rc2 and handwritten notes, as I don't have a serial cable to my U5):
PROMLIB: Sun IEEE Boot Prom 'OBP 3.25.3 2000/06/29 14:12'
PROMLIB: Root node compatible:
*** the following line can't be seen in dmesg after rc2 has booted
console [earlyprom0] enabled
Linux version 2.6.25-rc3 (mikpe@sparge) (gcc version 4.2.3) #1 Mon Feb 25 18:49:41 CET 2008
ARCH: SUN4U
Ethernet address: 08:00:20:fd:ec:1f
[0000000200000000-fffff80000400000] page_structs&2144 node=0 entry=0/0 [0000000200000000-fffff80000800000] page_structs&2144 node=0 entry=1/0 [0000000200000000-fffff80000c00000] page_structs&2144 node=0 entry=2/0 [0000000200000000-fffff80001000000] page_structs&2144 node=0 entry=3/0
OF stdout device is: /pci@1f,0/pci@1,1/SUNW,m64B@2
PROM: Built device tree with 46617 bytes of memory.
On node 0 totalpages: 32299
Normal zone: 335 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 31964 pages, LIFO batch:7
Movable zone: 0 pages used for memmap
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 31964 Kernel command line: ro root=/dev/sda5
PID hash table entries: 1024 (order: 10, 8192 bytes)
clocksource: mult[28000] shift[16]
clockevent: mult[66666666] shift[32]
Console: colour dummy device 80x25
*** the following line can't be seen in dmesg after rc2 has booted
console handover: boot [earlyprom0] -> real [tty0]At this point rc3 hangs hard and won't even respond to sysrq.
Another difference is that with rc2 the first few lines of kernel output while the console is still in OF mode either aren't shown or disappear quickly since the switch to the framebuffer occurs within a fraction of a second after the kernel has been loaded. With rc3 the kernel output (the text shown above) in the OF-mode console is very very slow.
Quoted 4 lines
(I should have quoted my .config here but I forgot to bring it. It's exactly the same in rc2 and rc3, however.)
I'll try some
rc2->rc3bisecting later.
Minor update: rc2-git7 has the slow initial console behaviour, but successfully switches to the framebuffer. rc2-git8 however hangs in the console handover. So I'll bisect git7->git8 next.
/Mikael
From: Mikael Pettersson <[email protected]>
Date: Tue, 26 Feb 2008 09:55:50 +0100
Quoted 3 lines
Minor update: rc2-git7 has the slow initial console behaviour, but successfully switches to the framebuffer. rc2-git8 however hangs in the console handover. So I'll bisect
git7->git8next.
Thanks for doing this research.
David Miller
Acked-by Ingo MolnarFrom: Mikael Pettersson <[email protected]>
Date: Tue, 26 Feb 2008 09:55:50 +0100
Quoted 3 lines
Minor update: rc2-git7 has the slow initial console behaviour, but successfully switches to the framebuffer. rc2-git8 however hangs in the console handover. So I'll bisect
git7->git8next.
Between the VT layer registering it's console and the atyfb driver initializing we get a crash, and it happens on all sparc64 systems. It is caused by this commit and I am working on a fix:
commit a0c1e9073ef7428a14309cba010633a6cd6719ea
Author: Thomas Gleixner <[email protected]> Date: Sat Feb 23 15:23:57 2008 -0800
futex: runtime enable pi and robust functionality
Not all architectures implement futex_atomic_cmpxchg_inatomic(). The default implementation returns -ENOSYS, which is currently not handled inside of the futex guts.
Futex PI calls and robust list exits with a held futex result in an endless loop in the futex code on architectures which have no support.
Fixing up every place where futex_atomic_cmpxchg_inatomic() is called would add a fair amount of extra if/else constructs to the already complex code. It is also not possible to disable the robust feature before user space tries to register robust lists.
Compile time disabling is not a good idea either, as there are already architectures with runtime detection of futex_atomic_cmpxchg_inatomic support.
Detect the functionality at runtime instead by calling cmpxchg_futex_value_locked() with a NULL pointer from the futex initialization code. This is guaranteed to fail, but the call of futex_atomic_cmpxchg_inatomic() happens with pagefaults disabled.
On architectures, which use the asm-generic implementation or have a runtime CPU feature detection, a -ENOSYS return value disables the PI/robust features.
On architectures with a working implementation the call returns -EFAULT and the PI/robust features are enabled.
The relevant syscalls return -ENOSYS and the robust list exit code is blocked, when the detection fails.
Fixes http://lkml.org/lkml/2008/2/11/149
Originally reported by: Lennart Buytenhek
include/linux/futex.h +1 -0
@@ -167,6 +167,7 @@ union futex_key {#ifdef CONFIG_FUTEX extern void exit_robust_list (struct task_struct *curr);extern void exit_pi_state_list (struct task_struct *curr); +extern int futex_cmpxchg_enabled;#else static inline void exit_robust_list (struct task_struct *curr) {
kernel/futex.c +34 -4
@@ -60,6 +60,8 @@#include "rtmutex_common.h" +int __read_mostly futex_cmpxchg_enabled; +#define FUTEX_HASHBITS (CONFIG_BASE_SMALL ? 4 : 8)/*
@@ -469,6 +471,8 @@ void exit_pi_state_list(struct task_struct *curr)struct futex_hash_bucket *hb;union futex_key key; +if (!futex_cmpxchg_enabled) +return ;/* * We are a ZOMBIE and nobody can enqueue itself on * pi_state_list anymore, but we have to be careful
@@ -1870,6 +1874,8 @@ asmlinkage long sys_set_robust_list(struct robust_list_head __user *head, size_t len) { +if (!futex_cmpxchg_enabled) +return -ENOSYS ;/* * The kernel knows only one size for now: */
@@ -1894,6 +1900,9 @@ sys_get_robust_list(int pid, struct robust_list_head __user * __user *head_ptr,struct robust_list_head __user *head;unsigned long ret; +if (!futex_cmpxchg_enabled) +return -ENOSYS ; +if (!pid) head = current->robust_list;else {
@@ -1997,6 +2006,9 @@ void exit_robust_list(struct task_struct *curr)unsigned long futex_offset;int rc; +if (!futex_cmpxchg_enabled) +return ; +/* * Fetch the list head (which was registered earlier, via * sys_set_robust_list()):
@@ -2051,7 +2063,7 @@ void exit_robust_list(struct task_struct *curr)long do_futex (u32 __user *uaddr,int op,u32 val,ktime_t *timeout,u32 __user *uaddr2,u32 val2,u32 val3) { -int ret; +int ret = -ENOSYS ;int cmd = op &FUTEX_CMD_MASK ;struct rw_semaphore *fshared =NULL ;
@@ -2083,13 +2095,16 @@ long do_futex(u32 __user *uaddr, int op, u32 val, ktime_t *timeout, ret =futex_wake_op (uaddr, fshared, uaddr2, val, val2, val3);break ;case FUTEX_LOCK_PI : - ret =futex_lock_pi (uaddr, fshared, val, timeout,0 ); +if (futex_cmpxchg_enabled) + ret =futex_lock_pi (uaddr, fshared, val, timeout,0 );break ;case FUTEX_UNLOCK_PI : - ret =futex_unlock_pi (uaddr, fshared); +if (futex_cmpxchg_enabled) + ret =futex_unlock_pi (uaddr, fshared);break ;case FUTEX_TRYLOCK_PI : - ret =futex_lock_pi (uaddr, fshared,0 , timeout,1 ); +if (futex_cmpxchg_enabled) + ret =futex_lock_pi (uaddr, fshared,0 , timeout,1 );break ;default : ret = -ENOSYS ;
@@ -2145,8 +2160,23 @@ static struct file_system_type futex_fs_type = {static int __init init (void ) { +u32 curval;int i; +/* +* This will fail and we want it. Some arch implementations do +* runtime detection of the futex_atomic_cmpxchg_inatomic() +* functionality. We want to know that before we call in any +* of the complex code paths. Also we want to prevent +* registration of robust lists in that case. NULL is +* guaranteed to fault and we get -EFAULT on functional +* implementation, the non functional ones will return +* -ENOSYS. +*/ + curval =cmpxchg_futex_value_locked (NULL ,0 ,0 ); +if (curval == -EFAULT ) + futex_cmpxchg_enabled =1 ; +for (i =0 ; i <ARRAY_SIZE (futex_queues); i++) {plist_head_init (&futex_queues[i].chain, &futex_queues[i].lock);spin_lock_init (&futex_queues[i].lock);
kernel/futex_compat.c +9 -0
@@ -54,6 +54,9 @@ void compat_exit_robust_list(struct task_struct *curr)compat_long_t futex_offset;int rc; +if (!futex_cmpxchg_enabled) +return ; +/* * Fetch the list head (which was registered earlier, via * sys_set_robust_list()):
@@ -115,6 +118,9 @@ asmlinkage long compat_sys_set_robust_list(struct compat_robust_list_head __user *head, compat_size_t len) { +if (!futex_cmpxchg_enabled) +return -ENOSYS ; +if (unlikely (len !=sizeof (*head)))return -EINVAL ;
@@ -130,6 +136,9 @@ compat_sys_get_robust_list(int pid, compat_uptr_t __user *head_ptr,struct compat_robust_list_head __user *head;unsigned long ret; +if (!futex_cmpxchg_enabled) +return -ENOSYS ; +if (!pid) head = current->compat_robust_list;else {
From: David Miller <[email protected]>
Date: Tue, 26 Feb 2008 16:49:00 -0800 (PST)
[ Thomas, forgot to CC: you earlier, changeset
a0c1e9073ef7428a14309cba010633a6cd6719ea ("futex: runtime enable pi and robust functionality") broke sparc64. ]
Quoted 11 lines
From: Mikael Pettersson <[email protected]>
Date: Tue, 26 Feb 2008 09:55:50 +0100Quoted 3 lines
Minor update: rc2-git7 has the slow initial console behaviour, but successfully switches to the framebuffer. rc2-git8 however hangs in the console handover. So I'll bisect
git7->git8next.Between the VT layer registering it's console and the atyfb driver initializing we get a crash, and it happens on all sparc64 systems. It is caused by this commit and I am working on a fix:
The following patch will let things "work" but the trick being used here by the FUTEX layer is borderline valid in my opinion.
Basically for 10+ years on sparc64 we've had this check here in the fault path, which makes sure that if we're processing an exception table entry we really, truly, are doing an access to userspace from the kernel. Otherwise we OOPS.
What the FUTEX checking code is doing now is doing a "user" access with set_fs(KERNEL_DS) since it runs from the kernel bootup early init sequence. And this is illegal according to the existing checks.
When we do set_fs(KERNEL_DS) then pass a "user" pointer down into a system call or something like that, we give it a pointer that "cannot fault". So if we get into the fault handling path here for a case like that we really do want to scream and print out an OOPS message in my opinion.
I realize that not many platforms other than sparc64 can check for things this precisely, but it's something to consider.
Did this FUTEX change go into -stable too?
arch/sparc64/mm/fault.c +2 -10
@@ -244,16 +244,8 @@ static void do_kernel_fault(struct pt_regs *regs, int si_code, int fault_code,if (regs->tstate &TSTATE_PRIV ) {const struct exception_table_entry *entry; -if (asi ==ASI_P && (insn &0xc0800000 ) ==0xc0800000 ) { -if (insn &0x2000 ) - asi = (regs->tstate >>24 ); -else - asi = (insn >>5 ); - } - -/* Look in asi.h: All _S asis have LS bit set */ -if ((asi &0x1 ) && - (entry =search_exception_tables (regs->tpc))) { + entry =search_exception_tables (regs->tpc); +if (entry) { regs->tpc = entry->fixup; regs->tnpc = regs->tpc +4 ;return ;
Quoted 9 lines · David Miller
What the FUTEX checking code is doing now is doing a "user" access with
set_fs(KERNEL_DS)since it runs from the kernel bootup early init sequence. And this is illegal according to the existing checks.When we do
set_fs(KERNEL_DS)then pass a "user" pointer down into a system call or something like that, we give it a pointer that "cannot fault". So if we get into the fault handling path here for a case like that we really do want to scream and print out an OOPS message in my opinion.
So it would be correct to set_fs(USER_DS) then do the check and switch back to KERNEL_DS ?
Quoted 2 lines
I realize that not many platforms other than sparc64 can check for things this precisely, but it's something to consider.
Hmm, I missed that.
Quoted 1 line
Did this FUTEX change go into -stable too?
It's queued, AFAIK
Thanks,
tglx
From: Thomas Gleixner <[email protected]>
Date: Wed, 27 Feb 2008 09:02:22 +0100 (CET)
Quoted 5 lines · David Miller
Quoted 4 lines
What the FUTEX checking code is doing now is doing a "user" access with
set_fs(KERNEL_DS)since it runs from the kernel bootup early init sequence. And this is illegal according to the existing checks.
Quoted 8 lines
Quoted 5 lines
When we do
set_fs(KERNEL_DS)then pass a "user" pointer down into a system call or something like that, we give it a pointer that "cannot fault". So if we get into the fault handling path here for a case like that we really do want to scream and print out an OOPS message in my opinion.So it would be correct to
set_fs(USER_DS)then do the check and switch back toKERNEL_DS?
No, I'm saying it would be better not to take faults purposefully in the kernel address space. We don't have a usable user address space setup at this point in the boot, so using USER_DS would be even worse.
I think I'll just add a different version of the sanity check to this sparc64 code later on, one that will take into consideration this KERNEL_DS case because I can see how it could be useful in other circumstances.
Quoted 3 lines
Quoted 1 line
Did this FUTEX change go into -stable too?
It's queued, AFAIK
Crap, I'll need to push my fix there too.
Quoted 14 lines · David Miller
From: Thomas Gleixner <[email protected]>
Date: Wed, 27 Feb 2008 09:02:22 +0100 (CET)Quoted 11 lines · David Miller
Quoted 9 lines
What the FUTEX checking code is doing now is doing a "user" access with
set_fs(KERNEL_DS)since it runs from the kernel bootup early init sequence. And this is illegal according to the existing checks.When we do
set_fs(KERNEL_DS)then pass a "user" pointer down into a system call or something like that, we give it a pointer that "cannot fault". So if we get into the fault handling path here for a case like that we really do want to scream and print out an OOPS message in my opinion.
Quoted 5 lines
Quoted 2 lines
So it would be correct to
set_fs(USER_DS)then do the check and switch back toKERNEL_DS?No, I'm saying it would be better not to take faults purposefully in the kernel address space.
I would have preferred not to. The hassle is that we need to figure out, whether it works or not _before_ any user space program can use the interfaces. We could omit the check for archs where the in_atomic_cmpxchg is guaranteed to be functional.
Quoted 2 lines
We don't have a usable user address space
setup at this point in the boot, so usingUSER_DSwould be even worse.
Ouch, yes. Stupid me.
Quoted 4 lines
I think I'll just add a different version of the sanity check to this sparc64 code later on, one that will take into consideration this
KERNEL_DScase because I can see how it could be useful in other circumstances.
Ok.
Thanks,
tglx
Quoted 56 lines · David Miller
From: David Miller <[email protected]>
Date: Tue, 26 Feb 2008 16:49:00 -0800 (PST)[ Thomas, forgot to CC: you earlier, changeset
a0c1e9073ef7428a14309cba010633a6cd6719ea("futex: runtime enable pi and robust functionality") broke sparc64. ]Quoted 11 lines
From: Mikael Pettersson <[email protected]>
Date: Tue, 26 Feb 2008 09:55:50 +0100Quoted 3 lines
Minor update: rc2-git7 has the slow initial console behaviour, but successfully switches to the framebuffer. rc2-git8 however hangs in the console handover. So I'll bisect
git7->git8next.Between the VT layer registering it's console and the atyfb driver initializing we get a crash, and it happens on all sparc64 systems. It is caused by this commit and I am working on a fix:
The following patch will let things "work" but the trick being used here by the FUTEX layer is borderline valid in my opinion.
Basically for 10+ years on sparc64 we've had this check here in the fault path, which makes sure that if we're processing an exception table entry we really, truly, are doing an access to userspace from the kernel. Otherwise we OOPS.
What the FUTEX checking code is doing now is doing a "user" access with
set_fs(KERNEL_DS)since it runs from the kernel bootup early init sequence. And this is illegal according to the existing checks.When we do
set_fs(KERNEL_DS)then pass a "user" pointer down into a system call or something like that, we give it a pointer that "cannot fault". So if we get into the fault handling path here for a case like that we really do want to scream and print out an OOPS message in my opinion.I realize that not many platforms other than sparc64 can check for things this precisely, but it's something to consider.
Did this FUTEX change go into -stable too?
arch/sparc64/mm/fault.c +0
-5@@ -244,16 +244,8 @@ static void do_kernel_fault(struct pt_regs *regs, int si_code, int fault_code,if (regs->tstate &TSTATE_PRIV ) {const struct exception_table_entry *entry; -if (asi =ASI_P && (insn &0xc0800000 ) =0xc0800000 ) { -if (insn &0x2000 ) - asi = (regs->tstate >>24 ); -else - asi = (insn >>5 );
arch/sparc64/mm/fault.c static void do_kernel_fault(struct pt_regs *regs, int si_code, int fault_code, · line 251
Quoted 10 lines
arch/sparc64/mm/fault.c +2
-5- } - -/* Look in asi.h: All _S asis have LS bit set */ -if ((asi &0x1 ) && - (entry =search_exception_tables (regs->tpc))) { + entry =search_exception_tables (regs->tpc); +if (entry) { regs->tpc = entry->fixup; regs->tnpc = regs->tpc +4 ;return ;
Thanks, 2.6.25-rc3 plus this patch works fine on my Ultra5.
/Mikael
From: Mikael Pettersson <[email protected]>
Date: Wed, 27 Feb 2008 20:16:17 +0100
Quoted 1 line
Thanks, 2.6.25-rc3 plus this patch works fine on my Ultra5.
Thank you for testing.
Quoted 12 lines · David Miller
From: Mikael Pettersson <[email protected]>
Date: Tue, 26 Feb 2008 09:55:50 +0100Quoted 3 lines
Minor update: rc2-git7 has the slow initial console behaviour, but successfully switches to the framebuffer. rc2-git8 however hangs in the console handover. So I'll bisect
git7->git8next.Between the VT layer registering it's console and the atyfb driver initializing we get a crash, and it happens on all sparc64 systems. It is caused by this commit and I am working on a fix:
Quoted 5 lines
commit
a0c1e9073ef7428a14309cba010633a6cd6719eaAuthor: Thomas Gleixner <[email protected]> Date: Sat Feb 23 15:23:57 2008 -0800futex: runtime enable pi and robust functionality
My git7->git8 bisection yesterday independently also arrived at that specific commit as being the culprit.
Bracketing the offending cmpxchg_futex_value_locked(NULL, 0, 0) call with #if 0 .. #endif was enough to make my kernel boot.
I'll try your do_kernel_fault() patch later today.
/Mikael
From: Mikael Pettersson <[email protected]>
Date: Mon, 25 Feb 2008 21:41:03 +0100
Quoted 6 lines
Another difference is that with rc2 the first few lines of kernel output while the console is still in OF mode either aren't shown or disappear quickly since the switch to the framebuffer occurs within a fraction of a second after the kernel has been loaded. With rc3 the kernel output (the text shown above) in the OF-mode console is very very slow.
Yes that's a new feature. Until we switch over to the "real" console we print the log messages using the firmware console routines.
This way if an early crash or similar happens, you'll see it and be able to report it instead of having to report with "-p" on the command line.
I'll fire up my ultra5 and try to figure out what's wrong with the atyfb framebuffer driver, that's where it's dying.