2.6.4 on Alpha uninterruptible sleep of processes
Hi All,
I installed 2.6.4 for the first time on my alpha and run into big troubles.
After some time, new started processes are in a uninterruptible sleep state. It seems every process that opens a file hangs. E.g. I can't tail -f /var/log/messages
witch ~ # uptime
09:44:29 up 14:51, 6 users, load average: 116.99, 115.78, 110.12
Attached the output of ps -aux and /proc/slabinfo.
I don't know which further informations are needed.
cat /proc/vmstat:
nr_dirty 35nr_writeback 0nr_unstable 0nr_page_table_pages 1477nr_mapped 23651nr_slab 2390
pgpgin 463751
pgpgout 692219
pswpin 4906
pswpout 50947
pgalloc 2795267
pgfree 2795470
pgactivate 32874
pgdeactivate 78053
pgfault 10497618
pgmajfault 2668
pgscan 348418
pgrefill 762245
pgsteal 96252
pginodesteal 22kswapd_steal 8564kswapd_inodesteal 0
pageoutrun 95
allocstall 2415
pgrotated 49666
I'm going to reboot now
Regards
Marc
Quoted 3 lines · Marc Giger
After some time, new started processes are in a uninterruptible sleep state. It seems every process that opens a file hangs. E.g. I can't tail -f
/var/log/messages
My fault. There was a typo in the semaphore patch. Does this work for you?
Ivan.
arch/alpha/kernel/semaphore.c +1 -1
@@ -29,7 +29,7 @@ static inline int __sem_update_count(str __asm__ __volatile__("1: ldl_l %0,%2\n" -" cmovgt %0,%0,%1\n" +" cmovgt %0,0,%1\n" " addl %1,%3,%1\n" " stl_c %1,%2\n" " beq %1,2f\n"
arch/alpha/kernel/semaphore.c static inline int __sem_update_count(str · line 32
Quoted 2 lines · Ivan Kokshaysky
arch/alpha/kernel/semaphore.c +1
-1-" cmovgt %0,%0,%1\n" +" cmovgt %0,0,%1\n"
Oops. This is wrong, please ignore.
Will investigate further.
Ivan.
Hi Ivan,
Quoted 6 lines · On Fri, 12 Mar 2004 18:41:15 +0300 Ivan Kokshaysky <[email protected]>…
arch/alpha/kernel/semaphore.c static inline int __sem_update_count(str · line 32
Quoted 2 lines · Ivan Kokshaysky
arch/alpha/kernel/semaphore.c +1
-1-" cmovgt %0,%0,%1\n" +" cmovgt %0,0,%1\n" Oops. This is wrong, please ignore.
Will investigate further.
Too late. Already applied, compiled and booted. Read your message and rebooted to 2.4:-)
Another question:
Why is there no option to compile a preemptive kernel? Not possible on alpha or nobody interested to code or...?
Perhaps you can answer yet another question:
In 2.4.23 /prc/meminfo shows always
Buffers: 0 kB
Is it normal on alpha? 2.6.4 showed a value > 0
Thank you!
Regards
Marc
Quoted 2 lines · Marc Giger
Too late. Already applied, compiled and booted. Read your message and rebooted to 2.4:-)
Well, you can try the appended patch to see whether it's a semaphore problem or not.
BTW, what alpha system do you have?
Quoted 2 lines
Why is there no option to compile a preemptive kernel? Not possible on alpha or nobody interested to code or...?
The answer is here:
http://bugzilla.kernel.org/show_bug.cgi?id=397
Quoted 5 lines
In 2.4.23 /prc/meminfo shows always
Buffers: 0 kB
Is it normal on alpha? 2.6.4 showed a value > 0
No idea. In 2.4.25 I have a non-zero value as well:
Buffers: 7288 kB
Ivan.
include/asm-alpha/semaphore.h +90 -26
@@ -16,7 +16,10 @@#include <linux/rwsem.h> struct semaphore { -atomic_t count; +/* Careful, inline assembly knows about the position of these two. */ +atomic_t count __attribute__((aligned (8 ))); +atomic_t waking;/* biased by -1 */ +wait_queue_head_t wait;#if WAITQUEUE_DEBUG long __magic;
@@ -30,18 +33,18 @@ struct semaphore {#endif #define __SEMAPHORE_INITIALIZER (name,count) \ - { ATOMIC_INIT(count), \ + { ATOMIC_INIT(count), ATOMIC_INIT(-1), \ __WAIT_QUEUE_HEAD_INITIALIZER((name).wait) \ __SEM_DEBUG_INIT(name) } -#define __MUTEX_INITIALIZER (name) \ +#define __MUTEX_INITIALIZER (name) \ __SEMAPHORE_INITIALIZER(name,1) -#define __DECLARE_SEMAPHORE_GENERIC (name,count) \ +#define __DECLARE_SEMAPHORE_GENERIC (name,count) \ struct semaphore name = __SEMAPHORE_INITIALIZER(name,count) -#define DECLARE_MUTEX (name) __DECLARE_SEMAPHORE_GENERIC(name,1) -#define DECLARE_MUTEX_LOCKED (name) __DECLARE_SEMAPHORE_GENERIC(name,0) +#define DECLARE_MUTEX (name) __DECLARE_SEMAPHORE_GENERIC(name,1) +#define DECLARE_MUTEX_LOCKED (name) __DECLARE_SEMAPHORE_GENERIC(name,0)static inline void sema_init (struct semaphore *sem,int val) {
@@ -52,6 +55,7 @@ static inline void sema_init(struct sema*/ atomic_set (&sem->count, val); +atomic_set (&sem->waking,-1 );init_waitqueue_head (&sem->wait);#if WAITQUEUE_DEBUG sem->__magic = (long )&sem->__magic;
@@ -103,42 +107,102 @@ static inline int __down_interruptible(s/* * down_trylock returns 0 on success, 1 if we failed to get the lock. +* +* We must manipulate count and waking simultaneously and atomically. +* Do this by using ll/sc on the pair of 32-bit words. */ -static inline int __down_trylock (struct semaphore *sem) +static inline int __down_trylock (struct semaphore * sem) { -long ret; +long ret, tmp, tmp2, sub; -/* "Equivalent" C: +/* "Equivalent" C. Note that we have to do this all without +(taken) branches in order to be a valid ll/sc sequence. do { -ret = ldl_l; ---ret; -if (ret < 0) -break; -ret = stl_c = ret; -} while (ret == 0); +tmp = ldq_l; +sub = 0x0000000100000000; +ret = ((int)tmp <= 0); // count <= 0 ? +// Note that if count=0, the decrement overflows into +// waking, so cancel the 1 loaded above. Also cancel +// it if the lock was already free. +if ((int)tmp >= 0) sub = 0; // count >= 0 ? +ret &= ((long)tmp < 0); // waking < 0 ? +sub += 1; +if (ret) break; +tmp -= sub; +tmp = stq_c = tmp; +} while (tmp == 0); */ + __asm__ __volatile__( -"1: ldl_l %0,%1\n" -" subl %0,1,%0\n" -" blt %0,2f\n" -" stl_c %0,%1\n" -" beq %0,3f\n" -" mb\n" -"2:\n" +"1: ldq_l %1,%4\n" +" lda %3,1\n" +" addl %1,0,%2\n" +" sll %3,32,%3\n" +" cmple %2,0,%0\n" +" cmovge %2,0,%3\n" +" cmplt %1,0,%2\n" +" addq %3,1,%3\n" +" and %0,%2,%0\n" +" bne %0,2f\n" +" subq %1,%3,%1\n" +" stq_c %1,%4\n" +" beq %1,3f\n" +"2: mb\n" ".subsection 2\n" "3: br 1b\n" ".previous" - :"=&r" (ret),"=m" (sem->count) - :"m" (sem->count)); + :"=&r" (ret),"=&r" (tmp),"=&r" (tmp2),"=&r" (sub) + :"m" (*sem) + :"memory" ); -return ret <0 ; +return ret; }static inline void __up (struct semaphore *sem) { -if (unlikely (atomic_inc_return (&sem->count) <=0 )) +long ret, tmp, tmp2, tmp3; + +/* We must manipulate count and waking simultaneously and atomically. +Otherwise we have races between up and __down_failed_interruptible +waking up on a signal. + +"Equivalent" C. Note that we have to do this all without +(taken) branches in order to be a valid ll/sc sequence. + +do { +tmp = ldq_l; +ret = (int)tmp + 1; // count += 1; +tmp2 = tmp & 0xffffffff00000000; // extract waking +if (ret <= 0) // still sleepers? +tmp2 += 0x0000000100000000; // waking += 1; +tmp = ret & 0x00000000ffffffff; // insert count +tmp |= tmp2; // insert waking; +tmp = stq_c = tmp; +} while (tmp == 0); +*/ + + __asm__ __volatile__( +" mb\n" +"1: ldq_l %1,%4\n" +" addl %1,1,%0\n" +" zapnot %1,0xf0,%2\n" +" addq %2,%5,%3\n" +" cmovle %0,%3,%2\n" +" zapnot %0,0x0f,%1\n" +" bis %1,%2,%1\n" +" stq_c %1,%4\n" +" beq %1,3f\n" +"2:\n" +".subsection 2\n" +"3: br 1b\n" +".previous" + :"=&r" (ret),"=&r" (tmp),"=&r" (tmp2),"=&r" (tmp3) + :"m" (*sem),"r" (0x0000000100000000 ) + :"memory" ); + +if (unlikely (ret <=0 ))__up_wakeup (sem); }
arch/alpha/kernel/semaphore.c +121 -79
258 lines not on this page · raw message
Quoted 7 lines · On Fri, 12 Mar 2004 22:46:49 +0300 Ivan Kokshaysky <[email protected]>…
Quoted 2 lines · Marc Giger
Too late. Already applied, compiled and booted. Read your message and rebooted to 2.4:-)
Well, you can try the appended patch to see whether it's a semaphore problem or not.
BTW, what alpha system do you have?
Right now I'm recompiling the kernel. So you say this patch isn't a fix but a test? This time I have additionally "semaphore debugging" enabled, perhaps it's useful for you.
I will inform you as soon as I get new infos.
Quoted 6 lines
Quoted 3 lines
Why is there no option to compile a preemptive kernel? Not possible on alpha or nobody interested to code or...?
The answer is here:
http://bugzilla.kernel.org/show_bug.cgi?id=397
That's no answer, that's a statement:-) Do know the exactly reason why it should be a bad idea? Is it mostly a bad idea on alpha?
Quoted 9 lines
Quoted 6 lines
In 2.4.23 /prc/meminfo shows always
Buffers: 0 kB
Is it normal on alpha? 2.6.4 showed a value > 0
No idea. In 2.4.25 I have a non-zero value as well:
Buffers: 7288 kB
Interestingly. I'm still waiting to the acl patch for 2.4.25. I think I will rediff it myself.
Thank you for your work!
Regards
Marc
Quoted 2 lines · Marc Giger
Right now I'm recompiling the kernel. So you say this patch isn't a fix but a test?
Yes. That patch just reverts new alpha semaphore stuff which went into 2.6.4.
Quoted 2 lines
This time I have additionally "semaphore debugging" enabled, perhaps it's useful for you.
Thanks, this might be helpful.
Quoted 5 lines
Quoted 2 lines
The answer is here:
http://bugzilla.kernel.org/show_bug.cgi?id=397That's no answer, that's a statement:-) Do know the exactly reason why it should be a bad idea? Is it mostly a bad idea on alpha?
Hmm, I haven't discussed that with Richard, so I can't speak for him :-) IMHO, the benefits of the kernel preempt support in general are more than doubtful, the level of complexity that it adds to the kernel code is just unacceptable.
Ivan.
Hi Ivan,
Quoted 11 lines · On Sat, 13 Mar 2004 02:01:41 +0300 Ivan Kokshaysky <[email protected]>…
Quoted 2 lines · Marc Giger
Right now I'm recompiling the kernel. So you say this patch isn't a fix but a test?
Yes. That patch just reverts new alpha semaphore stuff which went into 2.6.4.
Quoted 2 lines
This time I have additionally "semaphore debugging" enabled, perhaps it's useful for you.
Thanks, this might be helpful.
Hmm, I couldn't boot the kernel with enabled "semaphore debugging". It hangs directly after aboot. No messages, nothing. Do I something wrong? Now I've booted 2.6.4 without debugging.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
The answer is here:
http://bugzilla.kernel.org/show_bug.cgi?id=397That's no answer, that's a statement:-) Do know the exactly reason why it should be a bad idea? Is it mostly a bad idea on alpha?
Hmm, I haven't discussed that with Richard, so I can't speak for him :-) IMHO, the benefits of the kernel preempt support in general are more than doubtful, the level of complexity that it adds to the kernel code is just unacceptable.
Ok, but I read somewhere exactly the opposite (lkml?).
The statement was something like the following: "Preempt doesn't need much more infrastrucure in kernel code, because the needed locking mechanism is already there (SMP)."
So I'm confused now:-) But I understand that every little more complexity is not for free. More task switches etc...
Thank you for the infos.
greets
Marc
Quoted 2 lines · Marc Giger
Hmm, I couldn't boot the kernel with enabled "semaphore debugging". It hangs directly after aboot. No messages, nothing. Do I something wrong?
No. Indeed, "semaphore debugging" is completely useless since printk() itself is trying to acquire the console_sem, which in turn causes another debugging printk() and so on.
I think this option should be removed...
Quoted 1 line
Now I've booted 2.6.4 without debugging.
And does the last patch makes things better?
Ivan.
Hi Ivan,
Quoted 5 lines · On Sun, 14 Mar 2004 17:06:27 +0300 Ivan Kokshaysky <[email protected]>…
Quoted 3 lines · Marc Giger
Hmm, I couldn't boot the kernel with enabled "semaphore debugging". It hangs directly after aboot. No messages, nothing. Do I something wrong?
Quoted 8 lines
No. Indeed, "semaphore debugging" is completely useless since
printk()itself is trying to acquire theconsole_sem, which in turn causes another debuggingprintk()and so on.
I think this option should be removed...Quoted 1 line
Now I've booted 2.6.4 without debugging.
And does the last patch makes things better?
No, it doesn't. After some hours it has got the same problems.
It's interesting that this happens after some hours of uptime and not immediately.
Regards
Marc
Quoted 1 line · Marc Giger
No, it doesn't. After some hours it has got the same problems.
Fine, at least semaphores changes are innocent. :-)
Interesting - I wasn't able to reproduce these problems with recent kernels on my boxes (including one lx164)...
Ivan.
Quoted 7 lines · On Mon, 15 Mar 2004 14:51:45 +0300 Ivan Kokshaysky <[email protected]>…
Quoted 1 line · Marc Giger
No, it doesn't. After some hours it has got the same problems.
Fine, at least semaphores changes are innocent. :-)
Interesting - I wasn't able to reproduce these problems with recent kernels on my boxes (including one lx164)...
How long did you let your machine run? In my case, it has to run the whole night until it happens. I don't know if it helps but I think the first processes that are in uninterruptible sleep are apache and mysql. Also, as you can see in my first e-mail (ps -aux output), the pdflush and kswapd0 are in in uninterruptible sleep state.
Regards
Marc
Quoted 1 line · Marc Giger
How long did you let your machine run?
$ uptime 01:40:40 up 3 days, 6:31, 4 users, load average: 25.23, 24.16, 23.73
It's with unpatched 2.6.4. Before that the machine was running 2.6.1-rc1 for 2 months.
Quoted 1 line
In my case, it has to run the whole night until it happens.
Perhaps there is a memory leak somewhere, and your systems just starts swapping.
Quoted 4 lines
I don't know if it helps but I think the
first processes that are in uninterruptible sleep are apache and mysql. Also, as you can see in my first e-mail (ps -aux output), the pdflush and kswapd0 are in in uninterruptible sleep state.
Well, I can see something like that when I compile kernel with "make -j 15". The system starts swapping like crazy, most processes are in the D-state waiting for disk, but all goes back to normal after compilation is finished. What's wrong with it? :-)
Ivan.
Quoted 6 lines · Marc Giger
Quoted 1 line
How long did you let your machine run?
$ uptime 01:40:40 up 3 days, 6:31, 4 users, load average: 25.23, 24.16, 23.73
Quoted 7 lines
It's with unpatched 2.6.4. Before that the machine was running 2.6.1-rc1 for 2 months.
Quoted 1 line
In my case, it has to run the whole night until it happens.
Perhaps there is a memory leak somewhere, and your systems just starts swapping.
No, that isn't the case. The memory usage stays constant and it has no swap used at all.
Quoted 6 lines
Quoted 5 lines
I don't know if it helps but I think the
first processes that are in uninterruptible sleep are apache and mysql. Also, as you can see in my first e-mail (ps -aux output), the pdflush and kswapd0 are in in uninterruptible sleep state.
Quoted 6 lines
Well, I can see something like that when I compile kernel with "make -j 15".
The system starts swapping like crazy, most processes are in the D-state waiting for disk, but all goes back to normal after compilation is finished.
What's wrong with it? :-)
Nothing when it would switch back to a normal state.:-) As I already mentioned when processes begins to hang, all file operations like tail -f /var/log/messages etc. then hangs too:-( Not even a login or a proper shutdown will work. I have to press CTRL-SYSRQ-X
So I think it is something wrong with 2.6 on alpha:-) I have absolutely no problems with 2.4.
Regards
Marc
Signature · 2 lines
+++ NEU bei GMX und erstmalig in Deutschland: TÜV-geprüfter Virenschutz +++ 100% Virenerkennung nach Wildlist. Infos: http://www.gmx.net/virenschutz
Quoted 2 lines · Marc Giger
So I think it is something wrong with 2.6 on alpha:-) I have absolutely no problems with 2.4.
I'm running 2.6.3 on my SX164 with no problems at all. It's running web and mail servers, hosts a few BK repos, and lately it's been busy compiling a bunch of stuff. Current uptime is 13 days after kernel upgrade. I've been running 2.6 kernels since January without a glitch, so there's nothing wrong with 2.6 in general on Alphas in general, at least. I haven't dared try 2.6.4 after reading this thread.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 7 lines · On Fri, 12 Mar 2004 22:46:49 +0300 Ivan Kokshaysky <[email protected]>…
Quoted 2 lines · Marc Giger
Too late. Already applied, compiled and booted. Read your message and rebooted to 2.4:-)
Well, you can try the appended patch to see whether it's a semaphore problem or not.
BTW, what alpha system do you have?
Oh, it's an LX164 ev56 @ 533Mhz