CONFIG_PREEMPT_REALTIME, 'Fully Preemptible Kernel', VP-2.6.9-rc4-mm1-T4

PATCH In review [email protected] 67 participants 947 messages Oct 11, 2004 – Feb 7, 2005 6 messages not in the archive Page 5 of 5

Subsystems (28)

Networking Network drivers Memory management Media (V4L/DVB) +24 more subsystems

lore

Share & tools

lore

Florian Schmidt

2004-11-23 14:42 UTC #744 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

In reply to Ingo Molnar

Quoted 9 lines · On Tue, 23 Nov 2004 16:21:26 +0100 Ingo Molnar <[email protected]> wrote:

Quoted 6 lines · Florian Schmidt
~$ ps -C jack_test -cmL
PID LWP CLS PRI TTY TIME CMD
  988     - -     - pts/1    00:00:00 jack_test
    -   988 TS   20 -        00:00:00 -
    -   989 FF   99 -        00:00:00 -

Quoted 6 lines

Quoted 2 lines
So when you ctrl-z out of jack_test you cause its process() thread to be suspended, too, thus jackd cannot finish processing its graph.
so in theory any scheduling delay of PID 988 in the above setup (the SCHED_OTHER task) should not be able to negatively influence jackd, correct?

correct

Quoted 2 lines

In fact, does in this particular jack_test case PID 988 do anything substantial?

Well, it registers the client with jackd, sets up the ports, registers the process() callback and then simply goes to sleep() for the desired runtime of the program. All these are non RT ops and should never be able to cause any xruns.

All the work is done by the process() callback which is called by libjack in a SCHED_FIFO thread. The process() callback is called once for each buffer that jackd processes.

I cannot explain the detailed mechanism of how jackd wakes its clients and communicates with them myself too well, so i'll leave this to Paul Davis (CC'ed). Care to elaborate, Paul?

flo

Paul Davis

2004-12-01 13:58 UTC #745 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 29 lines

Quoted 9 lines · On Tue, 23 Nov 2004 16:21:26 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines · Florian Schmidt
~$ ps -C jack_test -cmL
PID LWP CLS PRI TTY TIME CMD
  988     - -     - pts/1    00:00:00  jack_test
    -   988 TS   20 -        00:00:00 -
    -   989 FF   99 -        00:00:00 -
Quoted 6 lines
Quoted 2 lines
So when you ctrl-z out of jack_test you cause its process() thread to be suspended, too, thus jackd cannot finish processing its graph.
so in theory any scheduling delay of PID 988 in the above setup (the SCHED_OTHER task) should not be able to negatively influence jackd, correct?
correct
Quoted 2 lines
In fact, does in this particular jack_test case PID 988 do anything substantial?
Well, it registers the client with jackd, sets up the ports, registers the process() callback and then simply goes to sleep() for the desired runtime of the program. All these are non RT ops and should never be able to cause any xruns.

Quoted 7 lines

All the work is done by the process() callback which is called by libjack in a SCHED_FIFO thread. The process() callback is called once for each buffer that jackd processes.
I cannot explain the detailed mechanism of how jackd wakes its clients and communicates with them myself too well, so i'll leave this to Paul Davis (CC'ed). Care to elaborate, Paul?

sorry for the delay on this, it ended up in a mail folder that i don't check very often.

jackd wakes clients by writing a single byte to a FIFO. the first client is sleeping on the other side of the FIFO. when that client is done, it writes a single byte to another FIFO. either another client is sleeping on the other side of that FIFO, or if there are no other clients, jackd is waiting for the client there.

the "chain" of wakeup FIFOs is rearranged every time the graph execution order is modified (e.g. by new connections being established between clients that requires a different execution order).

the chain is executed every time jackd is woken by its "backend" client, typically the ALSA client that waits on the fd's corresponding to an audio interface to be readable and/or writable. in other words, every interrupt from the audio interface.

we know that writes to FIFOs are not really RT-safe, but they are the closest thing linux has at this time. i have outlined an idea to ingo that florian and i cooked up one evening on IRC that would provide true RT-safe IPC mechanisms, but as i recall, he didn't seem to think that much of it :)

--p

Ingo Molnar

2004-12-01 14:37 UTC #746 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 1 line · Paul Davis

we know that writes to FIFOs are not really RT-safe, [...]

in kernels -V0.7.30-9 or later they are RT-safe when PREEMPT_RT is enabled.

also, the problem is that jackd uses _named_ fifos, which are tied to the raw FS and might trigger journalling activities. Normal pipes (unnamed fifos) would not cause such problems. Would it be possible to change jackd to use a pair of pipes, instead of a fifo?

Quoted 3 lines

[...] i have outlined an idea to ingo that florian and i cooked up one evening on IRC that would provide true RT-safe IPC mechanisms, but as i recall, he didn't seem to think that much of it :)

actually, my answer (sent on Nov 1) was:

Quoted 3 lines

futexes are nearly lock-free. [and even those locks are short-held so combined with priority-inheritance they should be lockfree in essence.] Would futexes suit your purposes?

to which suggestion i got no reply yet :-)

Ingo

Paul Davis

2004-12-01 14:56 UTC #747 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 2 lines

in kernels -V0.7.30-9 or later they are RT-safe when PREEMPT_RT is enabled.

thats good to hear.

Quoted 4 lines

also, the problem is that jackd uses _named_ fifos, which are tied to the raw FS and might trigger journalling activities. Normal pipes (unnamed fifos) would not cause such problems. Would it be possible to change jackd to use a pair of pipes, instead of a fifo?

i.e. pipe(2) rather than mkfifo(2) ?

it would be a complete pain because the pipes have to be "discoverable" across processes. we would have to do fd passing, which is still really quite ugly in linux (and other *nix systems). it would quite difficult, though not impossible.

Quoted 11 lines

Quoted 3 lines
[...] i have outlined an idea to ingo that florian and i cooked up one evening on IRC that would provide true RT-safe IPC mechanisms, but as i recall, he didn't seem to think that much of it :)
actually, my answer (sent on Nov 1) was:
Quoted 3 lines
futexes are nearly lock-free. [and even those locks are short-held so combined with priority-inheritance they should be lockfree in essence.] Would futexes suit your purposes?
to which suggestion i got no reply yet :-)

i am still trying to find the time to investigate futexes. they seem close to the desired object, but have a slightly more general semantic than i can fit into my head right now;)

--p

Ingo Molnar

2004-12-01 15:54 UTC #748 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 11 lines · Paul Davis

Quoted 4 lines
also, the problem is that jackd uses _named_ fifos, which are tied to the raw FS and might trigger journalling activities. Normal pipes (unnamed fifos) would not cause such problems. Would it be possible to change jackd to use a pair of pipes, instead of a fifo?
i.e. pipe(2) rather than mkfifo(2) ?
it would be a complete pain because the pipes have to be "discoverable" across processes. we would have to do fd passing, which is still really quite ugly in linux (and other *nix systems). it would quite difficult, though not impossible.

yeah. And i think mkfifo(2) objects ought to behave atomically as well, it's an unfortunate side-effect of atime/mtime inode semantics that they can block.

your point is correct, the best way to have a system-wide namespace for synchronization objects is ... the filesystem hierarchy. If you create a unix domain socket then you can distribute your pipe fds, but that's indeed somewhat painful.

Ingo

Paul Davis

2004-12-01 16:08 UTC #749 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 4 lines

your point is correct, the best way to have a system-wide namespace for synchronization objects is ... the filesystem hierarchy. If you create a unix domain socket then you can distribute your pipe fds, but that's indeed somewhat painful.

this is where Mach ports come in. they were designed to be passed around from process to process, painlessly, but without any system wide namespace. you can create ports that can be looked up by anyone, but not all ports are required meet this condition. this makes it easy to set up a private communication channel between two processes.

a bit like futexes :)

--p

Esben Nielsen

2004-12-01 16:17 UTC #750 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

In reply to Ingo Molnar

Quoted 15 lines · Ingo Molnar

Quoted 11 lines · Paul Davis
Quoted 4 lines
also, the problem is that jackd uses _named_ fifos, which are tied to the raw FS and might trigger journalling activities. Normal pipes (unnamed fifos) would not cause such problems. Would it be possible to change jackd to use a pair of pipes, instead of a fifo?
i.e. pipe(2) rather than mkfifo(2) ?
it would be a complete pain because the pipes have to be "discoverable" across processes. we would have to do fd passing, which is still really quite ugly in linux (and other *nix systems). it would quite difficult, though not impossible.

Quoted 9 lines

yeah. And i think mkfifo(2) objects ought to behave atomically as well, it's an unfortunate side-effect of atime/mtime inode semantics that they can block.
your point is correct, the best way to have a system-wide namespace for synchronization objects is ... the filesystem hierarchy. If you create a unix domain socket then you can distribute your pipe fds, but that's indeed somewhat painful.

Don't worry about setting up the stuff. Once you have the pipe it ought to be RT in the usage of read/write, but setting it up is something you is something you do "under boot", just as allocating memory and other non-real-time stuff.

Quoted 2 lines

Ingo

Esben

Ingo Molnar

2004-12-01 21:24 UTC #751 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 4 lines · Esben Nielsen

Don't worry about setting up the stuff. Once you have the pipe it ought to be RT in the usage of read/write, but setting it up is something you is something you do "under boot", just as allocating memory and other non-real-time stuff.

actually, the main problem with fifos was they were _not_ atomic even in read/write (i myself fully expected them to be that, but they arent). That's the bug/misfeature that i fixed in the latest kernels.

Ingo

Chris Friesen

2004-12-01 21:41 UTC #752 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 2 lines · Ingo Molnar

actually, the main problem with fifos was they were _not_ atomic even in
read/write (i myself fully expected them to be that, but they arent). That's the bug/misfeature that i fixed in the latest kernels.

Whoa. pipes (ie from the pipe() call) don't read/write atomicly? Even if you write less than a page?

When was this fixed?

Chris

Esben Nielsen

2004-12-01 16:13 UTC #753 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

In reply to Paul Davis

Quoted 6 lines · Paul Davis

[...]
Quoted 5 lines
Quoted 4 lines
futexes are nearly lock-free. [and even those locks are short-held so combined with priority-inheritance they should be lockfree in essence.] Would futexes suit your purposes?

Quoted 6 lines

Quoted 1 line
to which suggestion i got no reply yet :-)
i am still trying to find the time to investigate futexes. they seem close to the desired object, but have a slightly more general semantic than i can fit into my head right now;)

I looked into them just to see if they could be used for user-space priority inheritance. I saw that it takes (read-)lock on mmap_sem. Isn't that potentially held for a long time? I don't know how the memory subsystem works at all but my guess is that any changing of the process's memory is done with mmpa_sem locked. The only way to prevent that is 1) mlockall() and 2) stop increasing your heap.

I.e. you can't have one thread running real-time using a futex while another runs non-real-time allocating memory in the same process.

Am I correct?

Another problem is the hashing of the futex. That inherently has a non-deterministic timing behaviour. The more applications use futex'es the slower this hashing gets :-( The slowdown might in practise be very low because looking up in the global hash table can't take very many us!

Can't it just use a file-descriptor in the system call to look up the futex instead of the hashing? That is RT-O(1), right? But, ofcourse, then it is hard having a futex in shared mem.

Quoted 1 line

--p

Esben

Eran Mann

2004-11-22 08:48 UTC #754 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

In reply to Ingo Molnar

Quoted 4 lines · Ingo Molnar

i have released the -V0.7.30-2 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/

Hi,
I´m seeing latencies of up to ~2000 microseconds. see attached traces file for a small sample. I think I´m missing something obvious config-wise but I don´t know what...
The ´load´ during the traces consisted of a kernel build in a gnome-terminal, and 2 browser windows with a heavy site (Flash ads etc.) in each. This load causes a >1 ms latency every 5 minutes on average. After the kernel build ended the rate dropped dramatically to ~2 traces an hour.
The traces were from V-0.7.29-5 but I´ve seen these latencies in all RT kernels I tested (2.6.9-mm1-RT-V0.2 was the first). I´ll try V0.7.30-2 next. The machine is a PIII 733 Mhz, 256MB RAM, IDE disks.
I attached the traces and a partial .config.
Relevant items in /proc/sys/kernel:

/proc/sys/kernel/kernel_preemption :
1
...
/proc/sys/kernel/osrelease :
2.6.10-rc2-mm2-V0.7.29-5
...
/proc/sys/kernel/panic :
0
/proc/sys/kernel/panic_on_oops :
0
/proc/sys/kernel/pid_max :
32768
/proc/sys/kernel/preempt_max_latency :
1737
/proc/sys/kernel/preempt_thresh :
1000
/proc/sys/kernel/preempt_wakeup_timing :
1
...
/proc/sys/kernel/real-root-dev :
0
/proc/sys/kernel/sem :
250 32000 32 128
/proc/sys/kernel/shmall :
2097152
/proc/sys/kernel/shmmax :
33554432
/proc/sys/kernel/shmmni :
4096
/proc/sys/kernel/sysrq :
1
/proc/sys/kernel/tainted :
0
/proc/sys/kernel/threads-max :
4095
/proc/sys/kernel/trace_enabled :
1
/proc/sys/kernel/trace_freerunning :
0
/proc/sys/kernel/trace_print_at_crash :
1
/proc/sys/kernel/trace_user_triggered :
0
/proc/sys/kernel/trace_verbose :
0
/proc/sys/kernel/unknown_nmi_panic :
0

Let me know if some more information is required.
Thank,
Eran Mann

# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
CONFIG_MPENTIUMIII=y
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
# CONFIG_SMP is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
# CONFIG_X86_MCE_P4THERMAL is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=y
......
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=m
CONFIG_RTC_HISTOGRAM=m
# CONFIG_GEN_RTC is not set
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
......
#
# Profiling support
#
CONFIG_PROFILING=y
CONFIG_OPROFILE=m

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
CONFIG_WAKEUP_TIMING=y
CONFIG_CRITICAL_PREEMPT_TIMING=y
CONFIG_CRITICAL_IRQSOFF_TIMING=y
CONFIG_CRITICAL_TIMING=y
CONFIG_LATENCY_TIMING=y
CONFIG_LATENCY_TRACE=y
CONFIG_MCOUNT=y
CONFIG_RT_DEADLOCK_DETECT=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
# CONFIG_KGDB is not set
.....
#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

Nov 21 19:22:42 eran kernel: (multiload-apple/5506/CPU#0): 1694 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1694 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: multiload-apple/5506, uid:500 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
=> ended at: finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): (115) ((116))
131 88000005 0.000ms (+0.000ms): (5506) ((131))
131 88000005 0.001ms (+0.000ms): try_to_wake_up (wake_up_process) 131 88000005 0.001ms (+0.000ms): (0) ((1))
131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
131 88000003 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
131 88000002 0.003ms (+1.687ms): dequeue_task (deactivate_task)
5506 80000002 1.691ms (+0.001ms): __switch_to (__schedule)
5506 80000002 1.692ms (+0.000ms): (131) ((5506))
5506 80000002 1.693ms (+0.000ms): (116) ((115))
5506 80000002 1.693ms (+0.000ms): finish_task_switch (__schedule) 5506 80000001 1.693ms (+0.000ms): trace_stop_sched_switched (finish_task_switch) 5506 80000001 1.693ms (+0.005ms): (5506) ((115))
5506 80000001 1.699ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================

Nov 21 19:30:58 eran kernel: (firefox-bin/14054/CPU#0): 1991 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1991 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: firefox-bin/14054, uid:500 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
=> ended at: finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): (115) ((116))
131 88000005 0.000ms (+0.000ms): <000036e6> ((131))
131 88000005 0.000ms (+0.000ms): try_to_wake_up (wake_up_process) 131 88000005 0.001ms (+0.000ms): (0) ((1))
131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
131 88000003 0.001ms (+0.000ms): preempt_schedule (__up_mutex)
131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
131 88000002 0.003ms (+1.986ms): dequeue_task (deactivate_task)
14054 80000002 1.989ms (+0.000ms): __switch_to (__schedule)
14054 80000002 1.989ms (+0.000ms): (131) (<000036e6>)
14054 80000002 1.989ms (+0.000ms): (116) ((115))
14054 80000002 1.990ms (+0.000ms): finish_task_switch (__schedule) 14054 80000001 1.990ms (+0.000ms): trace_stop_sched_switched (finish_task_switch) 14054 80000001 1.990ms (+0.006ms): <000036e6> ((115))
14054 80000001 1.996ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================

Nov 21 19:31:44 eran kernel: (IRQ 0/2/CPU#0): 1213 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1213 us, entries: 23 (23)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: IRQ 0/2, uid:0 nice:0 policy:1 rt_prio:49
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
    3 88010003 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
    3 88010002 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
    3 88010002 0.000ms (+0.000ms): (50) ((105))
    3 88010002 0.000ms (+0.000ms): (2) ((3))
    3 88010002 0.000ms (+0.000ms): try_to_wake_up (wake_up_process)
    3 88010002 0.001ms (+0.000ms): (0) ((1))
    3 88010001 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
    3 88010001 0.001ms (+0.000ms): wake_up_process (redirect_hardirq)
    3 88010000 0.001ms (+0.000ms): preempt_schedule (__do_IRQ)
    3 88010000 0.002ms (+0.000ms): irq_exit (do_IRQ)
    3 88000001 0.002ms (+0.000ms): do_softirq (irq_exit)
    3 88000001 0.002ms (+0.000ms): __do_softirq (do_softirq)
    3 88000000 0.002ms (+0.000ms): preempt_schedule_irq (need_resched)
    3 98000000 0.003ms (+0.000ms): __schedule (preempt_schedule_irq)
    3 98000000 0.003ms (+0.000ms): profile_hit (__schedule)
    3 98000001 0.003ms (+0.000ms): sched_clock (__schedule)
    2 80000002 0.004ms (+1.207ms): __switch_to (__schedule)
    2 80000002 1.212ms (+0.000ms): (3) ((2))
    2 80000002 1.212ms (+0.000ms): (105) ((50))
    2 80000002 1.212ms (+0.000ms): finish_task_switch (__schedule)
    2 80000001 1.212ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
    2 80000001 1.212ms (+0.006ms): (2) ((50))
    2 80000001 1.219ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================
Nov 21 19:36:18 eran kernel: (firefox-bin/14054/CPU#0): 1881 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1881 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: firefox-bin/14054, uid:500 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
=> ended at: finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): (115) ((116))
131 88000005 0.000ms (+0.000ms): <000036e6> ((131))
131 88000005 0.001ms (+0.000ms): try_to_wake_up (wake_up_process) 131 88000005 0.001ms (+0.000ms): (0) ((1))
131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
131 88000003 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
131 88000002 0.003ms (+1.875ms): dequeue_task (deactivate_task)
14054 80000002 1.878ms (+0.001ms): __switch_to (__schedule)
14054 80000002 1.880ms (+0.000ms): (131) (<000036e6>)
14054 80000002 1.880ms (+0.000ms): (116) ((115))
14054 80000002 1.880ms (+0.000ms): finish_task_switch (__schedule) 14054 80000001 1.880ms (+0.000ms): trace_stop_sched_switched (finish_task_switch) 14054 80000001 1.881ms (+0.008ms): <000036e6> ((115))
14054 80000001 1.890ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================

Nov 21 19:37:18 eran kernel: (firefox-bin/14054/CPU#0): 1347 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1347 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: firefox-bin/14054, uid:500 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
=> ended at: finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): (115) ((116))
131 88000005 0.000ms (+0.000ms): <000036e6> ((131))
131 88000005 0.000ms (+0.000ms): try_to_wake_up (wake_up_process) 131 88000005 0.001ms (+0.000ms): (0) ((1))
131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
131 88000003 0.001ms (+0.000ms): preempt_schedule (__up_mutex)
131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
131 88000002 0.003ms (+0.000ms): dequeue_task (deactivate_task)
14054 80000002 0.003ms (+0.000ms): __switch_to (__schedule)
14054 80000002 0.004ms (+0.000ms): (131) (<000036e6>)
14054 80000002 0.004ms (+0.000ms): (116) ((115))
14054 80000002 0.004ms (+0.000ms): finish_task_switch (__schedule) 14054 80000001 0.005ms (+1.342ms): trace_stop_sched_switched (finish_task_switch) 14054 80000001 1.347ms (+0.103ms): <000036e6> ((115))
14054 80000001 1.450ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================

Nov 21 19:49:21 eran kernel: (gnome-terminal/5434/CPU#0): 1683 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1683 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: gnome-terminal/5434, uid:500 nice:0 policy:0 rt_prio:0
    -----------------

=> started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
=> ended at: finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000005 0.000ms (+0.000ms): (115) ((116))
131 88000005 0.001ms (+0.000ms): (5434) ((131))
131 88000005 0.001ms (+0.000ms): try_to_wake_up (wake_up_process) 131 88000005 0.001ms (+0.000ms): (0) ((1))
131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up) 131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
131 88000003 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
131 88000002 0.003ms (+1.676ms): dequeue_task (deactivate_task)
5434 80000002 1.680ms (+0.001ms): __switch_to (__schedule)
5434 80000002 1.681ms (+0.000ms): (131) ((5434))
5434 80000002 1.682ms (+0.000ms): (116) ((115))
5434 80000002 1.682ms (+0.000ms): finish_task_switch (__schedule) 5434 80000001 1.682ms (+0.000ms): trace_stop_sched_switched (finish_task_switch) 5434 80000001 1.682ms (+0.006ms): (5434) ((115))
5434 80000001 1.689ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================

Nov 21 19:52:18 eran kernel: (IRQ 0/2/CPU#0): 2009 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 2009 us, entries: 37 (37)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: IRQ 0/2, uid:0 nice:0 policy:1 rt_prio:49
    -----------------

=> started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
=> ended at: finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
783 88010003 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up) 783 88010002 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up) 783 88010002 0.000ms (+0.000ms): (50) ((54))
783 88010002 0.000ms (+0.000ms): (2) ((783))
783 88010002 0.000ms (+0.000ms): try_to_wake_up (wake_up_process) 783 88010002 0.001ms (+0.000ms): (0) ((1))
783 88010001 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up) 783 88010001 0.001ms (+0.000ms): wake_up_process (redirect_hardirq) 783 88010000 0.001ms (+0.000ms): preempt_schedule (__do_IRQ) 783 88010000 0.002ms (+0.000ms): irq_exit (do_IRQ)
783 88000001 0.002ms (+0.000ms): do_softirq (irq_exit)
783 88000001 0.002ms (+0.000ms): __do_softirq (do_softirq)
783 88000001 0.003ms (+0.000ms): wake_up_process (do_softirq) 783 88000001 0.003ms (+0.000ms): try_to_wake_up (wake_up_process) 783 88000001 0.003ms (+0.000ms): task_rq_lock (try_to_wake_up) 783 88000002 0.003ms (+0.000ms): activate_task (try_to_wake_up) 783 88000002 0.004ms (+0.000ms): sched_clock (activate_task) 783 88000002 0.004ms (+0.000ms): recalc_task_prio (activate_task) 783 88000002 0.004ms (+0.000ms): effective_prio (recalc_task_prio) 783 88000002 0.004ms (+0.000ms): enqueue_task (activate_task) 783 88000002 0.005ms (+0.000ms): (105) ((54))
783 88000002 0.005ms (+0.000ms): (3) ((783))
783 88000002 0.005ms (+0.000ms): try_to_wake_up (wake_up_process) 783 88000002 0.006ms (+1.999ms): (0) ((1))
783 88000001 2.005ms (+0.000ms): preempt_schedule (try_to_wake_up) 783 88000001 2.005ms (+0.000ms): wake_up_process (do_softirq) 783 88000000 2.006ms (+0.000ms): preempt_schedule_irq (need_resched) 783 98000000 2.006ms (+0.000ms): __schedule (preempt_schedule_irq) 783 98000000 2.006ms (+0.000ms): profile_hit (__schedule)
783 98000001 2.007ms (+0.000ms): sched_clock (__schedule)

    2 80000002 2.007ms (+0.000ms): __switch_to (__schedule)
    2 80000002 2.008ms (+0.000ms): (783) ((2))
    2 80000002 2.008ms (+0.000ms): (54) ((50))
    2 80000002 2.008ms (+0.000ms): finish_task_switch (__schedule)
    2 80000001 2.008ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
    2 80000001 2.009ms (+0.006ms): (2) ((50))
    2 80000001 2.015ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)

Ingo Molnar

2004-11-22 08:59 UTC #755 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 10 lines · Eran Mann

Quoted 4 lines · Ingo Molnar
i have released the -V0.7.30-2 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
Hi,
I´m seeing latencies of up to ~2000 microseconds. see attached traces file for a small sample. I think I´m missing something obvious config-wise but I don´t know what...

131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule) 131 88000002 0.003ms (+1.687ms): dequeue_task (deactivate_task)
5506 80000002 1.691ms (+0.001ms): __switch_to (__schedule)

this seems to be hardware-generated. As you can see it from the trace, the codepath between __schedule()'s deactive_task() and __switch_to() has all interrupts and preemption disabled. The O(1) scheduler there has constant overhead and that codepath should at most take ~1 usec.

(there is one exception, if both LATENCY_TRACING and RT_DEADLOCK_DETECT are enabled in -V0.7.30-2 and later kernels then the overhead within __schedule is O(nr_running), because the tracer adds entries for every runnable task. But this is not the case for your trace because then you'd see those entries in /proc/latency_trace.)

Quoted 5 lines

The ´load´ during the traces consisted of a kernel build in a gnome-terminal, and 2 browser windows with a heavy site (Flash ads etc.) in each. This load causes a >1 ms latency every 5 minutes on average. After the kernel build ended the rate dropped dramatically to ~2 traces an hour.

this seems to imply IDE DMA related hardware overhead. Apparently what happens is that with certain motherboards/chipsets, if IDE DMA happens then that DMA transfer _completely locks up_ the system bus. Nothing happens, and the CPU is stalled in essence until the end of the DMA request.

there's nothing the kernel can do about a hardware latency like that, but you can try to work it around. Mark H. Johnson has reported up to 500 usec latencies that had a similar pattern as yours, and he has experimented with lesser DMA modes (udma2?) via hdparm. YMMV and be careful with hdparm settings.

Quoted 3 lines

The traces were from V-0.7.29-5 but I´ve seen these latencies in all RT kernels I tested (2.6.9-mm1-RT-V0.2 was the first). I´ll try V0.7.30-2 next. The machine is a PIII 733 Mhz, 256MB RAM, IDE disks.

this very strongly implies some sort of hardware overhead. Btw., the likely reason why this often shows up within __schedule() is that 1) it's a very common operation, especially on the -RT kernel 2) we do a TLB flush there, which can be quite memory-intense, so if the system bus (the memory bus) is locked up, there is a high likelyhood that this function generates a cachemiss.

Ingo

Eran Mann

2004-11-22 13:40 UTC #756 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 6 lines · Ingo Molnar

Quoted 5 lines · Eran Mann
Hi,
I´m seeing latencies of up to ~2000 microseconds. see attached traces file for a small sample. I think I´m missing something obvious config-wise but I don´t know what...

...

Quoted 4 lines

this seems to imply IDE DMA related hardware overhead. Apparently what
happens is that with certain motherboards/chipsets, if IDE DMA happens
then that DMA transfer _completely locks up_ the system bus. Nothing happens, and the CPU is stalled in essence until the end of the DMA request.

....

Quoted 1 line

Ingo

Right on.
After hdparm -d0 I see maximum latency of 35 us after a full kernel build with a few GUI apps in the background. I´ll try to find a reasonable compromise.

Signature · 1 line

Eran Mann

Ingo Molnar

2004-11-22 13:45 UTC #757 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 11 lines · Eran Mann

Quoted 3 lines
Quoted 3 lines
I´m seeing latencies of up to ~2000 microseconds. see attached traces file for a small sample. I think I´m missing something obvious config-wise but I don´t know what...
...
Quoted 5 lines
this seems to imply IDE DMA related hardware overhead. Apparently what happens is that with certain motherboards/chipsets, if IDE DMA happens then that DMA transfer _completely locks up_ the system bus. Nothing happens, and the CPU is stalled in essence until the end of the DMA request.

Quoted 4 lines

Right on.
After hdparm -d0 I see maximum latency of 35 us after a full kernel build with a few GUI apps in the background. I´ll try to find a reasonable compromise.

it might make sense to report this to the hw vendor as well, as these latencies dont occur at _every_ IDE DMA, it might be some sort of chipset (or BIOS) bug they might want to see resolved as well (if this isnt a ship-and-forget vendor). 2 msec stalls are not nice to a fair number of applications.

Ingo

Eran Mann

2004-11-23 08:28 UTC #758 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Quoted 5 lines · Ingo Molnar

Quoted 2 lines · Eran Mann
Right on.
After hdparm -d0 I see maximum latency of 35 us after a full kernel build with a few GUI apps in the background. I´ll try to find a reasonable compromise.

Quoted 7 lines

it might make sense to report this to the hw vendor as well, as these latencies dont occur at _every_ IDE DMA, it might be some sort of chipset (or BIOS) bug they might want to see resolved as well (if this isnt a ship-and-forget vendor). 2 msec stalls are not nice to a fair number of applications.
Ingo

It´s a rather old white-box machine with a noname VIA-based motherboard, So I don´t really have whom to report the problem to. On the other hand with udma2 I see latencies < 170 us which seems reasonable. Thanks for the advice.
Eran.

Ingo Molnar

2004-11-23 22:58 UTC #759 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

In reply to Ingo Molnar

i have released the -V0.7.30-9 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

most importantly it includes a JACK related latency fix. With Florian Schmidt's great detective work we honed in on a big latency source within JACK: the use of named pipes (fifos) on journalled filesystems.

This issue has been empirically identified before (and is mentioned in the JACK howto) but has never been given high enough prominence. It turns out that the atime updates done while read()ing or write()ing named pipes causes the delays - it may under certain circumstances call out into the journalling code. It may block even on non-journalled filesystems.

To work this issue around, when PREEMPT_RT is enabled the -30-9 kernel skips atime updates on named-fifos. (it's pretty pointless anyway.) Alternative userspace workarounds are to put the fifos on tmpfs/ramfs, or to mark the filesystem noatime,nodiratime.

those experiencing xruns under JACK should definitely try the -30-9 kernel.

Changes since -V0.7.30-2:

- named fifo reads/writes are now atomic, whenever possible

- fixed pi_lock related SMP & CRITICAL_IRQSOFF_TIMING lockups, this could resolve the lockups reported by Mark H. Johnson.

- fixed one more PI buglet: wake up the new owner _after_ restoring the priority of the old owner.

- made the NMI oopser more robust - it should print out some message in pretty much any locking scenario.

- added the blocker device used by Esben Nielsen's pi_test suite.

- added user-triggerable ALSA xrun tracing to the patch: if a sound IO channel has xrun_debug enabled in /proc then user_trace_stop() will be called before printing the xrun message, and the current trace will be saved to /proc/latency_trace. This is a 'one-shot' tracing method for now. I can be activated via:

echo 1 > /proc/asound/card0/pcm0p/xrun_debug

echo 1 > /proc/sys/kernel/trace_user_triggered echo 1 > /proc/sys/kernel/trace_freerunning echo 0 > /proc/sys/kernel/preempt_max_latency echo 0 > /proc/sys/kernel/preempt_thresh echo 0 > /proc/sys/kernel/preempt_wakeup_timing

./gettimeofday 0 1

gettimeofday.c is attached below. The JACK fifo xrun source was found via this tracing facility.

to create a -V0.7.30-9 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-9

Ingo

-- gettimeofday.c:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include <linux/unistd.h>

int main (int argc, char **argv)

{
	if (argc != 3) {
		printf("usage: gettimeofday <val1> <val2>\n");
		exit(0);
	}
	gettimeofday(atol(argv[1]), atol(argv[2]));

	return 0;
}

K.R. Foley

2004-11-23 18:53 UTC #760 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

Quoted 3 lines · Ingo Molnar

i have released the -V0.7.30-9 Real-Time Preemption patch, which can be downloaded from the usual place:

A couple of observations that I would like to share:

1) With all of the tracing compiled in, there is a measurable amount of additional latency, even when the tracing is disabled via the proc interfaces. So if you want to measure best-case (or worst-case) latencies turn off the tracing facilities to do so. For example, using a blocking read, the worst case latency from interrupt to read: wo/tracing ~68 usec, w/tracing ~110 usec. I am attaching two histograms generated by the rtc histogram by running a slightly modified version of realfeel.

2) Also there is a big difference in latencies between blocking reads and using signals to notify the process that data is available. I am also attaching a histogram generated by the rtc histogram by running amlat. Part of the numbers in the > 10 msec bucket are into the hundreds of msecs.

All of these numbers were generated using a comparable load from stress-kernel.

No tracing using read:
rtc latency histogram of {realfeel/5162, 671921 samples}:
3 221135
4 32396
5 38047
6 62650
7 99079
8 62347
9 47702
10 37572
11 27323
12 16734
13 9889
14 6081
15 3607
16 2214
17 1440
18 784
19 494
20 355
21 269
22 215
23 186
24 163
25 124
26 123
27 93
28 85
29 85
30 90
31 80
32 61
33 72
34 47
35 53
36 49
37 39
38 29
39 26
40 22
41 15
42 19
43 18
44 23
45 15
46 12
47 4
48 10
49 12
50 4
51 4
52 4
53 3
54 2
55 4
56 3
57 2
59 1
60 1
61 1
62 1
67 1
68 2

Tracing on using read:

Nov 22 19:37:39 porky kernel: rtc latency histogram of {realfeel/6592, 487037 samples}:
Nov 22 19:37:39 porky kernel: 13 10
Nov 22 19:37:39 porky kernel: 14 88142
Nov 22 19:37:39 porky kernel: 15 113256
Nov 22 19:37:39 porky kernel: 16 77884
Nov 22 19:37:39 porky kernel: 17 60492
Nov 22 19:37:39 porky kernel: 18 41451
Nov 22 19:37:39 porky kernel: 19 23587
Nov 22 19:37:39 porky kernel: 20 15409
Nov 22 19:37:39 porky kernel: 21 11975
Nov 22 19:37:39 porky kernel: 22 14672
Nov 22 19:37:39 porky kernel: 23 11843
Nov 22 19:37:39 porky kernel: 24 8318
Nov 22 19:37:39 porky kernel: 25 5595
Nov 22 19:37:39 porky kernel: 26 3546
Nov 22 19:37:39 porky kernel: 27 2057
Nov 22 19:37:39 porky kernel: 28 1192
Nov 22 19:37:39 porky kernel: 29 819
Nov 22 19:37:39 porky kernel: 30 682
Nov 22 19:37:39 porky kernel: 31 665
Nov 22 19:37:39 porky kernel: 32 662
Nov 22 19:37:39 porky kernel: 33 540
Nov 22 19:37:39 porky kernel: 34 494
Nov 22 19:37:39 porky kernel: 35 433
Nov 22 19:37:39 porky kernel: 36 290
Nov 22 19:37:39 porky kernel: 37 227
Nov 22 19:37:39 porky kernel: 38 189
Nov 22 19:37:39 porky kernel: 39 129
Nov 22 19:37:39 porky kernel: 40 110
Nov 22 19:37:39 porky kernel: 41 78
Nov 22 19:37:39 porky kernel: 42 85
Nov 22 19:37:39 porky kernel: 43 102
Nov 22 19:37:39 porky kernel: 44 314
Nov 22 19:37:39 porky kernel: 45 189
Nov 22 19:37:39 porky kernel: 46 137
Nov 22 19:37:39 porky kernel: 47 105
Nov 22 19:37:39 porky kernel: 48 147
Nov 22 19:37:39 porky kernel: 49 101
Nov 22 19:37:39 porky kernel: 50 99
Nov 22 19:37:39 porky kernel: 51 93
Nov 22 19:37:39 porky kernel: 52 61
Nov 22 19:37:39 porky kernel: 53 57
Nov 22 19:37:39 porky kernel: 54 44
Nov 22 19:37:39 porky kernel: 55 50
Nov 22 19:37:39 porky kernel: 56 49
Nov 22 19:37:39 porky kernel: 57 50
Nov 22 19:37:39 porky kernel: 58 48
Nov 22 19:37:39 porky kernel: 59 49
Nov 22 19:37:39 porky kernel: 60 41
Nov 22 19:37:39 porky kernel: 61 31
Nov 22 19:37:39 porky kernel: 62 23
Nov 22 19:37:39 porky kernel: 63 19
Nov 22 19:37:39 porky kernel: 64 23
Nov 22 19:37:39 porky kernel: 65 27
Nov 22 19:37:39 porky kernel: 66 18
Nov 22 19:37:39 porky kernel: 67 17
Nov 22 19:37:39 porky kernel: 68 15
Nov 22 19:37:39 porky kernel: 69 16
Nov 22 19:37:39 porky kernel: 70 17
Nov 22 19:37:39 porky kernel: 71 11
Nov 22 19:37:39 porky kernel: 72 11
Nov 22 19:37:39 porky kernel: 73 19
Nov 22 19:37:40 porky kernel: 74 15
Nov 22 19:37:40 porky kernel: 75 24
Nov 22 19:37:40 porky kernel: 76 26
Nov 22 19:37:40 porky kernel: 77 14
Nov 22 19:37:40 porky kernel: 78 15
Nov 22 19:37:40 porky kernel: 79 17
Nov 22 19:37:40 porky kernel: 80 19
Nov 22 19:37:40 porky kernel: 81 12
Nov 22 19:37:40 porky kernel: 82 4
Nov 22 19:37:40 porky kernel: 83 7
Nov 22 19:37:40 porky kernel: 84 9
Nov 22 19:37:40 porky kernel: 85 8
Nov 22 19:37:40 porky kernel: 86 11
Nov 22 19:37:40 porky kernel: 87 8
Nov 22 19:37:40 porky kernel: 88 4
Nov 22 19:37:40 porky kernel: 89 3
Nov 22 19:37:40 porky kernel: 90 1
Nov 22 19:37:40 porky kernel: 91 5
Nov 22 19:37:40 porky kernel: 92 5
Nov 22 19:37:40 porky kernel: 94 3
Nov 22 19:37:40 porky kernel: 95 1
Nov 22 19:37:40 porky kernel: 96 2
Nov 22 19:37:40 porky kernel: 97 2
Nov 22 19:37:40 porky kernel: 98 2
Nov 22 19:37:40 porky kernel: 99 1
Nov 22 19:37:40 porky kernel: 101 1
Nov 22 19:37:40 porky kernel: 107 2
Nov 22 19:37:40 porky kernel: 110 1

No tracing using signals:
rtc latency histogram of {amlat/7215, 535020 samples}:
4 1
9 18916
10 125448
11 29422
12 41496
13 22222
14 15344
15 12256
16 10935
17 11157
18 11192
19 11456
20 12994
21 13832
22 14704
23 14330
24 12625
25 11837
26 11839
27 11206
28 10307
29 10057
30 10183
31 13115
32 11438
33 11897
34 11559
35 7817
36 5823
37 4660
38 3944
39 3106
40 2763
41 2424
42 1961
43 1729
44 1348
45 1154
46 984
47 801
48 682
49 537
50 467
51 345
52 288
53 214
54 182
55 162
56 131
57 117
58 135
59 118
60 111
61 120
62 118
63 89
64 103
65 95
66 92
67 85
68 62
69 45
70 61
71 39
72 29
73 40
74 28
75 25
76 25
77 18
78 18
79 11
80 13
81 9
82 13
83 10
84 11
85 5
86 2
87 4
88 2
89 3
90 2
91 5
92 2
93 1
94 2
95 4
96 3
97 2
98 1
99 3
103 1
104 1
108 2
118 1
7860 1
9999 43

K.R. Foley

2004-11-23 19:08 UTC #761 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

In reply to Ingo Molnar

Quoted 3 lines · Ingo Molnar

i have released the -V0.7.30-9 Real-Time Preemption patch, which can be downloaded from the usual place:

A couple of observations that I would like to share:

One other thing regarding my last email. The numbers were generated using -V0.7.30-3, not -9. Sorry.

Florian Schmidt

2004-11-24 00:34 UTC #762 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

In reply to Ingo Molnar

Quoted 15 lines · On Tue, 23 Nov 2004 18:58:23 +0100 Ingo Molnar <[email protected]> wrote:

- added user-triggerable ALSA xrun tracing to the patch: if a sound IO channel has xrun_debug enabled in /proc then user_trace_stop() will be called before printing the xrun message, and the current trace will be saved to /proc/latency_trace. This is a 'one-shot' tracing method for now. I can be activated via:
echo 1 > /proc/asound/card0/pcm0p/xrun_debug
echo 1 > /proc/sys/kernel/trace_user_triggered echo 1 > /proc/sys/kernel/trace_freerunning echo 0 > /proc/sys/kernel/preempt_max_latency echo 0 > /proc/sys/kernel/preempt_thresh echo 0 > /proc/sys/kernel/preempt_wakeup_timing

Quoted 4 lines

./gettimeofday 0 1
gettimeofday.c is attached below. The JACK fifo xrun source was found via this tracing facility.

Hi, i have some problem with unresolved symbols loading my alsa sound card driver with this kernel version. At first i suspected an unclean build, but then i did make clean bzImage modules and the unresolved symbols persist (i have wakeup/nonpreemptible/interrupts-off tracing enabled (see .config)):

snd_pcm: Unknown symbol user_trace_stop
snd_ac97_codec: Unknown symbol snd_interval_refine
snd_ac97_codec: Unknown symbol snd_pcm_hw_rule_add
snd_cs46xx: Unknown symbol snd_ac97_write_cache
snd_cs46xx: Unknown symbol snd_pcm_new
snd_cs46xx: Unknown symbol snd_pcm_lib_preallocate_pages_for_all
snd_cs46xx: Unknown symbol snd_pcm_format_unsigned
snd_cs46xx: Unknown symbol snd_pcm_format_physical_width
snd_cs46xx: Unknown symbol snd_ac97_mixer
snd_cs46xx: Unknown symbol snd_ac97_bus
snd_cs46xx: Unknown symbol snd_pcm_lib_malloc_pages
snd_cs46xx: Unknown symbol snd_pcm_lib_ioctl
snd_cs46xx: Unknown symbol snd_pcm_lib_free_pages
snd_cs46xx: Unknown symbol snd_pcm_set_ops
snd_cs46xx: Unknown symbol snd_pcm_hw_constraint_list
snd_cs46xx: Unknown symbol snd_pcm_format_big_endian
snd_cs46xx: Unknown symbol snd_pcm_lib_preallocate_free_for_all
snd_cs46xx: Unknown symbol snd_pcm_period_elapsed
snd_cs46xx: Unknown symbol snd_ac97_write
snd_cs46xx: Unknown symbol snd_ac97_read
snd_cs46xx: Unknown symbol snd_pcm_format_width

.config attached

flo

Ingo Molnar

2004-11-24 02:17 UTC #763 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

Quoted 7 lines · Florian Schmidt

Hi, i have some problem with unresolved symbols loading my alsa sound card driver with this kernel version. At first i suspected an unclean build, but then i did make clean bzImage modules and the unresolved symbols persist (i have wakeup/nonpreemptible/interrupts-off tracing enabled (see .config)):
snd_pcm: Unknown symbol user_trace_stop

does adding this line to kernel/latency.c resolve it?:

  EXPORT_SYMBOL(user_trace_stop);

Ingo

Florian Schmidt

2004-11-24 02:48 UTC #764 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

Quoted 9 lines · On Wed, 24 Nov 2004 04:19:08 +0100 Ingo Molnar <[email protected]> wrote:

Quoted 6 lines · Florian Schmidt
Hi, i have some problem with unresolved symbols loading my alsa sound card driver with this kernel version. At first i suspected an unclean build, but then i did make clean bzImage modules and the unresolved symbols persist (i have wakeup/nonpreemptible/interrupts-off tracing enabled (see .config)):

Quoted 5 lines

Quoted 1 line
snd_pcm: Unknown symbol user_trace_stop
does adding this line to kernel/latency.c resolve it?:
  EXPORT_SYMBOL(user_trace_stop);

yes, modules load fine now. thanks.

flo

Lee Revell

2004-11-24 00:59 UTC #765 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

In reply to Ingo Molnar

Quoted 4 lines · Ingo Molnar

i have released the -V0.7.30-9 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/

I have notices some weird interactivity issues with this. These are also present in T3.

The symptom is that CPU bound tasks like kernel compiles will starve I/O bound tasks like evolution for a _long_ time. If I have a kernel build and external modules building at the same time and Evolution goes to "Update message list...", it can sit and spin with a blank message pane for a minute or two. If I suspend the builds, the message list renders immediately.

It seems like the build process is constantly preempting the Evolution process, preventing the latter from making much progress. The build on the other hand progresses fine.

AIUI I/O bound, interactive tasks like a mail client should get scheduled in preference to CPU bound tasks like builds. The scheduler has heuristics to distinguish the two types of tasks and boosts the dynamic priority of the former, right? It seems like exactly the opposite is happening.

Another possibility is that Evolution really DOES use so many cycles to generate the message list that it looks like a CPU bound process to the kernel. Unfortunately I think this seems most likely. For example, evolution still consumes a hell of a lot of CPU at a low nice value. It just makes other tasks stall this way.

Do I need to just find a better mail client?

Lee

Ingo Molnar

2004-11-24 02:45 UTC #766 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

Quoted 9 lines · Lee Revell

Quoted 4 lines · Ingo Molnar
i have released the -V0.7.30-9 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
I have notices some weird interactivity issues with this. These are also present in T3.

Quoted 6 lines

The symptom is that CPU bound tasks like kernel compiles will starve I/O bound tasks like evolution for a _long_ time. If I have a kernel build and external modules building at the same time and Evolution goes to "Update message list...", it can sit and spin with a blank message pane for a minute or two. If I suspend the builds, the message list renders immediately.

could you try the vanilla -rc2-mm2 kernel (with PREEMPT enabled), does it behave in such a way too? At first sight this could be a property of the upstream scheduler, but maybe it's special to PREEMPT_RT.

Ingo

Steven Rostedt

reviewer

2004-11-24 14:09 UTC #767 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

Quoted 10 lines · Ingo Molnar

Quoted 7 lines · Lee Revell
The symptom is that CPU bound tasks like kernel compiles will starve I/O bound tasks like evolution for a _long_ time. If I have a kernel build and external modules building at the same time and Evolution goes to "Update message list...", it can sit and spin with a blank message pane for a minute or two. If I suspend the builds, the message list renders immediately.

Quoted 4 lines

could you try the vanilla -rc2-mm2 kernel (with PREEMPT enabled), does it behave in such a way too? At first sight this could be a property of the upstream scheduler, but maybe it's special to PREEMPT_RT.

Have you notice this behavior with other interactive (I/O) tasks, such as bash. Evolution is quite a big utility, and might be doing something in the background. If you see the same behavior with bash then there is no doubt that the compile is slowing down an I/O intensive task.

Another variable can be memory. Are you running this on something with adequate memory, or is you harddrive churning like mad and you're constantly thrashing the swap space?

-- Steve

Lee Revell

2004-11-24 17:34 UTC #768 lore raw

kernel builds starving evolution process - scheduler issue? (was Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9)

Quoted 16 lines · Steven Rostedt

Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Lee Revell
The symptom is that CPU bound tasks like kernel compiles will starve I/O bound tasks like evolution for a _long_ time. If I have a kernel build and external modules building at the same time and Evolution goes to "Update message list...", it can sit and spin with a blank message pane for a minute or two. If I suspend the builds, the message list renders immediately.
Quoted 4 lines
could you try the vanilla -rc2-mm2 kernel (with PREEMPT enabled), does it behave in such a way too? At first sight this could be a property of the upstream scheduler, but maybe it's special to PREEMPT_RT.

Quoted 5 lines

Have you notice this behavior with other interactive (I/O) tasks, such as bash. Evolution is quite a big utility, and might be doing something in the background. If you see the same behavior with bash then there is no doubt that the compile is slowing down an I/O intensive task.

No. Only evolution (2.0) exhibits the problem. But, it looks like evolution uses a comparable amount of CPU to a kernel build just updating the message list. All stracing it shows me is that it spends a hell of a lot of time polling(). I think this might be a bloat issue.

Quoted 4 lines

Another variable can be memory. Are you running this on something with adequate memory, or is you harddrive churning like mad and you're constantly thrashing the swap space?

No, I have plenty of RAM (512M). I am using a 600Mhz C3 so the system is probably CPU bound. But, it seems like evolution should make a little more progress. I often find myself having to background all build processes for a few seconds to let the message list render. Once the list renders, and I resume the builds, evolution is more or less usable. Running gtk-gnutella in the background will also make evolution horribly slow.

Running the offending, CPU bound processes at a high nice value solves the problem. But now I am wasting half my fscking cycles "Updating
message list...". Grr.

Lee

Ingo Molnar

2004-11-24 09:14 UTC #769 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-10

In reply to Ingo Molnar

i have released the -V0.7.30-10 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

the most important fixes are the ones to the priority inheritance logic (affecting the latency of RT tasks), discovered and reported by Esben Nielsen. I also found two more PI bugs running the new pi_test2 code from Esben.

Changes since -V0.7.30-9:

- PI fixes:

- the waiter->prio field caused wrong priority settings upon unlock, resulting in PI bugs in the nested-locking case.

- use rt_task() when determining PI tasks, not p->policy.

- in the blocking-on-blocked-task nesting case both promote now-RT tasks to the pi_waiters list and queue them to the head of the wait list, and demote now-non-RT tasks from the pi_waiters list and queue them to the tail of the wait list.

- PI-debugging blocker device update from Esben Nielsen

- module build fix: export user_trace_stop symbol, this fixes the error reported by Florian Schmidt

- tracer fix: in the default !freerunning tracing mode, if the trace buffer overflows (this is relatively rare, but can happen) then the tracer overwrote kernel memory that leads to lockups/kernel crashes. Maybe this bug was also the source of the truncated trace bug reported by Mark H. Johnson?

- reduce tracing overhead within schedule() when !tracing_enabled.

to create a -V0.7.30-10 tree from scratch, the patching order is:

Ingo

Ingo Molnar

2004-11-24 10:26 UTC #770 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

i have released the -V0.7.31-0 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is a merge of the -30-10 patch to 2.6.10-rc2-mm3. There are no other changes.

to create a -V0.7.31-0 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.31-0

Ingo

Gene Heskett

2004-11-24 20:32 UTC #771 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

Quoted 9 lines · Ingo Molnar

i have released the -V0.7.31-0 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is a merge of the -30-10 patch to 2.6.10-rc2-mm3. There are no other changes.
to create a -V0.7.31-0 tree from scratch, the patching order is:

Except by the time I got there, it was 31-3.

Quoted 10 lines

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz
2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-r
c2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-r
c2-mm3-V0.7.31-0
Ingo

It built relatively uneventfully, set at 4 for the preempt behaviour, and with a couple of minor additions to the dmesg, booted about the same.

The keyboard actually feels snappier in kmail as it tends to hang pretty bad while spamassassin is running. So far, everything I've tried has just worked, which is a long ways from the last time I tried it Ingo, congratulations & a tip of the hat to you. Its come a heck of a long ways in the last 3 weeks.

Now, since this is based on the rc2-mm3 tree, I note that I've got all the irq shareing back in place that was fixed for the most part in rc2-plain with the kernel argument 'acpi_skip_timer_override', which did not seem to be effective for later than 2.6.10-rc2 builds.

I'm also running the cfq scheduler as that also seems to level the cpu hogs quite a bit here, although it did slow down the amanda run last night (not on this kernel, but -rc2-bk8) by an hour or so since gzip was pretty badly starved by cfq. That doesn't hurt my feelings as the machine now remains usable while amanda is running. Running the anticipatory scheduler, amanda made this 2800XP a real sickly dog.

What else can I report on that you would like to see? Take a look at my attached .config and shoot me a list.

Gawd this is smooth!

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.29% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.10-rc2-mm3-V0.7.31-3
# Wed Nov 24 10:29:07 2004
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
# CONFIG_CLEAN_COMPILE is not set
CONFIG_BROKEN=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_LOG_BUF_SHIFT=16
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
CONFIG_MK7=y
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
# CONFIG_SMP is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_UP_APIC=y
# CONFIG_X86_UP_IOAPIC is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_TSC=y
# CONFIG_X86_MCE is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
CONFIG_X86_CPUID=y

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
CONFIG_HIGHPTE=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
CONFIG_HAVE_DEC_LOCK=y
# CONFIG_REGPARM is not set

#
# Performance-monitoring counters support
#
# CONFIG_PERFCTR is not set
CONFIG_KERN_PHYS_OFFSET=1
# CONFIG_KEXEC is not set

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support #
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
# CONFIG_ACPI_SLEEP is not set
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
# CONFIG_ACPI_BUTTON is not set CONFIG_ACPI_VIDEO=y
# CONFIG_ACPI_FAN is not set CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
# CONFIG_X86_PM_TIMER is not set
# CONFIG_ACPI_CONTAINER is not set

#
# APM (Advanced Power Management) BIOS Support
#
CONFIG_APM=y
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not set
CONFIG_APM_RTC_IS_GMT=y
# CONFIG_APM_ALLOW_INTS is not set
CONFIG_APM_REAL_MODE_POWER_OFF=y

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PC-card bridges
#
CONFIG_PCMCIA_PROBE=y

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_AOUT is not set
CONFIG_BINFMT_MISC=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_PC_CML1=y
# CONFIG_PARPORT_SERIAL is not set
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_OTHER is not set
CONFIG_PARPORT_1284=y

#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
CONFIG_ISAPNP=y
CONFIG_PNPBIOS=y
CONFIG_PNPBIOS_PROC_FS=y
CONFIG_PNPACPI=y

#
# Block devices
#
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_LBD is not set
# CONFIG_CDROM_PKTCDVD is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives #
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set CONFIG_BLK_DEV_IDEDISK=y
# CONFIG_IDEDISK_MULTI_MODE is not set CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_IDEPCI_SHARE_IRQ is not set
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_GENERIC is not set
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
CONFIG_BLK_DEV_AMD74XX=y
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
CONFIG_CHR_DEV_SG=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs #
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_CPQFCTS is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PCI2000 is not set
# CONFIG_SCSI_PCI2220I is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA6322 is not set
# CONFIG_SCSI_SEAGATE is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
# CONFIG_BLK_DEV_MD is not set
# CONFIG_BLK_DEV_DM is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_IP_TCPDIAG=y
# CONFIG_IP_TCPDIAG_IPV6 is not set
# CONFIG_IPV6 is not set
# CONFIG_NETFILTER is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
# CONFIG_NET_CLS_ROUTE is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_KGDBOE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NETPOLL_RX is not set
# CONFIG_NETPOLL_TRAP is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=y
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_ETHERTAP is not set
# CONFIG_NET_SB1000 is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
CONFIG_FORCEDETH=m
# CONFIG_CS89x0 is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_NET_POCKET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
CONFIG_R8169=m
# CONFIG_R8169_NAPI is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1600
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1200
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input I/O drivers
#
# CONFIG_GAMEPORT is not set
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_INPORT is not set
# CONFIG_MOUSE_LOGIBM is not set
# CONFIG_MOUSE_PC110PAD is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_UINPUT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=2
CONFIG_SERIAL_8250_EXTENDED=y
# CONFIG_SERIAL_8250_MANY_PORTS is not set
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
# CONFIG_SERIAL_8250_MULTIPORT is not set
# CONFIG_SERIAL_8250_RSA is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=y
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
# CONFIG_TIPAR is not set

#
# IPMI
#
CONFIG_IPMI_HANDLER=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=y
# CONFIG_IPMI_SI is not set
# CONFIG_IPMI_WATCHDOG is not set
# CONFIG_IPMI_POWEROFF is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
CONFIG_HW_RANDOM=y
# CONFIG_NVRAM is not set
CONFIG_RTC=y
CONFIG_RTC_HISTOGRAM=y
CONFIG_BLOCKER=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_INTEL_MCH is not set
CONFIG_AGP_NVIDIA=y
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_GAMMA is not set
# CONFIG_DRM_R128 is not set
CONFIG_DRM_RADEON=y
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
# CONFIG_HANGCHECK_TIMER is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=y
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_ELEKTOR is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
CONFIG_I2C_ISA=y
CONFIG_I2C_NFORCE2=y
# CONFIG_I2C_PARPORT is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set
# CONFIG_I2C_PCA_ISA is not set

#
# Hardware Sensors Chip support
#
CONFIG_I2C_SENSOR=y
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_VIA686A is not set
CONFIG_SENSORS_W83781D=y
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set

#
# Other I2C Chip support
#
CONFIG_SENSORS_EEPROM=m
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_RTC8564 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
CONFIG_VIDEO_DEV=m

#
# Video For Linux
#

#
# Video Adapters
#
CONFIG_VIDEO_BT848=m
# CONFIG_VIDEO_PMS is not set
# CONFIG_VIDEO_BWQCAM is not set
# CONFIG_VIDEO_CQCAM is not set
# CONFIG_VIDEO_W9966 is not set
# CONFIG_VIDEO_CPIA is not set
# CONFIG_VIDEO_SAA5246A is not set
# CONFIG_VIDEO_SAA5249 is not set
# CONFIG_TUNER_3036 is not set
# CONFIG_VIDEO_STRADIS is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_VIDEO_ZR36120 is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DPC is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_OVCAMCHIP is not set

#
# Radio Adapters
#
# CONFIG_RADIO_CADET is not set
# CONFIG_RADIO_RTRACK is not set
# CONFIG_RADIO_RTRACK2 is not set
# CONFIG_RADIO_AZTECH is not set
# CONFIG_RADIO_GEMTEK is not set
# CONFIG_RADIO_GEMTEK_PCI is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_MAESTRO is not set
# CONFIG_RADIO_SF16FMI is not set
# CONFIG_RADIO_SF16FMR2 is not set
# CONFIG_RADIO_TERRATEC is not set
# CONFIG_RADIO_TRUST is not set
# CONFIG_RADIO_TYPHOON is not set
# CONFIG_RADIO_ZOLTRIX is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEO_BUF=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_IR=m

#
# Graphics support
#
CONFIG_FB=y
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y
# CONFIG_FB_HGA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON_OLD is not set
CONFIG_FB_RADEON=y
CONFIG_FB_RADEON_I2C=y
# CONFIG_FB_RADEON_DEBUG is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE is not set

#
# Logo configuration
#
# CONFIG_LOGO is not set

#
# Sound
#
CONFIG_SOUND=y

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
# CONFIG_SND_SEQ_DUMMY is not set
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
# CONFIG_SND_MTPAV is not set
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m

#
# ISA devices
#
# CONFIG_SND_AD1816A is not set
# CONFIG_SND_AD1848 is not set
# CONFIG_SND_CS4231 is not set
# CONFIG_SND_CS4232 is not set
# CONFIG_SND_CS4236 is not set
# CONFIG_SND_ES968 is not set
# CONFIG_SND_ES1688 is not set
# CONFIG_SND_ES18XX is not set
# CONFIG_SND_GUSCLASSIC is not set
# CONFIG_SND_GUSEXTREME is not set
# CONFIG_SND_GUSMAX is not set
# CONFIG_SND_INTERWAVE is not set
# CONFIG_SND_INTERWAVE_STB is not set
# CONFIG_SND_OPTI92X_AD1848 is not set
# CONFIG_SND_OPTI92X_CS4231 is not set
# CONFIG_SND_OPTI93X is not set
# CONFIG_SND_SB8 is not set
# CONFIG_SND_SB16 is not set
# CONFIG_SND_SBAWE is not set
# CONFIG_SND_WAVEFRONT is not set
# CONFIG_SND_ALS100 is not set
# CONFIG_SND_AZT2320 is not set
# CONFIG_SND_CMI8330 is not set
# CONFIG_SND_DT019X is not set
# CONFIG_SND_OPL3SA2 is not set
# CONFIG_SND_SGALAXY is not set
# CONFIG_SND_SSCAPE is not set

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VX222 is not set

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_SPLIT_ISO is not set
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_UHCI_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_BLUETOOTH_TTY is not set
# CONFIG_USB_MIDI is not set
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=y
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_RW_DETECT is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_HP8200e is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_USB_HIDINPUT=y
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_HPUSBSCSI is not set

#
# USB Multimedia devices
#
CONFIG_USB_DABUSB=m
CONFIG_USB_VICAM=m
CONFIG_USB_DSBR=m
CONFIG_USB_IBMCAM=m
CONFIG_USB_KONICAWC=m
CONFIG_USB_OV511=m
CONFIG_USB_SE401=m
CONFIG_USB_SN9C102=m
CONFIG_USB_STV680=m

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=y
# CONFIG_USB_SERIAL_CONSOLE is not set
# CONFIG_USB_SERIAL_GENERIC is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
CONFIG_USB_SERIAL_PL2303=y
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OMNINET is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_TIGL is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETKIT is not set
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_TEST is not set

#
# USB ATM/DSL drivers
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_XATTR is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_QUOTA is not set
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
# CONFIG_DEVPTS_FS_XATTR is not set
# CONFIG_TMPFS is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_EXPORTFS is not set
CONFIG_SMB_FS=y
# CONFIG_SMB_NLS_DEFAULT is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=y

#
# Profiling support
#
# CONFIG_PROFILING is not set

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_DEBUG_PREEMPT=y
CONFIG_WAKEUP_TIMING=y
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set
# CONFIG_CRITICAL_IRQSOFF_TIMING is not set
CONFIG_LATENCY_TIMING=y
# CONFIG_LATENCY_TRACE is not set
CONFIG_RT_DEADLOCK_DETECT=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_USE_FRAME_POINTER is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
# CONFIG_KGDB is not set

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
# CONFIG_CRYPTO is not set

#
# Library routines
#
CONFIG_CRC_CCITT=y
CONFIG_CRC32=y
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

Rui Nuno Capela

2004-11-26 22:51 UTC #772 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

In reply to Ingo Molnar

Quoted 9 lines · Ingo Molnar

i have released the -V0.7.31-0 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is a merge of the -30-10 patch to 2.6.10-rc2-mm3. There are no other changes.

I have a problem. Better said, one half-of-a-problem :)

I've been testing the RT patches on both of my personal machines, one laptop (P4/UP) and a desktop (P4/SMT). That you probably already know.

On the P4/UP side everything has evolved smoothly, with the only major quirk now being the loopback device hanging while on mkinitrd. It's not a system lockup, only the "mkinitrd" and "mount -o loop" processes gets stuck (distro is Mandrake 10.1c). OTOH, audio performance when regarding jackd low-latency has reached such levels never dreamt before. To seal my confidence on the RT I've committed to be the primary kernel that boots by default and production. I'm happy here, so let's get to the topic.

On the P4/SMP/HT side, history has tought quite a different tale. It was already late on the VP era when this was even able to boot to the login-prompt. Then it suffered from all sorts of lockups and starvations when able to start jackd. Then happily, all that has been ironed out. One last thing, at the moment, that "reliably" locks up the machine is accessing the floppy-disk (dev/fd0). Yes, I still have one here, and it was just yesterday that I've tried to mount on it and bang! power-off and a cold-boot follows. Reproducibility? ALWAYS is often enough. Nothing shows up via serial console.

OTOH, my confidence goes down the drain when I compare the jackd low-latency performance between the latest RT-V0.7.31-3 kernel and the one supplied from SUSE 9.2 Pro (2.6.8-24). I have been checking and double-checking this too far many times with even stressful workloads:
SUSE's non-RT kernel has an edge over the latest RT ones. Jackd XRUN rates are pretty low and on the same level (e.g. less than 5 per hour with the default jack_test3.1 test), but SUSE's 2.6.8-24 is consistently on par of RT-V0.7.31-3, and even better if the RT kernel is built with some preempt-debugging options set.

Is this black-magic or what? :)

Oh well. But let's get back to reality :) How can I help on fixing this floppy showstopper? I've tried with almost every debug option set and nothing is dumped either on syslog or serial console. The only visible thing is that, once the floppy starts spinning (LED is on) the machine freezes. Weird.

Nuff said.

Cheers.

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Ingo Molnar

2004-11-26 19:29 UTC #773 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

Quoted 5 lines · Rui Nuno Capela

last thing, at the moment, that "reliably" locks up the machine is accessing the floppy-disk (dev/fd0). Yes, I still have one here, and it was just yesterday that I've tried to mount on it and bang! power-off and a cold-boot follows. Reproducibility? ALWAYS is often enough. Nothing shows up via serial console.

will take a look.

Quoted 2 lines

[...] Jackd XRUN rates are pretty low and on the same level (e.g. less than 5 per hour with the default jack_test3.1 test), [...]

could you post the jack_test summary outputs?

Quoted 5 lines

Oh well. But let's get back to reality :) How can I help on fixing this floppy showstopper? I've tried with almost every debug option set and nothing is dumped either on syslog or serial console. The only visible thing is that, once the floppy starts spinning (LED is on) the machine freezes. Weird.

how hard of a freeze is it? I.e. if you log in over the text console, and do:

	chrt -f 99 -p `pidof 'IRQ 1'`
	chrt -f 99 -p $$

can you access the sysrq keys after the freeze happens? If not, can you access them if you do:

echo 1 > /proc/sys/kernel/debug_direct_keyboard

? And finally, if the above experiments suggest that it's a hard lockup, do you have a working NMI watchdog? (i.e. do the NMI counts in /proc/interrupt increase on all CPUs?)

Ingo

Rui Nuno Capela

2004-11-26 23:42 UTC #774 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

Hi Ingo,

Quoted 11 lines · Rui Nuno Capela

Quoted 6 lines
last thing, at the moment, that "reliably" locks up the machine is accessing the floppy-disk (dev/fd0). Yes, I still have one here, and it was just yesterday that I've tried to mount on it and bang! power-off and a cold-boot follows. Reproducibility? ALWAYS is often enough. Nothing shows up via serial console.
will take a look.

Thanks, as always ;)

Quoted 5 lines

Quoted 2 lines
[...] Jackd XRUN rates are pretty low and on the same level (e.g. less than 5 per hour with the default jack_test3.1 test), [...]
could you post the jack_test summary outputs?

Of course, but only later tonight (12 hours from now?). Sorry.

Quoted 6 lines

Quoted 5 lines
Oh well. But let's get back to reality :) How can I help on fixing this floppy showstopper? I've tried with almost every debug option set and nothing is dumped either on syslog or serial console. The only visible thing is that, once the floppy starts spinning (LED is on) the machine freezes. Weird.

Quoted 7 lines

how hard of a freeze is it? I.e. if you log in over the text console, and do:
	chrt -f 99 -p `pidof 'IRQ 1'`
	chrt -f 99 -p $$
can you access the sysrq keys after the freeze happens?

The lockup is pretty hard indeed. Complete lockup. No sysrq, not even any output thru serial console. The only action that has some visible effect is turning the power/reset switch off :)

Quoted 8 lines

If not, can you access them if you do:
echo 1 > /proc/sys/kernel/debug_direct_keyboard
? And finally, if the above experiments suggest that it's a hard lockup, do you have a working NMI watchdog? (i.e. do the NMI counts in /proc/interrupt increase on all CPUs?)

Yes, nmi_watchdog=1 was set but have to double-check if the NMI counts does really pump on /proc/interrupts. Will retry and check later, again.

Bye.

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Ingo Molnar

2004-11-27 07:26 UTC #775 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

Quoted 11 lines · Rui Nuno Capela

Quoted 7 lines
how hard of a freeze is it? I.e. if you log in over the text console, and do:
	chrt -f 99 -p `pidof 'IRQ 1'`
	chrt -f 99 -p $$
can you access the sysrq keys after the freeze happens?
The lockup is pretty hard indeed. Complete lockup. No sysrq, not even any output thru serial console. The only action that has some visible effect is turning the power/reset switch off :)

note that unless you try the above, or the debug_direct_keyboard switch, 'soft' lockups will have the same symptoms: no sysrq, no serial console, an apparently hung system. So unless you've done the equivalent already, please try my suggestions.

Ingo

Rui Nuno Capela

2004-11-27 07:18 UTC #776 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

Ingo Molnar

Quoted 15 lines · Rui Nuno Capela

Quoted 4 lines
Quoted 4 lines
how hard of a freeze is it? I.e. if you log in over the text console, and do:
Quoted 8 lines
Quoted 4 lines
	chrt -f 99 -p `pidof 'IRQ 1'`
	chrt -f 99 -p $$
can you access the sysrq keys after the freeze happens?
The lockup is pretty hard indeed. Complete lockup. No sysrq, not even any output thru serial console. The only action that has some visible effect is turning the power/reset switch off :)

Quoted 7 lines

note that unless you try the above, or the debug_direct_keyboard switch, 'soft' lockups will have the same symptoms: no sysrq, no serial console, an apparently hung system. So unless you've done the equivalent already, please try my suggestions.
Ingo

Yes Master :)

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Ingo Molnar

2004-11-27 00:34 UTC #777 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

In reply to Ingo Molnar

Quoted 4 lines · Ingo Molnar

Quoted 2 lines
[...] Jackd XRUN rates are pretty low and on the same level (e.g. less than 5 per hour with the default jack_test3.1 test), [...]
could you post the jack_test summary outputs?

also, could you change JACKD_PRIO from 20 to 50? Otherwise normal IRQ handlers (IDE, etc.) will preempt jackd.

(the test-clients inherit this prio 50 setting, right?)

Ingo

Ingo Molnar

2004-11-27 06:41 UTC #778 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

In reply to Rui Nuno Capela

the -RT patchset doesnt properly detect my fd0 device, so there's definitely something broken in that area. The unpatched -rc2-mm3 kernel detects it fine. Might be an effect of IRQ threading - the floppy hardware/driver is a fragile beast.

Ingo

Ingo Molnar

2004-11-27 04:14 UTC #779 lore raw

PATCH 1/1 floppy boot-time detection fix

Quoted 4 lines · Ingo Molnar

the -RT patchset doesnt properly detect my fd0 device, so there's definitely something broken in that area. The unpatched -rc2-mm3 kernel detects it fine. Might be an effect of IRQ threading - the floppy hardware/driver is a fragile beast.

found the bug that causes the fd detection failure. It's a generic race in the upstream floppy driver, which happens to work by chance in the vanilla kernel but breaks when IRQ and softirq threading is enabled:

when the FDC hardware is initialized, it sometimes generates a floppy interrupt right away - without being told to. This interrupt can hit the detection code that executes right after the initialization code, in particular it can get intermixed with user_reset_fdc() that the detection code uses. The fd driver is fundamentally single-threaded when it comes to handling events: an unexpected irq that arrives in the wrong moment can confuse the reset_fdc() code, which, with softirq and hardirq threading on, executes in keventd.

in the stock kernel this stale irq doesnt seem to hit the detection code in the wrong moment, but i think under certain circumstances it may still happen. One of the typical incarnations of the race was the following message:

reset set in interrupt, calling c0258400

and googling for "reset set in interrupt, calling" does turn up a fair number of bootlogs (most of them 2.4 ones) that show such a detection failure, so i think upstream wants to have the fix too.

the fix is simple: delay a bit after initialization, to make sure the stale irq does not interfere with the detection code. It will be safely ignored, since do_floppy is still NULL. It might look sloppy that i went for a delay, but delay i think it is better than waiting for the irq to occur, because i dont think there's a guarantee that fdc initialization triggers an interrupt, so waiting for it could hang the boot process. A delay OTOH is totally harmless.

The attached patch implements this fix, which resolves the detection problem on my testbox.

here's again how a failure looks like:

Floppy drive(s): fd0 is 1.44M
reset set in interrupt, calling c0258400
floppy0: no floppy controllers found

and this is how it works with the fix:

Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077

i've tested this on vanilla 2.6.10-rc2-mm3 too (to make sure this doesnt break the floppy driver), and it should work fine in -BK too.

(this does not solve the irq threading related SMP lockup though, i'm attacking that problem next - now that my fd0 gets detected fine ;-) )

Ingo

Signed-off-by Ingo Molnar <[email protected]>

drivers/block/floppy.c renamed from drivers/block/floppy.c.orig +7 -0

@@ -4504,6 +4578,13 @@ int __init floppy_init(void)
 		floppy_track_buffer = NULL;
 		max_buffer_sectors = 0;
 	}
+	/*
+	 * Small 10 msec delay to let through any interrupt that
+	 * initialization might have triggered, to not
+	 * confuse detection:
+	 */
+	current->state = TASK_UNINTERRUPTIBLE;
+	schedule_timeout(HZ/100 + 1);
 
 	for (i = 0; i < N_FDC; i++) {
 		fdc = i;

Ingo Molnar

2004-11-26 19:55 UTC #780 lore raw

Re: [patch, 2.6.10-rc2] floppy boot-time detection fix

Quoted 2 lines · Ingo Molnar

(this does not solve the irq threading related SMP lockup though, i'm attacking that problem next - now that my fd0 gets detected fine ;-) )

in fact, i cannot reproduce the SMP lockup anymore and the floppy works just fine. Maybe the stale irq, even if detection went fine, mis-programmed the controller, which ended up totally locking up upon the first attempted IO?

Ingo

Ingo Molnar

2004-11-27 06:48 UTC #781 lore raw

Re: [patch, 2.6.10-rc2] floppy boot-time detection fix

Quoted 5 lines · Ingo Molnar

Quoted 2 lines
(this does not solve the irq threading related SMP lockup though, i'm attacking that problem next - now that my fd0 gets detected fine ;-) )
in fact, i cannot reproduce the SMP lockup anymore and the floppy works just fine. [...]

it turns out that this was the side-effect of me doing idle=poll.

The reason for the lockup was a bug in the -RT patch, it broke disable_hlt() which resulted in all CPUs spinning in idle with interrupts disabled - with an obvious hard lockup as a result. Fixed it in my tree.

Ingo

Ingo Molnar

2004-11-26 21:25 UTC #782 lore raw

Re: [patch, 2.6.10-rc2] floppy boot-time detection fix

In reply to Ingo Molnar

Review

Quoted 3 lines

(quoted diff) +2 -0

+	current->state = TASK_UNINTERRUPTIBLE;
+	schedule_timeout(HZ/100 + 1);

should use msleep() of course:

Signed-off-by Ingo Molnar <[email protected]>

drivers/block/floppy.c renamed from drivers/block/floppy.c.orig +6 -0

@@ -4504,6 +4504,12 @@ int __init floppy_init(void)
 		floppy_track_buffer = NULL;
 		max_buffer_sectors = 0;
 	}
+	/*
+	 * Small 10 msec delay to let through any interrupt that
+	 * initialization might have triggered, to not
+	 * confuse detection:
+	 */
+	msleep(10);
 
 	for (i = 0; i < N_FDC; i++) {
 		fdc = i;

Arjan van de Ven

2004-11-27 00:50 UTC #783 lore raw

Re: [patch, 2.6.10-rc2] floppy boot-time detection fix

In reply to Ingo Molnar

drivers/block/floppy.c int __init floppy_init(void) · line 4587

Quoted 13 lines · Ingo Molnar

drivers/block/floppy.c renamed from drivers/block/floppy.c.orig +7 -0

@@ -4504,6 +4578,13 @@ int __init floppy_init(void)
 		floppy_track_buffer = NULL;
 		max_buffer_sectors = 0;
 	}
+	/*
+	 * Small 10 msec delay to let through any interrupt that
+	 * initialization might have triggered, to not
+	 * confuse detection:
+	 */
+	current->state = TASK_UNINTERRUPTIBLE;
+	schedule_timeout(HZ/100 + 1);

how about using msleep() ?

Rui Nuno Capela

2004-11-27 06:44 UTC #784 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

In reply to Ingo Molnar

Quoted 7 lines · Ingo Molnar

the -RT patchset doesnt properly detect my fd0 device, so there's definitely something broken in that area. The unpatched -rc2-mm3 kernel detects it fine. Might be an effect of IRQ threading - the floppy hardware/driver is a fragile beast.

But it works flawlessly on my laptop (P4/UP). Could it be SMP/HT related?

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Ingo Molnar

2004-11-27 07:14 UTC #785 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-0

Quoted 10 lines · Rui Nuno Capela

Quoted 6 lines · Ingo Molnar
the -RT patchset doesnt properly detect my fd0 device, so there's definitely something broken in that area. The unpatched -rc2-mm3 kernel detects it fine. Might be an effect of IRQ threading - the floppy hardware/driver is a fragile beast.
But it works flawlessly on my laptop (P4/UP). Could it be SMP/HT related?

yeah, could be, the failure i'm seeing is on SMP. On SMP if you have interrupt threads then concurrency is higher. fd0 gets detected fine when i turn IRQ threading off.

Ingo

Ingo Molnar

2004-12-03 20:59 UTC #786 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

In reply to Ingo Molnar

i have released the -V0.7.32-0 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is a fixes-mostly release with one new feature:

implemented global RT-task balancing on SMP systems, which improves the latency of RT tasks on SMP systems. The basic problem was that the 2.6 kernel has per-CPU runqueues. In the current design there is no guarantee that if an RT task starves another, lower-prio (or same-prio) RT task in a given local runqueue, that the starved task will ever be migrated to another CPU: it has to wait for the higher-prio task to finish. In short, task migration on SMP is fundamentally non-RT and priority-insensitive. In particular the workloads and latencies reported by Mark H. Johnson reflect such SMP scheduling artifacts.

the new global RT-task balancing feature solves this problem by tracking the 'RT overload' situation (when there is more than one RT tasks on a CPU) and makes other CPUs 'pull' RT tasks (and only RT tasks) immediately when such a situation occurs.

To give an example, in the following scheduling scenario:

CPU#0 CPU#1

task-A SCHED_FIFO prio 30 task-C SCHED_FIFO prio 30 [curr]
task-B SCHED_FIFO prio 40 [curr]

task-B is the currently executing task on CPU#0, task-C is the currently executing task on CPU#1. Now on the vanilla 2.6 kernel, if task-C blocks, there's no guarantee that task-A will be run there - if there's a SCHED_NORMAL task on CPU#1's runqueue then it will run indefinitely. With global RT-balancing task-A will be scheduled on CPU#1 immediately after task-C leaves it.

furthermore, if in the same scenario, if e.g. a RT-prio 35 task-D is woken up on CPU#0, the vanilla 2.6 scheduler will not move it to CPU#1, even though it could preempt the prio 30 task-C there. With global RT-balancing this will happen and task-C will be preempted immediately and task-D runs.

on low RT load (the common case) the scheduler behaves like the stock scheduler - the new logic only kicks in if a CPU runqueue has 2 or more RT tasks running at once.

anyway, while the feature is stable on my SMP testboxes, this is still a nontrivial ~200-lines delta in the scheduler so there might be problems. UP should not be affected by this.

other changes since -V0.7.32-20:

- local-APIC shutdown fix: this should solve some of the 'reboot hangs' reports.

- more tracing fixes - might fix the 'truncated traces' problems.

- reduce the NMI watchdog frequency from 10 KHz to 1000 Hz.

- dont report futex reschedules as atomicity violations

to create a -V0.7.32-0 tree from scratch, the patching order is:

Ingo

Ingo Molnar

2004-12-03 21:04 UTC #787 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Quoted 2 lines

to create a -V0.7.32-0 tree from scratch, the patching order is:

that should read:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.32-0

Ingo

Rui Nuno Capela

2004-12-04 22:35 UTC #788 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

In reply to Ingo Molnar

Quoted 4 lines · Ingo Molnar

i have released the -V0.7.32-0 Real-Time Preemption patch, which can be downloaded from the usual place:

I have a bug to report, it shows on both of my machines (SMP and UP) now running RT-V0.7.32-2. It seems to be present also on previous RT releases, and don't even know if it's upstream.

When one usb-storage flash stick is first time unplugged:

usb 4-6: USB disconnect, address 4
slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't free all objects

 [<c010361f>] dump_stack+0x23/0x25 (20)
 [<c01467d7>] kmem_cache_destroy+0x103/0x1aa (28)
 [<c026e77a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
 [<c026f5b1>] scsi_host_dev_release+0x37/0xe1 (104)
 [<c023c6a9>] device_release+0x7a/0x7c (32)
 [<c01efd90>] kobject_cleanup+0x87/0x89 (28)
 [<c01f07eb>] kref_put+0x52/0xef (40)
 [<c01efdcc>] kobject_put+0x25/0x27 (16)
 [<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage] (16)
 [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
 [<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
 [<c023d915>] device_release_driver+0x75/0x77 (28)
 [<c023db18>] bus_remove_device+0x53/0x82 (20)
 [<c023cae1>] device_del+0x4b/0x9b (20)
 [<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
 [<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
 [<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
 [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
 [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
 [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)

Then, just when the same is plugged in again:

usb 4-6: new high speed USB device using ehci_hcd and address 5
kmem_cache_create: duplicate cache scsi_cmd_cache
BUG at mm/slab.c:1447!

------------[ cut here ]------------
kernel BUG at mm/slab.c:1447!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in: nls_utf8 nls_cp860 vfat fat nls_base usb_storage

ohci1394 ieee1394 ehci_hcd usbhid uhci_hcd intel_mch_agp agpgart evdev wacom snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event snd_seq snd_usb_usx2y snd_usb_lib snd_hwdep snd_cs46xx snd_rawmidi snd_seq_device snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd soundcore snd_page_alloc realtime commoncap w83781d i2c_sensor i2c_isa i2c_i801 i2c_core sk98lin subfs dm_mod usbcore

CPU:    1
EIP:    0060:[<c01462e9>]    Not tainted VLI
EFLAGS: 00010286   (2.6.10-rc2-mm3-RT-V0.7.32-2.0smp)

EIP is at kmem_cache_create+0x4df/0x69c
eax: 00000017 ebx: f7cd9710 ecx: c05c5d00 edx: f77b8000
esi: c030973a edi: c030973a ebp: f77b9d10 esp: f77b9cd8
ds: 007b es: 007b ss: 0068 preempt: 00000001
Process khubd (pid: 1477, threadinfo=f77b8000 task=f777c2b0)

Stack: c02f2054 c02f63aa 000005a7 00000565 f77b9d00 f7cd10c0 c0000000
ffffff80
       00000001 f7cd1080 00000080 c03418a0 f7000400 f7000428 f77b9d3c
c026e662
       c030972b 00000180 00000080 00002000 00000000 00000000 f7000490
00000000
Call Trace:
 [<c01035f4>] show_stack+0xb4/0xbc (28)
 [<c010378a>] show_registers+0x169/0x1de (56)
 [<c010399c>] die+0x10b/0x191 (68)
 [<c0103e19>] do_invalid_op+0xba/0xc4 (204)
 [<c0103237>] error_code+0x2b/0x30 (116)
 [<c026e662>] scsi_setup_command_freelist+0x94/0x115 (44)
 [<c026f8ca>] scsi_host_alloc+0x26f/0x399 (124)
 [<f8cf173f>] usb_stor_acquire_resources+0x6c/0x10a [usb_storage] (28)
 [<f8cf1c64>] storage_probe+0x1b8/0x259 [usb_storage] (40)
 [<f89ca07a>] usb_probe_interface+0x62/0x7b [usbcore] (28)
 [<c023d723>] driver_probe_device+0x2f/0x70 (24)
 [<c023d7ae>] device_attach+0x4a/0xa5 (40)
 [<c023da89>] bus_add_device+0x4d/0x89 (28)
 [<c023c9a6>] device_add+0xb1/0x137 (40)
 [<f89d1ba6>] usb_set_configuration+0x3ca/0x444 [usbcore] (84)
 [<f89cc822>] usb_new_device+0xa5/0x164 [usbcore] (44)
 [<f89cd6b0>] hub_port_connect_change+0x221/0x426 [usbcore] (56)
 [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
 [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
 [<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
Code: e8 57 5e fd ff b8 08 4f be c0 e8 c0 d1 fe ff c7 44 24 08 a7 05 00 00

c7 44 24 04 aa 63 2f c0 c7 04 24 54 20 2f c0 e8 31 5e fd ff <0f> 0b a7 05 aa 63 2f c0 eb b9 a1 44 1a 33 c0 c7 44 24 04 d0 00

After this, any USB related operation is doomed, if not the entire system.

Bye.

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Ingo Molnar

2004-12-04 22:47 UTC #789 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Quoted 11 lines · Rui Nuno Capela

Quoted 4 lines · Ingo Molnar
i have released the -V0.7.32-0 Real-Time Preemption patch, which can be downloaded from the usual place:
I have a bug to report, it shows on both of my machines (SMP and UP) now running RT-V0.7.32-2. It seems to be present also on previous RT releases, and don't even know if it's upstream.
When one usb-storage flash stick is first time unplugged:

hm, doesnt seem to be directly related to -RT. Could you try the vanilla -rc2-mm3 kernel, does it trigger there too?

Ingo

Rui Nuno Capela

2004-12-04 23:41 UTC #790 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Quoted 9 lines · Ingo Molnar

Quoted 6 lines · Rui Nuno Capela
Quoted 4 lines · Ingo Molnar
i have released the -V0.7.32-0 Real-Time Preemption patch, which can be downloaded from the usual place:

Quoted 9 lines

Quoted 5 lines
I have a bug to report, it shows on both of my machines (SMP and UP) now running RT-V0.7.32-2. It seems to be present also on previous RT releases, and don't even know if it's upstream.
When one usb-storage flash stick is first time unplugged:
hm, doesnt seem to be directly related to -RT. Could you try the vanilla -rc2-mm3 kernel, does it trigger there too?

No, it doesn't. Just tested on 2.6.10-rc2-mm3 (vanilla) and nothing wrong happens. I can plug and unplug the USB flashram stick, and again, and again, and everything stays fine.

On 2.6.10-rc2-mm3-RT-V0.7.32-2 I get trouble after the first time I unplug the thingie. From then on I rather not touch it back again ;)

Take care.

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

K.R. Foley

2004-12-04 23:55 UTC #791 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

In reply to Ingo Molnar

Quoted 13 lines · Ingo Molnar

Quoted 10 lines · Rui Nuno Capela
Quoted 3 lines · Ingo Molnar
i have released the -V0.7.32-0 Real-Time Preemption patch, which can be downloaded from the usual place:
I have a bug to report, it shows on both of my machines (SMP and UP) now running RT-V0.7.32-2. It seems to be present also on previous RT releases, and don't even know if it's upstream.

Quoted 8 lines

Quoted 1 line
When one usb-storage flash stick is first time unplugged:
hm, doesnt seem to be directly related to -RT. Could you try the vanilla -rc2-mm3 kernel, does it trigger there too?
Ingo

Gene Heskett reported a very similar problem yesterday with the subject:
Re:2.6.10-rc2-mm3-V0.7.31-19

Gene Heskett

2004-12-05 03:10 UTC #792 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Quoted 12 lines · K.R. Foley

Quoted 11 lines · Ingo Molnar
Quoted 9 lines · Rui Nuno Capela
Quoted 2 lines · Ingo Molnar
i have released the -V0.7.32-0 Real-Time Preemption patch, which can be downloaded from the usual place:
I have a bug to report, it shows on both of my machines (SMP and UP) now running RT-V0.7.32-2. It seems to be present also on previous RT releases, and don't even know if it's upstream.
When one usb-storage flash stick is first time unplugged:

Quoted 7 lines

Quoted 4 lines
hm, doesnt seem to be directly related to -RT. Could you try the vanilla -rc2-mm3 kernel, does it trigger there too?
Ingo
Gene Heskett reported a very similar problem yesterday with the
subject: Re:2.6.10-rc2-mm3-V0.7.31-19

Yes, and I can confirm that it does not do it right now, running the UP version of 2.6.10-rc3. I just tried it and this is all I got in the log:
-----------------

Dec  4 22:06:38 coyote kernel: usb 3-2.2: new full speed USB device using ohci_hcd and address 7
Dec  4 22:06:38 coyote kernel: scsi0 : SCSI emulation for USB Mass Storage devices
Dec  4 22:06:44 coyote kernel:   Vendor: OLYMPUS   Model: C-3020ZOOM(U)     Rev: 1.00
Dec  4 22:06:44 coyote kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Dec  4 22:06:44 coyote kernel: Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
Dec  4 22:06:44 coyote scsi.agent[26069]: disk at /devices/pci0000:00/0000:00:02.1/usb3/3-2/3-2.2/3-2.2:1.0/host0/target0:0:0/0:0:0:0
Dec  4 22:06:44 coyote kernel: SCSI device sda: 128000 512-byte hdwr sectors (66 MB)
Dec  4 22:06:44 coyote kernel: sda: assuming Write Enabled
Dec  4 22:06:44 coyote kernel: sda: assuming drive cache: write through
Dec  4 22:06:44 coyote kernel: SCSI device sda: 128000 512-byte hdwr sectors (66 MB)
Dec  4 22:06:44 coyote kernel: sda: assuming Write Enabled
Dec  4 22:06:44 coyote kernel: sda: assuming drive cache: write through
Dec  4 22:06:44 coyote kernel:  sda: sda1
Dec  4 22:06:44 coyote kernel: Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0
Dec  4 22:06:51 coyote kernel: usb 3-2.2: USB disconnect, address 7

-----------------

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Gene Heskett

2004-12-05 03:04 UTC #793 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

In reply to Rui Nuno Capela

Quoted 3 lines · Rui Nuno Capela

Quoted 2 lines · Ingo Molnar
i have released the -V0.7.32-0 Real-Time Preemption patch, which can be downloaded from the usual place:

I can confirm essentially this same bug, this rather voluminous output took place after I turned off my camera and unplugged the usb cable from it. At that point all other usb stuff was spotty, even the mouse till I rebooted.

Quoted 92 lines

I have a bug to report, it shows on both of my machines (SMP and UP) now running RT-V0.7.32-2. It seems to be present also on previous RT releases, and don't even know if it's upstream.
When one usb-storage flash stick is first time unplugged:
usb 4-6: USB disconnect, address 4
slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't free all objects
[<c010361f>] dump_stack+0x23/0x25 (20)
[<c01467d7>] kmem_cache_destroy+0x103/0x1aa (28)
[<c026e77a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
[<c026f5b1>] scsi_host_dev_release+0x37/0xe1 (104)
[<c023c6a9>] device_release+0x7a/0x7c (32)
[<c01efd90>] kobject_cleanup+0x87/0x89 (28)
[<c01f07eb>] kref_put+0x52/0xef (40)
[<c01efdcc>] kobject_put+0x25/0x27 (16)
[<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage]
(16) [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
[<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
[<c023d915>] device_release_driver+0x75/0x77 (28)
[<c023db18>] bus_remove_device+0x53/0x82 (20)
[<c023cae1>] device_del+0x4b/0x9b (20)
[<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
[<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
[<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
[<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
[<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
[<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
Then, just when the same is plugged in again:
usb 4-6: new high speed USB device using ehci_hcd and address 5
kmem_cache_create: duplicate cache scsi_cmd_cache
BUG at mm/slab.c:1447!
------------[ cut here ]------------
kernel BUG at mm/slab.c:1447!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in: nls_utf8 nls_cp860 vfat fat nls_base usb_storage
ohci1394 ieee1394 ehci_hcd usbhid uhci_hcd intel_mch_agp agpgart evdev wacom snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event snd_seq snd_usb_usx2y snd_usb_lib snd_hwdep snd_cs46xx snd_rawmidi snd_seq_device snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd soundcore snd_page_alloc realtime commoncap w83781d i2c_sensor i2c_isa i2c_i801 i2c_core sk98lin subfs dm_mod usbcore
CPU:    1
EIP:    0060:[<c01462e9>]    Not tainted VLI
EFLAGS: 00010286   (2.6.10-rc2-mm3-RT-V0.7.32-2.0smp)
EIP is at kmem_cache_create+0x4df/0x69c
eax: 00000017 ebx: f7cd9710 ecx: c05c5d00 edx: f77b8000
esi: c030973a edi: c030973a ebp: f77b9d10 esp: f77b9cd8
ds: 007b es: 007b ss: 0068 preempt: 00000001
Process khubd (pid: 1477, threadinfo=f77b8000 task=f777c2b0)
Stack: c02f2054 c02f63aa 000005a7 00000565 f77b9d00 f7cd10c0
c0000000 ffffff80
      00000001 f7cd1080 00000080 c03418a0 f7000400 f7000428
f77b9d3c c026e662
      c030972b 00000180 00000080 00002000 00000000 00000000
f7000490 00000000
Call Trace:
[<c01035f4>] show_stack+0xb4/0xbc (28)
[<c010378a>] show_registers+0x169/0x1de (56)
[<c010399c>] die+0x10b/0x191 (68)
[<c0103e19>] do_invalid_op+0xba/0xc4 (204)
[<c0103237>] error_code+0x2b/0x30 (116)
[<c026e662>] scsi_setup_command_freelist+0x94/0x115 (44)
[<c026f8ca>] scsi_host_alloc+0x26f/0x399 (124)
[<f8cf173f>] usb_stor_acquire_resources+0x6c/0x10a [usb_storage]
(28) [<f8cf1c64>] storage_probe+0x1b8/0x259 [usb_storage] (40)
[<f89ca07a>] usb_probe_interface+0x62/0x7b [usbcore] (28)
[<c023d723>] driver_probe_device+0x2f/0x70 (24)
[<c023d7ae>] device_attach+0x4a/0xa5 (40)
[<c023da89>] bus_add_device+0x4d/0x89 (28)
[<c023c9a6>] device_add+0xb1/0x137 (40)
[<f89d1ba6>] usb_set_configuration+0x3ca/0x444 [usbcore] (84)
[<f89cc822>] usb_new_device+0xa5/0x164 [usbcore] (44)
[<f89cd6b0>] hub_port_connect_change+0x221/0x426 [usbcore] (56)
[<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
[<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
[<c01009b1>] kernel_thread_helper+0x5/0xb (142893076)
Code: e8 57 5e fd ff b8 08 4f be c0 e8 c0 d1 fe ff c7 44 24 08 a7 05
00 00 c7 44 24 04 aa 63 2f c0 c7 04 24 54 20 2f c0 e8 31 5e fd ff <0f> 0b a7 05 aa 63 2f c0 eb b9 a1 44 1a 33 c0 c7 44 24 04 d0 00
After this, any USB related operation is doomed, if not the entire system.
Bye.

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Esben Nielsen

2004-12-05 23:19 UTC #794 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

In reply to Ingo Molnar

Quoted 3 lines · Ingo Molnar

[...]
on low RT load (the common case)

In the system I deal with on my day job, 50% of the CPU load is from RT tasks!

In fact, I think that is quite normal for industrial applications. Those tend to be PLC-like: You run your scan every 10 ms. First you read your input, then you do your calculations then you set your output. You do the same thing every time. You can thus safely tick around at a CPU load close to 100%!!

This is quite opposit to how telecom real-time systems works: Those are usually event based. I.e. they process things when they occur. The load can vary a lot depending on how much traffic has to go through the device.

Esben

Ingo Molnar

2004-12-06 13:17 UTC #795 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Quoted 8 lines · Esben Nielsen

Quoted 3 lines · Ingo Molnar
[...]
on low RT load (the common case)
In the system I deal with on my day job, 50% of the CPU load is from RT tasks!

i'm not sure what point you are trying to make, but 'low RT load' in this context means up to a load of 1.0. (i.e. one or less tasks running on average)

also, this only applies to SMP systems.

thirdly, even if the new balancing code kicks in, the overhead is not that bad, and it's for a purpose: we rather want an RT task to run on a free CPU.

Ingo

Esben Nielsen

2004-12-06 15:04 UTC #796 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Quoted 9 lines · Ingo Molnar

Quoted 6 lines · Esben Nielsen
Quoted 3 lines · Ingo Molnar
[...]
on low RT load (the common case)

Quoted 7 lines

Quoted 2 lines
In the system I deal with on my day job, 50% of the CPU load is from RT tasks!
i'm not sure what point you are trying to make, but 'low RT load' in this context means up to a load of 1.0. (i.e. one or less tasks running on average)

The point I want to make is that RT tasks is not always of the type which reacts to some interrupt, does a small job and then goes to sleep. Sometimes you also want larger jobs done realtime. Forinstance an application waking up every 10ms, running for 5ms before going to sleep again. That ofcourse gives you a RT-load <= 1.0.

In more complicated systems you can have a task running every 5ms, another at every 25ms and another at every 100ms. That gives you a RT-load of up to 3! And it is a even common case because you try to syncronize the tasks, i.e. you wake all 3 tasks up at the same time but then you know that on your typical UP system they will be executed by priority, i.e. first the 5ms one, then the 25ms one and last the 100ms.

So my point was: Low RT-load might not be the common case on specific systems. It is on the desktop where you want to run some multimedia multimedia task(s) and have fast CPU compared to the job you need done in RT but in embedded devices you do not pick such a big CPU and you often use a large part on it in you RT control-loops.

Quoted 6 lines

also, this only applies to SMP systems.
thirdly, even if the new balancing code kicks in, the overhead is not that bad, and it's for a purpose: we rather want an RT task to run on a free CPU.

Good to hear. I was going to ask if the overhead wasn't too big :-)

However, I do have a feeling that it is rather expensive to move tasks around. If I was going to design a real-time application on a 2 CPU SMP system, I think I would CPU lock my RT-tasks to make the system more deterministic. I.e. I would probably "port" the above mentioned application by putting the 5 ms task on one CPU, the 25ms on the other and the 100 ms on the whatever CPU has the less CPU load. Then the other non-real-time tasks in the system can wander around as they please.

But that is ofcourse not something you can do unless you know the specific hardware system you will run your application on. That is not the case when you make multimedia software for the PC. I.e. in that case moving RT-tasks around makes perfect sense!

Quoted 2 lines

Ingo

Esben

Ingo Molnar

2004-12-06 15:28 UTC #797 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.32-0

Quoted 2 lines · Esben Nielsen

So my point was: Low RT-load might not be the common case on specific systems. [...]

i did not suggest it was. The reason why i mentioned it was to point out that _non-RT usage_ does not see any overhead, i.e. ordinary Linux boxes. (which nevertheless do run RT tasks occasionally.)

of course in the RT-specific it can be common - Mark's test is one such workload. If it werent widespread i'd not try to solve the problem...

Ingo

Ingo Molnar

2004-12-07 13:29 UTC #798 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-4

In reply to Ingo Molnar

i have released the -V0.7.32-4 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release.

Changes since -V0.7.32-2:

- fixed a seqlock related xtime_lock lockup scenario - this could explain the SMP lockups reported by Mark H. Johnson.

- fixed a small buglet in the new SMP RT-balancing code, which could lead to bad balancing in certain rare cases.

to create a -V0.7.32-4 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.32-4

Ingo

Ingo Molnar

2004-12-07 14:11 UTC #799 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

i have released the -V0.7.32-6 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this too is a fixes-only release.

Changes since -V0.7.32-4:

- fixed a lock_kernel()-re-enables-interrupts bug reported by Daniel Walker. The fix is to allow down() from irqs-off sections (and save/restore irq flags) as long as there's no real contention on the semaphore.

- fixed a /proc/latency_trace formatting bug reported by Mark H. Johnson.

to create a -V0.7.32-6 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm3/2.6.10-rc2-mm3.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm3-V0.7.32-6

Ingo

K.R. Foley

2004-12-08 04:31 UTC #800 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 3 lines · Ingo Molnar

i have released the -V0.7.32-6 Real-Time Preemption patch, which can be downloaded from the usual place:

Ingo,

Could you explain what the attached trace means. It looks to me like the trace starts in try_to_wake_up when we are trying to wake amlat, but then before we finish we get a hit on IRQ 8 and run the IRQ handler??? Or do I somehow have it backwards? :)

preemption latency trace v1.1.3 on 2.6.10-rc2-mm3-V0.7.32-9
--------------------------------------------------------------------
latency: 39 us, #42/42 | (M:rt VP:0, KP:1, SP:1 HP:1 #P:2)

    -----------------
    | task: IRQ 8-677 (uid:0 nice:-5 policy:1 rt_prio:99)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> hardirq         
               || / _---=> softirq         
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
   amlat-4973  0-h.3    0�s : __trace_start_sched_wakeup (try_to_wake_up)
   amlat-4973  0-h.3    1�s : _raw_spin_unlock (try_to_wake_up)
   amlat-4973  0-h.2    1�s : preempt_schedule (try_to_wake_up)
   amlat-4973  0        2�s : wake_up_process <IRQ 8-677> (0 1): 
   amlat-4973  0-h.2    2�s : try_to_wake_up (wake_up_process)
   amlat-4973  0-h.2    2�s : _raw_spin_unlock (try_to_wake_up)
   amlat-4973  0-h.1    3�s : preempt_schedule (try_to_wake_up)
   amlat-4973  0-h.1    3�s : wake_up_process (redirect_hardirq)
   amlat-4973  0-h.1    4�s : _raw_spin_unlock (__do_IRQ)
   amlat-4973  0-h..    4�s : preempt_schedule (__do_IRQ)
   amlat-4973  0-h..    4�s : irq_exit (do_IRQ)
   amlat-4973  0-h.1    5�s : do_softirq (irq_exit)
   amlat-4973  0-h.1    5�s : __do_softirq (do_softirq)
   amlat-4973  0-h..    6�s : preempt_schedule_irq (need_resched)
   amlat-4973  0-h..    6�s : __schedule (preempt_schedule_irq)
   amlat-4973  0-h.1    7�s : sched_clock (__schedule)
   amlat-4973  0-h.1    8�s : _raw_spin_lock_irq (__schedule)
   amlat-4973  0-h.1    8�s : _raw_spin_lock_irqsave (__schedule)
   amlat-4973  0-h.2   10�s : pull_rt_tasks (__schedule)
   amlat-4973  0-h.2   10�s : find_next_bit (pull_rt_tasks)
   amlat-4973  0-h.2   11�s+: find_next_bit (pull_rt_tasks)
   amlat-4973  0-..2   13�s : trace_array (__schedule)
   amlat-4973  0       14�s : __schedule <IRQ 8-677> (0 1): 
   amlat-4973  0       14�s+: __schedule <amlat-4973> (1 2): 
   amlat-4973  0       18�s+: __schedule <<unknown-792> (39 3a): 
   amlat-4973  0       21�s : __schedule <<unknown-4> (69 6e): 
   amlat-4973  0       21�s : __schedule <<unknown-4854> (73 78): 
   amlat-4973  0-..2   22�s+: trace_array (__schedule)
   IRQ 8-677   0-..2   31�s : __switch_to (__schedule)
   IRQ 8-677   0       32�s : schedule <amlat-4973> (1 0): 
   IRQ 8-677   0-..2   32�s : finish_task_switch (__schedule)
   IRQ 8-677   0-..2   33�s : smp_send_reschedule_allbutself (finish_task_switch)
   IRQ 8-677   0-..2   33�s : __bitmap_weight (smp_send_reschedule_allbutself)
   IRQ 8-677   0-..2   34�s : __send_IPI_shortcut (smp_send_reschedule_allbutself)
   IRQ 8-677   0-..2   35�s : _raw_spin_unlock (finish_task_switch)
   IRQ 8-677   0-..1   35�s : trace_stop_sched_switched (finish_task_switch)
   IRQ 8-677   0       36�s : finish_task_switch <IRQ 8-677> (0 0): 
   IRQ 8-677   0-..1   36�s+: _raw_spin_lock_irqsave (trace_stop_sched_switched)
   IRQ 8-677   0-..1   43�s : trace_stop_sched_switched (finish_task_switch)

vim:ft=help

Ingo Molnar

2004-12-08 08:37 UTC #801 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 9 lines · K.R. Foley

Could you explain what the attached trace means. It looks to me like the trace starts in try_to_wake_up when we are trying to wake amlat, but then before we finish we get a hit on IRQ 8 and run the IRQ handler??? Or do I somehow have it backwards? :)
   amlat-4973  0-h.3    0?s : __trace_start_sched_wakeup (try_to_wake_up)
   amlat-4973  0-h.3    1?s : _raw_spin_unlock (try_to_wake_up)
   amlat-4973  0-h.2    1?s : preempt_schedule (try_to_wake_up)
   amlat-4973  0        2?s : wake_up_process <IRQ 8-677> (0 1): 

this portion shows that amlat-4973 woke up IRQ_8-677. Subsequently the scheduler picked it from a list of 5 tasks:

Quoted 8 lines

   amlat-4973  0-..2   13?s : trace_array (__schedule)
   amlat-4973  0       14?s : __schedule <IRQ 8-677> (0 1): 
   amlat-4973  0       14?s+: __schedule <amlat-4973> (1 2): 
   amlat-4973  0       18?s+: __schedule <<unknown-792> (39 3a): 
   amlat-4973  0       21?s : __schedule <<unknown-4> (69 6e): 
   amlat-4973  0       21?s : __schedule <<unknown-4854> (73 78): 
   amlat-4973  0-..2   22?s+: trace_array (__schedule)
   IRQ 8-677   0-..2   31?s : __switch_to (__schedule)

IRQ_8's RT priority was 1, amlat's priority was 2, so IRQ-8 got selected. (there were also other, SCHED_NORMAL tasks with pid 792, 4 and 4854 in the queue but they did not get selected) [ Note that in reality the O(1) scheduler only considered IRQ_8 when picking the next task, it's the tracer that listed all runnable tasks, to make it easier to validate scheduler logic. This 'list all runnable tasks at schedule() time' tracing is only done if both tracing and rw-deadlock detection is enabled.]

in this trace you can see the new RT global balancing in the works as well:

Quoted 5 lines

   IRQ 8-677   0       32?s : schedule <amlat-4973> (1 0): 
   IRQ 8-677   0-..2   32?s : finish_task_switch (__schedule)
   IRQ 8-677   0-..2   33?s : smp_send_reschedule_allbutself (finish_task_switch)
   IRQ 8-677   0-..2   33?s : __bitmap_weight (smp_send_reschedule_allbutself)
   IRQ 8-677   0-..2   34?s : __send_IPI_shortcut (smp_send_reschedule_allbutself)

here the scheduler noticed that a higher-prio RT task (IRQ_8) preempted a lower-prio but still RT task (amlat), and sent an IPI (inter-processor interrupt) to another CPU in the system so that amlat can run on the other CPU.

Ingo

K.R. Foley

2004-12-08 16:08 UTC #802 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 8 lines · Ingo Molnar

Quoted 4 lines · K.R. Foley
Could you explain what the attached trace means. It looks to me like the trace starts in try_to_wake_up when we are trying to wake amlat, but then before we finish we get a hit on IRQ 8 and run the IRQ handler??? Or do I somehow have it backwards? :)

Thank you. I really did have it backwards. The thing that confused me was that trace_start... gets called with the task that we are trying to wake up. I didn't follow the trace code far enough to realize that it later starts getting task info from current instead of p. :) This all makes more sense now.

I am still confused about one thing, unrelated to this. If RT tasks never expire and thus are never moved to the expired array??? Does that imply that we never switch the active and expired arrays? If so how do tasks that do expire get moved back into the active array?

Quoted 47 lines

Quoted 5 lines
  amlat-4973  0-h.3    0?s : __trace_start_sched_wakeup (try_to_wake_up)
  amlat-4973  0-h.3    1?s : _raw_spin_unlock (try_to_wake_up)
  amlat-4973  0-h.2    1?s : preempt_schedule (try_to_wake_up)
  amlat-4973  0        2?s : wake_up_process <IRQ 8-677> (0 1): 
this portion shows that amlat-4973 woke up IRQ_8-677. Subsequently the scheduler picked it from a list of 5 tasks:
Quoted 8 lines
  amlat-4973  0-..2   13?s : trace_array (__schedule)
  amlat-4973  0       14?s : __schedule <IRQ 8-677> (0 1): 
  amlat-4973  0       14?s+: __schedule <amlat-4973> (1 2): 
  amlat-4973  0       18?s+: __schedule <<unknown-792> (39 3a): 
  amlat-4973  0       21?s : __schedule <<unknown-4> (69 6e): 
  amlat-4973  0       21?s : __schedule <<unknown-4854> (73 78): 
  amlat-4973  0-..2   22?s+: trace_array (__schedule)
  IRQ 8-677   0-..2   31?s : __switch_to (__schedule)
IRQ_8's RT priority was 1, amlat's priority was 2, so IRQ-8 got selected. (there were also other, SCHED_NORMAL tasks with pid 792, 4 and 4854 in the queue but they did not get selected) [ Note that in reality the O(1) scheduler only considered IRQ_8 when picking the next task, it's the tracer that listed all runnable tasks, to make it easier to validate scheduler logic. This 'list all runnable tasks at schedule() time' tracing is only done if both tracing and rw-deadlock detection is enabled.]
in this trace you can see the new RT global balancing in the works as well:
Quoted 5 lines
  IRQ 8-677   0       32?s : schedule <amlat-4973> (1 0): 
  IRQ 8-677   0-..2   32?s : finish_task_switch (__schedule)
  IRQ 8-677   0-..2   33?s : smp_send_reschedule_allbutself (finish_task_switch)
  IRQ 8-677   0-..2   33?s : __bitmap_weight (smp_send_reschedule_allbutself)
  IRQ 8-677   0-..2   34?s : __send_IPI_shortcut (smp_send_reschedule_allbutself)
here the scheduler noticed that a higher-prio RT task (IRQ_8) preempted a lower-prio but still RT task (amlat), and sent an IPI (inter-processor interrupt) to another CPU in the system so that amlat can run on the other CPU.
Ingo

Lee Revell

2004-12-08 16:18 UTC #803 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 4 lines · K.R. Foley

I am still confused about one thing, unrelated to this. If RT tasks never expire and thus are never moved to the expired array??? Does that imply that we never switch the active and expired arrays? If so how do tasks that do expire get moved back into the active array?

I think that RT tasks use a completely different scheduling mechanism that bypasses the active/expired array.

Lee

K.R. Foley

2004-12-08 16:52 UTC #804 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 11 lines · Lee Revell

Quoted 1 line · K.R. Foley
I am still confused about one thing, unrelated to this. If RT tasks never expire and thus are never moved to the expired array??? Does that imply that we never switch the active and expired arrays? If so how do tasks that do expire get moved back into the active array?
I think that RT tasks use a completely different scheduling mechanism that bypasses the active/expired array.
Lee

Please don't misunderstand. I am not arguing with you because obviously I am not really intimate with this code, but if the above statement is true then I am even more confused than I thought. I don't see any such distinctions in the scheduler code. In fact it looks to me like the whole scheduler is built on the premise of allowing RT tasks to be just like other tasks with a few exceptions, one of which is that RT tasks never hit the expired task array.

Lee Revell

2004-12-08 16:58 UTC #805 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 15 lines · K.R. Foley

Quoted 8 lines · Lee Revell
Quoted 4 lines · K.R. Foley
I am still confused about one thing, unrelated to this. If RT tasks never expire and thus are never moved to the expired array??? Does that imply that we never switch the active and expired arrays? If so how do tasks that do expire get moved back into the active array?
Quoted 6 lines
I think that RT tasks use a completely different scheduling mechanism that bypasses the active/expired array.
Lee

Quoted 7 lines

Please don't misunderstand. I am not arguing with you because obviously I am not really intimate with this code, but if the above statement is true then I am even more confused than I thought. I don't see any such distinctions in the scheduler code. In fact it looks to me like the whole scheduler is built on the premise of allowing RT tasks to be just like other tasks with a few exceptions, one of which is that RT tasks never hit the expired task array.

No, you are probably right, I am the one who is confused.

Lee

Ingo Molnar

2004-12-09 09:03 UTC #806 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

In reply to K.R. Foley

Quoted 4 lines · K.R. Foley

[...] it looks to me like the whole scheduler is built on the premise of allowing RT tasks to be just like other tasks with a few exceptions, one of which is that RT tasks never hit the expired task array.

this is more or less correct, and we are trying to keep RT scheduling 'integrated' into the SCHED_NORMAL scheduler as long as it's practical.

but Lee is correct too in that the scheduling behavior of RT tasks and SCHED_NORMAL tasks is completely different. But on the implementational level the distinction is less stark and boils down to a few branches here and there, while 90% of the scheduling code is shared.

to answer your question: it is true that if there is an RT task active then we never switch the arrays. That's how RT tasks work: they run until they want. That's why it needs privileges to use RT scheduling, and that's why a buggy RT task can lock up the system. The 'array switching' mechanism is part of the 10% of code that is only used by non-RT tasks. SCHED_FIFO tasks dont have any timeslices, they run until they deschedule voluntarily. SCHED_RR tasks have a notion of timeslices but they only yield to RR tasks on their own priority level, which is implemented without an array-switch. [the array-switch implements fair scheduling between different-priority SCHED_NORMAL tasks - this is a fundamentally harder problem which necessiates more work from the scheduler.]

(btw., the 'global RT balancing' SMP code i recently added, and the priority inheritance scheduler code increases the 10% of non-shared scheduling code to perhaps 15% or so. Not always is it possible to share code.)

Ingo

K.R. Foley

2004-12-09 06:11 UTC #807 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

In reply to K.R. Foley

K.R. Foley wrote:
<snip>

Quoted 3 lines

I am still confused about one thing, unrelated to this. If RT tasks never expire and thus are never moved to the expired array??? Does that imply that we never switch the active and expired arrays? If so how do tasks that do expire get moved back into the active array?

OK dumb question. I am going out to get my own personal brown paper bag, since I seem to be wearing it so often. I forgot tasks get removed from the runqueue when they are sleeping, etc. so the active array should empty most of the time. However, with more RT tasks and interactive tasks being thrown back into the active queue I could see this POSSIBLY occasionally starving a few processes???

kr
<snip>

Ingo Molnar

2004-12-09 12:12 UTC #808 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 6 lines · K.R. Foley

OK dumb question. I am going out to get my own personal brown paper bag, since I seem to be wearing it so often. I forgot tasks get removed from the runqueue when they are sleeping, etc. so the active array should empty most of the time. However, with more RT tasks and interactive tasks being thrown back into the active queue I could see this POSSIBLY occasionally starving a few processes???

interactive tasks do get thrown back, but they wont ever preempt RT tasks. RT tasks themselves can starve any lower-prio process indefinitely. Interactive tasks can starve other tasks up to a certain limit, which is defined via STARVATION_LIMIT, at which point we empty the active array and perform an array switch. (also see EXPIRED_STARVING())

Ingo

K.R. Foley

2004-12-09 14:50 UTC #809 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 11 lines · Ingo Molnar

Quoted 6 lines · K.R. Foley
OK dumb question. I am going out to get my own personal brown paper bag, since I seem to be wearing it so often. I forgot tasks get removed from the runqueue when they are sleeping, etc. so the active array should empty most of the time. However, with more RT tasks and interactive tasks being thrown back into the active queue I could see this POSSIBLY occasionally starving a few processes???

Quoted 9 lines

interactive tasks do get thrown back, but they wont ever preempt RT tasks. RT tasks themselves can starve any lower-prio process indefinitely. Interactive tasks can starve other tasks up to a certain limit, which is defined via STARVATION_LIMIT, at which point we empty the active array and perform an array switch. (also see EXPIRED_STARVING())
Ingo

Understood. BTW, I wouldn't consider some possible starvation of lower priority, non-realtime tasks to be incorrect behavior for a realtime system. The comments in the above email as well as previous emails were not intended as complaints or questions of correctness. They were more just thoughts generated while thinking about some of the reports of non-realtime tasks being starved.

Steven Rostedt

reviewer

2004-12-08 17:14 UTC #810 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

In reply to Ingo Molnar

Hi Ingo,

I found a race condition in slab.c, but I'm still trying to figure out exactly how it's playing out. This has to do with dynamic loading and unloading of caches. I have a small test case that simulates the problem at http://home.stny.rr.com/rostedt/tests/sillycaches.tgz

This was done on:

# uname -r
2.6.10-rc2-mm3-V0.7.32-9

I have a module that creates a cache to allocate objects from. When you unload the module, it deallocates the objects and then destroys the cache. But with your patched kernel I get the following output, and the system then goes into an unstable state. That is the system will crash at a latter time. Usually when dealing with caches.

Here's the output:

slab error in kmem_cache_destroy(): cache `silly_stuff': Can't free all objects

 [<c0103953>] dump_stack+0x23/0x30 (20)
 [<c014929f>] kmem_cache_destroy+0xff/0x1a0 (28)
 [<d081e10d>] mkcache_cleanup+0x1d/0x21 [sillymod] (12)
 [<c013a711>] sys_delete_module+0x161/0x1a0 (100)
 [<c0102a00>] syscall_call+0x7/0xb (-8124)
---------------------------

| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c01383ed>] .... print_traces+0x1d/0x60
.....[<c0103953>] .. ( <= dump_stack+0x23/0x30)

I've done some extra testing and found that if I wait between the frees and the destroying of the cache, everything works fine. This problem happens because it seems that the objects in the slab are being freed in a batch style and they don't get freed on the destroy. I put prints in to see more information and found that in kmem_cache_destroy, it calls __cache_shrink, which calls drain_cpu_caches (obvious from code), but what my prints show, is that when it gets down to drain_array_locked (it gets in the function) that ac->avail is zero. I need to read more into the details of how the slab works, but you can take a look too.

By the way, 2.6.10-rc2-mm3 does not have a problem with this.

Thanks,

-- Steve

Rui Nuno Capela

2004-12-08 18:19 UTC #811 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 44 lines · Steven Rostedt

I found a race condition in slab.c, but I'm still trying to figure out exactly how it's playing out. This has to do with dynamic loading and unloading of caches. I have a small test case that simulates the problem at http://home.stny.rr.com/rostedt/tests/sillycaches.tgz
This was done on:
# uname -r
2.6.10-rc2-mm3-V0.7.32-9
I have a module that creates a cache to allocate objects from. When you unload the module, it deallocates the objects and then destroys the cache. But with your patched kernel I get the following output, and the system then goes into an unstable state. That is the system will crash at a latter time. Usually when dealing with caches.
Here's the output:
slab error in kmem_cache_destroy(): cache `silly_stuff': Can't free all objects
 [<c0103953>] dump_stack+0x23/0x30 (20)
 [<c014929f>] kmem_cache_destroy+0xff/0x1a0 (28)
 [<d081e10d>] mkcache_cleanup+0x1d/0x21 [sillymod] (12)
 [<c013a711>] sys_delete_module+0x161/0x1a0 (100)
 [<c0102a00>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c01383ed>] .... print_traces+0x1d/0x60
.....[<c0103953>] .. ( <= dump_stack+0x23/0x30)
I've done some extra testing and found that if I wait between the frees and the destroying of the cache, everything works fine. This problem happens because it seems that the objects in the slab are being freed in a batch style and they don't get freed on the destroy. I put prints in to see more information and found that in kmem_cache_destroy, it calls __cache_shrink, which calls drain_cpu_caches (obvious from code), but what my prints show, is that when it gets down to drain_array_locked (it gets in the function) that ac->avail is zero. I need to read more into the details of how the slab works, but you can take a look too.

Quoted 2 lines

By the way, 2.6.10-rc2-mm3 does not have a problem with this.

AFAICS this seems to be exactly the bug I've reported recently, about when an usb-storage flashram stick is first time unplugged.

Good show Steven :) Hope it helps.

Cheers.

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Steven Rostedt

reviewer

2004-12-08 19:04 UTC #812 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 7 lines · Rui Nuno Capela

Quoted 6 lines · Steven Rostedt
I found a race condition in slab.c, but I'm still trying to figure out exactly how it's playing out. This has to do with dynamic loading and unloading of caches. I have a small test case that simulates the problem at http://home.stny.rr.com/rostedt/tests/sillycaches.tgz

Quoted 5 lines

Quoted 5 lines
This was done on:
# uname -r
2.6.10-rc2-mm3-V0.7.32-9

<snip>

Found the culprit!!! I did a diff of 2.6.10-rc2-mm3 to 2.6.10-rc2-mm3-V0.7.32-9 and found this in slab.c:
----------------------------
+#ifndef CONFIG_PREEMPT_RT
+/*
+ * Executes in an IRQ context:
+ */
static void do_drain(void *arg)

 {         kmem_cache_t *cachep = (kmem_cache_t*)arg;
        struct array_cache *ac;
+       int cpu = smp_processor_id();
         check_irq_off();
-       ac = ac_data(cachep);
+       ac = ac_data(cachep, cpu);
        spin_lock(&cachep->spinlock);
        free_block(cachep, &ac_entry(ac)[0], ac->avail);
        spin_unlock(&cachep->spinlock);
        ac->avail = 0;

}
+#endif

static void drain_cpu_caches(kmem_cache_t *cachep) {
+#ifndef CONFIG_PREEMPT_RT

        smp_call_function_all_cpus(do_drain, cachep);
+#endif
        check_irq_on();

--------------------------------
(I have CONFIG_PREEMPT_RT defined :-)

I then put in

static void drain_cpu_caches(kmem_cache_t *cachep) {

 #ifndef CONFIG_PREEMPT_RT
        smp_call_function_all_cpus(do_drain, cachep);
 #endif
        check_irq_on();
        spin_lock_irq(&cachep->spinlock);
+       {
+               struct array_cache *ac;
+               ac = ac_data(cachep, smp_processor_id());
+               free_block(cachep, &ac_entry(ac)[0], ac->avail);
+               ac->avail = 0;

+ }

To see what would happen, and this indeed fixed the problem. At least didn't cause the problem to appear after a few tests.

Obviously, this is not the right answer, and Ingo, since I don't know exactly what you are accomplishing with the added cpu changes, I think you are probably better at writing a patch than I.

Which brings up another point.

In places like kmem_cache_create you have cpu = _smp_processor_id(); and way down near the bottom, you use it. Being a preemptable kernel, can't that process jump cpus in the time being? So isn't that in itself a race condition?

Thanks,

-- Steve

Rui,

Try adding the following in slab.c

slab.c.new 2004-12-08 13:58:40.000000000 -0500 renamed from slab.c 2004-12-08 09:27:10.000000000 -0500 +6 -0

@@ -1533,6 +1533,12 @@
 #ifndef CONFIG_PREEMPT_RT
        smp_call_function_all_cpus(do_drain, cachep);
 #endif
+       {
+               struct array_cache *ac;
+               ac = ac_data(cachep, smp_processor_id());
+               free_block(cachep, &ac_entry(ac)[0], ac->avail);
+               ac->avail = 0;
+       }
        check_irq_on();
        spin_lock_irq(&cachep->spinlock);
        if (cachep->lists.shared)

and see if this fixes your usb problems. I would say that this is not a proper fix and especially for a SMP system. But if it fixes your problem then you know this is the solution.

Rui Nuno Capela

2004-12-08 21:41 UTC #813 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 36 lines · Steven Rostedt

Quoted 8 lines
Quoted 7 lines · Steven Rostedt
I found a race condition in slab.c, but I'm still trying to figure out exactly how it's playing out. This has to do with dynamic loading and unloading of caches. I have a small test case that simulates the problem at
http://home.stny.rr.com/rostedt/tests/sillycaches.tgz
Quoted 5 lines
Quoted 5 lines
This was done on:
# uname -r
2.6.10-rc2-mm3-V0.7.32-9
Rui,
Try adding the following in slab.c
slab.c.new 2004-12-08 13:58:40.000000000 -0500 renamed from slab.c 2004-12-08 09:27:10.000000000 -0500 +6 -0
@@ -1533,6 +1533,12 @@
 #ifndef CONFIG_PREEMPT_RT
        smp_call_function_all_cpus(do_drain, cachep);
 #endif
+       {
+               struct array_cache *ac;
+               ac = ac_data(cachep, smp_processor_id());
+               free_block(cachep, &ac_entry(ac)[0], ac->avail);
+               ac->avail = 0;
+       }
        check_irq_on();
        spin_lock_irq(&cachep->spinlock);
        if (cachep->lists.shared)

Quoted 4 lines

and see if this fixes your usb problems. I would say that this is not a proper fix and especially for a SMP system. But if it fixes your problem then you know this is the solution.

Almost there, perhaps not...

It doesn't solve the problem completely, if not at all. What was kind of a deterministic failure now seems probabilistic: the fault still occur on unplugging the usb-storage stick, but not everytime as before.

Did try several times, reboot included, and now it fails after unplugging a second or a third time. Never needed to replug/unplug more than 3 times for it to show up, however.

Here is one sample, taken on the patched RT-V0.7.32-9:

usb 4-5: USB disconnect, address 7
slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't free
all objects

  [<c010361f>] dump_stack+0x23/0x25 (20)
  [<c014669f>] kmem_cache_destroy+0x103/0x1aa (28)
  [<c026e61a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
  [<c026f451>] scsi_host_dev_release+0x37/0xe1 (104)
  [<c023c569>] device_release+0x7a/0x7c (32)
  [<c01efc50>] kobject_cleanup+0x87/0x89 (28)
  [<c01f06ab>] kref_put+0x52/0xef (40)
  [<c01efc8c>] kobject_put+0x25/0x27 (16)
  [<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage] (16)
  [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
  [<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
  [<c023d7d5>] device_release_driver+0x75/0x77 (28)
  [<c023d9d8>] bus_remove_device+0x53/0x82 (20)
  [<c023c9a1>] device_del+0x4b/0x9b (20)
  [<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
  [<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
  [<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
  [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
  [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
  [<c01009b1>] kernel_thread_helper+0x5/0xb (150118420)

Bye now.

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Steven Rostedt

reviewer

2004-12-08 22:12 UTC #814 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 7 lines · Rui Nuno Capela

Almost there, perhaps not...
It doesn't solve the problem completely, if not at all. What was kind of a deterministic failure now seems probabilistic: the fault still occur on unplugging the usb-storage stick, but not everytime as before.

OK, so I would say that this is part of a fix, but there are others. There are lots of changes done to the slab.c file by Ingo. The change I made (and that is just a quick patch, it needs real work), was only in a place that was obvious that there could be problems.

Are you running an SMP machine? If so, than the patch I gave you is definitely not enough.

Ingo really scares me with all the removing of local_irq_disables in the rt mode. I'm not sure exactly what is going on there, and why they can, or should be removed. Ingo?

Quoted 21 lines

Did try several times, reboot included, and now it fails after unplugging a second or a third time. Never needed to replug/unplug more than 3 times for it to show up, however.
Here is one sample, taken on the patched RT-V0.7.32-9:
usb 4-5: USB disconnect, address 7
slab error in kmem_cache_destroy(): cache `scsi_cmd_cache': Can't free
all objects
  [<c010361f>] dump_stack+0x23/0x25 (20)
  [<c014669f>] kmem_cache_destroy+0x103/0x1aa (28)
  [<c026e61a>] scsi_destroy_command_freelist+0x97/0xa8 (28)
  [<c026f451>] scsi_host_dev_release+0x37/0xe1 (104)
  [<c023c569>] device_release+0x7a/0x7c (32)
  [<c01efc50>] kobject_cleanup+0x87/0x89 (28)
  [<c01f06ab>] kref_put+0x52/0xef (40)
  [<c01efc8c>] kobject_put+0x25/0x27 (16)
  [<f8cf1843>] usb_stor_release_resources+0x66/0xca [usb_storage] (16)
  [<f8cf1d93>] storage_disconnect+0x8e/0x9b [usb_storage] (16)
  [<f89ca117>] usb_unbind_interface+0x84/0x86 [usbcore] (28)
  [<c023d7d5>] device_release_driver+0x75/0x77 (28)

Quoted 9 lines

  [<c023d9d8>] bus_remove_device+0x53/0x82 (20)
  [<c023c9a1>] device_del+0x4b/0x9b (20)
  [<f89d142a>] usb_disable_device+0xf5/0x10a [usbcore] (28)
  [<f89cc61c>] usb_disconnect+0xc8/0x164 [usbcore] (40)
  [<f89cd77e>] hub_port_connect_change+0x2ef/0x426 [usbcore] (56)
  [<f89cda7b>] hub_events+0x1c6/0x39d [usbcore] (56)
  [<f89cdc89>] hub_thread+0x37/0x109 [usbcore] (96)
  [<c01009b1>] kernel_thread_helper+0x5/0xb (150118420)

Unfortunately this really doesn't help. The problem occurs earlier than this. What happened was that there are still slabs out there that need to be freed that were postponed to a later time and not done with the kmem_cache_free call. So this dump only tells you that. The backtrace unfortunately doesn't give us any more clues. It's all in the slab.c code.
(Of course if the driver really did not free the objects then it's not slab.c's fault, and why Ingo may not have thought it was related to RT)

Tschuess,

-- Steve

Ingo Molnar

2004-12-09 09:32 UTC #815 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

Quoted 9 lines · Steven Rostedt

Quoted 7 lines · Rui Nuno Capela
Almost there, perhaps not...
It doesn't solve the problem completely, if not at all. What was kind of a deterministic failure now seems probabilistic: the fault still occur on unplugging the usb-storage stick, but not everytime as before.

Quoted 7 lines

OK, so I would say that this is part of a fix, but there are others. There are lots of changes done to the slab.c file by Ingo. The change I made (and that is just a quick patch, it needs real work), was only in a place that was obvious that there could be problems.
Are you running an SMP machine? If so, than the patch I gave you is definitely not enough.

one of Rui's boxes is an SMP system - which would explain why the bug goes from an 'always crash' to 'spurious crash'. (if Rui's laptop triggers this problem too then there must be something else going on as well.)

Quoted 3 lines

Ingo really scares me with all the removing of local_irq_disables in the rt mode. I'm not sure exactly what is going on there, and why they can, or should be removed. Ingo?

it is done so that the SLAB code can be fully preempted too. The SLAB code is of central importance to the -RT project, if it's not fully preemptible then that has a ripple effect on other subsystems (timer, signal code, file handling, etc.).

So while making it fully preemptible was quite challenging (==dangerous, scary), i couldnt just keep the SLAB using raw spinlocks, due to the locking dependencies. (nor did i have any true inner desire to keep it non-preemptible - the point of PREEMPT_RT is to have everything preemptible. I want to see how much preemption the Linux kernel can take =B-) It has held up surprisingly well i have to say.)

to make the SLAB code fully preemptible, there were two main aspects that i had to fix:

1) irq context execution
2) process preemption

in the -RT kernel all IRQ contexts execute in a separate process context, so the SLAB code is never called from a true IRQ context - hence problem #1 is solved. As far as #1 is concerned, the local_irq_disable()s are not needed anymore.

the other aspect is process<->process preemption - which can still occur in the -RT kernel (and is the whole point of the PREEMPT_RT feature). This means that the per-CPU assumptions within slab.c break.

To solve this i've turned the unlocked per-CPU SLAB code to be controlled by the cachep->spinlock. (on RT only - on non-RT kernels the SLAB code should be largely unmodified - this is why all that _rt and _nort API trickery is done.) Since the SLAB code is thus locked by cachep->spinlock on PREEMPT_RT, other tasks cannot interfere with the internal data structures.

Finally, there was still the problem of the use of smp_processor_id() - the non-RT SLAB code (rightfully) assumes that smp_processor_id() is constant, but this is not true for the RT code - which can be preempted anytime (still holding the spinlock of course) and can be migrated to another CPU.

To solve this problem i am saving smp_processor_id() once, before we use any per-CPU data structure for the first time, and this constant CPU ID value is cached and used throughout the whole SLAB processing pass.

[ Since in the RT case we lock the cachep exclusively, it's not a problem if the 'old' CPU's ID is used as an index - as long as the index is consistent. Most of the time the current CPU's ID will be used so we preserve most of the performance advantages (==cache-hotness) of per-CPU SLABs on SMP systems too. (except for the locking, which is serialized on RT.) ]

SLAB draining was an oversight - it's mainly called when there is VM pressure (which is not a stricly necessary feature, so i disabled it), but i forgot about the module-unload case where it's a correctness feature. Your patch is a good starting point, i'll try to fix it on SMP too.

Ingo

Ingo Molnar

2004-12-09 13:13 UTC #816 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 5 lines · Ingo Molnar

SLAB draining was an oversight - it's mainly called when there is VM pressure (which is not a stricly necessary feature, so i disabled it), but i forgot about the module-unload case where it's a correctness feature. Your patch is a good starting point, i'll try to fix it on SMP too.

here's the full patch against a recent tree, or download the -32-12 patch from the usual place:

http://redhat.com/~mingo/realtime-preempt/

Rui, Gene, does this fix the module unload crash you are seeing?

Ingo

mm/slab.c renamed from mm/slab.c.orig +18 -9

@@ -1509,22 +1509,26 @@ static void smp_call_function_all_cpus(v
 static void drain_array_locked(kmem_cache_t* cachep,
 				struct array_cache *ac, int force);
 
-#ifndef CONFIG_PREEMPT_RT
-/*
- * Executes in an IRQ context:
- */
-static void do_drain(void *arg)
+static void do_drain_cpu(kmem_cache_t *cachep, int cpu)
 {
-	kmem_cache_t *cachep = (kmem_cache_t*)arg;
 	struct array_cache *ac;
-	int cpu = smp_processor_id();
 
 	check_irq_off();
-	ac = ac_data(cachep, cpu);
+
 	spin_lock(&cachep->spinlock);
+	ac = ac_data(cachep, cpu);
 	free_block(cachep, &ac_entry(ac)[0], ac->avail);
-	spin_unlock(&cachep->spinlock);
 	ac->avail = 0;
+	spin_unlock(&cachep->spinlock);
+}
+
+#ifndef CONFIG_PREEMPT_RT
+/*
+ * Executes in an IRQ context:
+ */
+static void do_drain(void *arg)
+{
+	do_drain_cpu((kmem_cache_t*)arg, smp_processor_id());
 }
 #endif

@@ -1532,6 +1536,11 @@ static void drain_cpu_caches(kmem_cache_
 {
 #ifndef CONFIG_PREEMPT_RT
 	smp_call_function_all_cpus(do_drain, cachep);
+#else
+	int cpu;
+
+	for_each_online_cpu(cpu)
+		do_drain_cpu(cachep, cpu);
 #endif
 	check_irq_on();
 	spin_lock_irq(&cachep->spinlock);

Gene Heskett

2004-12-09 14:24 UTC #817 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 7 lines · Ingo Molnar

Quoted 5 lines · Ingo Molnar
SLAB draining was an oversight - it's mainly called when there is VM pressure (which is not a stricly necessary feature, so i disabled it), but i forgot about the module-unload case where it's a correctness feature. Your patch is a good starting point, i'll try to fix it on SMP too.

Quoted 6 lines

here's the full patch against a recent tree, or download the -32-12 patch from the usual place:
http://redhat.com/~mingo/realtime-preempt/
Rui, Gene, does this fix the module unload crash you are seeing?

I'm still on -9 here Ingo, and I just rmmod'ed eeprom with no ill effects. I'd built -10 last night but as kde was trying to build, didn't reboot. I just got -12, so I'll do it and reboot to it next.

Or am I the wrong Gene?

Quoted 58 lines

Ingo

mm/slab.c renamed from mm/slab.c.orig +18 ~~-10~~

@@ -1509,22 +1509,26 @@ static void smp_call_function_all_cpus(v
 static void drain_array_locked(kmem_cache_t* cachep,
    struct array_cache *ac, int force);
 
-#ifndef CONFIG_PREEMPT_RT
-/*
- * Executes in an IRQ context:
- */
-static void do_drain(void *arg)
+static void do_drain_cpu(kmem_cache_t *cachep, int cpu)
 {
-	kmem_cache_t *cachep = (kmem_cache_t*)arg;
 	struct array_cache *ac;
-	int cpu = smp_processor_id();
 
 	check_irq_off();
-	ac = ac_data(cachep, cpu);
+
 	spin_lock(&cachep->spinlock);
+	ac = ac_data(cachep, cpu);
 	free_block(cachep, &ac_entry(ac)[0], ac->avail);
-	spin_unlock(&cachep->spinlock);
 	ac->avail = 0;
+	spin_unlock(&cachep->spinlock);
+}
+
+#ifndef CONFIG_PREEMPT_RT
+/*
+ * Executes in an IRQ context:
+ */
+static void do_drain(void *arg)
+{
+	do_drain_cpu((kmem_cache_t*)arg, smp_processor_id());
 }
 #endif

@@ -1532,6 +1536,11 @@ static void drain_cpu_caches(kmem_cache_
 {
 #ifndef CONFIG_PREEMPT_RT
 	smp_call_function_all_cpus(do_drain, cachep);
+#else
+	int cpu;
+
+	for_each_online_cpu(cpu)
+		do_drain_cpu(cachep, cpu);
 #endif
 	check_irq_on();
 	spin_lock_irq(&cachep->spinlock);
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Steven Rostedt

reviewer

2004-12-09 14:33 UTC #818 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

In reply to Ingo Molnar

Quoted 8 lines · Ingo Molnar

Quoted 5 lines · Ingo Molnar
SLAB draining was an oversight - it's mainly called when there is VM pressure (which is not a stricly necessary feature, so i disabled it), but i forgot about the module-unload case where it's a correctness feature. Your patch is a good starting point, i'll try to fix it on SMP too.

Quoted 4 lines

here's the full patch against a recent tree, or download the -32-12 patch from the usual place:
http://redhat.com/~mingo/realtime-preempt/

Ingo,

I just tried out your changes with both my sillycaches test as well as my real modules that had the original problems. They both work fine.

I'll ever reboot my main machine now (SMP) and run your kernel there too, and see how it works.

Thanks,

-- Steve

Steven Rostedt

reviewer

2004-12-09 19:20 UTC #819 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 8 lines · Steven Rostedt

Quoted 6 lines · Ingo Molnar
* Ingo Molnar <[email protected]> wrote:
here's the full patch against a recent tree, or download the -32-12 patch from the usual place:
http://redhat.com/~mingo/realtime-preempt/

Quoted 7 lines

Ingo,
I just tried out your changes with both my sillycaches test as well as my real modules that had the original problems. They both work fine.
I'll ever reboot my main machine now (SMP) and run your kernel there too, and see how it works.

Hi Ingo,

I just tried it on my main machine (SMP Dual Athlon MP, with a Gig of RAM), and I just got the following dump on boot-up. It never made it to a prompt.

Freeing unused kernel memory: 216k freed

BUG: sleeping function called from invalid context IRQ 14(814) at
arch/i386/mm/highmem.c:5
in_atomic():0 [00000000], irqs_disabled():1
 [<c011c064>] __might_sleep+0xd4/0xf0 (8)
 [<c011749f>] kmap+0x1f/0x50 (36)
 [<c014ffb8>] bounce_copy_vec+0x28/0x70 (16)
 [<c01500bc>] copy_to_high_bio_irq+0x5c/0x70 (32)
 [<c0150221>] __bounce_end_io_read+0x41/0x50 (28)
 [<c0150258>] bounce_end_io_read+0x28/0x30 (20)
 [<c01686b9>] bio_endio+0x59/0x80 (12)
 [<c025a2bc>] ide_end_request+0x2c/0xc0 (16)
 [<c024bc02>] __end_that_request_first+0x1c2/0x230 (12)
 [<c01386cf>] up_mutex+0xaf/0x100 (16)
 [<c025a197>] __ide_end_request+0x77/0x170 (36)
 [<c025a2f5>] ide_end_request+0x65/0xc0 (36)
 [<c0260810>] task_end_request+0x40/0x80 (36)
 [<c0260941>] task_in_intr+0xf1/0x110 (24)
 [<c0260850>] task_in_intr+0x0/0x110 (20)
 [<c025bda1>] ide_intr+0xe1/0x170 (12)
 [<c0140e3b>] handle_IRQ_event+0x5b/0xd0 (32)
 [<c0141683>] do_hardirq+0xa3/0x100 (48)
 [<c01417f6>] do_irqd+0x116/0x1e0 (36)
 [<c01416e0>] do_irqd+0x0/0x1e0 (44)
 [<c0135e37>] kthread+0xb7/0xc0 (4)
 [<c0135d80>] kthread+0x0/0xc0 (28)
 [<c01012e5>] kernel_thread_helper+0x5/0x10 (16)
---------------------------

| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c01398e7>] .... print_traces+0x17/0x50
.....[<00000000>] .. ( <= 0x0)

This looks like it was triggered by bounce_copy_vec calling kmap_atomic which is now just kmap with irqs disabled. Does this need to change to __kmap_atomic? Is this also used to make things more preemptible, and start removing the local_irq_saves? I'd like to know so that you don't need to make the patches yourself and I can handle things like this, but I need to know what the general ideas are. Also, am I the only one that has highmem support enabled, because this looks like this bug would have been triggered by anyone.

Thanks,

-- Steve

john cooper

2004-12-09 20:36 UTC #820 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 3 lines · Steven Rostedt

...Also, am I the only one that
has highmem support enabled, because this looks like this bug would have been triggered by anyone.

I have not encountered this in -12 with CONFIG_HIGHMEM4G running on an SMP Opteron with 1GB memory. Details attached.

-john

Signature · 1 line

[email protected]

Steven Rostedt

reviewer

2004-12-09 22:19 UTC #821 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 8 lines · john cooper

Quoted 3 lines · Steven Rostedt
...Also, am I the only one that
has highmem support enabled, because this looks like this bug would have been triggered by anyone.
I have not encountered this in -12 with CONFIG_HIGHMEM4G running on an SMP Opteron with 1GB memory. Details attached.

Hi John,

Could you do me a big favor? Put a print in mm/highmem.c bounce_copy_vec to see if you get into it. If you don't then it seems that my system is triggering this and it just so happens that yours does not. Looking at my dump, it shows that there may have been some contention in the ide interrupt.

I let it run for a little longer and the system does eventually get to a login prompt, but I'm looking into this further.

-- Steve

john cooper

2004-12-09 23:12 UTC #822 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 3 lines · Steven Rostedt

Could you do me a big favor? Put a print in mm/highmem.c bounce_copy_vec to see if you get into it. If you don't then it seems that my system is triggering this and it just so happens that yours does not.

Did so. For whatever reason I don't appear to be getting into bounce_copy_vec() during bootup as you seem to be.

-john

Signature · 1 line

[email protected]

Ingo Molnar

2004-12-09 22:10 UTC #823 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

In reply to Steven Rostedt

Quoted 8 lines · Steven Rostedt

This looks like it was triggered by bounce_copy_vec calling kmap_atomic which is now just kmap with irqs disabled. Does this need to change to __kmap_atomic? Is this also used to make things more preemptible, and start removing the local_irq_saves? I'd like to know so that you don't need to make the patches yourself and I can handle things like this, but I need to know what the general ideas are. Also, am I the only one that has highmem support enabled, because this looks like this bug would have been triggered by anyone.

the fix would be to find the place that disabled interrupts, and to check that it's safe to change it to local_irq_disable_nort() (or whatever other variant is used). Usually it's safe.

Ingo

Steven Rostedt

reviewer

2004-12-10 06:11 UTC #824 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 11 lines · Ingo Molnar

Quoted 8 lines · Steven Rostedt
This looks like it was triggered by bounce_copy_vec calling kmap_atomic which is now just kmap with irqs disabled. Does this need to change to __kmap_atomic? Is this also used to make things more preemptible, and start removing the local_irq_saves? I'd like to know so that you don't need to make the patches yourself and I can handle things like this, but I need to know what the general ideas are. Also, am I the only one that has highmem support enabled, because this looks like this bug would have been triggered by anyone.

Quoted 3 lines

the fix would be to find the place that disabled interrupts, and to check that it's safe to change it to local_irq_disable_nort() (or whatever other variant is used). Usually it's safe.

Hi Ingo,

Here's your fix. I haven't seen anything else cause the bug, and since it uses local_irq_save, I guess the bounce_copy_vec can be called with interrupts disabled. Since the kmap_atomic (or just kmap) checks for that, I don't think I need more than what I've done.

Second, my ethernet doesn't work, and it really seems to be some kind of interrupt trouble. It sends out ARPs but doesn't see them come back, and it also doesn't seem to know that it sent them out. I get the following:

NETDEV WATCHDOG: eth0: transmit timed out
eth0: transmit timed out, tx_status 00 status e601.
diagnostics: net 0ccc media 8880 dma 0000003a fifo 0000
eth0: Interrupt posted but not delivered -- IRQ blocked by another device?
Flags; bus-master 1, dirty 33(1) current 33(1) Transmit list 00000000 vs. f75012a0.

  0: @f7501200  length 80000043 status 8c010043
  1: @f75012a0  length 8000007a status 0c01007a
  2: @f7501340  length 8000002a status 0001002a
  3: @f75013e0  length 80000098 status 0c010098
  4: @f7501480  length 8000002a status 0001002a
  5: @f7501520  length 8000002a status 0001002a
  6: @f75015c0  length 8000002a status 0001002a
  7: @f7501660  length 8000002a status 0001002a
  8: @f7501700  length 80000043 status 0c010043
  9: @f75017a0  length 80000043 status 0c010043
  10: @f7501840  length 8000004f status 0c01004f
  11: @f75018e0  length 8000004f status 0c01004f
  12: @f7501980  length 80000043 status 0c010043
  13: @f7501a20  length 8000007a status 0c01007a
  14: @f7501ac0  length 80000098 status 0c010098
  15: @f7501b60  length 8000002a status 8001002a

I have a (from lspci)
0000:02:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)

I can look into this further to see what the problem is. One funny note, on the vanilla kernel, my eth0 is at interrupt 177, but on the rt patched kernel its swapped with the sound card and is at interrupt 169. Well it's getting too late for me now (its 1am my time (01:00 for you European folks ;-) , and I need to get up at 6:30 am). Tomorrow, I'll hack on it some more.

Oh, and here's the highmem patch:

highmem.c +2 -2

Index: mm/highmem.c
===================================================================
@@ -240,11 +240,11 @@
 	unsigned long flags;
 	unsigned char *vto;
 
-	local_irq_save(flags);
+	local_irq_save_nort(flags);
 	vto = kmap_atomic(to->bv_page, KM_BOUNCE_READ);
 	memcpy(vto + to->bv_offset, vfrom, to->bv_len);
 	kunmap_atomic(vto, KM_BOUNCE_READ);
-	local_irq_restore(flags);
+	local_irq_restore_nort(flags);
 }
 
 #else /* CONFIG_HIGHMEM */

-- Steve

Ingo Molnar

2004-12-10 11:06 UTC #825 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 6 lines · Steven Rostedt

Hi Ingo,
Here's your fix. I haven't seen anything else cause the bug, and since it uses local_irq_save, I guess the bounce_copy_vec can be called with interrupts disabled. Since the kmap_atomic (or just kmap) checks for that, I don't think I need more than what I've done.

yeah, it looks good to me too - thx. I've uploaded -32-16 with your fix included.

Ingo

Ingo Molnar

2004-12-10 11:13 UTC #826 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

In reply to Steven Rostedt

Quoted 34 lines · Steven Rostedt

Second, my ethernet doesn't work, and it really seems to be some kind of interrupt trouble. It sends out ARPs but doesn't see them come back, and it also doesn't seem to know that it sent them out. I get the following:
NETDEV WATCHDOG: eth0: transmit timed out
eth0: transmit timed out, tx_status 00 status e601.
diagnostics: net 0ccc media 8880 dma 0000003a fifo 0000
eth0: Interrupt posted but not delivered -- IRQ blocked by another device?
Flags; bus-master 1, dirty 33(1) current 33(1) Transmit list 00000000 vs. f75012a0.
  0: @f7501200  length 80000043 status 8c010043
  1: @f75012a0  length 8000007a status 0c01007a
  2: @f7501340  length 8000002a status 0001002a
  3: @f75013e0  length 80000098 status 0c010098
  4: @f7501480  length 8000002a status 0001002a
  5: @f7501520  length 8000002a status 0001002a
  6: @f75015c0  length 8000002a status 0001002a
  7: @f7501660  length 8000002a status 0001002a
  8: @f7501700  length 80000043 status 0c010043
  9: @f75017a0  length 80000043 status 0c010043
  10: @f7501840  length 8000004f status 0c01004f
  11: @f75018e0  length 8000004f status 0c01004f
  12: @f7501980  length 80000043 status 0c010043
  13: @f7501a20  length 8000007a status 0c01007a
  14: @f7501ac0  length 80000098 status 0c010098
  15: @f7501b60  length 8000002a status 8001002a 
I have a (from lspci)
0000:02:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)

Quoted 6 lines

I can look into this further to see what the problem is. One funny note, on the vanilla kernel, my eth0 is at interrupt 177, but on the rt patched kernel its swapped with the sound card and is at interrupt
169. Well it's getting too late for me now (its 1am my time (01:00 for you European folks ;-) , and I need to get up at 6:30 am). Tomorrow, I'll hack on it some more.

yeah, please check this - you are the first one to report this issue.

A good first step would be to go switch to a non-PREEMPT_RT preemption model (but to keep the -RT codebase) and see whether the breakage is related to that. If the breakage goes away with say PREEMPT_DESKTOP then i'd suggest to enable PREEMPT_HARDIRQS and PREEMPT_SOFTIRQS (but keep PREEMPT_DESKTOP) - do these alone trigger the breakage? E.g. there was an obscure timing bug in the floppy driver that only triggered with PREEMPT_HARDIRQS enabled. So it's not out of question that there's some other driver bug/race in hiding. The other possibility is some generic -RT kernel breakage - like the SLAB issue was.

Ingo

K.R. Foley

2004-12-10 16:33 UTC #827 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 56 lines · Ingo Molnar

Quoted 33 lines · Steven Rostedt
Second, my ethernet doesn't work, and it really seems to be some kind of interrupt trouble. It sends out ARPs but doesn't see them come back, and it also doesn't seem to know that it sent them out. I get the following:
NETDEV WATCHDOG: eth0: transmit timed out
eth0: transmit timed out, tx_status 00 status e601.
diagnostics: net 0ccc media 8880 dma 0000003a fifo 0000
eth0: Interrupt posted but not delivered -- IRQ blocked by another device?
Flags; bus-master 1, dirty 33(1) current 33(1) Transmit list 00000000 vs. f75012a0.
 0: @f7501200  length 80000043 status 8c010043
 1: @f75012a0  length 8000007a status 0c01007a
 2: @f7501340  length 8000002a status 0001002a
 3: @f75013e0  length 80000098 status 0c010098
 4: @f7501480  length 8000002a status 0001002a
 5: @f7501520  length 8000002a status 0001002a
 6: @f75015c0  length 8000002a status 0001002a
 7: @f7501660  length 8000002a status 0001002a
 8: @f7501700  length 80000043 status 0c010043
 9: @f75017a0  length 80000043 status 0c010043
 10: @f7501840  length 8000004f status 0c01004f
 11: @f75018e0  length 8000004f status 0c01004f
 12: @f7501980  length 80000043 status 0c010043
 13: @f7501a20  length 8000007a status 0c01007a
 14: @f7501ac0  length 80000098 status 0c010098
 15: @f7501b60  length 8000002a status 8001002a 
I have a (from lspci) 0000:02:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)
Quoted 6 lines
I can look into this further to see what the problem is. One funny note, on the vanilla kernel, my eth0 is at interrupt 177, but on the rt patched kernel its swapped with the sound card and is at interrupt
169. Well it's getting too late for me now (its 1am my time (01:00 for you European folks ;-) , and I need to get up at 6:30 am). Tomorrow, I'll hack on it some more.
yeah, please check this - you are the first one to report this issue.
A good first step would be to go switch to a non-PREEMPT_RT preemption model (but to keep the -RT codebase) and see whether the breakage is related to that. If the breakage goes away with say PREEMPT_DESKTOP then i'd suggest to enable PREEMPT_HARDIRQS and PREEMPT_SOFTIRQS (but keep PREEMPT_DESKTOP) - do these alone trigger the breakage? E.g. there was an obscure timing bug in the floppy driver that only triggered with PREEMPT_HARDIRQS enabled. So it's not out of question that there's some other driver bug/race in hiding. The other possibility is some generic -RT kernel breakage - like the SLAB issue was.

Quoted 2 lines

Ingo

I actually have this same card in the system I am sending this from currently running 2.6.10-rc2-mm3-V0.7.32-12 #15 SMP.

Steven Rostedt

reviewer

2004-12-10 18:04 UTC #828 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 6 lines · K.R. Foley

Quoted 5 lines
Quoted 5 lines
I have a (from lspci)
0000:02:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)

<snip>

Quoted 2 lines

I actually have this same card in the system I am sending this from currently running 2.6.10-rc2-mm3-V0.7.32-12 #15 SMP.

I have a dual Athlon MP system with Tyan S2466N4 Motherboard. It looks more of a problem with the interrupt controller. I downloaded -17 and tried that, but now it hangs on starting up cups and the last message from the kernel is:

lp0: using parport0 (interrupt-driven).

It looks like a pretty bad lock up, since I know exactly when it happens since the cursor stops blinking.

I'm right now compiling the PREEMPT_DESKTOP kernel to see if that gives me the same problems.

-- Steve

Steven Rostedt

reviewer

2004-12-11 02:26 UTC #829 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

In reply to Ingo Molnar

Quoted 7 lines · Ingo Molnar

Quoted 5 lines · Steven Rostedt
Second, my ethernet doesn't work, and it really seems to be some kind of interrupt trouble. It sends out ARPs but doesn't see them come back, and it also doesn't seem to know that it sent them out. I get the following:

<snip>

Quoted 3 lines

Quoted 1 line
I'll hack on it some more.
yeah, please check this - you are the first one to report this issue.

Hi Ingo, I found the problem! and I now know why John Cooper didn't have this problem too. I have CONFIG_PCI_MSI defined. I don't know why, I must have seen the option a while ago and said to myself "That looks cool, lets try it". Since I started with the config file of the vanilla kernel with your rt patches, it was still on.

Anyways, what is happening is that the io_apic code is mapping irqs to vectors, and your code didn't account for it. So here's my patch.

i386/kernel/io_apic.c +8 -2

Index: arch/i386/kernel/io_apic.c
===================================================================
@@ -1942,12 +1942,14 @@
 
 static void end_level_ioapic_irq(unsigned int irq)
 {
+#ifndef CONFIG_PCI_MSI
 	if (!(irq_desc[irq].status & (IRQ_DISABLED | IRQ_INPROGRESS)) &&
 							irq_desc[irq].action)
+#endif
 		unmask_IO_APIC_irq(irq);
 }
 
-#else /* !CONFIG_PREEMPT_HARDIRQS || !CONFIG_SMP */
+#else /* !CONFIG_PREEMPT_HARDIRQS */
 
 static void mask_and_ack_level_ioapic_irq(unsigned int irq)
 {

@@ -2035,7 +2037,11 @@
 {
 	int irq = vector_to_irq(vector);
 
-	end_level_ioapic_irq(irq);
+#if defined(CONFIG_PREEMPT_HARDIRQS)
+	if (!(irq_desc[vector].status & (IRQ_DISABLED | IRQ_INPROGRESS)) &&
+							irq_desc[vector].action)
+#endif
+		end_level_ioapic_irq(irq);
 }
 
 static void enable_level_ioapic_vector(unsigned int vector)

--------------------

I also removed the comment "!CONFIG_SMP" since it really wasn't correct. So I can get back to looking at other things. This also may explain why my system would hang with my usb printer attached (the usb interrupts were vectored too). I'll plug my printer back in and see if it works now. I'll let you know if I have any more problems.

Thanks,

-- Steve

Steven Rostedt

reviewer

2004-12-11 03:02 UTC #830 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 1 line · Steven Rostedt

Hi Ingo, I found the problem! and I now know why John Cooper didn't

^^^^^^^^^^^
Sorry, I meant K.R.Foley,
Since he's the one with the same ethernet card as me.

Quoted 4 lines

have this problem too. I have CONFIG_PCI_MSI defined. I don't know why, I must have seen the option a while ago and said to myself "That looks cool, lets try it". Since I started with the config file of the vanilla kernel with your rt patches, it was still on.

Fernando Lopez-Lezcano

2004-12-11 07:39 UTC #831 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

In reply to Steven Rostedt

Quoted 15 lines · Steven Rostedt

Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Steven Rostedt
Second, my ethernet doesn't work, and it really seems to be some kind of interrupt trouble. It sends out ARPs but doesn't see them come back, and it also doesn't seem to know that it sent them out. I get the following:
<snip>
Quoted 3 lines
Quoted 1 line
I'll hack on it some more.
yeah, please check this - you are the first one to report this issue.

Quoted 8 lines

Hi Ingo, I found the problem! and I now know why John Cooper didn't have this problem too. I have CONFIG_PCI_MSI defined. I don't know why, I must have seen the option a while ago and said to myself "That looks cool, lets try it". Since I started with the config file of the vanilla kernel with your rt patches, it was still on.
Anyways, what is happening is that the io_apic code is mapping irqs to vectors, and your code didn't account for it. So here's my patch.

Can't wait to try the patch, I don't have CONFI_PCI_MSI defined in the configurations I use. I've had problems with a network card (R8169 driver) for a while (I think I reported it), the interrupts were being ignored. Hopefully the same problem...

-- Fernando

Steven Rostedt

reviewer

2004-12-11 12:30 UTC #832 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 4 lines · Fernando Lopez-Lezcano

Can't wait to try the patch, I don't have CONFI_PCI_MSI defined in the configurations I use. I've had problems with a network card (R8169 driver) for a while (I think I reported it), the interrupts were being ignored. Hopefully the same problem...

Hi Fernando,

You may have the same problem but the patch I sent won't solve it. My patch only is a problem if you have CONFIG_PCI_MSI defined. But I'm sure there exists other instances that threading hardirqs might not work properly with other configurations. Send me your .config, and if I get time I'll take a look. (also your /proc/cpuinfo might help).

Before you send this, make sure that it is the hardirqs that's the problem. Switch to PREEMPT_DESKTOP and make sure hardirqs are not threaded. If the problem goes away, then this may be your problem. If it does not, I'm afraid that it's something else, and you don't need to send me anything.

-- Steve

Fernando Lopez-Lezcano

2004-12-13 23:35 UTC #833 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 12 lines · Steven Rostedt

Quoted 4 lines · Fernando Lopez-Lezcano
Can't wait to try the patch, I don't have CONFI_PCI_MSI defined in the configurations I use. I've had problems with a network card (R8169 driver) for a while (I think I reported it), the interrupts were being ignored. Hopefully the same problem...
You may have the same problem but the patch I sent won't solve it. My patch only is a problem if you have CONFIG_PCI_MSI defined. But I'm sure there exists other instances that threading hardirqs might not work properly with other configurations. Send me your .config, and if I get time I'll take a look. (also your /proc/cpuinfo might help).

Quoted 3 lines

Before you send this, make sure that it is the hardirqs that's the problem. Switch to PREEMPT_DESKTOP and make sure hardirqs are not threaded.

[The following is all done booting into 0.7.32-19, interrupt scheduling and priorities unchanged from the defaults]

I'm using PREEMPT_DESKTOP. I don't know how to force the kernel to not thread hardirqs. What I did (maybe the same thing?) is to boot single user, then turn off /proc/sys/kernel/hardirq_preempt, and then start the network. It works (the network). And then I tried the same thing with hardirq_preempt=1 and still worked when booting single user... :-(

So, these are the interrupts after booting single user:
CPU0

  0:      36426    IO-APIC-edge  timer  0/35956
  1:        143    IO-APIC-edge  i8042  0/143
  4:          0    IO-APIC-edge  KGDB-stub  0/0
  8:          1    IO-APIC-edge  rtc  0/1
  9:          0   IO-APIC-level  acpi  0/0
 12:        100    IO-APIC-edge  i8042  0/100
 14:         26    IO-APIC-edge  ide0  1/24
 17:         59   IO-APIC-level  libata, libata  0/59
 20:       1462   IO-APIC-level  libata  0/1462
 21:          0   IO-APIC-level  ehci_hcd, uhci_hcd, uhci_hcd, uhci_hcd,
uhci_hcd  0/0
NMI:       7544 
LOC:      36254 
ERR:          0
MIS:          0

Here's the extra one I have after I start the network

16: 7 IO-APIC-level eth0 0/7

(network works). I then telinit 3, no changes in interrupts (except for the count numbers), network continues working.

I then telinit 5 and the network dies. This is the added interrupt:

11: 0 IO-APIC-edge radeon@PCI:1:0:0 0/0

I tried repeating the whole thing but going through all the services in the transition from level 3 to level 5 one by one, and nothing happened to the network so it must be X. I then rebooted into single user, started the network and loaded the radeon kernel module alone and the network was not affected.

So, this is what I get in dmesg when I go into level 5:

NET: Registered protocol family 10
Disabled Privacy Extensions on device c03e0ec0(lo)
IPv6 over IPv4 tunneling driver
divert: not allocating divert_blk for non-ethernet device sit0
eth0: no IPv6 routers present
[drm] Initialized radeon 1.11.0 20020828 on minor 0:
ACPI: PCI interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
agpgart: Found an AGP 3.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V3 device at 0000:00:00.0 into 4x mode
agpgart: Putting AGP V3 device at 0000:01:00.0 into 4x mode [drm] Loading R200 Microcode
irq 16: nobody cared!

 [<c01041e3>] dump_stack+0x23/0x30 (20)
 [<c014d480>] __report_bad_irq+0x30/0xa0 (24)
 [<c014d590>] note_interrupt+0x70/0xb0 (32)
 [<c014d30c>] do_hardirq+0x13c/0x150 (40)
 [<c014d399>] do_irqd+0x79/0xb0 (32)
 [<c013bf7a>] kthread+0xaa/0xb0 (48)
 [<c0101335>] kernel_thread_helper+0x5/0x10 (153411604)
---------------------------

| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c014d2aa>] .... do_hardirq+0xda/0x150
.....[<c014d399>] .. ( <= do_irqd+0x79/0xb0)
.. [<c014045d>] .... print_traces+0x1d/0x60
.....[<c01041e3>] .. ( <= dump_stack+0x23/0x30)

handlers:
[<f88e3f00>] (rtl8169_interrupt+0x0/0x1a0 [r8169]) Disabling IRQ #16

Anything else I could test?

Quoted 3 lines

If the problem goes away, then this may be your problem. If it does not, I'm afraid that it's something else, and you don't need to send me anything.

Let me know if you still want the .config...
-- Fernando

Ingo Molnar

2004-12-15 09:52 UTC #834 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

Quoted 5 lines · Fernando Lopez-Lezcano

[The following is all done booting into 0.7.32-19, interrupt scheduling and priorities unchanged from the defaults]
I'm using PREEMPT_DESKTOP. I don't know how to force the kernel to not thread hardirqs. [...]

this is now moot for your case, but here's how you can disable hardirq
threading: you can disable CONFIG_PREEMPT_HARDIRQS in the .config, or you can disable it via the hardirq-preempt=0 boot-time kernel flag.

Ingo

Ingo Molnar

2004-12-11 09:57 UTC #835 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

In reply to Steven Rostedt

Quoted 2 lines · Steven Rostedt

Anyways, what is happening is that the io_apic code is mapping irqs to vectors, and your code didn't account for it. So here's my patch.

ah .. thanks, great debugging!

Ingo

Rui Nuno Capela

2004-12-09 14:44 UTC #836 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-12

In reply to Ingo Molnar

Quoted 7 lines · Ingo Molnar

Quoted 6 lines
SLAB draining was an oversight - it's mainly called when there is VM pressure (which is not a stricly necessary feature, so i disabled it), but i forgot about the module-unload case where it's a correctness feature. Your patch is a good starting point, i'll try to fix it on SMP too.

Quoted 7 lines

here's the full patch against a recent tree, or download the -32-12 patch from the usual place:
http://redhat.com/~mingo/realtime-preempt/
Rui, Gene, does this fix the module unload crash you are seeing?

Tested with RT-V0.7.32-12 after some usb-storage plug/unplug bonanza. All seems to be OK, at least on my laptop (P4/UP).

Cheers.

Signature · 2 lines

rncbc aka Rui Nuno Capela
[email protected]

Steven Rostedt

reviewer

2004-12-09 13:36 UTC #837 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

In reply to Ingo Molnar

Quoted 16 lines · Ingo Molnar

Quoted 3 lines · Steven Rostedt
Ingo really scares me with all the removing of local_irq_disables in the rt mode. I'm not sure exactly what is going on there, and why they can, or should be removed. Ingo?
it is done so that the SLAB code can be fully preempted too. The SLAB code is of central importance to the -RT project, if it's not fully preemptible then that has a ripple effect on other subsystems (timer, signal code, file handling, etc.).
So while making it fully preemptible was quite challenging (==dangerous, scary), i couldnt just keep the SLAB using raw spinlocks, due to the locking dependencies. (nor did i have any true inner desire to keep it non-preemptible - the point of PREEMPT_RT is to have everything preemptible. I want to see how much preemption the Linux kernel can take =B-) It has held up surprisingly well i have to say.)

<snip>

Quoted 2 lines

Ingo

Ingo,

Thanks for the write up. It really clears things up for me. Now I understand your approach, not only for slabs, but other areas of the kernel. Once again, thanks for the explanation.

-- Steve

Ingo Molnar

2004-12-09 09:06 UTC #838 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.32-6

In reply to Steven Rostedt

Quoted 6 lines · Steven Rostedt

Hi Ingo,
I found a race condition in slab.c, but I'm still trying to figure out exactly how it's playing out. This has to do with dynamic loading and unloading of caches. I have a small test case that simulates the problem at http://home.stny.rr.com/rostedt/tests/sillycaches.tgz

good catch! When i converted slab.c to RT i mistakenly thought that SLAB flushing (draining) is only an SMP optimization (which i thus generously disabled), but i forgot about module unloading. This could indeed explain some of the unresolved bugs in the -RT patchset.

Ingo

Ingo Molnar

2004-12-14 13:28 UTC #839 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

i have released the -V0.7.33-0 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:

- due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode, but there's one interesting new feature: -rc3-mm1 introduced the ->unlocked_ioctl method which is now an official way to do BKL-less ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this facility. The ALSA/sound guys might be interested in these bits. Thus another chunk of -RT could go upstream.

- IO-APIC/MSI fix from Steven Rostedt.

- fixed a tracer bug which would produce a kernel warning and an empty /proc/latency_trace if the trace buffer overflows.

to create a -V0.7.33-0 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc3.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc3/2.6.10-rc3-mm1/2.6.10-rc3-mm1.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc3-mm1-V0.7.33-0

Ingo

Steven Rostedt

reviewer

2004-12-14 19:34 UTC #840 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

[RFC]

Ingo,

Any thought about adding a one shot timer for the system?

I know there's been talk about this in the mainline (actually going on right now in the dynamic Hz thread), but I figured that this would especially be good for a RT system and not be worried about the HZ settings.

An example would be to have the timer interrupt be implemented as a one shot timer, and be able to register actions to times with it. The jiffy calculations could just be an event that is registered to go off once every HZ. The timer interrupt would then just have to look at all the events that have expired (using something like the TSC to determine what expires), execute their registered actions, and if need be, the action could add itself back on the event queue (the jiffy timer would do this), and then the timer would set itself to go off at the next event. I believe something like this was done by the utime patch (now with the KURT project).

This way the RT processes would not need to depend on the HZ settings.

I'd be willing to implement this, since I'm doing it regardless for a client of mine. But I would like to know your input on this.

Thanks,

-- Steve

Lee Revell

2004-12-14 20:10 UTC #841 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 6 lines · Steven Rostedt

[RFC]
Ingo,
Any thought about adding a one shot timer for the system?

Isn't this what George Anzinger is working on?

http://sourceforge.net/projects/high-res-timers/

Lee

Steven Rostedt

reviewer

2004-12-14 20:46 UTC #842 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 5 lines · Lee Revell

Quoted 4 lines · Steven Rostedt
[RFC]
Ingo,

Quoted 9 lines

Quoted 2 lines
Any thought about adding a one shot timer for the system?
Isn't this what George Anzinger is working on?
http://sourceforge.net/projects/high-res-timers/
Lee

A quick look at this looks like this is what I was looking for. I'd need to review the code in a more detailed aspect but first glance, Yes, this is what I want.

Now, since High Res-timers and RT seem to go together, what are the plans for merging these? If this is indeed what I need, then I'll be doing it to myself, but if these can merge in the public domain, then I would say, all the better.

-- Steve

Ingo Molnar

2004-12-14 21:18 UTC #843 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 14 lines · Steven Rostedt

Quoted 4 lines
Quoted 4 lines
[RFC]
Ingo,
Quoted 9 lines
Quoted 2 lines
Any thought about adding a one shot timer for the system?
Isn't this what George Anzinger is working on?
http://sourceforge.net/projects/high-res-timers/
Lee

Quoted 7 lines

A quick look at this looks like this is what I was looking for. I'd need to review the code in a more detailed aspect but first glance, Yes, this is what I want.
Now, since High Res-timers and RT seem to go together, what are the plans for merging these? If this is indeed what I need, then I'll be doing it to myself, [...]

i've been thinking about it on and off. If you would/could try it that would certainly help. RT for Linux is a dance of many small steps.

the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.

Ingo

Lee Revell

2004-12-14 21:48 UTC #844 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 7 lines · Ingo Molnar

i've been thinking about it on and off. If you would/could try it that would certainly help. RT for Linux is a dance of many small steps.
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.

OK, I'll give this a shot. It would certainly help on my underpowered EPIA system, where my tests show 2.1% residency for the timer ISR with HZ=1000. On a system like this I would expect the difference to be perceptible with a regular desktop workload.

Lee

Ingo Molnar

2004-12-14 21:54 UTC #845 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 9 lines · Lee Revell

Quoted 7 lines · Ingo Molnar
i've been thinking about it on and off. If you would/could try it that would certainly help. RT for Linux is a dance of many small steps.
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.

Quoted 4 lines

OK, I'll give this a shot. It would certainly help on my underpowered EPIA system, where my tests show 2.1% residency for the timer ISR with HZ=1000. On a system like this I would expect the difference to be perceptible with a regular desktop workload.

a warning: it's probably quite complex merging work - while the two projects work on different conceptual issues, they touch the same code.

Ingo

Lee Revell

2004-12-14 21:58 UTC #846 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 11 lines · Ingo Molnar

Quoted 9 lines · Lee Revell
Quoted 7 lines · Ingo Molnar
i've been thinking about it on and off. If you would/could try it that would certainly help. RT for Linux is a dance of many small steps.
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.

Quoted 8 lines

Quoted 4 lines
OK, I'll give this a shot. It would certainly help on my underpowered EPIA system, where my tests show 2.1% residency for the timer ISR with HZ=1000. On a system like this I would expect the difference to be perceptible with a regular desktop workload.
a warning: it's probably quite complex merging work - while the two projects work on different conceptual issues, they touch the same code.

Yeah, I expected this, I'll wait for George to update it. I am testing with 2.6.9 vanilla for the time being.

Lee

George Anzinger

2004-12-14 21:53 UTC #847 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Quoted 26 lines · Ingo Molnar

Quoted 14 lines · Steven Rostedt
Quoted 4 lines
Quoted 4 lines
[RFC]
Ingo,
Quoted 9 lines
Quoted 2 lines
Any thought about adding a one shot timer for the system?
Isn't this what George Anzinger is working on?
http://sourceforge.net/projects/high-res-timers/
Lee
Quoted 7 lines
A quick look at this looks like this is what I was looking for. I'd need to review the code in a more detailed aspect but first glance, Yes, this is what I want.
Now, since High Res-timers and RT seem to go together, what are the plans for merging these? If this is indeed what I need, then I'll be doing it to myself, [...]

Quoted 7 lines

i've been thinking about it on and off. If you would/could try it that would certainly help. RT for Linux is a dance of many small steps.
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.

Well, I guess I am just backward :) I plan to port it to the current RT today or tomorrow (if all goes well). I will then work on the changes needed to get it into -mm. Guess I will be supporting two versions for a bit...

Signature · 2 lines

George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

Steven Rostedt

reviewer

2004-12-14 22:01 UTC #848 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 1 line · George Anzinger

Ingo Molnar wrote:

...

Quoted 8 lines

Quoted 8 lines
Quoted 3 lines
Now, since High Res-timers and RT seem to go together, what are the plans for merging these? If this is indeed what I need, then I'll be doing it to myself, [...]
i've been thinking about it on and off. If you would/could try it that would certainly help. RT for Linux is a dance of many small steps.

Quoted 9 lines

Quoted 4 lines
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Well, I guess I am just backward :) I plan to port it to the current RT today or tomorrow (if all goes well). I will then work on the changes needed to get it into -mm. Guess I will be supporting two versions for a bit...

I need what you've been working on, so I'm available to help get the two together. Since you two (Ingo and George) are the creators of these, I'll let you fight it out at how to go about this. If it ends up being two patches for you George, then I would be glad to help maintain the RT/HigRes one.

-- Steve

Ingo Molnar

2004-12-14 22:33 UTC #849 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to George Anzinger

Quoted 9 lines · George Anzinger

Quoted 4 lines
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Well, I guess I am just backward :) I plan to port it to the current RT today or tomorrow (if all goes well). I will then work on the changes needed to get it into -mm. Guess I will be supporting two versions for a bit...

very good - i can carry it along in -RT, and your VST bits are certainly an immediate bonus feature for the non-hard-RT (=laptop, desktop, audio) folks too.

Ingo

Lee Revell

2004-12-14 22:14 UTC #850 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Quoted 4 lines · Ingo Molnar

the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.

Among other things I think Paul Davis mentioned that George's high res timer patch would make it possible for JACK to send MIDI clock. This would be a huge improvement.

Lee

Ingo Molnar

2004-12-15 00:18 UTC #851 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 9 lines · Lee Revell

Quoted 4 lines · Ingo Molnar
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Among other things I think Paul Davis mentioned that George's high res timer patch would make it possible for JACK to send MIDI clock. This would be a huge improvement.

<clueless question> roughly what latency/accuracy requirements does the MIDI clock have, and why is it an advantage if Linux generates it? What generates it otherwise - external MIDI hardware? Or was the problem mainly not latency/accuracy but that Linux couldnt generate a finegrained enough clock?

Ingo

Lee Revell

2004-12-14 22:50 UTC #852 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 12 lines · Ingo Molnar

Quoted 9 lines · Lee Revell
Quoted 4 lines · Ingo Molnar
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Among other things I think Paul Davis mentioned that George's high res timer patch would make it possible for JACK to send MIDI clock. This would be a huge improvement.

Quoted 5 lines

<clueless question> roughly what latency/accuracy requirements does the MIDI clock have, and why is it an advantage if Linux generates it? What generates it otherwise - external MIDI hardware? Or was the problem mainly not latency/accuracy but that Linux couldnt generate a finegrained enough clock?

Being able to send or receive MIDI clock is a common feature for hardware and software samplers, sequencers, etc. It just allows more flexibility in your setup. I think currently Linux can receive MIDI clock better than it can send.

To answer the last question I think the clock was not fine grained enough. But as far as timing requirements I don't actually know the details. Paul?

Lee

Paul Davis

2004-12-14 23:01 UTC #853 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 10 lines

Quoted 10 lines
Quoted 9 lines · Ingo Molnar
Quoted 4 lines
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Among other things I think Paul Davis mentioned that George's high res timer patch would make it possible for JACK to send MIDI clock. This would be a huge improvement.

Quoted 5 lines

Quoted 5 lines
<clueless question> roughly what latency/accuracy requirements does the MIDI clock have, and why is it an advantage if Linux generates it? What generates it otherwise - external MIDI hardware? Or was the problem mainly not latency/accuracy but that Linux couldnt generate a finegrained enough clock?

the latter. to send MIDI Clock or MIDI Timecode requires an interrupt source that is not locked to jiffie-ish intervals or power-of-2 related intervals. For example, MTC requires sending 2 bytes roughly every 0.8msec. Sending them every msec isn't good enough, in general.

my understanding of the HRT patch is that it allows the timer to be reprogrammed to elapse with nanosecond resolution. i don't understand why linus has been so reluctant to move linux in this direction, other than it being hard to fit into the existing fixed interval timer framework.

--p

Ingo Molnar

2004-12-15 09:33 UTC #854 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 4 lines · Paul Davis

the latter. to send MIDI Clock or MIDI Timecode requires an interrupt source that is not locked to jiffie-ish intervals or power-of-2 related intervals. For example, MTC requires sending 2 bytes roughly every 0.8msec. Sending them every msec isn't good enough, in general.

yeah, i can understand that - 20% slower music is bad, even to kernel hackers ;-)

Quoted 5 lines

my understanding of the HRT patch is that it allows the timer to be reprogrammed to elapse with nanosecond resolution. i don't understand why linus has been so reluctant to move linux in this direction, other than it being hard to fit into the existing fixed interval timer framework.

i dont think there's any particular type of feeling from Linus towards HRT - it's always the quality of the patch (and the underlying fundamental issues) that controls, plus 'demand'. So integrating this stuff is not a matter of will, it's a matter of having good enough code.

the current 'fixed interval timer framework' is one reason that makes Linux scale so well on big networked boxes, which can easily have millions of timers running (per CPU). Sub-jiffy solutions i've seen so far tended to concentrate on making sub-jiffies work, while keeping fixed interval timers only as a side-effect. That's not how usage demands it - right now fixed interval timers are king in terms of usage (and will be king even if we had HRT integrated), so subjiffy timers must be very precisely engineered to not hurt fixed interval timers. Plus if we touch the timer code then maybe it could be made more deterministic (cascading isnt quite deterministic right now) - which further complicates things. It's not a simple and clear-cut task for sure.

Ingo

Lee Revell

2004-12-15 16:23 UTC #855 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 9 lines · Ingo Molnar

Quoted 4 lines · Paul Davis
the latter. to send MIDI Clock or MIDI Timecode requires an interrupt source that is not locked to jiffie-ish intervals or power-of-2 related intervals. For example, MTC requires sending 2 bytes roughly every 0.8msec. Sending them every msec isn't good enough, in general.
yeah, i can understand that - 20% slower music is bad, even to kernel hackers ;-)

It's a bit worse than that. If there is excessive jitter in your MIDI clock/MTC then your other devices will fail to lock on to it at all - it does not degrade gracefully.

Lee

linux-os

2004-12-14 23:28 UTC #856 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Lee Revell

Quoted 24 lines · Lee Revell

Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Lee Revell
Quoted 4 lines · Ingo Molnar
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Among other things I think Paul Davis mentioned that George's high res timer patch would make it possible for JACK to send MIDI clock. This would be a huge improvement.
Quoted 5 lines
<clueless question> roughly what latency/accuracy requirements does the MIDI clock have, and why is it an advantage if Linux generates it? What generates it otherwise - external MIDI hardware? Or was the problem mainly not latency/accuracy but that Linux couldnt generate a finegrained enough clock?
Being able to send or receive MIDI clock is a common feature for hardware and software samplers, sequencers, etc. It just allows more flexibility in your setup. I think currently Linux can receive MIDI clock better than it can send.

Quoted 6 lines

To answer the last question I think the clock was not fine grained enough. But as far as timing requirements I don't actually know the details. Paul?
Lee

When I use Cakewalk Home-Studio to record Music from my MIDI piano, I notice that the clock-resolution shown is several orders of magnitude better than anything a PC can generate! I haven't got a clue where this information comes from. It is in seconds, starting at 1 (not zero, don't know why) and has resolution of microseconds with no missing codes. This is on a M$ PC.

This generates a highly-accurate "time ruler". One can back-up through this time and resolve samples that appear synchronous but can be displaced in time with apparent resolution much better than the 38,400 baud-rate of MIDI. I don't know how they do it, but this is the MIDI "sample-clock". It has to be virtual because there isn't any hardware on a PC that can duplicate it.

It is likely that they use a software PLL with some periodic updates from a timer-tick, but it sure looks impressive to see "real-time" with such resolution on a PC.

I'm pretty sure that if Cakewalk decided to port Home Studio to Linux, they would be able to do it with no technical hurdles. Its just that, for Music, most use Apple and cheapskates like me use PCs running M$.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.
98.36% of all statistics are fiction.

Paul Davis

2004-12-15 01:57 UTC #857 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 3 lines

When I use Cakewalk Home-Studio to record Music from my MIDI piano, I notice that the clock-resolution shown is several orders of magnitude better than anything a PC can generate! I haven't got

incoming timing is a solved problem. the problem is scheduling outgoing data, which requires an interrupt source of some kind.

Quoted 1 line

a clue where this information comes from. It is in seconds, starting

they almost certainly use either the sample clock of the audio interface interpolated/estimated using a DLL (Delay Locked Loop).

Quoted 4 lines

I'm pretty sure that if Cakewalk decided to port Home Studio to Linux, they would be able to do it with no technical hurdles. Its just that, for Music, most use Apple and cheapskates like me use PCs running M$.

Real cheapskates would use Linux with Rosegarden, Ardour et al :)

--p

Gene Heskett

2004-12-15 02:49 UTC #858 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to linux-os

Quoted 12 lines · linux-os

Quoted 2 lines · Lee Revell
Quoted 1 line
<clueless question> roughly what latency/accuracy requirements
Lee
When I use Cakewalk Home-Studio to record Music from my MIDI piano, I notice that the clock-resolution shown is several orders of magnitude better than anything a PC can generate! I haven't got a clue where this information comes from. It is in seconds, starting at 1 (not zero, don't know why) and has resolution of microseconds with no missing codes. This is on a M$ PC.

Quoted 7 lines

This generates a highly-accurate "time ruler". One can back-up through this time and resolve samples that appear synchronous but can be displaced in time with apparent resolution much better than the 38,400 baud-rate of MIDI. I don't know how they do it, but this is the MIDI "sample-clock". It has to be virtual because there isn't any hardware on a PC that can duplicate it.

Midi's baud rate isn't 38,400, its 31,250. Check your references. It should be in any keyboards manual.

Quoted 3 lines

It is likely that they use a software PLL with some periodic updates from a timer-tick, but it sure looks impressive to see "real-time" with such resolution on a PC.

Well, it is pretty slow for a non multiuser os equipt with todays hardware.

Quoted 9 lines

I'm pretty sure that if Cakewalk decided to port Home Studio to Linux, they would be able to do it with no technical hurdles. Its just that, for Music, most use Apple and cheapskates like me use PCs running M$.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.

I don't want to bring politics into this group it has no place
here. None, nada. But, since you mentioned it, screw Ashcroft
and the camel that ride in on him. He has done more to bring true democracy to its knees in 4 years than all others before him in our 230 years.

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Gene Heskett

2004-12-15 02:38 UTC #859 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Quoted 12 lines · Ingo Molnar

Quoted 10 lines · Lee Revell
Quoted 5 lines · Ingo Molnar
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Among other things I think Paul Davis mentioned that George's high res timer patch would make it possible for JACK to send MIDI clock. This would be a huge improvement.

Quoted 7 lines

<clueless question> roughly what latency/accuracy requirements does the MIDI clock have, and why is it an advantage if Linux generates it? What generates it otherwise - external MIDI hardware? Or was the problem mainly not latency/accuracy but that Linux couldnt generate a finegrained enough clock?
Ingo

I'm not sure of the exact reasons, Ingo. But the midi clock is a bit of an odd man out in the normal progression of baud rates, its 31,250 baud, in case you didn't already know that.

A lot of systems resort to a hardware timer driving a seriel shift register as I doubt if one could guarantee writing to a single bit port with an interval error of much more than 5% bit to bit, and cumulatively less than that for the overall byte. It can be done though, we have a program for the coco's that run on a .89 mhz clock that can actually drive the seriel port well enough to run a midi keyboard plugged into it. I've run it, and it has enough spare time that I can steer some instruments to a second homemade midi pack plugged into its expansion interface at the same time & no descernable squiggles in the beat of the music.

Did you see my comment about the later versions seeming to slow seti a wee bit? Other than that, I'm in love with this, the whole system just plain feels better. The only thing on my wish list right now is to be able to shut tvtime up, it grows the system log about a megabyte a minute with its missed read reports. Or is it tvtime that needs help?

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

K.R. Foley

2004-12-15 15:24 UTC #860 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 40 lines · Gene Heskett

Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Lee Revell
Quoted 5 lines · Ingo Molnar
the two projects are obviously complementary and i have no intention to reinvent the wheel in any way. Best would be to bring hires timers up to upstream-mergable state (independently of the -RT patch) and ask Andrew to include it in -mm, then i'd port -RT to it automatically.
Among other things I think Paul Davis mentioned that George's high res timer patch would make it possible for JACK to send MIDI clock. This would be a huge improvement.
Quoted 7 lines
<clueless question> roughly what latency/accuracy requirements does the MIDI clock have, and why is it an advantage if Linux generates it? What generates it otherwise - external MIDI hardware? Or was the problem mainly not latency/accuracy but that Linux couldnt generate a finegrained enough clock?
Ingo
I'm not sure of the exact reasons, Ingo. But the midi clock is a bit of an odd man out in the normal progression of baud rates, its 31,250 baud, in case you didn't already know that.
A lot of systems resort to a hardware timer driving a seriel shift register as I doubt if one could guarantee writing to a single bit port with an interval error of much more than 5% bit to bit, and cumulatively less than that for the overall byte. It can be done though, we have a program for the coco's that run on a .89 mhz clock that can actually drive the seriel port well enough to run a midi keyboard plugged into it. I've run it, and it has enough spare time that I can steer some instruments to a second homemade midi pack plugged into its expansion interface at the same time & no descernable squiggles in the beat of the music.

Quoted 7 lines

Did you see my comment about the later versions seeming to slow seti a wee bit? Other than that, I'm in love with this, the whole system just plain feels better. The only thing on my wish list right now is to be able to shut tvtime up, it grows the system log about a megabyte a minute with its missed read reports. Or is it tvtime that needs help?

Are you referring to the "Read missed before next interrupt" messages? If so, you can disable this by disabling the rtc histogram under:
Device Drivers --> Character devices --> Real Time Clock Histogram Support.

Gene Heskett

2004-12-15 16:34 UTC #861 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 8 lines · K.R. Foley

Quoted 6 lines · Gene Heskett
Did you see my comment about the later versions seeming to slow seti a wee bit? Other than that, I'm in love with this, the whole system just plain feels better. The only thing on my wish list right now is to be able to shut tvtime up, it grows the system log about a megabyte a minute with its missed read reports. Or is it tvtime that needs help?

Quoted 4 lines

Are you referring to the "Read missed before next interrupt" messages? If so, you can disable this by disabling the rtc histogram under: Device Drivers --> Character devices --> Real Time Clock Histogram Support.

Ok, thats building now. Silly Q: Can that be turned back on with a setting in /proc or /sys?

Quoted 1 line

kr

Thanks.

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

K.R. Foley

2004-12-15 16:39 UTC #862 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 11 lines · Gene Heskett

Quoted 9 lines · K.R. Foley
Quoted 6 lines · Gene Heskett
Did you see my comment about the later versions seeming to slow seti a wee bit? Other than that, I'm in love with this, the whole system just plain feels better. The only thing on my wish list right now is to be able to shut tvtime up, it grows the system log about a megabyte a minute with its missed read reports. Or is it tvtime that needs help?

Quoted 9 lines

Quoted 4 lines
Are you referring to the "Read missed before next interrupt" messages? If so, you can disable this by disabling the rtc histogram under: Device Drivers --> Character devices --> Real Time Clock Histogram Support.
Ok, thats building now. Silly Q: Can that be turned back on with a setting in /proc or /sys?

Not currently.

Fernando Lopez-Lezcano

2004-12-14 20:08 UTC #863 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Quoted 5 lines · Ingo Molnar

i have released the -V0.7.33-0 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/

Quoted 8 lines

this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
- due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode, but there's one interesting new feature: -rc3-mm1 introduced the ->unlocked_ioctl method which is now an official way to do BKL-less ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this facility. The ALSA/sound guys might be interested in these bits. Thus another chunk of -RT could go upstream.

If I build a cvs version of alsa without those patches, will it work on top of the 0.7.33 kernel? Or do I have to try to isolate the patches and apply them to current alsa cvs?

-- Fernando

Ingo Molnar

2004-12-14 20:36 UTC #864 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 3 lines · Fernando Lopez-Lezcano

If I build a cvs version of alsa without those patches, will it work on top of the 0.7.33 kernel? Or do I have to try to isolate the patches and apply them to current alsa cvs?

these are the modified files:

linux/sound/core/oss/pcm_oss.c.orig
linux/sound/core/oss/mixer_oss.c.orig
linux/sound/core/pcm_lib.c.orig
linux/sound/core/control.c.orig
linux/sound/core/seq/oss/seq_oss.c.orig
linux/sound/core/seq/seq_clientmgr.c.orig
linux/sound/core/hwdep.c.orig
linux/sound/core/pcm_native.c.orig
linux/sound/core/timer.c.orig
linux/sound/core/info.c.orig
linux/sound/core/rawmidi.c.orig

cvs alsa should work just fine, but to get optimal results you might want to try to apply the above changes (all linux/sound/ changes) to alsa-cvs.

Ingo

Fernando Lopez-Lezcano

2004-12-14 23:28 UTC #865 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Quoted 14 lines · Ingo Molnar

i have released the -V0.7.33-0 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
- due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode, but there's one interesting new feature: -rc3-mm1 introduced the ->unlocked_ioctl method which is now an official way to do BKL-less ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this facility. The ALSA/sound guys might be interested in these bits. Thus another chunk of -RT could go upstream.

Quoted 4 lines

- IO-APIC/MSI fix from Steven Rostedt.
- fixed a tracer bug which would produce a kernel warning and an empty /proc/latency_trace if the trace buffer overflows.

I don't know which change did it, but I have network connectivity in my athlon64 test box with 0.7.33-0! Woohoo! [*]
Thanks...
-- Fernando

[*] as reported before: going up to runlevel 5 killed the network, up to leve 3 everything was fine, I was guessing probably due to some problem with X and the radeon driver.

Florian Schmidt

2004-12-15 00:28 UTC #866 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 5 lines · On 14 Dec 2004 15:21:56 -0800 Fernando Lopez-Lezcano…

I don't know which change did it, but I have network connectivity in my athlon64 test box with 0.7.33-0! Woohoo! [*]
Thanks...
-- Fernando

I thought i'd just chime in. 33-0 runs excellent here (i have no debugging or timing enabled, but xrun performance is really good (i have seen none but app caused ones (only mild load though, will push it a bit harder though tomorrow))..

Flo

Signature · 2 lines

Palimm Palimm!
http://affenbande.org/~tapas/

Lee Revell

2004-12-15 01:19 UTC #867 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Fernando Lopez-Lezcano

Quoted 2 lines · Fernando Lopez-Lezcano

I don't know which change did it, but I have network connectivity in my athlon64 test box with 0.7.33-0! Woohoo! [*]

Wait, this works on x84-64 now? There was a recent report on LAU that it didn't compile.

Lee

Fernando Lopez-Lezcano

2004-12-15 02:05 UTC #868 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 6 lines · Lee Revell

Quoted 2 lines · Fernando Lopez-Lezcano
I don't know which change did it, but I have network connectivity in my athlon64 test box with 0.7.33-0! Woohoo! [*]
Wait, this works on x84-64 now? There was a recent report on LAU that it didn't compile.

The machine has an athlon64 but it is running 32 bit fc2. I have not tried to build (yet) on 64 bit fcx.

-- Fernando

Ingo Molnar

2004-12-15 09:09 UTC #869 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 10 lines · Fernando Lopez-Lezcano

Quoted 6 lines · Lee Revell
Quoted 2 lines · Fernando Lopez-Lezcano
I don't know which change did it, but I have network connectivity in my athlon64 test box with 0.7.33-0! Woohoo! [*]
Wait, this works on x84-64 now? There was a recent report on LAU that it didn't compile.
The machine has an athlon64 but it is running 32 bit fc2. I have not tried to build (yet) on 64 bit fcx.

x64 wont work for now, it needs some work to make threaded timer IRQs work.

Ingo

Andrew Walrond

2004-12-15 10:18 UTC #870 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

Quoted 4 lines · Ingo Molnar

x64 wont work for now, it needs some work to make threaded timer IRQs work.

Don't leave it too long; I'm pretty sure I'm not the only x64 user wanting to test your -RT stuff.

I'm beginning to develop 32bit-envy ;)

Andrew Walrond

Lee Revell

2004-12-15 16:51 UTC #871 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Quoted 2 lines · Ingo Molnar

x64 wont work for now, it needs some work to make threaded timer IRQs work.

But the timer IRQ can only be threaded with PREEMPT_RT, right? Seems like PREEMPT_DESKTOP should work. Older VP patches worked on x64 (S7, T3) IIRC.

Lee

Fernando Lopez-Lezcano

2004-12-17 00:49 UTC #872 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Fernando Lopez-Lezcano

Quoted 15 lines · Fernando Lopez-Lezcano

Quoted 14 lines · Ingo Molnar
i have released the -V0.7.33-0 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
- due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode, but there's one interesting new feature: -rc3-mm1 introduced the ->unlocked_ioctl method which is now an official way to do BKL-less ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this facility. The ALSA/sound guys might be interested in these bits. Thus another chunk of -RT could go upstream.

Quoted 7 lines

Quoted 4 lines
- IO-APIC/MSI fix from Steven Rostedt.
- fixed a tracer bug which would produce a kernel warning and an empty /proc/latency_trace if the trace buffer overflows.
I don't know which change did it, but I have network connectivity in my athlon64 test box with 0.7.33-0! Woohoo! [*]

But I'm still having problems with a dual athlon mp machine while trying to boot 0.7.33-04 (same as before - this has not booted for quite a while with the realtime patches). This is what I see:

pnp: 00:02: ioport range 0xe400-0xe4fe could not be reserved

BUG: scheduling while atomic: swapper/0x10000000/1
caller is preempt_schedule_irq+0x34/0x50
 [<c021053c3>] dump_stack+0x23/-x30 (20)
 [<c037438c>] __schedule+0xd5c/0xdb0 (124)
 [<c037432b>] preempt_schedul_irq+0x34/0x50 (12)
 [<c010432b>] need_resched+0x20/0x29 (-7416)
---------------------------

| preempt count: 10000001 ]
| 1-level deep critical section nesting:
---------------------------------------
.. [<c01472ed>] .... print_traces+0x1d/0x60
.....[<c01053c3>] .. ( <= dump_stack+0x23/0x30)

BUG at kernel/latency.c:1292! --------------------[ cut here ]--------------------- kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

=================
Booting with:
hardirq_preempt=0
softirq_preempt=0

ide1 at 0x170-0x177,0x376 on irq 15

BUG: scheduling while atomic: swapper/0x10000000/1
caller is preempt_schedule_irq+0x34/0x50
 [<>] dump_stack=0x23/0x30 (20)
 [<>] __schedule+0x5dc/0xdb0 (124)
 [<>] preempt_sched_irq+0x34/0x50 (12)
 [<>] need_resched+0x20/0x29 (-7376)
---------------------------

BUG at kernel/latency.c:1292! --------------------[ cut here ]--------------------- kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

=============
Booting with:
acpi=off

usbcore: registered new driver hub
BUG scheduling while atomic: swapper......
....
kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

========
hardirq_preempt=0
softirq_preempt=0
acpi=off

usbcore: registered new driver hub
BUG scheduling while atomic: swapper......
....
kernel BUG at kernel/latency.c:1292!
invalid operand: 0000 [#1]
PREEMPT_SMP

=============
Booting with:
hardirq_preempt=0
softirq_preempt=0
kernel_preempt=0
acpi=off

Same........
Suggestions?
-- Fernando

Magnus Määttä

2004-12-15 20:48 UTC #873 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Hi Ingo,

Quoted 14 lines · Ingo Molnar

i have released the -V0.7.33-0 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a port from -rc2-mm3 to -rc3-mm1. Changes:
- due to 2.6.10 release work the -mm kernel now is in fixes-mostly mode, but there's one interesting new feature: -rc3-mm1 introduced the ->unlocked_ioctl method which is now an official way to do BKL-less ioctls. I changed the ALSA ->ioctl_bkl changes in -RT to use this facility. The ALSA/sound guys might be interested in these bits. Thus another chunk of -RT could go upstream.

Quoted 4 lines

- IO-APIC/MSI fix from Steven Rostedt.
- fixed a tracer bug which would produce a kernel warning and an empty /proc/latency_trace if the trace buffer overflows.

I got this deadlock with V0.7.33-2:

==========================================
[ BUG: lock recursion deadlock detected! |
------------------------------------------
already locked: [c04d63e0] {rtc_task_lock}
.. held by: IRQ 8: 305 [df45c030, 52]
... acquired at: rtc_interrupt+0x137/0x230

------------------------------
| showing all locks held by: | (IRQ 8/305 [df45c030, 52]):
------------------------------

#001: [c04d63e0] {rtc_task_lock}
... acquired at: rtc_interrupt+0x137/0x230

#002: [df4f16f8] {&timer->lock}
... acquired at: _snd_timer_stop+0x8f/0x180

-{current task's backtrace}----------------->

 [<c0104273>] dump_stack+0x23/0x30 (20)
 [<c013d011>] check_deadlock+0x141/0x2d0 (48)
 [<c013dc1a>] task_blocks_on_lock+0x7a/0x100 (36)
 [<c041a847>] __down_mutex+0x237/0x440 (88)
 [<c013f86b>] __spin_lock+0x4b/0x60 (24)
 [<c013f8dd>] _spin_lock_irq+0x1d/0x20 (16)
 [<c02b60d9>] rtc_control+0x29/0x80 (32)
 [<c034efef>] rtctimer_stop+0x2f/0x70 (28)
 [<c034c8fc>] snd_timer_interrupt+0x2fc/0x380 (64)
 [<c034f057>] rtctimer_interrupt+0x27/0x40 (20)
 [<c02b4f08>] rtc_interrupt+0x158/0x230 (32)
 [<c014f40a>] handle_IRQ_event+0x6a/0xf0 (52)
 [<c014fdcb>] do_hardirq+0xab/0x130 (40)
 [<c014fec0>] do_irqd+0x70/0xb0 (32)
 [<c013bcc6>] kthread+0xa6/0xf0 (48)
 [<c0101365>] kernel_thread_helper+0x5/0x10 (548970516)
---------------------------

| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c041a94a>] .... __down_mutex+0x33a/0x440
.....[<c013f86b>] .. ( <= __spin_lock+0x4b/0x60)
.. [<c01435bb>] .... print_traces+0x1b/0x50
.....[<c0104273>] .. ( <= dump_stack+0x23/0x30)

showing all tasks:

s            init:    1 [c1467970, 116] (not blocked)
s           IRQ 0:    2 [c1467320,  50] (not blocked)
s     ksoftirqd/0:    3 [c1466cd0, 105] (not blocked)
s       desched/0:    4 [c1466680, 105] (not blocked)
s        events/0:    5 [c1466030,  98] (not blocked)
s         khelper:    6 [df687970, 111] (not blocked)
s         kthread:   11 [df687320, 110] (not blocked)
s          kacpid:   20 [df686cd0, 110] (not blocked)
s           IRQ 9:   21 [df686680,  51] (not blocked)
s       kblockd/0:  117 [c15d4680, 110] (not blocked)
s           khubd:  125 [c15e9320, 115] (not blocked)
s         pdflush:  226 [c15e8680, 117] (not blocked)
s         pdflush:  227 [c15e9970, 116] (not blocked)
s         kswapd0:  228 [c15d4030, 117] (not blocked)
s           aio/0:  229 [df686030, 112] (not blocked)
s           IRQ 8:  305 [df45c030,  52] (not blocked)
s         kseriod:  309 [df45c680, 117] (not blocked)
s          IRQ 12:  315 [df45ccd0,  53] (not blocked)
s          IRQ 14:  405 [df45d320,  54] (not blocked)
s          IRQ 15:  407 [df46b970,  55] (not blocked)
s        khpsbpkt:  420 [df46b320, 115] blocked on: [c04ddfc4] {khpsbpkt_sig.lock}
.. held by:          khpsbpkt:  420 [df46b320, 115]
... acquired at:  dma_trm_tasklet+0x89/0x1a0
s          IRQ 10:  430 [c15d5970,  56] (not blocked)
s     knodemgrd_0:  431 [c15e8cd0, 116] blocked on: [df49bba8] {&hi->reset_sem}
.. held by:       knodemgrd_0:  431 [c15e8cd0, 116]
... acquired at:  highlevel_host_reset+0x4b/0x60
s          IRQ 11:  433 [df45d970,  57] (not blocked)
s           IRQ 1:  451 [c15e8030,  58] (not blocked)
s        krxtimod:  481 [c15fd970, 118] (not blocked)
s          krxiod:  482 [c15d4cd0, 117] (not blocked)
s         krxsecd:  483 [c15fd320, 117] (not blocked)
s       kafstimod:  487 [c15d5320, 120] (not blocked)
s      kafsasyncd:  488 [df46a680, 120] (not blocked)
s      reiserfs/0:  489 [df46acd0, 111] (not blocked)
s           udevd:  580 [de854030, 113] (not blocked)
s           IRQ 6: 4119 [dde00030,  59] (not blocked)
s       ipw2200/0: 4284 [dde01320, 110] (not blocked)
s       syslog-ng: 7503 [dcb30cd0, 115] (not blocked)
s             gpm: 7629 [dcb26cd0, 117] (not blocked)
s         portmap: 8374 [c15fc680, 116] (not blocked)
s        rpciod/0: 8386 [deefb970, 111] (not blocked)
s           lockd: 8387 [dde00cd0, 117] (not blocked)
s            sshd: 8439 [ddbbc680, 120] (not blocked)
s            cron: 8481 [ddbbc030, 116] (not blocked)
s           login: 8502 [c15fccd0, 117] (not blocked)
s          agetty: 8503 [ddb8e680, 116] (not blocked)
s          agetty: 8504 [ddbbd970, 116] (not blocked)
s          agetty: 8505 [deefa680, 116] (not blocked)
s          agetty: 8506 [dcb31320, 116] (not blocked)
s          agetty: 8507 [ddbbccd0, 116] (not blocked)
s            bash: 8588 [de855970, 117] (not blocked)
s             kdm: 8653 [deefb320, 116] (not blocked)

? X: 8656 [deefacd0, 116] (not blocked)
s kdm: 8657 [c15fc030, 117] (not blocked)
s startkde: 8681 [ddbbd320, 118] (not blocked)
s kdeinit: 8714 [dde01970, 116] (not blocked)
s kdeinit: 8717 [de854680, 115] (not blocked)
s kdeinit: 8719 [dcb26030, 116] (not blocked)
s kdeinit: 8722 [df46a030, 116] (not blocked)
s kdeinit: 8731 [de855320, 116] (not blocked)
s kdeinit: 8735 [ddb8f970, 115] (not blocked)
s kwrapper: 8736 [dde00680, 116] (not blocked)
s kdeinit: 8738 [ddb8ecd0, 116] (not blocked)
s kdeinit: 8739 [deefa030, 116] (not blocked)
s kdeinit: 8741 [dcb30030, 115] (not blocked)
s kdeinit: 8743 [dcb27320, 115] (not blocked)
s kdeinit: 8745 [dcb30680, 116] (not blocked)
s kdeinit: 8746 [dcb27970, 116] (not blocked)
s kdeinit: 8748 [dcb26680, 115] (not blocked)
s kdeinit: 8752 [ddb8f320, 115] (not blocked)
s korgac: 8753 [dcb31970, 116] (not blocked)
s kalarmd: 8755 [d7398cd0, 119] (not blocked)
s kdeinit: 8756 [ddb8e030, 116] (not blocked)
s bash: 8757 [de854cd0, 117] (not blocked)
s firefox: 8765 [d7399320, 117] (not blocked)
s firefox-bin: 8778 [d7399970, 115] (not blocked)
s firefox-bin: 8782 [d5c53970, 116] (not blocked)
s firefox-bin: 8784 [d7398680, 116] (not blocked)
X netstat: 8790 [d5c52cd0, 118] (not blocked)
s ssh: 8811 [d5c53320, 117] (not blocked)
s bash: 8812 [d5c52680, 117] (not blocked)
s bash: 8820 [d5c52030, 117] (not blocked)
s bash: 8828 [d7398030, 117] (not blocked)
s bash: 8836 [d3161970, 117] (not blocked)
s bash: 8844 [d3160cd0, 116] (not blocked)
s bash: 8852 [d3161320, 116] (not blocked)
s bash: 8860 [d3160680, 116] (not blocked)
s bash: 8868 [d3160030, 116] (not blocked)
s ssh: 8876 [d328d320, 117] (not blocked)
s ssh: 8877 [d328c030, 117] (not blocked)
s bash: 8880 [d328c680, 119] (not blocked)
s jackd: 8881 [d328ccd0, 120] (not blocked)
s jackd: 8882 [d328d970, 116] (not blocked)
s jackd: 8883 [d1f6f970, 0] (not blocked)
s jackd: 8884 [d1f6f320, 9] (not blocked)
s qsynth: 8887 [d1f6ecd0, 116] (not blocked)
s qsynth: 8888 [c8095970, 10] (not blocked)
? qsynth: 8889 [c8095320, 116] (not blocked)
s ksnapshot: 8890 [d1f6e030, 116] (not blocked)
s ladccad: 8908 [c8094cd0, 116] (not blocked)
s ladccad: 8910 [c8094680, 10] (not blocked)
s ladccad: 8911 [d1f6e680, 116] (not blocked)
s ladccad: 8912 [c5769970, 116] (not blocked)
s ladccad: 8919 [c5769320, 116] (not blocked)
s ladccad: 8920 [c5768cd0, 123] (not blocked)
s ladccad: 8909 [c8094030, 120] (not blocked)
D rosegarden: 8923 [c5768030, 116] blocked on: [c3f12418] {&q->lock}
.. held by: X: 8656 [deefacd0, 116]
... acquired at: __wake_up+0x51/0x60
R rosegardenseque: 8924 [c5768680, 116] (not blocked) s rosegardenseque: 8925 [c30af970, 115] (not blocked) s rosegardenseque: 8926 [c30af320, 10] (not blocked)

---------------------------
| showing all locks held: |
---------------------------

#001: [c062af2c] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#002: [c062ab50] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#003: [c062ad24] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#004: [c062b574] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#005: [c062b198] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#006: [c062b36c] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#007: [c062bbbc] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#008: [c062b7e0] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#009: [c062b9b4] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#010: [c062c204] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#011: [c062be28] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#012: [c062bffc] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#013: [c062c84c] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#014: [c062c470] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#015: [c062c644] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#016: [c062ce94] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#017: [c062cab8] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#018: [c062cc8c] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#019: [c062d4dc] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#020: [c062d100] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#021: [c062d2d4] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#022: [c062db24] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#023: [c062d748] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#024: [c062d91c] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#025: [c062e16c] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#026: [c062dd90] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#027: [c062df64] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#028: [c062e7b4] {&hwif->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x9d/0x180

#029: [c062e3d8] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#030: [c062e5ac] {&drive->gendev_rel_sem}
.. held by: init: 1 [c1467970, 116]
... acquired at: init_hwif_data+0x16f/0x180

#031: [ded6fcf0] {&tty->atomic_read}
.. held by: agetty: 8503 [ddb8e680, 116]
... acquired at: read_chan+0x477/0x7c0

#032: [dedb9cf0] {&tty->atomic_read}
.. held by: agetty: 8505 [deefa680, 116]
... acquired at: read_chan+0x477/0x7c0

#033: [dd710cf0] {&tty->atomic_read}
.. held by: agetty: 8506 [dcb31320, 116]
... acquired at: read_chan+0x477/0x7c0

#034: [dca2acf0] {&tty->atomic_read}
.. held by: agetty: 8507 [ddbbccd0, 116]
... acquired at: read_chan+0x477/0x7c0

#035: [dd3bfcf0] {&tty->atomic_read}
.. held by: agetty: 8504 [ddbbd970, 116]
... acquired at: read_chan+0x477/0x7c0

#036: [c1773cf0] {&tty->atomic_read}
.. held by: bash: 8588 [de855970, 117]
... acquired at: read_chan+0x477/0x7c0

#037: [d35decf0] {&tty->atomic_read}
.. held by: bash: 8868 [d3160030, 116]
... acquired at: read_chan+0x477/0x7c0

#038: [d30c4cf0] {&tty->atomic_read}
.. held by: bash: 8860 [d3160680, 116]
... acquired at: read_chan+0x477/0x7c0

#039: [d306fcf0] {&tty->atomic_read}
.. held by: bash: 8852 [d3161320, 116]
... acquired at: read_chan+0x477/0x7c0

#040: [d3ba3cf0] {&tty->atomic_read}
.. held by: bash: 8844 [d3160cd0, 116]
... acquired at: read_chan+0x477/0x7c0

#041: [c3f12418] {&q->lock}
.. held by: X: 8656 [deefacd0, 116]
... acquired at: __wake_up+0x51/0x60

#042: [c3f135d8] {&sk->sk_callback_lock}
.. held by: rosegarden: 8923 [c5768030, 116]
... acquired at: sock_def_readable+0x25/0xb0

#043: [d125631c] {&q->lock}
.. held by: qsynth: 8889 [c8095320, 116]
... acquired at: __wake_up+0x51/0x60

#044: [c04d63e0] {rtc_task_lock}
.. held by: IRQ 8: 305 [df45c030, 52]
... acquired at: rtc_interrupt+0x137/0x230

#045: [df4f16f8] {&timer->lock}
.. held by: IRQ 8: 305 [df45c030, 52]
... acquired at: _snd_timer_stop+0x8f/0x180
=============================================

[ turning off deadlock detection. Please report this trace. ]

Regards,
Magnus Määttä

Signature · 1 line

Pushing 40 is exercise enough.

Bill Huey (hui)

2005-01-04 06:41 UTC #874 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc3-mm1-V0.7.33-0

In reply to Ingo Molnar

Quoted 8 lines · Ingo Molnar

to create a -V0.7.33-0 tree from scratch, the patching order is:
http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc3.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc3/2.6.10-rc3-mm1/2.6.10-rc3-mm1.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc3-mm1-V0.7.33-0

Forward port of this patch to 2.6.10-mm1 here:

	http://people.lynuxworks.com/~bhuey/realtime/Ingo_forward_port-mm1-V0.7.33-04

Obviously, you'll need to patch a plain 2.6.10 with -mm1 from Andrew Morton, but folks should be able to do this by now. ;)

You'll have to apply Ingo's patch so that it gets rejects and then apply this patch on top of it so that it resolves those issues. It's a bit sloppy, but this'll at least be somewhat workable until Ingo comes back and pounds us with patches. :)

bill

Ingo Molnar

2005-01-04 09:46 UTC #875 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.10-mm1-V0.7.34-00

Quoted 4 lines · Bill Huey

You'll have to apply Ingo's patch so that it gets rejects and then apply this patch on top of it so that it resolves those issues. It's a bit sloppy, but this'll at least be somewhat workable until Ingo comes back and pounds us with patches. :)

i've uploaded my port:

http://redhat.com/~mingo/realtime-preempt/

this is mainly a straight port from 2.6.10-rc3-mm1 to 2.6.10-mm1, plus i picked up a post-2.6.10-mm1 audio-driver buildsystem fix-patch. Please (re-)report any new or old regressions.

to create a -V0.7.34-00 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10/2.6.10-mm1/2.6.10-mm1.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-mm1-V0.7.34-00

Ingo

Bill Huey (hui)

2005-01-04 10:49 UTC #876 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-mm1-V0.7.34-00

Quoted 3 lines · Ingo Molnar

this is mainly a straight port from 2.6.10-rc3-mm1 to 2.6.10-mm1, plus i picked up a post-2.6.10-mm1 audio-driver buildsystem fix-patch. Please (re-)report any new or old regressions.

Build failure.

LD arch/i386/kernel/cpu/mcheck/built-in.o
kernel/time.c: In function `sys_gettimeofday':
kernel/time.c:164: error: parse error before ')' token distcc[2235] ERROR: compile kernel/time.c on localhost failed make[1]: *** [kernel/time.o] Error 1
make[1]: *** Waiting for unfinished jobs....
CC mm/truncate.o

bill

Ingo Molnar

2005-01-04 10:52 UTC #877 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-mm1-V0.7.34-00

Quoted 6 lines · Bill Huey

Quoted 3 lines · Ingo Molnar
this is mainly a straight port from 2.6.10-rc3-mm1 to 2.6.10-mm1, plus i picked up a post-2.6.10-mm1 audio-driver buildsystem fix-patch. Please (re-)report any new or old regressions.
Build failure.

does -01 work better? There was some leftover debugging stuff that got generated into the patch by mistake.

Ingo

Felipe Alfaro Solana

2005-01-04 10:55 UTC #878 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-mm1-V0.7.34-00

In reply to Ingo Molnar

Quoted 16 lines · Ingo Molnar

Quoted 4 lines · Bill Huey
You'll have to apply Ingo's patch so that it gets rejects and then apply this patch on top of it so that it resolves those issues. It's a bit sloppy, but this'll at least be somewhat workable until Ingo comes back and pounds us with patches. :)
i've uploaded my port:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a straight port from 2.6.10-rc3-mm1 to 2.6.10-mm1, plus i picked up a post-2.6.10-mm1 audio-driver buildsystem fix-patch. Please (re-)report any new or old regressions.

Quoted 8 lines

to create a -V0.7.34-00 tree from scratch, the patching order is:
http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10/2.6.10-mm1/2.6.10-mm1.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-mm1-V0.7.34-00

  AS [M]  arch/i386/crypto/aes-i586-asm.o
  CC [M]  arch/i386/crypto/aes.o
  LD [M]  arch/i386/crypto/aes-i586.o
  CC      kernel/sched.o
  CC      kernel/fork.o
  CC      kernel/exec_domain.o
  CC      kernel/panic.o
  CC      kernel/printk.o
  CC      kernel/profile.o
  CC      kernel/exit.o
  CC      kernel/itimer.o
  CC      kernel/time.o

kernel/time.c: In function `sys_gettimeofday':
kernel/time.c:164: error: syntax error before ')' token make[3]: *** [kernel/time.o] Error 1
make[2]: *** [kernel] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.70899 (%build)

kernel/time.c renamed from kernel/time.c.old +1 -1

@@ -161,7 +161,7 @@
 " sti; cli; mov %ax,%ss; mov %ss,%ax; mov %ax,%ss; "
 " cli; mov %ax,%ss; "
 " cli; mov %ax,%ss; cli; "
-" mov %ax,%ss; mov %ax,%ss; sti; sti; sti; sti");
+" mov %ax,%ss; mov %ax,%ss; sti; sti; sti; sti";
 #ifdef CONFIG_LATENCY_TRACE
 	if (!tv && ((long)tz == 1))
 		return user_trace_start();

Florian Schmidt

2005-01-06 18:49 UTC #879 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-mm1-V0.7.34-00

In reply to Ingo Molnar

Quoted 16 lines · On Tue, 4 Jan 2005 10:45:18 +0100 Ingo Molnar <[email protected]> wrote:

Quoted 4 lines · Bill Huey
You'll have to apply Ingo's patch so that it gets rejects and then apply this patch on top of it so that it resolves those issues. It's a bit sloppy, but this'll at least be somewhat workable until Ingo comes back and pounds us with patches. :)
i've uploaded my port:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a straight port from 2.6.10-rc3-mm1 to 2.6.10-mm1, plus i picked up a post-2.6.10-mm1 audio-driver buildsystem fix-patch. Please (re-)report any new or old regressions.

Quoted 5 lines

to create a -V0.7.34-00 tree from scratch, the patching order is:
http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10/2.6.10-mm1/2.6.10-mm1.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-mm1-V0.7.34-00

Hi,

something is wrong with 34-01 and ALSA:

make modules_install gives me:

if [ -r System.map ]; then /sbin/depmod -ae -F System.map 2.6.10-mm1-V0.7.34-01; fi
WARNING: /lib/modules/2.6.10-mm1-V0.7.34-01/kernel/sound/pci/ac97/snd-ac97-codec.ko needs unknown symbol snd_ac97_restore_status
WARNING: /lib/modules/2.6.10-mm1-V0.7.34-01/kernel/sound/pci/ac97/snd-ac97-codec.ko needs unknown symbol snd_ac97_restore_iec958

and consequently after booting into the kernel modprobe'ing the alsa modules fails:

~/tmp/src/linux-2.6.10-realtime-preempt-2.6.10-mm1-V0.7.34-01$ sudo modprobe snd_cs46xx
WARNING: Error inserting snd_ac97_codec (/lib/modules/2.6.10-mm1-V0.7.34-01/kernel/sound/pci/ac97/snd-ac97-codec.ko): Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting snd_cs46xx (/lib/modules/2.6.10-mm1-V0.7.34-01/kernel/sound/pci/cs46xx/snd-cs46xx.ko): Unknown symbol in module, or unknown parameter (see dmesg)

from dmesg:

snd_ac97_codec: Unknown symbol snd_ac97_restore_status
snd_ac97_codec: Unknown symbol snd_ac97_restore_iec958
snd_cs46xx: Unknown symbol snd_ac97_write_cache
snd_cs46xx: Unknown symbol snd_ac97_mixer
snd_cs46xx: Unknown symbol snd_ac97_bus
snd_cs46xx: Unknown symbol snd_ac97_write
snd_cs46xx: Unknown symbol snd_ac97_read

.config attached

Flo

Signature · 2 lines

Palimm Palimm!
http://affenbande.org/~tapas/

Tom Rini

2005-01-07 19:30 UTC #880 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-mm1-V0.7.34-00

In reply to Ingo Molnar

Quoted 8 lines · Ingo Molnar

Quoted 4 lines · Bill Huey
You'll have to apply Ingo's patch so that it gets rejects and then apply this patch on top of it so that it resolves those issues. It's a bit sloppy, but this'll at least be somewhat workable until Ingo comes back and pounds us with patches. :)

Quoted 7 lines

i've uploaded my port:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a straight port from 2.6.10-rc3-mm1 to 2.6.10-mm1, plus i picked up a post-2.6.10-mm1 audio-driver buildsystem fix-patch. Please (re-)report any new or old regressions.

Here's a handful of little things to fix issues in the patch for when you try and use the patchset on an architecture that doesn't (yet) work.

- cycles_t is defined in <asm/timex.h>, but that wasn't part of <linux/irq.h> previously (breaks ppc32, I _think_).
- debug_direct_keyboard & such only exist on GENERIC_HARDIRQ arches (and I would guess make sense on ones you have a console & !USB keyboard on).
- The stubs for init_hardirqs / early_init_hardirqs were in a conflicting state (as seen on ppc32, which is GENERIC_HARDIRQ, but not yet supported). I think this gets them down to what was intended.

Signed-off-by Tom Rini <[email protected]>

include/linux/irq.h +5 -1

@@ -21,6 +21,7 @@
 
 #include <asm/irq.h>
 #include <asm/ptrace.h>
+#include <asm/timex.h>
 
 /*
  * IRQ line status.

@@ -100,11 +101,14 @@ extern void report_bad_irq(unsigned int 
 extern int can_request_irq(unsigned int irq, unsigned long irqflags);
 
 extern void early_init_hardirqs(void);
-extern void init_hardirqs(void);
 extern void init_irq_proc(void);
 extern cycles_t irq_timestamp(unsigned int irq);
 #else
 static inline void early_init_hardirqs(void) { }
+#endif
+#if defined(CONFIG_GENERIC_HARDIRQS) && defined(CONFIG_PREEMPT_HARDIRQS)
+extern void init_hardirqs(void);
+#else
 static inline void init_hardirqs(void) { }
 #endif

include/linux/sched.h +2 -0

@@ -47,7 +47,9 @@ extern void direct_timer_interrupt(struc
 extern struct semaphore kernel_sem;
 #endif
 
+#ifdef CONFIG_GENERIC_HARDIRQS
 extern int debug_direct_keyboard;
+#endif
 
 #ifdef CONFIG_RT_DEADLOCK_DETECT
   extern void deadlock_trace_off(void);

kernel/irq/manage.c +0 -6

@@ -527,10 +527,6 @@ void __init init_hardirqs(void)
 
 #else
 
-void __init init_hardirqs(void)
-{
-}
-
 static int start_irq_thread(int irq, struct irq_desc *desc)
 {
 	return 0;

@@ -545,5 +541,3 @@ void __init early_init_hardirqs(void)
 	for (i = 0; i < NR_IRQS; i++)
 		init_waitqueue_head(&irq_desc[i].wait_for_handler);
 }
-
-

kernel/sched.c +2 -0

@@ -5409,8 +5409,10 @@ void __might_sleep(char *file, int line)
 
 	if ((in_atomic() || irqs_disabled()) &&
 	    system_state == SYSTEM_RUNNING) {
+#ifdef CONFIG_GENERIC_HARDIRQS
 		if (debug_direct_keyboard && hardirq_count())
 			return;
+#endif
 		if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
 			return;
 		prev_jiffy = jiffies;

kernel/sysctl.c +2 -0

@@ -391,6 +391,7 @@ static ctl_table kern_table[] = {
 		.proc_handler	= &proc_dointvec,
 	},
 #endif
+#ifdef CONFIG_GENERIC_HARDIRQS
 	{
 		.ctl_name	= KERN_PANIC,
 		.procname	= "debug_direct_keyboard",

@@ -399,6 +400,7 @@ static ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= &proc_dointvec,
 	},
+#endif
 	{
 		.ctl_name	= KERN_CORE_USES_PID,
 		.procname	= "core_uses_pid",

include/linux/rt_lock.h +2 -1

@@ -3,7 +3,6 @@
 
 #include <linux/config.h>
 #include <linux/list.h>
-#include <asm/atomic.h>
 
 /*
  * These are the basic SMP spinlocks, allowing only a single CPU anywhere.

@@ -26,6 +25,8 @@ typedef struct {
 # define RAW_SPIN_LOCK_UNLOCKED (raw_spinlock_t) __RAW_SPIN_LOCK_UNLOCKED
 #endif
 
+#include <asm/atomic.h>
+
 /*
  * Read-write spinlocks, allowing multiple readers
  * but only one writer.

include/linux/rt_lock.h +1 -2

@@ -25,8 +25,6 @@ typedef struct {
 # define RAW_SPIN_LOCK_UNLOCKED (raw_spinlock_t) __RAW_SPIN_LOCK_UNLOCKED
 #endif
 
-#include <asm/atomic.h>
-
 /*
  * Read-write spinlocks, allowing multiple readers
  * but only one writer.

@@ -173,6 +171,7 @@ typedef struct {
 
 
 #ifdef CONFIG_PREEMPT_RT
+#include <asm/atomic.h>
 
 /*
  * semaphores - an RT-mutex plus the semaphore count:

Signature · 2 lines

Tom Rini
http://gate.crashing.org/~trini/

Lee Revell

2005-01-07 19:40 UTC #881 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-mm1-V0.7.34-00

Quoted 2 lines · Tom Rini

Here's a handful of little things to fix issues in the patch for when you try and use the patchset on an architecture that doesn't (yet) work.

Speaking of which, I guess no one ever stepped up & fixed the patch for x86_64? AFAICT it should not be too hard to get PREEMPT_DESKTOP working, as the previous VP patch sets worked, then the timer interrupt threading changes broke it. But you can't thread the timer irq with PREEMPT_DESKTOP anyway.

Unfortunately I can only offer moral support as I don't have the hardware...

Lee

Ingo Molnar

2005-01-26 08:10 UTC #882 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-04

In reply to Tom Rini

Quoted 12 lines · Tom Rini

Here's a handful of little things to fix issues in the patch for when you try and use the patchset on an architecture that doesn't (yet) work.
- cycles_t is defined in <asm/timex.h>, but that wasn't part of <linux/irq.h> previously (breaks ppc32, I _think_).
- debug_direct_keyboard & such only exist on GENERIC_HARDIRQ arches (and I would guess make sense on ones you have a console & !USB keyboard on).
- The stubs for init_hardirqs / early_init_hardirqs were in a conflicting state (as seen on ppc32, which is GENERIC_HARDIRQ, but not yet supported). I think this gets them down to what was intended.

Quoted 1 line

Signed-off-by Tom Rini <[email protected]>

thanks - i have applied all of these and have released the -2.6.11-rc2-V0.7.36-04 patch which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

The -04 patch should also fix the down_write_interruptible() related build error reported by Lee Revell and others.

Ingo

Lee Revell

2005-02-01 20:01 UTC #883 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-04

Quoted 8 lines · Ingo Molnar

thanks - i have applied all of these and have released the -2.6.11-rc2-V0.7.36-04 patch which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
The -04 patch should also fix the down_write_interruptible() related build error reported by Lee Revell and others.

Assuming it's still available, what is the config option to get the "User-space atomicity debugging" feature? This feature is extremely useful for debugging complex JACK clients, several Linux audio developers have asked me about it but I can't find the config option anymore.

Lee

Ingo Molnar

2005-02-01 20:17 UTC #884 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-04

Quoted 5 lines · Lee Revell

Assuming it's still available, what is the config option to get the "User-space atomicity debugging" feature? This feature is extremely useful for debugging complex JACK clients, several Linux audio developers have asked me about it but I can't find the config option anymore.

it's always-on in the -RT tree (it's a pretty low-overhead thing). I havent changed the mechanism so the jackd hacks from a couple of weeks ago should still work.

Ingo

Lee Revell

2005-02-01 20:31 UTC #885 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-04

Quoted 8 lines · Ingo Molnar

Quoted 5 lines · Lee Revell
Assuming it's still available, what is the config option to get the "User-space atomicity debugging" feature? This feature is extremely useful for debugging complex JACK clients, several Linux audio developers have asked me about it but I can't find the config option anymore.

Quoted 4 lines

it's always-on in the -RT tree (it's a pretty low-overhead thing). I havent changed the mechanism so the jackd hacks from a couple of weeks ago should still work.

OK. So for application triggered tracing you need LATENCY_TRACING enabled, as described here:

http://lkml.org/lkml/2004/10/29/312

Lee

Ingo Molnar

2005-02-01 20:44 UTC #886 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-04

Quoted 4 lines · Lee Revell

OK. So for application triggered tracing you need LATENCY_TRACING enabled, as described here:
http://lkml.org/lkml/2004/10/29/312

correct, that too should still work fine - with the small change that there's now a separate flag to active it:

echo 1 > /proc/sys/kernel/trace_user_triggered # default: 0

it is an orthogonal mechanism to atomicity-debugging.

since i wrote the above mail 3 months ago, a number of improvements have been done to the tracer. There are a handful of modifier feature-flags to the tracer, which can be used to get additional functionality. Here's a quick summary:

echo 1 > /proc/sys/kernel/trace_freerunning # default: 0

will get a 'freerunning' tracer which never stops (and overwrites the oldest entries if the trace gets full). Especially with long latencies this in some cases can be more informative.

this flag:

echo 0 > /proc/sys/kernel/mcount_enabled # default: 1

causes the tracer to record only key kernel events (schedule/wakeup events, etc.), not every kernel function call. This might be useful if you want to see the bigger picture and want to validate scheduling logic on a bigger scale, spanning a much longer timeframe.

and if you have stability problems, this flag might be handy:

echo 1 > /proc/sys/kernel/trace_freerunning # default: 0

it will dump the current kernel trace to the kernel console if a crash happens - obviously only useful with a serial console or netconsole. It's a big dump but can make some bugs much easier to debug.

on SMP:

echo 1 > /proc/sys/kernel/trace_all_cpus # default: 0

this flag will cause all activity from all CPUs to be included in the trace. This can be useful if it is suspected that a particular latency was caused not by the CPU where the latency triggers, but by some other CPU.

Ingo

Lee Revell

2005-02-01 23:45 UTC #887 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-04

Quoted 5 lines · Ingo Molnar

Quoted 3 lines · Lee Revell
OK. So for application triggered tracing you need LATENCY_TRACING enabled, as described here:

Quoted 8 lines

Quoted 1 line
http://lkml.org/lkml/2004/10/29/312
correct, that too should still work fine - with the small change that there's now a separate flag to active it:
echo 1 > /proc/sys/kernel/trace_user_triggered # default: 0
it is an orthogonal mechanism to atomicity-debugging.

OK. Rereading my old mail, it looks like there were some possibly unresolved false positives with the userspace atomicity debugger.

Here's one I get from alsaplayer. Would more information be required to know if this is a false positive?

alsaplayer:5718 userspace BUG: scheduling in user-atomic context!

 [<c0102a97>] dump_stack+0x17/0x20 (20)
 [<c026268c>] schedule+0x6c/0x100 (24)
 [<c026330c>] rwsem_down_read_failed+0x9c/0x170 (48)
 [<c01277f5>] .text.lock.futex+0x7/0xb2 (44)
 [<c01276bf>] do_futex+0x4f/0x80 (28)
 [<c01277ba>] sys_futex+0xca/0xe0 (68)
 [<c0102457>] syscall_call+0x7/0xb (-4028)

(gdb) bt

#0  0x4117c4ec in __lll_mutex_unlock_wake () from /lib/tls/libpthread.so.0
#1  0x41179a69 in _L_mutex_unlock_26 () from /lib/tls/libpthread.so.0
#2  0x0824d3c0 in ?? ()
#3  0xb7ef3958 in ?? ()
#4  0x41179a60 in pthread_mutex_unlock () from /lib/tls/libpthread.so.0
#5  0x41179a60 in pthread_mutex_unlock () from /lib/tls/libpthread.so.0
#6  0x08057a30 in CorePlayer::Read32 (this=0x1, data=0xb7508008, samples=64) at CorePlayer.cpp:1076
#7  0x08057f89 in CorePlayer::streamer_func (arg=0x824d3c0, data=0x824cc80, size=128) at CorePlayer.cpp:1257
#8  0xb7ffcd52 in process (nframes=64, arg=0x824b258) at jack.cpp:350
#9  0xb7ef99f9 in jack_client_thread (arg=0x824bb48) at client.c:1264

#10 0x41177b63 in start_thread () from /lib/tls/libpthread.so.0 #11 0x410f0c4a in clone () from /lib/tls/libc.so.6

The backtrace is incomplete because I was unable to reproduce the problem with the debug glibc.

Lee

Ingo Molnar

2005-02-02 07:06 UTC #888 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-04

Quoted 5 lines · Lee Revell

OK. Rereading my old mail, it looks like there were some possibly unresolved false positives with the userspace atomicity debugger.
Here's one I get from alsaplayer. Would more information be required to know if this is a false positive?

this is a false positive that only triggers if PREEMPT_RT is disabled. Does the patch below fix it? (i've also updated the -37-03 patch that includes it.)

Ingo

lib/rwsem.c renamed from lib/rwsem.c.orig +5 -0

@@ -169,6 +169,8 @@ rwsem_down_failed_common(struct rw_semap
 
 	/* wait to be given the lock */
 	for (;;) {
+		unsigned long nosched_flag = current->flags & PF_NOSCHED;
+	
 		if ((sleep_state == TASK_INTERRUPTIBLE) &&
 						signal_pending(current)) {
 			spin_lock(&sem->wait_lock);

@@ -181,7 +183,10 @@ rwsem_down_failed_common(struct rw_semap
 		}
 		if (!waiter->task)
 			break;
+
+		current->flags &= ~PF_NOSCHED;
 		schedule();
+		current->flags |= nosched_flag;
 		set_task_state(tsk, sleep_state);
 	}

Ingo Molnar

2005-01-15 13:37 UTC #889 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.11-rc1-V0.7.35-00

In reply to Ingo Molnar

i have released the -V0.7.35-00 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

The two dozen split out latency patches (including PREEMPT_BKL) that were in -mm are in BK now, so i've rebased the -RT patchset to 2.6.11-rc1. It would be nice to check for regressions (or the lack of them), to make sure everything latency-related has been properly merged upstream from -mm. (so that a new splitup cycle can start.)

to create a -V0.7.34-00 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.11-rc1.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.11-rc1-V0.7.35-00

Ingo

K.R. Foley

2005-01-15 15:16 UTC #890 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc1-V0.7.35-00

Quoted 11 lines · Ingo Molnar

i have released the -V0.7.35-00 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
The two dozen split out latency patches (including PREEMPT_BKL) that were in -mm are in BK now, so i've rebased the -RT patchset to 2.6.11-rc1. It would be nice to check for regressions (or the lack of them), to make sure everything latency-related has been properly merged upstream from -mm. (so that a new splitup cycle can start.)

I've not done much testing with it yet but I do have it built and running on my slower SMP system. Working on my UP sytem now.

Gene Heskett

2005-01-15 19:29 UTC #891 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc1-V0.7.35-01

In reply to Ingo Molnar

Quoted 5 lines · Ingo Molnar

i have released the -V0.7.35-00 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/

By the time I got there is was 35-01 :-)

I have 1 squawk and one question.

The squawk is the extra "-RT" in the Makefiles Extra-Version, I made it 3 times and failed to boot to it with my script before that light came on.

And, what happened to drivers/ieee1394/amdtp? Its not even in the xconfig choices now. And I use it with my Sony Digital8 camera.

Also, and this is a more general squawk, I have to unload all those ieee1394 stuffs, turn on the camera, and then reload it all before the camera is recognized and usable. Ditto if I have it working, then turn the camera off, before I can access the camera again I have to rmmod all that stuff and reload it again. This is probably something that can be done with hotplug, but the last time I played with that it was lockup the whole machine disaster.

This was true with 2.6.11-rc1, but I haven't checked that stuff yet after getting this to boot just now.

Quoted 21 lines

The two dozen split out latency patches (including PREEMPT_BKL) that were in -mm are in BK now, so i've rebased the -RT patchset to 2.6.11-rc1. It would be nice to check for regressions (or the lack of them), to make sure everything latency-related has been properly merged upstream from -mm. (so that a new splitup cycle can start.)
to create a -V0.7.34-00 tree from scratch, the patching order is:
http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.11-rc1.bz
2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.11-r
c1-V0.7.35-00
Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

K.R. Foley

2005-01-21 20:39 UTC #892 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc1-V0.7.35-00

In reply to Ingo Molnar

Quoted 11 lines · Ingo Molnar

i have released the -V0.7.35-00 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
The two dozen split out latency patches (including PREEMPT_BKL) that were in -mm are in BK now, so i've rebased the -RT patchset to 2.6.11-rc1. It would be nice to check for regressions (or the lack of them), to make sure everything latency-related has been properly merged upstream from -mm. (so that a new splitup cycle can start.)

Ingo,

I just thought I would mention something that I have been experiencing with the last few versions of your patches on my workstation at the office. On my 2.6 dual xeon, if I have CONFIG_MICROCODE=m I have had a heck of a time getting the -V0.7.35-02,03 kernels to boot. After disabling CONFIG_MICROCODE -V0.7.35-03 boots fine now. I tried this after finding the following in my log after a successful boot of -V0.7.35-01.

Jan 21 11:53:50 swdev14 kernel: IA-32 Microcode Update Driver: v1.14 <[email protected]>
Jan 21 11:53:50 swdev14 kernel: BUG: sleeping function called from invalid context hotplug(3795) at kernel/rt.c:1443
Jan 21 11:53:50 swdev14 kernel: BUG: scheduling while atomic: hotplug/0x00010000/3796
Jan 21 11:53:50 swdev14 smartd[4274]: smartd version 5.33 [i386-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Jan 21 11:53:51 swdev14 kernel: in_atomic():1 [00010000], irqs_disabled():1
Jan 21 11:53:51 swdev14 smartd[4274]: Home page is http://smartmontools.sourceforge.net/
Jan 21 11:53:51 swdev14 kernel:  [<c010366a>] <6>microcode: CPU3 updated from revision 0x0 to 0x38, date = 06042003
Jan 21 11:53:51 swdev14 smartd[4274]: Opened configuration file /etc/smartd.conf
Jan 21 11:53:51 swdev14 kernel: dump_stack+0x23/0x25 (20)
Jan 21 11:53:51 swdev14 smartd[4274]: Configuration file /etc/smartd.conf parsed.
Jan 21 11:53:51 swdev14 kernel:  [<c01180fa>] __might_sleep+0xd8/0xf3caller is schedule+0x38/0x12f
Jan 21 11:53:51 swdev14 smartd[4274]: Device: /dev/hda, opened
Jan 21 11:53:51 swdev14 kernel:  (36)
Jan 21 11:53:51 swdev14 kernel:  [<c0133bd0>]  [<c010366a>] __spin_lock+0x34/0x50 (24)
Jan 21 11:53:51 swdev14 kernel:  [<c0133c66>] _spin_lock_irqsave+0x1d/0x21dump_stack+0x23/0x25 (20)
Jan 21 11:53:51 swdev14 kernel:  [<c02a2376>] __schedule+0xa86/0xd95 (16)
Jan 21 11:53:51 swdev14 smartd[4274]: Device: /dev/hda, found in smartd database.
Jan 21 11:53:51 swdev14 kernel:  [<e08c872f>]  (120)
Jan 21 11:53:51 swdev14 kernel:  [<c02a26bd>] do_update_one+0x4e/0xb7 [microcode] (44)
Jan 21 11:53:51 swdev14 smartd[4274]: Device: /dev/hda, is SMART capable. Adding to "monitor" list.
Jan 21 11:53:51 swdev14 kernel:  [<c010e6d0>] schedule+0x38/0x12fsmp_call_function_interrupt+0x45/0x65 (24)
Jan 21 11:53:51 swdev14 smartd[4274]: Monitoring 1 ATA and 0 SCSI devices
Jan 21 11:53:51 swdev14 kernel:  [<c01031ec>] call_function_interrupt+0x1c/0x24 (36)
Jan 21 11:53:51 swdev14 kernel:  [<c02a38f2>] __down_mutex+0x2aa/0x35a (176)
Jan 21 11:53:51 swdev14 kernel:  [<c0167f5f>]  (88)
Jan 21 11:53:51 swdev14 kernel: sys_fstat64+0x37/0x3d [<c0133be2>]  (100)
Jan 21 11:53:51 swdev14 smartd[4276]: smartd has fork()ed into background mode. New PID=4276.
Jan 21 11:53:51 swdev14 smartd: smartd startup succeeded
Jan 21 11:53:51 swdev14 kernel: __spin_lock+0x46/0x50 (24)
Jan 21 11:53:51 swdev14 kernel:  [<c0133c66>] _spin_lock_irqsave+0x1d/0x21 [<c0102804>] syscall_call+0x7/0xb (16)
Jan 21 11:53:51 swdev14 kernel:  [<e08c872f>]  (-8124)
Jan 21 11:53:51 swdev14 kernel: do_update_one+0x4e/0xb7 [microcode] (44)
Jan 21 11:53:51 swdev14 kernel:  [<c010e6d0>] ---------------------------
Jan 21 11:53:51 swdev14 kernel: smp_call_function_interrupt+0x45/0x65 (24)
Jan 21 11:53:52 swdev14 kernel:  [<c01031ec>] call_function_interrupt+0x1c/0x24| preempt count: 00010001 ]
Jan 21 11:53:52 swdev14 kernel: | 1-level deep critical section nesting:
Jan 21 11:53:52 swdev14 kernel: ----------------------------------------
Jan 21 11:53:52 swdev14 kernel: .. [<c0136ebd>] ....  (124)
Jan 21 11:53:52 swdev14 kernel: print_traces+0x1b/0x52
Jan 21 11:53:52 swdev14 kernel: .....[<c010366a>] ..   ( <= dump_stack+0x23/0x25)
Jan 21 11:53:52 swdev14 kernel:  [<c01333f5>] up_mutex+0x89/0x100
Jan 21 11:53:52 swdev14 sshd:  succeeded
Jan 21 11:53:52 swdev14 kernel:  (40)
Jan 21 11:53:52 swdev14 kernel:  [<c014eadc>] <3>BUG: scheduling while atomic: hotplug/0x00010000/3795
Jan 21 11:53:52 swdev14 kernel: caller is schedule+0x38/0x12f
Jan 21 11:53:52 swdev14 kernel: do_no_page+0x197/0x3b6 (64)
Jan 21 11:53:52 swdev14 kernel:  [<c014ef6e>] handle_mm_fault+0x15a/0x191 [<c010366a>] dump_stack+0x23/0x25 (48)
Jan 21 11:53:52 swdev14 kernel:  [<c0112291>]  (20)
Jan 21 11:53:52 swdev14 kernel: do_page_fault+0x1fc/0x61b (216)
Jan 21 11:53:52 swdev14 kernel:  [<c0103293>] error_code+0x2b/0x30 [<c02a2376>] __schedule+0xa86/0xd95 (-518059332)
Jan 21 11:53:52 swdev14 kernel: ---------------------------
Jan 21 11:53:52 swdev14 kernel: | preempt count: 00010001 ]
Jan 21 11:53:52 swdev14 xinetd: xinetd startup succeeded
Jan 21 11:53:52 swdev14 kernel: | 1-level deep critical section nesting:
Jan 21 11:53:52 swdev14 kernel: ----------------------------------------
Jan 21 11:53:52 swdev14 kernel: .. [<c0136ebd>] ....  (120)
Jan 21 11:53:52 swdev14 ntpdate[4310]: can't find host wizard
Jan 21 11:53:52 swdev14 kernel: print_traces+0x1b/0x52
Jan 21 11:53:52 swdev14 kernel: .....[<c010366a>] ..   ( <= dump_stack+0x23/0x25)
Jan 21 11:53:52 swdev14 kernel:  [<c02a26bd>] schedule+0x38/0x12f
Jan 21 11:53:52 swdev14 kernel:  (36)
Jan 21 11:53:52 swdev14 kernel:  [<c02a38f2>] __down_mutex+0x2aa/0x35a (88)
Jan 21 11:53:52 swdev14 kernel:  [<c0133be2>] __spin_lock+0x46/0x50 (24)
Jan 21 11:53:52 swdev14 kernel:  [<c0133c66>] _spin_lock_irqsave+0x1d/0x21 (16)
Jan 21 11:53:52 swdev14 kernel:  [<e08c872f>] do_update_one+0x4e/0xb7 [microcode] (44)
Jan 21 11:53:52 swdev14 kernel:  [<c010e6d0>] smp_call_function_interrupt+0x45/0x65 (24)
Jan 21 11:53:52 swdev14 kernel:  [<c01031ec>] call_function_interrupt+0x1c/0x24 (176)
Jan 21 11:53:52 swdev14 kernel:  [<c0167f5f>] sys_fstat64+0x37/0x3d (100)
Jan 21 11:53:52 swdev14 kernel:  [<c0102804>] syscall_call+0x7/0xb (-8124)
Jan 21 11:53:52 swdev14 kernel: ---------------------------
Jan 21 11:53:52 swdev14 kernel: | preempt count: 00010001 ]
Jan 21 11:53:52 swdev14 kernel: | 1-level deep critical section nesting:
Jan 21 11:53:52 swdev14 kernel: ----------------------------------------
Jan 21 11:53:52 swdev14 kernel: .. [<c0136ebd>] .... print_traces+0x1b/0x52
Jan 21 11:53:52 swdev14 kernel: .....[<c010366a>] ..   ( <= dump_stack+0x23/0x25)

Ingo Molnar

2005-01-22 12:29 UTC #893 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

In reply to Ingo Molnar

i have released the -V0.7.36-00 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

this is mainly a merge to 2.6.11-rc2.

There was alot of merging to be done due to Thomas Gleixner's spinlock/rwlock cleanups making it into upstream and due to the upstream spinlock changes, and there were some networking related conflicts as well, so these areas might introduce new regressions.

the patch includes a fix that should resolve the microcode-update related boot-time crash reported by K.R. Foley. It also includes a verify_mm_writelocked() fix from Daniel Walker.

to create a -V0.7.36-00 tree from scratch, the patching order is:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.11-rc2.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.11-rc2-V0.7.36-00

Ingo

Gene Heskett

2005-01-22 21:23 UTC #894 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

Quoted 6 lines · Ingo Molnar

i have released the -V0.7.36-00 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a merge to 2.6.11-rc2.

Humm, by the time I went after the patch it was up to -02.

And I'm getting a couple of error exits:
-------------------
net/sched/sch_generic.c: In function `qdisc_restart':
net/sched/sch_generic.c:128: error: label `requeue' used but not defined
CC drivers/pci/setup-bus.o
make[2]: *** [net/sched/sch_generic.o] Error 1 make[1]: *** [net/sched] Error 2
make[1]: *** Waiting for unfinished jobs....
-------------------

And
-------------------
LD net/sunrpc/built-in.o
make: *** [net] Error 2
make: *** Waiting for unfinished jobs....
-------------------
So obviously I'm not running it. :-)

One other item I don't think is related, in the last version (35-01) I had svn'd a new ieee1396 sub-directory from that ieee1394.org site into the drivers tree, and since it was less than a week old and worked right well, I just copied it over into the new kernel tree & reran the configs after renameing the existing ieee1394 to ieee1394-orig.

Quoted 19 lines

There was alot of merging to be done due to Thomas Gleixner's spinlock/rwlock cleanups making it into upstream and due to the upstream spinlock changes, and there were some networking related conflicts as well, so these areas might introduce new regressions.
the patch includes a fix that should resolve the microcode-update related boot-time crash reported by K.R. Foley. It also includes a verify_mm_writelocked() fix from Daniel Walker.
to create a -V0.7.36-00 tree from scratch, the patching order is:
http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.11-rc2.bz
2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.11-r
c2-V0.7.36-00
Ingo

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

andyliu

2005-01-23 09:27 UTC #895 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

hi , ingo

i am trying to understand your patch,but the patch file is so long and complex. i am wondering is there some documents about your patch?

thanks very much for your patch and your work.

Quoted 71 lines · Gene Heskett

Quoted 6 lines · Ingo Molnar
i have released the -V0.7.36-00 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a merge to 2.6.11-rc2.
Humm, by the time I went after the patch it was up to -02.
And I'm getting a couple of error exits:
-------------------
net/sched/sch_generic.c: In function `qdisc_restart':
net/sched/sch_generic.c:128: error: label `requeue' used but not defined
CC drivers/pci/setup-bus.o
make[2]: *** [net/sched/sch_generic.o] Error 1 make[1]: *** [net/sched] Error 2
make[1]: *** Waiting for unfinished jobs....
-------------------
And
-------------------
LD net/sunrpc/built-in.o
make: *** [net] Error 2
make: *** Waiting for unfinished jobs....
-------------------
So obviously I'm not running it. :-)
One other item I don't think is related, in the last version (35-01) I had svn'd a new ieee1396 sub-directory from that ieee1394.org site into the drivers tree, and since it was less than a week old and worked right well, I just copied it over into the new kernel tree & reran the configs after renameing the existing ieee1394 to ieee1394-orig.
Quoted 11 lines
There was alot of merging to be done due to Thomas Gleixner's spinlock/rwlock cleanups making it into upstream and due to the upstream spinlock changes, and there were some networking related conflicts as well, so these areas might introduce new regressions.
the patch includes a fix that should resolve the microcode-update related boot-time crash reported by K.R. Foley. It also includes a verify_mm_writelocked() fix from Daniel Walker.
to create a -V0.7.36-00 tree from scratch, the patching order is:
Quoted 8 lines
http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.10.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.11-rc2.bz
2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.11-r
c2-V0.7.36-00
Ingo
--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 1 line

Yours andyliu

Ingo Molnar

2005-01-23 11:31 UTC #896 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

Quoted 6 lines · andyliu

hi , ingo
i am trying to understand your patch,but the patch file is so long and complex. i am wondering is there some documents about your patch?
:)

well, it mainly offers the PREEMPT_RT feature, which is a 'no compromises' variant of kernel preemption: virtually everything (including normal spinlocked sections) is preemptable, with the goal of providing hard-realtime category ~10-20 usecs maximum scheduling latency guarantees on a typical PC (or embedded platform). Those long and complex changes are almost all needed to achieve this goal.

this tree is mainly an experiment to see what it takes to achieve that latency goal, and to see how much of that can go upstream (without having to decide whether upstream wants to have the PREEMPT_RT feature or not). (A couple of dozen patches were already split out of this patch and are in the current upstream kernel - they already made a latency difference for the 2.6.10 kernel.)

Ingo

Gene Heskett

2005-01-23 14:41 UTC #897 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

Quoted 17 lines · Ingo Molnar

Quoted 7 lines · andyliu
hi , ingo
i am trying to understand your patch,but the patch file is so long and complex. i am wondering is there some documents about your patch?
:)
well, it mainly offers the PREEMPT_RT feature, which is a 'no compromises' variant of kernel preemption: virtually everything (including normal spinlocked sections) is preemptable, with the goal of providing hard-realtime category ~10-20 usecs maximum scheduling latency guarantees on a typical PC (or embedded platform). Those long and complex changes are almost all needed to achieve this goal.

Quoted 8 lines

this tree is mainly an experiment to see what it takes to achieve that latency goal, and to see how much of that can go upstream (without having to decide whether upstream wants to have the PREEMPT_RT feature or not). (A couple of dozen patches were already split out of this patch and are in the current upstream kernel - they already made a latency difference for the 2.6.10 kernel.)
Ingo

Hijacking the thread here Ingo, but did you see my build failure message of yesterday?

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

andyliu

2005-01-24 01:02 UTC #898 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

hi Gene

i am lucky enough to have 2.6.11-rc2-V0.7.36-02 compile and run in my i386 host. below is my config

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.11-rc2-RT-V0.7.36-02
# Mon Jan 24 08:45:39 2005
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCK_KERNEL=y

#
# General setup
#
CONFIG_LOCALVERSION="RT"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not set
CONFIG_EMBEDDED=y
# CONFIG_KALLSYMS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
# CONFIG_SHMEM is not set
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_TINY_SHMEM=y

#
# Loadable module support
#
CONFIG_MODULES=y
# CONFIG_MODULE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
# CONFIG_HPET_TIMER is not set
CONFIG_SMP=y
CONFIG_NR_CPUS=2
CONFIG_SCHED_SMT=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_IRQBALANCE=y
CONFIG_HAVE_DEC_LOCK=y
# CONFIG_REGPARM is not set

#
# Power management options (ACPI, APM)
#
# CONFIG_PM is not set

#
# ACPI (Advanced Configuration and Power Interface) Support #
# CONFIG_ACPI is not set
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BLACKLIST_YEAR=0

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PC-card bridges
#
CONFIG_PCMCIA_PROBE=y

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=y
CONFIG_BINFMT_MISC=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
# CONFIG_ISAPNP is not set
# CONFIG_PNPBIOS is not set

#
# Block devices
#
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_INITRAMFS_SOURCE=""
CONFIG_LBD=y
# CONFIG_CDROM_PKTCDVD is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives #
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_CMD640=y
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
CONFIG_BLK_DEV_RZ1000=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=y
# CONFIG_SCSI_PROC_FS is not set

#
# SCSI support type (disk, tape, CD-ROM)
#
# CONFIG_BLK_DEV_SD is not set
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
# CONFIG_CHR_DEV_SG is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs #
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Multi-device support (RAID and LVM)
#
# CONFIG_MD is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
# CONFIG_NETLINK_DEV is not set
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_IP_TCPDIAG=y
# CONFIG_IP_TCPDIAG_IPV6 is not set

#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=y
# CONFIG_IP_NF_CT_ACCT is not set
# CONFIG_IP_NF_CONNTRACK_MARK is not set
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
# CONFIG_IP_NF_FTP is not set
# CONFIG_IP_NF_IRC is not set
# CONFIG_IP_NF_TFTP is not set
# CONFIG_IP_NF_AMANDA is not set
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_LIMIT=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_MAC=y
CONFIG_IP_NF_MATCH_PKTTYPE=y
CONFIG_IP_NF_MATCH_MARK=y
CONFIG_IP_NF_MATCH_MULTIPORT=y
CONFIG_IP_NF_MATCH_TOS=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_DSCP=y
CONFIG_IP_NF_MATCH_AH_ESP=y
CONFIG_IP_NF_MATCH_LENGTH=y
CONFIG_IP_NF_MATCH_TTL=y
CONFIG_IP_NF_MATCH_TCPMSS=y
CONFIG_IP_NF_MATCH_HELPER=y
CONFIG_IP_NF_MATCH_STATE=y
CONFIG_IP_NF_MATCH_CONNTRACK=y
CONFIG_IP_NF_MATCH_OWNER=y
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_REALM is not set
# CONFIG_IP_NF_MATCH_SCTP is not set
# CONFIG_IP_NF_MATCH_COMMENT is not set
# CONFIG_IP_NF_MATCH_HASHLIMIT is not set
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_IP_NF_TARGET_TCPMSS=y
CONFIG_IP_NF_NAT=y
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_SAME=y
# CONFIG_IP_NF_NAT_SNMP_BASIC is not set
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_TOS=y
CONFIG_IP_NF_TARGET_ECN=y
CONFIG_IP_NF_TARGET_DSCP=y
CONFIG_IP_NF_TARGET_MARK=y
CONFIG_IP_NF_TARGET_CLASSIFY=y
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=y
CONFIG_IP_NF_ARPFILTER=y
CONFIG_IP_NF_ARP_MANGLE=y

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
# CONFIG_NET_CLS_ROUTE is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_NET_SB1000 is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_CS89x0 is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_NET_POCKET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
CONFIG_S2IO=m
# CONFIG_S2IO_NAPI is not set
# CONFIG_2BUFF_MODE is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
# CONFIG_RTC is not set
CONFIG_BLOCKER=y
# CONFIG_GEN_RTC is not set
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
CONFIG_AGP_INTEL=y
CONFIG_AGP_INTEL_MCH=m
# CONFIG_AGP_NVIDIA is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I830 is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HANGCHECK_TIMER is not set

#
# I2C support
#
# CONFIG_I2C is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
# CONFIG_FB is not set
# CONFIG_VIDEO_SELECT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Sound
#
# CONFIG_SOUND is not set

#
# USB support
#
# CONFIG_USB is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_QUOTA is not set
# CONFIG_DNOTIFY is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set

#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set

#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_NTFS_FS is not set

#
# Network File Systems
#
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_EXPORTFS is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_UTF8 is not set

#
# Profiling support
#
# CONFIG_PROFILING is not set

#
# Kernel hacking
#
# CONFIG_DEBUG_KERNEL is not set
# CONFIG_DEBUG_PREEMPT is not set
CONFIG_WAKEUP_TIMING=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set
# CONFIG_CRITICAL_IRQSOFF_TIMING is not set
CONFIG_LATENCY_TIMING=y
# CONFIG_LATENCY_TRACE is not set
CONFIG_RT_DEADLOCK_DETECT=y
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_USE_FRAME_POINTER is not set
CONFIG_EARLY_PRINTK=y
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
# CONFIG_CRYPTO is not set

#
# Hardware crypto devices
#

#
# Library routines
#
# CONFIG_CRC_CCITT is not set
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y

Quoted 40 lines · Gene Heskett

Quoted 17 lines · Ingo Molnar
Quoted 7 lines · andyliu
hi , ingo
i am trying to understand your patch,but the patch file is so long and complex. i am wondering is there some documents about your patch?
:)
well, it mainly offers the PREEMPT_RT feature, which is a 'no compromises' variant of kernel preemption: virtually everything (including normal spinlocked sections) is preemptable, with the goal of providing hard-realtime category ~10-20 usecs maximum scheduling latency guarantees on a typical PC (or embedded platform). Those long and complex changes are almost all needed to achieve this goal.
Quoted 8 lines
this tree is mainly an experiment to see what it takes to achieve that latency goal, and to see how much of that can go upstream (without having to decide whether upstream wants to have the PREEMPT_RT feature or not). (A couple of dozen patches were already split out of this patch and are in the current upstream kernel - they already made a latency difference for the 2.6.10 kernel.)
Ingo
Hijacking the thread here Ingo, but did you see my build failure message of yesterday?
--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attorneys please note, additions to this message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Signature · 1 line

Yours andyliu

Gene Heskett

2005-01-24 02:56 UTC #899 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

Quoted 2 lines · andyliu

hi Gene

Thanks, I'll give it a shot and see if it fits my hardware.

Quoted 1,105 lines

i am lucky enough to have 2.6.11-rc2-V0.7.36-02 compile and run in my i386 host. below is my config
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.11-rc2-RT-V0.7.36-02
# Mon Jan 24 08:45:39 2005
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCK_KERNEL=y
#
# General setup
#
CONFIG_LOCALVERSION="RT"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not set
CONFIG_EMBEDDED=y
# CONFIG_KALLSYMS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
# CONFIG_SHMEM is not set
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_TINY_SHMEM=y
#
# Loadable module support
#
CONFIG_MODULES=y
# CONFIG_MODULE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
# CONFIG_HPET_TIMER is not set
CONFIG_SMP=y
CONFIG_NR_CPUS=2
CONFIG_SCHED_SMT=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_IRQBALANCE=y
CONFIG_HAVE_DEC_LOCK=y
# CONFIG_REGPARM is not set
#
# Power management options (ACPI, APM)
#
# CONFIG_PM is not set
#
# ACPI (Advanced Configuration and Power Interface) Support #
# CONFIG_ACPI is not set
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BLACKLIST_YEAR=0
#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set
#
# PC-card bridges
#
CONFIG_PCMCIA_PROBE=y
#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set
#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=y
CONFIG_BINFMT_MISC=y
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set
#
# Parallel port support
#
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_PC_CML1=y
# CONFIG_PARPORT_SERIAL is not set
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_OTHER is not set
# CONFIG_PARPORT_1284 is not set
#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set
#
# Protocols
#
# CONFIG_ISAPNP is not set
# CONFIG_PNPBIOS is not set
#
# Block devices
#
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_INITRAMFS_SOURCE=""
CONFIG_LBD=y
# CONFIG_CDROM_PKTCDVD is not set
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_ATA_OVER_ETH is not set
#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
#
# Please see Documentation/ide.txt for help/info on IDE drives #
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set
#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_CMD640=y
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
CONFIG_BLK_DEV_RZ1000=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set
#
# SCSI device support
#
CONFIG_SCSI=y
# CONFIG_SCSI_PROC_FS is not set
#
# SCSI support type (disk, tape, CD-ROM)
#
# CONFIG_BLK_DEV_SD is not set
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
# CONFIG_CHR_DEV_SG is not set
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs #
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
#
# SCSI Transport Attributes
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set
#
# Multi-device support (RAID and LVM)
#
# CONFIG_MD is not set
#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set
#
# I2O device support
#
# CONFIG_I2O is not set
#
# Networking support
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
# CONFIG_NETLINK_DEV is not set
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_IP_TCPDIAG=y
# CONFIG_IP_TCPDIAG_IPV6 is not set
#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=y
# CONFIG_IP_NF_CT_ACCT is not set
# CONFIG_IP_NF_CONNTRACK_MARK is not set
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
# CONFIG_IP_NF_FTP is not set
# CONFIG_IP_NF_IRC is not set
# CONFIG_IP_NF_TFTP is not set
# CONFIG_IP_NF_AMANDA is not set
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_LIMIT=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_MAC=y
CONFIG_IP_NF_MATCH_PKTTYPE=y
CONFIG_IP_NF_MATCH_MARK=y
CONFIG_IP_NF_MATCH_MULTIPORT=y
CONFIG_IP_NF_MATCH_TOS=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_DSCP=y
CONFIG_IP_NF_MATCH_AH_ESP=y
CONFIG_IP_NF_MATCH_LENGTH=y
CONFIG_IP_NF_MATCH_TTL=y
CONFIG_IP_NF_MATCH_TCPMSS=y
CONFIG_IP_NF_MATCH_HELPER=y
CONFIG_IP_NF_MATCH_STATE=y
CONFIG_IP_NF_MATCH_CONNTRACK=y
CONFIG_IP_NF_MATCH_OWNER=y
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_REALM is not set
# CONFIG_IP_NF_MATCH_SCTP is not set
# CONFIG_IP_NF_MATCH_COMMENT is not set
# CONFIG_IP_NF_MATCH_HASHLIMIT is not set
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_IP_NF_TARGET_TCPMSS=y
CONFIG_IP_NF_NAT=y
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_SAME=y
# CONFIG_IP_NF_NAT_SNMP_BASIC is not set
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_TOS=y
CONFIG_IP_NF_TARGET_ECN=y
CONFIG_IP_NF_TARGET_DSCP=y
CONFIG_IP_NF_TARGET_MARK=y
CONFIG_IP_NF_TARGET_CLASSIFY=y
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=y
CONFIG_IP_NF_ARPFILTER=y
CONFIG_IP_NF_ARP_MANGLE=y
#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
# CONFIG_NET_CLS_ROUTE is not set
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_NET_SB1000 is not set
#
# ARCnet devices
#
# CONFIG_ARCNET is not set
#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set
#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_CS89x0 is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_NET_POCKET is not set
#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
CONFIG_S2IO=m
# CONFIG_S2IO_NAPI is not set
# CONFIG_2BUFF_MODE is not set
#
# Token Ring devices
#
# CONFIG_TR is not set
#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set
#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
#
# ISDN subsystem
#
# CONFIG_ISDN is not set
#
# Telephony Support
#
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set
#
# Input I/O drivers
#
# CONFIG_GAMEPORT is not set
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_INPORT is not set
# CONFIG_MOUSE_LOGIBM is not set
# CONFIG_MOUSE_PC110PAD is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=y
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
# CONFIG_TIPAR is not set
#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set
#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
# CONFIG_RTC is not set
CONFIG_BLOCKER=y
# CONFIG_GEN_RTC is not set
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
#
# Ftape, the floppy tape device driver
#
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
CONFIG_AGP_INTEL=y
CONFIG_AGP_INTEL_MCH=m
# CONFIG_AGP_NVIDIA is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I830 is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HANGCHECK_TIMER is not set
#
# I2C support
#
# CONFIG_I2C is not set
#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set
#
# Misc devices
#
# CONFIG_IBM_ASM is not set
#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set
#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set
#
# Graphics support
#
# CONFIG_FB is not set
# CONFIG_VIDEO_SELECT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
#
# Sound
#
# CONFIG_SOUND is not set
#
# USB support
#
# CONFIG_USB is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information
#
#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set
#
# MMC/SD Card support
#
# CONFIG_MMC is not set
#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set
#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_QUOTA is not set
# CONFIG_DNOTIFY is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
# CONFIG_DEVPTS_FS_XATTR is not set
# CONFIG_TMPFS is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
#
# Network File Systems
#
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_EXPORTFS is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_UTF8 is not set
#
# Profiling support
#
# CONFIG_PROFILING is not set
#
# Kernel hacking
#
# CONFIG_DEBUG_KERNEL is not set
# CONFIG_DEBUG_PREEMPT is not set
CONFIG_WAKEUP_TIMING=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set
# CONFIG_CRITICAL_IRQSOFF_TIMING is not set
CONFIG_LATENCY_TIMING=y
# CONFIG_LATENCY_TRACE is not set
CONFIG_RT_DEADLOCK_DETECT=y
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_USE_FRAME_POINTER is not set
CONFIG_EARLY_PRINTK=y
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
#
# Cryptographic options
#
# CONFIG_CRYPTO is not set
#
# Hardware crypto devices
#
#
# Library routines
#
# CONFIG_CRC_CCITT is not set
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
On Sun, 23 Jan 2005 09:40:56 -0500, Gene Heskett
Quoted 40 lines · <[email protected]> wrote:
Quoted 25 lines · Ingo Molnar
Quoted 7 lines · andyliu
hi , ingo
i am trying to understand your patch,but the patch file is so long and complex. i am wondering is there some documents about your patch?
:)
well, it mainly offers the PREEMPT_RT feature, which is a 'no compromises' variant of kernel preemption: virtually everything (including normal spinlocked sections) is preemptable, with the goal of providing hard-realtime category ~10-20 usecs maximum scheduling latency guarantees on a typical PC (or embedded platform). Those long and complex changes are almost all needed to achieve this goal.
this tree is mainly an experiment to see what it takes to achieve that latency goal, and to see how much of that can go upstream (without having to decide whether upstream wants to have the PREEMPT_RT feature or not). (A couple of dozen patches were already split out of this patch and are in the current upstream kernel - they already made a latency difference for the 2.6.10 kernel.)
Quoted 1 line
Ingo
Hijacking the thread here Ingo, but did you see my build failure message of yesterday?
--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attorneys please note, additions to this message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Ingo Molnar

2005-01-24 08:03 UTC #900 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

In reply to Gene Heskett

Quoted 8 lines · Gene Heskett

Quoted 6 lines · Ingo Molnar
i have released the -V0.7.36-00 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a merge to 2.6.11-rc2.

Quoted 7 lines

Humm, by the time I went after the patch it was up to -02.
And I'm getting a couple of error exits:
-------------------
net/sched/sch_generic.c: In function `qdisc_restart':
net/sched/sch_generic.c:128: error: label `requeue' used but not defined

indeed - !PREEMPT_RT compilation broke. I've uploaded -03 with the fix (and other fixes).

Ingo

Gene Heskett

2005-01-24 10:42 UTC #901 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

Quoted 17 lines · Ingo Molnar

Quoted 8 lines · Gene Heskett
Quoted 6 lines · Ingo Molnar
i have released the -V0.7.36-00 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is mainly a merge to 2.6.11-rc2.
Quoted 7 lines
Humm, by the time I went after the patch it was up to -02.
And I'm getting a couple of error exits:
-------------------
net/sched/sch_generic.c: In function `qdisc_restart':
net/sched/sch_generic.c:128: error: label `requeue' used but not defined

Quoted 4 lines

indeed - !PREEMPT_RT compilation broke. I've uploaded -03 with the fix (and other fixes).
Ingo

Thanks a bunch for your work, Ingo. I'm having fun running this stuff, and it really does seem to help everything but kmail/spamassassin.

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.32% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Lee Revell

2005-01-26 01:08 UTC #902 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.36-00

In reply to Ingo Molnar

Quoted 2 lines · Ingo Molnar

indeed - !PREEMPT_RT compilation broke. I've uploaded -03 with the fix (and other fixes).

-03 still does not compile with CONFIG_PREEMPT_DESKTOP:

rlrevell@mindpipe:~/kernel-source/linux-2.6.11-rc2$ make
  CHK     include/linux/version.h
make[1]: `arch/i386/kernel/asm-offsets.s' is up to date.
  CHK     include/linux/compile.h
  CHK     usr/initramfs_list
  CC      net/core/rtnetlink.o
net/core/rtnetlink.c: In function `rtnl_lock_interruptible':
net/core/rtnetlink.c:63: warning: implicit declaration of function `down_write_interruptible'
  LD      net/core/built-in.o
  LD      net/built-in.o
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
net/built-in.o(.text+0x10f99): In function `rtnl_lock_interruptible':
net/core/rtnetlink.c:63: undefined reference to `down_write_interruptible'
make: *** [.tmp_vmlinux1] Error 1

Lee

Ingo Molnar

2005-02-01 20:14 UTC #903 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

In reply to Ingo Molnar

i have released the -V0.7.37-02 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

the big change in the patch is increased architecture support: most notable i've merged the MIPS patch from Manish Lachwani. Also, the x64 port should be working again. (To make architecture merges easier in the future the timer interrupt is not threaded anymore - if this shows latency problems then we'll try to solve it on other ways.)

to create a -V0.7.37-02 tree from scratch, the patching order is:

Ingo

Eugeny S. Mints

2005-02-03 20:46 UTC #904 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

Quoted 4 lines · Ingo Molnar

i have released the -V0.7.37-02 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/

Minor fix for deadlock tracer: "...acquired at XXX" may print caller's of an up() eip instead of eip of caller of a down() in case a lock was initally contended before deadlock is detected.

Seems actual for 37-03 as well. pathc is attached.

Eugeny

rt.c renamed from rt.c.orig +1 -1

@@ -734,7 +734,7 @@
 
 	list_del_init(&waiter->pi_list);
 
-	set_new_owner(lock, old_owner, new_owner, eip);
+	set_new_owner(lock, old_owner, new_owner, waiter->eip);
 	/* Don't touch waiter after ->task has been NULLed */
 	mb();
 	waiter->task = NULL;

Ingo Molnar

2005-02-03 20:56 UTC #905 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

Quoted 10 lines · Eugeny S. Mints

Quoted 4 lines · Ingo Molnar
i have released the -V0.7.37-02 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
Minor fix for deadlock tracer: "...acquired at XXX" may print caller's of an up() eip instead of eip of caller of a down() in case a lock was initally contended before deadlock is detected.
Seems actual for 37-03 as well. patch is attached.

ah - this might explain some of the weirder deadlock traces. Thanks and applied.

Ingo

Steven Rostedt

reviewer

2005-02-04 02:00 UTC #906 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

In reply to Ingo Molnar

Quoted 10 lines · Ingo Molnar

i have released the -V0.7.37-02 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
the big change in the patch is increased architecture support: most notable i've merged the MIPS patch from Manish Lachwani. Also, the x64 port should be working again. (To make architecture merges easier in the future the timer interrupt is not threaded anymore - if this shows latency problems then we'll try to solve it on other ways.)

Ingo,

I was just about to send you a note about why the timer interrupt as a thread was a problem, but you changed it here, so it really isn't anymore. But since your reason is for architecture, and not for this reason, I'm posting it anyway.

Here was what I was about to send:

I was wondering why the timer interrupt is a thread, because this can cause some unexpected results.

Say if you have two high priority processes A and B. A is higher than B and both are higher than the timer interrupt (I'm on x86 so it's IRQ0)

B is doing lots of busy work, and A does a little then sleeps a little.

This is what is happening. Once A sleeps, B goes off doing lots of busy work, and A doesn't wake up until B is done, although A had a timeout, that it missed.

What's going on is that the jiffies are not incrementing. After the first time that IRQ 0 goes goes off, the interrupt is disabled. So we don't get any more interrupts for the timer, and B gets to run as much as it wants leaving A in the dust.

So my question to you is why even bother having the timer interrupt as thread. Maybe make a tasklet or something to do more of it's work, but let the interrupt go off as much as it wants. Otherwise, you can have the problem mentioned above. If the fix to this is to either change the timer interrupt to either NODELAY or just have the processes lower in priority than the timer, then what's the point to having the timer as a thread?

OK that was the note I was about to send, but like I stated, it isn't a problem now that the timer interrupt is back to a hard interrupt. I just showing this to you so you can see the real problem. Maybe I'm missing something, and maybe I'm not. I'll try to write up something that shows the problem with the timer interrupt as a thread.

Thanks,

-- Steve

Steven Rostedt

reviewer

2005-02-04 02:23 UTC #907 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

Quoted 6 lines

OK that was the note I was about to send, but like I stated, it isn't a problem now that the timer interrupt is back to a hard interrupt. I just showing this to you so you can see the real problem. Maybe I'm missing something, and maybe I'm not. I'll try to write up something that shows the problem with the timer interrupt as a thread.

Well, I wrote something up and I found a test case that locks the machine (on 2.6.11-rc2-V0.7.37-03). I've only tried this on 2.6.10-rc3-mm1, where it works. I don't have anymore time to look into this today, so I haven't tried it on vanilla 2.6.11-rc2.

The attached file was going to be something that shows what happens when you have two processes with a higher priority than the timer interrupt. But unfortunately it locked up 36-03. I tried it on 37-03 too, and it locked it up as well. But if I modify this program a little, it runs fine on both.

Here's the case: I have two processes waiting on a semaphore to go to zero. Once it does, they both run spin loops, where one sleeps, and the other just spins for a duration. The parent of these processes, set their priorities very high and then zeros the semaphore.

Here's the problem: If I raise the priorities of the processes before zeroing the semaphore, the machine hangs. If I zero the semaphore before raising the priorities, the program runs fine.

You would think that if the problem happened the other way around, it could be just that one of the high priority spinners is starving everything else. But the problem occurs if the spinners are SLEEPING!

Attached is the code that locks up the machine. I'd look into it but I've already done my 16 hours today ;-)

-- Steve

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <sys/sem.h>
#include <sys/shm.h>
#include <sys/time.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <sched.h>
#include <signal.h>

#define __USE_GNU
#include <sys/ipc.h>

static int int_spinner (int i);
static int spinner (int i);

typedef int (*callfunc_t)(int);

#define NR_PROCS 2
pid_t pids[NR_PROCS];
callfunc_t dofunc[NR_PROCS] = {
int_spinner,
spinner,
};

key_t key;
int semid = -1;
int shmid = -1;
int *flags;
pid_t timer_pid;
int timer_prio;

void cleanup(void)

{
	int i;

	for (i=0; i < NR_PROCS; i++) {
		if (pids[0])
			kill(pids[i],SIGKILL);
	}

	if (semid >= 0)
		semctl(semid, 0, IPC_RMID);
	if (shmid >= 0)
		shmctl(shmid, IPC_RMID, NULL);
}

void double_to_tv(double d, struct timeval *tv)

{
	tv->tv_sec = (int)d;
	tv->tv_usec = (int) ((d - (double)tv->tv_sec) * 1000000.0);
	printf ("sec:%d usecs:%d\n",(int)tv->tv_sec,(int)tv->tv_usec);
}

void catchall(int sig)

{
	cleanup();
	psignal(sig,"Caught: ");
	exit(-1);
}

static int compare_timeval(const struct timeval *a, const struct timeval *b)

{
	return (a->tv_sec > b->tv_sec) ? 1 :
		(a->tv_sec < b->tv_sec) ? -1:
		(a->tv_usec > b->tv_usec) ? 1:
		(a->tv_usec < b->tv_usec) ? -1:
		0;
}

static void add_timeval(const struct timeval *a, const struct timeval *b, struct timeval *c) {

	c->tv_usec = a->tv_usec + b->tv_usec;
	c->tv_sec = a->tv_sec + b->tv_sec;
	while (c->tv_usec > 1000000) {
		c->tv_usec -= 1000000;
		c->tv_sec++;
	}

}

static void sub_timeval(const struct timeval *a, const struct timeval *b, struct timeval *c) {

	c->tv_usec = a->tv_usec - b->tv_usec;
	c->tv_sec = a->tv_sec - b->tv_sec;
	while (c->tv_usec < 0) {
		c->tv_usec += 1000000;
		c->tv_sec--;
	}

}

static int int_spinner(int i)

{
	struct sembuf sops;
	int semid;
	struct timeval starttv;
	struct timeval endtv;
	struct timeval tv;
	struct timeval lasttv;
	struct timeval nexttv;
	time_t t;
	char timebuf[30];
	unsigned long long starttsc, nowtsc;

	memset(&sops,0,sizeof(sops));
	
	if ((semid = semget(key,1,0)) < 0) {
		perror("semget");
		return -1;
	}
	if ((shmid = shmget(key,30,0600)) < 0) {
		perror("shmget");
		return -1;
	}
	if ((flags = shmat(shmid, NULL, 0)) == (void*)-1) {
		perror("shmat");
		return -1;
	}
	
	printf("int_spinner %d grabbing waiting on sem\n",i);
	if (semop(semid, &sops, 1) < 0) {
		perror("semop");
		return -1;
	}
	printf("int_spinner %d got semaphore\n",i);


	if (gettimeofday(&starttv,NULL) < 0) {
		perror("gettimeofday");
		return -1;
	}

	endtv.tv_sec = 10;
	endtv.tv_usec = 0;

	add_timeval(&starttv,&endtv,&endtv);

	t = time(NULL);
	ctime_r(&t,timebuf);
	timebuf[strlen(timebuf)-1] = 0;

	printf("spinner %d starting loop (%s)\n",i,timebuf);
	lasttv = starttv;

	nexttv = starttv;
	nexttv.tv_sec += 3;

	printf("A sleeping\n");
	(*flags)++;
	sleep(1);
	printf("A a wake\n");
	
	asm ("rdtsc" : "=A"(starttsc));

	do {
		asm ("rdtsc" : "=A"(nowtsc));
		if (nowtsc - starttsc > 7000000000ULL) {
			printf("spinner %d now - start > 7000000000\n",i);
			break;
		}
		if (gettimeofday(&tv,NULL) < 0) {
			perror("gettimeofday (in loop)");
			return -1;
		}
		if (compare_timeval(&tv,&nexttv) > 0) {
			printf("A sleeping\n");
			(*flags)++;
			sleep(1);
			printf("A a wake\n");
			nexttv.tv_sec += 3;
		}
		sub_timeval(&tv,&lasttv,&lasttv);
		lasttv = tv;
	} while(compare_timeval(&endtv,&tv) > 0);

	t = time(NULL);
	ctime_r(&t,timebuf);
	timebuf[strlen(timebuf)-1] = 0;
	printf("spinner %d ended loop flags=%d (%s)\n",i,*flags,timebuf);
	

	return 0;
}

static int spinner(int i)

{
	struct sembuf sops;
	int semid;
	struct timeval starttv;
	struct timeval endtv;
	struct timeval tv;
	struct timeval lasttv;
	struct timeval deltatv;
	time_t t;
	char timebuf[30];
	unsigned long long starttsc, nowtsc;
	int last_flag;

	memset(&sops,0,sizeof(sops));
	
	if ((semid = semget(key,1,0)) < 0) {
		perror("semget");
		return -1;
	}
	if ((shmid = shmget(key,30,0600)) < 0) {
		perror("shmget");
		return -1;
	}
	if ((flags = shmat(shmid, NULL, 0)) == (void*)-1) {
		perror("shmat");
		return -1;
	}
	


	printf("spinner %d grabbing waiting on sem\n",i);
	if (semop(semid, &sops, 1) < 0) {
		perror("semop");
		return -1;
	}

	printf("spinner %d got semaphore\n",i);

	if (gettimeofday(&starttv,NULL) < 0) {
		perror("gettimeofday");
		return -1;
	}

	memset (&deltatv,0,sizeof(deltatv));

	endtv.tv_sec = 10;
	endtv.tv_usec = 0;

	add_timeval(&starttv,&endtv,&endtv);

	t = time(NULL);
	ctime_r(&t,timebuf);
	timebuf[strlen(timebuf)-1] = 0;

	printf("spinner %d starting loop flags=%d (%s)\n",i,*flags,timebuf);
	lasttv = starttv;
	asm ("rdtsc" : "=A"(starttsc));

	last_flag = *flags;
	do {
		asm ("rdtsc" : "=A"(nowtsc));
		if (nowtsc - starttsc > 7000000000ULL) {
			printf("spinner %d now - start > 7000000000\n",i);
			break;
		}
		if (gettimeofday(&tv,NULL) < 0) {
			perror("gettimeofday (in loop)");
			return -1;
		}
		if (*flags != last_flag) {
			printf("spinner %d running again (flags=%d)!\n",i,*flags);
			last_flag = *flags;
		}			

		sub_timeval(&tv,&lasttv,&lasttv);
		if (compare_timeval(&deltatv,&lasttv) < 0)
			deltatv = lasttv;
		lasttv = tv;
	} while(compare_timeval(&endtv,&tv) > 0);
	t = time(NULL);
	ctime_r(&t,timebuf);
	timebuf[strlen(timebuf)-1] = 0;
	printf("spinner %d ended loop (%s) flags=%d (%d.%06d secs delta)\n",
	       i,timebuf,*flags,(int)deltatv.tv_sec,(int)deltatv.tv_usec);
	

	return 0;
}

int set_realtime_priority(pid_t pid, unsigned int prio)

{
	struct sched_param p;
	int ret;

	memset(&p,0,sizeof(p));
	p.sched_priority = prio;
	if ((ret = sched_setscheduler(pid,SCHED_FIFO,&p)) < 0) {
		perror("sched_setscheduler");
	}
	return ret;
}

int main (int argc, char **argv)

{
	int ret=0;
	int i;
	int nr_procs=0;
	struct sched_param p;
	struct sembuf sops;
	unsigned int max;
	
	key = ftok(argv[0],123);

	if (argc >=2) {
		timer_pid = atoi(argv[1]);
		if (sched_getparam(timer_pid,&p) < 0) {
			perror("sched_getparam");
			goto out;
		}
		timer_prio = p.sched_priority;
		printf("timer priority is %d\n",timer_prio);
	}

		

	if ((semid = semget(key,1,IPC_CREAT|IPC_EXCL|0600)) < 0) {
		perror("semget");
		exit (-1);
	}
	if ((shmid = shmget(key,30,IPC_CREAT|IPC_EXCL|0600)) < 0) {
		perror("shmget");
		goto out;
	}
	if ((flags = shmat(shmid, NULL, 0)) == (void*)-1) {
		perror("shmat");
		goto out;
	}
	

	/* Grab the semaphore before anyone else can take it. */
	memset(&sops,0,sizeof(sops));
//	sops.sem_flg = SEM_UNDO;
	sops.sem_op = 1;
	if (semop(semid, &sops, 1) < 0) {
		perror("semop");
		ret = -1;
		goto out;
	}

	for (i=0; i < NR_PROCS; i++) {
		if ((pids[i] = fork()) < 0) {
			perror("fork");
			ret = -1;
			goto out;
		} else if (pids[i] == 0) {
			/* child */
			ret = dofunc[i](i);
			exit(ret);
		}
		nr_procs++;
		/* parent */
	}

	
	signal(SIGINT,catchall);
	signal(SIGILL,catchall);
	signal(SIGFPE,catchall);
	signal(SIGSEGV,catchall);
	signal(SIGBUS,catchall);


	max = sched_get_priority_max(SCHED_FIFO);
	printf("max priority is %d\n",max);
	
	sleep(1);

#if 1
	ret = -1;
	printf("setting priorities\n");
	for (i=0; i < NR_PROCS; i++) {
		printf("setting %d to prio %d\n",i,max-(1+i));
		if (set_realtime_priority(pids[0],max-(1 + i)))
			goto out;
	}

#endif

	printf("parent zeroing semaphore\n");
	sops.sem_op = -1;
       	if (semop(semid,&sops,1) < 0) {
		perror("semop");
		ret = -1;
		goto out;
	}


	while (nr_procs) {
		int status;
		pid_t pid;
		if ((pid = wait(&status)) < 0) {
			perror("wait");
			break;
		}
		for (i=0; i < NR_PROCS; i++) {
			if (pids[i] == pid) {
				pids[i] = 0;
				nr_procs--;
			}
		}
		
	}
	printf("flags = %d\n",*flags);
 out:
	cleanup();
	exit(ret);
}

Steven Rostedt

reviewer

2005-02-05 06:03 UTC #908 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

Quoted 3 lines · Steven Rostedt

Here's the problem: If I raise the priorities of the processes before zeroing the semaphore, the machine hangs. If I zero the semaphore before raising the priorities, the program runs fine.

Hi Ingo,

I've looked into this and found where the deadlock occurs. Actually it is a starving of processes. I finally got around to trying it on 2.6.11-rc3, and it doesn't have a problem.

Anyway, here's the scoop.

The parent process that zeros the semaphore to allow the child process to run after the parent upped the priority of the child really high, gets stuck in the following:

In update_queue in ipc/sem.c (I don't have the correct line numbers, cause I haven't stripped out the debug yet). at the point of:
q->status = IN_WAKEUP;
/*
* Continue scanning. The next operation
* that must be checked depends on the type of the
* completed operation:
* - if the operation modified the array, then
* restart from the head of the queue and
* check for threads that might be waiting
* for semaphore values to become 0.
* - if the operation didn't modify the array,
* then just continue.
*/
if (q->alter)
n = sma->sem_pending;
else
n = q->next;
wake_up_process(q->sleeper);

Here is where it locks up.

			/* hands-off: q will disappear immediately after
			 * writing q->status.
			 */
			q->status = error;

I also found that the high priority child is running in an infinite while loop in sys_semtimedop in the same file, at the following:

	while(unlikely(error == IN_WAKEUP)) {
		cpu_relax();
		error = queue.status;
	}

So, what looks to be happening is that as soon as the parent wakes up the child, the child preempts the parent, and hits this while loop. But since the child is a realtime task, with the highest priority of the system, it starves the system. Of course this is a UP and I don't think this will show a problem on an SMP machine.

I can't think of a solution right now, so I'll just pass it on to you ;-). Once again it's late and I'm going to bed.

Thanks,

-- Steve

Ingo Molnar

2005-02-05 07:59 UTC #909 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

Quoted 10 lines · Steven Rostedt

	while(unlikely(error == IN_WAKEUP)) {
		cpu_relax();
		error = queue.status;
	}
So, what looks to be happening is that as soon as the parent wakes up the child, the child preempts the parent, and hits this while loop. But since the child is a realtime task, with the highest priority of the system, it starves the system. Of course this is a UP and I don't think this will show a problem on an SMP machine.

hm - i had a fix in this area in the -V0.7 series. Then i thought this is a performance fix only and dropped it eventually, but could you give it a go - does it fix the deadlock?

Ingo

ipc/sem.c renamed from ipc/sem.c.orig +6 -0

@@ -359,12 +371,18 @@ static void update_queue (struct sem_arr
 			struct sem_queue *n;
 			remove_from_queue(sma,q);
 			n = q->next;
+			/*
+			 * Make sure that the wakeup doesnt preempt
+			 * _this_ CPU prematurely. (on PREEMPT_RT)
+			 */
+			preempt_disable();
 			q->status = IN_WAKEUP;
 			wake_up_process(q->sleeper);
 			/* hands-off: q will disappear immediately after
 			 * writing q->status.
 			 */
 			q->status = error;
+			preempt_enable();
 			q = n;
 		} else {
 			q = q->next;

Steven Rostedt

reviewer

2005-02-05 14:33 UTC #910 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

Quoted 5 lines · Ingo Molnar

hm - i had a fix in this area in the -V0.7 series. Then i thought this is a performance fix only and dropped it eventually, but could you give it a go - does it fix the deadlock?
Ingo

Yep, it worked! I tried a similar fix earlier but I put the preempt disable before the setting of q->status (duh!) and it didn't work. But it was late and I was tired of looking at it. I was about to say that I already tried it, but then noticed the placement of preempt_disable, and thought, I better try yours anyway. Well, it seems to fix it. By the way, I just put in the disable and enable in -37. I haven't gotten to your 38 yet, but this fixed 37.

Thanks,

-- Steve

Quoted 22 lines

ipc/sem.c renamed from ipc/sem.c.orig +6 -0

@@ -359,12 +371,18 @@ static void update_queue (struct sem_arr
 			struct sem_queue *n;
 			remove_from_queue(sma,q);
 			n = q->next;
+			/*
+			 * Make sure that the wakeup doesnt preempt
+			 * _this_ CPU prematurely. (on PREEMPT_RT)
+			 */
+			preempt_disable();
 			q->status = IN_WAKEUP;
 			wake_up_process(q->sleeper);
 			/* hands-off: q will disappear immediately after
 			 * writing q->status.
 			 */
 			q->status = error;
+			preempt_enable();
 			q = n;
 		} else {
 			q = q->next;

Ingo Molnar

2005-02-07 09:23 UTC #911 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.11-rc2-V0.7.37-02

Quoted 8 lines · Steven Rostedt

Quoted 5 lines · Ingo Molnar
hm - i had a fix in this area in the -V0.7 series. Then i thought this is a performance fix only and dropped it eventually, but could you give it a go - does it fix the deadlock?
Ingo

Quoted 7 lines

Yep, it worked! I tried a similar fix earlier but I put the preempt disable before the setting of q->status (duh!) and it didn't work. But it was late and I was tired of looking at it. I was about to say that I already tried it, but then noticed the placement of preempt_disable, and thought, I better try yours anyway. Well, it seems to fix it. By the way, I just put in the disable and enable in -37. I haven't gotten to your 38 yet, but this fixed 37.

good - i've merged this into the -38-03 release.

Ingo

Fernando Lopez-Lezcano

2004-11-24 21:20 UTC #912 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

In reply to Ingo Molnar

Quoted 6 lines · Ingo Molnar

i have released the -V0.7.30-9 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
this is a fixes-only release.

Using PREEMPT_DESKTOP I see a irq related problem with my network interface:

cat /proc/interrupts
CPU0

  0:     134209    IO-APIC-edge  timer  0/34148
  1:        221    IO-APIC-edge  i8042  0/221
  8:          1    IO-APIC-edge  rtc  0/1
  9:          0   IO-APIC-level  acpi  0/0
 11:          0    IO-APIC-edge  radeon@PCI:1:0:0  0/0
 12:       2215    IO-APIC-edge  i8042  0/2215
 14:        650    IO-APIC-edge  ide0  2/648
 16:     100000   IO-APIC-level  eth0  0/0
 17:         59   IO-APIC-level  libata, libata  0/59
 18:          0   IO-APIC-level  ICE1712  0/0
 20:      15160   IO-APIC-level  libata  0/15160
 21:          0   IO-APIC-level  ehci_hcd, uhci_hcd, uhci_hcd, uhci_hcd,
uhci_hcd  0/0
NMI:          0 
LOC:     134034 
ERR:          0
MIS:          0

relevant portion of dmesg:
ip_tables: (C) 2000-2002 Netfilter core team
ip_conntrack version 2.1 (8192 buckets, 65536 max) - 304 bytes per conntrack
r8169 Gigabit Ethernet driver 1.6LK loaded
ACPI: PCI interrupt 0000:00:0b.0[A] -> GSI 16 (level, low) -> IRQ 16
divert: allocating divert_blk for eth0
eth0: Identified chip type is 'RTL8169s/8110s'.
eth0: RTL8169 at 0xf88b6f00, 00:0c:76:b3:c2:43, IRQ 16 IRQ#16 thread RT prio: 40.
hm: ioapic cache empty for irq 16 (e:00000000/d:00010000) 0001a9c1
r8169: eth0: link up
Bluetooth: Core ver 2.7
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.5
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM ver 1.3
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM TTY layer initialized
IRQ#4 thread RT prio: 39.
parport_pc: Ignoring new-style parameters in presence of obsolete ones
parport0: PC-style at 0x378 (0x778) [PCSPP,TRISTATE,EPP]
parport0: irq 7 detected
lp0: using parport0 (polling).
lp0: console ready
NET: Registered protocol family 10
Disabled Privacy Extensions on device c03d7e80(lo)
IPv6 over IPv4 tunneling driver
divert: not allocating divert_blk for non-ethernet device sit0
eth0: no IPv6 routers present
[drm] Initialized radeon 1.11.0 20020828 on minor 0:
ACPI: PCI interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
agpgart: Found an AGP 3.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V3 device at 0000:00:00.0 into 4x mode
agpgart: Putting AGP V3 device at 0000:01:00.0 into 4x mode [drm] Loading R200 Microcode
IRQ#11 thread RT prio: 38.
irq 16: nobody cared!

 [<c0104173>] dump_stack+0x23/0x30 (20)
 [<c0147970>] __report_bad_irq+0x30/0xa0 (24)
 [<c0147a80>] note_interrupt+0x70/0xb0 (32)
 [<c01477dc>] do_hardirq+0x13c/0x150 (40)
 [<c0147889>] do_irqd+0x99/0xd0 (32)
 [<c0139fda>] kthread+0xaa/0xb0 (48)
 [<c0101335>] kernel_thread_helper+0x5/0x10 (153083924)
---------------------------

| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c014777a>] .... do_hardirq+0xda/0x150
.....[<c0147889>] .. ( <= do_irqd+0x99/0xd0)
.. [<c013c98d>] .... print_traces+0x1d/0x60
.....[<c0104173>] .. ( <= dump_stack+0x23/0x30)

handlers:
[<f892ce60>] (rtl8169_interrupt+0x0/0x140 [r8169]) Disabling IRQ #16
ACPI: PCI interrupt 0000:00:07.0[A] -> GSI 18 (level, low) -> IRQ 18 IRQ#18 thread RT prio: 37.
hm: ioapic cache empty for irq 18 (e:00000000/d:00010000) 0001a9c9

output of lspci -v for the card:
00:0b.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)

	Subsystem: Micro-Star International Co., Ltd.: Unknown device 702c
	Flags: bus master, 66Mhz, medium devsel, latency 32, IRQ 16
	I/O ports at d000 [size=cffa0000]
	Memory at cfffef00 (32-bit, non-prefetchable) [size=256]
	Expansion ROM at 00020000 [disabled]
	Capabilities: [dc] Power Management version 2

-- Fernando

Ingo Molnar

2004-11-24 21:25 UTC #913 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

Quoted 5 lines · Fernando Lopez-Lezcano

Using PREEMPT_DESKTOP I see a irq related problem with my network interface:
IRQ#11 thread RT prio: 38.
irq 16: nobody cared!

Quoted 7 lines

 [<c0104173>] dump_stack+0x23/0x30 (20)
 [<c0147970>] __report_bad_irq+0x30/0xa0 (24)
 [<c0147a80>] note_interrupt+0x70/0xb0 (32)
 [<c01477dc>] do_hardirq+0x13c/0x150 (40)
 [<c0147889>] do_irqd+0x99/0xd0 (32)
 [<c0139fda>] kthread+0xaa/0xb0 (48)
 [<c0101335>] kernel_thread_helper+0x5/0x10 (153083924)

does it otherwise get detected and does it work fine afterwards?

Ingo

Fernando Lopez-Lezcano

2004-11-24 23:03 UTC #914 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-9

Quoted 4 lines · Ingo Molnar

Quoted 2 lines · Fernando Lopez-Lezcano
Using PREEMPT_DESKTOP I see a irq related problem with my network interface:

(I don't think this is only with PREEMPT_DESKTOP, on a previous kernel _RT was giving the same error)

Quoted 11 lines

Quoted 9 lines

IRQ#11 thread RT prio: 38.
irq 16: nobody cared!

 [<c0104173>] dump_stack+0x23/0x30 (20)
 [<c0147970>] __report_bad_irq+0x30/0xa0 (24)
 [<c0147a80>] note_interrupt+0x70/0xb0 (32)
 [<c01477dc>] do_hardirq+0x13c/0x150 (40)
 [<c0147889>] do_irqd+0x99/0xd0 (32)
 [<c0139fda>] kthread+0xaa/0xb0 (48)
 [<c0101335>] kernel_thread_helper+0x5/0x10 (153083924)

does it otherwise get detected and does it work fine afterwards?

Anything network related activity hangs (for example, trying to ping). Otherwise the machine seems to work.

Same kernel on my laptop loads the network driver with no problems (different driver, e100), but has problems with the usb uhci-hcd driver. This is when the driver is loaded:

USB Universal Host Controller Interface driver v2.2
ACPI: PCI interrupt 0000:00:1d.0[A] -> GSI 9 (level, low) -> IRQ 9 uhci_hcd 0000:00:1d.0: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.0 to 64 uhci_hcd 0000:00:1d.0: irq 9, io base 0x1800
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1 hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 9
ACPI: PCI interrupt 0000:00:1d.1[B] -> GSI 9 (level, low) -> IRQ 9 uhci_hcd 0000:00:1d.1: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.1 to 64 uhci_hcd 0000:00:1d.1: irq 9, io base 0x1820
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2 hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 9
ACPI: PCI interrupt 0000:00:1d.2[C] -> GSI 9 (level, low) -> IRQ 9 uhci_hcd 0000:00:1d.2: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.2 to 64 uhci_hcd 0000:00:1d.2: irq 9, io base 0x1840
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3 hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
usb 3-1: new full speed USB device using uhci_hcd and address 2 Initializing USB Mass Storage driver...
scsi0 : SCSI emulation for USB Mass Storage devices
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb-storage: device found at 2
usb-storage: waiting for device to settle before scanning inserting floppy driver for 2.6.9-1.520.1rV0.7.30_9.ll.rhfc2.ccrma inserting floppy driver for 2.6.9-1.520.1rV0.7.30_9.ll.rhfc2.ccrma
Vendor: Sony Model: MSC-U03 Rev: 1.00
Type: Direct-Access ANSI SCSI revision: 00
Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0, type 0
usb-storage: device scan complete
Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0 uhci_hcd 0000:00:1d.0: remove, state 1
usb usb1: USB disconnect, address 1
Device not ready. Make sure there is a disc in the drive. Device not ready. Make sure there is a disc in the drive. inserting floppy driver for 2.6.9-1.520.1rV0.7.30_9.ll.rhfc2.ccrma uhci_hcd 0000:00:1d.0: USB bus 1 deregistered
uhci_hcd 0000:00:1d.1: remove, state 1
usb usb2: USB disconnect, address 1
Device not ready. Make sure there is a disc in the drive. Device not ready. Make sure there is a disc in the drive. inserting floppy driver for 2.6.9-1.520.1rV0.7.30_9.ll.rhfc2.ccrma

Also, after the driver is loaded I get a "subdevfs not supported in kernel" message when I try to:
mount -t usbdevfs usbdevfs /proc/bus/usb
But I do have (I think) the right kernel options for this to work...

And this is when I try to remove it:

uhci_hcd 0000:00:1d.1: USB bus 2 deregistered
uhci_hcd 0000:00:1d.2: remove, state 1
usb usb3: USB disconnect, address 1
usb 3-1: USB disconnect, address 2
rmmod/3999: BUG in do_drain at mm/slab.c:1522

 [<c0104173>] dump_stack+0x23/0x30 (20)
 [<c0151fc9>] do_drain+0xb9/0xc0 (44)
 [<c0151ece>] smp_call_function_all_cpus+0x2e/0x70 (28)
 [<c0151ff1>] drain_cpu_caches+0x21/0x90 (24)
 [<c0152079>] __cache_shrink+0x19/0x160 (36)
 [<c01522cf>] kmem_cache_destroy+0xaf/0x1c0 (28)
 [<e094657b>] scsi_destroy_command_freelist+0x6b/0xa0 [scsi_mod] (28)
 [<e0947917>] scsi_host_dev_release+0x37/0xa0 [scsi_mod] (36)
 [<c0271725>] device_release+0x85/0x90 (32)
 [<c01ef505>] kobject_cleanup+0x95/0xa0 (28)
 [<c01f00f5>] kref_put+0x45/0x100 (40)
 [<c01ef556>] kobject_put+0x26/0x30 (16)
 [<e0adfd18>] usb_stor_release_resources+0x78/0x150 [usb_storage] (24)
 [<e0ae0294>] storage_disconnect+0xa4/0xb1 [usb_storage] (20)
 [<c02ba207>] usb_unbind_interface+0x87/0x90 (28)
 [<c0272e46>] device_release_driver+0x86/0x90 (28)
 [<c0273109>] bus_remove_device+0x89/0xd0 (28)
 [<c0271c14>] device_del+0x74/0xb0 (28)
 [<c02c1fd8>] usb_disable_device+0xb8/0x100 (28)
 [<c02bcb76>] usb_disconnect+0xc6/0x1a0 (40)
 [<c02bcc18>] usb_disconnect+0x168/0x1a0 (40)
 [<c02c50d5>] usb_hcd_pci_remove+0x85/0x1c0 (36)
 [<c01fbec6>] pci_device_remove+0x46/0x50 (16)
 [<c0272e46>] device_release_driver+0x86/0x90 (28)
 [<c0272e7b>] driver_detach+0x2b/0x40 (20)
 [<c02733c1>] bus_remove_driver+0x71/0xc0 (28)
 [<c02739e9>] driver_unregister+0x19/0x30 (16)
 [<c01fc15c>] pci_unregister_driver+0x1c/0x30 (16)
 [<e0ac2147>] uhci_hcd_cleanup+0x17/0x68 [uhci_hcd] (16)
 [<c013eec6>] sys_delete_module+0x146/0x190 (96)
 [<c01032a1>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------

| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c0151ec3>] .... smp_call_function_all_cpus+0x23/0x70
.....[<c0151ff1>] .. ( <= drain_cpu_caches+0x21/0x90)
.. [<c013c98d>] .... print_traces+0x1d/0x60
.....[<c0104173>] .. ( <= dump_stack+0x23/0x30)

inserting floppy driver for 2.6.9-1.520.1rV0.7.30_9.ll.rhfc2.ccrma inserting floppy driver for 2.6.9-1.520.1rV0.7.30_9.ll.rhfc2.ccrma uhci_hcd 0000:00:1d.2: USB bus 3 deregistered rmmod/3999: BUG in do_drain at mm/slab.c:1522

 [<c0104173>] dump_stack+0x23/0x30 (20)
 [<c0151fc9>] do_drain+0xb9/0xc0 (44)
 [<c0151ece>] smp_call_function_all_cpus+0x2e/0x70 (28)
 [<c0151ff1>] drain_cpu_caches+0x21/0x90 (24)
 [<c0152079>] __cache_shrink+0x19/0x160 (36)
 [<c01522cf>] kmem_cache_destroy+0xaf/0x1c0 (28)
 [<e0ac2154>] uhci_hcd_cleanup+0x24/0x68 [uhci_hcd] (16)
 [<c013eec6>] sys_delete_module+0x146/0x190 (96)
 [<c01032a1>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------

-- Fernando

A message that is not in the archive (sent off-list or not yet received): 1100732223.3472.10.camel@localhost

Ingo Molnar

2004-11-18 14:52 UTC #916 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

Quoted 9 lines · Christian Meder

after successfully running the last couple of rt patches on my Dell Inspiron laptop I thought I'd give it a try on my combined vdr/router box which is probably more interesting from a rt point of view. This box is bridging wireless/ADSL and working as a digital vdr using the kernel DVB-S drivers.
I got the appended logging messages with the appended config. Is there anything else I should provide for debugging purposes or are the messages just harmless ?

the messages mean that i havent converted the bridge code's RCU locking to PREEMPT_RT yet. I've done this in the -V0.7.28-2 patch that i've just uploaded to:

http://redhat.com/~mingo/realtime-preempt/

does bridging work fine with this patch, and if yes, do you get any (other) warning messages?

Ingo

Christian Meder

2004-11-18 15:12 UTC #917 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

Quoted 12 lines · Ingo Molnar

Quoted 9 lines · Christian Meder
after successfully running the last couple of rt patches on my Dell Inspiron laptop I thought I'd give it a try on my combined vdr/router box which is probably more interesting from a rt point of view. This box is bridging wireless/ADSL and working as a digital vdr using the kernel DVB-S drivers.
I got the appended logging messages with the appended config. Is there anything else I should provide for debugging purposes or are the messages just harmless ?

Quoted 8 lines

the messages mean that i havent converted the bridge code's RCU locking to PREEMPT_RT yet. I've done this in the -V0.7.28-2 patch that i've just uploaded to:
http://redhat.com/~mingo/realtime-preempt/
does bridging work fine with this patch, and if yes, do you get any (other) warning messages?

Thanks, will try soon and report. There was another trace in my log of the vdr/router box which seemed unrelated to the bridging traces.

Christian

Nov 17 22:44:41 verena kernel: dvb-ttpci: found av7110-0.
Nov 17 22:44:43 verena kernel: ves1x93: Detected ves1893a rev2
Nov 17 22:44:43 verena kernel: DVB: registering frontend 0 (VES1893)...
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel:
==========================================
Nov 17 22:44:44 verena kernel: [ BUG: lock recursion deadlock detected!
|
Nov 17 22:44:44 verena kernel:
------------------------------------------
Nov 17 22:44:44 verena kernel: already locked:  [c9c8fa10] {&fe->sem}
Nov 17 22:44:44 verena kernel: .. held by:               vdr: 2147
[ca3a6950, 118]
Nov 17 22:44:44 verena kernel: ... acquired at:  dvb_frontend_start
+0x75/0x100
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: ------------------------------
Nov 17 22:44:44 verena kernel: | showing all locks held by: |  (vdr/2147
[ca3a6950, 118]):
Nov 17 22:44:44 verena kernel: ------------------------------
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #001:             [c9c8fa10] {&fe->sem}
Nov 17 22:44:44 verena kernel: ... acquired at:  dvb_frontend_start
+0x75/0x100
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #002:             [c045e444]
{kernel_sem.lock}
Nov 17 22:44:44 verena kernel: ... acquired at:  lock_kernel+0x27/0x50
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: -{current task's
backtrace}----------------->
Nov 17 22:44:44 verena kernel:  [check_deadlock+608/640] check_deadlock
+0x260/0x280 (8)
Nov 17 22:44:44 verena kernel:  [dvb_frontend_ioctl+171/832]
dvb_frontend_ioctl+0xab/0x340 (28)
Nov 17 22:44:44 verena kernel:  [dvb_frontend_ioctl+171/832]
dvb_frontend_ioctl+0xab/0x340 (8)
Nov 17 22:44:44 verena kernel:  [task_blocks_on_lock+243/256]
task_blocks_on_lock+0xf3/0x100 (12)
Nov 17 22:44:44 verena kernel:  [dvb_frontend_ioctl+171/832]
dvb_frontend_ioctl+0xab/0x340 (16)
Nov 17 22:44:44 verena kernel:  [__down_interruptible+534/1072]
__down_interruptible+0x216/0x430 (4)
Nov 17 22:44:44 verena kernel:  [dvb_frontend_ioctl+171/832]
dvb_frontend_ioctl+0xab/0x340 (4)
Nov 17 22:44:44 verena kernel:  [pg0+264556013/1068020736]
av7110_send_fw_cmd+0x4d/0xc0 [dvb_ttpci] (12)
Nov 17 22:44:44 verena kernel:  [up+190/256] up+0xbe/0x100 (24)
Nov 17 22:44:44 verena kernel:  [down_interruptible+173/480]
down_interruptible+0xad/0x1e0 (36)
Nov 17 22:44:44 verena kernel:  [lru_cache_add_active+13/64]
lru_cache_add_active+0xd/0x40 (4)
Nov 17 22:44:44 verena kernel:  [dvb_frontend_ioctl+171/832]
dvb_frontend_ioctl+0xab/0x340 (40)
Nov 17 22:44:44 verena kernel:  [__kmalloc+133/272] __kmalloc+0x85/0x110
(52)
Nov 17 22:44:44 verena kernel:  [dvb_usercopy+132/272] dvb_usercopy
+0x84/0x110 (32)
Nov 17 22:44:44 verena kernel:  [__down+470/800] __down+0x1d6/0x320 (44)
Nov 17 22:44:44 verena kernel:  [lock_kernel+39/80] lock_kernel
+0x27/0x50 (4)
Nov 17 22:44:44 verena kernel:  [__down_mutex+470/864] __down_mutex
+0x1d6/0x360 (12)
Nov 17 22:44:44 verena kernel:  [fget+47/96] fget+0x2f/0x60 (4)
Nov 17 22:44:44 verena kernel:  [__down_mutex+470/864] __down_mutex
+0x1d6/0x360 (4)
Nov 17 22:44:44 verena kernel:  [kmem_cache_free+53/208] kmem_cache_free
+0x35/0xd0 (4)
Nov 17 22:44:44 verena kernel:  [down+173/480] down+0xad/0x1e0 (48)
Nov 17 22:44:44 verena kernel:  [fget+72/96] fget+0x48/0x60 (16)
Nov 17 22:44:44 verena kernel:  [dvb_generic_ioctl+62/80]
dvb_generic_ioctl+0x3e/0x50 (24)
Nov 17 22:44:44 verena kernel:  [dvb_frontend_ioctl+0/832]
dvb_frontend_ioctl+0x0/0x340 (8)
Nov 17 22:44:44 verena kernel:  [sys_ioctl+184/528] sys_ioctl+0xb8/0x210
(12)
Nov 17 22:44:44 verena kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
(36)
Nov 17 22:44:44 verena kernel: ---------------------------
Nov 17 22:44:44 verena kernel: | preempt count: 00000002 ]
Nov 17 22:44:44 verena kernel: | 2-level deep critical section nesting:
Nov 17 22:44:44 verena kernel: ----------------------------------------
Nov 17 22:44:44 verena kernel: .. [__down_interruptible+1057/1072] ....
__down_interruptible+0x421/0x430
Nov 17 22:44:44 verena kernel: .....[<00000000>] ..   ( <= 0x0)
Nov 17 22:44:44 verena kernel: .. [print_traces+13/64] .... print_traces
+0xd/0x40
Nov 17 22:44:44 verena kernel: .....[<00000000>] ..   ( <= 0x0)
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: showing all tasks:
Nov 17 22:44:44 verena kernel: s            init:    1 [c1239370, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           IRQ 0:    2 [c1238d00,  50]
(not blocked)
Nov 17 22:44:44 verena kernel: s     ksoftirqd/0:    3 [c1238690, 105]
(not blocked)
Nov 17 22:44:44 verena kernel: s       desched/0:    4 [c1238020, 105]
(not blocked)
Nov 17 22:44:44 verena kernel: s        events/0:    5 [cf7db390,  98]
(not blocked)
Nov 17 22:44:44 verena kernel: s         khelper:    6 [cf7dad20, 113]
(not blocked)
Nov 17 22:44:44 verena kernel: s         kthread:   11 [cf7da6b0, 110]
(not blocked)
Nov 17 22:44:44 verena kernel: s          kacpid:   19 [cf7da040, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s          IRQ 11:   20 [cf64d3b0,  58]
(not blocked)
Nov 17 22:44:44 verena kernel: s       kblockd/0:   91 [cf64cd40, 110]
(not blocked)
Nov 17 22:44:44 verena kernel: s           khubd:  104 [cf64c6d0, 115]
(not blocked)
Nov 17 22:44:44 verena kernel: s         pdflush:  183 [cf64c060, 117]
(not blocked)
Nov 17 22:44:44 verena kernel: s         pdflush:  184 [cf6c53d0, 115]
(not blocked)
Nov 17 22:44:44 verena kernel: s           aio/0:  186 [cf6c46f0, 112]
(not blocked)
Nov 17 22:44:44 verena kernel: s         kswapd0:  185 [cf6c4d60, 125]
(not blocked)
Nov 17 22:44:44 verena kernel: s           IRQ 8:  775 [cf6c4080,  56]
(not blocked)
Nov 17 22:44:44 verena kernel: s          IRQ 12:  785 [c139ed80,  54]
(not blocked)
Nov 17 22:44:44 verena kernel: s           IRQ 7:  804 [c139e710, 112]
(not blocked)
Nov 17 22:44:44 verena kernel: s         kseriod:  779 [c139f3f0, 125]
(not blocked)
Nov 17 22:44:44 verena kernel: s           IRQ 6:  807 [c139e0a0,  51]
(not blocked)
Nov 17 22:44:44 verena kernel: s          IRQ 14:  821 [c13e9450,  52]
(not blocked)
Nov 17 22:44:44 verena kernel: s          IRQ 15:  824 [c13dc0e0,  53]
(not blocked)
Nov 17 22:44:44 verena kernel: s       scsi_eh_0:  837 [c13dcdc0, 117]
blocked on: [c13e1fac] {sem.lock}
Nov 17 22:44:44 verena kernel: .. held by:         scsi_eh_0:  837
[c13dcdc0, 117]
Nov 17 22:44:44 verena kernel: ... acquired at:  scsi_error_handler
+0x50/0x100
Nov 17 22:44:44 verena kernel: s           IRQ 1:  864 [c13cd410,  55]
(not blocked)
Nov 17 22:44:44 verena kernel: s       kjournald:  903 [c13e8770, 115]
(not blocked)
Nov 17 22:44:44 verena kernel: s           IRQ 5: 1074 [c13dc750,  57]
(not blocked)
Nov 17 22:44:44 verena kernel: s          IRQ 10: 1116 [c13e8100,  59]
(not blocked)
Nov 17 22:44:44 verena kernel: s       kjournald: 1153 [cf439470, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s         portmap: 1320 [ce6ace20, 115]
(not blocked)
Nov 17 22:44:44 verena kernel: s           IRQ 4: 1328 [c13cc0c0, 112]
(not blocked)
Nov 17 22:44:44 verena kernel: s           IRQ 3: 1331 [cf438e00, 112]
(not blocked)
Nov 17 22:44:44 verena kernel: s         syslogd: 1433 [c13ccda0, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           klogd: 1436 [ce6ac7b0, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s            pppd: 1448 [cdbb2e40, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           pppoe: 1451 [c13dd430, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           gnugk: 1484 [c13e8de0, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           gnugk: 1508 [cd366e60, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s           gnugk: 1509 [cd3667f0, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           gnugk: 1510 [cd366180, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s           gnugk: 1511 [cc5014f0, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s           inetd: 1489 [cdbb34b0, 122]
(not blocked)
Nov 17 22:44:44 verena kernel: s         jabberd: 1492 [ce6ad490, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           ip-up: 1502 [cd3674d0, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s       run-parts: 1504 [cdbb27d0, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s            exim: 1506 [cdbb2160, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s            exim: 1507 [cf438120, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s         jabberd: 1516 [cc5001a0, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s         jabberd: 1519 [c13cc730, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1907 [cc500e80, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1908 [cc58eea0, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1909 [cc58e830, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1910 [cf438790, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1911 [cc58e1c0, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1912 [cc7a7530, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1913 [cc7a6ec0, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s            nfsd: 1914 [cc7a6850, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s           lockd: 1916 [cc7a61e0, 121]
(not blocked)
Nov 17 22:44:44 verena kernel: s        rpciod/0: 1917 [cc783550, 111]
(not blocked)
Nov 17 22:44:44 verena kernel: s      rpc.mountd: 1920 [cc782ee0, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s      postmaster: 1962 [cc58f510, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s      postmaster: 1966 [cc7fc890, 121]
(not blocked)
Nov 17 22:44:44 verena kernel: s      postmaster: 1967 [cc7fcf00, 122]
(not blocked)
Nov 17 22:44:44 verena kernel: s            slpd: 1981 [cc7fd570, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s            sshd: 1987 [cc500810, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s   vdrconvert.sh: 1996 [cbcd8f20, 135]
(not blocked)
Nov 17 22:44:44 verena kernel: s       rpc.statd: 2004 [cc782870, 117]
(not blocked)
Nov 17 22:44:44 verena kernel: s            Xvfb: 2005 [cbcd9590, 131]
(not blocked)
Nov 17 22:44:44 verena kernel: s             atd: 2018 [cbcd8240, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s            cron: 2021 [cc782200, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s           fcron: 2024 [cbee6f40, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s          apache: 2031 [cbee68d0, 116]
(not blocked)
Nov 17 22:44:44 verena kernel: s             vdr: 2039 [cb1f4f60, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s           getty: 2040 [cb1f48f0, 117]
(not blocked)
Nov 17 22:44:44 verena kernel: s           getty: 2043 [ce6ac140, 117]
(not blocked)
Nov 17 22:44:44 verena kernel: s           getty: 2044 [cbee75b0, 117]
(not blocked)
Nov 17 22:44:44 verena kernel: s           getty: 2045 [cbcd88b0, 117]
(not blocked)
Nov 17 22:44:44 verena kernel: s           getty: 2046 [cb1f55d0, 117]
(not blocked)
Nov 17 22:44:44 verena kernel: s           lircd: 2059 [cbee6260, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s     wait2enc.py: 2060 [cb2815f0, 135]
(not blocked)
Nov 17 22:44:44 verena kernel: s          apache: 2063 [cb280f80, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s          apache: 2064 [cb280910, 120]
(not blocked)
Nov 17 22:44:44 verena kernel: s          apache: 2065 [cb2802a0, 121]
(not blocked)
Nov 17 22:44:44 verena kernel: s          apache: 2066 [caead610, 122]
(not blocked)
Nov 17 22:44:44 verena kernel: s          apache: 2067 [caeacfa0, 123]
(not blocked)
Nov 17 22:44:44 verena kernel: s    vdradmind.pl: 2078 [caeac2c0, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s          runvdr: 2079 [cb1f4280, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: s         arm_mon: 2119 [ca3a7630, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: s           sleep: 2139 [c9de0fe0, 136]
(not blocked)
Nov 17 22:44:44 verena kernel: s             vdr: 2147 [ca3a6950, 118]
(not blocked)
Nov 17 22:44:44 verena kernel: R             vdr: 2148 [caeac930, 119]
(not blocked)
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: ---------------------------
Nov 17 22:44:44 verena kernel: | showing all locks held: |
Nov 17 22:44:44 verena kernel: ---------------------------
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #001:             [c053ee6c]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #002:             [c053ea90]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #003:             [c053ec64]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #004:             [c053f4b4]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #005:             [c053f0d8]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #006:             [c053f2ac]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #007:             [c053fafc]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #008:             [c053f720]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #009:             [c053f8f4]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #010:             [c0540144]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #011:             [c053fd68]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #012:             [c053ff3c]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #013:             [c054078c]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #014:             [c05403b0]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #015:             [c0540584]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #016:             [c0540dd4]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #017:             [c05409f8]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #018:             [c0540bcc]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #019:             [c054141c]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #020:             [c0541040]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #021:             [c0541214]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #022:             [c0541a64]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #023:             [c0541688]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #024:             [c054185c]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #025:             [c05420ac]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #026:             [c0541cd0]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #027:             [c0541ea4]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #028:             [c05426f4]
{&hwif->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x8e/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #029:             [c0542318]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #030:             [c05424ec]
{&drive->gendev_rel_sem}
Nov 17 22:44:44 verena kernel: .. held by:           swapper:    0
[c044cd80, 140]
Nov 17 22:44:44 verena kernel: ... acquired at:  init_hwif_data
+0x160/0x180
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #031:             [ccd3dcf0]
{&tty->atomic_read}
Nov 17 22:44:44 verena kernel: .. held by:             getty: 2040
[cb1f48f0, 117]
Nov 17 22:44:44 verena kernel: ... acquired at:  read_chan+0x6f1/0x750
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #032:             [cd66acf0]
{&tty->atomic_read}
Nov 17 22:44:44 verena kernel: .. held by:             getty: 2044
[cbee75b0, 117]
Nov 17 22:44:44 verena kernel: ... acquired at:  read_chan+0x6f1/0x750
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #033:             [cda85cf0]
{&tty->atomic_read}
Nov 17 22:44:44 verena kernel: .. held by:             getty: 2045
[cbcd88b0, 117]
Nov 17 22:44:44 verena kernel: ... acquired at:  read_chan+0x6f1/0x750
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #034:             [cdfe6cf0]
{&tty->atomic_read}
Nov 17 22:44:44 verena kernel: .. held by:             getty: 2046
[cb1f55d0, 117]
Nov 17 22:44:44 verena kernel: ... acquired at:  read_chan+0x6f1/0x750
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #035:             [cb22ccf0]
{&tty->atomic_read}
Nov 17 22:44:44 verena kernel: .. held by:             getty: 2043
[ce6ac140, 117]
Nov 17 22:44:44 verena kernel: ... acquired at:  read_chan+0x6f1/0x750
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #036:             [c9c8fa10] {&fe->sem}
Nov 17 22:44:44 verena kernel: .. held by:               vdr: 2147
[ca3a6950, 118]
Nov 17 22:44:44 verena kernel: ... acquired at:  dvb_frontend_start
+0x75/0x100
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: #037:             [c045e444]
{kernel_sem.lock}
Nov 17 22:44:44 verena kernel: .. held by:               vdr: 2147
[ca3a6950, 118]
Nov 17 22:44:44 verena kernel: ... acquired at:  lock_kernel+0x27/0x50
Nov 17 22:44:44 verena kernel:
=============================================
Nov 17 22:44:44 verena kernel: 
Nov 17 22:44:44 verena kernel: [ turning off deadlock detection. Please

report this trace. ]

Signature · 6 lines

Christian Meder, email: [email protected]

The Way-Seeking Mind of a tenzo is actualized 
by rolling up your sleeves.

                (Eihei Dogen Zenji)

Ingo Molnar

2004-11-18 20:29 UTC #918 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

Quoted 2 lines · Christian Meder

Thanks, will try soon and report. There was another trace in my log of the vdr/router box which seemed unrelated to the bridging traces.

does the patch below fix that message?

Ingo

drivers/media/dvb/dvb-core/dvb_frontend.c renamed from drivers/media/dvb/dvb-core/dvb_frontend.c.orig2 +3 -3

@@ -658,7 +658,7 @@ static void dvb_frontend_stop (struct dv
 		printk("dvb_frontend_stop: thread PID %d already died\n",
 				fe->thread_pid);
 		/* make sure the mutex was not held by the thread */
-		init_MUTEX (&fe->sem);
+		sema_init_nocheck (&fe->sem, 1);
 		return;
 	}

@@ -1127,10 +1127,10 @@ dvb_register_frontend (int (*ioctl) (str
 
 	memset (fe, 0, sizeof (struct dvb_frontend_data));
 
-	init_MUTEX (&fe->sem);
+	sema_init_nocheck (&fe->sem, 1);
 	init_waitqueue_head (&fe->wait_queue);
 	init_waitqueue_head (&fe->events.wait_queue);
-	init_MUTEX (&fe->events.sem);
+	sema_init_nocheck (&fe->events.sem, 1);
 	fe->events.eventw = fe->events.eventr = 0;
 	fe->events.overflow = 0;
 	fe->module = module;

Christian Meder

2004-11-18 22:45 UTC #919 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

Quoted 6 lines · Ingo Molnar

Quoted 2 lines · Christian Meder
Thanks, will try soon and report. There was another trace in my log of the vdr/router box which seemed unrelated to the bridging traces.
does the patch below fix that message?

Yesss. This patch applied on top of 0.7.29-0 removes all warning messages (bridging/dvb) in my vdr/router box.

Thanks,

Christian

Signature · 6 lines

Christian Meder, email: [email protected]

The Way-Seeking Mind of a tenzo is actualized 
by rolling up your sleeves.

                (Eihei Dogen Zenji)

Christian Meder

2004-11-18 22:38 UTC #920 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

In reply to Ingo Molnar

Quoted 12 lines · Ingo Molnar

Quoted 9 lines · Christian Meder
after successfully running the last couple of rt patches on my Dell Inspiron laptop I thought I'd give it a try on my combined vdr/router box which is probably more interesting from a rt point of view. This box is bridging wireless/ADSL and working as a digital vdr using the kernel DVB-S drivers.
I got the appended logging messages with the appended config. Is there anything else I should provide for debugging purposes or are the messages just harmless ?

Quoted 8 lines

the messages mean that i havent converted the bridge code's RCU locking to PREEMPT_RT yet. I've done this in the -V0.7.28-2 patch that i've just uploaded to:
http://redhat.com/~mingo/realtime-preempt/
does bridging work fine with this patch, and if yes, do you get any (other) warning messages?

I'm running 0.7.29 now on the vdr/router box and bridging is working fine and there are no new warning messages.

Thanks,

Christian

Signature · 6 lines

Christian Meder, email: [email protected]

The Way-Seeking Mind of a tenzo is actualized 
by rolling up your sleeves.

                (Eihei Dogen Zenji)

Adam Heath

2004-11-08 22:22 UTC #921 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.19

In reply to Ingo Molnar

Quoted 6 lines · Ingo Molnar

i have released the -V0.7.19 Real-Time Preemption patch, which can be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/

Quoted 8 lines

this release includes fixes only.
Changes since -V0.7.18:
- fixed a merge bug introduced in -V0.7.18, breaking bit-spinlocks used by ext3's journalling code. This could/should fix the kjournald crash reported by Adam Heath, Gunther Persoons and Eran Mann. Bug triggered on !SMP kernels only.

The last kernel I tried was v0.7.13, so I doubt it was a recent introduction.

Will try something newer soon.

Gene Heskett

2004-10-22 21:52 UTC #922 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-U10

In reply to Ingo Molnar

Quoted 9 lines · Ingo Molnar

i have released the -U10 Real-Time Preemption patch, which can be downloaded from:
http://redhat.com/~mingo/realtime-preempt/
this is purely a rebasing of -U9.3 to 2.6.9-mm1.
to create a -U10 tree from scratch, the patching order is:

Quoted 8 lines

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
+
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9/2.
6.9-mm1/2.6.9-mm1.bz2 +
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-mm
1-U10
Ingo

I teased 2 gallons of latex primer out of the cans, making it jump up onto some outbuildings, so I'm a bit late with the report, which is that it spit out a whole flurry of scheduleing while atomic messages when I tried it just now, and finally hung, requiring I exersize the reset button long before it got to rc.sysinit. The best I could do might be a screen snapshot, but I haven't figured out howto shut the flippin flash off and use the available light, or try and scribble it all down. And that would lead to violent deaths if I was a doctor trying to write prescriptions. :-(

As an experiment to see if I could lay bleeding edge, I bled. :)

Signature · 8 lines

Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.28% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Alexander Batyrshin

2004-10-22 18:46 UTC #923 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U9.3

In reply to Ingo Molnar

Hi Ingo,
U9.3 (defconfig) died with trace:

------------[ cut here ]------------
kernel BUG at kernel/sched.c:784!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU:    0
EIP:    0060:[<c011873d>]    Not tainted VLI
EFLAGS: 00010002   (2.6.9-rc4-mm1-RT-U9.3)

EIP is at resched_task+0x80/0x8a
eax: 00000001 ebx: c1674000 ecx: dfb578f0 edx: 00000000
esi: c16712c0 edi: 00000000 ebp: c167bc48 esp: c167bc3c
ds: 007b es: 007b ss: 0068 preempt: 00000003
Process ksoftirqd/0 (pid: 3, threadinfo=c167a000 task=c1670660)

Stack: c1403820 c1403820 00000000 c167bc94 c0118b78 c16712c0 c1433820 00000000
       00000000 00000000 00000001 00000100 c1433820 c1433820 00000001 
00000000
       00000000 00000001 00000082 00000001 de9d3e60 00000001 c167bcd8 
c0133c7b
Call Trace:
 [<c0118b78>] try_to_wake_up+0x1f3/0x26b (20)
 [<c0133c7b>] autoremove_wake_function+0x2f/0x57 (76)
 [<c011a766>] __wake_up_common+0x3f/0x5e (28)
 [<c011a7d0>] __wake_up+0x4b/0x62 (40)
 [<c034e08c>] sock_def_readable+0x8e/0x90 (52)
 [<c0388dbe>] tcp_child_process+0xe6/0xec (28)
 [<c0384df7>] tcp_v4_do_rcv+0x123/0x162 (36)
 [<c0134079>] _mutex_lock+0x2c/0x3b (16)
 [<c0385513>] tcp_v4_rcv+0x6dd/0x92c (16)
 [<c03c0e30>] svc_revisit+0x27/0x154 (48)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (40)
 [<c0368609>] ip_local_deliver_finish+0xb6/0x1b2 (4)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (12)
 [<c035f607>] nf_hook_slow+0xdc/0x12e (20)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (28)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (28)
 [<c0367fcb>] ip_local_deliver+0x208/0x226 (4)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (24)
 [<c0368828>] ip_rcv_finish+0x123/0x2b3 (20)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (32)
 [<c035f607>] nf_hook_slow+0xdc/0x12e (20)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (28)
 [<c036846a>] ip_rcv+0x481/0x56a (32)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (24)
 [<c0354e68>] netif_receive_skb+0x117/0x1dd (28)
 [<c0354ff7>] process_backlog+0xc9/0x1cb (36)
 [<c03551b2>] net_rx_action+0xb9/0x1ed (48)
 [<c01242d9>] ___do_softirq+0xe1/0x130 (36)
 [<c0124923>] ksoftirqd+0x0/0xda (40)
 [<c01243fb>] _do_softirq+0x4b/0x4d (4)
 [<c01249c5>] ksoftirqd+0xa2/0xda (16)
 [<c013376d>] kthread+0xb7/0xbd (24)
 [<c01336b6>] kthread+0x0/0xbd (28)
 [<c0103375>] kernel_thread_helper+0x5/0xb (16)

preempt count: 00000004
. 4-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x1d/0xa5 [<00000000>] / (0x0 [<00000000>]) .. entry 2: _spin_lock+0x19/0x6d [<00000000>] / (0x0 [<00000000>]) .. entry 3: _spin_lock_irqsave+0x1d/0xa5 [<00000000>] / (0x0 [<00000000>]) .. entry 4: print_traces+0x17/0x4e [<00000000>] / (0x0 [<00000000>])

Ingo Molnar

2004-10-22 20:10 UTC #924 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U9.3

how about U10.3? That has the BKL fix too.

Ingo

Quoted 63 lines · Alexander Batyrshin

Hi Ingo,
U9.3 (defconfig) died with trace:

------------[ cut here ]------------
kernel BUG at kernel/sched.c:784!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU:    0
EIP:    0060:[<c011873d>]    Not tainted VLI
EFLAGS: 00010002   (2.6.9-rc4-mm1-RT-U9.3)

Stack: c1403820 c1403820 00000000 c167bc94 c0118b78 c16712c0 c1433820 
00000000
       00000000 00000000 00000001 00000100 c1433820 c1433820 00000001 
00000000
       00000000 00000001 00000082 00000001 de9d3e60 00000001 c167bcd8 
c0133c7b
Call Trace:
 [<c0118b78>] try_to_wake_up+0x1f3/0x26b (20)
 [<c0133c7b>] autoremove_wake_function+0x2f/0x57 (76)
 [<c011a766>] __wake_up_common+0x3f/0x5e (28)
 [<c011a7d0>] __wake_up+0x4b/0x62 (40)
 [<c034e08c>] sock_def_readable+0x8e/0x90 (52)
 [<c0388dbe>] tcp_child_process+0xe6/0xec (28)
 [<c0384df7>] tcp_v4_do_rcv+0x123/0x162 (36)
 [<c0134079>] _mutex_lock+0x2c/0x3b (16)
 [<c0385513>] tcp_v4_rcv+0x6dd/0x92c (16)
 [<c03c0e30>] svc_revisit+0x27/0x154 (48)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (40)
 [<c0368609>] ip_local_deliver_finish+0xb6/0x1b2 (4)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (12)
 [<c035f607>] nf_hook_slow+0xdc/0x12e (20)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (28)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (28)
 [<c0367fcb>] ip_local_deliver+0x208/0x226 (4)
 [<c0368553>] ip_local_deliver_finish+0x0/0x1b2 (24)
 [<c0368828>] ip_rcv_finish+0x123/0x2b3 (20)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (32)
 [<c035f607>] nf_hook_slow+0xdc/0x12e (20)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (28)
 [<c036846a>] ip_rcv+0x481/0x56a (32)
 [<c0368705>] ip_rcv_finish+0x0/0x2b3 (24)
 [<c0354e68>] netif_receive_skb+0x117/0x1dd (28)
 [<c0354ff7>] process_backlog+0xc9/0x1cb (36)
 [<c03551b2>] net_rx_action+0xb9/0x1ed (48)
 [<c01242d9>] ___do_softirq+0xe1/0x130 (36)
 [<c0124923>] ksoftirqd+0x0/0xda (40)
 [<c01243fb>] _do_softirq+0x4b/0x4d (4)
 [<c01249c5>] ksoftirqd+0xa2/0xda (16)
 [<c013376d>] kthread+0xb7/0xbd (24)
 [<c01336b6>] kthread+0x0/0xbd (28)
 [<c0103375>] kernel_thread_helper+0x5/0xb (16)

Adam Heath

2004-10-19 19:04 UTC #925 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U6

In reply to Ingo Molnar

Quoted 1 line · Ingo Molnar

i have released the -U6 Real-Time Preemption patch:

(xterm/1219/CPU#0): new 39188 us maximum-latency critical section.
=> started at timestamp 71898423: <call_console_drivers+0x76/0x140>
=> ended at timestamp 71937611: <finish_task_switch+0x43/0xb0>

 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c01324de>] check_preempt_timing+0x15e/0x270
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c02a5717>] __sched_text_start+0x2d7/0x5d0
 [<c02a684f>] down_write+0x12f/0x1e0
 [<c0113104>] mcount+0x14/0x18
 [<c02a684f>] down_write+0x12f/0x1e0
 [<c014dd45>] remove_vm_struct+0x45/0xb0
 [<c014fde2>] exit_mmap+0xf2/0x120
 [<c0119b96>] mmput+0x46/0xf0
 [<c0166d8f>] exec_mmap+0xaf/0x140
 [<c0166ffd>] flush_old_exec+0xfd/0x7f0
 [<c015b917>] vfs_read+0xe7/0x140
 [<c0113104>] mcount+0x14/0x18
 [<c0185d7f>] load_elf_binary+0x30f/0xbd0
 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c01db739>] up_read+0xf9/0x140
 [<c01323d8>] check_preempt_timing+0x58/0x270
 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c0131c6d>] __mcount+0x1d/0x20
 [<c0185a70>] load_elf_binary+0x0/0xbd0
 [<c0167aba>] search_binary_handler+0x19a/0x2e0
 [<c0167dac>] do_execve+0x1ac/0x260
 [<c0104b07>] sys_execve+0x47/0xd0
 [<c0106013>] syscall_call+0x7/0xb

preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: __sched_text_start+0x3b/0x5d0 / (down_write+0x12f/0x1e0) .. entry 2: print_traces+0x1d/0x80 / (dump_stack+0x23/0x30)

=> dump-end timestamp 71938197

Michal Schmidt

2004-10-19 21:10 UTC #926 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U6

In reply to Ingo Molnar

Quoted 9 lines · Ingo Molnar

i have released the -U6 Real-Time Preemption patch:
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U6
[...]
- deadlock bug: fix networking deadlock reported by Matthew L Foster. Restructured the way the RT-RCU locking of ptype_lock is done - it's cleaner and more obvious now (besides being correct). This could also fix the deadlock reported by Michal Schmidt.

Yes, this fixed the deadlock for me.
I'm now going to try -U7.

Michal

Adam Heath

2004-10-20 16:35 UTC #927 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U6

In reply to Ingo Molnar

Quoted 4 lines · Ingo Molnar

i have released the -U6 Real-Time Preemption patch:
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U6

Got these high-latency values during the night on U6(haven't booted U8 yet).

IRQ 5/431/CPU#0): 612 us critical section violates 100 us threshold.
=> started at timestamp 4167601478: <call_console_drivers+0x76/0x140>
=> ended at timestamp 4167602090: <finish_task_switch+0x43/0xb0>

 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c01324de>] check_preempt_timing+0x15e/0x270
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c02a5717>] __sched_text_start+0x2d7/0x5d0
 [<c0113104>] mcount+0x14/0x18
 [<c013bdfa>] do_irqd+0x5a/0x80
 [<c01309ea>] kthread+0xaa/0xb0
 [<c013bda0>] do_irqd+0x0/0x80
 [<c0130940>] kthread+0x0/0xb0
 [<c0104099>] kernel_thread_helper+0x5/0xc

preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: __sched_text_start+0x3b/0x5d0 / (do_irqd+0x5a/0x80) .. entry 2: print_traces+0x1d/0x80 / (dump_stack+0x23/0x30)

=> dump-end timestamp 4167602447

(IRQ 5/431/CPU#0): 34875 us critical section violates 100 us threshold.
=> started at timestamp 4167608224: <call_console_drivers+0x76/0x140>
=> ended at timestamp 4167643099: <finish_task_switch+0x43/0xb0>

 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c01324de>] check_preempt_timing+0x15e/0x270
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c02a5717>] __sched_text_start+0x2d7/0x5d0
 [<c0113104>] mcount+0x14/0x18
 [<c013bdfa>] do_irqd+0x5a/0x80
 [<c01309ea>] kthread+0xaa/0xb0
 [<c013bda0>] do_irqd+0x0/0x80
 [<c0130940>] kthread+0x0/0xb0
 [<c0104099>] kernel_thread_helper+0x5/0xc

preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: __sched_text_start+0x3b/0x5d0 / (do_irqd+0x5a/0x80) .. entry 2: print_traces+0x1d/0x80 / (dump_stack+0x23/0x30)

=> dump-end timestamp 4167643459

(IRQ 1/18/CPU#0): 30560 us critical section violates 100 us threshold.
=> started at timestamp 4167647182: <call_console_drivers+0x76/0x140>
=> ended at timestamp 4167677742: <finish_task_switch+0x43/0xb0>

 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c01324de>] check_preempt_timing+0x15e/0x270
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c02a5717>] __sched_text_start+0x2d7/0x5d0
 [<c0113104>] mcount+0x14/0x18
 [<c013bdfa>] do_irqd+0x5a/0x80
 [<c01309ea>] kthread+0xaa/0xb0
 [<c013bda0>] do_irqd+0x0/0x80
 [<c0130940>] kthread+0x0/0xb0
 [<c0104099>] kernel_thread_helper+0x5/0xc

preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: __sched_text_start+0x3b/0x5d0 / (do_irqd+0x5a/0x80) .. entry 2: print_traces+0x1d/0x80 / (dump_stack+0x23/0x30)

=> dump-end timestamp 4167678099

(bash/10595/CPU#0): 33546 us critical section violates 100 us threshold.
=> started at timestamp 4167681248: <call_console_drivers+0x76/0x140>
=> ended at timestamp 4167714794: <finish_task_switch+0x43/0xb0>

 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c01324de>] check_preempt_timing+0x15e/0x270
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c01327f0>] sub_preempt_count+0x60/0x90
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c0117ca3>] finish_task_switch+0x43/0xb0
 [<c02a5717>] __sched_text_start+0x2d7/0x5d0
 [<c02a684f>] down_write+0x12f/0x1e0
 [<c0113104>] mcount+0x14/0x18
 [<c02a684f>] down_write+0x12f/0x1e0
 [<c01182bb>] lock_kernel+0x2b/0x40
 [<c016f122>] sys_ioctl+0x52/0x230
 [<c0106013>] syscall_call+0x7/0xb

=> dump-end timestamp 4167715168

Kevin Hilman

2004-10-19 12:57 UTC #928 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U5

In reply to Ingo Molnar

Quoted 1 line · Ingo Molnar

i have released the -U5 Real-Time Preemption patch:

[...]

Quoted 5 lines

the generic semaphore implementation (which uses rwsems) makes it possible to use the deadlock detection mechanism for all the mutex types we currently have: semaphores, rw-semaphores, spinlock-mutexes and rwlock-mutexes. Another benefit is that PREEMPT_REALTIME becomes much more portable this way. (although it's still x86-only at the moment.)

Speaking of portability, is anyone yet working on ports to other platforms? I'm particularily interested in ARM.

If anyone has started on ARM, I'd be glad to help out.

Kevin
http://hilman.org/kevin/

Ingo Molnar

2004-10-19 13:02 UTC #929 lore raw

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U5

Quoted 8 lines · Kevin Hilman

Quoted 5 lines
the generic semaphore implementation (which uses rwsems) makes it possible to use the deadlock detection mechanism for all the mutex types we currently have: semaphores, rw-semaphores, spinlock-mutexes and rwlock-mutexes. Another benefit is that PREEMPT_REALTIME becomes much more portable this way. (although it's still x86-only at the moment.)
Speaking of portability, is anyone yet working on ports to other platforms? I'm particularily interested in ARM.

i'll do x64 a couple of days after stability has been reached. I dont know of any ARM efforts though.

a good starting point would be to enhance the generic-hardirqs framework for ARM. Generic-hardirqs is a portion of the PREEMPT_REALTIME patch that i've split out and submitted upstream, and which is expected to be merged into 2.6.10. The generic irq subsystem makes the irq-threading feature really simple and maintainable. So for PREEMPT_REALTIME to work on ARM the first step is to enable generic-hardirqs on ARM. (which is far from simple though.)

Ingo

Florian Schmidt

2004-10-15 11:07 UTC #930 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U1

In reply to Ingo Molnar

Quoted 4 lines · On Thu, 14 Oct 2004 16:31:31 +0200 Ingo Molnar <[email protected]> wrote:

i have released the -U1 PREEMPT_REALTIME patch:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U1

Ok,

with the help of Paul Davis i think i have found what's causing the jackd FP exception. It seems to be a bug in the kernel when PREEMPT_REALTIME is enabled:

~$ cat /proc/cpuinfo|grep cpu
cpu family : 6
cpu MHz : 0.001
cpuid level : 1

Mhz == 0.001? Hrmm. No wonder jackd was freaking out in its timing code.. The real cpu speed is 1.2ghz.

flo

P.S.: Will retry with U3, to see if this persists.

dmesg output:

Linux version 2.6.9-rc4-mm1-VP-U1-RT ([email protected]) (gcc version 3.3.5 (Debian 1:3.3.5-1)) #3 Thu Oct 14 22:40:26 CEST 2004 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000030000000 (usable)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ffee0000 - 00000000fff00000 (reserved)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
768MB LOWMEM available.
On node 0 totalpages: 196608
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 192512 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
DMI 2.3 present.
Built 1 zonelists
Initializing CPU#0
Kernel command line: BOOT_IMAGE=2.6.9-U0 ro root=1601
PID hash table entries: 4096 (order: 12, 65536 bytes)
Detected 1195.144 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 776332k/786432k available (1673k kernel code, 9644k reserved, 482k data, 340k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay loop... 2367.48 BogoMIPS (lpj=1183744)
Security Scaffold v1.0.0 initialized
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: After generic identify, caps: 0183f9ff c1c7f9ff 00000000 00000000
CPU: After vendor identify, caps: 0183f9ff c1c7f9ff 00000000 00000000
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 256K (64 bytes/line)
CPU: After all inits, caps: 0183f9ff c1c7f9ff 00000000 00000020
CPU: AMD Athlon(tm) Processor stepping 02
Enabling fast FPU save and restore... done.
Checking 'hlt' instruction... OK.
ksoftirqd started up.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfdb01, last bus=1
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
Uncovering SIS18 that hid as a SIS503 (compatible=1)
Enabling SiS 96x SMBus.
PCI: Using IRQ router SIS [1039/0018] at 0000:00:02.0
PCI: IRQ 0 for device 0000:00:02.1 doesn't match PIRQ mask - try pci=usepirqmask
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
Initializing Cryptographic API
Real Time Clock Driver v1.12
Non-volatile memory driver v1.2
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
SIS5513: IDE controller at PCI slot 0000:00:02.5
SIS5513: chipset revision 208
SIS5513: not 100% native mode: will probe irqs later
SIS5513: SiS735 ATA 100 (2nd gen) controller

    ide0: BM-DMA at 0xff00-0xff07, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xff08-0xff0f, BIOS settings: hdc:DMA, hdd:DMA

Probing IDE interface ide0...
hda: IC35L060AVER07-0, ATA DISK drive
elevator: using anticipatory as default io scheduler
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: ST340823A, ATA DISK drive
hdd: TDK CDRW121032, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
Probing IDE interface ide2...
ide2: Wait for ready failed before probe !
Probing IDE interface ide3...
ide3: Wait for ready failed before probe !
Probing IDE interface ide4...
ide4: Wait for ready failed before probe !
Probing IDE interface ide5...
ide5: Wait for ready failed before probe !
hda: max request size: 128KiB
hda: 120103200 sectors (61492 MB) w/1916KiB Cache, CHS=65535/16/63, UDMA(100)
hda: cache flushes not supported
hda: hda1 hda2 hda3
hdc: max request size: 128KiB
hdc: Host Protected Area detected.

	current capacity is 78165360 sectors (40020 MB)
	native  capacity is 78165361 sectors (40020 MB)

hdc: Host Protected Area disabled.
hdc: 78165361 sectors (40020 MB) w/1024KiB Cache, CHS=65535/16/63, UDMA(33)
hdc: cache flushes not supported
hdc: hdc1 hdc2
hdd: ATAPI 32X CD-ROM CD-R/RW drive, 2048kB Cache, DMA
Uniform CD-ROM driver Revision: 3.20
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
input: ImExPS/2 Logitech Explorer Mouse on isa0060/serio1
input: PC Speaker
NET: Registered protocol family 2
IP: routing cache hash table of 2048 buckets, 64Kbytes
TCP: Hash tables configured (established 65536 bind 37449)
NET: Registered protocol family 1
NET: Registered protocol family 17
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 340k freed
kjournald starting. Commit interval 5 seconds
Adding 289160k swap on /dev/hda3. Priority:-1 extents:1 EXT3-fs warning: maximal mount count reached, running e2fsck is recommended EXT3 FS on hdc1, internal journal
PCI: Found IRQ 5 for device 0000:00:0f.0
sis900.c: v1.08.07 11/02/2003
PCI: Found IRQ 10 for device 0000:00:03.0
eth0: Realtek RTL8201 PHY transceiver found at address 1.
eth0: Using transceiver found at address 1 as default
eth0: SiS 900 PCI Fast Ethernet at 0xdc00, IRQ 10, 00:d0:09:e9:c1:0f.
CSLIP: code copyright 1989 Regents of the University of California PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
kjournald starting. Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on hda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on hdc2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.

complete /proc/cpuinfo

processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 4
model name : AMD Athlon(tm) Processor
stepping : 2
cpu MHz : 0.001
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips : 2367.48

Ingo Molnar

2004-10-15 11:43 UTC #931 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U1

Quoted 15 lines · Florian Schmidt

Quoted 3 lines
i have released the -U1 PREEMPT_REALTIME patch:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U1
Ok,
with the help of Paul Davis i think i have found what's causing the jackd FP exception. It seems to be a bug in the kernel when PREEMPT_REALTIME is enabled:
~$ cat /proc/cpuinfo|grep cpu
cpu family : 6
cpu MHz : 0.001
cpuid level : 1

Quoted 6 lines

Mhz == 0.001? Hrmm. No wonder jackd was freaking out in its timing code.. The real cpu speed is 1.2ghz.
flo
P.S.: Will retry with U3, to see if this persists.

ah ... good eyes. Seems to be working fine here:

saturn:~> cat /proc/cpuinfo | grep -i mhz
cpu MHz : 2051.126
saturn:~> uname -a
Linux saturn 2.6.9-rc4-mm1-VP-U4 #288 SMP Fri Oct 15 12:31:38 CEST 2004

but it could easily be happening on some CPUs only. Let me know if that problem persists. Fortunately i think it will be at most a detection problem, not some FPU breakage that i initially suspected.

it could be the following thing: if you got an smp_processor_id() warning _in the CPU detection code_ in earlier PREEMPT_REALTIME kernels then the kernel could easily see that the CPU is extremely slow, because it didnt manage to do much work (due to the long printout...). So i'd say if this happens again it's most likely a debug printout in the 'calibrating delay loop' phase.

Ingo

Florian Schmidt

2004-10-15 12:09 UTC #932 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U1

Quoted 11 lines · On Fri, 15 Oct 2004 13:44:05 +0200 Ingo Molnar <[email protected]> wrote:

Quoted 1 line
cpu MHz : 0.001
ah ... good eyes. Seems to be working fine here:
saturn:~> cat /proc/cpuinfo | grep -i mhz
cpu MHz : 2051.126
saturn:~> uname -a
Linux saturn 2.6.9-rc4-mm1-VP-U4 #288 SMP Fri Oct 15 12:31:38 CEST 2004
but it could easily be happening on some CPUs only. Let me know if that problem persists.

Same problem with U3.

~$ uname -a
Linux mango.fruits.de 2.6.9-rc4-mm1-VP-U3-RT #1 Fri Oct 15 13:45:00 CEST 2004 i686 GNU/Linux ~$ cat /proc/cpuinfo |grep MHz
cpu MHz : 0.001

Quoted 9 lines

Fortunately i think it will be at most a detection problem, not some FPU breakage that i initially suspected.
it could be the following thing: if you got an smp_processor_id() warning _in the CPU detection code_ in earlier PREEMPT_REALTIME kernels then the kernel could easily see that the CPU is extremely slow, because it didnt manage to do much work (due to the long printout...). So i'd say if this happens again it's most likely a debug printout in the 'calibrating delay loop' phase.

I see. btw: i built this one with CONFIG_PREEMPT_TIMING and CONFIG_LATENCY_TRACE and, naturally, this also throws the timing code of the critical section timing off:

Linux version 2.6.9-rc4-mm1-VP-U3-RT ([email protected]) (gcc version 3.3.5 (Debian 1:3.3.5-1)) #1 Fri Oct 15 13:45:00 CEST 2004 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000030000000 (usable)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ffee0000 - 00000000fff00000 (reserved)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
768MB LOWMEM available.
On node 0 totalpages: 196608
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 192512 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
DMI 2.3 present.
Built 1 zonelists
Initializing CPU#0
Kernel command line: BOOT_IMAGE=2.6.9-U3-RT ro root=1601
PID hash table entries: 4096 (order: 12, 65536 bytes)
(swapper/0): new 436746 us maximum-latency critical section.
=> started at: <start_kernel+0x39/0x1c0>
=> ended at: <cond_resched+0x23/0x80>

 [<c012ea8c>] touch_preempt_timing+0x3c/0x40
 [<c012e9b0>] check_preempt_timing+0x160/0x200
 [<c02a3e23>] cond_resched+0x23/0x80
 [<c012ea8c>] touch_preempt_timing+0x3c/0x40
 [<c02a3e23>] cond_resched+0x23/0x80
 [<c02a3e23>] cond_resched+0x23/0x80
 [<c012d899>] _mutex_lock+0x19/0x40
 [<c011dad0>] tasklet_hi_action+0x0/0x70
 [<c010b51a>] get_cmos_time+0x1a/0x1e0
 [<c03228e3>] start_kernel+0xc3/0x1c0
 [<c0112240>] mcount+0x14/0x18
 [<c0326ba0>] time_init+0x10/0x70
 [<c011dad0>] tasklet_hi_action+0x0/0x70
 [<c03228e3>] start_kernel+0xc3/0x1c0
 [<c03225a0>] unknown_bootoption+0x0/0x160

preempt count: 1
entry 1: start_kernel+0x39/0x1c0 / (0xc010019f) Detected 1195.144 MHz processor.
Using tsc for high-res timesource
(swapper/0): new 597854 us maximum-latency critical section.
=> started at: <cond_resched+0x23/0x80>
=> ended at: <cond_resched+0x23/0x80>

 [<c012ea8c>] touch_preempt_timing+0x3c/0x40
 [<c012e9b0>] check_preempt_timing+0x160/0x200
 [<c02a3e23>] cond_resched+0x23/0x80
 [<c012ea8c>] touch_preempt_timing+0x3c/0x40
 [<c02a3e23>] cond_resched+0x23/0x80
 [<c02a3e23>] cond_resched+0x23/0x80
 [<c012d899>] _mutex_lock+0x19/0x40
 [<c012d916>] _mutex_lock_irqsave+0x16/0x20
 [<c01f0347>] tty_register_ldisc+0x37/0xb0
 [<c0333367>] console_init+0x27/0x50
 [<c03228e8>] start_kernel+0xc8/0x1c0
 [<c03225a0>] unknown_bootoption+0x0/0x160

preempt count: 1
entry 1: start_kernel+0x39/0x1c0 / (0xc010019f)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 776156k/786432k available (1685k kernel code, 9820k reserved, 485k data, 344k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay loop... 2351.10 BogoMIPS (lpj=1175552)
Security Scaffold v1.0.0 initialized
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: After generic identify, caps: 0183f9ff c1c7f9ff 00000000 00000000
CPU: After vendor identify, caps: 0183f9ff c1c7f9ff 00000000 00000000
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 256K (64 bytes/line)
CPU: After all inits, caps: 0183f9ff c1c7f9ff 00000000 00000020
CPU: AMD Athlon(tm) Processor stepping 02
Enabling fast FPU save and restore... done.
Checking 'hlt' instruction... OK.
ksoftirqd started up.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfdb01, last bus=1
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
Uncovering SIS18 that hid as a SIS503 (compatible=1)
Enabling SiS 96x SMBus.
PCI: Using IRQ router SIS [1039/0018] at 0000:00:02.0
PCI: IRQ 0 for device 0000:00:02.1 doesn't match PIRQ mask - try pci=usepirqmask
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
Initializing Cryptographic API
Real Time Clock Driver v1.12
Non-volatile memory driver v1.2
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
SIS5513: IDE controller at PCI slot 0000:00:02.5
SIS5513: chipset revision 208
SIS5513: not 100% native mode: will probe irqs later
SIS5513: SiS735 ATA 100 (2nd gen) controller

    ide0: BM-DMA at 0xff00-0xff07, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xff08-0xff0f, BIOS settings: hdc:DMA, hdd:DMA

	current capacity is 78165360 sectors (40020 MB)
	native  capacity is 78165361 sectors (40020 MB)

hdc: Host Protected Area disabled.
hdc: 78165361 sectors (40020 MB) w/1024KiB Cache, CHS=65535/16/63, UDMA(33)
hdc: cache flushes not supported
hdc: hdc1 hdc2
hdd: ATAPI 32X CD-ROM CD-R/RW drive, 2048kB Cache, DMA
Uniform CD-ROM driver Revision: 3.20
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
input: ImExPS/2 Logitech Explorer Mouse on isa0060/serio1
input: PC Speaker
NET: Registered protocol family 2
IP: routing cache hash table of 2048 buckets, 64Kbytes
TCP: Hash tables configured (established 65536 bind 37449)
NET: Registered protocol family 1
NET: Registered protocol family 17
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 344k freed
kjournald starting. Commit interval 5 seconds
Adding 289160k swap on /dev/hda3. Priority:-1 extents:1 EXT3-fs warning: maximal mount count reached, running e2fsck is recommended EXT3 FS on hdc1, internal journal
PCI: Found IRQ 5 for device 0000:00:0f.0
sis900.c: v1.08.07 11/02/2003
PCI: Found IRQ 10 for device 0000:00:03.0
eth0: Realtek RTL8201 PHY transceiver found at address 1.
eth0: Using transceiver found at address 1 as default
eth0: SiS 900 PCI Fast Ethernet at 0xdc00, IRQ 10, 00:d0:09:e9:c1:0f.
CSLIP: code copyright 1989 Regents of the University of California PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
kjournald starting. Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on hda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on hdc2, internal journal
EXT3-fs: mounted filesystem with ordered data mode. (S40networking/210): new 698390 us maximum-latency critical section.
=> started at: <kernel_fpu_begin+0x21/0x60>
=> ended at: <_mmx_memcpy+0x131/0x180>

 [<c012ec41>] sub_preempt_count+0x71/0x90
 [<c012e9b0>] check_preempt_timing+0x160/0x200
 [<c01e48d1>] _mmx_memcpy+0x131/0x180
 [<c010c44e>] kernel_fpu_begin+0xe/0x60
 [<c012ec41>] sub_preempt_count+0x71/0x90
 [<c01e48d1>] _mmx_memcpy+0x131/0x180
 [<c01e48d1>] _mmx_memcpy+0x131/0x180
 [<c01edbe5>] vgacon_scroll+0x245/0x260
 [<c01fe33a>] scrup+0xda/0xf0
 [<c0112240>] mcount+0x14/0x18
 [<c01ffe82>] lf+0x72/0x80
 [<c0201b20>] do_con_trol+0xa90/0xc30
 [<c01fef3b>] hide_softcursor+0xb/0x70
 [<c0201f25>] do_con_write+0x265/0x720
 [<c0202a0b>] con_write+0x3b/0x50
 [<c0202a65>] con_put_char+0x45/0x50
 [<c01f4b15>] opost+0xa5/0x1d0
 [<c0112240>] mcount+0x14/0x18
 [<c01f7083>] write_chan+0x1b3/0x220
 [<c0114ff0>] default_wake_function+0x0/0x20
 [<c0112240>] mcount+0x14/0x18
 [<c0114ff0>] default_wake_function+0x0/0x20
 [<c0114f8f>] lock_kernel+0x2f/0x50
 [<c01f169f>] tty_write+0x12f/0x1e0
 [<c01f6ed0>] write_chan+0x0/0x220
 [<c01f1750>] redirected_tty_write+0x0/0xb0
 [<c015584a>] vfs_write+0xca/0x140
 [<c0112240>] mcount+0x14/0x18
 [<c0155990>] sys_write+0x50/0x80
 [<c010603b>] syscall_call+0x7/0xb

preempt count: 1
entry 1: kernel_fpu_begin+0x21/0x60 / (_mmx_memcpy+0x36/0x180)

Daniel Walker

2004-10-14 18:58 UTC #933 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

In reply to Ingo Molnar

Quoted 6 lines · Ingo Molnar

To solve all these fundamental problems, i improved/fixed/changed all of these locking methods to be preemption-friendly. Most of the time it was necessary to introduce an additional API variant because e.g. rcu_read_lock() is anonymous (it doesnt identify the data protected), so i introduced a variant that takes the write-lock as an argument. In the PREEMPT_REALTIME case we can thus properly serialize on that lock.

When I was reviewing this it seemed like it would be possible to keep RCU anonymous by moving the callback processing out of the tasklet . The reason it was moved into a tasklet was to reduce latency. But if you serialize it like you have, aren't you removing all the benefits of the RCU type lock in those section that are converted to the new API ?

Quoted 5 lines

For per-cpu variables i introduced a new API variant that creates a spinlock-array for the per-cpu-variable, and users must make sure the cpu field doesnt change. Migration to another CPU can happen within the critical section, but 'statistically' the variable is still per-CPU and update correctness is fully preserved.

Why not have a per cpu mutex instead of a per variable per cpu mutex? I'm not sure what the trade off are, except size.

Daniel

Ingo Molnar

2004-10-14 19:31 UTC #934 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 6 lines · Daniel Walker

When I was reviewing this it seemed like it would be possible to keep RCU anonymous by moving the callback processing out of the tasklet . The reason it was moved into a tasklet was to reduce latency. But if you serialize it like you have, aren't you removing all the benefits of the RCU type lock in those section that are converted to the new API ?

only if compiling for PREEMPT_REALTIME. Given the overhead of PREEMPT_REALTIME i'm not sure RCU matters that much. But the nicest would be Dipankar's preemptible-RCU patch.

Quoted 8 lines

Quoted 5 lines
For per-cpu variables i introduced a new API variant that creates a spinlock-array for the per-cpu-variable, and users must make sure the cpu field doesnt change. Migration to another CPU can happen within the critical section, but 'statistically' the variable is still per-CPU and update correctness is fully preserved.
Why not have a per cpu mutex instead of a per variable per cpu mutex? I'm not sure what the trade off are, except size.

well, nesting would be one issue. What if such a section gets preempted on this CPU and another task tries to use the same mutex? Per-var-per-cpu mutexes seemed like the most orthogonal extension to the existing concept. Keeping the original Linux locking semantics intact seems like the primary mission, at least until the full scope is mapped.

Ingo

Dipankar Sarma

2004-10-14 20:23 UTC #935 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 10 lines · Ingo Molnar

Quoted 6 lines · Daniel Walker
When I was reviewing this it seemed like it would be possible to keep RCU anonymous by moving the callback processing out of the tasklet . The reason it was moved into a tasklet was to reduce latency. But if you serialize it like you have, aren't you removing all the benefits of the RCU type lock in those section that are converted to the new API ?

Quoted 4 lines

only if compiling for PREEMPT_REALTIME. Given the overhead of PREEMPT_REALTIME i'm not sure RCU matters that much. But the nicest would be Dipankar's preemptible-RCU patch.

I am swamped this week and racing against time to get some other pending things done in time. I will look at the issue of RCU with PREEMPT_REALTIME next week and try to help out.

Thanks
Dipankar

Bill Huey (hui)

2004-10-14 21:27 UTC #936 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

In reply to Daniel Walker

Quoted 5 lines · Daniel Walker

When I was reviewing this it seemed like it would be possible to keep RCU anonymous by moving the callback processing out of the tasklet . The reason it was moved into a tasklet was to reduce latency. But if you serialize it like you have, aren't you removing all the benefits of the RCU type lock in those section that are converted to the new API ?

What Ingo is doing now is mostly like a temporary fix for dealing with this issue. Simple backing with a normal mutex should be sufficient for protecting that access. RCU is still an open problem.

Quoted 2 lines

Why not have a per cpu mutex instead of a per variable per cpu mutex? I'm not sure what the trade off are, except size.

It's a read-mostly read/write lock. N number of real processors can do N number of read locks. That structure needs to be emulated somehow and a per CPU mutex is probably the correct method of getting it. It's just a matter of how. I did suggest something in my project announcement.

I don't know if it's crack smoking or not. :)

bill

Daniel Walker

2004-10-14 20:06 UTC #937 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

In reply to Ingo Molnar

I'm not sure about this one ..

------------[ cut here ]------------
kernel BUG at fs/buffer.c:1360!
invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU:    0
EIP:    0060:[<c01619c1>]    Not tainted VLI
EFLAGS: 00010002   (2.6.9-rc4-mm1-VP-U1)

EIP is at __find_get_block+0xe1/0x100
eax: 00000001 ebx: cfd30c14 ecx: cffdc600 edx: 00000000
esi: 0005709b edi: 00000000 ebp: cfd30b70 esp: cfd30b54
ds: 007b es: 007b ss: 0068 preempt: 00000001
Process kjournald (pid: 786, threadinfo=cfd30000 task=cfd26040)

Stack: 00000002 00000000 0005709b 00000000 cfd30c14 0005709b 00000000
cfd30b94
       c01619fe cffeab80 0005709b 00000000 00001000 cfd30c14 00000002
cfd30c44
       cfd30bac c0161a99 cffeab80 0005709b 00000000 00001000 cfd30bd4
c019aff0
Call Trace:
 [<c01619fe>] __getblk+0x1e/0x60
 [<c0161a99>] __bread+0x19/0x40
 [<c019aff0>] ext3_get_branch+0x70/0x100
 [<c019b61a>] ext3_get_block_handle+0x7a/0x2e0
 [<c026b1ee>] as_choose_req+0xe/0x1e0
 [<c026bc5f>] as_update_arq+0x1f/0x60
 [<c019b8c3>] ext3_get_block+0x43/0x80
 [<c0163735>] generic_block_bmap+0x35/0x40
 [<c0134c73>] __mcount+0x13/0x20
 [<c019c26d>] ext3_bmap+0xd/0xa0
 [<c01797c5>] bmap+0x45/0x60
 [<c019c2dc>] ext3_bmap+0x7c/0xa0
 [<c019b880>] ext3_get_block+0x0/0x80
 [<c01797c5>] bmap+0x45/0x60
 [<c01af8c2>] journal_bmap+0x42/0xa0
 [<c0134c73>] __mcount+0x13/0x20
 [<c0134249>] _mutex_unlock+0x9/0x60
 [<c01af827>] journal_next_log_block+0x47/0xa0
 [<c0113d30>] mcount+0x14/0x18
 [<c01af832>] journal_next_log_block+0x52/0xa0
 [<c01af939>] journal_get_descriptor_buffer+0x19/0xc0
 [<c01ac4ec>] journal_commit_transaction+0xf6c/0x13e0
 [<c01aedee>] kjournald+0xce/0x260
 [<c013555c>] sub_preempt_count+0x7c/0xa0
 [<c0133d00>] autoremove_wake_function+0x0/0x60
 [<c03b2a33>] _spin_unlock_irq+0x13/0x40
 [<c0133d00>] autoremove_wake_function+0x0/0x60
 [<c0119459>] schedule_tail+0x19/0x60
 [<c01aece0>] commit_timeout+0x0/0x20
 [<c01aed20>] kjournald+0x0/0x260
 [<c0103339>] kernel_thread_helper+0x5/0xc
Code: 45 14 39 43 10 75 a0 85 ff 74 17 8b 45 e4 89 f9 8d 14 b8 8b 42 fc

89 02 83 ea 04

Ingo Molnar

2004-10-14 20:43 UTC #938 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 31 lines · Daniel Walker

I'm not sure about this one ..

------------[ cut here ]------------
kernel BUG at fs/buffer.c:1360!

EIP is at __find_get_block+0xe1/0x100

Call Trace:
 [<c01619fe>] __getblk+0x1e/0x60
 [<c0161a99>] __bread+0x19/0x40
 [<c019aff0>] ext3_get_branch+0x70/0x100
 [<c019b61a>] ext3_get_block_handle+0x7a/0x2e0
 [<c026b1ee>] as_choose_req+0xe/0x1e0
 [<c026bc5f>] as_update_arq+0x1f/0x60
 [<c019b8c3>] ext3_get_block+0x43/0x80
 [<c0163735>] generic_block_bmap+0x35/0x40
 [<c0134c73>] __mcount+0x13/0x20
 [<c019c26d>] ext3_bmap+0xd/0xa0
 [<c01797c5>] bmap+0x45/0x60
 [<c019c2dc>] ext3_bmap+0x7c/0xa0
 [<c019b880>] ext3_get_block+0x0/0x80
 [<c01797c5>] bmap+0x45/0x60
 [<c01af8c2>] journal_bmap+0x42/0xa0
 [<c0134c73>] __mcount+0x13/0x20
 [<c0134249>] _mutex_unlock+0x9/0x60
 [<c01af827>] journal_next_log_block+0x47/0xa0
 [<c0113d30>] mcount+0x14/0x18
 [<c01af832>] journal_next_log_block+0x52/0xa0
 [<c01af939>] journal_get_descriptor_buffer+0x19/0xc0
 [<c01ac4ec>] journal_commit_transaction+0xf6c/0x13e0
 [<c01aedee>] kjournald+0xce/0x260

Quoted 8 lines

 [<c013555c>] sub_preempt_count+0x7c/0xa0
 [<c0133d00>] autoremove_wake_function+0x0/0x60
 [<c03b2a33>] _spin_unlock_irq+0x13/0x40
 [<c0133d00>] autoremove_wake_function+0x0/0x60
 [<c0119459>] schedule_tail+0x19/0x60
 [<c01aece0>] commit_timeout+0x0/0x20
 [<c01aed20>] kjournald+0x0/0x260
 [<c0103339>] kernel_thread_helper+0x5/0xc

this is a weird one. This is the first message, right? I've reviewed bh_lru_lock/unlock and cannot spot anything that could be wrong there.

Ingo

Radoslaw Szkodzinski

2004-10-14 22:24 UTC #939 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

In reply to Ingo Molnar

Quoted 9 lines · Ingo Molnar

i'm pleased to announce a significantly improved version of the Real-Time Preemption (PREEMPT_REALTIME) feature that i have been working towards in the past couple of weeks.
the patch (against 2.6.9-rc4-mm1) can be downloaded from:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U0

I'm getting some scheduling_while_atomic messages concerning Reiser4. They're non-fatal, please look into them.

scheduling while atomic: rc/0x04000001/908
caller is cond_resched+0x4c/0x69

 [<c0105666>] dump_stack+0x1e/0x20
 [<c035e40c>] schedule+0x94/0x3f9
 [<c035ec0d>] cond_resched+0x4c/0x69
 [<c012bda7>] _rw_mutex_read_lock+0x1f/0x33
 [<c019e7dd>] cbk_cache_scan_slots+0x5c/0x276
 [<c019ea22>] cbk_cache_search+0x2b/0x5e
 [<c019d84d>] coord_by_handle+0x12/0x29
 [<c019d831>] object_lookup+0xce/0xd8
 [<c01c8dad>] find_entry+0x123/0x2b7
 [<c01c788d>] lookup_name_hashed+0xc3/0x143
 [<c01c7997>] lookup_hashed+0x3e/0xc0
 [<c01a6bdd>] reiser4_lookup+0x83/0x10e
 [<c015e02c>] real_lookup+0x74/0xf4
 [<c015e2cb>] do_lookup+0x5e/0x9d
 [<c015ec7c>] link_path_walk+0x972/0xde6
 [<c015f49e>] path_lookup+0x19a/0x1a6
 [<c015f63a>] __user_walk+0x31/0x53
 [<c015a16d>] vfs_stat+0x1e/0x53
 [<c015a890>] sys_stat64+0x19/0x32
 [<c010509d>] sysenter_past_esp+0x52/0x71
scheduling while atomic: sh/0x04000001/4346
caller is cond_resched+0x4c/0x69
 [<c0105666>] dump_stack+0x1e/0x20
 [<c035e40c>] schedule+0x94/0x3f9
 [<c035ec0d>] cond_resched+0x4c/0x69
 [<c012bda7>] _rw_mutex_read_lock+0x1f/0x33
 [<c019e7dd>] cbk_cache_scan_slots+0x5c/0x276
 [<c019ea22>] cbk_cache_search+0x2b/0x5e
 [<c019d84d>] coord_by_handle+0x12/0x29
 [<c019d831>] object_lookup+0xce/0xd8
 [<c01cfc83>] find_file_item+0x127/0x1bc
 [<c01d1b2e>] read_unix_file+0x2e4/0x48d
 [<c01a74e8>] reiser4_read+0x90/0xaf
 [<c0150abb>] vfs_read+0xe0/0x126
 [<c015b39d>] kernel_read+0x4a/0x57
 [<c017aaeb>] load_elf_binary+0x303/0xba1
 [<c015bfe2>] search_binary_handler+0xd6/0x1f1
 [<c015c2fb>] do_execve+0x1fe/0x2af
 [<c0103c1c>] sys_execve+0x3f/0x91
 [<c010509d>] sysenter_past_esp+0x52/0x71

scheduling while atomic: gcc/0x04000001/5587
caller is cond_resched+0x4c/0x69

 [<c0105666>] dump_stack+0x1e/0x20
 [<c035e40c>] schedule+0x94/0x3f9
 [<c035ec0d>] cond_resched+0x4c/0x69
 [<c012bda7>] _rw_mutex_read_lock+0x1f/0x33
 [<c019e7dd>] cbk_cache_scan_slots+0x5c/0x276
 [<c019ea22>] cbk_cache_search+0x2b/0x5e
 [<c019d84d>] coord_by_handle+0x12/0x29
 [<c019d831>] object_lookup+0xce/0xd8
 [<c01c8dad>] find_entry+0x123/0x2b7
 [<c01c8acb>] rem_entry_hashed+0x75/0x1d5
 [<c01c97cb>] unlink_common+0xd4/0x1ca
 [<c01a6fe0>] unlink_file+0x48/0x75
 [<c01a703a>] reiser4_unlink+0x2d/0x37
 [<c0160d12>] vfs_unlink+0x1d1/0x225
 [<c0160e24>] sys_unlink+0xbe/0x145
 [<c010509d>] sysenter_past_esp+0x52/0x71
scheduling while atomic: xinit/0x04000001/5628
caller is cond_resched+0x4c/0x69
 [<c0105666>] dump_stack+0x1e/0x20
 [<c035e40c>] schedule+0x94/0x3f9
 [<c035ec0d>] cond_resched+0x4c/0x69
 [<c012bda7>] _rw_mutex_read_lock+0x1f/0x33
 [<c019e7dd>] cbk_cache_scan_slots+0x5c/0x276
 [<c019ea22>] cbk_cache_search+0x2b/0x5e
 [<c019d84d>] coord_by_handle+0x12/0x29
 [<c019d831>] object_lookup+0xce/0xd8
 [<c01cfc83>] find_file_item+0x127/0x1bc
 [<c01d1550>] readpage_unix_file+0xb1/0x35b
 [<c01a7961>] reiser4_readpage+0x4d/0x7d
 [<c0134e1e>] page_cache_read+0x72/0xea
 [<c013505d>] filemap_nopage+0x1c7/0x391
 [<c01d21bf>] unix_file_filemap_nopage+0x59/0x84
 [<c01429c9>] do_no_page+0xb5/0x32f
 [<c0142db6>] handle_mm_fault+0x91/0x164
 [<c01133db>] do_page_fault+0x20c/0x654
 [<c0105299>] error_code+0x2d/0x38

Karim Yaghmour

2004-10-14 22:42 UTC #940 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

In reply to Ingo Molnar

Quoted 1 line · Ingo Molnar

http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U0

I notice:
+ local_irq_save(flags);
+ ____trace(&__get_cpu_var(trace), eip, parent_eip);
+ local_irq_restore(flags);

Why not use the lockless logging available in relayfs, you'll avoid the interrupt disabling altogether since, umm ... it's lockless.

Karim

Signature · 3 lines

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

Ingo Molnar

2004-10-14 23:49 UTC #941 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 11 lines · Karim Yaghmour

Quoted 1 line · Ingo Molnar
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U0
I notice:
(quoted diff) +3 -0
+	local_irq_save(flags);
+	____trace(&__get_cpu_var(trace), eip, parent_eip);
+	local_irq_restore(flags);
Why not use the lockless logging available in relayfs, you'll avoid the interrupt disabling altogether since, umm ... it's lockless.

i just added something ad-hoc. I wanted it to be accurate across interrupt entries. I have not looked at the relayfs locking but how does it solve that? Also, cli/sti makes it obviously SMP-safe and is pretty cheap on all x86 CPUs. (Also, i didnt want to use preempt_disable/enable because the tracer interacts with that code quite heavily.)

Ingo

Karim Yaghmour

2004-10-15 00:02 UTC #942 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 1 line · Ingo Molnar

i just added something ad-hoc.

Yes, I understood as much. I'm suggesting it because a lot of people who need such ad-hoc functionality could easily be using relayfs.

Quoted 3 lines

I wanted it to be accurate across
interrupt entries. I have not looked at the relayfs locking but how does it solve that?

cmpxchg (basically: try reserve; if fail retry; else write), with per-cpu buffers.

Quoted 3 lines

Also, cli/sti makes it obviously SMP-safe and is pretty cheap on all x86 CPUs. (Also, i didnt want to use preempt_disable/enable because the tracer interacts with that code quite heavily.)

No preempt_disable/enable found in the lockless logging in relayfs.

Karim

Signature · 3 lines

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

Roland Dreier

2004-10-15 00:34 UTC #943 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Karim> cmpxchg (basically: try reserve; if fail retry; else Karim> write), with per-cpu buffers.

Not sure if I really understand the context where Ingo would use this, but this lockless scheme doesn't seem to be safe for realtime; the retry can potentially happen an arbitrary number of times.

- Roland

Karim Yaghmour

2004-10-15 01:14 UTC #944 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 3 lines · Roland Dreier

Not sure if I really understand the context where Ingo would use this, but this lockless scheme doesn't seem to be safe for realtime; the retry can potentially happen an arbitrary number of times.

In theory. In practice it doesn't often happen twice and very rarely more than that.

Karim

Signature · 3 lines

Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || [email protected] || 1-866-677-4546

Robert Wisniewski

2004-10-15 02:01 UTC #945 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

In reply to Roland Dreier

Theoretically a problem, in practice not, i.e., good enough for soft/normal real-time, not hard real-time; probably wouldn't want my heart monitor on it, but then I wouldn't be using Linux for that either :-)

Robert Wisniewski
The K42 MP OS Project
Advanced Operating Systems
Scalable Parallel Systems
IBM T.J. Watson Research Center
914-945-3181
http://www.research.ibm.com/K42/
[email protected]

Quoted 8 lines · Roland Dreier

Karim> cmpxchg (basically: try reserve; if fail retry; else Karim> write), with per-cpu buffers.
Not sure if I really understand the context where Ingo would use this, but this lockless scheme doesn't seem to be safe for realtime; the retry can potentially happen an arbitrary number of times.
- Roland

Lee Revell

2004-10-15 02:26 UTC #946 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 3 lines · Robert Wisniewski

Theoretically a problem, in practice not, i.e., good enough for soft/normal real-time, not hard real-time; probably wouldn't want my heart monitor on it, but then I wouldn't be using Linux for that either :-)

Also, the issue here is how we do debug logging. You would presumably not use this at all in production.

Lee

Robert Wisniewski

2004-10-15 12:28 UTC #947 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 9 lines · Lee Revell

Quoted 3 lines · Robert Wisniewski
Theoretically a problem, in practice not, i.e., good enough for soft/normal real-time, not hard real-time; probably wouldn't want my heart monitor on it, but then I wouldn't be using Linux for that either :-)
Also, the issue here is how we do debug logging. You would presumably not use this at all in production.
Lee

Yes actually you would. If the tracing subsystem is designed correctly you leave it in for production systems and enable it when you need to find a problem. The reason is because many times you can not reproduce a problem someone in production is seeing in your environment. In addition to the LTT/Relayfs and K42 tracing work, lots of of tracing work/papers suggest leaving it in all the time. Most commercial operating systems have made the investment to correctly design tracing facilities so they are available. LTT in combination with relayfs could fulfill that role for Linux.

Robert Wisniewski
The K42 MP OS Project
Advanced Operating Systems
Scalable Parallel Systems
IBM T.J. Watson Research Center
914-945-3181
http://www.research.ibm.com/K42/
[email protected]

Ingo Molnar

2004-10-15 06:57 UTC #948 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

In reply to Karim Yaghmour

Quoted 4 lines · Karim Yaghmour

Quoted 1 line
i just added something ad-hoc.
Yes, I understood as much. I'm suggesting it because a lot of people who need such ad-hoc functionality could easily be using relayfs.

the latency tracer is pretty specialized for a number of reasons, i'm not sure there's a good match between the two. If relayfs were in the mainline kernel i'd consider reusing parts of it.

Quoted 6 lines

Quoted 3 lines
I wanted it to be accurate across
interrupt entries. I have not looked at the relayfs locking but how does it solve that?
cmpxchg (basically: try reserve; if fail retry; else write), with per-cpu buffers.

this still does not solve all problems related to irq entries: if the IRQ interrups the tracing code after a 'successful reserve' but before the 'else write' point, and the trace is printed/saved from an interrupt, then there will be an incomplete entry in the trace.

also, there is the problem of timestamp atomicity: if an IRQ interrupts the tracing code and the trace timestamp is taken in the 'else' branch then a time-reversal situation can occur: the entry will have a timestamp _larger_ than the IRQ trace-entries. With cli/sti all tracing entries occur atomically: either fully or not at all.

Quoted 5 lines

Quoted 3 lines
Also, cli/sti makes it obviously SMP-safe and is pretty cheap on all x86 CPUs. (Also, i didnt want to use preempt_disable/enable because the tracer interacts with that code quite heavily.)
No preempt_disable/enable found in the lockless logging in relayfs.

it would have to do that on PREEMPT_REALTIME. The irq flag solves both the races, the predictability problem and the preemption problem nicely.

Ingo

Robert Wisniewski

2004-10-15 12:33 UTC #949 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 11 lines · Ingo Molnar

Quoted 4 lines · Karim Yaghmour
Quoted 1 line
i just added something ad-hoc.
Yes, I understood as much. I'm suggesting it because a lot of people who need such ad-hoc functionality could easily be using relayfs.
the latency tracer is pretty specialized for a number of reasons, i'm not sure there's a good match between the two. If relayfs were in the mainline kernel i'd consider reusing parts of it.

:-) it's nice we all have a sense of humor.

Quoted 5 lines

Quoted 5 lines
Quoted 4 lines
I wanted it to be accurate across
interrupt entries. I have not looked at the relayfs locking but how does it solve that?

Quoted 7 lines

Quoted 2 lines
cmpxchg (basically: try reserve; if fail retry; else write), with per-cpu buffers.
this still does not solve all problems related to irq entries: if the IRQ interrups the tracing code after a 'successful reserve' but before the 'else write' point, and the trace is printed/saved from an interrupt, then there will be an incomplete entry in the trace.

That is incorrect. The system behavior needed to generate an incomplete entry is far more complicated and unlikely than what you describe.

Quoted 11 lines

also, there is the problem of timestamp atomicity: if an IRQ interrupts the tracing code and the trace timestamp is taken in the 'else' branch then a time-reversal situation can occur: the entry will have a timestamp _larger_ than the IRQ trace-entries. With cli/sti all tracing entries occur atomically: either fully or not at all.
Quoted 4 lines
Quoted 3 lines
Also, cli/sti makes it obviously SMP-safe and is pretty cheap on all x86 CPUs. (Also, i didnt want to use preempt_disable/enable because the tracer interacts with that code quite heavily.)

Quoted 6 lines

Quoted 1 line
No preempt_disable/enable found in the lockless logging in relayfs.
it would have to do that on PREEMPT_REALTIME. The irq flag solves both the races, the predictability problem and the preemption problem nicely.
Ingo

If you do not care about performance then you're probably right, this is fine. If you are concerned about the time it takes to go through the sequence of code, then probably not.

Robert Wisniewski
The K42 MP OS Project
Advanced Operating Systems
Scalable Parallel Systems
IBM T.J. Watson Research Center
914-945-3181
http://www.research.ibm.com/K42/
[email protected]

Ingo Molnar

2004-10-15 13:10 UTC #950 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 11 lines · Robert Wisniewski

Quoted 7 lines
Quoted 2 lines
cmpxchg (basically: try reserve; if fail retry; else write), with per-cpu buffers.
this still does not solve all problems related to irq entries: if the IRQ interrups the tracing code after a 'successful reserve' but before the 'else write' point, and the trace is printed/saved from an interrupt, then there will be an incomplete entry in the trace.
That is incorrect. The system behavior needed to generate an incomplete entry is far more complicated and unlikely than what you describe.

ah, but i'm talking about actual first-hand experience, not supposition. It happens quite easily with latency traces (which are saved/printed from IRQ entries) and it can be very annoying to analyze. My first tracers tried to do things without the IRQ flag, so i've seen both methods.

and lets not forget this other issue:

Quoted 8 lines

Quoted 8 lines
also, there is the problem of timestamp atomicity: if an IRQ interrupts the tracing code and the trace timestamp is taken in the 'else' branch then a time-reversal situation can occur: the entry will have a timestamp _larger_ than the IRQ trace-entries. With cli/sti all tracing entries occur atomically: either fully or not at all.
Quoted 1 line
Quoted 1 line
Also, cli/sti makes it obviously SMP-safe and is pretty

                                                      ^^^^^^^^^^

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
cheap on all x86 CPUs. (Also, i didnt want to use preempt_disable/enable

        ^^^^^^^^^^^^^^^^^^^^^^

Quoted 4 lines

Quoted 4 lines
Quoted 3 lines
Quoted 1 line
because the tracer interacts with that code quite heavily.)
No preempt_disable/enable found in the lockless logging in relayfs.

Quoted 8 lines

Quoted 4 lines
it would have to do that on PREEMPT_REALTIME. The irq flag solves both the races, the predictability problem and the preemption problem nicely.
Ingo
If you do not care about performance then you're probably right, this is fine. If you are concerned about the time it takes to go through the sequence of code, then probably not.

see the portion i highlighted above. CPU makers are busy making cli/sti as fast as possible. To make sure i tested it on a typical x86 box:

    # ./cli-latency
    CLI+STI latency: 8 cycles

Since the trace entry can be filled in a constant amount of time there's no reason not to make use of that extra silicon that makes fast cli/sti possible! How many trace entries can you generate per second via relayfs, on a typical PC? Have you ever measured this?

in fact on a modern CPU cli/sti is very likely faster than a cmpxchgl for the following reason: the cmpxchgl generates a read dependency on the cacheline which must be fetched in. A single cachemiss can cost _alot_ in comparison, 200 cycles easily. While in the cli/sti case we stream out to a new cacheline in a linear fashion which is nicely optimized by write-allocate cache policies in modern CPUs. No cachemisses on the trace buffer! Just simple streaming out of data.

i challenge you to change your code to use cli/sti and compare it with the cmpxchgl variant doing some heavy tracing on a modern PC. Please just _do_ it and come back with numbers. I have done my own measurements.

Ingo

Robert Wisniewski

2004-10-15 14:53 UTC #951 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 15 lines · Ingo Molnar

Quoted 11 lines · Robert Wisniewski
Quoted 7 lines
Quoted 2 lines
cmpxchg (basically: try reserve; if fail retry; else write), with per-cpu buffers.
this still does not solve all problems related to irq entries: if the IRQ interrups the tracing code after a 'successful reserve' but before the 'else write' point, and the trace is printed/saved from an interrupt, then there will be an incomplete entry in the trace.
That is incorrect. The system behavior needed to generate an incomplete entry is far more complicated and unlikely than what you describe.

Quoted 5 lines

ah, but i'm talking about actual first-hand experience, not supposition. It happens quite easily with latency traces (which are saved/printed from IRQ entries) and it can be very annoying to analyze. My first tracers tried to do things without the IRQ flag, so i've seen both methods.

This means that other code you've written has this happen, it doesn't mean the fundamental model is broken. Also, if what you claim is true and there really is this contention, then it both means that 1) there are many many other higher priority tasks in the system than the one you are trying to trace, and 2) it's questionable whether you want to use locks.

Quoted 45 lines

and lets not forget this other issue:
Quoted 8 lines
Quoted 8 lines
also, there is the problem of timestamp atomicity: if an IRQ interrupts the tracing code and the trace timestamp is taken in the 'else' branch then a time-reversal situation can occur: the entry will have a timestamp _larger_ than the IRQ trace-entries. With cli/sti all tracing entries occur atomically: either fully or not at all.
Quoted 1 line
Quoted 1 line
Also, cli/sti makes it obviously SMP-safe and is pretty
                                                      ^^^^^^^^^^
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
cheap on all x86 CPUs. (Also, i didnt want to use preempt_disable/enable
        ^^^^^^^^^^^^^^^^^^^^^^
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
because the tracer interacts with that code quite heavily.)
No preempt_disable/enable found in the lockless logging in relayfs.
Quoted 8 lines
Quoted 4 lines
it would have to do that on PREEMPT_REALTIME. The irq flag solves both the races, the predictability problem and the preemption problem nicely.
Ingo
If you do not care about performance then you're probably right, this is fine. If you are concerned about the time it takes to go through the sequence of code, then probably not.
see the portion i highlighted above. CPU makers are busy making cli/sti as fast as possible. To make sure i tested it on a typical x86 box:
    # ./cli-latency
    CLI+STI latency: 8 cycles
Since the trace entry can be filled in a constant amount of time there's no reason not to make use of that extra silicon that makes fast cli/sti possible! How many trace entries can you generate per second via relayfs, on a typical PC? Have you ever measured this?
in fact on a modern CPU cli/sti is very likely faster than a cmpxchgl for the following reason: the cmpxchgl generates a read dependency on the cacheline which must be fetched in. A single cachemiss can cost _alot_ in comparison, 200 cycles easily. While in the cli/sti case we stream out to a new cacheline in a linear fashion which is nicely optimized by write-allocate cache policies in modern CPUs. No cachemisses on the trace buffer! Just simple streaming out of data.

Quoted 6 lines

i challenge you to change your code to use cli/sti and compare it with the cmpxchgl variant doing some heavy tracing on a modern PC. Please just _do_ it and come back with numbers. I have done my own measurements.
Ingo

This is an interesting analysis, do you have a paper you can point to, or can you post the numbers from, and what the setup of the experiment was, that you ran. Sounds interesting.

Robert Wisniewski
The K42 MP OS Project
Advanced Operating Systems
Scalable Parallel Systems
IBM T.J. Watson Research Center
914-945-3181
http://www.research.ibm.com/K42/
[email protected]

Ingo Molnar

2004-10-15 14:59 UTC #952 lore raw

Re: [patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

Quoted 22 lines · Robert Wisniewski

Quoted 15 lines · Ingo Molnar
Quoted 11 lines · Robert Wisniewski
Quoted 7 lines
Quoted 2 lines
cmpxchg (basically: try reserve; if fail retry; else write), with per-cpu buffers.
this still does not solve all problems related to irq entries: if the IRQ interrups the tracing code after a 'successful reserve' but before the 'else write' point, and the trace is printed/saved from an interrupt, then there will be an incomplete entry in the trace.
That is incorrect. The system behavior needed to generate an incomplete entry is far more complicated and unlikely than what you describe.
Quoted 5 lines
ah, but i'm talking about actual first-hand experience, not supposition. It happens quite easily with latency traces (which are saved/printed from IRQ entries) and it can be very annoying to analyze. My first tracers tried to do things without the IRQ flag, so i've seen both methods.

Quoted 5 lines

This means that other code you've written has this happen, it doesn't mean the fundamental model is broken. Also, if what you claim is true and there really is this contention, then it both means that 1) there are many many other higher priority tasks in the system than the one you are trying to trace, and 2) it's questionable whether you want to use locks.

_interrupts_. The latency tracer does traces like:

00000002 0.022ms (+0.000ms): mark_page_accessed (zap_pte_range) 00000002 0.022ms (+0.000ms): page_remove_rmap (zap_pte_range) 00000002 0.022ms (+0.000ms): free_page_and_swap_cache (zap_pte_range) 00000002 0.022ms (+0.001ms): put_page (zap_pte_range) 00010002 0.023ms (+0.000ms): do_IRQ (zap_pte_range) 00010002 0.023ms (+0.000ms): do_IRQ (<00000000>)
00010003 0.024ms (+0.004ms): mask_and_ack_8259A (do_IRQ) 00010003 0.029ms (+0.000ms): redirect_hardirq (do_IRQ) 00010000 0.029ms (+0.000ms): handle_IRQ_event (do_IRQ)

and i just pointed out why i didnt use relayfs.

Ingo