Skip to contentSkip to conversation

Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

PATCH In review Ingo Molnar 3 participants 11 messages Jun 7, 2005 – Jun 8, 2005

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Real-Time Preemption, -RT-2.6.12-rc6-V0.7.47-29

i have released the -V0.7.47-29 Real-Time Preemption patch, which can be downloaded from the usual place:

http://redhat.com/~mingo/realtime-preempt/

it's a fixes and speedups release. Changes since -47-20:

- x64 fixes (Michal Schmidt)

- cpufreq fix (Esben Nielsen)

- CONFIG_RT_DEADLOCK_DETECT build fix (Michal Schmidt)

- MAX_USER_RT_PRIO fix (Steven Rostedt)

there are more microoptimizations to the spin_lock/unlock hotpath:

- the caching of mutex_getprio() priority in p->normal_prio

- the mutex lock/unlock paths are now all fall-through. (Found a gcc bug, it mishandles __builtin_expect() in certain cases and produces correct but suboptimal code - we are working it around now.)

- reduced the amount of recursive preemption-counter bumps via the use of raw spinlocks

- rely on the preemption-counter instead of IRQs-off sections

These changes brought the PREEMPT_RT overhead significantly down on hackbench workloads (clearly a worst-case test for PREEMPT_RT overhead).

Would be interesting to see what kind of system time overhead PREEMPT_RT now causes for e.g. jack_test workloads.

to build a -V0.7.47-29 tree, the following patches should to be applied:

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.11.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.12-rc6.bz2
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.12-rc6-V0.7.47-29

Ingo

#1 lore raw
Quoted 4 lines · Ingo Molnar

ere are more microoptimizations to the spin_lock/unlock hotpath:

- the caching of mutex_getprio() priority in p->normal_prio

Quoted 8 lines

- the mutex lock/unlock paths are now all fall-through. (Found a gcc bug, it mishandles __builtin_expect() in certain cases and produces correct but suboptimal code - we are working it around now.)

- reduced the amount of recursive preemption-counter bumps via the use of raw spinlocks

- rely on the preemption-counter instead of IRQs-off sections

There is a local_irq_enable missing someplace in UP ..

BUG: scheduling with irqs disabled: khelper/0x00000000/5
caller is __down_mutex+0x276/0x440
 [<c03a88d6>] __down_mutex+0x276/0x440 (4)
 [<c03a75cc>] schedule+0xec/0x100 (4)
 [<c03a88d6>] __down_mutex+0x276/0x440 (12)
 [<c012b290>] remove_wait_queue+0x10/0x40 (64)
 [<c0130e02>] __spin_lock+0x22/0x40 (32)
 [<c012b290>] remove_wait_queue+0x10/0x40 (8)
 [<c0130e8c>] _spin_lock_irqsave+0xc/0x20 (12)
 [<c012b290>] remove_wait_queue+0x10/0x40 (8)
 [<c012b290>] remove_wait_queue+0x10/0x40 (4)
 [<c0126a2b>] worker_thread+0x14b/0x280 (20)
 [<c01265a0>] __call_usermodehelper+0x0/0x60 (16)
 [<c0112d20>] default_wake_function+0x0/0x40 (28)
 [<c0112d20>] default_wake_function+0x0/0x40 (32)
 [<c01268e0>] worker_thread+0x0/0x280 (36)
 [<c012af24>] kthread+0x84/0xc0 (4)
 [<c012aea0>] kthread+0x0/0xc0 (20)
 [<c010132d>] kernel_thread_helper+0x5/0x18 (16)
#2 lore raw
Quoted 7 lines · Daniel Walker
Quoted 1 line

- rely on the preemption-counter instead of IRQs-off sections

There is a local_irq_enable missing someplace in UP ..

BUG: scheduling with irqs disabled: khelper/0x00000000/5
caller is __down_mutex+0x276/0x440
 [<c03a88d6>] __down_mutex+0x276/0x440 (4)

.config please.

Ingo

#3 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 5 lines

There is a local_irq_enable missing someplace in UP ..

BUG: scheduling with irqs disabled: khelper/0x00000000/5
caller is __down_mutex+0x276/0x440
 [<c03a88d6>] __down_mutex+0x276/0x440 (4)

.config please.

ok, managed to reproduce.

Ingo

#4 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · Ingo Molnar
Quoted 2 lines

There is a local_irq_enable missing someplace in UP ..

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
BUG: scheduling with irqs disabled: khelper/0x00000000/5
caller is __down_mutex+0x276/0x440
 [<c03a88d6>] __down_mutex+0x276/0x440 (4)

.config please.

ok, managed to reproduce.

ok, fixed it and uploaded the -47-30 patch with the fix.

Ingo

#5 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 6 lines · Ingo Molnar
Quoted 5 lines

There is a local_irq_enable missing someplace in UP ..

BUG: scheduling with irqs disabled: khelper/0x00000000/5
caller is __down_mutex+0x276/0x440
 [<c03a88d6>] __down_mutex+0x276/0x440 (4)
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

.config please.

ok, managed to reproduce.

ok, fixed it and uploaded the -47-30 patch with the fix.

Ingo

I just built this one, with RT but without the statistical stuff enabled. While rc6 works fine with tvtime-0.99, this gives a no video blue screen, audio ok.

Its also found a phantom usb-serial-audio device and fouled up kmix with two devices to control. This causes a slash thru the icon for kmix on the toolbar. The lsusb -v output is many pages in length for this mis-identified device as I believe thats the vendor:product for a labtec webcam that is plugged in. 100% el-cheapo model.

config, dmesg saved if you'd like to see them.

Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
#6 lore raw
Quoted 3 lines · Gene Heskett

I just built this one, with RT but without the statistical stuff enabled. While rc6 works fine with tvtime-0.99, this gives a no video blue screen, audio ok.

could you check PREEMPT_DESKTOP too, with softirq/hardirq threading enabled/disabled?

Ingo

#7 lore raw
Quoted 9 lines · Ingo Molnar
Quoted 3 lines · Gene Heskett

I just built this one, with RT but without the statistical stuff enabled. While rc6 works fine with tvtime-0.99, this gives a no video blue screen, audio ok.

could you check PREEMPT_DESKTOP too, with softirq/hardirq threading enabled/disabled?

Ingo

Ok, next build with preempt mode 3, hard & softirq's theaded, as is RCU.

Kmix now normal, tvtime still fubar.

Build with mode 2 and it looks as if its all working:

[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT_DESKTOP is not set
# CONFIG_PREEMPT_RT is not set CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
# CONFIG_PREEMPT_BKL is not set

Now, I note that going to the #2 mode (voluntary) turned off threaded RCU's, so I'm going to leave that off and try a mode 3 again. BRB.

And that makes tvtime's video fail with a blue screen, audio ok..

Mode 2, FWIW, makes for quite jerky card motions while playing AisleRiot, the solitaire game.

Quoted 5 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
#8 lore raw
Quoted 11 lines · Gene Heskett

[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT_DESKTOP is not set
# CONFIG_PREEMPT_RT is not set CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
# CONFIG_PREEMPT_BKL is not set

Now, I note that going to the #2 mode (voluntary) turned off threaded RCU's, so I'm going to leave that off and try a mode 3 again. BRB.

please try mode 3 and disable softirq/hardirq threading. If that fixes things, could you check which of the two options (CONFIG_PREEMPT_SOFTIRQS or CONFIG_PREEMPT_HARDIRQS) causes which type of regression?

Quoted 4 lines

And that makes tvtime's video fail with a blue screen, audio ok..

Mode 2, FWIW, makes for quite jerky card motions while playing AisleRiot, the solitaire game.

this doesnt happen on PREEMPT_DESKTOP or PREEMPT_RT?

Ingo

#9 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 9 lines · Gene Heskett

[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT_DESKTOP is not set
# CONFIG_PREEMPT_RT is not set CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
# CONFIG_PREEMPT_BKL is not set

Quoted 8 lines
Quoted 3 lines

Now, I note that going to the #2 mode (voluntary) turned off threaded RCU's, so I'm going to leave that off and try a mode 3 again. BRB.

please try mode 3 and disable softirq/hardirq threading. If that fixes things, could you check which of the two options (CONFIG_PREEMPT_SOFTIRQS or CONFIG_PREEMPT_HARDIRQS) causes which type of regression?

Ok, they are both off, mode 3 and no RCU threading:

[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config # CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set CONFIG_PREEMPT_DESKTOP=y
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_SOFTIRQS is not set
# CONFIG_PREEMPT_HARDIRQS is not set
# CONFIG_PREEMPT_RCU is not set
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set

And tvtime seems normal. However, when the video works (and this is a known seperate problem with the cx88 driver) then the audio is highly 'aliased' making for very tinny sound until an internal restart with new stds (or 3, sometimes it comes back after a few seconds) is done. There is a cx88_video.c patch that went by a few days ago, but even thats not 100% "bulletproof" :-)

Quoted 6 lines
Quoted 4 lines

And that makes tvtime's video fail with a blue screen, audio ok..

Mode 2, FWIW, makes for quite jerky card motions while playing AisleRiot, the solitaire game.

this doesnt happen on PREEMPT_DESKTOP or PREEMPT_RT?

Its subjectively better, but plain old rc6 is the best in this regard. And I note that as I play with the soft & hard irq threading off, in mode 3, it improves to be equ to rc6 after about 5 or 6 moves. Like the cfq scheduler is dynamicly adjusting priorities or something. Thats my default scheduler for the last 2-3 months, should I be retrying the others?

On to make one with softirq threading turned on, BRB. With this:
[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config # CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set CONFIG_PREEMPT_DESKTOP=y
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
# CONFIG_PREEMPT_HARDIRQS is not set
# CONFIG_PREEMPT_RCU is not set
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set

tvtime seems normal. So for the next build, turn on CONFIG_PREEMPT_RCU=y for effects since I note that its turned on along with hardirq threading by CONFIG_PREEMPT_RT=y. BRB (well, 15 minutes by the time I reboot).

Speaking of reboot, I saw a 'failed to contact something' message go flying by in the shutdown phase the last time around this loop.

With:
[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# grep PREEMPT .config # CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set CONFIG_PREEMPT_DESKTOP=y
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
# CONFIG_PREEMPT_HARDIRQS is not set CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set

tvtime works ok with RCU threading on.

But a "service ntpd restart" which failed during the boot, failed the 2nd time befrore I did the startx, and here's the 3rd restart:
[root@coyote linux-2.6.12-rc6-RT-V0.7.47-30]# service ntpd restart

Shutting down ntpd:                                        [  OK  ]
ntpd: Synchronizing with time server:                      [FAILED]
Starting ntpd:  -U ntp -p /var/run/ntpd.pid -g             [  OK  ]

According to the lights on the router, it pounded on the server about 10 times at 1 second intervals before it output the failed message.

So we are flushing out some bugs here. The missus just came back, she's been out to the sawbones with her emphesema so my playtime today is about used up. Many thanks for the quick responses.

Quoted 2 lines

Ingo

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
#10 lore raw
In reply to Ingo Molnar
Quoted 5 lines · Ingo Molnar
Quoted 3 lines
Quoted 1 line

.config please.

ok, managed to reproduce.

ok, fixed it and uploaded the -47-30 patch with the fix.

I'm glad cause I don't think I have the .config any more ..

Daniel