- changes to timers still broken - we don't oops anymore
Subsystems (4)
It just caused sshd to hang. I don't know why Here's what strace reports:
Sshd is stuck in 'D' and a child in zombie state. The machine has been up for 2 days 18 hours 50 mins.
Somehow, although those changes fix the kernel from oopsing, its now causing other tty issues somewhere.
Any ideas?
Shawn.
-----Original Message-----
From: Shawn Starr [mailto:[email protected]]
Sent: Saturday, April 05, 2003 7:35 PM
To: 'Andrew Morton'
Cc: '[email protected]'; '[email protected]'; '[email protected]'
Subject: RE: [OOPS][2.5.66bk9+] run_timer_softirq - IRQ Mishandlings - New OOPS w/ timer
Andrew, since the new timer changes made, Things are looking solid so far. We might want to begin adding things into BK-current no?
Sshd hasn't hung, ttys haven't hung either.
Shawn.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Shawn" == Shawn Starr <[email protected]> writes:
Shawn> It just caused sshd to hang. I don't know why Here's what Shawn> strace reports: Sshd is stuck in 'D' and a child in zombie Shawn> state. The machine has been up for 2 days 18 hours 50 mins.
I may have missed some emails on this topic, so I apologize if this objection was already answered. In any case, if you're running with Andrew's patch that added a del_timer_sync to release_dev(), I think I see how that could cause problems. Here's what I said in my previous email:
From looking at workqueue.c (especially the comment in queue_delayed_work() that says "Increase nr_queued so that the flush function knows that there's something pending."), it seems like flush_scheduled_work() should wait until even delayed work is done. Given that, I don't think the del_timer_sync() should be there -- wouldn't flush_scheduled_work() block forever, since nr_queued can never reach 0 now?
Also, I didn't see an answer to the worry I expressed below:
I don't see how it's _ever_ safe to call flush_scheduled_work(). The comment in workqueue.c before flush_workqueue() says "NOTE: if work is being added to the queue constantly by some other context then this function might block indefinitely." But flush_scheduled_work() is flushing the keventd_wq, which other code will definitely add work to. If we're unlucky, flush_scheduled_work() could block forever. Am I just being paranoid?
I admit that flush_scheduled_work() is unlikely to cause problems in most real world situations, but I'd be curious to know if anyone thinks this could ever lead to a livelock under some strange condition.
- Roland
Quoted 2 lines · Shawn Starr
It just caused sshd to hang.
What is "it"?
I receive rather a lot of email and am dependent on people helping me out a bit with context. I have lost the plot on this one.
Quoted 4 lines
I don't know why Here's what strace reports:
Sshd is stuck in 'D' and a child in zombie state. The machine has been up for 2 days 18 hours 50 mins.
a sysrq-T trace here would help.
Oh, the timer changes made caused sshd to hang It appears.
When it next does this I will sysctl+T and report the output.
It's still hard to track this problem, but at least its not causing the box to hang anymore, so part of the problem is solved :-)
Shawn.
-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Sunday, April 06, 2003 4:38 PM
To: Shawn Starr
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [BUG][2.5.66bk9+] - changes to timers still broken - we don't oops anymore
Quoted 2 lines · Shawn Starr
It just caused sshd to hang.
What is "it"?
I receive rather a lot of email and am dependent on people helping me out a bit with context. I have lost the plot on this one.
Quoted 4 lines
I don't know why Here's what strace reports:
Sshd is stuck in 'D' and a child in zombie state. The machine has been up for 2 days 18 hours 50 mins.
a sysrq-T trace here would help.
[BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
In reply to Andrew MortonPatches Applied to 2.5.66-bk9:
==============================
2.5.64-sysctld.patch (from rmk):
--------------------------------
Quoted 1 line
From [email protected] Fri Mar 14 15:19:17 2003
Date: Fri, 14 Mar 2003 19:53:22 +0000
From: Russell King <[email protected]>
To: Shawn Starr <[email protected]>
Subject: Re: [BUG][2.5.64bk4] Weird problem with 2 PCs
Quoted 2 lines · Shawn Starr
2.4 isn't locking up when I turn on the other box. This is going to be hard to track down. Will kgdb help me out?
Ok, here's a patch. Make sure you have sysrq enabled, and, after the crash, <alt>-<sysrq>-d on the keyboard of the crashed machine should dump all the kernel messages.
kernel/printk.c Fri Mar 14 19:42:48 2003 renamed from kernel/printk.c Tue Feb 11 16:10:58 2003 +22 -0
@@ -675,3 +675,25 @@ tty->driver.write(tty, 0, msg, strlen(msg)); return; } + +#include <linux/sysrq.h> + +static void +show_msg_info(int key, struct pt_regs *regs, struct tty_struct *tty) +{ + call_console_drivers(log_end - logged_chars, log_end); +} + +static struct sysrq_key_op msg_info_op = { + .handler = show_msg_info, + .help_msg = "Dumpmsgs", + .action_msg = "Kernel Messages", +}; + +static int __init dbg_init(void) +{ + register_sysrq_key('d', &msg_info_op); + return 0; +} + +__initcall(dbg_init);
timerfind.diff (from akpm):
---------------------------
fs/open.c | 33 +++++++++++++++++++++++++++++++++
include/linux/timer.h | 3 ++- mm/slab.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 1 deletion(-)
diff -puN include/linux/timer.h~freed-timer-finder include/linux/timer.h --- 25/include/linux/timer.h~freed-timer-finder 2003-03-29 14:10:15.000000000 -0800
+++ 25-akpm/include/linux/timer.h 2003-03-29 14:10:26.000000000 -0800
@@ -8,11 +8,12 @@
struct tvec_t_base_s;
struct timer_list {
+ unsigned long magic;
+
struct list_head entry;
unsigned long expires;
spinlock_t lock;- unsigned long magic;
void (*function)(unsigned long);
unsigned long data;mm/slab.c 2003-03-29 14:16:23.000000000 -0800 renamed from mm/slab.c~freed-timer-finder 2003-03-29 14:10:30.000000000 -0800 +22 -0
@@ -800,6 +800,35 @@ static void poison_obj(kmem_cache_t *cac *(unsigned char *)(addr+size-1) = POISON_END; } +static void timer_hunt(kmem_cache_t *cachep, void *addr) +{ + int size = cachep->objsize; + void *p; + + if (cachep->flags & SLAB_RED_ZONE) { + addr += BYTES_PER_WORD; + size -= 2*BYTES_PER_WORD; + } + if (cachep->flags & SLAB_STORE_USER) { + size -= BYTES_PER_WORD; + } + + for (p = addr; p < addr + size; p += sizeof(unsigned long)) { + unsigned long *laddr = p; + + if (*laddr == TIMER_MAGIC) { + struct timer_list *timer; + + timer = (struct timer_list *)laddr; + if (timer_pending(timer)) { + printk("free of pending timer at %p\n",
timer);
+ printk("function=%p\n", timer->function);
+ dump_stack();
+ }
+ }
+ }
+}
+
static void *fprob(unsigned char* addr, unsigned int size)
{
unsigned char *end;
@@ -1603,6 +1632,7 @@ static inline void *cache_free_debugchec
else
cachep->dtor(objp, cachep, 0);
}
+ timer_hunt(cachep, objp);
if (cachep->flags & SLAB_POISON)
poison_obj(cachep, objp, POISON_AFTER);#endif
fs/open.c 2003-03-29 14:21:20.000000000 -0800 renamed from fs/open.c~freed-timer-finder 2003-03-29 14:17:32.000000000 -0800 +33 -0
@@ -793,11 +793,44 @@ void fd_install(unsigned int fd, struct write_unlock(&files->file_lock); } +#include <linux/timer.h> + +struct foo_thing { + int a; + struct timer_list t; + int b; +}; + +static void my_foo(unsigned long data) +{ + printk("the handler\n"); +} + +static void timer_thing(void) +{ + static int did_it; + struct foo_thing *f; + + if (did_it) + return; + did_it = 1; + + f = kmalloc(sizeof(*f), GFP_KERNEL); + init_timer(&f->t); + f->t.expires = jiffies + HZ; + f->t.function = my_foo; + add_timer(&f->t); + kfree(f); +} + asmlinkage long sys_open(const char * filename, int flags, int mode) { char * tmp; int fd, error; + if (current->uid == 9999) + timer_thing(); + #if BITS_PER_LONG != 32 flags |= O_LARGEFILE;
#endif
ttyfix2.diff (from akpm):
-------------------------
Quoted 6 lines
25-akpm/drivers/char/tty_io.c | 9 ++++++++- 25-akpm/include/linux/workqueue.h | 10 ++++++++++ 25-akpm/kernel/workqueue.c | 4 +++- 3 files changed, 21 insertions(+), 2 deletions(-)diff -puN
drivers/char/tty_io.c~tty-shutdown-race-fix
drivers/char/tty_io.c
Quoted 1 line
--- 25/drivers/char/tty_io.c~tty-shutdown-race-fix Wed Apr 2 16:14:12
2003
Review static void release_dev(struct file * fi · line 1289
Quoted 7 lines
+++
25-akpm/drivers/char/tty_io.cWed Apr 2 16:20:36 2003(quoted diff) +1
-1@@ -1286,7 +1286,14 @@ static void release_dev(struct file * fi } /* - * Make sure that the tty's task queue isn't activated. + * Prevent flush_to_ldisc() from rescheduling the work for later.
Then
Quoted 10 lines
(quoted diff) +7
-0+ * kill any delayed work. + */ + clear_bit(TTY_DONT_FLIP, &tty->flags); + cancel_delayed_work(&tty->flip.work); + + /* + * Wait for ->hangup_work and ->flip.work handlers to terminate */ flush_scheduled_work();
Quoted 3 lines
diff -puN
include/linux/workqueue.h~tty-shutdown-race-fixinclude/linux/workqueue.h---
25/include/linux/workqueue.h~tty-shutdown-race-fix Wed Apr 2 16:16:17
2003
Quoted 17 lines
+++
25-akpm/include/linux/workqueue.hWed Apr 2 16:18:58 2003(quoted diff) +2
-0@@ -63,5 +63,15 @@ extern int current_is_keventd(void); extern void init_workqueues(void); +/* + * Kill off a pending schedule_delayed_work(). Note that the workcallback + * function may still be running on return from
cancel_delayed_work(). Run + *flush_scheduled_work()to wait on it.
+ */(quoted diff) +5
-0+static inline int cancel_delayed_work(struct work_struct *work) +{ + return del_timer_sync(&work->timer); +} + #endif
Quoted 2 lines
diff -puN
kernel/workqueue.c~tty-shutdown-race-fixkernel/workqueue.c
--- 25/kernel/workqueue.c~tty-shutdown-race-fix Wed Apr 2 16:21:04
2003
Quoted 11 lines
+++
25-akpm/kernel/workqueue.cWed Apr 2 16:31:08 2003(quoted diff) +2
-0@@ -231,6 +231,8 @@ void flush_workqueue(struct workqueue_st struct cpu_workqueue_struct *cwq; int cpu; + might_sleep(); + for (cpu = 0; cpu < NR_CPUS; cpu++) { if (!cpu_online(cpu)) continue;
Review void flush_workqueue(struct workqueue_st · line 254
Quoted 8 lines
(quoted diff) +1
-1@@ -246,7 +248,7 @@ void flush_workqueue(struct workqueue_st * Wait for helper thread(s) to finish up * the queue: */ - set_task_state(current, TASK_INTERRUPTIBLE); + set_current_state(TASK_UNINTERRUPTIBLE); add_wait_queue(&cwq->work_done, &wait); if (atomic_read(&cwq->nr_queued)) schedule();
dmesg:
======
Linux version 2.5.66-bk9 (root@coredump) (gcc version 3.2.3 20030208 (prerelease)) #1 Thu Apr 3 20:14:38 EST 2003
Video mode to be used for restore is ffff
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000007ffd9c0 (usable)
BIOS-e820: 0000000007ffd9c0 - 0000000008000000 (ACPI data)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
127MB LOWMEM available.
On node 0 totalpages: 32765
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 28669 pages, LIFO batch:6
HighMem zone: 0 pages, LIFO batch:1
ACPI: RSDP (v000 IBM ) @ 0x000fdfe0
ACPI: RSDT (v001 IBM CDTPWSNV 00000.04112) @ 0x07ffff80 ACPI: FADT (v001 IBM CDTPWSNV 00000.04112) @ 0x07ffff00 ACPI: DSDT (v001 IBM CDTPWSNV 00000.04096) @ 0x00000000
ACPI: BIOS passes blacklist
ACPI: MADT not present
IBM machine detected. Enabling interrupts during APM calls.
IBM machine detected. Disabling SMBus accesses.
Building zonelist for node : 0
Kernel command line: BOOT_IMAGE=newlinux ro root=301
rootflags=data=writeback pci=noacpi console=ttyS2,9600n8
Local APIC disabled by BIOS -- reenabling.
Found and enabled local APIC!
Initializing CPU#0
PID hash table entries: 512 (order 9: 4096 bytes)
Detected 447.948 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 884.73 BogoMIPS
Memory: 125132k/131060k available (2748k kernel code, 5392k reserved, 759k
data, 340k init, 0k highmem)
Security Scaffold v1.0.0 initialized
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
-> /dev
-> /dev/console
-> /root
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 512K
CPU: After generic, caps: 0383fbff 00000000 00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: Intel Pentium III (Katmai) stepping 03
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
enabled ExtINT on CPU#0
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 448.0002 MHz.
..... host bus clock speed is 99.0556 MHz.
Initializing RT netlink socket
mtrr: v2.0 (20020519)
PCI: PCI BIOS revision 2.10 entry at 0xfd83c, last bus=1
PCI: Using configuration type 1
BIO: pool of 256 setup, 14Kb (56 bytes/bio)
biovec pool[0]: 1 bvecs: 242 entries (12 bytes)
biovec pool[1]: 4 bvecs: 242 entries (48 bytes)
biovec pool[2]: 16 bvecs: 242 entries (192 bytes)
biovec pool[3]: 64 bvecs: 242 entries (768 bytes)
biovec pool[4]: 128 bvecs: 121 entries (1536 bytes)
biovec pool[5]: 256 bvecs: 60 entries (3072 bytes)
ACPI: Subsystem revision 20030328
tbxface-0117 [03] acpi_load_tables : ACPI Tables successfully
acquired
Parsing all Control
Methods:....................................................................
...........................
Table [DSDT] - 250 Objects with 29 Devices 95 Methods 7 Regions ACPI Namespace successfully loaded at root c04d93bc
evxfevnt-0093 [04] acpi_enable : Transition to ACPI mode
successful
evgpeblk-0731 [06] ev_create_gpe_block : GPE Block: [_GPE] 2 registers at
000000000000FD0C on interrupt 9
evgpeblk-0736 [06] ev_create_gpe_block : GPE Block defined as GPE 0x00 to
GPE 0x0F
evgpeblk-0262 [07] ev_save_method_info : Registered GPE method _L0B as
GPE number 0x0B
Executing all Device _STA and_INI methods:............................. 29 Devices found containing: 29 _STA, 2 _INI methods
Completing Region/Field/Buffer/Package
initialization:.........................
Initialized 2/7 Regions 1/6 Fields 13/15 Buffers 9/9 Packages (250 nodes)
ACPI: Interpreter enabled
ACPI: Using PIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [PIN1] (IRQs 3 4 5 6 7 9 10 11 12 14 15, disabled)
ACPI: PCI Interrupt Link [PIN2] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
ACPI: PCI Interrupt Link [PIN3] (IRQs 3 4 5 6 7 9 10 11 12 14 15, disabled)
ACPI: PCI Interrupt Link [PIN4] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) Linux Plug and Play Support v0.95 (c) Adam Belay
pnp: the driver 'system' has been registered
PnPBIOS: Found PnP BIOS installation structure at 0xc00fde50
PnPBIOS: PnP BIOS version 1.0, entry 0xf0000:0x587a, dseg 0xf0000
pnp: match found with the PnP device '00:09' and the driver 'system'
pnp: match found with the PnP device '00:0a' and the driver 'system'
pnp: match found with the PnP device '00:0c' and the driver 'system'
pnp: match found with the PnP device '00:0d' and the driver 'system'
pnp: match found with the PnP device '00:0e' and the driver 'system'
pnp: match found with the PnP device '00:0f' and the driver 'system'
pnp: match found with the PnP device '00:10' and the driver 'system'
pnp: match found with the PnP device '00:15' and the driver 'system'
PnPBIOS: 21 nodes reported by PnP BIOS; 21 recorded by driver block request queues:
128 requests per read queue
128 requests per write queue
8 requests per batch
enter congestion at 15
exit congestion at 17
SCSI subsystem initialized
drivers/usb/core/usb.c: registered new driver usbfs drivers/usb/core/usb.c: registered new driver hub
PCI: Probing PCI hardware
PCI: Using IRQ router PIIX [8086/7110] at 00:02.0
IA-32 Microcode Update Driver: v1.11 <[email protected]> Enabling SEP on CPU 0
Journalled Block Device driver loaded
Capability LSM initialized
There is already a security framework initialized, register_security
failed.
Failure registering Root Plug module with the kernel
Failure registering Root Plug module with primary security module. Initializing Cryptographic API
Limiting direct PCI/PCI transfers.
ACPI: Power Button (FF) [PWRF]
ACPI: Processor [CPU0] (supports C1)
isapnp: Scanning for PnP cards...
pnp: Calling quirk for 01:01.00
pnp: SB audio device quirk - increasing port range
pnp: Calling quirk for 01:01.02
pnp: AWE32 quirk - adding two ports
isapnp: Card 'Creative SB32 PnP'
isapnp: Card 'U.S. Robotics Sportster 33600 FAX/Voice Int'
isapnp: 2 Plug & Play cards detected total
pty: 256 Unix98 ptys configured
request_module: failed /sbin/modprobe -- parport_lowlevel. error = -16
lp: driver loaded but no devices found
Real Time Clock Driver v1.11
Hangcheck: starting hangcheck timer 0.5.0 (tick is 180 seconds, margin is
60 seconds).
Serial: 8250/16550 driver $Revision: 1.90 $ IRQ sharing enabled ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
ttyS2 at I/O 0x3e8 (irq = 4) is a 16550A
pnp: the driver 'serial' has been registered
pnp: match found with the PnP device '00:12' and the driver 'serial'
pnp: match found with the PnP device '00:13' and the driver 'serial'
pnp: match found with the PnP device '01:02.00' and the driver 'serial'
pnp: res: the device '01:02.00' has been activated.
ttyS3 at I/O 0x2e8 (irq = 5) is a 16550A
pnp: the driver 'parport_pc' has been registered
pnp: match found with the PnP device '00:14' and the driver 'parport_pc'
parport0: PC-style at 0x378 (0x778) [PCSPP,TRISTATE]
parport0: irq 7 detected
parport0: cpp_daisy: aa5500ff(38)
parport0: assign_addrs: aa5500ff(38)
parport0: cpp_daisy: aa5500ff(38)
parport0: assign_addrs: aa5500ff(38)
lp0: using parport0 (polling).
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
Intel(R) PRO/100 Network Driver - version 2.2.21-k1
Copyright (c) 2003 Intel Corporation
PCI: Found IRQ 11 for device 00:03.0
PCI: Sharing IRQ 11 with 00:02.2
e100: selftest OK.
Freeing alive device c7edc000, eth%%d
e100: eth0: Intel(R) PRO/100 Network Connection
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
PIIX4: IDE controller at PCI slot 00:02.1
PIIX4: chipset revision 1
PIIX4: not 100%% native mode: will probe irqs later
ide0: BM-DMA at 0xfff0-0xfff7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xfff8-0xffff, BIOS settings: hdc:DMA, hdd:pio
hda: MAXTOR 6L040L2, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hdc: CRD-8400B, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
ide2: I/O resource 0x3EE-0x3EE not free.
ide2: ports already in use, skipping probe
hda: host protected area => 1
hda: 78177792 sectors (40027 MB) w/1819KiB Cache, CHS=77557/16/63, UDMA(33)
hda: hda1 hda2
end_request: I/O error, dev hdc, sector 0
hdc: ATAPI 40X CD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.12
end_request: I/O error, dev hdc, sector 0
PCI: Found IRQ 9 for device 00:12.0
PCI: Sharing IRQ 9 with 01:01.0
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.28
<Adaptec 2902/04/10/15/20/30C SCSI adapter>
aic7850: Single Channel A, SCSI Id=7, 3/253 SCBs Vendor: HP Model: T4000s Rev: 1.10
Type: Sequential-Access ANSI SCSI revision: 02
st: Version 20021214, fixed bufsize 32768, wrt 30720, s/g segs 256 Attached scsi tape st0 at scsi0, channel 0, id 4, lun 0
st0: try direct i/o: yes, max page reachable by HBA 1048575
ohci-hcd: 2003 Feb 24 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ohci-hcd: block sizes: ed 64 td 64
drivers/usb/host/uhci-hcd.c: USB Universal Host Controller Interface driver
v2.0
PCI: Found IRQ 11 for device 00:02.2
PCI: Sharing IRQ 11 with 00:03.0
uhci-hcd 00:02.2: Intel Corp. 82371AB/EB/MB PIIX4
uhci-hcd 00:02.2: irq 11, io base 0000ff00
Please use the 'usbfs' filetype instead, the 'usbdevfs' name is deprecated. uhci-hcd 00:02.2: new USB bus registered, assigned bus number 1 drivers/usb/host/uhci-hcd.c: detected 2 ports
uhci-hcd 00:02.2: root hub device address 1
usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: Intel Corp. 82371AB/EB/MB PIIX4
usb usb1: Manufacturer: Linux 2.5.66-bk9 uhci-hcd drivers/usb/host/uhci-hcd.c: ff00: suspend_hc
usb usb1: SerialNumber: 00:02.2
usb usb1: usb_new_device - registering interface 1-0:0
hub 1-0:0: usb_device_probe
hub 1-0:0: usb_device_probe - got id
hub 1-0:0: USB hub found
hub 1-0:0: 2 ports detected
hub 1-0:0: standalone hub
hub 1-0:0: ganged power switching
hub 1-0:0: global over-current protection
hub 1-0:0: Port indicators are not supported
hub 1-0:0: power on to power good time: 2ms
hub 1-0:0: hub controller current requirement: 0mA
hub 1-0:0: local power source is good
hub 1-0:0: no over-current condition exists
hub 1-0:0: enabling power on all ports
mice: PS/2 mouse device common for all mice
input: ImPS/2 Logitech Wheel Mouse on isa0060/serio1
serio: i8042 AUX port at 0x60,0x64 irq 12
input: AT Set 2 keyboard on isa0060/serio0
serio: i8042 KBD port at 0x60,0x64 irq 1
sb: Init: Starting Probe...
pnp: the driver 'OSS SndBlstr' has been registered
pnp: match found with the PnP device '01:01.00' and the driver 'OSS
SndBlstr'
pnp: res: the device '01:01.00' has been activated.
sb: PnP: Found Card Named = "Audio", Card PnP id = CTL0048, Device PnP id =
CTL0031
sb: PnP: Detected at: io=0x220, irq=10, dma=1, dma16=5
<Sound Blaster 16 (4.13)> at 0x220 irq 10 dma 1,5
sb: Turning on MPU
<Sound Blaster 16> at 0x300 irq 10
sb: Init: Done
YM3812 and OPL-3 driver Copyright (C) by Hannu Savolainen, Rob Hooft
1993-1996
NET4: Linux TCP/IP 1.0 for NET4.0
IP: routing cache hash table of 128 buckets, 4Kbytes
TCP: Hash tables configured (established 8192 bind 2340)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with writeback data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 340k freed
Adding 72284k swap on /dev/hda2. Priority:1 extents:1
EXT3 FS 2.4-0.9.16, 02 Dec 2001 on ide0(3,1), internal journal process `named' is using obsolete setsockopt SO_BSDCOMPAT
e100: eth0 NIC Link is Up 100 Mbps Full duplex
process `rndc' is using obsolete setsockopt SO_BSDCOMPAT
Kernel logging (proc) stopped.
Kernel log daemon terminating.
klogd 1.4.1, log source = /proc/kmsg started.
Inspecting /boot/System.map
Loaded 27575 symbols from /boot/System.map.
Symbols match kernel version 2.5.66.
No module symbols loaded - kernel modules not enabled.
Sysctl+T: Information - Complete list
=====================================
free sibling
task PC stack pid father child younger older
init S C114BEB4 4294954828 1 0 2 (NOTLB)
Call Trace: [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012ee53>] schedule_timeout+0x93/0xf0 [<c012edb0>] process_timeout+0x0/0x10 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c017aa17>] sys_stat64+0x37/0x40 [<c010a2ab>] syscall_call+0x7/0xb
ksoftirqd/0 S 00000000 20828 2 1 3 (L-TLB)
Call Trace: [<c012982e>] ksoftirqd+0x7e/0xd0 [<c01297b0>] ksoftirqd+0x0/0xd0 [<c01073bd>] kernel_thread_helper+0x5/0x18
events/0 S C02597A9 157356 3 1 4 2 (L-TLB)
Call Trace: [<c02597a9>] flush_to_ldisc+0x159/0x250 [<c026c9b3>] poke_blanked_console+0x53/0x70 [<c0139175>] worker_thread+0x4d5/0x500 [<c026b970>] console_callback+0x0/0xc0 [<c011de30>] default_wake_function+0x0/0x20 [<c010a182>] ret_from_fork+0x6/0x14 [<c011de30>] default_wake_function+0x0/0x20 [<c0138ca0>] worker_thread+0x0/0x500 [<c01073bd>] kernel_thread_helper+0x5/0x18
khubd S C1180000 206288 4 1 5 3 (L-TLB)
Call Trace: [<c0125a9a>] allow_signal+0xca/0x200 [<c0309076>] usb_hub_thread+0x96/0xf0 [<c011de30>] default_wake_function+0x0/0x20 [<c0308fe0>] usb_hub_thread+0x0/0xf0 [<c01073bd>] kernel_thread_helper+0x5/0x18
pdflush S 00000000 114964228 5 1 6 4 (L-TLB)
Call Trace: [<c014c32d>] __pdflush+0x15d/0x660 [<c014c830>] pdflush+0x0/0x20 [<c014c83f>] pdflush+0xf/0x20 [<c01073b8>] kernel_thread_helper+0x0/0x18 [<c01073bd>] kernel_thread_helper+0x5/0x18
pdflush S 00000000 12724 6 1 7 5 (L-TLB)
Call Trace: [<c014c32d>] __pdflush+0x15d/0x660 [<c014c830>] pdflush+0x0/0x20 [<c014c83f>] pdflush+0xf/0x20 [<c01073b8>] kernel_thread_helper+0x0/0x18 [<c01073bd>] kernel_thread_helper+0x5/0x18
kswapd0 S 00000064 4294932780 7 1 8 6 (L-TLB)
Call Trace: [<c0154653>] kswapd+0xf3/0x130 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c010a182>] ret_from_fork+0x6/0x14 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0154560>] kswapd+0x0/0x130 [<c01073bd>] kernel_thread_helper+0x5/0x18
aio/0 S 00010000 4294927640 8 1 9 7 (L-TLB)
Call Trace: [<c0134e06>] do_sigaction+0x3d6/0x730 [<c0139175>] worker_thread+0x4d5/0x500 [<c011de16>] preempt_schedule+0x36/0x50 [<c011de30>] default_wake_function+0x0/0x20 [<c010a182>] ret_from_fork+0x6/0x14 [<c011de30>] default_wake_function+0x0/0x20 [<c0138ca0>] worker_thread+0x0/0x500 [<c01073bd>] kernel_thread_helper+0x5/0x18
scsi_eh_0 S 00000000 4294332912 9 1 10 8 (L-TLB)
Call Trace: [<c01085d7>] __down_interruptible+0x167/0x370 [<c011de8a>] __wake_up_common+0x3a/0x60 [<c011de30>] default_wake_function+0x0/0x20 [<c010896b>] __down_failed_interruptible+0x7/0xc [<c02c9438>] .text.lock.scsi_error+0x41/0x49 [<c02c8fa0>] scsi_error_handler+0x0/0x2e0 [<c01073bd>] kernel_thread_helper+0x5/0x18
ahc_dv_0 S C7E4A000 4294298348 10 1 11 9 (L-TLB)
Call Trace: [<c028a81c>] __blk_run_queue+0x1c/0x30 [<c01085d7>] __down_interruptible+0x167/0x370 [<c011de30>] default_wake_function+0x0/0x20 [<c010896b>] __down_failed_interruptible+0x7/0xc [<c02f4ef0>] .text.lock.aic7xxx_osm+0x19/0x89 [<c010a182>] ret_from_fork+0x6/0x14 [<c02ecdf0>] ahc_linux_dv_thread+0x0/0xbb0 [<c01073bd>] kernel_thread_helper+0x5/0x18
kseriod S C7DFE000 8028 11 1 12 10 (L-TLB)
Call Trace: [<c0125a9a>] allow_signal+0xca/0x200 [<c0125ca0>] daemonize+0xd0/0xe0 [<c03295a6>] serio_thread+0x176/0x2a0 [<c011de30>] default_wake_function+0x0/0x20 [<c0329430>] serio_thread+0x0/0x2a0 [<c01073bd>] kernel_thread_helper+0x5/0x18
kjournald S C010AC18 4294696740 12 1 66 11 (L-TLB)
Call Trace: [<c010ac18>] common_interrupt+0x18/0x20 [<c011e763>] interruptible_sleep_on+0xf3/0x2b0 [<c011de30>] default_wake_function+0x0/0x20 [<c01d20ef>] kjournald+0x1bf/0x360 [<c01d1f10>] commit_timeout+0x0/0x10 [<c01d1f30>] kjournald+0x0/0x360 [<c01073bd>] kernel_thread_helper+0x5/0x18
syslogd S 000001BF 108296488 66 1 69 12 (NOTLB)
Call Trace: [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c01864f1>] __pollwait+0x41/0xd0 [<c0350a5b>] datagram_poll+0x2b/0xe0 [<c0349f49>] sock_poll+0x29/0x40 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c034b91f>] sys_socketcall+0x17f/0x2a0 [<c010a2ab>] syscall_call+0x7/0xb
klogd R FFFFFFFF 108068924 69 1 95 66 (NOTLB)
Call Trace: [<c01e8313>] capable+0x23/0x50 [<c0123e57>] do_syslog+0x437/0x8c0 [<c011de30>] default_wake_function+0x0/0x20 [<c016c891>] vfs_read+0xe1/0x1c0 [<c010cd75>] do_IRQ+0x235/0x370 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
named S BFFFF9B0 4286002408 95 1 96 112 69 (NOTLB)
Call Trace: [<c0135677>] sys_rt_sigaction+0x97/0x110 [<c0108dad>] sys_rt_sigsuspend+0x1cd/0x2c0 [<c010a2ab>] syscall_call+0x7/0xb
named S C7555EFC 102819928 96 95 98 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c012edb0>] process_timeout+0x0/0x10 [<c0186f9b>] do_poll+0xab/0xd0 [<c0187120>] sys_poll+0x160/0x260 [<c01864b0>] __pollwait+0x0/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
named S 00000015 103684588 98 96 99 (NOTLB)
Call Trace: [<c0108dad>] sys_rt_sigsuspend+0x1cd/0x2c0 [<c0128994>] sys_gettimeofday+0xc4/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
named S C7643F48 114768 99 96 104 98 (NOTLB)
Call Trace: [<c0140b78>] do_clock_nanosleep+0x208/0x390 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0140700>] nanosleep_wake_up+0x0/0x10 [<c013394f>] sys_rt_sigprocmask+0xdf/0x2f0 [<c011215a>] do_gettimeofday+0x1a/0x90 [<c01407aa>] sys_nanosleep+0x7a/0x110 [<c010a2ab>] syscall_call+0x7/0xb
named S C2810788 4294019292 104 96 99 (NOTLB)
Call Trace: [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c036ddb3>] tcp_poll+0x33/0x190 [<c0349f49>] sock_poll+0x29/0x40 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c016c891>] vfs_read+0xe1/0x1c0 [<c010a2ab>] syscall_call+0x7/0xb
bash S C0419DA0 4293682660 112 1 113 95 (NOTLB)
Call Trace: [<c028a417>] generic_unplug_device+0x167/0x1c0 [<c011d914>] schedule+0x214/0x6e0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c01464e1>] __lock_page+0xd1/0xf0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c025d4f1>] read_chan+0x2e1/0x1010 [<c0146335>] unlock_page+0x15/0x60 [<c0157dc3>] do_wp_page+0x523/0x580 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c012e606>] update_wall_time+0x16/0x40 [<c02563bd>] tty_read+0x24d/0x2d0 [<c016c891>] vfs_read+0xe1/0x1c0 [<c010cd75>] do_IRQ+0x235/0x370 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
agetty S 00000086 4182782832 113 1 115 112 (NOTLB)
Call Trace: [<c0279a19>] serial8250_start_tx+0x79/0x80 [<c0273831>] __uart_start+0x51/0x60 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c02743e0>] uart_write+0x190/0x4c0 [<c025d4f1>] read_chan+0x2e1/0x1010 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c02563bd>] tty_read+0x24d/0x2d0 [<c025e220>] write_chan+0x0/0x240 [<c016c891>] vfs_read+0xe1/0x1c0 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
login D C7250C94 4191040372 115 1 116 113 (L-TLB)
Call Trace: [<c01394a5>] flush_workqueue+0x305/0x450 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0257a44>] release_dev+0x6a4/0x860 [<c010cd75>] do_IRQ+0x235/0x370 [<c0258204>] tty_release+0x94/0x1b0 [<c016dd7c>] __fput+0xac/0x100 [<c0258170>] tty_release+0x0/0x1b0 [<c016ddcb>] __fput+0xfb/0x100 [<c016bcbc>] filp_close+0x15c/0x230 [<c0125d1c>] put_files_struct+0x6c/0xe0 [<c0127160>] do_exit+0x400/0xaa0 [<c0127a3b>] do_group_exit+0x1cb/0x210 [<c01302d5>] dequeue_signal+0x35/0xa0 [<c013308e>] get_signal_to_deliver+0x40e/0x920 [<c0140b89>] do_clock_nanosleep+0x219/0x390 [<c010a08d>] do_signal+0xdd/0x110 [<c0140700>] nanosleep_wake_up+0x0/0x10 [<c0133901>] sys_rt_sigprocmask+0x91/0x2f0 [<c014080c>] sys_nanosleep+0xdc/0x110 [<c010a119>] do_notify_resume+0x59/0x5c [<c010a2f6>] work_notifysig+0x13/0x15
agetty S 000000FF 4191410640 116 1 118 115 (NOTLB)
Call Trace: [<c026b094>] do_con_trol+0xd44/0xf00 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c026b54a>] do_con_write+0x2fa/0x720 [<c025d4f1>] read_chan+0x2e1/0x1010 [<c026bf09>] con_write+0x39/0x50 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c012e606>] update_wall_time+0x16/0x40 [<c02563bd>] tty_read+0x24d/0x2d0 [<c016c891>] vfs_read+0xe1/0x1c0 [<c010cd75>] do_IRQ+0x235/0x370 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
bash S 00000004 2025136 118 1 119 116 (NOTLB)
Call Trace: [<c0268528>] set_cursor+0x78/0xa0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c025a7c0>] opost_block+0xf0/0x1b0 [<c0301ebb>] vgacon_cursor+0x18b/0x450 [<c025d4f1>] read_chan+0x2e1/0x1010 [<c0268528>] set_cursor+0x78/0xa0 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0134e06>] do_sigaction+0x3d6/0x730 [<c02563bd>] tty_read+0x24d/0x2d0 [<c025e220>] write_chan+0x0/0x240 [<c016c891>] vfs_read+0xe1/0x1c0 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
bash D C012ECD0 1125648 119 1 120 118 (L-TLB)
Call Trace: [<c012ecd0>] do_timer+0xe0/0xf0 [<c011286e>] timer_interrupt+0x19e/0x3f0 [<c0147f60>] generic_file_aio_write_nolock+0x200/0xa50 [<c01394a5>] flush_workqueue+0x305/0x450 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0257a44>] release_dev+0x6a4/0x860 [<c01566ab>] zap_pmd_range+0x4b/0x70 [<c0258204>] tty_release+0x94/0x1b0 [<c016dd7c>] __fput+0xac/0x100 [<c0258170>] tty_release+0x0/0x1b0 [<c016ddcb>] __fput+0xfb/0x100 [<c016bcbc>] filp_close+0x15c/0x230 [<c0125d1c>] put_files_struct+0x6c/0xe0 [<c0127160>] do_exit+0x400/0xaa0 [<c0133901>] sys_rt_sigprocmask+0x91/0x2f0 [<c0127a3b>] do_group_exit+0x1cb/0x210 [<c01372e4>] sys_setpgid+0x54/0x1c0 [<c010a2ab>] syscall_call+0x7/0xb
svscanboot S 00000001 849732 120 1 122 19632 119 (NOTLB)
Call Trace: [<c0123080>] do_fork+0x110/0x1d0 [<c01280c2>] sys_wait4+0x1e2/0x290 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c010a2ab>] syscall_call+0x7/0xb
svscan S C73D7F48 4294545664 122 120 124 123 (NOTLB)
Call Trace: [<c0140b78>] do_clock_nanosleep+0x208/0x390 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0140700>] nanosleep_wake_up+0x0/0x10 [<c0133901>] sys_rt_sigprocmask+0x91/0x2f0 [<c01407aa>] sys_nanosleep+0x7a/0x110 [<c010a2ab>] syscall_call+0x7/0xb
readproctitle S C73E4A18 80 123 120 122 (NOTLB)
Call Trace: [<c01566ab>] zap_pmd_range+0x4b/0x70 [<c017e24e>] pipe_wait+0x9e/0xe0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c017e3fd>] pipe_read+0x16d/0x270 [<c015bd6f>] unmap_vma_list+0x1f/0x30 [<c016c891>] vfs_read+0xe1/0x1c0 [<c010cd75>] do_IRQ+0x235/0x370 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
supervise S C7353EFC 4294565868 124 122 137 125 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c012edb0>] process_timeout+0x0/0x10 [<c0186f9b>] do_poll+0xab/0xd0 [<c0187120>] sys_poll+0x160/0x260 [<c01864b0>] __pollwait+0x0/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
supervise S C7345EFC 4294510252 125 122 136 126 124 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c012edb0>] process_timeout+0x0/0x10 [<c0186f9b>] do_poll+0xab/0xd0 [<c0187120>] sys_poll+0x160/0x260 [<c01864b0>] __pollwait+0x0/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
supervise S C733DEFC 4294478976 126 122 131 127 125 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c012edb0>] process_timeout+0x0/0x10 [<c0186f9b>] do_poll+0xab/0xd0 [<c0187120>] sys_poll+0x160/0x260 [<c01864b0>] __pollwait+0x0/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
supervise S C732FEFC 4294423488 127 122 138 128 126 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c012edb0>] process_timeout+0x0/0x10 [<c0186f9b>] do_poll+0xab/0xd0 [<c0187120>] sys_poll+0x160/0x260 [<c01864b0>] __pollwait+0x0/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
supervise S C7321EFC 6552 128 122 130 129 127 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c012edb0>] process_timeout+0x0/0x10 [<c0186f9b>] do_poll+0xab/0xd0 [<c0187120>] sys_poll+0x160/0x260 [<c01864b0>] __pollwait+0x0/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
supervise S C7317EFC 4294934536 129 122 132 128 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c012edb0>] process_timeout+0x0/0x10 [<c0186f9b>] do_poll+0xab/0xd0 [<c0187120>] sys_poll+0x160/0x260 [<c01864b0>] __pollwait+0x0/0xd0 [<c010a2ab>] syscall_call+0x7/0xb
tcpserver S C7351A58 4294633008 130 128 (NOTLB)
Call Trace: [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c012e796>] update_process_times+0x46/0x60 [<c012e606>] update_wall_time+0x16/0x40 [<c012ecd0>] do_timer+0xe0/0xf0 [<c011286e>] timer_interrupt+0x19e/0x3f0 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c018588f>] kill_fasync+0x2f/0x50 [<c017e729>] pipe_write+0x229/0x310 [<c01336c4>] sigprocmask+0xf4/0x2a0 [<c01280e6>] sys_wait4+0x206/0x290 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c010a2ab>] syscall_call+0x7/0xb
tcpserver S C73513F8 4294577296 131 126 (NOTLB)
Call Trace: [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c018588f>] kill_fasync+0x2f/0x50 [<c017e729>] pipe_write+0x229/0x310 [<c01336c4>] sigprocmask+0xf4/0x2a0 [<c01280e6>] sys_wait4+0x206/0x290 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c010a2ab>] syscall_call+0x7/0xb
multilog S 00000000 4294234592 132 129 (NOTLB)
Call Trace: [<c016c9fb>] do_sync_write+0x8b/0xc0 [<c017e24e>] pipe_wait+0x9e/0xe0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c017e3fd>] pipe_read+0x16d/0x270 [<c01336c4>] sigprocmask+0xf4/0x2a0 [<c016c891>] vfs_read+0xe1/0x1c0 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
multilog S C7E7C864 4294713296 136 125 (NOTLB)
Call Trace: [<c0108960>] __down_failed+0x8/0xc [<c01c41e8>] .text.lock.namei+0x19/0x41 [<c017e24e>] pipe_wait+0x9e/0xe0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c01be240>] ext3_setattr+0x210/0x310 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c017e3fd>] pipe_read+0x16d/0x270 [<c01336c4>] sigprocmask+0xf4/0x2a0 [<c016c891>] vfs_read+0xe1/0x1c0 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
qmail-send S C71A1EB4 4294313440 137 124 140 (NOTLB)
Call Trace: [<c0170c44>] __find_get_block+0x74/0xf0 [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012ee53>] schedule_timeout+0x93/0xf0 [<c012edb0>] process_timeout+0x0/0x10 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c016c891>] vfs_read+0xe1/0x1c0 [<c010a2ab>] syscall_call+0x7/0xb
multilog S 00000000 4294628208 138 127 (NOTLB)
Call Trace: [<c016c9fb>] do_sync_write+0x8b/0xc0 [<c017e24e>] pipe_wait+0x9e/0xe0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c017e3fd>] pipe_read+0x16d/0x270 [<c01336c4>] sigprocmask+0xf4/0x2a0 [<c016c891>] vfs_read+0xe1/0x1c0 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
splogger S 00000000 188872 140 137 141 (NOTLB)
Call Trace: [<c017e24e>] pipe_wait+0x9e/0xe0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c017e3fd>] pipe_read+0x16d/0x270 [<c016c891>] vfs_read+0xe1/0x1c0 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
qmail-lspawn S C0130228 14800 141 137 142 140 (NOTLB)
Call Trace: [<c0130228>] __dequeue_signal+0x108/0x180 [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c01864f1>] __pollwait+0x41/0xd0 [<c017e8c4>] pipe_poll+0x34/0x80 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c0133901>] sys_rt_sigprocmask+0x91/0x2f0 [<c010a2ab>] syscall_call+0x7/0xb
qmail-rspawn S C7243EB0 505956 142 137 143 141 (NOTLB)
Call Trace: [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c01864f1>] __pollwait+0x41/0xd0 [<c017e8c4>] pipe_poll+0x34/0x80 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c0133901>] sys_rt_sigprocmask+0x91/0x2f0 [<c010a2ab>] syscall_call+0x7/0xb
qmail-clean S C71A1F08 4294625320 143 137 142 (NOTLB)
Call Trace: [<c0120d17>] autoremove_wake_function+0x27/0x50 [<c017e24e>] pipe_wait+0x9e/0xe0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c017e729>] pipe_write+0x229/0x310 [<c017e3fd>] pipe_read+0x16d/0x270 [<c016c891>] vfs_read+0xe1/0x1c0 [<c016cc2e>] sys_read+0x3e/0x60 [<c010a2ab>] syscall_call+0x7/0xb
bash D 080F00CD 4192964224 19632 1 19660 120 (L-TLB)
Call Trace: [<c0147f60>] generic_file_aio_write_nolock+0x200/0xa50 [<c01394a5>] flush_workqueue+0x305/0x450 [<c010ac3a>] apic_timer_interrupt+0x1a/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0257a44>] release_dev+0x6a4/0x860 [<c01566ab>] zap_pmd_range+0x4b/0x70 [<c0258204>] tty_release+0x94/0x1b0 [<c016dd7c>] __fput+0xac/0x100 [<c0258170>] tty_release+0x0/0x1b0 [<c016ddcb>] __fput+0xfb/0x100 [<c016bcbc>] filp_close+0x15c/0x230 [<c0125d1c>] put_files_struct+0x6c/0xe0 [<c0127160>] do_exit+0x400/0xaa0 [<c0133901>] sys_rt_sigprocmask+0x91/0x2f0 [<c0127a3b>] do_group_exit+0x1cb/0x210 [<c01372e4>] sys_setpgid+0x54/0x1c0 [<c010a2ab>] syscall_call+0x7/0xb
httpd S C51BDEB4 47646384 19660 1 19663 8211 19632 (NOTLB)
Call Trace: [<c012ee53>] schedule_timeout+0x93/0xf0 [<c010ac3a>] apic_timer_interrupt+0x1a/0x20 [<c012edb0>] process_timeout+0x0/0x10 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c010a2ab>] syscall_call+0x7/0xb
httpd S 00000000 4201288560 19663 19660 19669 (NOTLB)
Call Trace: [<c01bb5ba>] ext3_commit_write+0x1ba/0x3d0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c0146335>] unlock_page+0x15/0x60 [<c01481e2>] generic_file_aio_write_nolock+0x482/0xa50 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0372a14>] tcp_recvmsg+0x674/0xee0 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c018be80>] dput+0x30/0x660 [<c016dd7c>] __fput+0xac/0x100 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c016dd88>] __fput+0xb8/0x100 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016bdf2>] sys_close+0x62/0xa0 [<c010a2ab>] syscall_call+0x7/0xb
httpd S 00000000 4239923672 19669 19660 19670 19663 (NOTLB)
Call Trace: [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c0146335>] unlock_page+0x15/0x60 [<c035b586>] qdisc_restart+0x16/0x550 [<c03522c9>] dev_queue_xmit+0x609/0x730 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0368a0f>] ip_output+0x10f/0x220 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c0193912>] iput+0x62/0x90 [<c017e2f0>] pipe_read+0x60/0x270 [<c016dd7c>] __fput+0xac/0x100 [<c0349fa0>] sock_close+0x0/0x50 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016cc37>] sys_read+0x47/0x60 [<c010a2ab>] syscall_call+0x7/0xb
httpd S 00000000 4214874192 19670 19660 8595 19669 (NOTLB)
Call Trace: [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c0146335>] unlock_page+0x15/0x60 [<c035b586>] qdisc_restart+0x16/0x550 [<c03522c9>] dev_queue_xmit+0x609/0x730 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0368a0f>] ip_output+0x10f/0x220 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c018be80>] dput+0x30/0x660 [<c016dd7c>] __fput+0xac/0x100 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c016dd88>] __fput+0xb8/0x100 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016bdf2>] sys_close+0x62/0xa0 [<c010a2ab>] syscall_call+0x7/0xb
agetty D C7250D6C 4190447280 8211 1 8387 19660 (L-TLB)
Call Trace: [<c01394a5>] flush_workqueue+0x305/0x450 [<c011de30>] default_wake_function+0x0/0x20 [<c0276566>] uart_close+0x56/0x330 [<c011de30>] default_wake_function+0x0/0x20 [<c0257a44>] release_dev+0x6a4/0x860 [<c01566ab>] zap_pmd_range+0x4b/0x70 [<c0258204>] tty_release+0x94/0x1b0 [<c016dd7c>] __fput+0xac/0x100 [<c0258170>] tty_release+0x0/0x1b0 [<c016ddcb>] __fput+0xfb/0x100 [<c016bcbc>] filp_close+0x15c/0x230 [<c0125d1c>] put_files_struct+0x6c/0xe0 [<c0127160>] do_exit+0x400/0xaa0 [<c025d537>] read_chan+0x327/0x1010 [<c0127a3b>] do_group_exit+0x1cb/0x210 [<c01302d5>] dequeue_signal+0x35/0xa0 [<c013308e>] get_signal_to_deliver+0x40e/0x920 [<c010a08d>] do_signal+0xdd/0x110 [<c02563bd>] tty_read+0x24d/0x2d0 [<c025e220>] write_chan+0x0/0x240 [<c016c891>] vfs_read+0xe1/0x1c0 [<c010a119>] do_notify_resume+0x59/0x5c [<c010a2f6>] work_notifysig+0x13/0x15
sshd S C014A681 4217195792 8387 1 8524 8211 (NOTLB)
Call Trace: [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c014a8ea>] __get_free_pages+0x1a/0x50 [<c01864f1>] __pollwait+0x41/0xd0 [<c036ddb3>] tcp_poll+0x33/0x190 [<c0349f49>] sock_poll+0x29/0x40 [<c0186793>] do_select+0x123/0x250 [<c01864b0>] __pollwait+0x0/0xd0 [<c0186c16>] sys_select+0x326/0x560 [<c0109128>] restore_sigcontext+0x128/0x140 [<c010a2ab>] syscall_call+0x7/0xb
bash D C04CDC68 4233453816 8524 1 8387 (L-TLB)
Call Trace: [<c010cd75>] do_IRQ+0x235/0x370 [<c01394a5>] flush_workqueue+0x305/0x450 [<c010ac18>] common_interrupt+0x18/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0257a44>] release_dev+0x6a4/0x860 [<c01566ab>] zap_pmd_range+0x4b/0x70 [<c0258204>] tty_release+0x94/0x1b0 [<c016dd7c>] __fput+0xac/0x100 [<c0258170>] tty_release+0x0/0x1b0 [<c016ddcb>] __fput+0xfb/0x100 [<c016bcbc>] filp_close+0x15c/0x230 [<c0125d1c>] put_files_struct+0x6c/0xe0 [<c0127160>] do_exit+0x400/0xaa0 [<c0130228>] __dequeue_signal+0x108/0x180 [<c0127a3b>] do_group_exit+0x1cb/0x210 [<c013032a>] dequeue_signal+0x8a/0xa0 [<c013308e>] get_signal_to_deliver+0x40e/0x920 [<c010a08d>] do_signal+0xdd/0x110 [<c01356c3>] sys_rt_sigaction+0xe3/0x110 [<c01092f6>] sys_sigreturn+0x1b6/0x2a0 [<c010a119>] do_notify_resume+0x59/0x5c [<c010a2f6>] work_notifysig+0x13/0x15
httpd S 00000000 1100 8595 19660 8597 19670 (NOTLB)
Call Trace: [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c0146335>] unlock_page+0x15/0x60 [<c035b586>] qdisc_restart+0x16/0x550 [<c03522c9>] dev_queue_xmit+0x609/0x730 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0368a0f>] ip_output+0x10f/0x220 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c0193912>] iput+0x62/0x90 [<c017e2f0>] pipe_read+0x60/0x270 [<c016dd7c>] __fput+0xac/0x100 [<c0349fa0>] sock_close+0x0/0x50 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016cc37>] sys_read+0x47/0x60 [<c010a2ab>] syscall_call+0x7/0xb
httpd S C041F8FC 10923632 8597 19660 8598 8595 (NOTLB)
Call Trace: [<c014a424>] buffered_rmqueue+0xc4/0x280 [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c014e3b4>] cache_init_objs+0x144/0x160 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c018be80>] dput+0x30/0x660 [<c016dd7c>] __fput+0xac/0x100 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c01b7e70>] ext3_release_file+0x0/0x70 [<c016dd88>] __fput+0xb8/0x100 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016bdf2>] sys_close+0x62/0xa0 [<c010a2ab>] syscall_call+0x7/0xb
httpd S 00000000 4281504624 8598 19660 8601 8597 (NOTLB)
Call Trace: [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c0146335>] unlock_page+0x15/0x60 [<c035b586>] qdisc_restart+0x16/0x550 [<c03522c9>] dev_queue_xmit+0x609/0x730 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0368a0f>] ip_output+0x10f/0x220 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c0193912>] iput+0x62/0x90 [<c017e2f0>] pipe_read+0x60/0x270 [<c016dd7c>] __fput+0xac/0x100 [<c0349fa0>] sock_close+0x0/0x50 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016cc37>] sys_read+0x47/0x60 [<c010a2ab>] syscall_call+0x7/0xb
httpd S C041F8FC 4278766908 8601 19660 8602 8598 (NOTLB)
Call Trace: [<c014a424>] buffered_rmqueue+0xc4/0x280 [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c014e3b4>] cache_init_objs+0x144/0x160 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c0193912>] iput+0x62/0x90 [<c017e2f0>] pipe_read+0x60/0x270 [<c016dd7c>] __fput+0xac/0x100 [<c0349fa0>] sock_close+0x0/0x50 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016cc37>] sys_read+0x47/0x60 [<c010a2ab>] syscall_call+0x7/0xb
httpd S 00000000 36396268 8602 19660 8621 8601 (NOTLB)
Call Trace: [<c035b586>] qdisc_restart+0x16/0x550 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c034df33>] kfree_skbmem+0x13/0x70 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0372a14>] tcp_recvmsg+0x674/0xee0 [<c0368a0f>] ip_output+0x10f/0x220 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d2c>] tcp_close+0x6ec/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c0193912>] iput+0x62/0x90 [<c017e2f0>] pipe_read+0x60/0x270 [<c016dd7c>] __fput+0xac/0x100 [<c0349fa0>] sock_close+0x0/0x50 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016cc37>] sys_read+0x47/0x60 [<c010a2ab>] syscall_call+0x7/0xb
httpd S C041F8FC 4270070224 8621 19660 8622 8602 (NOTLB)
Call Trace: [<c014a424>] buffered_rmqueue+0xc4/0x280 [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c014a681>] __alloc_pages+0xa1/0x2f0 [<c014e3b4>] cache_init_objs+0x144/0x160 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c0193912>] iput+0x62/0x90 [<c017e2f0>] pipe_read+0x60/0x270 [<c016dd7c>] __fput+0xac/0x100 [<c0349fa0>] sock_close+0x0/0x50 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016cc37>] sys_read+0x47/0x60 [<c010a2ab>] syscall_call+0x7/0xb
httpd S 00000000 11785072 8622 19660 8621 (NOTLB)
Call Trace: [<c029a0d0>] e100_xmit_frame+0x1a0/0x240 [<c012eea5>] schedule_timeout+0xe5/0xf0 [<c0146335>] unlock_page+0x15/0x60 [<c035b586>] qdisc_restart+0x16/0x550 [<c03522c9>] dev_queue_xmit+0x609/0x730 [<c0374a23>] wait_for_connect+0x383/0x450 [<c0368a0f>] ip_output+0x10f/0x220 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0120cf0>] autoremove_wake_function+0x0/0x50 [<c0348fb8>] sock_alloc_inode+0x18/0xc0 [<c0374dc9>] tcp_accept+0x2d9/0x570 [<c0190b90>] alloc_inode+0xd0/0x180 [<c0397da5>] inet_accept+0x35/0x3d0 [<c0373d46>] tcp_close+0x706/0xcc0 [<c03492ac>] sockfd_lookup+0x1c/0x80 [<c034aaf4>] sys_accept+0xa4/0x180 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c034907b>] sock_destroy_inode+0x1b/0x20 [<c0190c80>] destroy_inode+0x40/0x60 [<c0193912>] iput+0x62/0x90 [<c017e2f0>] pipe_read+0x60/0x270 [<c016dd7c>] __fput+0xac/0x100 [<c0349fa0>] sock_close+0x0/0x50 [<c034b88d>] sys_socketcall+0xed/0x2a0 [<c016cc37>] sys_read+0x47/0x60 [<c010a2ab>] syscall_call+0x7/0xb
-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Sunday, April 06, 2003 4:38 PM
To: Shawn Starr
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [BUG][2.5.66bk9+] - changes to timers still broken - we don't oops anymore
Quoted 11 lines
What is "it"?
I receive rather a lot of email and am dependent on people helping me out a bit with context. I have lost the plot on this one.
I don't know why Here's what strace reports:
Sshd is stuck in 'D' and a child in zombie state. The machine has been up for 2 days 18 hours 50 mins.
a sysrq-T trace here would help.
Re: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
Quoted 12 lines · Shawn Starr
login D C7250C94 4191040372 115 1 116 113 (L-TLB)
Call Trace: [<c01394a5>] flush_workqueue+0x305/0x450 [<c011de30>] default_wake_function+0x0/0x20 [<c011de30>] default_wake_function+0x0/0x20 [<c0257a44>] release_dev+0x6a4/0x860 [<c010cd75>] do_IRQ+0x235/0x370 [<c0258204>] tty_release+0x94/0x1b0 [<c016dd7c>] __fput+0xac/0x100 [<c0258170>] tty_release+0x0/0x1b0 [<c016ddcb>] __fput+0xfb/0x100
Well it does look like you've hit the flush_workqueue livelock.
Can you run with this (rather hastily tested) patch?
kernel/workqueue.c | 80 ++++++++++++++++++++++++++--------------------------- 1 files changed, 40 insertions(+), 40 deletions(-)
kernel/workqueue.c renamed from kernel/workqueue.c~flush_workqueue-hang-fix +40 -40
@@ -27,13 +27,20 @@#include <linux/slab.h> /* -* The per-CPU workqueue: +* The per-CPU workqueue. +* +* The sequnece counters are for flush_scheduled_work(). It wants to wait +* until until all currently-scheduled works are completed, but it doesn't +* want to be livelocked by new, incoming ones. So it waits until +* remove_sequence is >= the initial insert_sequence */ struct cpu_workqueue_struct {spinlock_t lock; -atomic_t nr_queued; +long remove_sequence;/* Least-recently added (next to run) */ +long insert_sequence;/* Next to add */ +struct list_head worklist;wait_queue_head_t more_work;wait_queue_head_t work_done;
@@ -71,10 +78,9 @@ int queue_work(struct workqueue_struct *spin_lock_irqsave (&cwq->lock, flags);list_add_tail (&work->entry, &cwq->worklist); -atomic_inc (&cwq->nr_queued); -spin_unlock_irqrestore (&cwq->lock, flags); - + cwq->insert_sequence++;wake_up (&cwq->more_work); +spin_unlock_irqrestore (&cwq->lock, flags); ret =1 ; }put_cpu ();
@@ -93,11 +99,13 @@ static void delayed_work_timer_fn(unsign*/ spin_lock_irqsave (&cwq->lock, flags);list_add_tail (&work->entry, &cwq->worklist); + cwq->insert_sequence++;wake_up (&cwq->more_work);spin_unlock_irqrestore (&cwq->lock, flags); } -int queue_delayed_work (struct workqueue_struct *wq,struct work_struct *work,unsigned long delay) +int queue_delayed_work (struct workqueue_struct *wq, +struct work_struct *work,unsigned long delay) {int ret =0 , cpu =get_cpu ();struct timer_list *timer = &work->timer;
@@ -107,18 +115,11 @@ int queue_delayed_work(struct workqueue_BUG_ON (timer_pending (timer));BUG_ON (!list_empty (&work->entry)); -/* -* Increase nr_queued so that the flush function -* knows that there's something pending. -*/ -atomic_inc (&cwq->nr_queued); work->wq_data = cwq; - timer->expires = jiffies + delay; timer->data = (unsigned long )work; timer->function = delayed_work_timer_fn;add_timer (timer); - ret =1 ; }put_cpu ();
@@ -135,7 +136,8 @@ static inline void run_workqueue(struct*/ spin_lock_irqsave (&cwq->lock, flags);while (!list_empty (&cwq->worklist)) { - struct work_struct *work = list_entry(cwq->worklist.next, struct work_struct, entry); +struct work_struct *work =list_entry (cwq->worklist.next, + struct work_struct, entry);void (*f) (void *) = work->func;void *data = work->data;
@@ -146,14 +148,9 @@ static inline void run_workqueue(structclear_bit (0 , &work->pending);f (data); -/* -* We only wake up 'work done' waiters (flush) when -* the last function has been fully processed. -*/ -if (atomic_dec_and_test (&cwq->nr_queued)) -wake_up (&cwq->work_done); -spin_lock_irqsave (&cwq->lock, flags); + cwq->remove_sequence++; +wake_up (&cwq->work_done); }spin_unlock_irqrestore (&cwq->lock, flags); }
@@ -223,8 +220,13 @@ static int worker_thread(void *__startup* Forces execution of the workqueue and blocks until its completion. * This is typically used in driver shutdown handlers. * -* NOTE: if work is being added to the queue constantly by some other -* context then this function might block indefinitely. +* This function will sample each workqueue's current insert_sequence number and +* will sleep until the head sequence is greater than or equal to that. This +* means that we sleep until all works which were queued on entry have been +* handled, but we are not livelocked by new incoming ones. +* +* This function used to run the workqueues itself. Now we just wait for the +* helper threads to do it. */ void flush_workqueue (struct workqueue_struct *wq) {
@@ -234,28 +236,25 @@ void flush_workqueue(struct workqueue_stmight_sleep ();for (cpu =0 ; cpu <NR_CPUS ; cpu++) { +DEFINE_WAIT (wait); +long sequence_needed; +if (!cpu_online (cpu))continue ; cwq = wq->cpu_wq + cpu; -if (atomic_read (&cwq->nr_queued)) { -DECLARE_WAITQUEUE (wait, current); +spin_lock_irq (&cwq->lock); + sequence_needed = cwq->insert_sequence; -if (!list_empty (&cwq->worklist)) -run_workqueue (cwq); - -/* -* Wait for helper thread(s) to finish up -* the queue: -*/ -set_current_state (TASK_UNINTERRUPTIBLE ); -add_wait_queue (&cwq->work_done, &wait); -if (atomic_read (&cwq->nr_queued)) -schedule (); -else -set_task_state (current,TASK_RUNNING ); -remove_wait_queue (&cwq->work_done, &wait); +while (sequence_needed - cwq->remove_sequence >0 ) { +prepare_to_wait (&cwq->work_done, &wait, +TASK_UNINTERRUPTIBLE ); +spin_unlock_irq (&cwq->lock); +schedule (); +spin_lock_irq (&cwq->lock); } +finish_wait (&cwq->work_done, &wait); +spin_unlock_irq (&cwq->lock); } }
@@ -281,7 +280,8 @@ struct workqueue_struct *create_workqueuspin_lock_init (&cwq->lock); cwq->wq = wq; cwq->thread =NULL ; -atomic_set (&cwq->nr_queued,0 ); + cwq->insert_sequence =0 ; + cwq->remove_sequence =0 ;INIT_LIST_HEAD (&cwq->worklist);init_waitqueue_head (&cwq->more_work);init_waitqueue_head (&cwq->work_done);
_
Re: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
Andrew> Well it does look like you've hit the flush_workqueue Andrew> livelock.
I don't think it's really livelock. I think it's just the fact that his kernel (with your tty-shutdown-race-fix patch) does del_timer_sync() without decrementing nr_queued() and so flush_workqueue() never returns.
Still, I like the idea of this patch, since it resolves the livelock. But I don't think the implementation is quite right. insert_sequence doesn't get incremented until delayed_work_timer_fn(). That means that a driver (tty_io.c, for example) could call schedule_delayed_work(), then call flush_scheduled_work() before delayed_work_timer_fn() has run for that work.
In that case schedule_delayed_work() could return immediately because insert_sequence and remove_sequence are (probably) equal. Then delayed_work_timer_fn() runs after the driver exits, and we're back with the original problem (running a freed timer).
It should be pretty easy to rejigger the patch so that it works correctly, just by moving the cwq->insert_sequence++ from delayed_work_timer_fn() into queue_delayed_work() (right before the add_timer(), say). I'm still not positive that this covers everything; I need to think a little harder.
In any case, I think we still have to do something to fix release_dev() in tty_io.c. It seems we should at least add the clear_bit(TTY_DONT_FLIP, &tty->flags); however, I'm not familiar enough with how the tty driver works to know whether TTY_DONT_FLIP could get set again (while we're waiting for flush_scheduled_work()). If so we would also need something along the lines of cancel_delayed_work(&tty->flip.work).
Shawn, I think the patch I just posted a little while ago (with a fixed cancel_delayed_work() implementation) is more likely to cure your tty hanging right now. However, I think something along the lines of this patch from Andrew is a better solution in the long run.
- Roland
Re: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
Quoted 7 lines · Roland Dreier
Still, I like the idea of this patch, since it resolves the livelock. But I don't think the implementation is quite right.
insert_sequencedoesn't get incremented untildelayed_work_timer_fn(). That means that a driver (tty_io.c, for example) could callschedule_delayed_work(), then callflush_scheduled_work()beforedelayed_work_timer_fn()has run for that work.
The driver needs to run cancel_delayed_work() before calling flush_scheduled_work(). The tty patch is already doing that, and I think that plugs the holes.
Here's a full changelog.
The workqueue code currently has a notion of a per-cpu queue being "busy". flush_scheduled_work()'s responsibility is to wait for a queue to be not busy.
Problem is, flush_scheduled_work() can easily hang up.
- The workqueue is deemed "busy" when there are pending (timer-based) works. But if someone repeatedly schedules new work in the delayed callback, the queue will never fall idle, and flush_scheduled_work() will not complete.
- If someone reschedules work (not delayed work) in the work function, that too will cause the queue to never go idle, and flush_scheduled_work() will not terminate.
So what this patch does is:
- Create a new "cancel_delayed_work()" which will try to kill off any timer-based works.
- Change flush_scheduled_work() so that it is immune to people re-adding work in the work callout handler.
We can do this by recognising that the caller does *not* want to wait until the workqueue is "empty". The caller merely wants to wait until all works which were pending at the time flush_scheduled_work() was called have completed.
The patch uses a couple of sequence numbers for that.
So now, if someone wants to reliably remove delayed work they should do:
/* * Make sure that my work-callback will no longer schedule new work */ my_driver_is_shutting_down = 1; /* * Kill off any pending delayed work */ cancel_delayed_work(&my_work); /* * OK, there will be no new works scheduled. But there may be one * currently queued or in progress. So wait for that to complete. */ flush_scheduled_work();
And change the flush_workqueue() sleep to be uninterruptible. We worry that delivery of a signal may cause the wait to return too early.
Re: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
In reply to Shawn StarrI am completely convinced that the patch you are using (which I believe is the same as the one Andrew calls "tty-shutdown-race-fix.patch") is the problem. What happens is that release_dev() in tty_io.c calls cancel_delayed_work(), which calls del_timer_sync() without decrementing nr_queued for keventd_wq.
When flush_scheduled_work() gets called it sleeps on the work_done waitqueue. The only place work_done gets woken up is in run_workqueue, and it only happens if atomic_dec_and_test(&cwq->nr_queued) decrements nr_queued to 0. But after calling cancel_delayed_work(), that can never happen (we deleted the timer that was going to add the work that we're waiting for).
It seems to me that the implementation of cancel_delayed_work() is not quite right. We need to decrement nr_queued if we actually stopped the work from being added to the workqueue.
Andrew, I've never seen a reply from you about this, can you tell me if I'm missing something here?
By the way, I assume that the process below is the one that's hung:
bash D C04CDC68 4233453816 8524 1 8387 (L-TLB)
Call Trace:
[<c010cd75>] do_IRQ+0x235/0x370
[<c01394a5>] flush_workqueue+0x305/0x450
[<c010ac18>] common_interrupt+0x18/0x20
[<c011de30>] default_wake_function+0x0/0x20
[<c011de30>] default_wake_function+0x0/0x20
[<c0257a44>] release_dev+0x6a4/0x860
[<c01566ab>] zap_pmd_range+0x4b/0x70
[<c0258204>] tty_release+0x94/0x1b0
[<c016dd7c>] __fput+0xac/0x100
[<c0258170>] tty_release+0x0/0x1b0It would seem to be stuck in the flush_workqueue() called from release_dev(), just as I would expect.
Shawn, can you try the patch below instead of Andrew's ttyfix2?
- Roland
===== drivers/char/tty_io.c 1.72 vs edited =====
drivers/char/tty_io.c +8 -1
@@ -1286,8 +1286,15 @@ }/* -* Make sure that the tty's task queue isn't activated. +* Prevent flush_to_ldisc() from rescheduling the work for later. Then +* kill any delayed work. */ +clear_bit (TTY_DONT_FLIP , &tty->flags); +cancel_delayed_work (&tty->flip.work); + +/* +* Wait for ->hangup_work and ->flip.work handlers to terminate +*/ flush_scheduled_work ();/*
===== include/linux/workqueue.h 1.4 vs edited =====
include/linux/workqueue.h +7 -0
@@ -63,5 +63,12 @@extern void init_workqueues (void ); +/* +* Kill off a pending schedule_delayed_work(). Note that the work callback +* function may still be running on return from cancel_delayed_work(). Run +* flush_scheduled_work() to wait on it. +*/ +extern int cancel_delayed_work (struct work_struct *work); +#endif
===== kernel/workqueue.c 1.6 vs edited =====
kernel/workqueue.c +19 -1
@@ -125,6 +125,24 @@return ret; } +int cancel_delayed_work (struct work_struct *work) { +struct cpu_workqueue_struct *cwq = work->wq_data; +int ret; + + ret =del_timer_sync (&work->timer); +if (ret) { +/* +* Wake up 'work done' waiters (flush) if we just +* removed the last thing on the workqueue. +*/ +if (atomic_dec_and_test (&cwq->nr_queued)) +wake_up (&cwq->work_done); + + } + +return ret; +} +static inline void run_workqueue (struct cpu_workqueue_struct *cwq) {unsigned long flags;
@@ -378,5 +396,5 @@EXPORT_SYMBOL (schedule_work);EXPORT_SYMBOL (schedule_delayed_work); +EXPORT_SYMBOL (cancel_delayed_work);EXPORT_SYMBOL (flush_scheduled_work); -
Re: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
Quoted 4 lines · Roland Dreier
Andrew, I've never seen a reply from you about this, can you tell me if I'm missing something here?
No, I agree. I don't think pending delayed work should contribute to the count at all.
If someone wants to synchronise with the workqueue system they should cancel any delayed work which they own (via cancel_scheduled_work) and then wait on any currently-queued works via flush_scheduled_work().
So flush_scheduled_work() only needs to care about currently-queued works, not the ones which are pending a timer event.
And flush_scheduled_work() needs to be taught to not lock up if someone keeps re-adding work.
That's what my patch did (I think; it was a quicky)
Re: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
Andrew> No, I agree. I don't think pending delayed work should Andrew> contribute to the count at all.
Andrew> If someone wants to synchronise with the workqueue system Andrew> they should cancel any delayed work which they own (via Andrew> cancel_scheduled_work) and then wait on any Andrew> currently-queued works via flush_scheduled_work().
Andrew> So flush_scheduled_work() only needs to care about Andrew> currently-queued works, not the ones which are pending a Andrew> timer event.
Andrew> And flush_scheduled_work() needs to be taught to not lock Andrew> up if someone keeps re-adding work.
Ah, I see... your patch that added insert_sequence and remove_sequence was intended to apply on top of the patch that adds cancel_delayed_work().
Please ignore the reply to your patch that I just sent, I misunderstood what you were trying to do.
Shawn, I think if you add Andrew's most recent patch on top of what you were running with, your problem should probably be fixed.
Sorry for the extra noise.
- Roland
RE: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
Applying akpm's patch now, expect a result in 1-2 days since that's how long it takes to begin to destabilize :-)
Shawn.
-----Original Message-----
From: Roland Dreier [mailto:[email protected]]
Sent: Wednesday, April 09, 2003 12:52 AM
To: Andrew Morton
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [BUG][2.5.66bk9+] - tty hangings - patches, dmesg & sysctl+T info
Importance: High
Andrew> No, I agree. I don't think pending delayed work should Andrew> contribute to the count at all.
Andrew> If someone wants to synchronise with the workqueue system Andrew> they should cancel any delayed work which they own (via Andrew> cancel_scheduled_work) and then wait on any Andrew> currently-queued works via flush_scheduled_work().
Andrew> So flush_scheduled_work() only needs to care about Andrew> currently-queued works, not the ones which are pending a Andrew> timer event.
Andrew> And flush_scheduled_work() needs to be taught to not lock Andrew> up if someone keeps re-adding work.
Ah, I see... your patch that added insert_sequence and remove_sequence was intended to apply on top of the patch that adds cancel_delayed_work().
Please ignore the reply to your patch that I just sent, I misunderstood what you were trying to do.
Shawn, I think if you add Andrew's most recent patch on top of what you were running with, your problem should probably be fixed.
Sorry for the extra noise.
- Roland