2.6.18-rc3-mm2

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc3/2.6.18-rc3-mm2/

- 2.6.18-rc3-mm1 gets mysterious udev timeouts during boot and crashes in NFS. This kernel reverts the patches which were causing that.

Changes since 2.6.18-rc3-mm1:

+revert-x86_64-mm-i386-remove-lock-section.patch

Revert patch which caues udev timeouts.

-knfsd-make-rpc-threads-pools-numa-aware-fix.patch

Folded into knfsd-make-rpc-threads-pools-numa-aware.patch

+revert-knfsd-make-rpc-threads-pools-numa-aware.patch

Revert patch which causes nfs crashes.

All 1136 patches:

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc3/2.6.18-rc3-mm2/patch-list

Hi,

I get this error during the build.

kernel/built-in.o: In function `bacct_add_tsk':
/usr/src/linux-mm/kernel/tsacct.c:39: undefined reference to `__divdi3' make[1]: *** [.tmp_vmlinux1] Error 1
make: *** [_all] Error 2

I'll try with CONFIG_TASKSTATS disabled.

Regards,
Michal

Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

Sounds likes we are trying to do a 64 bit division since timespec_to_ns() returns a 64 bit value.

Here's a compile tested patch to fix the problem

---

 kernel/tsacct.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletion(-)

diff -puN kernel/tsacct.c~tsacct-build-fix kernel/tsacct.c
@@ -36,7 +36,8 @@ void bacct_add_tsk(struct taskstats *sta
 	do_posix_clock_monotonic_gettime(&uptime);
 	ts = timespec_sub(uptime, current->group_leader->start_time);
 	/* rebase elapsed time to usec */
-	stats->ac_etime = (timespec_to_ns(&ts))/NSEC_PER_USEC;
+	stats->ac_etime = (ts.tv_sec * USEC_PER_SEC) +
+				(ts.tv_nsec / NSEC_PER_USEC);
 	stats->ac_btime = xtime.tv_sec - ts.tv_sec;
 	if (thread_group_leader(tsk)) {
 		stats->ac_exitcode = tsk->exit_code;

_

	Regards,
	Balbir Singh,
	Linux Technology Center,
	IBM Software Labs

Hi,

It doesn't apply
cat patches/tsacct1.patch | patch -p1 --dry-run
patching file kernel/tsacct.c
Hunk #1 FAILED at 36.
1 out of 1 hunk FAILED -- saving rejects to file kernel/tsacct.c.rej

Andrew's csa-basic-accounting-over-taskstats-fix.patch fix compilation problem.

Regards,
Michal

Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)

Yeah, thats it! I did not see the fix in mm-commits.

Thanks for pointing to the fix.

	Balbir Singh,
	Linux Technology Center,
	IBM Software Labs

there's something more, I had a load of the following while playing with UML, full dmesg and config are
http://oioio.altervista.org/linux/config-2.6.18-rc3-mm2-1
http://oioio.altervista.org/linux/dmesg-2.6.18-rc3-mm2-1

[  781.988000] ------------[ cut here ]------------
[  781.988000] kernel BUG at mm/vmscan.c:383!
[  781.988000] invalid opcode: 0000 [#1]
[  781.988000] 4K_STACKS PREEMPT 
[  781.988000] last sysfs file: /devices/system/cpu/cpu0/cpufreq/ondemand/ignore_nice_load
[  781.988000] Modules linked in: ipv6 nfsd exportfs lockd sunrpc ipt_MASQUERADE iptable_nat ip_nat xt_tcpudp xt_state ip_conntrack iptable_filter ip_tables x_tables jfs aes dm_crypt dm_mod rtc sony_acpi tun psmouse sonypi speedstep_ich speedstep_lib freq_table cpufreq_conservative cpufreq_ondemand cpufreq_powersave sd_mod usb_storage scsi_mod usbhid pcmcia snd_intel8x0 snd_ac97_codec snd_ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_timer intel_agp agpgart i2c_i801 uhci_hcd usbcore evdev e100 mii yenta_socket rsrc_nonstatic pcmcia_core snd soundcore snd_page_alloc pcspkr
[  781.988000] CPU:    0
[  781.988000] EIP:    0060:[<c014c4d8>]    Not tainted VLI
[  781.988000] EFLAGS: 00210203   (2.6.18-rc3-mm2-1 #1) 
[  781.988000] EIP is at remove_mapping+0xe8/0x120
[  781.988000] eax: c0374120   ebx: c11e2a80   ecx: c0374120   edx: 000000d0
[  781.988000] esi: c0374120   edi: cfea0f78   ebp: cfea0e04   esp: cfea0df8
[  781.988000] ds: 007b   es: 007b   ss: 0068
[  781.988000] Process kswapd0 (pid: 134, ti=cfea0000 task=cfe9e030 task.ti=cfea0000)
[  781.988000] Stack: c11e2a80 c11e2a80 c0374120 cfea0f14 c014cbab c0374120 c11e2a80 cfea0f78 
[  781.988000]        c0373d60 c0373e2c 00000020 00000020 00000000 00000020 00000000 00000000 
[  781.988000]        c0374120 00000001 00000000 c101a860 c0373c20 00000000 00000001 c0463168 
[  781.988000] Call Trace:
[  781.988000]  [<c014cbab>] shrink_inactive_list+0x69b/0x920
[  781.988000]  [<c014cec2>] shrink_zone+0x92/0xe0
[  781.988000]  [<c014d1f1>] kswapd+0x2e1/0x430
[  781.988000]  [<c012ee26>] kthread+0xe6/0xf0
[  781.988000]  [<c0101005>] kernel_thread_helper+0x5/0x10
[  781.988000] DWARF2 unwinder stuck at kernel_thread_helper+0x5/0x10
[  781.988000] Leftover inexact backtrace:
[  781.988000]  [<c0103a06>] show_stack_log_lvl+0xb6/0x100
[  781.988000]  [<c0103c2f>] show_registers+0x1df/0x290
[  781.988000]  [<c01041aa>] die+0x13a/0x310
[  781.988000]  [<c01047dd>] do_trap+0x9d/0x100
[  781.988000]  [<c0104c41>] do_invalid_op+0xa1/0xb0
[  781.988000]  [<c031a4a9>] error_code+0x39/0x40
[  781.988000]  [<c014cbab>] shrink_inactive_list+0x69b/0x920
[  781.988000]  [<c014cec2>] shrink_zone+0x92/0xe0
[  781.988000]  [<c014d1f1>] kswapd+0x2e1/0x430
[  781.988000]  [<c012ee26>] kthread+0xe6/0xf0
[  781.988000]  [<c0101005>] kernel_thread_helper+0x5/0x10
[  781.988000] Code: 89 e0 25 00 f0 ff ff ff 48 14 8b 40 08 31 d2 a8 08 74 bc e8 6b be 1c 00 31 d2 eb b3 8d b4 26 00 00 00 00 8b 53 0c e9 51 ff ff ff <0f> 0b 7f 01 4e 66 33 c0 e9 2c ff ff ff 0f 0b 7e 01 4e 66 33 c0 
[  781.988000] EIP: [<c014c4d8>] remove_mapping+0xe8/0x120 SS:ESP 0068:cfea0df8
[  781.988000]  <0>------------[ cut here ]------------
[  782.292000] kernel BUG at mm/vmscan.c:383!
...
[  782.292000]  <0>------------[ cut here ]------------
[  782.564000] kernel BUG at mm/vmscan.c:383!
...
[  809.588000] ------------[ cut here ]------------
[  809.588000] kernel BUG at mm/vmscan.c:383!
...
[  809.588000]  <0>------------[ cut here ]------------
[  811.748000] kernel BUG at mm/vmscan.c:383!
...
[  811.748000]  <0>------------[ cut here ]------------
[  814.128000] kernel BUG at mm/vmscan.c:383!
...
[  814.128000]  <0>------------[ cut here ]------------
[  815.272000] kernel BUG at mm/vmscan.c:383!
...
[  815.272000]  <0>------------[ cut here ]------------
[  816.116000] kernel BUG at mm/vmscan.c:383!
...
[  816.856000]  <0>------------[ cut here ]------------
[  817.120000] kernel BUG at mm/vmscan.c:383!

mattia
:wq!

You are so right: the minor fix below is needed.

Sorry, someone else will have to help with all that nuisance.

remove_mapping() must check against page_mapping(page):
&swapper_space is implicit, never actually stored in page->mapping.

@@ -380,7 +380,7 @@ static pageout_t pageout(struct page *pa
 int remove_mapping(struct address_space *mapping, struct page *page)
 {
 	BUG_ON(!PageLocked(page));
-	BUG_ON(mapping != page->mapping);
+	BUG_ON(mapping != page_mapping(page));
 
 	write_lock_irq(&mapping->tree_lock);
 

Thanks now it runs ok (since ~30 minutes now).
Hot-fix? :)

mattia
:wq!

hum.

-tycho kernel: ata1.00: configured for UDMA/33
+tycho kernel: ata1.00: configured for UDMA/100

That looks nice.

-tycho kernel: ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x18C8 irq 15 -tycho kernel: scsi1 : ata_piix
-tycho kernel: ata2: port disabled. ignoring.
-tycho kernel: ATA: abnormal status 0xFF on port 0x177

So does that.

-tycho kernel: input: PS/2 Mouse as /class/input/input1 -tycho kernel: input: AlpsPS/2 ALPS GlidePoint as /class/input/input2

That's not so good.

Dmitry, do you have anything in there which might have caused that?

Perhaps hdaps-handle-errors-from-input_register_device.patch is triggering for some reason. Fabio, it'd be useful if you could add this, see if it triggers:

@@ -1007,6 +1007,10 @@ int input_register_device(struct input_d
  fail3:	sysfs_remove_group(&dev->cdev.kobj, &input_dev_id_attr_group);
  fail2:	sysfs_remove_group(&dev->cdev.kobj, &input_dev_attr_group);
  fail1:	class_device_del(&dev->cdev);
+	if (error) {
+		printk(KERN_ERR "%s failed: %d\n", __FUNCTION__, error);
+		dump_stack();
+	}
 	return error;
 }
 EXPORT_SYMBOL(input_register_device);

_

Hmm, I'd be more concerned with i8042-get-rid-of-polling-timer patch... Anyway, can I have dmesg from boot with i8042.debug=1, please? Make sure you have big log biffer.

Dmitry

Hi.

Bingo! Reverting remove-polling-timer-from-i8042-v2.patch did the trick. Now I'm running 2.6.18-rc3-mm2 + hot-fixes :-)

Still interested in dmesg with i8042.debug=1 ?

Ciao.
Fabio

Yes, _with_ the i8042 polling patch applied. Do you have PNP support enabled?

Dmitry

]--snip--[

I've got one for you (attached).

Greetings,
Rafael

Thnk you, I think I see what the problem is. Rafael, could you please try booting with i8042.nomux and tell me if mouse starts working.

Fabio, do you have a multiplexing controller as well?

Dmitry

Hi Dmitry.

Well, I don't even know what this means :-(
How do I know?

However, it's a HP laptop, model name Pavillion DV4378EA.

Fabio

Yep, you do have it:

Could you please try booting with i8042.nomux and tell me if it works?

Thanks!

Dmitry

Hi.

Yup, it works.

Ciao.
Fabio

Fabio, Rafael,

Could you please try applying the patch below on top of -rc3-mm2 and see if it works without needing i8042.nomux?

Thank you!

Dmitry

Signed-off-by: Dmitry Torokhov <[email protected]>
---

 drivers/input/serio/i8042.c |    2 +-
 1 files changed, 1 insertion(+), 1 deletion(-)

Index: work/drivers/input/serio/i8042.c
===================================================================
--- work.orig/drivers/input/serio/i8042.c
+++ work/drivers/input/serio/i8042.c
@@ -435,7 +435,7 @@ static int i8042_enable_mux_ports(void)
 		i8042_command(&param, I8042_CMD_AUX_ENABLE);
 	}
 
-	return 0;
+	return i8042_enable_aux_port();
 }
 
 /*

Yes, it does.

Thanks,
Rafael

Hi Dmitry.

Yes, it works for me too. However, Andrew put a revert patch for remove-polling-timer-from-i8042-v2.patch in his hot-fixes directory. So, which one should be considered the correct fix?

Ciao.
Fabio

Thank you for testing.

I'd rather have him replace reverting patch with this one. Removing polling timer is needed for tickless operation.

Dmitry

It's a touchpad, but I guess that doesn't make a difference?

Rafael

Hi.

Please find the compressed log attached. And no, I don't have PNP support enabled.
Hope this helps.

Fabio

My box's (Asus L5D, x86_64) keyboard doesn't work on this kernel at all, even if I boot with init=/bin/bash. On the 2.6.18-rc2-mm1 it worked.

Unfortunately I have no indication what can be wrong, no oopses, no error messages in dmesg, nothing.

Right now I'm doing a binary search for the offending patch.

Greetings,
Rafael

Thanks. I'd zoom in on hdaps-handle-errors-from-input_register_device.patch and git-input.patch.

None of these, but close: remove-polling-timer-from-i8042-v2.patch breaks things here. [FYI, the box is booted with "noapic", because the IRQ sharing doesn't work otherwise due to a BIOS issue, so it may be related.]

Attached is the dmesg output with i8042.debug=1 for Dmitry. It's from 2.6.18-rc3 with -mm2 partially applied (up to and including logips2pp-fix-mx300-button-layout.patch). I'll apply the rest tonight, after I find the patch that broke suspend for me.

BTW, I couldn't test -rc4, because I don't use git and there's no standalone version so far. I hope it will be available?

[Now, I have an emergency to handle, so I won't be reachable before tonight, I think.]

Greetings,
Rafael

Unfortunately this one is git-block.patch. I have no idea which part of it may break the suspend.

It hangs during suspend, right after the memory has been shrunk, when devices should be suspended. After pressing SysRq-P it shows it's spinning in the idle thread and then hangs hard.

Greetings,
Rafael

ow, that tree is pretty huge at present.

OK, thanks for doing that. I'll drop git-block until we can get it sorted.

I think I know what it is, hang on.

Jens Axboe

Can I please have dmesg with i8042.debug=1?

Dmitry

Btw, does 2.6.18-rc4 work?

Dmitry

Okay here, done some fdisk partition manipulation and didn't lose any filesystems or any other nasties. ;) Dual boot 'doze, so stuffing around with NTFS (ro) as well as NFS (rw).

Some odd looking IRQ reassignments (Via chipset), I've put up -rc3 -> -rc3-mm2 dmesg diff, as well as dmesg and config on
<http://bugsplatter.mine.nu/test/linux-2.6/sempro/> if anyone curious.

Grant.

I tried it and guess what :)... swsusp doesn't work :@.

This time I was able to dump process states with sysrq-t:
http://www.fi.muni.cz/~xslaby/sklad/ide2.gif

My guess is ide2/2.0 dies (hpt370 driver), since last thing kernel prints is suspending device 2.0

diff of dmesgs:
@@ -1,4 +1,4 @@
-Linux version 2.6.18-rc2-mm1 (ku@bellona) (gcc version 4.1.1 20060721 (Red Hat 4.1.1-13)) #155 SMP Tue Aug 1 01:17:45 CEST 2006
+Linux version 2.6.18-rc3-mm2 (ku@bellona) (gcc version 4.1.1 20060802 (Red Hat 4.1.1-14)) #157 SMP Sun Aug 6 19:38:53 CEST 2006
 BIOS-provided physical RAM map:
 sanitize start
 sanitize end

@@ -49,7 +49,7 @@
 Enabling APIC mode:  Flat.  Using 1 I/O APICs
 Using ACPI (MADT) for SMP configuration information
 Allocating PCI resources starting at 50000000 (gap: 40000000:bec00000)
-Detected 2736.278 MHz processor.
+Detected 2736.289 MHz processor.
 Built 1 zonelists.  Total pages: 262128
 Kernel command line: ro root=/dev/hda2 reboot=w vga=1 2
 mapped APIC to ffffd000 (fee00000)

@@ -57,14 +57,22 @@
 Enabling fast FPU save and restore... done.
 Enabling unmasked SIMD FPU exception support... done.
 Initializing CPU#0
-CPU 0 irqstacks, hard=c0505000 soft=c0502000
+CPU 0 irqstacks, hard=c0509000 soft=c0506000
 PID hash table entries: 4096 (order: 12, 16384 bytes)
 Console: colour VGA+ 80x50
 Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
 Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
-Memory: 1034488k/1048512k available (2514k kernel code, 13456k reserved, 1349k data, 200k init, 131008k highmem)
+Memory: 1034472k/1048512k available (2522k kernel code, 13472k reserved, 1353k data, 204k init, 131008k highmem)
+virtual kernel memory layout:
+    fixmap  : 0xfff90000 - 0xfffff000   ( 444 kB)
+    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
+    vmalloc : 0xf8800000 - 0xff7fe000   ( 111 MB)
+    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
+      .init : 0xc04ce000 - 0xc0501000   ( 204 kB)
+      .data : 0xc03768d2 - 0xc04c8ff8   (1353 kB)
+      .text : 0xc0100000 - 0xc03768d2   (2522 kB)
 Checking if this processor honours the WP bit even in supervisor mode... Ok.
-Calibrating delay using timer specific routine.. 5476.47 BogoMIPS (lpj=10952942)
+Calibrating delay using timer specific routine.. 5476.48 BogoMIPS (lpj=10952969)
 Mount-cache hash table entries: 512
 CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000 

00004400 00000000 00000000
CPU: After vendor identify, caps: bfebfbff 00000000 00000000 00000000 00004400
00000000 00000000
@@ -82,9 +90,9 @@
CPU0: Intel(R) Pentium(R) 4 CPU 2.60GHz stepping 09
SMP alternatives: switching to SMP code
Booting processor 1/1 eip 3000
-CPU 1 irqstacks, hard=c0506000 soft=c0503000
+CPU 1 irqstacks, hard=c050a000 soft=c0507000
Initializing CPU#1
-Calibrating delay using timer specific routine.. 5472.77 BogoMIPS (lpj=10945546) +Calibrating delay using timer specific routine.. 5472.79 BogoMIPS (lpj=10945581)
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000
00004400 00000000 00000000
CPU: After vendor identify, caps: bfebfbff 00000000 00000000 00000000 00004400
00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
@@ -96,15 +104,15 @@
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Pentium(R) 4 CPU 2.60GHz stepping 09
-Total of 2 processors activated (10949.24 BogoMIPS).
+Total of 2 processors activated (10949.27 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 2 CPUs: passed.
Brought up 2 CPUs
-migration_cost=111
+migration_cost=1
NET: Registered protocol family 16
ACPI: bus type pci registered
-PCI: PCI BIOS revision 2.10 entry at 0xfb670, last bus=2
+PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
@@ -189,7 +197,7 @@
ACPI: PCI Interrupt 0000:02:01.0[A] -> GSI 21 (level, low) -> IRQ 19
HPT370: chipset revision 3
HPT370: no clock data saved by BIOS
-HPT370: DPLL base: 48 MHz, f_CNT: 146, assuming 33 MHz PCI
+HPT370: DPLL base: 48 MHz, f_CNT: 148, assuming 33 MHz PCI
HPT370: using 33 MHz PCI clock
HPT370: 100% native mode on irq 19

     ide2: BM-DMA at 0x9000-0x9007, BIOS settings: hde:DMA, hdf:pio

@@ -243,7 +251,7 @@
usb usb1: new device found, idVendor=0000, idProduct=0000
usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: EHCI Host Controller
-usb usb1: Manufacturer: Linux 2.6.18-rc2-mm1 ehci_hcd
+usb usb1: Manufacturer: Linux 2.6.18-rc3-mm2 ehci_hcd
usb usb1: SerialNumber: 0000:00:1d.7
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
@@ -257,7 +265,7 @@
usb usb2: new device found, idVendor=0000, idProduct=0000
usb usb2: new device strings: Mfr=3, Product=2, SerialNumber=1 usb usb2: Product: UHCI Host Controller
-usb usb2: Manufacturer: Linux 2.6.18-rc2-mm1 uhci_hcd
+usb usb2: Manufacturer: Linux 2.6.18-rc3-mm2 uhci_hcd
usb usb2: SerialNumber: 0000:00:1d.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
@@ -270,7 +278,7 @@
usb usb3: new device found, idVendor=0000, idProduct=0000
usb usb3: new device strings: Mfr=3, Product=2, SerialNumber=1 usb usb3: Product: UHCI Host Controller
-usb usb3: Manufacturer: Linux 2.6.18-rc2-mm1 uhci_hcd
+usb usb3: Manufacturer: Linux 2.6.18-rc3-mm2 uhci_hcd
usb usb3: SerialNumber: 0000:00:1d.1
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
@@ -283,7 +291,7 @@
usb usb4: new device found, idVendor=0000, idProduct=0000
usb usb4: new device strings: Mfr=3, Product=2, SerialNumber=1 usb usb4: Product: UHCI Host Controller
-usb usb4: Manufacturer: Linux 2.6.18-rc2-mm1 uhci_hcd
+usb usb4: Manufacturer: Linux 2.6.18-rc3-mm2 uhci_hcd
usb usb4: SerialNumber: 0000:00:1d.2
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
@@ -296,7 +304,7 @@
usb usb5: new device found, idVendor=0000, idProduct=0000
usb usb5: new device strings: Mfr=3, Product=2, SerialNumber=1 usb usb5: Product: UHCI Host Controller
-usb usb5: Manufacturer: Linux 2.6.18-rc2-mm1 uhci_hcd
+usb usb5: Manufacturer: Linux 2.6.18-rc3-mm2 uhci_hcd
usb usb5: SerialNumber: 0000:00:1d.3
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
@@ -312,8 +320,8 @@
input: Wacom Graphire2 4x5 as /class/input/input0
usbcore: registered new interface driver wacom /l/latest/xxx/drivers/usb/input/wacom.c: v1.45:USB Wacom Graphire and Wacom
Intuos tablet driver
-serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
+serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
it87: Found IT8712F chip at 0x290, revision 5
md: raid0 personality registered for level 0
@@ -325,6 +333,7 @@
No soundcards found.
oprofile: using NMI interrupt.
ip_conntrack version 2.4 (8191 buckets, 65528 max) - 208 bytes per conntrack
+input: AT Translated Set 2 keyboard as /class/input/input1
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP bic registered
NET: Registered protocol family 1
@@ -336,13 +345,15 @@
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
+EXT3-fs: INFO: recovery required on readonly filesystem. +EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
+EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
-Freeing unused kernel memory: 200k freed
-input: AT Translated Set 2 keyboard as /class/input/input1 +Freeing unused kernel memory: 204k freed
ieee1394: Initialized config rom entry `ip1394'
-hdc: ATAPI 63X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33) +hdc: ATAPI CD-ROM drive, 0kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
hdd: ATAPI 40X DVD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33)
ACPI: PCI Interrupt 0000:02:05.0[A] -> GSI 21 (level, low) -> IRQ 19
@@ -387,3 +398,5 @@
EXT3 FS on md0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 506036k swap on /dev/hda3. Priority:-1 extents:1 across:506036k
+JBD: barrier-based sync failed on hda2 - disabling barriers +JBD: barrier-based sync failed on md0 - disabling barriers

regards,

<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Does it go away if you revert this? ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc3/2.6.18-rc3-mm2/broken-out/ide-reprogram-disk-pio-timings-on-resume.patch

That should only affect resume, not suspend, but it does mess around with ide power management. Is this maybe happening on the *second* suspend?

I found that git-block.patch broke the suspend for me. Still have no idea what's up with it.

Rafael

Can you apply this on top of -mm and see if that fixes it?

index d2339e9..db647a9 100644
@@ -390,7 +390,7 @@ void ide_end_drive_cmd (ide_drive_t *dri
 			args[5] = hwif->INB(IDE_HCYL_REG);
 			args[6] = hwif->INB(IDE_SELECT_REG);
 		}
-	} else if (rq->cmd_type & REQ_TYPE_ATA_TASKFILE) {
+	} else if (rq->cmd_type == REQ_TYPE_ATA_TASKFILE) {
 		ide_task_t *args = (ide_task_t *) rq->special;
 		if (rq->errors == 0)
 			rq->errors = !OK_STAT(stat,READY_STAT,BAD_STAT);

Jens Axboe

It doesn't solve the problem for me.

<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Ok, thanks for testing, I'll try and reproduce it here.

Jens Axboe

I suspect elevator changes. The wait_for_completion is not woken in ide-io by ll_rw_blk. But I don't understand block layer too much. Where the blk_end_sync_rq should be called from (why is not called at all)?

regards,

<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

The ide changes are far more likely, it's probably missing a completion.

It's called from ->end_io() in end_that_request_last().

Jens Axboe

Actually I think the commit f74bf2e6b415588e562fdcfdd454d587eb33cd46 (Remove ->waiting member from struct request) is wrong, because generic_ide_suspend() uses the end_of_io member of rq to pass the PM data to ide_do_drive_cmd() where the pointer gets overwritten by &wait (must_wait is "true", because action == ide_wait). Previously &wait was stored in rq->waiting and it didn't overwrite the PM data.

Haven't tested yet, though.

Greetings,
Rafael

Indeed, that looks broken now. That must be what is screwing it up. With the former patch applied, did cdrom detection still look funny to you?

I'll concoct a fix for that breakage.

Jens Axboe

Something like this.

index db647a9..38479a2 100644
@@ -141,7 +141,7 @@ enum {
 
 static void ide_complete_power_step(ide_drive_t *drive, struct request *rq, u8 stat, u8 error)
 {
-	struct request_pm_state *pm = rq->end_io_data;
+	struct request_pm_state *pm = rq->data;
 
 	if (drive->media != ide_disk)
 		return;

@@ -164,7 +164,7 @@ static void ide_complete_power_step(ide_
 
 static ide_startstop_t ide_start_power_step(ide_drive_t *drive, struct request *rq)
 {
-	struct request_pm_state *pm = rq->end_io_data;
+	struct request_pm_state *pm = rq->data;
 	ide_task_t *args = rq->special;
 
 	memset(args, 0, sizeof(*args));

@@ -421,7 +421,7 @@ void ide_end_drive_cmd (ide_drive_t *dri
 			}
 		}
 	} else if (blk_pm_request(rq)) {
-		struct request_pm_state *pm = rq->end_io_data;
+		struct request_pm_state *pm = rq->data;
 #ifdef DEBUG_PM
 		printk("%s: complete_power_step(step: %d, stat: %x, err: %x)\n",
 			drive->name, rq->pm->pm_step, stat, err);

@@ -933,7 +933,7 @@ #endif
 
 static void ide_check_pm_state(ide_drive_t *drive, struct request *rq)
 {
-	struct request_pm_state *pm = rq->end_io_data;
+	struct request_pm_state *pm = rq->data;
 
 	if (blk_pm_suspend_request(rq) &&
 	    pm->pm_step == ide_pm_state_start_suspend)

@@ -1018,7 +1018,7 @@ #endif
 		    rq->cmd_type == REQ_TYPE_ATA_TASKFILE)
 			return execute_drive_cmd(drive, rq);
 		else if (blk_pm_request(rq)) {
-			struct request_pm_state *pm = rq->end_io_data;
+			struct request_pm_state *pm = rq->data;
 #ifdef DEBUG_PM
 			printk("%s: start_power_step(step: %d)\n",
 				drive->name, rq->pm->pm_step);

index d7b4499..0fd1e1c 100644
@@ -1219,7 +1219,7 @@ static int generic_ide_suspend(struct de
 	memset(&args, 0, sizeof(args));
 	rq.cmd_type = REQ_TYPE_PM_SUSPEND;
 	rq.special = &args;
-	rq.end_io_data = &rqpm;
+	rq.data = &rqpm;
 	rqpm.pm_step = ide_pm_state_start_suspend;
 	rqpm.pm_state = state.event;
 

@@ -1238,7 +1238,7 @@ static int generic_ide_resume(struct dev
 	memset(&args, 0, sizeof(args));
 	rq.cmd_type = REQ_TYPE_PM_RESUME;
 	rq.special = &args;
-	rq.end_io_data = &rqpm;
+	rq.data = &rqpm;
 	rqpm.pm_step = ide_pm_state_start_resume;
 	rqpm.pm_state = PM_EVENT_ON;
 

Jens Axboe

Hm, I'm not sure what you mean ...

Looks good, I'll give it a try.

Rafael

-hdc: ATAPI 63X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33) +hdc: ATAPI CD-ROM drive, 0kB Cache, UDMA(33)

But perhaps that wasn't you?

Thanks!

Jens Axboe

No, that wasn't me. :-)

It fixes this particular issue for me, but your first patch (appended) is also needed to prevent the box from hanging later during the resume (when it tries to save the image).

Thanks,
Rafael

--

 drivers/ide/ide-io.c |    2 +-
 1 files changed, 1 insertion(+), 1 deletion(-)

Index: linux-2.6.18-rc3-mm2/drivers/ide/ide-io.c
===================================================================
@@ -402,7 +402,7 @@ void ide_end_drive_cmd (ide_drive_t *dri
 			args[5] = hwif->INB(IDE_HCYL_REG);
 			args[6] = hwif->INB(IDE_SELECT_REG);
 		}
-	} else if (rq->cmd_type & REQ_TYPE_ATA_TASKFILE) {
+	} else if (rq->cmd_type == REQ_TYPE_ATA_TASKFILE) {
 		ide_task_t *args = (ide_task_t *) rq->special;
 		if (rq->errors == 0)
 			rq->errors = !OK_STAT(stat,READY_STAT,BAD_STAT);

Yes certainly, that's a separate bug, sorry if I didn't make that clear. Both fixes are in the block repo now, so next -mm should work fine again.

Jens Axboe

It was me and it's OK.

And even this is OK.

I'm just curious, what
@@ -387,3 +398,5 @@
EXT3 FS on md0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 506036k swap on /dev/hda3. Priority:-1 extents:1 across:506036k
+JBD: barrier-based sync failed on hda2 - disabling barriers +JBD: barrier-based sync failed on md0 - disabling barriers

means. Another bug?

thanks,

<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Good.

Jens Axboe

No change.

Nope, the first one.

But hdc is not on the hpt3xx controller.

regards,

<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

It seems that the command line on x86_64 is being truncated during boot:

Bootdata ok (command line is root=/dev/sda1 ro profile=2 console=tty0 console=ttyS0,57600 autobench_args: root=/dev/sda1 ABAT:1154470592 profile=2) [...]
Kernel command line: root=/dev/sda1 ro profile=2 console=tty0 console=ttyS0,57600 autobench_args: root=/dev/sda1 ABAT:1154470592 profile=2 [...]

elm3b6:~# cat /proc/cmdline
root=/dev/sda1

This seems to be occuring around the parse_args area.

Will try and track it down.

-apw

in mm right?

Don't bother, it is likely "early-param" (the patch from hell). I'll investigate.

-Andi

Following up myself ...

Are you sure it's a regression? 2.6.17 does the same and we always had that 255 character limit (I tried to increase it once, but it broke some old lilo setups)

i386 should be the same btw.

-Andi

Its not being truncated at 255 characters, its being truncated at the first space. This is coming out of parse_args, which dumps '\0's into the command_line as it rips it apart. We now only have one copy of the command line (in x86_64) instead of two, so we now expose this trashed copy in /proc/cmdline.

-apw

I don't see this in my version; so it's likely fixed already. I did quite a lot of changes on this patch already.

Please test

ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/patches/early-param

-Andi

Easier said than done as the original version is unwilling to revert. Looking at the replacement patch it has the same fix I have been testing to restore the original dual buffer semantic. So I think it would fix the problem we're seeing here. I'll follow up to this email with the incremental patch I tested with 2.6.18-rc2-mm2.

-apw

x86_64 dirty fix to restore dual command line store

Ok, It seems that the patch below effectivly removes the second copy of the command line. This means that any modification to the 'working' command line (as returned from setup_arch) is incorrectly visible in userspace via /proc/cmdline.

	x86_64-mm-early-param.patch

This patch restores the second copy. Its probabally not the right way to fix this long term.

---

diff -upN reference/arch/x86_64/kernel/setup.c current/arch/x86_64/kernel/setup.c
@@ -378,7 +378,8 @@ void __init setup_arch(char **cmdline_p)
 	early_identify_cpu(&boot_cpu_data);
 
 	parse_early_param();
-	*cmdline_p = saved_command_line;
+	memcpy(command_line, saved_command_line, COMMAND_LINE_SIZE);
+	*cmdline_p = command_line;
 
 	finish_e820_parsing();
 

Sorry for the side question but why is setup_arch adding things back on the cmdline in the first place? What do you see in /proc/cmdline?

Thanks,
Keith

Sorry for the ping. I read some more lkml and the context for this patch was filled in.

Thanks,
Keith

Well I've narroed it down to the following patch from Andrew:

x86_64-mm-early-param.patch

Basically, that leads setup_arch to return saved_command_line as _the_ command_line. We then run parse_args() against it which assumes it may irrevocabaly change command_line. Previous to this patch saved_command_line and command_line were separate and this was not an issue.

It feels like we should be following the model in the newly added parse_early_parms() and taking a local copy of the command_line here.

-apw

Not me. My only contribution to that patch was to scrog the changelog ;) I'll be fixing that sometime.

I think that patch doesn't have a future, although Andi hasn't yet dropped it.

I fixed all known bugs (but hasn't reached your tree it) and right now it looks good to not be a drop.

Of course more testing will tell.

-Andi

acpi_force can become static.

@@ -37,7 +37,7 @@
 #include <asm/io.h>
 #include <asm/mpspec.h>
 
-int __initdata acpi_force = 0;
+static int __initdata acpi_force = 0;
 
 #ifdef	CONFIG_ACPI
 int acpi_disabled = 0;

Both patches added thanks

-Andi

enable_local_apic can now become static.

---

 arch/i386/kernel/apic.c |   13 ++++++++++++-
 include/asm-i386/apic.h |   12 ------------
 2 files changed, 12 insertions(+), 13 deletions(-)

@@ -16,20 +16,8 @@
 #define APIC_VERBOSE 1
 #define APIC_DEBUG   2
 
-extern int enable_local_apic;
 extern int apic_verbosity;
 
-static inline void lapic_disable(void)
-{
-	enable_local_apic = -1;
-	clear_bit(X86_FEATURE_APIC, boot_cpu_data.x86_capability);
-}
-
-static inline void lapic_enable(void)
-{
-	enable_local_apic = 1;
-}
-
 /*
  * Define the default level of output to be very little
  * This can be turned up by using apic=verbose for more

@@ -52,7 +52,18 @@
 /*
  * Knob to control our willingness to enable the local APIC.
  */
-int enable_local_apic __initdata = 0; /* -1=force-disable, +1=force-enable */
+static int enable_local_apic __initdata = 0; /* -1=force-disable, +1=force-enable */
+
+static inline void lapic_disable(void)
+{
+	enable_local_apic = -1;
+	clear_bit(X86_FEATURE_APIC, boot_cpu_data.x86_capability);
+}
+
+static inline void lapic_enable(void)
+{
+	enable_local_apic = 1;
+}
 
 /*
  * Debug level

This patch makes needlessly global code static.

---

BTW:
It doesn't seem to be intended that the new ipv4/fib_rules.c:fib4_rules_cleanup() is completely unused?

 include/net/ip6_fib.h              |    4 ----
 net/ipv4/cipso_ipv4.c              |    2 +-
 net/ipv4/fib_rules.c               |    4 ++--
 net/ipv6/fib6_rules.c              |    4 ++--
 net/ipv6/ip6_fib.c                 |    6 +++---
 net/ipv6/route.c                   |    6 +++---
 net/netlabel/netlabel_domainhash.c |    4 ++--
 7 files changed, 13 insertions(+), 17 deletions(-)

@@ -60,7 +60,7 @@
  * if in practice there are a lot of different DOIs this list should
  * probably be turned into a hash table or something similar so we
  * can do quick lookups. */
-DEFINE_SPINLOCK(cipso_v4_doi_list_lock);
+static DEFINE_SPINLOCK(cipso_v4_doi_list_lock);
 static struct list_head cipso_v4_doi_list = LIST_HEAD_INIT(cipso_v4_doi_list);
 
 /* Label mapping cache */

@@ -101,8 +101,8 @@
 	return err;
 }
 
-int fib4_rule_action(struct fib_rule *rule, struct flowi *flp, int flags,
-		     struct fib_lookup_arg *arg)
+static int fib4_rule_action(struct fib_rule *rule, struct flowi *flp,
+			    int flags, struct fib_lookup_arg *arg)
 {
 	int err = -EAGAIN;
 	struct fib_table *tbl;

@@ -66,8 +66,8 @@
 	return (struct dst_entry *) arg.result;
 }
 
-int fib6_rule_action(struct fib_rule *rule, struct flowi *flp,
-		     int flags, struct fib_lookup_arg *arg)
+static int fib6_rule_action(struct fib_rule *rule, struct flowi *flp,
+			    int flags, struct fib_lookup_arg *arg)
 {
 	struct rt6_info *rt = NULL;
 	struct fib6_table *table;

@@ -192,10 +192,6 @@
 					     struct in6_addr *daddr, int dst_len,
 					     struct in6_addr *saddr, int src_len);
 
-extern void			fib6_clean_tree(struct fib6_node *root,
-						int (*func)(struct rt6_info *, void *arg),
-						int prune, void *arg);
-
 extern void			fib6_clean_all(int (*func)(struct rt6_info *, void *arg),
 					       int prune, void *arg);
 

@@ -1169,9 +1169,9 @@
  *	ignoring pure split nodes) will be scanned.
  */
 
-void fib6_clean_tree(struct fib6_node *root,
-		     int (*func)(struct rt6_info *, void *arg),
-		     int prune, void *arg)
+static void fib6_clean_tree(struct fib6_node *root,
+			    int (*func)(struct rt6_info *, void *arg),
+			    int prune, void *arg)
 {
 	struct fib6_cleaner_t c;
 

@@ -613,8 +613,8 @@
 	return rt;
 }
 
-struct rt6_info *ip6_pol_route_input(struct fib6_table *table, struct flowi *fl,
-				     int flags)
+static struct rt6_info *ip6_pol_route_input(struct fib6_table *table,
+					    struct flowi *fl, int flags)
 {
 	struct fib6_node *fn;
 	struct rt6_info *rt, *nrt;

@@ -872,7 +872,7 @@
 }
 
 static struct dst_entry *ndisc_dst_gc_list;
-DEFINE_SPINLOCK(ndisc_lock);
+static DEFINE_SPINLOCK(ndisc_lock);
 
 struct dst_entry *ndisc_dst_alloc(struct net_device *dev, 
 				  struct neighbour *neigh,

@@ -50,11 +50,11 @@
 /* Domain hash table */
 /* XXX - updates should be so rare that having one spinlock for the entire
  * hash table should be okay */
-DEFINE_SPINLOCK(netlbl_domhsh_lock);
+static DEFINE_SPINLOCK(netlbl_domhsh_lock);
 static struct netlbl_domhsh_tbl *netlbl_domhsh = NULL;
 
 /* Default domain mapping */
-DEFINE_SPINLOCK(netlbl_domhsh_def_lock);
+static DEFINE_SPINLOCK(netlbl_domhsh_def_lock);
 static struct netlbl_dom_map *netlbl_domhsh_def = NULL;
 
 /*

From: Adrian Bunk <[email protected]>
Date: Mon, 7 Aug 2006 17:49:47 +0200

Looks reasonable, applied.

I'll kill it off.

IPv4 can't be built as a module and therefore there is no relevant exit or module load error path for ipv4 for which this function should be called.

Thanks.

This patch contains the following cleanups:
- make needlessly global code static
- use C99 struct initializers

---

The {cia,geode_aes}_{setkey,encrypt,decryt} prototype confusion both sparse and gcc are giveng warnings about should also be fixed.

 drivers/crypto/geode-aes.c |   12 ++++++------
 drivers/crypto/geode-aes.h |    2 --
 2 files changed, 6 insertions(+), 8 deletions(-)

@@ -37,6 +37,4 @@
   u8 iv[AES_IV_LENGTH];
 };
 
-unsigned int geode_aes_crypt(struct geode_aes_op *);
-
 #endif

@@ -114,7 +114,7 @@
 	AWRITE((status & 0xFF) | AES_INTRA_PENDING, AES_INTR_REG);
 }
 
-unsigned int
+static unsigned int
 geode_aes_crypt(struct geode_aes_op *op)
 {
 	u32 flags = 0;

@@ -361,7 +361,7 @@
 	return ret;
 }
 
-struct pci_device_id geode_aes_tbl[] = {
+static struct pci_device_id geode_aes_tbl[] = {
 	{ PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_LX_AES, PCI_ANY_ID, PCI_ANY_ID} ,
 	{ 0, }
 };

@@ -369,10 +369,10 @@
 MODULE_DEVICE_TABLE(pci, geode_aes_tbl);
 
 static struct pci_driver geode_aes_driver = {
-	name:      "Geode LX AES",
-	id_table:  geode_aes_tbl,
-	probe:     geode_aes_probe,
-	remove:    __devexit_p(geode_aes_remove)
+	.name		= "Geode LX AES",
+	.id_table	= geode_aes_tbl,
+	.probe		= geode_aes_probe,
+	.remove		= __devexit_p(geode_aes_remove)
 };
 
 static int __devinit

Hello,

after resume from ram (tested in single user), I can type commands for a few seconds (time is variable), the processes get stuck in io_schedule. Poorman's screenshots are here:
http://oioio.altervista.org/linux/dsc03448.jpg
http://oioio.altervista.org/linux/dsc03449.jpg

.config:
http://oioio.altervista.org/linux/config-2.6.18-rc3-mm2-1

Anything useful I could add?

mattia
:wq!

That probably measn that the device or device driver has got itself into a sick state and IO completions aren't occurring.

Which storage device (and which device driver) is being used here?

BTW: I tried to reverse ide-reprogram-disk-pio-timings-on-resume.patch with no luck.

A dmesg is available here (apart from the already resolved BUGs the boot process is meaningful):
http://oioio.altervista.org/linux/dmesg-2.6.18-rc3-mm2-1

[    3.168000] ICH3M: chipset revision 1
[    3.168000] ICH3M: not 100% native mode: will probe irqs later
[    3.168000]     ide0: BM-DMA at 0x1860-0x1867, BIOS settings: hda:DMA, hdb:pio
[    3.168000]     ide1: BM-DMA at 0x1868-0x186f, BIOS settings: hdc:pio, hdd:pio
[    3.168000] Probing IDE interface ide0...
[    3.460000] hda: FUJITSU MHV2080AH, ATA DISK drive
[    4.132000] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[    4.136000] Probing IDE interface ide1...
[    4.704000] Probing IDE interface ide1...
[    5.272000] hda: max request size: 128KiB
[    5.344000] hda: 156301488 sectors (80026 MB) w/8192KiB Cache, CHS=65535/16/63, UDMA(100)
[    5.348000] hda: cache flushes supported
[    5.352000]  hda: hda1 hda2 hda3 hda4 < hda5 hda6 >

lspci reports:
00:1f.1 IDE interface: Intel Corporation 82801CAM IDE U100 (rev 01) (prog-if 8a [Master SecP PriP])
Subsystem: Sony Corporation VAIO PCG-GR214EP/GR214MP/GR215MP/GR314MP/GR315MP
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin A routed to IRQ 255
Region 0: I/O ports at <ignored>
Region 1: I/O ports at <ignored>
Region 2: I/O ports at <ignored>
Region 3: I/O ports at <ignored>
Region 4: I/O ports at 1860 [size=16]
Region 5: Memory at d0000000 (32-bit, non-prefetchable) [size=1K]

mattia
:wq!

reverting git-block.patch (plus a couple more to make the thing build) let me resume correctly (2 cycles already).

Suggestion taken from the "swsusp regression" sub-thread.

mattia
:wq!

This patch removes three no longer used functions (that are even generating gcc warnings).

This patch doesn't look right, but it is the result of 58e5528ee464d38040b9489e10033c9387a10d56 in git-netdev...

---

 drivers/net/wireless/bcm43xx/bcm43xx_main.c |   33 --------------------
 1 file changed, 33 deletions(-)

@@ -3194,39 +3194,6 @@
 	bcm43xx_clear_keys(bcm);
 }
 
-static int bcm43xx_rng_read(struct hwrng *rng, u32 *data)
-{
-	struct bcm43xx_private *bcm = (struct bcm43xx_private *)rng->priv;
-	unsigned long flags;
-
-	spin_lock_irqsave(&(bcm)->irq_lock, flags);
-	*data = bcm43xx_read16(bcm, BCM43xx_MMIO_RNG);
-	spin_unlock_irqrestore(&(bcm)->irq_lock, flags);
-
-	return (sizeof(u16));
-}
-
-static void bcm43xx_rng_exit(struct bcm43xx_private *bcm)
-{
-	hwrng_unregister(&bcm->rng);
-}
-
-static int bcm43xx_rng_init(struct bcm43xx_private *bcm)
-{
-	int err;
-
-	snprintf(bcm->rng_name, ARRAY_SIZE(bcm->rng_name),
-		 "%s_%s", KBUILD_MODNAME, bcm->net_dev->name);
-	bcm->rng.name = bcm->rng_name;
-	bcm->rng.data_read = bcm43xx_rng_read;
-	bcm->rng.priv = (unsigned long)bcm;
-	err = hwrng_register(&bcm->rng);
-	if (err)
-		printk(KERN_ERR PFX "RNG init failed (%d)\n", err);
-
-	return err;
-}
-
 static int bcm43xx_shutdown_all_wireless_cores(struct bcm43xx_private *bcm)
 {
 	int ret = 0;

Hm, can't find that commit in a tree.
I looked at linus', netdev-2.6.

But one thing is for sure. This patch is _wrong_. ;)

NACK.

Greetings Michael.

It's in netdev-2.6.git#ALL that gets included in -mm.

And it seems to be your fault. ;-)

commit 58e5528ee464d38040b9489e10033c9387a10d56

Author: Michael Buesch <[email protected]>
Date:   Sat Jul 8 22:02:18 2006 +0200

[PATCH] bcm43xx: init routine rewrite

Rewrite of the bcm43xx initialization routines.
This fixes several issues:
* up-down-up-down-up... stale data issue
(May fix some DHCP issues)
* Fix the init vs IRQ handler race (and remove the workaround)
* Fix init for cards with multiple cores (APHY)
As softmac has no internal PHY handling (unlike dscape), this adds the file "phymode" to sysfs.
The active PHY can be selected by writing either a, b or g to this file. Current PHY can be determined by reading from it.
* Fix the controller restart code.
Controller restart can now also be triggered through echo 1 > /debug/bcm43xx/ethX/restart

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Uh, oh. I'm trapped.

Ah, I guessed it.
This was caused by some merge-race ;)
Will send a fix for this, soon.

Greetings Michael.

It's clearly in netdev-2.6.git#upstream:

commit 58e5528ee464d38040b9489e10033c9387a10d56

Author: Michael Buesch <[email protected]>
Date:   Sat Jul 8 22:02:18 2006 +0200

[PATCH] bcm43xx: init routine rewrite

Rewrite of the bcm43xx initialization routines.
This fixes several issues:
* up-down-up-down-up... stale data issue
(May fix some DHCP issues)
* Fix the init vs IRQ handler race (and remove the workaround)
* Fix init for cards with multiple cores (APHY)
As softmac has no internal PHY handling (unlike dscape), this adds the file "phymode" to sysfs.
The active PHY can be selected by writing either a, b or g to this file. Current PHY can be determined by reading from it.
* Fix the controller restart code.
Controller restart can now also be triggered through echo 1 > /debug/bcm43xx/ethX/restart

Hi,

I get something like the appended on every attempt to unmount the reiserfs filesystem mounted on /tmp. The other reiserfs filesystems don't have such problems and this one didn't have them too with 2.6.18-rc2-mm1.

BUG: Dentry ffff810037c573e8{i=3,n=.reiserfs_priv} still in use (1) [unmount of reiserfs hdc7]
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at fs/dcache.c:611
invalid opcode: 0000 [1] PREEMPT
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 0
Modules linked in: ide_cd cdrom xt_pkttype ipt_LOG xt_limit usbserial asus_acpi thermal processor fan button battery ac snd_pcm_oss snd_mix

er_oss snd_seq snd_seq_device af_packet bcm43xx ieee80211softmac ieee80211 ieee80211_crypt pcmcia firmware_class ohci1394 ieee1394 skge yen ta_socket rsrc_nonstatic pcmcia_core usbhid ff_memless ip6t_REJECT xt_tcpudp ipt_REJECT xt_state snd_intel8x0 snd_ac97_codec snd_ac97_bus s nd_pcm snd_timer snd iptable_mangle soundcore iptable_nat ip_nat iptable_filter snd_page_alloc ip6table_mangle ehci_hcd ip_conntrack i2c_nf orce2 i2c_core ip_tables ohci_hcd ip6table_filter ip6_tables x_tables ipv6 parport_pc lp parport dm_mod
Pid: 9478, comm: umount Not tainted 2.6.18-rc3-mm2 #7

RIP: 0010:[<ffffffff802a6eb7>]  [<ffffffff802a6eb7>] shrink_dcache_for_umount_subtree+0x1d7/0x2b0
RSP: 0018:ffff810059291da8  EFLAGS: 00010296
RAX: 0000000000000062 RBX: ffff810037c573e8 RCX: 0000000000000003
RDX: 0000000000000008 RSI: ffff810037c627d8 RDI: 0000000000000001
RBP: ffff810059291dc8 R08: 0000000000000002 R09: ffffffff8022de59
R10: 0000000000000000 R11: 0000000000000001 R12: ffff810037c573e8
R13: ffff81005ddc4800 R14: ffff81005f539250 R15: ffff81005f0a8688
FS:  00002afc38e00b00(0000) GS:ffffffff808c2000(0000) knlGS:00000000558b4d00
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002ac8a49a1d40 CR3: 000000005f546000 CR4: 00000000000006e0
Process umount (pid: 9478, threadinfo ffff810059290000, task ffff810037c62080)
Stack:  ffff81005f0a8b10 ffff81005f0a8688 ffffffff80577a20 ffff810059291ea8
 ffff810059291de8 ffffffff802a6fc4 ffff81005f0a8688 ffffffff80577a20
 ffff810059291e18 ffffffff80293bb4 ffff81005f539250 ffff81005e09d140
Call Trace:
 [<ffffffff802a6fc4>] shrink_dcache_for_umount+0x34/0x70
 [<ffffffff80293bb4>] generic_shutdown_super+0x24/0x110
 [<ffffffff80293cd0>] kill_block_super+0x30/0x50
 [<ffffffff80293f81>] deactivate_super+0x81/0xa0
 [<ffffffff802ac008>] mntput_no_expire+0x58/0xa0
 [<ffffffff8029b83d>] path_release_on_umount+0x1d/0x30
 [<ffffffff802ad3f4>] sys_umount+0x274/0x290
 [<ffffffff80209d0e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83

Leftover inexact backtrace:

Code: 0f 0b 68 41 33 4a 80 c2 63 02 49 8b 5c 24 68 49 39 dc 75 05
RIP  [<ffffffff802a6eb7>] shrink_dcache_for_umount_subtree+0x1d7/0x2b0
 RSP <ffff810059291da8>

Thanks, Rafael. vfs-destroy-the-dentries-contributed-by-a-superblock-on-unmounting.patch added that BUG_ON().

Make sure all dentries refs are released before calling kill_block_super() so that the assumption that generic_shutdown_super() can completely destroy the dentry tree for there will be no external references holds true.

What was being done in the put_super() superblock op, is now done in the kill_sb() filesystem op instead, prior to calling kill_block_super().

This prevents the BUG_ON() in the reduced-locking dcache destroyer patch from barking at reiserfs.

I've tested this patch by creating a ReiserFS partition, mounting and unmounting it a few times, and doing things to its contents whilst it is mounted.

---

 fs/reiserfs/super.c |   19 +++++++++++++------
 1 files changed, 13 insertions(+), 6 deletions(-)

index 5567328..69eefe2 100644
@@ -430,22 +430,29 @@ int remove_save_link(struct inode *inode
 	return journal_end(&th, inode->i_sb, JOURNAL_PER_BALANCE_CNT);
 }
 
-static void reiserfs_put_super(struct super_block *s)
+static void reiserfs_kill_sb(struct super_block *s)
 {
-	int i;
-	struct reiserfs_transaction_handle th;
-	th.t_trans_id = 0;
-
 	if (REISERFS_SB(s)->xattr_root) {
 		d_invalidate(REISERFS_SB(s)->xattr_root);
 		dput(REISERFS_SB(s)->xattr_root);
+		REISERFS_SB(s)->xattr_root = NULL;
 	}
 
 	if (REISERFS_SB(s)->priv_root) {
 		d_invalidate(REISERFS_SB(s)->priv_root);
 		dput(REISERFS_SB(s)->priv_root);
+		REISERFS_SB(s)->priv_root = NULL;
 	}
 
+	kill_block_super(s);
+}
+
+static void reiserfs_put_super(struct super_block *s)
+{
+	int i;
+	struct reiserfs_transaction_handle th;
+	th.t_trans_id = 0;
+
 	/* change file system state to current state if it was mounted with read-write permissions */
 	if (!(s->s_flags & MS_RDONLY)) {
 		if (!journal_begin(&th, s, 10)) {

@@ -2300,7 +2307,7 @@ struct file_system_type reiserfs_fs_type
 	.owner = THIS_MODULE,
 	.name = "reiserfs",
 	.get_sb = get_super_block,
-	.kill_sb = kill_block_super,
+	.kill_sb = reiserfs_kill_sb,
 	.fs_flags = FS_REQUIRES_DEV,
 };
 

It didn't apply cleanly to -rc3-mm2 for me and produces the appended oops every time at the kernel startup (on x86_64).

Greetings,
Rafael

input: SynPS/2 Synaptics TouchPad as /class/input/input2
RAMDISK: ext2 filesystem found at block 0
RAMDISK: Loading 2000KiB [1 disk] into ram disk... done.

Unable to handle kernel NULL pointer dereference at 0000000000000510 RIP:
 [<ffffffff802edc73>] reiserfs_kill_sb+0x13/0xa0
PGD 0
Oops: 0000 [1] PREEMPT
last sysfs file: /block/hdc/range
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.18-rc3-mm2 #10
RIP: 0010:[<ffffffff802edc73>]  [<ffffffff802edc73>] reiserfs_kill_sb+0x13/0xa0
RSP: 0000:ffff81005ff27a98  EFLAGS: 00010292
RAX: 0000000000000000 RBX: ffff810037c3ad20 RCX: 0000000000000003
RDX: 0000000000000008 RSI: ffff81005ff08798 RDI: ffff810037c3ad20
RBP: ffff81005ff27aa8 R08: 0000000000000001 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff805778c0
R13: 0000000000000001 R14: ffff810037c23080 R15: ffff810037df8168
FS:  0000000000000000(0000) GS:ffffffff808c2000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000510 CR3: 0000000000201000 CR4: 00000000000006e0
Process swapper (pid: 1, threadinfo ffff81005ff26000, task ffff81005ff08040)
Stack:  ffff810037c3ad20 ffff810037c3ad20 ffff81005ff27ac8 ffffffff80294671
 00000000ffffffea ffff810037c3ad20 ffff81005ff27b38 ffffffff8029539f
 ffffffff802ee260 0000000000000000 ffffff00306d6172 ffff810037df8168
Call Trace:
 [<ffffffff80294671>] deactivate_super+0x81/0xa0
 [<ffffffff8029539f>] get_sb_bdev+0x12f/0x180
 [<ffffffff802ec653>] get_super_block+0x13/0x20
 [<ffffffff80294746>] vfs_kern_mount+0xb6/0x160
 [<ffffffff8029485a>] do_kern_mount+0x4a/0x70
 [<ffffffff802ae370>] do_mount+0x720/0x790
 [<ffffffff802ae474>] sys_mount+0x94/0xe0
 [<ffffffff808d5b75>] mount_block_root+0xf5/0x2a0
 [<ffffffff808d84d2>] initrd_load+0xc2/0x330
 [<ffffffff808d5e43>] prepare_namespace+0xc3/0x140
 [<ffffffff8020723c>] init+0x1dc/0x2c0
 [<ffffffff8020a706>] child_rip+0x8/0x12
DWARF2 unwinder stuck at child_rip+0x8/0x12
Leftover inexact backtrace:
 [<ffffffff80471edb>] _spin_unlock_irq+0x2b/0x60
 [<ffffffff8020a2c0>] restore_args+0x0/0x30
 [<ffffffff80207060>] init+0x0/0x2c0
 [<ffffffff8020a6fe>] child_rip+0x0/0x12


Code: 48 8b b8 10 05 00 00 48 85 ff 74 31 e8 9c a3 fb ff 48 8b 83
RIP  [<ffffffff802edc73>] reiserfs_kill_sb+0x13/0xa0
 RSP <ffff81005ff27a98>

CR2: 0000000000000510
<0>Kernel panic - not syncing: Attempted to kill init!

Can you send me your modified patch?

David

Hmmm... It works okay for me, but then I'm testing it on i686, not x86_64. Should I draw any meaning from you saying "(on x86_64)"?

Also, can you do:

gdb vmlinux

And then at the prompt, can you disassemble the reiserfs_kill_sb() function:

disas reiserfs_kill_sb

And send me the disassembly?

If I had to guess, I'd say that REISERFS_SB() returned a NULL pointer, and that sb->s_root is NULL. In which case generic_shutdown_super() will not invoke reiserfs_put_super().

Something that you can try is to modify reiserfs_kill_sb() to be:

	static void reiserfs_kill_sb(struct super_block *s)
	{
		if (REISERFS_SB(s) {
			if (REISERFS_SB(s)->xattr_root) {
				d_invalidate(REISERFS_SB(s)->xattr_root);
				dput(REISERFS_SB(s)->xattr_root);
				REISERFS_SB(s)->xattr_root = NULL;
			}

			if (REISERFS_SB(s)->priv_root) {
				d_invalidate(REISERFS_SB(s)->priv_root);
				dput(REISERFS_SB(s)->priv_root);
				REISERFS_SB(s)->priv_root = NULL;
			}
		}

		kill_block_super(s);
	}

That way the function will be able to kill a superblock that isn't fully initialised.

David

Make sure all dentries refs are released before calling kill_block_super() so that the assumption that generic_shutdown_super() can completely destroy the dentry tree for there will be no external references holds true.

What was being done in the put_super() superblock op, is now done in the kill_sb() filesystem op instead, prior to calling kill_block_super().

Changes made in [try #2]:

(*) reiserfs_kill_sb() now checks that the superblock FS info pointer is set
before trying to dereference it.

---

 fs/reiserfs/super.c |   31 ++++++++++++++++++++-----------
 1 files changed, 20 insertions(+), 11 deletions(-)

index 5567328..c6e327a 100644
@@ -430,22 +430,31 @@ int remove_save_link(struct inode *inode
 	return journal_end(&th, inode->i_sb, JOURNAL_PER_BALANCE_CNT);
 }
 
+static void reiserfs_kill_sb(struct super_block *s)
+{
+	if (REISERFS_SB(s)) {
+		if (REISERFS_SB(s)->xattr_root) {
+			d_invalidate(REISERFS_SB(s)->xattr_root);
+			dput(REISERFS_SB(s)->xattr_root);
+			REISERFS_SB(s)->xattr_root = NULL;
+		}
+
+		if (REISERFS_SB(s)->priv_root) {
+			d_invalidate(REISERFS_SB(s)->priv_root);
+			dput(REISERFS_SB(s)->priv_root);
+			REISERFS_SB(s)->priv_root = NULL;
+		}
+	}
+
+	kill_block_super(s);
+}
+
 static void reiserfs_put_super(struct super_block *s)
 {
 	int i;
 	struct reiserfs_transaction_handle th;
 	th.t_trans_id = 0;
 
-	if (REISERFS_SB(s)->xattr_root) {
-		d_invalidate(REISERFS_SB(s)->xattr_root);
-		dput(REISERFS_SB(s)->xattr_root);
-	}
-
-	if (REISERFS_SB(s)->priv_root) {
-		d_invalidate(REISERFS_SB(s)->priv_root);
-		dput(REISERFS_SB(s)->priv_root);
-	}
-
 	/* change file system state to current state if it was mounted with read-write permissions */
 	if (!(s->s_flags & MS_RDONLY)) {
 		if (!journal_begin(&th, s, 10)) {

@@ -2300,7 +2309,7 @@ struct file_system_type reiserfs_fs_type
 	.owner = THIS_MODULE,
 	.name = "reiserfs",
 	.get_sb = get_super_block,
-	.kill_sb = kill_block_super,
+	.kill_sb = reiserfs_kill_sb,
 	.fs_flags = FS_REQUIRES_DEV,
 };
 

This one works on my box just fine.

Thanks,
Rafael

Excellent, thanks.

David

Yum managed to get wedged: 'echo t > /proc/sysrq-trigger' says:

[ 4514.840000] yum           D D5C32AA0     0  4747   4430                     (NOTLB)
[ 4514.840000]        d5c3dda4 d5c3dd78 00000007 d5c32aa0 bd3ddd00 00000338 00000000 d5c32bc0
[ 4514.840000]        c1601628 d5c3dd9c 64600300 0000001f d5c3ddd8 d5c3ddd8 c1601628 d5c3ddac
[ 4514.840000]        c034fef8 d5c3ddb4 c0136e8e d5c3ddcc c0350026 c0136e58 d5c3ddd8 00000000
[ 4514.840000] Call Trace:
[ 4514.840000]  [<c034fef8>] io_schedule+0x25/0x44
[ 4514.840000]  [<c0136e8e>] sync_page+0x36/0x3a
[ 4514.840000]  [<c0350026>] __wait_on_bit_lock+0x30/0x58
[ 4514.840000]  [<c0136e44>] __lock_page+0x51/0x59
[ 4514.840000]  [<c013f099>] truncate_inode_pages_range+0x1de/0x230
[ 4514.840000]  [<c013f0f7>] truncate_inode_pages+0xc/0x11
[ 4514.840000]  [<c018ea12>] ext3_delete_inode+0x16/0xbd
[ 4514.840000]  [<c016798f>] generic_delete_inode+0xb6/0x130
[ 4514.840000]  [<c0167a1b>] generic_drop_inode+0x12/0x166
[ 4514.840000]  [<c01673f1>] iput+0x67/0x6a
[ 4514.840000]  [<c0165662>] dentry_iput+0x97/0xcc
[ 4514.840000]  [<c016613d>] dput+0x183/0x19c
[ 4514.840000]  [<c015f64f>] sys_renameat+0x17a/0x1d3
[ 4514.840000]  [<c015f6ba>] sys_rename+0x12/0x14
[ 4514.840000]  [<c0102849>] sysenter_past_esp+0x56/0x79

A careful check of the dmesg doesn't reveal anything particularly helpful, like an oops or other relevant kernel message.

Usually this means that there's an IO request in flight and it got lost somewhere. Device driver bug, IO scheduler bug, etc. Conceivably a lost interrupt (hardware bug, PCI setup bug, etc).

Which device driver and which IO sched are you using?

    (NOTLB)

00000000 d5c32bc0

c1601628 d5c3ddac

d5c3ddd8 00000000

Aug  9 13:33:13 turing-police kernel: [   11.297507] libata version 2.00 loaded.
Aug  9 13:33:14 turing-police kernel: [   11.297763] ata_piix 0000:00:1f.1: version 2.00ac6
Aug  9 13:33:14 turing-police kernel: [   11.297780] PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
Aug  9 13:33:14 turing-police kernel: [   11.299245] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
Aug  9 13:33:14 turing-police kernel: [   11.299786] ACPI: PCI Interrupt 0000:00:1f.1[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11
Aug  9 13:33:14 turing-police kernel: [   11.300638] PCI: Setting latency timer of device 0000:00:1f.1 to 64
Aug  9 13:33:15 turing-police kernel: [   11.300720] ata1: PATA max UDMA/100 cmd 0x1F0 ctl 0x3F6 bmdma 0xBFA0 irq 14
Aug  9 13:33:15 turing-police kernel: [   11.301381] scsi0 : ata_piix

...

Disk was running with 'cfq' scheduler. I checked the dmesg, and only odd thing was this:

Aug  9 14:30:24 turing-police kernel: [ 3535.720000] end_request: I/O error, dev fd0, sector 0

Wierd though - floppy and ATA are on different IRQs according to /proc/interrupts:

CPU0

  0:   11122651    XT-PIC-level    timer
  1:      12532    XT-PIC-level    i8042
  2:          0    XT-PIC-level    cascade
  5:     190651    XT-PIC-level    Intel 82801CA-ICH3
  6:          5    XT-PIC-level    floppy
  8:          1    XT-PIC-level    rtc
  9:          1    XT-PIC-level    acpi
 11:     238728    XT-PIC-level    uhci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, yenta, yenta, yenta, ohci1394, pcmcia2.0, eth3
 12:        114    XT-PIC-level    i8042
 14:     172656    XT-PIC-level    libata
 15:          0    XT-PIC-level    libata
NMI:          0 
LOC:          0 
ERR:          1
MIS:          0

(For the record, the laptop doesn't even *have* a floppy drive installed at the moment)

Red herring. yum just wedged again, this time with no reference to floppy drive. Same traceback. Anybody have anything to suggest before I start playing hunt-the-wumpus with a -mm bisection?

Hmm, I have the accurately same problem...
yum + CFQ + BLK_DEV_PIIX + nothing odd in dmesg

[ 3438.574864] yum           D 00000000     0 21659   3838 (NOTLB)
[ 3438.575098]        e5c09d24 00000001 c180f5a8 00000000 e5c09ce0 c01683e8 fe37c0bc 000002c4
[ 3438.575388]        00001000 00000001 c18fbbd0 0023001f 00000007 f26cc560 c1913560 fe4166d5
[ 3438.575713]        000002c4 0009a619 00000001 f26cc66c c180ec40 c04ff140 e5c09d14 c01fad44
[ 3438.576039] Call Trace:
[ 3438.576113]  [<c0373d3b>] io_schedule+0x26/0x30
[ 3438.576187]  [<c014653c>] sync_page+0x39/0x45
[ 3438.576260]  [<c0374401>] __wait_on_bit_lock+0x41/0x64
[ 3438.576333]  [<c01464ef>] __lock_page+0x57/0x5f
[ 3438.576405]  [<c014f5f2>] truncate_inode_pages_range+0x1b6/0x304
[ 3438.576480]  [<c014f76f>] truncate_inode_pages+0x2f/0x40
[ 3438.576553]  [<c01a7bc4>] ext3_delete_inode+0x29/0xf7
[ 3438.576627]  [<c017f26b>] generic_delete_inode+0x65/0xe7
[ 3438.576701]  [<c017f3aa>] generic_drop_inode+0xbd/0x173
[ 3438.576774]  [<c017ed25>] iput+0x6b/0x7b
[ 3438.576846]  [<c017cc57>] dentry_iput+0x68/0xb3
[ 3438.576919]  [<c017d99e>] dput+0x4f/0x19f
[ 3438.576990]  [<c0176164>] sys_renameat+0x1e0/0x212
[ 3438.577063]  [<c01761be>] sys_rename+0x28/0x2a
[ 3438.577135]  [<c01030fb>] syscall_call+0x7/0xb

regards,

<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Is yum the only process which was stuck in D state?

If so, I'd still be expecting a device driver/iosched bug.

If not, it's probably a vfs/fs deadlock.

oooh, same setup and same trace here, but no yum, see some screenshots here:
http://oioio.altervista.org/linux/dsc03448.jpg
http://oioio.altervista.org/linux/dsc03449.jpg

The use case for me was simply:
- boot (in single user for the 2 shots)
- suspend
- resume
- wait some seconds and do anything that accesses the disk

[...]

in my case anything accessing the disk, leading to lockup shortly

I reverted the full git-block.patch and I'm now using rc3-mm2 since then suspending to ram, disk and using my laptop for daily stuff:

reboot system boot 2.6.18-rc3-mm2-1 Tue Aug 8 00:02 - 19:30 (2+19:27)

PS: my previous pasts are here: http://lkml.org/lkml/2006/8/7/264
probably an unfortunate Cc list :)

mattia
:wq!

This is reiser ^^?!, so we can exclude fs? I have this behaviour on ext3.

regards,

<a href="http://www.fi.muni.cz/~xslaby/">Jiri Slaby</a>
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Not quite the same trace - the first few lines are the same, but your call to __lock_page() comes in via do_generic_mapping_read(), while Jiri and I are seeing the call to __lock_page() coming from truncate_inode_pages_range()....

The suspend+resume->hang bug is known and reputedly fixed.

The stuck-in-lock_page-without-having-done-resume bug is not known. Someone please try the deadline scheduler, or AS.

echo anticipatory >| /sys/block/sda/queue/scheduler yum -C update yum-updatesd

And yum still hung with the exact same backtrace, so it's not a CFQ bug.

Same problem here, with urpmi:

urpmi D CAC9EAA0 6112 29146 30655 (NOTLB)

       c813dda0 c0291e70 00000001 cac9eaa0 fe7d7800 000008ad 00000000 cac9ebac 
       c813ddd4 c1404e00 07efea00 00000005 c813ddd4 c813ddd4 c1404e00 c813dda8 
       c028fb6b c813ddb0 c01390ed c813ddc8 c02902ea c01390b7 c813ddd4 00000000 
Call Trace:
 [<c028fb6b>] io_schedule+0xe/0x16
 [<c01390ed>] sync_page+0x36/0x3a
 [<c02902ea>] __wait_on_bit_lock+0x30/0x58
 [<c01390a3>] __lock_page+0x51/0x59
 [<c01403f4>] truncate_inode_pages_range+0x1f8/0x24a
 [<c0140452>] truncate_inode_pages+0xc/0x12
 [<c018a22a>] ext3_delete_inode+0x16/0xc0
 [<c0168e93>] generic_delete_inode+0x61/0xcf
 [<c0168f13>] generic_drop_inode+0x12/0x13e
 [<c0168994>] iput+0x67/0x6a
 [<c0166bab>] dentry_iput+0x7c/0x97
 [<c016792d>] dput+0x152/0x16b
 [<c0160fe5>] sys_renameat+0x17a/0x1dd
 [<c016105a>] sys_rename+0x12/0x14
 [<c0102c39>] sysenter_past_esp+0x56/0x8d

This is 2.6.18-rc3-mm2 + 6 hot-fixes. CFQ scheduler. The RPM DB is located on /var which is an ext3 FS.

I think I broke my RPM db :-(.
~~
laurent

Well, after a detour into hardware issues (a dying fan ended up escalating into swapping a motherboard), I built 2.6.18-rc4-mm1 - unable to replicate the 'yum' hang on that. Somehow, I'm not feeling very motivated to do a bisect of -rc3-mm2 to find it, unless somebody thinks we should track it down just in case it's just in hiding....

[this is a resend, as the original message may be too big to reach the list...]

Hello,

On my system, a cron runs every day to check the integrity of installed RPMS, it runs "rpm -v" on each package, which computes MD5 hash for each installed file and compares this result, the file size and modification time with values stored in RPM database.

This is the workload. Since 2.6.18-rc3-mm2, this processus eats all the memory and triggers OOM.

On my system, "free -t" output normally looks like this ("cached" value is about half of RAM):

# free -t 
             total       used       free     shared    buffers     cached
Mem:        515032     508512       6520          0      22992     256032
-/+ buffers/cache:     229488     285544
Swap:      1116428        324    1116104
Total:     1631460     508836    1122624

After the rpm database check, "free -t" says:

             total       used       free     shared    buffers     cached
Mem:        515032     507124       7908          0       8132     398296
-/+ buffers/cache:     100696     414336
Swap:      1116428      34896    1081532
Total:     1631460     542020    1089440

And the value of "cached" won't decrease.

This evening, this process trigger OOM-killer. Here is its first report:

syslogd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
[show_trace+13/16] show_trace+0xd/0x10

 [<c0104c18>] show_trace+0xd/0x10
 [dump_stack+25/29] dump_stack+0x19/0x1d
 [<c0104c34>] dump_stack+0x19/0x1d
 [out_of_memory+93/422] out_of_memory+0x5d/0x1a6
 [<c013be03>] out_of_memory+0x5d/0x1a6
 [__alloc_pages+505/633] __alloc_pages+0x1f9/0x279
 [<c013d25f>] __alloc_pages+0x1f9/0x279
 [__do_page_cache_readahead+165/495] __do_page_cache_readahead+0xa5/0x1ef
 [<c013e71b>] __do_page_cache_readahead+0xa5/0x1ef
 [do_page_cache_readahead+66/80] do_page_cache_readahead+0x42/0x50
 [<c013ec64>] do_page_cache_readahead+0x42/0x50
 [filemap_nopage+412/882] filemap_nopage+0x19c/0x372
 [<c013afbe>] filemap_nopage+0x19c/0x372
 [__handle_mm_fault+540/1772] __handle_mm_fault+0x21c/0x6ec
 [<c014435d>] __handle_mm_fault+0x21c/0x6ec
 [do_page_fault+397/1158] do_page_fault+0x18d/0x486
 [<c0111e1f>] do_page_fault+0x18d/0x486
 [error_code+57/64] error_code+0x39/0x40
 [<c0293079>] error_code+0x39/0x40

Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:63
cpu 0 cold: high 62, batch 15 used:61
Active:1621 inactive:97987 dirty:0 writeback:33 unstable:0 free:1215 slab:23388 mapped:3 pagetables:446
DMA free:2068kB min:88kB low:108kB high:132kB active:0kB inactive:7432kB present:16384kB pages_scanned:11284 all_unreclaimable? yes lowmem_reserve[]: 0 495
Normal free:2792kB min:2804kB low:3504kB high:4204kB active:6484kB inactive:384516kB present:507824kB pages_scanned:670357
all_unreclaimable? yes
lowmem_reserve[]: 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2068kB
Normal: 0*4kB 1*8kB 6*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2792kB
Swap cache: add 109576, delete 109542, find 12933/22258, race 0+8
Free swap = 936452kB
Total swap = 1116428kB
Free swap: 936452kB
131052 pages of RAM
0 pages of HIGHMEM
2358 reserved pages
2668 pages shared
34 pages swap cached
0 pages dirty
33 pages writeback
3 pages mapped
23388 pages slab
446 pages pagetables
Out of Memory: Kill process 23392 (seamonkey-bin) score 48523 and children.
Out of memory: Killed process 23392 (seamonkey-bin).

I gather some data before the rpm database check and near the end of it:
- /proc/slabinfo
- /proc/slab_allocators
- /proc/meminfo
- free -t

Please look in http://laurent.riffard.free.fr/2.6.18-rc3-mm2. You'll find dmesg and .config too.

For information:

/proc/sys/vm/block_dump:0
/proc/sys/vm/dirty_background_ratio:10
/proc/sys/vm/dirty_expire_centisecs:3000
/proc/sys/vm/dirty_ratio:40
/proc/sys/vm/dirty_writeback_centisecs:500
/proc/sys/vm/drop_caches:0
/proc/sys/vm/laptop_mode:0
/proc/sys/vm/legacy_va_layout:0
/proc/sys/vm/lowmem_reserve_ratio:256
/proc/sys/vm/max_map_count:65536
/proc/sys/vm/min_free_kbytes:2896
/proc/sys/vm/nr_pdflush_threads:2
/proc/sys/vm/overcommit_memory:0
/proc/sys/vm/overcommit_ratio:50
/proc/sys/vm/page-cluster:3
/proc/sys/vm/panic_on_oom:0
/proc/sys/vm/percpu_pagelist_fraction:0
/proc/sys/vm/readahead_hit_rate:1
/proc/sys/vm/readahead_ratio:50
/proc/sys/vm/swappiness:60
/proc/sys/vm/swap_prefetch:1
/proc/sys/vm/swap_token_timeout:300
/proc/sys/vm/vdso_enabled:1
/proc/sys/vm/vfs_cache_pressure:100

# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev /dev tmpfs rw 0 0
/dev/vglinux1/lvroot / ext3 rw,data=ordered 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
none /dev/pts devpts rw 0 0
none /dev/shm tmpfs rw 0 0
none /proc/bus/usb usbfs rw 0 0
/dev/hda2 /boot ext2 rw 0 0
/dev/vglinux1/lvhome /home reiserfs rw 0 0
/dev/vglinux1/lvusr /usr reiserfs ro 0 0
/dev/vglinux1/lvvar /var ext3 rw,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
automount(pid1949) /vol autofs rw,fd=4,pgrp=1949,timeout=5,minproto=2,maxproto=4,indirect 0 0

~~
laurent

Yes, I was just trying to reproduce this. No luck so far. Will try your .config tomorrow.

It would be interesting to try disabling CONFIG_ADAPTIVE_READAHEAD - perhaps that got broken.

Also, are you able to determine whether the problem is specific to `rpm -V'? Are you able to make the leak trigger using other filesystem workloads?

If it's specific to `rpm -V' then perhaps direct-io is somehow causing pagecache leakage. That would be a bit odd.

btw, it's not necessary to go all the way to oom to work out if the pagecache leak is happening. After booting, do

echo 3 > /proc/sys/vm/drop_pagecache

and record the Cached' figure in /proc/meminfo. After running some test, run echo 3 > /proc/sys/vm/drop_pagecache' again and check /proc/meminfo:Cached. If it dodn't do gown to a similarly low figure, we're leaking pagecache.

btw2: please use /proc/meminfo output rather than free(1). Because free(1) shows less info, and it does mysterious mangling of the info which it does read in ways which confuse me.

Thanks.

I just try it: when CONFIG_ADAPTIVE_READAHEAD is disabled, /proc/meminfo:Cached is stable and never exceeded 230.000, the system didn't even try to swap.

$ cat /proc/meminfo   # taken a few minutes after the end of rpm -V
MemTotal:       515032 kB
MemFree:          6612 kB
Buffers:         42212 kB
Cached:         182236 kB
SwapCached:          0 kB
Active:         376256 kB
Inactive:        75468 kB
SwapTotal:     1116428 kB
SwapFree:      1116428 kB
Dirty:             272 kB
Writeback:           0 kB
AnonPages:      227260 kB
Mapped:          62812 kB
Slab:            44968 kB
PageTables:       2152 kB
NFS Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   1373944 kB
Committed_AS:   637400 kB
VmallocTotal:   515796 kB
VmallocUsed:      6916 kB
VmallocChunk:   508760 kB

Will try...

I played with these values and as far I can remember, I get some poor improvement. Will try to gather some data.

Ok

laurent

I enabled CONFIG_ADAPTIVE_READAHEAD again, I applied all the hot-fixes and I ran another test with "rpm -qaV". Please see attached vmstat.log (output of vmstat 5).

No luck. For example, "find /usr -type f -print0 | xargs -0 cat > /dev/null" does not trigger the problem.

You can find the strace output of "rpm -qaV" at
http://laurent.riffard.free.fr/2.6.18-rc3-mm2/rpm-qaV-strace.log
(427 kB). Maybe you can guess what's specific...

FWIW, here are the relevant parts of my FS configuration:

# df
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vglinux1-lvroot
                      249M  210M   31M  88% /
/dev/mapper/vglinux1-lvusr
                      3.5G  3.4G  113M  97% /usr
/dev/mapper/vglinux1-lvvar
                      485M  330M  145M  70% /var

# mount
/dev/mapper/vglinux1-lvroot on / type ext3 (rw)
/dev/mapper/vglinux1-lvusr on /usr type reiserfs (ro)
/dev/mapper/vglinux1-lvvar on /var type ext3 (rw)

Please see attached file "typescript" for results of `echo 3 > /proc/sys/vm/drop_pagecache'.

laurent

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0   7832  21764 256992    0    0   609   164  342   410 15  5 64 16
 1  1      0   7436  21796 257380    0    0    55    10  326   465  8  1 88  3
 6  0      0   6376  21580 257608    0    0   186     6  407   832 28  3 65  3
 1  0      0   6004  21660 257660    0    0     3   200  396   835 33  4 54  9
 5  0      0   5844  21196 257308    0    0     2    24  398   616 15  3 79  3
 1  0      0   5644  21252 257308    0    0     0    15  290   459 24  4 70  1
13  0      0   6324  21124 256724    0    0     0    15  345   506 13  2 82  2
 0  0      0   6076  21180 256728    0    0     0    16  313   450 13  4 83  0
 0  0      0   6340  21280 256452    0    0     2   136  337   583 12  3 81  5
 9  0      0   5844  21336 256368    0    0   599    50  312   722 36  8 46  9
 1  0      0   6552  21396 254576    0    0     0    25  298   421 21  4 73  3
 0  0      0   6600  21412 254576    0    0     0     5  278   311  0  0 99  1
 2  0      0   6608  21428 254576    0    0     0     4  287   317 13  0 86  1
 5  0      0   6180  20948 253128    0    0  1599     8  352   363  5  6 68 20
11  0     48   6468  19664 252692    0    0  2233    31  407   402 28 37  0 36
12  0     56   7072  18812 251388    0    0  2180    38  411   557 10 15  0 75
 2  1    100   6036  18640 252808    0    0  1964    13  395   475 12 22  0 66
 3  1    100   6500  18348 251612    0    0  2672    12  418   562  7 12  0 81
 0  1    100   6172  18868 251424    0    0  2882     8  430   526 14 23  0 63
 0  1    100  30844  18232 227256    0    0  5543   182  423   519 16  8  0 75
 0  1    100  20856  19080 236312    0    0  1800   150  413   509 10 20  0 70
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
16  1    100   6512  20244 246884    0    0  1818    22  431   547 11 20  0 69
 0  1    100   5784  20088 247468    0    0  1568     7  417   550  4 12  0 84
 0  1    100   6288  19216 246960    0    0  1755    22  465   690  2  8  0 90
 0  1    100   6252  21480 243592    0    0  1125    12  478   714  1  6  0 93
 2  1    324   6348  21868 243700    0   11  4179    44  436   584 22 16  0 62
 1  1    324   6656  19928 244304    0    9  2136   233  447   589 15 21  0 64
 1  0    324   5776  16264 249728    0    0  2630    12  423   562  6  9  0 85
 1  1    324   6332  15984 249788    0    0  2414    10  514   622  9 19  0 72
 0  1    324   5960  17548 247628    0   45  1910    55  548   827  5 14  0 81
 6  0    324   5628  16124 249856    0    0  7056     8  462   638  7  9  0 83
 1  1    324   6100  16724 248732    0    0  3387    34  450   589  9 19  0 72
14  0    324   5612  17324 247348    0    0  1763   140  401   457 23 15  0 61
 5  1    324   5884  17092 247912    0    0  2904    14  452   533 10 17  0 72
 1  2    324   6400  16480 247712    0    0  2326    11  475   657  7 10  0 83
 2  1    324   5884  12080 252036    0    0  2658    10  410   536  8 15  0 77
 0  1    324   6040  13120 249940    0    0  2333     7  502   699 10 22  0 68
 0  2    324   5100  11872 250996    0    0  4964    12  451   511 11 20  0 70
 0  2    324   5768  11416 251532    0    0  1941     8  507   743 11 10  0 79
 1  1    324   6044  10168 251904    0    0  1089    18  490   706  4 11  0 85
 0  1    324   6376   9988 252376    0    0  1691   161  448   538 11 19  0 70
 1  1    324   6196  10744 251104    0    0  1572     7  412   509 11 23  0 66
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  1    324   5752  11628 250872    0    0  1710     9  417   501 10 20  0 70
 0  1    324   6304  11988 248712    0    0  2603    13  410   530 10 21  0 69
11  0    324   6348  12304 248544    0    0  1836     7  401   476 12 28  0 60
 0  1    324   5764  12340 248936    0    0  2456    45  404   522 18 13  0 69
 0  2    324   6000  10240 249944    0    0  5880    21  434   570  9 11  0 80
 1  1    324   6724   9132 250416    0    0  3671   157  419   517  9 14  0 78
 2  2    324   5268   6836 254468    0    0  6737     9  401   461  8  7  0 85
 0  1    324   6320   4680 255556    0    0  6013    12  440   589  7  8  0 86
 1  2    324   5648   4300 256560    0    0  4519    24  426   528  5  5  0 91
 0  1    324   5980   3576 256784    0    0  7595    15  403   548  6  5  0 89
 1  1    324   6184   3944 255720    0    0  3220    23  497   548 19 13  0 68
 0  1    324   6556   4504 254428    0    0  4983   147  400   494  6 10  0 84
 1  2    324   6472   4900 253820    0    0  3204     7  504   718  5 11  0 84
 0  2    324   5948   4980 253632    0    0  3737     5  448   569  7 11  0 82
 1  0    324   5852   5732 253416    0    0  2330     4  453   630  4  6  0 90
 1  2    324   6108   6416 251724    0    0  2408    15  458   560  8 19  0 73
 0  2    324   5092   7188 251680    0    0  4300    19  459   551 19 13  0 68
 1  1    324   6052   6756 251084    0    0  3778    22  406   539  4  6  0 90
 0  1    324   6924   7612 248708    0    0  2527    13  412   505  9 20  0 71
 1  1    324   5836   7596 249760    0    0  1811    14  405   499  8 18  0 74
10  1    324   5116   6888 251356    0    0  5613    12  440   470  8 17  0 75
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
11  0    324   5820   7180 249852    0    0  1874     8  450   602  4 13  0 83
 8  1    324   6192   7456 248364    0    0  2632    10  474   665  6 17  0 77
 3  1    324   6072   8072 247480    0    0  1709    10  422   533 13 16  0 71
 0  2    324   5040   8640 247564    0    0  1947   153  406   480 10 25  0 65
 0  1    324   5844   9756 243252    0    0   560   145  409   545  3 16  0 81
 0  1    324   6532   9456 241720    0    0  2930    16  422   558  5  9  0 87
 1  1    324   6104   9864 241828    0    0  2284    11  433   532  8 24  0 68
 0  1    324   6228  11372 239524    0    0  1179    10  413   580  2  5  0 94
 1  1    324   6264  11916 237968    0    0  1646     9  438   655 11  6  0 83
 1  2    324   6140  12372 236400    0    0   763    10  413   586  7  6  0 87
 2  0    324   6132  12192 235672    0    0  2024    33  422   575  4 12  0 83
 1  1    324   5440   8616 240236    0    0  3090   173  417   521  5  5  0 91
 0  1    324   6028   8720 239780    0    0  3010   136  443   548  5  9  0 86
 1  2    324   5140   5976 242500    0    0  3109     7  405   504  9 15  0 76
 0  2    324   5776   4556 243948    0    0  1817    15  429   548  7 16  0 77
 0  1    324   6824   4752 242040    0    0  2128     7  459   572 14 19  0 68
 0  2    324   7612   3108 242756    0    0  7304    19  419   549 11  9  0 80
 0  2    324  10120   1416 242576    0    0 11255    23  398   504  9  5  0 87
 6  3    324   5484   1452 247192    0    0 10157    20  452   525  8  4  0 88
 3  2    628   9708   1292 243624    0   61  9840    78  432   516  8  4  0 88
 0  1    628  16944   1504 235840    0    0  8235    15  433   580  6  5  0 88
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2   8316   5788   2272 252928    0 1405  8242  1430  435   577 14  8  0 79
 1  2  10504   5396   1300 257208    0  570 12718   589  443   557  9  7  0 84
 0  2  10504   5052   2024 257324    6    0  5701     6  448   647  5  6  0 89
 0  1  10504   5896   2520 255568    0    0  2658     7  459   655  3  4  0 93
 1  1  10504  12028   3268 248416    0    0  2798     5  492   598  3  8  0 89
 0  1  10504   5728   4628 252512    0    0  2069     7  432   555  6 11  0 83
 0  2  10504   5196   4504 252460    0    0  5945    17  485   611 14 13  0 73
 0  1  10504   6492   4572 251628    0    0  2383     7  477   643  6 12  0 82
 0  1  10504   9500   3524 249332    0    0  3341   156  420   537 11 10  0 78
 2  2  10504   5528   4288 252256    0    0  1657   139  418   492  6 17  0 77
 2  1  10504   6336   4784 250884    0    0  3929    17  486   686  6  9  0 85
 4  1  10504   5068   5320 250336    0    0  1467     7  411   567  3  7  0 90
 1  2  10504   5900   5776 248864    0    0  2299    15  443   589  4 14  0 82
 0  2  10504   4924   5864 248360    0    0  2007    14  436   509 16 18  0 66
 1  2  10504   7208   5548 246200    0    0  3496    15  441   575  6 11  0 84
 2  1  10504   5760   5696 246916    0    0  1855    20  449   571  5 19  0 75
 2  2  10504   5136   4936 248112    0    0  2007   150  414   537  7 17  0 77
 1  1  10504   8068   4564 245372    0    0  2229     7  428   535  7 13  0 80
 0  1  10504   5804   4740 247520    0    0  2009    10  418   519 10 16  0 74
 1  2  10504   5340   4064 248556    0    0  2650    13  445   569 16 13  0 71
 0  3  10504   6108   2652 247384   96    0  3143    21  545   789  4 14  0 82
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  1  10504   9940   2504 243300   64    0  2900   174  451   564  6 14  0 81
 2  2  10504   5676   3056 246676    0    0  3172    12  463   539  7 12  0 82
 9  2  10504   5992   2192 247712    6    0  4282    23  449   510  6 14  0 81
 1  2  10504   6388   2536 247092    0    0  2561    12  438   520  6 13  0 82
 2  2  10504   5700   2440 247596    0    0  3427   143  420   491  6 11  0 83
 6  0  10504  10740   2576 242812    0    0  2113    24  444   539 14 17  0 70
 0  2  36280   7908   3040 245328    0 5053  1312  5218  397   482  4 12  0 84
 1  1  36280   6428   3696 252384    0    0  1447   145  412   450  6 16  0 78
 0  1  36280   5612   4768 263700   45    0  2387     8  447   549 11 11  0 78
 1  0  49140   6484   5336 279980    0 2550  4669  2571  418   525  5  8  0 86
 0  1  60032   6436   6232 288752    0 2178  2134  2321  463   579  7 14  0 79
 0  1  68940   6684   6612 297124    0 1714  3380  1761  439   572 14 14  0 73
 2  0  68940   5912   5168 299284   64  125  4599   142  465   605  5  9  0 86
 0  2  68940   4936   3292 301096   84   67  8721   272  456   544  8  4  0 88
 0  2  68940   5564   2152 298308  832    0  6982    15  492   635  7  5  0 87
 0  2  68940   5064   2640 298132   58    0  3503    22  516   570  9 16  0 75
 1  1  68940   6380   3188 296580   54    0  6451    20  528   719  7  9  0 84
 0  2  68940   5624   3664 297656   11    0  2985    13  509   726 14 15  0 71
 1  1  68940   5776   4556 296596    0    0  2888   104  448   621 11  7  0 83
 3  2  68940   5940   5520 294044   13    0  2147    12  457   628  5  8  0 87
 5  0  68940   5768   5344 295480    0    0  3767    14  471   584  7 10  0 83
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  1  68940   9360   6656 290372    0    0  1338     8  445   645  2  7  0 91
 1  1  68940   6052   7340 291884   13    0  2878     9  467   620  6 13  0 80
 0  5  68940   5784   7012 290236  368    0  3677     9  473   592  6 16  0 78
 2  4  68844   5616   5776 284660 1096    0  2550    14  470   791 11  6  0 82
 0  5  68748   5436   4916 284656  632    0  4851    15  519   654  8 11  0 82
 0  2  68748   6704   3704 283380  223    0  2349   286  458   536  8 12  0 80
 4  1  68700   5952   4868 281996  139    0  1422    26  429   913  8  8  0 84
 0  2  68700   4980   6196 281508    0    0  2389    10  446   578  5 13  0 82
 1  1  68700   6312   6840 282244    0    0  1510   351  433   513 11 15  0 74
 9  1  68700   5996   7040 282968    0    0  1870   154  422   529  9 18  0 73
 0  2  68700   5028   3528 286440  613    0  4406   158  446   509 12 10  0 77
 1  0  68700   7048   3744 285288    0    0  2023    14  433   554  6 15  0 78
 0  1  68700   5708   4228 285848    5    0  1833     9  401   544  7 14  0 79
 0  1  68700   5972   4336 284888    0    0  2396    13  443   640  6 12  0 82
 2  1  68700   6224   3592 284480  145    0  2652     8  419   558  8 14  0 78
 0  2  68368   6284   3920 283220   14    0  2418    19  413   524  5 10  0 85
 0  1  68368   6612   3284 283408   27    0  2166    18  422   564  7 15  0 78
 4  1  68368   5160   3308 284396   45    0  2654   158  534   724 13 14  0 73
 1  2  68368   5680   3548 283856    0    0  1983    14  431   579  4 10  0 86
 1  2  68368   4872   4808 282528    0    0  1676    15  500   772  3  7  0 90
 0  2  68368   5012   5108 280620    0    0  1883    15  529   779  4  7  0 89
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  1  68368   5204   2480 283340    0    0  2238   148  473   540  6  9  0 86
 0  2  68368   9236   1856 279660    0    0  2289    16  452   554  7 17  0 76
 1  1  75660   6572   2380 284684   70 1453  2144  1569  437   530 13 12  0 75
 1  2  77820   7436   3300 288556  108  432  1795   443  445   621  7 10  0 83
 1  1  91436   6404   2304 294856  105 2722  3682  2904  414   583  6  9  0 85
 0  2  95112   6376   3172 304564   54  731  2433   744  468   671  6  7  0 87
 0  2 107716   6612   2896 307048   32 2357  1912  2508  420   514  6  8  0 86
 0  1 107716   9788   3460 310060  271   13  2433    32  485   624 13 11  0 77
 1  1 107716  14764   4200 303920   30  191  2025   219  444   601  5  9  0 85
 3  0 107716   9780   2496 311956   62    0  6454   155  434   562 17 11  0 72
 2  2 107716   5244   2788 316508   42    0  2165    20  445   568  7 13  0 80
 1  1 107716   6256   3992 313108  114    0  1015    39  445   641  4 10  0 86
 5  0 107716   5932   4972 312024    6    0  1411    21  424   542  7 15  0 78
 0  1 107716   6616   5504 310784   12    0  2052    11  420   538  8 20  0 72
 8  1 107716   5464   5704 311064   18    0  2783   171  436   555  9 12  0 79
 1  1 107716   7372   4172 309104   15    0  1970   166  443   511 12 12  0 75
 8  0 107716   6136   3896 310768   57    0  2810    25  497   668  9 18  0 73
 0  2 107716   9500   4084 307548   19    0  2343    11  447   622  5  8  0 87
 0  2 107716   6416   3880 308504  402    0  3497     6  463   636  7  8  0 86
 1  1 107716   6256   2668 309544    6    0  2267    28  428   554  6 11  0 83
 1  2 107684   7452   3236 306504   74    0  1553    39  443   570  8 13  0 79
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  1 107684   6244   3872 307488    0    0  2003    10  502   698  7 13  0 80
 1  0 107684   9580   3740 304176    6    0  1265   163  465   666 10  4  0 86
 0  3 107684   6488   4808 305624   11    0  1057    25  497   741  4  6  0 90
 0  2 107684   5152   3104 308540    0    0  2937    32  485   731  3  9  0 87
 0  2 107676   6408   2200 308032    0    0  1898     9  491   731  2  6  0 92
 1  2 107644   6788   2232 308596    0    0  2660   175  505   653  4  8  0 88
 0  3 107644   5476   2600 309796    0    0  2472    10  448   509 12 11  0 77
 2  2 107644   6504   3076 307640   51    0  1821     4  450   580 13 11  0 77
 1  3 107644   5236   3212 308676    0    0  1884   248  467   634  4  9  0 87
 0  3 111752   5932   1212 312924    0  823  8258   857  412   496  7  8  0 85
 0  2 114556   5952   1988 317336   38  610  4539   618  524   542  4  8  0 88
 0  2 114892   5636   2352 317520  222  168  2421   182  467   566  5 10  0 85
 2  2 114892   9376   2916 312680    0    8  2557   296  492   593  6 16  0 78
 0  2 114892   5628   3272 316500   11    0  1875     9  393   490  8 16  0 76
 0  4 114876  15864   1868 308220  106    0  3600    48  439   531 13 10  0 77
 2  2 114876   5912   2324 317748    0    0  3231    19  496   578  7 12  0 81
 0  1 114876  12212   1736 312196    0    0  3968   156  458   642  4  5  0 92
 0  1 114828   5932   2772 317052    0    0  2401    17  481   562  5 12  0 83
 1  1 114828   8980   2716 313344    3    0  2818   145  427   525  6 18  0 76
 0  1 114828   7196   1876 315940    6    0  2419   143  428   515  5 15  0 79
 0  1 114828  13676   1632 310008    0    0  2838    15  484   671 21 10  0 69
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1 114828   6020   2640 316692    0    0  1875   139  477   671  3  5  0 93
 1  1 114828   8344   2780 313900    0    0  2409     9  508   597  3  9  0 88
 1  1 114828  10548   2340 311904    0    0  2403    16  465   607  6  9  0 85
 0  3 114828  11516   1588 310816    0    0  3214   144  533   642  3 11  0 86
 0  1 118216  11156   1532 311396   13  858  2947  1007  463   561  3  7  0 89
 3  0 118216   5628   2296 320392    0    0  1958   155  455   513 12 10  0 77
 2  1 118316   9044   1632 317676   27   20  3446    31  426   505 10 13  0 77
 0  1 118316  10204   1728 316808   19    0  3636   146  466   573  3  7  0 90
 0  2 118316   5232   2016 320572   25    0  2235   148  517   616  3  8  0 89
 1  2 118316  12040   2524 312484   75    0  1914   237  506   631  5  9  0 86
 0  1 129564   5744   3132 326928    4 2322  1504  2383  530   723  3 17  0 80
 2  2 145008   7172   3736 332112   13 3121  2155  3269  474   484  4 14  0 82
 7  0 145008   6192   4564 339080  111    0  2174    13  474   481 11 11  0 77
 0  3 145008   4964   3360 341816  103  145  2429   176  487   655  7 12  0 82
 1  2 145000   7592   4044 337696   38    0  1223   144  443   598  4  8  0 88
 4  0 145000   8824   2240 338332   38    0  3140    31  567   847  5 12  0 83
 2  1 145000   5888   4004 337880   29    0  1647    11  522   831  3 10  0 87
 1  1 146644   5588   3508 340152   42  521  1317   532  490   668  3  8  0 89
 1  1 149216   6700   1340 343796   34  543  3430   700  490   531  4 10  0 86
 0  2 149216   5132   1628 345248   46    0  3155   165  493   600 13  9  0 77
 3  2 149216   6428   2268 343724   22    0  2737   148  467   577  4  8  0 87
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  3 149216   5420   1396 345424   12    0  4620    14  571   540  5 12  0 83
 1  1 149216   9320   1384 341584   31    0  3746   150  500   550  5  9  0 86
 0  1 150404   5692   2932 346148   38  366  1736   376  475   621  3  7  0 90
 1  2 155024   4828   3704 351500   22 1214  1851  1222  482   607  5 11  0 85
 0  3 155024   5176   1688 351952  306    0  3980    20  499   590 13  8  0 79
 0  2 155024   7440   1972 346508  560    0  2552     7  442   466  4 12  0 84
 2  3 160384   6404   1488 344528 1704 1126  4411  1140  535   550  2 11  0 87
 0  2 165320   6364   2004 348096  417 1024  2608  1049  464   581 16 13  0 71
 2  9 165320   7936   3956 341600  371   34  2405   215  686  1127  5 11  0 84
 0  1 165320  10836   1612 341536  286    0  2974   165  503   673  8 14  0 78
 0  4 178592  10248   1612 342544  113 3203  2339  3480  484   483  3  8  0 88
 0  3 179000   5724   2444 360720  157  486  3948   498  516   587  5 10  0 85
 2  0 179476   6360   2912 362088  191  446  2384   481  426   507  6 15  0 79
 0  1 179476   5768   3156 363384    6    0  3091   177  479   592  6 12  0 82
 0  3 179476   6516   2740 360080  758    0  2864    19  413   477  7 19  0 73
 1  4 179476  16876   1960 346940  927    0  3272    20  444   445  4  9  0 87
 0  3 179476   7268   2064 358960  361    0  3637    14  495   421  2  5  0 94
 1  6 184984  20480   1336 343340 1442 2018  4298  2069  567   437  2 11  0 87
 2  4 184940   5964   1836 349220 1832    0  3067    10  489   473  1  2  0 97
 1  3 188860   6380   2168 355760  693  998  2519  1010  431   401  1  4  0 95
 1  5 188860   5140   2200 356168  596   68  3918   171  479   424  2  5  0 93
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1 10 188860  13248   1436 348680  711    0  5342   142  533   418  2  8  0 90
 0  2 129288  28484   1748 376972  449    0  6115    11  455   399  3  4  0 94
 0  4 129288  11804   2440 390368  525    0  3137    23  450   439  2  2  0 96
 1  2 129288   6232   2896 393752  157    0  4576   154  472   422  4  8  0 88
 0  4 129288   7288   3244 388452   69    0  3231    28  442   432  6  5  0 89
 1  4 129288   6284   3288 385520  648    0  5064   154  476   447  3  6  0 91
 0  3 129288  12344   3024 379112  406    0  3971    18  432   443  3  6  0 91
 1  3 129288   5048   2200 387492    5    0  5822   152  470   459  5  4  0 91
 0  6 129288  10612   2196 380552  261    0  4587    26  469   557  4  6  0 90
 0  4 129288   6188   3636 379296  546    0  2948     4  495   633  2  6  0 92
 3  2 129288   5200   4076 374308    0    0  2775    57  504   629 17  8  0 75
 0  3 129300  12860   1212 368272    0   86  3277   109  523   683 10 12  0 77
 0  7 131620   5848   1484 381180   28 1254  3532  1264  459   449  3  4  0 93
 0  3 134988   5760   1716 387916  152  901  1874   907  422   390  1  3  0 96
 0  5 135060   5668   3392 384512  319   33  4021    38  508   640  3  8  0 89
 0  4 135148   8252   3892 384572  152  456  2769   461  455   434  1  5  0 94
 0  4 135148   5928   2492 385356  439    0  3642     6  457   521  3  5  0 92
 0  3 135148   8728   1840 382620  337    0  3445   242  485   500  4  7  0 89
 1  4 135148   7368   1720 382348  212    0  3661   189  447   447  5  7  0 88
 0  5 135148  11452   1456 379176  240    0  3548   120  457   480  2  5  0 94
 2  5 135220   6084   1344 382488  479   96  4875   371  519   388  1  7  0 92
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  6 136072  14992   1292 375240  413  286  3405   292  498   415  1  6  0 93
 1  5 139580  13028   1592 380120  255 1215  3509  1216  479   415  1  4  0 95
 1  4 143036   5856   2000 392696  215  734  2893   741  459   439  2  4  0 94
 1  3 142944  11244   1540 385200  830    0  5063    12  510   451  1  6  0 92
 0  4 142944   7376   2024 387448  383    0  3758    21  476   477  3  6  0 92
 5  3 142944   9592   1476 385932  418    0  4704   281  506   444  3  8  0 88
 0  4 142944   4984   1240 390836  120    0  7614    18  460   450  6 10  0 84
 0  3 144816   7232   1460 390536   75  435  6089   710  518   476  3 11  0 86
 1  3 144816  11148   1364 386208  220    0  4383    12  493   475  4  7  0 89
 1  3 144816   6980   1356 389628  120    0  7460     9  465   443  3  6  0 91
 0  3 135236   5460   1492 393056  997    0  4203     1  479   474  1  4  0 95
 2  1 135236  12820   1196 380608  494    0  5389    81  484   437 12  5  0 83
 0  2 135236   7660   1236 379816  288    9  2314   133  427   632 21  4  0 74
 1  2 135792  10668   1116 375108   64  285  2584   314  450   514 12  8  0 80
 1  3 137580   6368   1252 381388   77  550  1674   562  420   471  8  3  0 88
 3  3 137688   5528   1280 382900  186   35  2373    43  490   519  4  3  0 93
 1  3 137688   6716   1272 381120  123    0  1849    11  459   552  4  2  0 94
 1  4 137688   5212   1228 381016  258    0  2119     9  481   545  3  2  0 95
 0  3 137688   5652   1232 380476  209    0  2296    12  468   536  3  3  0 94
 0  3 137688   7840   1184 376508  440    0  2768     7  482   537  3  4  0 93
 1  3 137640   5508   1308 379116   23    0  1742    17  471   532  2  3  0 95
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  3 141812   8940   1192 375624   46  878  1044   881  443   547  3  3  0 94
 2  2 141812   5740   1240 380424   64    0  1032     9  441   543  6  1  0 93
 3  3 141812   6488   1204 379116  132    0  1419    12  433   502 12  6  0 82
 1  3 141812   6984   1256 378996   61    0  2067    14  451   551  2  3  0 95
 0  4 141812   5436   1236 380220   59    0  1207    11  458   570  4  3  0 94
 1  2 141656   7140   1256 377964  122    0  1647    12  459   544  1  3  0 96
 3  1 141656   6520   1220 378380   38    0  1678    19  451   559  3  3  0 94
 0  3 141656   9060   1176 375664   74    0  1402    17  433   547  1  3  0 96
 0  2 141656   6548   1188 377876   65    0  1254    13  447   568  3  2  0 95
 0  3 141656   5284   1164 378924   25    0  1149     9  440   567  4  2  0 94
 0  2 141656   7308   1164 376756   23    0  1134    11  444   563  3  3  0 94
 2  2 145844   6332   1196 379020   29 1175   823  1184  436   491  2  3  0 96
 0  4 145860   5192   1260 382988  194    9  1189    19  443   528  7  3  0 90
 1  2 145860   6344   1224 379832  526    0  2200    11  427   527  2  3  0 95
 0  2 145860   6688   1212 380036   80    0  1108    13  453   582  4  3  0 93
 1  3 145860   5888   1176 380024  112    0   905    11  430   554  6  2  0 92
 1  3 145860   6076   1200 379068  150    0  1168    12  447   552  3  3  0 93
 1  4 145820   5444   1572 378508   67    0  1361     9  470   554  3  3  0 94
 0  2 145820   5880   1500 378328    4    0  1509    26  442   568  2  4  0 94
 2  3 145820   5088   1324 379244    2    0  1048    10  439   575  3  2  0 95
 0  2 149264   9216   1264 375084   23  910   781   927  436   470  2  4  0 95
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  2 149264   5976   1376 380968   77    0  1258     7  462   569  3  1  0 96
 1  2 149264   5156   1264 381316  191    0  1141    10  458   583  6  3  0 92
 3  1 151708   6296   1336 382808  217  735   830   746  435   461  4  3  0 93
 0  2 151708   7428   1256 381612  270    0  1657    16  442   513  6  3  0 91
 9  4 151708  11412   1148 373848 1302    0  4791    12  487   457  3  8  0 90
 1  4 151708   9100   1280 374496  708    0  4761    43  487   395  0 16  0 84
 2  7 155120   6404   1328 379824  701  850  4039   868  462   394  0  6  0 94
 1  9 157332   6028   1416 381316  379  773  2296   780  464   411  1  2  0 97
 1  7 162144   5800   1488 386068  682 1418  2066  1432  429   374  0  2  0 98
 1  3 162084  15036   1464 382352  501    0  4914    10  485   384  1  4  0 96
 1  2 162832  15412   1400 375144 1890  298  2765   353  492   446  1  2  0 97
 0  3 162832   4868   1180 381508  905    0  5570    17  486   446  6  5  0 88
 7  5 162832   9408   1172 375292  791    0  4077   110  500   500  3  6  0 92
 3  1 164976   5388   1208 378964  150  693  3724   714  442   389  4  6  0 90
 0  3 165976   5504   1260 379144  543  406  3707   456  468   408  1  6  0 93
 2  1 168148   7420   1240 377348  378  899  3979   932  462   415  1  7  0 92
 0  6 168148   5400   1244 381824  535   34  3930    41  451   353  0  4  0 96
 3  4 168148   6052   1264 381436  368    0  4577     8  494   364  1  4  0 96
 0  3 168044   6328   1308 379864  682    0  5438     9  491   417  1  4  0 94
 0  8 173256   6304   1248 380252  398 1447  3585  1459  499   414  1  6  0 94
 0  5 173204  11776   1348 379592  426    0  2335     6  466   384  1  2  0 97
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  7 173740  17092   1176 374852  813  481  5263   522  497   449  2  7  0 91
 0  4 177652   7972   1316 384596  274  934  2411   945  453   380  1  2  0 98
 0  7 180284   5408   1428 391244 1139  756  2901   769  506   483  1  3  0 96
 1  2 180284   7356   1360 386968  970    1  5561    22  491   524  5  3  0 92
 2  1 180284  14348   1304 377388 1193    0  3551    30  489   655 15  5  0 80
 2  2 180452   7892   1232 377548 2763  106  4475   136  558   533  1  4  0 96
 1  2 181920   5460   1224 377828 1624  677  2578   689  533   461  0  4  0 96
 0  4 182368   4956   1212 376308 1209  222  4513   236  509   447  1  7  0 92
 0  8 182856   8292   1228 375160  590  296  4027   313  620   541  0 10  0 89
 0 11 183652   6068   1308 374348  758  258  5703   281  544   460  0  8  0 91
 0 18 183936   5548   1324 380768  348 1819  2068  1823  520   385  0  3  0 97
 0 14 184264   5216   1436 385292  239  804  1343   808  441   385  0  1  0 99
 0 13 185220   5124   1508 389880  306 1182  1412  1185  435   379  0  1  0 99
 1  8 185720   5452   1628 393440  276  847  1207   848  432   380  0  1  0 99
 0  2 186348   5680   1684 396396  437  877  1138   881  432   403  0  1  0 99
 0  1 186360   6212   1604 388984 1954   43  3533    51  502   405  0  2  0 97
 0  2 186360   5096   1148 373472 5382    0  6009    17  703   839  2  5  0 94
 0  2 186528   9312   1180 376012  189  498  4450   509  492   319  1  8  0 92
 0  2 186552   6068   1264 378496  341   53  6235    61  494   370  0  5  0 95
 0  3 186552   8168   1168 371564 1699  126  5373   137  591   473  2  6  0 92
 0  7 186556   6912   1176 373708  736  302  6215   313  506   423  0  8  0 92
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1 16 186664   5952   1172 373924 1318 1937  8545  1964  875   750  0 10  0 90
 1 21 187840   8392   1344 382380  274 1131  1674  1132  801   357  0  5  0 95
 1 17 187844   7164   1428 385808  879 1120  2509  1122  452   356  0  1  0 99
 0 12 187844  13980   1396 381308  565  460  2919   460  444   305  0  4  0 96
 2  2 187844  12908   1368 374748 2693    0  5121     7  508   507  0  3  0 97
 0  3 187864   6620   1336 380064 1452  386  4387   395  450   396  1  4  0 96
 0  2 187864   9496   1284 376252 1416    0  4403     7  464   420  0  3  0 97
 2  3 187864   5840   1236 376328 1860    0  4385    26  476   509  1  4  0 95
 1  3 187864   5508   1208 376492 1230   27  4835    43  469   425  0  5  0 95
 0  3 187884   8156   1188 371684 1499  317  4517   325  535   457  0  6  0 94
 0  9 187920   6172   1232 373216 1359 1162  4482  1167  655   563  0  6  0 93
 0  8 187920   7816   1204 374276 1038 1803  6959  1807  550   441  0  6  0 93
 0  8 187920   5880   1336 386836  894 1761  3179  1761  763   559  0  3  0 97
 1  6 187920   6212   1428 390828 1630 1087  2732  1089  464   398  0  3  0 97
 1  5 187768   6468   1480 382820 2047    0  5389     4  477   462  0  4  0 96
 0  4 187768   7424   1328 378128 2008    0  4948    12  467   468  0  4  0 96
 1  4 187768  11104   1232 375308 1587    0  4737     6  476   443  0  4  0 96
 0  2 187716   6444   1272 378840 1687    0  4201    26  471   446  0  4  0 96
 0 27 187716   6404   1180 377732 1980    0  5786    22  536   500  0  7  0 92
 0 30 187752  15116   1412 379172  633 1651  3689  1668  523   438  0  7  0 93
 0 26 187756  19548   1464 376068  820   91  3885    91  622   589  0  3  0 97
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1 16 187756   5320   1684 384800 1182    0  2946     6  453   473  0  1  0 98
 1 15 187756   6428   1716 380392 1182    0  3135    38  450   484  2  4  0 94
 0 11 187756   5020   1508 383044  853    0  4106    28  451   440  0  3  0 96
 0 17 171176   5400   1520 385240  903    0  4191    19  447   390  0  4  0 96
 2 16 171056   9388   1456 380620  490  158  3684   173  451   357  0  5  0 95
 0 10 171056   6464   1512 384828  526    0  3919    20  439   308  0  3  0 97
 1 18 171312   5936   1356 381996  541  294  4295   339  461   345  1  5  0 95
 0 13 170796   6364   1496 387708  450  678  2102   680  441   297  0  2  0 98
 0 31 170796   8020   1488 383504 1046   62  2833    64  445   397  0  2  0 97
 0 10 170280   8580   1508 385948  628    0  3747     6  428   312  0  3  0 96
 1 18 169864  10348   1540 379860 1230    0  3646    31  437   391  1  4  0 95
 0 27 169900  11592   1428 378688  840  146  3733   173  449   347  0  4  0 96
 0 26 169772   6732   1476 382676  706    0  3210     6  463   354  0  2  0 97
 0 24 170372   7036   1400 383996  698  919  2596   931  441   353  0  5  0 95
 0 20  99548   6412   1728 393476  830   78  2803    84  452   387  1  2  0 97
 0 13  82212   6616   1908 397108  958  313  1906   322  447   349  0  2  0 97
 0  8  72732   8072   2152 398500  826   12  2248    24  428   364  1  2  0 97
 0  6  66136  16176   2260 391552  567    0  3060    44  412   367  1  4  0 95
 0  7  66084  11384   2348 394576  588    0  4641    22  446   347  1  5  0 94
 1  8  66084   5496   1800 398888  422    0  6239   428  448   320  1  5  0 94
 1  6  66088   5472   1676 398864  412    0  4112   184  439   293  0  3  0 97
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  5  60468  10752   1608 397652  787    1  4840   605  425   287 37 49  0 14
 1  1  32236  11324   1848 402152  284    0  1891    16  388   262 45 55  0  0
 2  1  32216   9504   2016 402372    2    0    48    25  281   145 47 53  0  0
 4  1  32216   6548   1956 400708    6    0  2164    14  328   197 45 55  0  0
 5  2  32216   5288   1892 400600    0    0  1474     8  311   201 45 55  0  0
 1  5  30188   6944   2328 402808  466    0  1175    22  385   394 42 58  0  0
 1  1  28248  10000   2524 397492   71    0   273    28  292   286 47 53  0  0
 1  1  28248  10016   2532 397492    0    0     0     5  254   141 42 58  0  0
 1  1  28136  10916   2612 397712   49    0    99    61  288   180 41 59  0  0
 1  1  28136  10964   2628 397712    0    0     0     6  254    97 40 60  0  0
 2  1  28136  11144   2632 397712    0    0     0     1  252    92 41 59  0  0
 2  1  28136  11168   2636 397712    0    0     0     1  253    95 39 61  0  0
 2  1  28136  11184   2640 397712    0    0     0     2  252    93 40 60  0  0
 1  1  28136  11200   2644 397712    0    0     0     1  252    95 40 60  0  0
 1  1  28136  11208   2648 397712    0    0     0     1  254    99 44 56  0  0
 3  1  28136  11208   2652 397712    0    0     0     1  251    92 43 57  0  0
 1  1  28136  11208   2656 397712    0    0     0     6  256    92 41 59  0  0
 1  1  28136  11208   2660 397712    0    0     0     1  251    93 37 63  0  0
 1  1  28136  11208   2664 397712    0    0     0     1  252    95 38 62  0  0
 1  1  28136  11208   2668 397712    0    0     0     1  251    93 39 61  0  0
 1  1  28128  10892   2684 397972   74    0   126     1  265   121 42 58  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  1  28128  10892   2692 397972    0    0     0     4  253    96 39 61  0  0
 1  1  28128  10908   2696 397972    0    0     0     1  253    93 43 57  0  0
 1  1  28084  10720   2704 398188    6    0    50     2  263   123 41 59  0  0
 1  1  28076  10720   2716 398188    3    0     3     3  257   111 41 59  0  0
 1  1  28076  10720   2724 398188    4    0     4     3  256   106 44 56  0  0
 1  1  28064  33288   1300 376968    1    0   190     1  283   119 33 67  0  0
 2  1  27500  33304   1356 377192  119    0   166     5  270   130 41 59  0  0

[root@antares test4]# date
ven aoû 11 10:50:10 CEST 2006

[root@antares test4]# cat /proc/memeinfo

MemTotal:       515032 kB
MemFree:          7756 kB
Buffers:          1348 kB
Cached:         376276 kB
SwapCached:      14852 kB
Active:          38408 kB
Inactive:       374000 kB
SwapTotal:     1116428 kB
SwapFree:       967164 kB
Dirty:              64 kB
Writeback:           0 kB
AnonPages:       30164 kB
Mapped:           7028 kB
Slab:            84120 kB
PageTables:       1956 kB
NFS Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   1373944 kB
Committed_AS:   493124 kB
VmallocTotal:   515796 kB
VmallocUsed:      6860 kB
VmallocChunk:   508788 kB

[root@antares test4]# echo 3 > /proc/sys/vm/drop_caches

[root@antares test4]# cat /proc/meminfo

MemTotal:       515032 kB
MemFree:          6000 kB
Buffers:          1336 kB
Cached:         380164 kB
SwapCached:      12680 kB
Active:          40532 kB
Inactive:       373024 kB
SwapTotal:     1116428 kB
SwapFree:       964720 kB
Dirty:              12 kB
Writeback:           0 kB
AnonPages:       31600 kB
Mapped:           7676 kB
Slab:            84016 kB
PageTables:       1960 kB
NFS Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   1373944 kB
Committed_AS:   493132 kB
VmallocTotal:   515796 kB
VmallocUsed:      6860 kB
VmallocChunk:   508788 kB

[root@antares test4]# date
ven aoû 11 10:50:56 CEST 2006

[root@antares test4]# vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  7 162140   5848   1572 390100   85  127  2205   213  418   514  9  8 22 61
 0  4 162084  16916   1428 380376  481    0  5819     8  490   388  1  5  0 95
 0  2 162084   5292   1588 386112 2135    0  4054    53  524   494  1  2  0 98
 0  1 162832  11388   1372 375764 1424  497  1963   540  462   395  1  3  0 96

I spent some time looking over what I thought was the obvious candidate, but alas, no cigar. Not surprising since Andrew can't reproduce it.

Mine is the plainest ext3 config imaginable.

It seems odd at the moment.

-Mike

I get no oom-killer action, but as soon as memory gets tight, I get something even more effective. rpm -qaV reliably emits the below.

kernel BUG at mm/vmscan.c:383!
invalid opcode: 0000 [#1]
4K_STACKS PREEMPT SMP
last sysfs file: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/resource

Modules linked in: xt_pkttype ipt_LOG xt_limit snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event eeprom snd_seq edd ip6t_REJECT xt_tcpudp ipt_REJECT xt_state iptable_mangle iptable_nat ip_nat iptable_filter ip6table_mangle ip_conntrack nfnetlink ip_tables ip6table_filter ip6_tables x_tables nls_iso8859_1 nls_cp437 nls_utf8 saa7134_dvb mt352 video_buf_dvb nxt200x tda1004x tuner snd_intel8x0 snd_ac97_codec snd_ac97_bus snd_pcm snd_timer sd_mod saa7134 bt878 i2c_i801 snd_page_alloc prism54 ir_kbd_i2c bttv video_buf ir_common ohci1394 snd_mpu401 snd_mpu401_uart btcx_risc tveeprom ieee1394 snd_rawmidi snd_seq_device snd soundcore
CPU:    1
EIP:    0060:[<c105a166>]    Not tainted VLI
EFLAGS: 00210203   (2.6.18-rc3-mm2-smp #162) 

EIP is at remove_mapping+0xa3/0xbf
eax: 80008009 ebx: c1e48200 ecx: c14ad9c0 edx: c1e48200
esi: c1e48200 edi: c14ad9c0 ebp: dffb7e14 esp: dffb7e08
ds: 007b es: 007b ss: 0068
Process kswapd0 (pid: 196, ti=dffb7000 task=dffb9a90 task.ti=dffb7000)

Stack: c1e48200 c1e48218 c14ad9c0 dffb7f28 c105a818 dffb7f18 00000000 dffb7f08 
       dffb7f10 c14ad680 c14ad690 dffb7f84 c14ad100 00000020 00000000 00000000 
       00000020 00000000 00000000 c14ad9c0 00000000 00000020 00000000 00000001 
Call Trace:
 [<c105a818>] shrink_inactive_list+0x696/0x8dc
 [<c105aaf0>] shrink_zone+0x92/0xe5
 [<c105b125>] kswapd+0x300/0x40e
 [<c10361d6>] kthread+0xe4/0xe8
 [<c1001005>] kernel_thread_helper+0x5/0xb
DWARF2 unwinder stuck at kernel_thread_helper+0x5/0xb
Leftover inexact backtrace:
 [<c1003f83>] show_stack_log_lvl+0xa6/0xcb
 [<c1004180>] show_registers+0x1d8/0x286
 [<c100437f>] die+0x151/0x333
 [<c10045d9>] do_trap+0x78/0xa3
 [<c1004f16>] do_invalid_op+0x97/0xa1
 [<c13e0369>] error_code+0x39/0x40
 [<c105a818>] shrink_inactive_list+0x696/0x8dc
 [<c105aaf0>] shrink_zone+0x92/0xe5
 [<c105b125>] kswapd+0x300/0x40e
 [<c10361d6>] kthread+0xe4/0xe8
 [<c1001005>] kernel_thread_helper+0x5/0xb
Code: f0 e8 46 88 ff ff 89 f8 e8 ba 5d 38 00 f0 ff 4e 04 b8 01 00 00 00 5b 5e 5f 5d c3 89 f8 e8 a5 5d 38 00 31 c0 eb d4 8b 56 0c eb 8d <0f> 0b 7f 01 6f 75 42 c1 89 f6 e9 6b ff ff ff 0f 0b 7e 01 6f 75 
EIP: [<c105a166>] remove_mapping+0xa3/0xbf SS:ESP 0068:dffb7e08
 <0>------------[ cut here ]------------

kernel BUG at mm/vmscan.c:383!
invalid opcode: 0000 [#2]
4K_STACKS PREEMPT SMP
last sysfs file: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/resource

Modules linked in: xt_pkttype ipt_LOG xt_limit snd_pcm_oss snd_mixer_oss snd_seq_midi snd_seq_midi_event eeprom snd_seq edd ip6t_REJECT xt_tcpudp ipt_REJECT xt_state iptable_mangle iptable_nat ip_nat iptable_filter ip6table_mangle ip_conntrack nfnetlink ip_tables ip6table_filter ip6_tables x_tables nls_iso8859_1 nls_cp437 nls_utf8 saa7134_dvb mt352 video_buf_dvb nxt200x tda1004x tuner snd_intel8x0 snd_ac97_codec snd_ac97_bus snd_pcm snd_timer sd_mod saa7134 bt878 i2c_i801 snd_page_alloc prism54 ir_kbd_i2c bttv video_buf ir_common ohci1394 snd_mpu401 snd_mpu401_uart btcx_risc tveeprom ieee1394 snd_rawmidi snd_seq_device snd soundcore
CPU:    0
EIP:    0060:[<c105a166>]    Not tainted VLI
EFLAGS: 00210203   (2.6.18-rc3-mm2-smp #162) 

EIP is at remove_mapping+0xa3/0xbf
eax: 80008009 ebx: c1e784a0 ecx: c14ad9c0 edx: c1e784a0
esi: c1e784a0 edi: c14ad9c0 ebp: dfda3ba0 esp: dfda3b94
ds: 007b es: 007b ss: 0068
Process rpm (pid: 6150, ti=dfda3000 task=dffca560 task.ti=dfda3000)

Stack: c1e784a0 c1e784b8 c14ad9c0 dfda3cb4 c105a818 dfda3ca4 00000008 dfda3c94 
       dfda3c9c c14ad680 c14ad690 dfda3cf4 c14ad100 00000020 00000000 00000000 
       00000020 00000000 00000000 c14ad9c0 00000000 00000020 00000000 00000001 
Call Trace:
 [<c105a818>] shrink_inactive_list+0x696/0x8dc
 [<c105aaf0>] shrink_zone+0x92/0xe5
 [<c105b68b>] try_to_free_pages+0x157/0x254
 [<c1055c9b>] __alloc_pages+0x155/0x2b4
 [<c1057595>] __do_page_cache_readahead+0x120/0x2a3
 [<c1057806>] ra_dispatch+0xee/0x100
 [<c1057d83>] page_cache_readahead_adaptive+0x3f4/0xb77
 [<c105349e>] filemap_nopage+0x41d/0x4ad
 [<c105e80d>] __handle_mm_fault+0x12e/0x8fb
 [<c101966a>] do_page_fault+0xdc/0x51f
 [<c13e0369>] error_code+0x39/0x40
 [<b7bc89cf>] 0xb7bc89cf
 [<c1003f83>] show_stack_log_lvl+0xa6/0xcb
 [<c1004180>] show_registers+0x1d8/0x286
 [<c100437f>] die+0x151/0x333
 [<c10045d9>] do_trap+0x78/0xa3
 [<c1004f16>] do_invalid_op+0x97/0xa1
 [<c13e0369>] error_code+0x39/0x40
 [<c105a818>] shrink_inactive_list+0x696/0x8dc
 [<c105aaf0>] shrink_zone+0x92/0xe5
 [<c105b68b>] try_to_free_pages+0x157/0x254
 [<c1055c9b>] __alloc_pages+0x155/0x2b4
 [<c1057595>] __do_page_cache_readahead+0x120/0x2a3
 [<c1057806>] ra_dispatch+0xee/0x100
 [<c1057d83>] page_cache_readahead_adaptive+0x3f4/0xb77
 [<c105349e>] filemap_nopage+0x41d/0x4ad
 [<c105e80d>] __handle_mm_fault+0x12e/0x8fb
 [<c101966a>] do_page_fault+0xdc/0x51f
 [<c13e0369>] error_code+0x39/0x40
Code: f0 e8 46 88 ff ff 89 f8 e8 ba 5d 38 00 f0 ff 4e 04 b8 01 00 00 00 5b 5e 5f 5d c3 89 f8 e8 a5 5d 38 00 31 c0 eb d4 8b 56 0c eb 8d <0f> 0b 7f 01 6f 75 42 c1 89 f6 e9 6b ff ff ff 0f 0b 7e 01 6f 75 
EIP: [<c105a166>] remove_mapping+0xa3/0xbf SS:ESP 0068:dfda3b94

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc3/2.6.18-rc3-mm2/hot-fixes/ ;)

Duh, I should have thought to look there first. Sorry.

Anyhoo, I can reproduce the problem. I now have ~800MB of cache that echo 3 > drop_caches doesn't help with, and I just started swapping.

MemTotal:      1032656 kB
MemFree:         42704 kB
Buffers:           648 kB
Cached:         825468 kB
SwapCached:      29312 kB
Active:          31196 kB
Inactive:       830144 kB
HighTotal:      131008 kB
HighFree:         3056 kB
LowTotal:       901648 kB
LowFree:         39648 kB
SwapTotal:     1028152 kB
SwapFree:       961356 kB
Dirty:             156 kB
Writeback:           0 kB
AnonPages:       17240 kB
Mapped:          10240 kB
Slab:           118536 kB
PageTables:       1876 kB
NFS Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   1544480 kB
Committed_AS:   266884 kB
VmallocTotal:   114680 kB
VmallocUsed:      5372 kB
VmallocChunk:   109216 kB

A typo was pinning pagecache. Fixes leak encountered with rpm -qaV.

@@ -1498,7 +1498,7 @@ retry_find:
 			page_cache_readahead_adaptive(mapping, ra,
 						file, NULL, NULL,
 						pgoff, pgoff, pgoff + 1);
-			page = find_lock_page(mapping, pgoff);
+			page = find_get_page(mapping, pgoff);
 		} else if (PageReadahead(page)) {
 			page_cache_readahead_adaptive(mapping, ra,
 						file, NULL, page,

Problem fixed here too. Thanks

laurent

Hi Andrew,

This patch aims at removing two implementations (spotted by Masatake YAMATO) of pseudo-rwlocks using a spinlock_t and an atomic_t. One in net/socket.c and another in net/bluetooth/af_bluetooth.c. I think that both could be converted to rwsems, saving some lines of code.

Regards,
Frederik

 net/dccp/ccid.c |   63 ++++++++++++------------------------------------------------
 net/socket.c    |   58 +++++++------------------------------------------------
 2 files changed, 21 insertions(+), 100 deletions(-)

@@ -12,48 +12,11 @@
  */
 
 #include "ccid.h"
+#include <linux/rwsem.h>
 
 static struct ccid_operations *ccids[CCID_MAX];
-#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT)
-static atomic_t ccids_lockct = ATOMIC_INIT(0);
-static DEFINE_SPINLOCK(ccids_lock);
+static DECLARE_RWSEM(ccids_sem);
 
-/*
- * The strategy is: modifications ccids vector are short, do not sleep and
- * veeery rare, but read access should be free of any exclusive locks.
- */
-static void ccids_write_lock(void)
-{
-	spin_lock(&ccids_lock);
-	while (atomic_read(&ccids_lockct) != 0) {
-		spin_unlock(&ccids_lock);
-		yield();
-		spin_lock(&ccids_lock);
-	}
-}
-
-static inline void ccids_write_unlock(void)
-{
-	spin_unlock(&ccids_lock);
-}
-
-static inline void ccids_read_lock(void)
-{
-	atomic_inc(&ccids_lockct);
-	spin_unlock_wait(&ccids_lock);
-}
-
-static inline void ccids_read_unlock(void)
-{
-	atomic_dec(&ccids_lockct);
-}
-
-#else
-#define ccids_write_lock() do { } while(0)
-#define ccids_write_unlock() do { } while(0)
-#define ccids_read_lock() do { } while(0)
-#define ccids_read_unlock() do { } while(0)
-#endif
 
 static kmem_cache_t *ccid_kmem_cache_create(int obj_size, const char *fmt,...)
 {

@@ -103,13 +66,13 @@ int ccid_register(struct ccid_operations
 	if (ccid_ops->ccid_hc_tx_slab == NULL)
 		goto out_free_rx_slab;
 
-	ccids_write_lock();
+	down_write(&ccids_sem);
 	err = -EEXIST;
 	if (ccids[ccid_ops->ccid_id] == NULL) {
 		ccids[ccid_ops->ccid_id] = ccid_ops;
 		err = 0;
 	}
-	ccids_write_unlock();
+	up_write(&ccids_sem);
 	if (err != 0)
 		goto out_free_tx_slab;
 

@@ -131,9 +94,9 @@ EXPORT_SYMBOL_GPL(ccid_register);
 
 int ccid_unregister(struct ccid_operations *ccid_ops)
 {
-	ccids_write_lock();
+	down_write(&ccids_sem);
 	ccids[ccid_ops->ccid_id] = NULL;
-	ccids_write_unlock();
+	up_write(&ccids_sem);
 
 	ccid_kmem_cache_destroy(ccid_ops->ccid_hc_tx_slab);
 	ccid_ops->ccid_hc_tx_slab = NULL;

@@ -152,15 +115,15 @@ struct ccid *ccid_new(unsigned char id, 
 	struct ccid_operations *ccid_ops;
 	struct ccid *ccid = NULL;
 
-	ccids_read_lock();
+	down_read(&ccids_sem);
 #ifdef CONFIG_KMOD
 	if (ccids[id] == NULL) {
 		/* We only try to load if in process context */
-		ccids_read_unlock();
+		up_read(&ccids_sem);
 		if (gfp & GFP_ATOMIC)
 			goto out;
 		request_module("net-dccp-ccid-%d", id);
-		ccids_read_lock();
+		down_read(&ccids_sem);
 	}
 #endif
 	ccid_ops = ccids[id];

@@ -170,7 +133,7 @@ #endif
 	if (!try_module_get(ccid_ops->ccid_owner))
 		goto out_unlock;
 
-	ccids_read_unlock();
+	up_read(&ccids_sem);
 
 	ccid = kmem_cache_alloc(rx ? ccid_ops->ccid_hc_rx_slab :
 				     ccid_ops->ccid_hc_tx_slab, gfp);

@@ -191,7 +154,7 @@ #endif
 out:
 	return ccid;
 out_unlock:
-	ccids_read_unlock();
+	up_read(&ccids_sem);
 	goto out;
 out_free_ccid:
 	kmem_cache_free(rx ? ccid_ops->ccid_hc_rx_slab :

@@ -235,10 +198,10 @@ static void ccid_delete(struct ccid *cci
 			ccid_ops->ccid_hc_tx_exit(sk);
 		kmem_cache_free(ccid_ops->ccid_hc_tx_slab,  ccid);
 	}
-	ccids_read_lock();
+	down_read(&ccids_sem);
 	if (ccids[ccid_ops->ccid_id] != NULL)
 		module_put(ccid_ops->ccid_owner);
-	ccids_read_unlock();
+	up_read(&ccids_sem);
 }
 
 void ccid_hc_rx_delete(struct ccid *ccid, struct sock *sk)

From: Frederik Deweerdt <[email protected]>
Date: Thu, 10 Aug 2006 14:13:36 +0200

The net/socket.c one has been converted to RCU by Stephen Hemminger already.

If the bluetooth case is in an important code path it should use RCU as well.

Sorry, I made a mistake there: net/bluetooth/af_bluetooth.c should read net/dccp/ccid.c. Does your comment regarding af_bluetooth.c applies to ccid.c as well?
Also, is there a place where I can find Stephen Hemminger's work?
- Note, this is pure curiosity, it can wait a kernel release or two :) -

Thanks,
Frederik

Just hit this one upon shutdown (no traces logged before then):

INIT: Sending processes the TERM signal INITStopping clamd: [FAILED]
Starting killall: Stopping clamd: [FAILED]
[ OK ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Syncing hardware clock to system time
Turning off swap:
Unmounting file systems: umount2: Device or resource busy
umount: /var/www/html: device is busy
umount2: Device or resource busy
umount: /var/www/html: device is busy