Skip to contentSkip to conversation

memcg: panic when rmdir()

BUG New Li Zefan 3 participants 15 messages Jan 16, 2009

Subsystems (2)
Memory management MM swap
lore
Share & tools
lore

Li Zefan

author
#0 lore raw

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

# mount -t cgroup -o memory xxx /mnt
# mkdir /mnt/0
# for pid in `cat /mnt/tasks`; do echo $pid > /mnt/0/tasks; done
# echo "low limit" > /mnt/0/tasks
# do whatever to allocate some memory
# swapoff -a
killed (by OOM)
# for pid in `cat /mnt/0/tasks`; do echo $pid > /mnt/tasks; done
# rmdir /mnt/0
------------[ cut here ]------------
WARNING: at kernel/res_counter.c:71 res_counter_uncharge_locked+0x25/0x36()
Hardware name: Aspire SA85
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]
Pid: 2548, comm: rmdir Tainted: G        W  2.6.29-rc1-mm1 #4
Call Trace:
 [<c042ecb6>] warn_slowpath+0x79/0x8f
 [<c04496ce>] ? clockevents_program_event+0xe0/0xef
 [<c0463a1b>] ? res_counter_charge+0x35/0xb0
 [<c04639b0>] ? res_counter_uncharge+0x29/0x5f
 [<c0463941>] res_counter_uncharge_locked+0x25/0x36
 [<c04639ba>] res_counter_uncharge+0x33/0x5f
 [<c049b9ef>] mem_cgroup_force_empty+0x21b/0x498
 [<c049c82f>] mem_cgroup_pre_destroy+0x12/0x14
 [<c0460776>] cgroup_rmdir+0x5e/0x27e
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] sys_rmdir+0x15/0x17
 [<c0403485>] sysenter_do_call+0x12/0x35
---[ end trace 4eaa2a86a8e2da24 ]---
------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:2517!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/irq
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]

Pid: 2548, comm: rmdir Tainted: G W (2.6.29-rc1-mm1 #4) Aspire SA85
EIP: 0060:[<c04607f2>] EFLAGS: 00210046 CPU: 1
EIP is at cgroup_rmdir+0xda/0x27e
EAX: f442b800 EBX: ed00b3c0 ECX: c04607ca EDX: 00000000
ESI: c0778dc0 EDI: 00200246 EBP: ed252f30 ESP: ed252f14
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmdir (pid: 2548, ti=ed252000 task=e19aa8c0 task.ti=ed252000)

Stack:
 00000000 f442b800 c0621d08 e18a8014 e2d96a00 fffffff0 e2ccadf0 ed252f44
 c04a67fe e2d96a00 00000000 0804ca00 ed252fa8 c04a7b8c ed269080 e2d745a0
 00002121 00000001 f46a3000 00000000 00000000 00000000 ed0a6d34 00000004
Call Trace:
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] ? vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] ? do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] ? sys_rmdir+0x15/0x17
 [<c0403485>] ? sysenter_do_call+0x12/0x35
Code: c8 fe ff 8b 43 40 8b 70 0c eb 3e 8b 46 28 8b 44 83 20 89 45 e8 8b 55 e8 8b 52 04 83 fa 01 89 55 e4 0f 8f 5f ff ff ff 85 d2 75 04 <0f> 0b eb fe 8b 45 e4 31 c9 8b 55 e8 f0 0f b1 4a 04 8b 55 e4 39
EIP: [<c04607f2>] cgroup_rmdir+0xda/0x27e SS:ESP 0068:ed252f14
---[ end trace 4eaa2a86a8e2da25 ]---

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw
Quoted 6 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

Quoted 10 lines
# mount -t cgroup -o memory xxx /mnt
# mkdir /mnt/0
# for pid in `cat /mnt/tasks`; do echo $pid > /mnt/0/tasks; done
# echo "low limit" > /mnt/0/tasks
# do whatever to allocate some memory
# swapoff -a
killed (by OOM)
# for pid in `cat /mnt/0/tasks`; do echo $pid > /mnt/tasks; done
# rmdir /mnt/0

Isn't this a problem Nishimura fixed today ?

could you try

memcg-get-put-parents-at-create-free.patch

in mm-commits ?

Sorry for inconvinience, I'll send you the patch in private mail if necessary.

Thanks,
-Kame

Quoted 36 lines
------------[ cut here ]------------
WARNING: at kernel/res_counter.c:71 res_counter_uncharge_locked+0x25/0x36()
Hardware name: Aspire SA85
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]
Pid: 2548, comm: rmdir Tainted: G        W  2.6.29-rc1-mm1 #4
Call Trace:
 [<c042ecb6>] warn_slowpath+0x79/0x8f
 [<c04496ce>] ? clockevents_program_event+0xe0/0xef
 [<c0463a1b>] ? res_counter_charge+0x35/0xb0
 [<c04639b0>] ? res_counter_uncharge+0x29/0x5f
 [<c0463941>] res_counter_uncharge_locked+0x25/0x36
 [<c04639ba>] res_counter_uncharge+0x33/0x5f
 [<c049b9ef>] mem_cgroup_force_empty+0x21b/0x498
 [<c049c82f>] mem_cgroup_pre_destroy+0x12/0x14
 [<c0460776>] cgroup_rmdir+0x5e/0x27e
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] sys_rmdir+0x15/0x17
 [<c0403485>] sysenter_do_call+0x12/0x35
---[ end trace 4eaa2a86a8e2da24 ]---
------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:2517!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/irq
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]

Pid: 2548, comm: rmdir Tainted: G W (2.6.29-rc1-mm1 #4) Aspire SA85
EIP: 0060:[<c04607f2>] EFLAGS: 00210046 CPU: 1
EIP is at cgroup_rmdir+0xda/0x27e
EAX: f442b800 EBX: ed00b3c0 ECX: c04607ca EDX: 00000000
ESI: c0778dc0 EDI: 00200246 EBP: ed252f30 ESP: ed252f14
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmdir (pid: 2548, ti=ed252000 task=e19aa8c0 task.ti=ed252000)

Quoted 20 lines
Stack:
 00000000 f442b800 c0621d08 e18a8014 e2d96a00 fffffff0 e2ccadf0 ed252f44
 c04a67fe e2d96a00 00000000 0804ca00 ed252fa8 c04a7b8c ed269080 e2d745a0
 00002121 00000001 f46a3000 00000000 00000000 00000000 ed0a6d34 00000004
Call Trace:
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] ? vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] ? do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] ? sys_rmdir+0x15/0x17
 [<c0403485>] ? sysenter_do_call+0x12/0x35
Code: c8 fe ff 8b 43 40 8b 70 0c eb 3e 8b 46 28 8b 44 83 20 89 45 e8 8b 55 e8 8b 52 04 83 fa 01 89 55 e4 0f 8f 5f ff ff ff 85 d2 75 04 <0f> 0b eb fe 8b 45 e4 31 c9 8b 55 e8 f0 0f b1 4a 04 8b 55 e4 39
EIP: [<c04607f2>] cgroup_rmdir+0xda/0x27e SS:ESP 0068:ed252f14
---[ end trace 4eaa2a86a8e2da25 ]---

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#2 lore raw
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

Quoted 10 lines
# mount -t cgroup -o memory xxx /mnt
# mkdir /mnt/0
# for pid in `cat /mnt/tasks`; do echo $pid > /mnt/0/tasks; done
# echo "low limit" > /mnt/0/tasks
# do whatever to allocate some memory
# swapoff -a
killed (by OOM)
# for pid in `cat /mnt/0/tasks`; do echo $pid > /mnt/tasks; done
# rmdir /mnt/0
Quoted 2 lines

Isn't this a problem Nishimura fixed today ?

Are you sure?

The changelog:
==========
The lifetime of struct cgroup and struct mem_cgroup is different and mem_cgroup has its own reference count for handling references from swap_cgroup.

This causes strange problem that the parent mem_cgroup dies while child mem_cgroup alive, and this problem causes a bug in case of use_hierarchy==1 because res_counter_uncharge climbs up the tree.
==========

I was not using hierarchy, and no "mem_cgroup dies while child mem_cgroup alive" in my test.

Anyway, I'll try.

Quoted 70 lines

could you try

memcg-get-put-parents-at-create-free.patch

in mm-commits ?

Sorry for inconvinience, I'll send you the patch in private mail if necessary.

Thanks,
-Kame

Quoted 56 lines
------------[ cut here ]------------
WARNING: at kernel/res_counter.c:71 res_counter_uncharge_locked+0x25/0x36()
Hardware name: Aspire SA85
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]
Pid: 2548, comm: rmdir Tainted: G        W  2.6.29-rc1-mm1 #4
Call Trace:
 [<c042ecb6>] warn_slowpath+0x79/0x8f
 [<c04496ce>] ? clockevents_program_event+0xe0/0xef
 [<c0463a1b>] ? res_counter_charge+0x35/0xb0
 [<c04639b0>] ? res_counter_uncharge+0x29/0x5f
 [<c0463941>] res_counter_uncharge_locked+0x25/0x36
 [<c04639ba>] res_counter_uncharge+0x33/0x5f
 [<c049b9ef>] mem_cgroup_force_empty+0x21b/0x498
 [<c049c82f>] mem_cgroup_pre_destroy+0x12/0x14
 [<c0460776>] cgroup_rmdir+0x5e/0x27e
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] sys_rmdir+0x15/0x17
 [<c0403485>] sysenter_do_call+0x12/0x35
---[ end trace 4eaa2a86a8e2da24 ]---
------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:2517!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/irq
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]

Pid: 2548, comm: rmdir Tainted: G W (2.6.29-rc1-mm1 #4) Aspire SA85
EIP: 0060:[<c04607f2>] EFLAGS: 00210046 CPU: 1
EIP is at cgroup_rmdir+0xda/0x27e
EAX: f442b800 EBX: ed00b3c0 ECX: c04607ca EDX: 00000000
ESI: c0778dc0 EDI: 00200246 EBP: ed252f30 ESP: ed252f14
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmdir (pid: 2548, ti=ed252000 task=e19aa8c0 task.ti=ed252000)

Stack:
 00000000 f442b800 c0621d08 e18a8014 e2d96a00 fffffff0 e2ccadf0 ed252f44
 c04a67fe e2d96a00 00000000 0804ca00 ed252fa8 c04a7b8c ed269080 e2d745a0
 00002121 00000001 f46a3000 00000000 00000000 00000000 ed0a6d34 00000004
Call Trace:
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] ? vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] ? do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] ? sys_rmdir+0x15/0x17
 [<c0403485>] ? sysenter_do_call+0x12/0x35
Code: c8 fe ff 8b 43 40 8b 70 0c eb 3e 8b 46 28 8b 44 83 20 89 45 e8 8b 55 e8 8b 52 04 83 fa 01 89 55 e4 0f 8f 5f ff ff ff 85 d2 75 04 <0f> 0b eb fe 8b 45 e4 31 c9 8b 55 e8 f0 0f b1 4a 04 8b 55 e4 39
EIP: [<c04607f2>] cgroup_rmdir+0xda/0x27e SS:ESP 0068:ed252f14
---[ end trace 4eaa2a86a8e2da25 ]---

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 20 lines · On Fri, 16 Jan 2009 14:24:39 +0800 Li Zefan <[email protected]> wrote:
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

# mount -t cgroup -o memory xxx /mnt
# mkdir /mnt/0
# for pid in `cat /mnt/tasks`; do echo $pid > /mnt/0/tasks; done
# echo "low limit" > /mnt/0/tasks
# do whatever to allocate some memory
# swapoff -a
killed (by OOM)
# for pid in `cat /mnt/0/tasks`; do echo $pid > /mnt/tasks; done
# rmdir /mnt/0
Quoted 5 lines
Quoted 2 lines

Isn't this a problem Nishimura fixed today ?

Are you sure?

Sorry, I didn't see BUG! line in you log.

-Kame

Quoted 10 lines

The changelog:
==========
The lifetime of struct cgroup and struct mem_cgroup is different and mem_cgroup has its own reference count for handling references from swap_cgroup.

This causes strange problem that the parent mem_cgroup dies while child mem_cgroup alive, and this problem causes a bug in case of use_hierarchy==1 because res_counter_uncharge climbs up the tree.
==========

Quoted 79 lines

I was not using hierarchy, and no "mem_cgroup dies while child mem_cgroup alive" in my test.

Anyway, I'll try.

Quoted 70 lines

could you try

memcg-get-put-parents-at-create-free.patch

in mm-commits ?

Sorry for inconvinience, I'll send you the patch in private mail if necessary.

Thanks,
-Kame

Quoted 56 lines
------------[ cut here ]------------
WARNING: at kernel/res_counter.c:71 res_counter_uncharge_locked+0x25/0x36()
Hardware name: Aspire SA85
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]
Pid: 2548, comm: rmdir Tainted: G        W  2.6.29-rc1-mm1 #4
Call Trace:
 [<c042ecb6>] warn_slowpath+0x79/0x8f
 [<c04496ce>] ? clockevents_program_event+0xe0/0xef
 [<c0463a1b>] ? res_counter_charge+0x35/0xb0
 [<c04639b0>] ? res_counter_uncharge+0x29/0x5f
 [<c0463941>] res_counter_uncharge_locked+0x25/0x36
 [<c04639ba>] res_counter_uncharge+0x33/0x5f
 [<c049b9ef>] mem_cgroup_force_empty+0x21b/0x498
 [<c049c82f>] mem_cgroup_pre_destroy+0x12/0x14
 [<c0460776>] cgroup_rmdir+0x5e/0x27e
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] sys_rmdir+0x15/0x17
 [<c0403485>] sysenter_do_call+0x12/0x35
---[ end trace 4eaa2a86a8e2da24 ]---
------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:2517!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/irq
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]

Pid: 2548, comm: rmdir Tainted: G W (2.6.29-rc1-mm1 #4) Aspire SA85
EIP: 0060:[<c04607f2>] EFLAGS: 00210046 CPU: 1
EIP is at cgroup_rmdir+0xda/0x27e
EAX: f442b800 EBX: ed00b3c0 ECX: c04607ca EDX: 00000000
ESI: c0778dc0 EDI: 00200246 EBP: ed252f30 ESP: ed252f14
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmdir (pid: 2548, ti=ed252000 task=e19aa8c0 task.ti=ed252000)

Stack:
 00000000 f442b800 c0621d08 e18a8014 e2d96a00 fffffff0 e2ccadf0 ed252f44
 c04a67fe e2d96a00 00000000 0804ca00 ed252fa8 c04a7b8c ed269080 e2d745a0
 00002121 00000001 f46a3000 00000000 00000000 00000000 ed0a6d34 00000004
Call Trace:
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] ? vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] ? do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] ? sys_rmdir+0x15/0x17
 [<c0403485>] ? sysenter_do_call+0x12/0x35
Code: c8 fe ff 8b 43 40 8b 70 0c eb 3e 8b 46 28 8b 44 83 20 89 45 e8 8b 55 e8 8b 52 04 83 fa 01 89 55 e4 0f 8f 5f ff ff ff 85 d2 75 04 <0f> 0b eb fe 8b 45 e4 31 c9 8b 55 e8 f0 0f b1 4a 04 8b 55 e4 39
EIP: [<c04607f2>] cgroup_rmdir+0xda/0x27e SS:ESP 0068:ed252f14
---[ end trace 4eaa2a86a8e2da25 ]---

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#4 lore raw
Quoted 23 lines · KAMEZAWA Hiroyuki
Quoted 20 lines · On Fri, 16 Jan 2009 14:24:39 +0800 Li Zefan <[email protected]> wrote:
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

# mount -t cgroup -o memory xxx /mnt
# mkdir /mnt/0
# for pid in `cat /mnt/tasks`; do echo $pid > /mnt/0/tasks; done
# echo "low limit" > /mnt/0/tasks
# do whatever to allocate some memory
# swapoff -a
killed (by OOM)
# for pid in `cat /mnt/0/tasks`; do echo $pid > /mnt/tasks; done
# rmdir /mnt/0
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Isn't this a problem Nishimura fixed today ?

Are you sure?

Sorry, I didn't see BUG! line in you log.

I've tested with Nishimura's patch applied, and as is expected, this bug is totally different from the one Nishimura has fixed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw
In reply to Li Zefan
Quoted 6 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

Quoted 10 lines
# mount -t cgroup -o memory xxx /mnt
# mkdir /mnt/0
# for pid in `cat /mnt/tasks`; do echo $pid > /mnt/0/tasks; done
# echo "low limit" > /mnt/0/tasks
# do whatever to allocate some memory
# swapoff -a
killed (by OOM)
# for pid in `cat /mnt/0/tasks`; do echo $pid > /mnt/tasks; done
# rmdir /mnt/0

Hmm, it seems css->refcnt is bad (css->refcnt < 0). maybe css_put is not called without css_get().

will chase. thank you for testing.

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
In reply to Li Zefan
Quoted 52 lines · Li Zefan

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

# mount -t cgroup -o memory xxx /mnt
# mkdir /mnt/0
# for pid in `cat /mnt/tasks`; do echo $pid > /mnt/0/tasks; done
# echo "low limit" > /mnt/0/tasks
# do whatever to allocate some memory
# swapoff -a
killed (by OOM)
# for pid in `cat /mnt/0/tasks`; do echo $pid > /mnt/tasks; done
# rmdir /mnt/0
------------[ cut here ]------------
WARNING: at kernel/res_counter.c:71 res_counter_uncharge_locked+0x25/0x36()
Hardware name: Aspire SA85
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]
Pid: 2548, comm: rmdir Tainted: G        W  2.6.29-rc1-mm1 #4
Call Trace:
 [<c042ecb6>] warn_slowpath+0x79/0x8f
 [<c04496ce>] ? clockevents_program_event+0xe0/0xef
 [<c0463a1b>] ? res_counter_charge+0x35/0xb0
 [<c04639b0>] ? res_counter_uncharge+0x29/0x5f
 [<c0463941>] res_counter_uncharge_locked+0x25/0x36
 [<c04639ba>] res_counter_uncharge+0x33/0x5f
 [<c049b9ef>] mem_cgroup_force_empty+0x21b/0x498
 [<c049c82f>] mem_cgroup_pre_destroy+0x12/0x14
 [<c0460776>] cgroup_rmdir+0x5e/0x27e
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] sys_rmdir+0x15/0x17
 [<c0403485>] sysenter_do_call+0x12/0x35
---[ end trace 4eaa2a86a8e2da24 ]---
------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:2517!
invalid opcode: 0000 [#1] PREEMPT SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/irq
Modules linked in: bridge stp llc autofs4 dm_mirror dm_region_hash dm_log dm_mod parport_pc button sg r8169 mii parport sata_sis pata_sis ata_generic libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan]

Pid: 2548, comm: rmdir Tainted: G W (2.6.29-rc1-mm1 #4) Aspire SA85
EIP: 0060:[<c04607f2>] EFLAGS: 00210046 CPU: 1
EIP is at cgroup_rmdir+0xda/0x27e
EAX: f442b800 EBX: ed00b3c0 ECX: c04607ca EDX: 00000000
ESI: c0778dc0 EDI: 00200246 EBP: ed252f30 ESP: ed252f14
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmdir (pid: 2548, ti=ed252000 task=e19aa8c0 task.ti=ed252000)

Quoted 16 lines
Stack:
 00000000 f442b800 c0621d08 e18a8014 e2d96a00 fffffff0 e2ccadf0 ed252f44
 c04a67fe e2d96a00 00000000 0804ca00 ed252fa8 c04a7b8c ed269080 e2d745a0
 00002121 00000001 f46a3000 00000000 00000000 00000000 ed0a6d34 00000004
Call Trace:
 [<c0621d08>] ? _spin_unlock+0x2c/0x41
 [<c04a67fe>] ? vfs_rmdir+0x5b/0x9c
 [<c04a7b8c>] ? do_rmdir+0x89/0xc8
 [<c04438ed>] ? up_read+0x1b/0x2e
 [<c0623de4>] ? do_page_fault+0x356/0x5ed
 [<c04a7c14>] ? sys_rmdir+0x15/0x17
 [<c0403485>] ? sysenter_do_call+0x12/0x35
Code: c8 fe ff 8b 43 40 8b 70 0c eb 3e 8b 46 28 8b 44 83 20 89 45 e8 8b 55 e8 8b 52 04 83 fa 01 89 55 e4 0f 8f 5f ff ff ff 85 d2 75 04 <0f> 0b eb fe 8b 45 e4 31 c9 8b 55 e8 f0 0f b1 4a 04 8b 55 e4 39
EIP: [<c04607f2>] cgroup_rmdir+0xda/0x27e SS:ESP 0068:ed252f14
---[ end trace 4eaa2a86a8e2da25 ]---

I've reproduced this bug in my environment too.
I'll dig this too.

Thank you for reporting a bug.

Daisuke Nishimura.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
In reply to Li Zefan
Quoted 6 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

Li-san, could you try this ? I myself can't reproduce the bug yet... ==

From: KAMEZAWA Hiroyuki <[email protected]>

Now, at swapoff, even while try_charge() fails, commit is executed. This is bug and make refcnt of cgroup_subsys_state minus, finally.

Signed-off-by KAMEZAWA Hiroyuki <[email protected]>

---

mm/swapfile.c +4 -1
Index: mmotm-2.6.29-Jan14/mm/swapfile.c
===================================================================
@@ -698,8 +698,10 @@ static int unuse_pte(struct vm_area_stru
 	pte_t *pte;
 	int ret = 1;
 
-	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr))
+	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr)) {
 		ret = -ENOMEM;
+		goto out_nolock;
+	}
 
 	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
 	if (unlikely(!pte_same(*pte, swp_entry_to_pte(entry)))) {

@@ -723,6 +725,7 @@ static int unuse_pte(struct vm_area_stru
 	activate_page(page);
 out:
 	pte_unmap_unlock(pte, ptl);
+out_nolock:
 	return ret;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Reviewed-by
#8 lore raw
Quoted 5 lines · KAMEZAWA Hiroyuki

From: KAMEZAWA Hiroyuki <[email protected]>

Now, at swapoff, even while try_charge() fails, commit is executed. This is bug and make refcnt of cgroup_subsys_state minus, finally.

Nice catch!

I think this bug can explain this problem I've seen. Commiting on trycharge failure will add the pc to the lru without a corresponding charge and refcnt.
And rmdir uncharges the pc(so we get WARNING: at kernel/res_counter.c:71) and decrements the refcnt(so we get BUG at kernel/cgroup.c:2517).

Even if the problem cannot be fixed by this patch, this patch is valid and needed.

Quoted 1 line
Signed-off-by KAMEZAWA Hiroyuki <[email protected]>
Reviewed-by Daisuke Nishimura <[email protected]>

I'll test it.

Thanks,
Daisuke Nishimura.

Quoted 18 lines

---
Index: mmotm-2.6.29-Jan14/mm/swapfile.c
===================================================================

mm/swapfile.c +3 -1
@@ -698,8 +698,10 @@ static int unuse_pte(struct vm_area_stru
 	pte_t *pte;
 	int ret = 1;
 
-	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr))
+	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr)) {
 		ret = -ENOMEM;
+		goto out_nolock;
+	}
 
 	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
 	if (unlikely(!pte_same(*pte, swp_entry_to_pte(entry)))) {

mm/swapfile.c static int unuse_pte(struct vm_area_stru · line 731

Quoted 7 lines
mm/swapfile.c +1 -0
@@ -723,6 +725,7 @@ static int unuse_pte(struct vm_area_stru
 	activate_page(page);
 out:
 	pte_unmap_unlock(pte, ptl);
+out_nolock:
 	return ret;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
Quoted 14 lines · On Fri, 16 Jan 2009 17:26:51 +0900 Daisuke Nishimura…
Quoted 5 lines · KAMEZAWA Hiroyuki

From: KAMEZAWA Hiroyuki <[email protected]>

Now, at swapoff, even while try_charge() fails, commit is executed. This is bug and make refcnt of cgroup_subsys_state minus, finally.

Nice catch!

I think this bug can explain this problem I've seen. Commiting on trycharge failure will add the pc to the lru without a corresponding charge and refcnt.
And rmdir uncharges the pc(so we get WARNING: at kernel/res_counter.c:71) and decrements the refcnt(so we get BUG at kernel/cgroup.c:2517).

Quoted 8 lines

Even if the problem cannot be fixed by this patch, this patch is valid and needed.

Quoted 1 line
Signed-off-by KAMEZAWA Hiroyuki <[email protected]>
Reviewed-by Daisuke Nishimura <[email protected]>

I'll test it.

I finally get how-to-reprocuce and confirmed this fixes the problem.

How-to-reproduce.

In shell-A
#mount -t cgroup none /opt/cgroup
#mkdir /opt/cgroup/xxx/
#echo 0 > /opt/cgroup/xxx/tasks
#Run malloc 100M on this and sleep. ---(*)

In shell-B.
#echo 40M > /opt/cgroup/xxx/memory.limit_in_bytes. Then, you'll see 60M of swap. #/sbin/swapoff -a
Then, you'll see OOM-Kill against (*) #echo shell-A > /opt/cgroup/tasks
make /opt/cgroup/xxx/ empty
#rmdir /opt/cgroup/xxx

=> panics.

I'll add this swap-off test to memcg-debug.txt later.

BTW, OOM against (*) itself seems also probelmatic. But simply disable oom-at-swapoff cannot be a workaround...

-Kame

Quoted 18 lines

Thanks,
Daisuke Nishimura.

Quoted 15 lines

---
Index: mmotm-2.6.29-Jan14/mm/swapfile.c
===================================================================

mm/swapfile.c +3 -1
@@ -698,8 +698,10 @@ static int unuse_pte(struct vm_area_stru
 	pte_t *pte;
 	int ret = 1;
 
-	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr))
+	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr)) {
 		ret = -ENOMEM;
+		goto out_nolock;
+	}
Quoted 11 lines

mm/swapfile.c static int unuse_pte(struct vm_area_stru · line 731

Quoted 10 lines
mm/swapfile.c +1 -0
 	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
 	if (unlikely(!pte_same(*pte, swp_entry_to_pte(entry)))) {

@@ -723,6 +725,7 @@ static int unuse_pte(struct vm_area_stru
 	activate_page(page);
 out:
 	pte_unmap_unlock(pte, ptl);
+out_nolock:
 	return ret;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw

PATCH 1/1 memcg: fix refcnt handling at swapoff

Quoted 17 lines · On Fri, 16 Jan 2009 18:12:35 +0900 KAMEZAWA Hiroyuki…

How-to-reproduce.

In shell-A
#mount -t cgroup none /opt/cgroup
#mkdir /opt/cgroup/xxx/
#echo 0 > /opt/cgroup/xxx/tasks
#Run malloc 100M on this and sleep. ---(*)

In shell-B.
#echo 40M > /opt/cgroup/xxx/memory.limit_in_bytes. Then, you'll see 60M of swap. #/sbin/swapoff -a
Then, you'll see OOM-Kill against (*) #echo shell-A > /opt/cgroup/tasks
make /opt/cgroup/xxx/ empty
#rmdir /opt/cgroup/xxx

Quoted 2 lines

=> panics.

I'll update how-to-test text under Documentation/ later. -Kame
==
From: KAMEZAWA Hiroyuki <[email protected]>

Now, at swapoff, even while try_charge() fails, commit is executed. This is bug and make refcnt of cgroup_subsys_state minus, finally.

Reported-by Li Zefan <[email protected]>
Tested-by Li Zefan <[email protected]>
Tested-by Daisuke Nishimura <[email protected]>
Signed-off-by KAMEZAWA Hiroyuki <[email protected]>
Reviewed-by Daisuke Nishimura <[email protected]>
---
 mm/swapfile.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
mm/swapfile.c +4 -1
Index: mmotm-2.6.29-Jan14/mm/swapfile.c
===================================================================
@@ -698,8 +698,10 @@ static int unuse_pte(struct vm_area_stru
 	pte_t *pte;
 	int ret = 1;
 
-	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr))
+	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr)) {
 		ret = -ENOMEM;
+		goto out_nolock;
+	}
 
 	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
 	if (unlikely(!pte_same(*pte, swp_entry_to_pte(entry)))) {

@@ -723,6 +725,7 @@ static int unuse_pte(struct vm_area_stru
 	activate_page(page);
 out:
 	pte_unmap_unlock(pte, ptl);
+out_nolock:
 	return ret;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Tested-by
#11 lore raw
In reply to Daisuke Nishimura
Quoted 14 lines · Daisuke Nishimura
Quoted 5 lines · KAMEZAWA Hiroyuki

From: KAMEZAWA Hiroyuki <[email protected]>

Now, at swapoff, even while try_charge() fails, commit is executed. This is bug and make refcnt of cgroup_subsys_state minus, finally.

Nice catch!

I think this bug can explain this problem I've seen. Commiting on trycharge failure will add the pc to the lru without a corresponding charge and refcnt.
And rmdir uncharges the pc(so we get WARNING: at kernel/res_counter.c:71) and decrements the refcnt(so we get BUG at kernel/cgroup.c:2517).

Quoted 7 lines

Even if the problem cannot be fixed by this patch, this patch is valid and needed.

Quoted 1 line
Signed-off-by KAMEZAWA Hiroyuki <[email protected]>
Reviewed-by Daisuke Nishimura <[email protected]>

I'll test it.

I've tested several times, but this problem didn't happen.

Tested-by Daisuke Nishimura <[email protected]>

Thanks,
Daisuke Nishimura.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#12 lore raw
Quoted 17 lines · On Fri, 16 Jan 2009 18:13:00 +0900 Daisuke Nishimura…
Quoted 16 lines · Daisuke Nishimura
Quoted 5 lines · KAMEZAWA Hiroyuki

From: KAMEZAWA Hiroyuki <[email protected]>

Now, at swapoff, even while try_charge() fails, commit is executed. This is bug and make refcnt of cgroup_subsys_state minus, finally.

Nice catch!

I think this bug can explain this problem I've seen. Commiting on trycharge failure will add the pc to the lru without a corresponding charge and refcnt.
And rmdir uncharges the pc(so we get WARNING: at kernel/res_counter.c:71) and decrements the refcnt(so we get BUG at kernel/cgroup.c:2517).

Even if the problem cannot be fixed by this patch, this patch is valid and needed.

Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Signed-off-by KAMEZAWA Hiroyuki <[email protected]>
Reviewed-by Daisuke Nishimura <[email protected]>

I'll test it.

I've tested several times, but this problem didn't happen.

Tested-by Daisuke Nishimura <[email protected]>

Thank you!, I'll send the patch to Andrew.

-Kame

Quoted 10 lines

Thanks,
Daisuke Nishimura.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author Tested-by
#13 lore raw
In reply to KAMEZAWA Hiroyuki
Quoted 11 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

Li-san, could you try this ? I myself can't reproduce the bug yet...

I've tested this patch, and the bug seems to disappear. :)

Tested-by Li Zefan <[email protected]>

I'm going to be off office, and I'll do more testing to confirm this next week.

Quoted 27 lines

==

From: KAMEZAWA Hiroyuki <[email protected]>

Now, at swapoff, even while try_charge() fails, commit is executed. This is bug and make refcnt of cgroup_subsys_state minus, finally.

Signed-off-by KAMEZAWA Hiroyuki <[email protected]>

---
Index: mmotm-2.6.29-Jan14/mm/swapfile.c
===================================================================

mm/swapfile.c +3 -1
@@ -698,8 +698,10 @@ static int unuse_pte(struct vm_area_stru
 	pte_t *pte;
 	int ret = 1;
 
-	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr))
+	if (mem_cgroup_try_charge_swapin(vma->vm_mm, page, GFP_KERNEL, &ptr)) {
 		ret = -ENOMEM;
+		goto out_nolock;
+	}
 
 	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
 	if (unlikely(!pte_same(*pte, swp_entry_to_pte(entry)))) {

mm/swapfile.c static int unuse_pte(struct vm_area_stru · line 731

Quoted 7 lines
mm/swapfile.c +1 -0
@@ -723,6 +725,7 @@ static int unuse_pte(struct vm_area_stru
 	activate_page(page);
 out:
 	pte_unmap_unlock(pte, ptl);
+out_nolock:
 	return ret;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw
Quoted 11 lines · On Fri, 16 Jan 2009 16:33:08 +0800 Li Zefan <[email protected]> wrote:
Quoted 10 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Fri, 16 Jan 2009 14:15:04 +0800 Li Zefan <[email protected]> wrote:

Found this when testing memory resource controller, can be triggered with:
- CONFIG_CGROUP_MEM_RES_CTLR_SWAP=n
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
- or CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y && boot with noswapaccount

Quoted 9 lines
Quoted 1 line

Li-san, could you try this ? I myself can't reproduce the bug yet...

I've tested this patch, and the bug seems to disappear. :)

Tested-by Li Zefan <[email protected]>

I'm going to be off office, and I'll do more testing to confirm this next week.

Thank you !

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>