how much memory does LVM need? oom-killer comes
I just did vgchange -an, and there came the oom-killer. This is all with 2.6.22.5 kernel.
The machine has 256 MB RAM; PV is almost 1.5 TB big:
# pvs PV VG Fmt Attr PSize PFree /dev/md0 superthecus lvm2 a- 1.46T 120.63G
After I do "vgchange -ay", the memory usage goes from ~17 MB to almost 180 MB - is it normal?
# vgchange -an vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013007b>] probe_irq_on+0x6c/0xea [<c013a2e6>] __handle_mm_fault+0x30e/0x68e [<c013971f>] follow_page+0xcf/0x11b [<c013a80b>] get_user_pages+0x1a5/0x25a [<c013afcc>] make_pages_present+0x79/0x8e [<c013cd70>] do_brk+0x1c7/0x1df [<c013ce30>] sys_brk+0xa8/0xc8 [<c0103aea>] syscall_call+0x7/0xb ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1720 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables Out of memory: kill process 2101 (rpc.statd) score 441 or a child Killed process 2101 (rpc.statd) klogd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013589e>] __do_page_cache_readahead+0x79/0x181 [<c0110581>] do_page_fault+0x272/0x515 [<c01329c9>] filemap_nopage+0x115/0x255 [<c013a0dc>] __handle_mm_fault+0x104/0x68e [<c011051b>] do_page_fault+0x20c/0x515 [<c0149452>] vfs_read+0xfa/0x10a [<c011030f>] do_page_fault+0x0/0x515 [<c028525a>] error_code+0x6a/0x70 [<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4 ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1721 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013007b>] probe_irq_on+0x6c/0xea [<c013a2e6>] __handle_mm_fault+0x30e/0x68e [<c013971f>] follow_page+0xcf/0x11b [<c013a80b>] get_user_pages+0x1a5/0x25a [<c013afcc>] make_pages_present+0x79/0x8e [<c013cd70>] do_brk+0x1c7/0x1df [<c013ce30>] sys_brk+0xa8/0xc8 [<c0103aea>] syscall_call+0x7/0xb ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1721 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013589e>] __do_page_cache_readahead+0x79/0x181 [<c028451a>] io_sc0130a54>] sync_page+0x0/0x3c [<c028470f>] __wait_on_bit_lock+0x4a/0x51 [<c0130a46>] __lock_page+0x57/0x5d [<c01329c9>] filemap_nopage+0x11 [<c013a0dc>] __handle_mm_fault+0x104/0x68e [<c011051b>] do_page_fault+0x20c/0x515 [<c0124062>] do_gettimeofday+0x2c/0xc4 [<c011030f>] do_page_fault+0x0/0x515 [<c028525a>] error_code+0x6a/0x70 [<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4 ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1722 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alndle_mm_fault+0x30e/0x68e [<c013971f>] follow_page+0xcf/0x11b [<c013a80b>] get_user_pages+0x1a5/0x25a [<c013afcc>] make_pages_present+0x79/0x8e [<c013cd70>] do_brk+0x1c7/0x1df [<c013ce30>] sys_brk+0xa8/0xc8 [<c0103aea>] syscall_call+0x7/0xb ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 ree:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1722 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013589e>] __do_page_cache_readahead+0x79/0x181 [<c028451a>] io_schedule+0xe/0x16 [<c0130a54>] sync_page+0x0/0x3c [<c028470f>] __wait_on_bit_lock+0x4a/0x51 [<c0130a46>] __lock_page+0x57/0x5d [<c01329c9>] filemap_nopage+0x115/0x255 [<c013a0dc>] __handle_mm_fault+0x104/0x68e [<c011051b>] do_page_fault+0x20c/0x515 [<c0124062>] do_gettimeofday+0x2c/0xc4 [<c011030f>] do_page_fault+0x0/0x515 [<c028525a>] error_code+0x6a/0x70 [<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4 ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1722 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013007b>] probe_irq_on+0x6c/0xea [<c013a2e6>] __handle_mm_fault+0x30e/0x68e [<c013971f>] follow_page+0xcf/0x11b [<c013a80b>] get_user_pages+0x1a5/0x25a [<c013afcc>] make_pages_present+0x79/0x8e [<c013cd70>] do_brk+0x1c7/0x1df [<c013ce30>] sys_brk+0xa8/0xc8 [<c0103aea>] syscall_call+0x7/0xb ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1722 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013589e>] __do_page_cache_readahead+0x79/0x181 [<c028451a>] io_schedule+0xe/0x16 [<c0130a54>] sync_page+0x0/0x3c [<c028470f>] __wait_on_bit_lock+0x4a/0x51 [<c0130a46>] __lock_page+0x57/0x5d [<c01329c9>] filemap_nopage+0x115/0x255 [<c013a0dc>] __handle_mm_fault+0x104/0x68e [<c011051b>] do_page_fault+0x20c/0x515 [<c0124062>] do_gettimeofday+0x2c/0xc4 [<c011030f>] do_page_fault+0x0/0x515 [<c028525a>] error_code+0x6a/0x70 [<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4 ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1722 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables vgchange invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013007b>] probe_irq_on+0x6c/0xea [<c013a2e6>] __handle_mm_fault+0x30e/0x68e [<c013971f>] follow_page+0xcf/0x11b [<c013a80b>] get_user_pages+0x1a5/0x25a [<c013afcc>] make_pages_present+0x79/0x8e [<c013cd70>] do_brk+0x1c7/0x1df [<c013ce30>] sys_brk+0xa8/0xc8 [<c0103aea>] syscall_call+0x7/0xb ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1722 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables rpc.idmapd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 [<c0133396>] out_of_memory+0x68/0x166 [<c01345f1>] __alloc_pages+0x1fb/0x285 [<c013589e>] __do_page_cache_readahead+0x79/0x181 [<c028451a>] io_schedule+0xe/0x16 [<c0130a54>] sync_page+0x0/0x3c [<c028470f>] __wait_on_bit_lock+0x4a/0x51 [<c0130a46>] __lock_page+0x57/0x5d [<c01329c9>] filemap_nopage+0x115/0x255 [<c013a0dc>] __handle_mm_fault+0x104/0x68e [<c011051b>] do_page_fault+0x20c/0x515 [<c0124062>] do_gettimeofday+0x2c/0xc4 [<c011030f>] do_page_fault+0x0/0x515 [<c028525a>] error_code+0x6a/0x70 [<c0280000>] unix_dgram_sendmsg+0x3a8/0x3b4 ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 14 Cold: hi: 30, btch: 7 usd: 18 Active:2017 inactive:490 dirty:0 writeback:0 unstable:0 free:16602 slab:40594 mapped:952 pagetables:56 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:4556kB inactive:916kB present:16256kB pages_scanned:8309 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61312kB min:61368kB low:76708kB high:92052kB active:3512kB inactive:1044kB present:239716kB pages_scanned:7494 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 0*16kB 72*32kB 40*64kB 19*128kB 9*256kB 9*512kB 4*1024kB 1*2048kB 10*4096kB = 61312kB Swap cache: add 5022, delete 5022, find 1982/2477, race 0+0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1722 pages shared 0 pages swap cached 0 pages dirty 0 pages writeback 952 pages mapped 40594 pages slab 56 pages pagetables Out of memory: kill process 1855 (portmap) score 422 or a child Killed process 1855 (portmap) Out of memory: kill process 2212 (bash) score 121 or a child Killed process 2260 (vgchange) Killed
Signature · 1 line
Tomasz Chmielewski
Quoted 9 lines · Tomasz Chmielewski
I just did vgchange -an, and there came the oom-killer. This is all with 2.6.22.5 kernel.
The machine has 256 MB RAM; PV is almost 1.5 TB big:
# pvs PV VG Fmt Attr PSize PFree /dev/md0 superthecus lvm2 a- 1.46T 120.63GAfter I do "vgchange -ay", the memory usage goes from ~17 MB to almost 180 MB - is it normal?
Can the memory usage be so high (~180 MB out of 256 MB) because I have two pretty big snapshots? Normally, this machine uses about 17 MB.
# lvs LV VG Attr LSize Origin Snap% Move Log Copy% backup superthecus owi-ao 1.20T backup-snap superthecus swi-a- 100.00G backup 30.15 backup-snap-resized superthecus swi-a- 30.00G backup 69.00 backup1 superthecus -wi-a- 3.00G backup1-sync-test superthecus -wi-a- 3.00G src superthecus -wi-a- 2.00G swap superthecus -wi-ao 3.00G
Unfortunately, I can't remove these snapshots, as oom-killer comes :(
Is there any solution to that, other than adding memory (which can be hard to do remotely)?
# lvremove /dev/superthecus/backup-snap-resized Do you really want to remove active logical volume "backup-snap-resized"? [y/n]: y printk: 11 messages suppressed. lvremove invoked oom-killer: gfp_mask=0x280d2, order=0, oomkilladj=0 [<c013338e>] out_of_memory+0x68/0x166 [<c01345e9>] __alloc_pages+0x1fb/0x285 [<c012fa09>] handle_edge_irq+0xb1/0xcc [<c013a2de>] __handle_mm_fault+0x30e/0x68e [<c0139717>] follow_page+0xcf/0x11b [<c013a803>] get_user_pages+0x1a5/0x25a [<c013afc4>] make_pages_present+0x79/0x8e [<c013cd68>] do_brk+0x1c7/0x1df [<c013ce28>] sys_brk+0xa8/0xc8 [<c0103aea>] syscall_call+0x7/0xb [<c0280000>] unix_stream_recvmsg+0x144/0x40c ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 90, btch: 15 usd: 13 Cold: hi: 30, btch: 7 usd: 4 Active:1374 inactive:675 dirty:0 writeback:0 unstable:0 free:16608 slab:40608 mapped:1007 pagetables:60 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:2124kB inactive:1760kB present:16256kB pages_scanned:5851 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61336kB min:61368kB low:76708kB high:92052kB active:3372kB inactive:940kB present:239716kB pages_scanned:938 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 1*8kB 31*16kB 91*32kB 35*64kB 15*128kB 6*256kB 10*512kB 4*1024kB 1*2048kB 10*4096kB = 61336kB Swap cache: add 5620, delete 5037, find 1791/2263, race 0+0 Free swap = 3140372kB Total swap = 3145720kB Free swap: 3140372kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1935 pages shared 583 pages swap cached 0 pages dirty 0 pages writeback 1007 pages mapped 40608 pages slab 60bles Out of memory: kill process 2160 (lvremove) score 198 or a child Killed process 2160 (lvremove) klogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0 [<c013338e>] out_of_memory+0x68/0x166 [<c01345e9>] __0x1fb/0x285 [<c47c7>] sock_def_readable+0x27/0x4b [<c013ffa6>] read_swap_cache_async+0x2f/0xac [<c0139040>] swapin_readahead+0x3a/0x58 [<c013a44c>] __handle_mm_fault+0x47c/0x68e [<c019a5b5>] cfq_completed_request+0x243/0x273 [<c011051b>] do_page_fault+0x20c/0x515 [<c0148b6c>] do_sync_write+0xc6/0x109 [<c011030f>] do_page_fault+0x0/0x515 [<c028510a>] error_code+0x6a/0x70 [<c012007b>] parse_args+0x89/0x20a [<c01149b9>] do_syslog+0x10c/0x2b4 [<c0121221>] autoremove_wake_function+0x0/0x35 [<c01742da>] kmsg_read+0x0/0x36 [<c01493d8>] vfs_read+0x88/0x10a [<c01496fb>] sys_read+0x41/0x67 [<c0103aea>] syscall_call+0x7/0xb ======================= Mem-info: DMA per-cpu: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: Hot: hi: 902 Cold: hi: 30, btch: 7 usd: 8 Active:1403 inactive:643 dirty:0 writeback:0 unstable:0 free:16606 slab:40608 mapped:1007 pagetables:60 bounce:0 DMA free:5096kB min:4160kB low:5200kB high:6240kB active:2124kB inactive:1760kB present:16256kB pages_scanned:5851 all_unreclaimable? yes lowmem_reserve[]: 0 234 Normal free:61328kB min:61368kB low:76708kB high:92052kB active:3488kB inactive:812kB present:239716kB pages_scanned:1002 all_unreclaimable? yes lowmem_reserve[]: 0 0 DMA: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 5096kB Normal: 0*4kB 0*8kB 31*16kB 91*32kB 35*64kB 15*128kB 6*256kB 10*512kB 4*1024kB 1*2048kB 10*4096kB = 61328kB Swap cache: add 5620, delete 5037, find 1791/2264, race 0+0 Free swap = 3140372kB Total swap = 3145720kB Free swap: 3140372kB 64496 pages of RAM 0 pages of HIGHMEM 1254 reserved pages 1933 pages shared 583 pages swap cached 0 pages dirty 0 pages writeback 1007 pages mapped 40608 pages slab 60 pages pagetables Killed
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 26 lines · Tomasz Chmielewski
Quoted 9 lines · Tomasz Chmielewski
I just did vgchange -an, and there came the oom-killer. This is all with 2.6.22.5 kernel.
The machine has 256 MB RAM; PV is almost 1.5 TB big:
# pvs PV VG Fmt Attr PSize PFree /dev/md0 superthecus lvm2 a- 1.46T 120.63GAfter I do "vgchange -ay", the memory usage goes from ~17 MB to almost 180 MB - is it normal?
Can the memory usage be so high (~180 MB out of 256 MB) because I have two pretty big snapshots? Normally, this machine uses about 17 MB.
# lvs LV VG Attr LSize Origin Snap% Move Log Copy% backup superthecus owi-ao 1.20T backup-snap superthecus swi-a- 100.00G backup 30.15 backup-snap-resized superthecus swi-a- 30.00G backup 69.00 backup1 superthecus -wi-a- 3.00G backup1-sync-test superthecus -wi-a- 3.00G src superthecus -wi-a- 2.00G swap superthecus -wi-ao 3.00G
Quoted 2 lines
Unfortunately, I can't remove these snapshots, as oom-killer comes :(
Is there any solution to that, other than adding memory (which can be hard to do remotely)?
Ouch - now it went so far that the machine only loops from a kernel panic to a kernel panic - that's really bad:
(...)
Out of memory: kill process 2220 (getty) score 24 or a child Killed process 2220 (getty)
Kernel panic - not syncing: Out of memory and no killable processes...
Rebooting in 20 seconds..
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 8 lines · Tomasz Chmielewski
Ouch - now it went so far that the machine only loops from a kernel panic to a kernel panic - that's really bad:
(...)
Out of memory: kill process 2220 (getty) score 24 or a child Killed process 2220 (getty)Kernel panic - not syncing: Out of memory and no killable processes...Rebooting in 20 seconds..
Again, replying to myself, as no one on this list seem to have a clue about problems with LVM snapshots... I added RAM, and was able to boot again.
Below is a quick try to estimate the memory usage of LVM snapshots; I'm using 2.6.22.6.
The machine has 256 MB RAM.
With these volumes and snapshots, almost 200 MB of memory is used:
# lvs LV VG Attr LSize Origin Snap% Move Log Copy% backup st owi-ao 1.20T backup-snap st swi-a- 100.00G backup 31.22 backup-snap-resized st swi-a- 30.00G backup 72.73 backup1 st -wi-a- 3.00G backup1-sync-test st -wi-a- 3.00G src st -wi-a- 2.00G swap st -wi-ao 3.00G
# free
total used free shared buffers cached
Mem: 512864 418380 94484 0 216036 5156
-/+ buffers/cache: 197188 315676After removing a 100 GB snapshot that is over 30% full, we reclaim almost 100 MB of memory:
# lvremove /dev/st/backup-snap
Do you really want to remove active logical volume "backup-snap"? [y/n]: y
Logical volume "backup-snap" successfully removed
# free
total used free shared buffers cached
Mem: 512864 328668 184196 0 220652 7480
-/+ buffers/cache: 100536 412328
Swap: 3145720 220 3145500After removing yet another snapshot - 30 GB, over 70% full - we reclaim 70 MB:
# lvremove /dev/st/backup-snap-resized Do you really want to remove active logical volume "backup-snap-resized"? [y/n]: y Logical volume "backup-snap-resized" successfully removed
# free
total used free shared buffers cached
Mem: 512864 335500 177364 0 296444 9436
-/+ buffers/cache: 29620 483244
Swap: 3145720 216 3145504So, as a rule of thumb, we can say that for each *used* GB of snapshot, ~3 MB of RAM is needed.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 9 lines · Tomasz Chmielewski
Quoted 5 lines
Out of memory: kill process 2220 (getty) score 24 or a child Killed process 2220 (getty)
Kernel panic - not syncing: Out of memory and no killable processes...Rebooting in 20 seconds..
Again, replying to myself, as no one on this list seem to have a clue about problems with LVM snapshots... I added RAM, and was able to boot again.
I don't have any answers, but I am very impressed by this kind of testing and grateful for the contribution. My thoughts on evil experiments focused on timing and component failure. Obviously, resource limits are extremely important as well.
I suppose ideal behaviour would be to refuse to create a snapshot when memory is "low". But defining "low" can be tricky.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 3 lines · Tomasz Chmielewski
Again, replying to myself, as no one on this list seem to have a clue about problems with LVM snapshots... I added RAM, and was able to boot again.
I may have an answer after all. I suspect that the chunksize parameter to lvcreate -s will affect memory use. A larger chunksize should use less memory.
Quoted 2 lines
So, as a rule of thumb, we can say that for each *used* GB of snapshot, ~3 MB of RAM is needed.
On my snapshots on Centos-5, the chunksize defaults to 8k. With your observation of 3MB per Gig of snapshot, that comes to 24 bytes per chunk. Quite reasonable. If you make your chunksize 512K (the maximum supported), memory use should change to 48K per Gig.
lvdisplay will show the chunksize for a snapshot.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 7 lines · Stuart D. Gathman
Quoted 5 lines · Tomasz Chmielewski
Quoted 5 lines
Out of memory: kill process 2220 (getty) score 24 or a child Killed process 2220 (getty)
Kernel panic - not syncing: Out of memory and no killable processes...Rebooting in 20 seconds..
Quoted 8 lines
Quoted 1 line
Again, replying to myself, as no one on this list seem to have a clue about problems with LVM snapshots... I added RAM, and was able to boot again.
I don't have any answers, but I am very impressed by this kind of testing
and grateful for the contribution. My thoughts on evil experiments
focused on timing and component failure. Obviously, resource limits are extremely important as well.I suppose ideal behaviour would be to refuse to create a snapshot when memory is "low". But defining "low" can be tricky.
Not really.
We can still have plenty memory when we want to create a snapshot - but memory can shrink later, as the snapshot fills.
As I understand, and as experimentation shown, memory usage increases as the snapshot gets filled, with 3 MB of RAM per 1 GB of snapshot filled (more or less).
As the memory used for a snapshot is not swappable, from a certain point, it is possible to realize "I'm in trouble, but I can't do anything".
Let's watch at the process once again:
1. Make a snapshot of some large volume (it's easier than making multiple small snapshots). We still have plenty of RAM, so there is no sense in refusing to create a snapshot.
2. Start filling the snapshot by either writing to the original, or to the snapshot. Memory usage increases.
3. As the snapshot fills, and you have less and less memory, from some point, you will no longer be able to remove the snapshot.
4. Now, you can only watch as the snapshot grows, your memory shrinks, the processes are getting killed, and eventually, the kernel panics. If you only have remote access, you're screwed.
Perhaps, in 1., instead of refusing to make a snapshot, make a printk, so that the admin knows what may happen? Is the memory used for a snapshot possible to calculate? Is it always ~3 MB RAM per 1 GB changes, or is it sometimes 1 MB, and sometimes 4 MB, depending on which data changes? If it remains more or less the same, add up all space reserved for snapshots, if it's more than we have, print a message.
BTW, wasn't it like that in some kernels before 2.6.22? With 2.6.18, I was getting a "cannot allocate memory" when I tried to create several snapshots. I guess it was a bug, though.
Ideally would be to make the memory needed for snapshots swappable, but I'm not sure the kernel would really like it.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 8 lines · Stuart D. Gathman
Quoted 1 line · Tomasz Chmielewski
Again, replying to myself, as no one on this list seem to have a clue about problems with LVM snapshots... I added RAM, and was able to boot again.
I may have an answer after all. I suspect that the chunksize parameter to lvcreate -s will affect memory use. A larger chunksize should use less memory.
Quoted 8 lines
Quoted 1 line
So, as a rule of thumb, we can say that for each *used* GB of snapshot, ~3 MB of RAM is needed.
On my snapshots on Centos-5, the chunksize defaults to 8k. With your observation of 3MB per Gig of snapshot, that comes to 24 bytes per chunk. Quite reasonable. If you make your chunksize 512K (the maximum supported), memory use should change to 48K per Gig.
lvdisplay will show the chunksize for a snapshot.
Indeed, it was 8 kB.
What is the meaning of the chunksize in snapshots?
With 512 kB chunksize, I guess the snapshot will fill faster than with 8 KB chunksize, given that the same amount of data changes?
Here is a quick test that shows it.
I created two 20 GB snapshots of a 20GB volume, one with 8kB, and the other with 512 kB chunksize.
Then, I created a 50 MB file on the original volume (out of /dev/zero):
build1 san1 owi-ao 20.00G build1-512 san1 swi-a- 20.00G build1 0.09 build1-8 san1 swi-a- 20.00G build1 0.08
Copying /bin, /boot and /lib (71 MB in total) into a new directory makes that difference bigger:
build1 san1 owi-ao 20.00G build1-512 san1 swi-a- 20.00G build1 0.76 build1-8 san1 swi-a- 20.00G build1 0.62
Copying a 1.2 GB /usr into a new directory shows this:
build1-512 san1 swi-a- 20.00G build1 7.40 build1-8 san1 swi-a- 20.00G build1 7.18
And finally, create 1000000 empty files:
build1 san1 owi-ao 20.00G build1-512 san1 swi-a- 20.00G build1 8.22 build1-8 san1 swi-a- 20.00G build1 7.91
The difference is not that big, usually it should be below 5%.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org