3.6.11: khubd issue SLAB related?
Hello,
I found this in dmesg:
[3663200.373989] pwc: Philips PCVC740K (ToUCam Pro)/PCVC840 (ToUCam II) USB webcam detected. [3663200.779617] khubd: page allocation failure: order:4, mode:0xd0 [3663200.779621] Pid: 274, comm: khubd Not tainted 3.6.11 #22 [3663200.779622] Call Trace: [3663200.779629] [<ffffffff810b5346>] ? warn_alloc_failed+0xf6/0x150 [3663200.779632] [<ffffffff8141c6de>] ? __alloc_pages_direct_compact+0x1ac/0x1be [3663200.779635] [<ffffffff810b7f89>] ? __alloc_pages_nodemask+0x609/0x820 [3663200.779638] [<ffffffff810e8796>] ? cache_alloc_refill+0x356/0x660 [3663200.779640] [<ffffffff810e8c7b>] ? kmem_cache_alloc+0xab/0xe0 [3663200.779643] [<ffffffffa00168da>] ? usb_pwc_probe+0x17a/0xaf0 [pwc] [3663200.779646] [<ffffffff812de513>] ? usb_probe_interface+0x113/0x200 [3663200.779650] [<ffffffff8128be2e>] ? driver_probe_device+0x6e/0x210 [3663200.779652] [<ffffffff8128c070>] ? __driver_attach+0xa0/0xa0 [3663200.779654] [<ffffffff8128a2de>] ? bus_for_each_drv+0x3e/0x80 [3663200.779656] [<ffffffff8128bd90>] ? device_attach+0x80/0x90 [3663200.779658] [<ffffffff8128b370>] ? bus_probe_device+0x80/0xb0 [3663200.779659] [<ffffffff812899d7>] ? device_add+0x5b7/0x670 [3663200.779662] [<ffffffff812dd3a8>] ? usb_set_configuration+0x4b8/0x720 [3663200.779664] [<ffffffff812e5a3b>] ? generic_probe+0x3b/0xa0 [3663200.779666] [<ffffffff8128be2e>] ? driver_probe_device+0x6e/0x210 [3663200.779668] [<ffffffff8128c070>] ? __driver_attach+0xa0/0xa0 [3663200.779670] [<ffffffff8128a2de>] ? bus_for_each_drv+0x3e/0x80 [3663200.779672] [<ffffffff8128bd90>] ? device_attach+0x80/0x90 [3663200.779674] [<ffffffff8128b370>] ? bus_probe_device+0x80/0xb0 [3663200.779675] [<ffffffff812899d7>] ? device_add+0x5b7/0x670 [3663200.779678] [<ffffffff81282d7f>] ? mix_pool_bytes.constprop.15+0x3f/0x60 [3663200.779681] [<ffffffff81421300>] ? _raw_spin_unlock_bh+0x20/0x20 [3663200.779683] [<ffffffff812d5d08>] ? usb_new_device+0x1d8/0x260 [3663200.779685] [<ffffffff812d640a>] ? hub_thread+0x67a/0x14d0 [3663200.779687] [<ffffffff8142021c>] ? __schedule+0x2ac/0x800 [3663200.779690] [<ffffffff810611ec>] ? __wake_up_common+0x4c/0x80 [3663200.779693] [<ffffffff81059b00>] ? abort_exclusive_wait+0xb0/0xb0 [3663200.779695] [<ffffffff812d5d90>] ? usb_new_device+0x260/0x260 [3663200.779697] [<ffffffff810591d5>] ? kthread+0x85/0x90 [3663200.779699] [<ffffffff814231f4>] ? kernel_thread_helper+0x4/0x10 [3663200.779701] [<ffffffff81059150>] ? kthread_freezable_should_stop+0x50/0x50 [3663200.779703] [<ffffffff814231f0>] ? gs_change+0xb/0xb [3663200.779704] Mem-Info: [3663200.779705] DMA per-cpu: [3663200.779706] CPU 0: hi: 0, btch: 1 usd: 0 [3663200.779708] CPU 1: hi: 0, btch: 1 usd: 0 [3663200.779709] CPU 2: hi: 0, btch: 1 usd: 0 [3663200.779710] CPU 3: hi: 0, btch: 1 usd: 0 [3663200.779711] DMA32 per-cpu: [3663200.779712] CPU 0: hi: 186, btch: 31 usd: 0 [3663200.779713] CPU 1: hi: 186, btch: 31 usd: 0 [3663200.779714] CPU 2: hi: 186, btch: 31 usd: 0 [3663200.779715] CPU 3: hi: 186, btch: 31 usd: 0 [3663200.779716] Normal per-cpu: [3663200.779717] CPU 0: hi: 186, btch: 31 usd: 0 [3663200.779718] CPU 1: hi: 186, btch: 31 usd: 0 [3663200.779719] CPU 2: hi: 186, btch: 31 usd: 0 [3663200.779720] CPU 3: hi: 186, btch: 31 usd: 0 [3663200.779723] active_anon:367320 inactive_anon:142633 isolated_anon:0 [3663200.779723] active_file:470023 inactive_file:522338 isolated_file:0 [3663200.779723] unevictable:0 dirty:89 writeback:0 unstable:0 [3663200.779723] free:87496 slab_reclaimable:79192 slab_unreclaimable:19937 [3663200.779723] mapped:18116 shmem:3960 pagetables:11525 bounce:0 [3663200.779728] DMA free:15904kB min:136kB low:168kB high:204kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15680kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[3663200.779730] lowmem_reserve[]: 0 2448 7483 7483 [3663200.779736] DMA32 free:169684kB min:22072kB low:27588kB
high:33108kB active_anon:196000kB inactive_anon:221500kB active_file:709152kB inactive_file:793704kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2507728kB mlocked:0kB dirty:12kB writeback:0kB mapped:19536kB shmem:3072kB slab_reclaimable:161628kB slab_unreclaimable:15436kB kernel_stack:488kB pagetables:4780kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[3663200.779737] lowmem_reserve[]: 0 0 5034 5034 [3663200.779742] Normal free:164396kB min:45372kB low:56712kB
high:68056kB active_anon:1273280kB inactive_anon:349032kB active_file:1170940kB inactive_file:1295648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5155036kB mlocked:0kB dirty:344kB writeback:0kB mapped:52928kB shmem:12768kB slab_reclaimable:155140kB slab_unreclaimable:64312kB kernel_stack:3312kB pagetables:41320kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[3663200.779744] lowmem_reserve[]: 0 0 0 0 [3663200.779746] DMA: 0*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15904kB [3663200.779751] DMA32: 23335*4kB 8814*8kB 365*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 169692kB [3663200.779757] Normal: 38925*4kB 1081*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 164396kB [3663200.779762] 1007244 total pagecache pages [3663200.779763] 10889 pages in swap cache [3663200.779764] Swap cache stats: add 125307, delete 114418, find 564015/570538 [3663200.779765] Free swap = 3908344kB [3663200.779766] Total swap = 4095984kB [3663200.799573] 1961968 pages RAM [3663200.799575] 75067 pages reserved [3663200.799576] 268951 pages shared [3663200.799577] 1601656 pages non-shared [3663200.799579] SLAB: Unable to allocate memory on node 0 (gfp=0xd0) [3663200.799581] cache: size-65536, object size: 65536, order: 4 [3663200.799583] node 0: slabs: 6/6, objs: 6/6, free: 0 [3663200.799585] pwc: Oops, could not allocate memory for pwc_device. [3663200.799593] Philips webcam: probe of 2-4:1.0 failed with error -12 [3663200.799619] usb 2-4: adding 2-4:1.1 (config #1, interface 1) [3663200.799807] snd-usb-audio 2-4:1.1: usb_probe_interface [3663200.799812] snd-usb-audio 2-4:1.1: usb_probe_interface - got id [3663200.867541] usb 2-4: adding 2-4:1.2 (config #1, interface 2) [3663200.867614] hub 2-0:1.0: state 7 ports 5 chg 0000 evt 0010
How can we fix this?
Udo
Quoted 3 lines · Udo van den Heuvel
Hello,
I found this in dmesg:
slabtop:
Active / Total Objects (% used) : 784982 / 981439 (80.0%) Active / Total Slabs (% used) : 62996 / 63077 (99.9%) Active / Total Caches (% used) : 119 / 196 (60.7%) Active / Total Size (% used) : 205966.20K / 236692.16K (87.0%)
Minimum / Average / Maximum Object : 0.02K / 0.24K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
217946 211509 97% 0.06K 3694 59 14776K size-64 158508 65908 41% 0.10K 4284 37 17136K buffer_head 121380 107155 88% 0.12K 4046 30 16184K size-128 110692 108707 98% 0.86K 27673 4 110692K ext4_inode_cache 98980 85163 86% 0.19K 4949 20 19796K dentry 37927 32219 84% 0.16K 1649 23 6596K vm_area_struct 36001 24087 66% 0.55K 5143 7 20572K radix_tree_node 25179 19801 78% 0.05K 327 77 1308K anon_vma_chain 22561 7716 34% 0.05K 293 77 1172K jbd2_inode 17696 12708 71% 0.03K 158 112 632K size-32 17260 11348 65% 0.19K 863 20 3452K size-192 13416 8703 64% 0.29K 1032 13 4128K nf_conntrack_ffffffff815d8840 12274 12046 98% 0.11K 361 34 1444K sysfs_dir_cache 11977 10117 84% 0.06K 203 59 812K anon_vma 11070 9712 87% 0.25K 738 15 2952K filp 10638 10457 98% 0.60K 1773 6 7092K proc_inode_cache 9156 9136 99% 0.13K 327 28 1308K ext4_groupinfo_4k 3762 3161 84% 0.60K 627 6 2508K shmem_inode_cache 3744 3553 94% 0.02K 26 144 104K dm_target_io 3680 3553 96% 0.04K 40 92 160K dm_io 3577 2991 83% 0.54K 511 7 2044K inode_cache 3524 3103 88% 1.00K 881 4 3524K size-1024 2756 2586 93% 0.07K 52 53 208K Acpi-Operand 2044 1929 94% 0.13K 73 28 292K inotify_inode_mark 1600 1439 89% 0.50K 200 8 800K size-512 1380 552 40% 0.19K 69 20 276K cred_jar 1092 910 83% 0.62K 182 6 728K sock_inode_cache 1029 979 95% 0.53K 147 7 588K idr_layer_cache 1020 537 52% 0.19K 51 20 204K bio-0 1012 953 94% 0.04K 11 92 44K Acpi-Namespace 860 839 97% 0.19K 43 20 172K ip_dst_cache 810 605 74% 0.12K 27 30 108K pid 808 303 37% 0.02K 4 202 16K ext4_io_page 788 775 98% 2.00K 394 2 1576K size-2048 782 585 74% 0.11K 23 34 92K task_delay_info 765 474 61% 0.25K 51 15 204K skbuff_head_cache 742 209 28% 0.07K 14 53 56K eventpoll_pwq 693 640 92% 0.81K 77 9 616K UNIX 605 591 97% 1.56K 121 5 968K task_struct 560 119 21% 0.03K 5 112 20K tcp_bind_bucket 500 209 41% 0.19K 25 20 100K eventpoll_epi 486 418 86% 0.81K 54 9 432K task_xstate 480 378 78% 0.25K 32 15 128K size-256 480 241 50% 0.19K 24 20 96K inet_peer_cache 408 189 46% 0.11K 12 34 48K jbd2_journal_head 400 299 74% 0.15K 16 25 64K dm_crypt_io 369 308 83% 4.00K 369 1 1476K biovec-256 368 304 82% 1.00K 92 4 368K signal_cache 360 220 61% 0.09K 9 40 36K blkdev_ioc 354 354 100% 4.00K 354 1 1416K size-4096 354 149 42% 0.06K 6 59 24K fs_cache 350 230 65% 0.50K 50 7 200K skbuff_fclone_cache 303 296 97% 2.06K 101 3 808K sighand_cache 289 284 98% 16.00K 289 1 4624K size-16384 285 228 80% 0.25K 19 15 76K mnt_cache 276 50 18% 0.04K 3 92 12K khugepaged_mm_slot 270 224 82% 0.12K 9 30 36K scsi_sense_cache 255 183 71% 0.25K 17 15 68K sgpool-8
99.9% used?
Please help!
Udo
Quoted 4 lines · Udo van den Heuvel
[3663200.799579] SLAB: Unable to allocate memory on node 0 (gfp=0xd0) [3663200.799581] cache: size-65536, object size: 65536, order: 4 [3663200.799583] node 0: slabs: 6/6, objs: 6/6, free: 0 [3663200.799585] pwc: Oops, could not allocate memory for pwc_device.
You have memory freed, but like your earlier page allocation failure on mount, it is too fragmented. This is an order-4 page allocation, the kmalloc() of struct pwc_device, and you don't even have any order-3 pages that are available. Are you running with CONFIG_COMPACTION enabled?
If not, enable it and try again, or get root and manually try to defragment memory by doing echo 1 > /proc/sys/vm/compact_memory.
If so, try killing a memory hogging process to free some memory and attempt to get some higher order pages available.
Quoted 11 lines · David Rientjes
Quoted 4 lines · Udo van den Heuvel
[3663200.799579] SLAB: Unable to allocate memory on node 0 (gfp=0xd0) [3663200.799581] cache: size-65536, object size: 65536, order: 4 [3663200.799583] node 0: slabs: 6/6, objs: 6/6, free: 0 [3663200.799585] pwc: Oops, could not allocate memory for pwc_device.You have memory freed, but like your earlier page allocation failure on mount, it is too fragmented. This is an order-4 page allocation, the
kmalloc()of structpwc_device, and you don't even have any order-3 pages that are available. Are you running withCONFIG_COMPACTIONenabled?
Yes.
Quoted 2 lines
If not, enable it and try again, or get root and manually try to defragment memory by doing echo 1 >
/proc/sys/vm/compact_memory.
This needs to be done regularly, I assume?
Quoted 2 lines
If so, try killing a memory hogging process to free some memory and attempt to get some higher order pages available.
It's a 8GB box (7.5GB available to software) with modest load. Mostly email & web browsing, some torrents, nntp stuff, etc. Nothing weird. Around 4.8G in buffers/cache right now.
Why is this issue occurring?
Why did I not see this in the past?
Udo
Quoted 5 lines · Udo van den Heuvel
Quoted 2 lines
If not, enable it and try again, or get root and manually try to defragment memory by doing echo 1 >
/proc/sys/vm/compact_memory.This needs to be done regularly, I assume?
It's done when the page allocation initially fails, so it shouldn't need to be done explicitly by you. I was hoping to see if that helped to solve the problem and something like deferred compaction wasn't involved.
Quoted 7 lines
Quoted 2 lines
If so, try killing a memory hogging process to free some memory and attempt to get some higher order pages available.
It's a 8GB box (7.5GB available to software) with modest load. Mostly email & web browsing, some torrents, nntp stuff, etc. Nothing weird. Around 4.8G in buffers/cache right now.
That may be true, but you only have about 300MB available at the time of failure.
Quoted 3 lines
Why is this issue occurring?
Why did I not see this in the past?
I already added Hans to the cc to see if the struct has grown recently.