btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
My server died last night during a btrfs send/receive to a btrfs radi5 array
Here are the logs. Is this anything known or with a possible workaround?
Thanks,
Marc
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1 Hardware name: System manufacturer P5KC/P5KC, BIOS 0502 05/24/2007 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffff88007ff7ce00 0000000000000000 0000000200000000 0000003000000000 ffff88007ff7ce00 Call Trace: [<ffffffff816090b3>] dump_stack+0x4e/0x7a [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854 [<ffffffff81014650>] ? dma_generic_alloc_coherent+0xa7/0x11c [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas] [<ffffffff812f9fa2>] ? blkg_path.isra.80.constprop.90+0x17/0x38 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas] [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas] [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas] [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas] [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e [<ffffffff814964da>] scsi_request_fn+0x4da/0x507 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb [<ffffffff812dfa7d>] submit_bio+0x112/0x131 [<ffffffff8128274c>] rmw_work+0x112/0x162 [<ffffffff8125073f>] worker_loop+0x168/0x4d8 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283 [<ffffffff8106bc56>] kthread+0xae/0xb6 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 171
CPU 1: hi: 186, btch: 31 usd: 190
active_anon:17298 inactive_anon:21061 isolated_anon:0
active_file:67491 inactive_file:94189 isolated_file:32
unevictable:1260 dirty:38914 writeback:49596 unstable:0
free:15999 slab_reclaimable:8198 slab_unreclaimable:9741
mapped:12981 shmem:1661 pagetables:2711 bounce:0
free_cma:0
Node 0 DMA free:8084kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:2040kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56080kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:374588kB unevictable:4940kB isolated(anon):0kB isolated(file):128kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197112kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38628kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 85*4kB (UEM) 22*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8100kB
Node 0 DMA32: 13004*4kB (M) 16*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56240kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
164139 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1 Hardware name: System manufacturer P5KC/P5KC, BIOS 0502 05/24/2007 0000000000000000 ffff88000549d690 ffffffff816090b3 0000000000000000 ffff88000549d718 ffffffff811037b0 00000001fffffffe ffff88000549d6c8 ffffffff8160e9c4 0000000200000000 0000003000000000 ffff88007ff7ce00 Call Trace: [<ffffffff816090b3>] dump_stack+0x4e/0x7a [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125 [<ffffffff8160e9c4>] ? _raw_spin_trylock+0x20/0x50 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854 [<ffffffff81093ce9>] ? console_unlock+0x2f6/0x302 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas] [<ffffffff811063d7>] ? get_page_from_freelist+0x549/0x71d [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas] [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas] [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas] [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas] [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e [<ffffffff814964da>] scsi_request_fn+0x4da/0x507 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b [<ffffffff812fc16d>] cfq_insert_request+0x391/0x3b5 [<ffffffff812e002f>] ? perf_trace_block_rq_with_error+0x45/0x14f [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e [<ffffffff812dc08c>] __elv_add_request+0x1fc/0x276 [<ffffffff812e1c6c>] blk_queue_bio+0x237/0x256 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb [<ffffffff812dfa7d>] submit_bio+0x112/0x131 [<ffffffff8128274c>] rmw_work+0x112/0x162 [<ffffffff8125073f>] worker_loop+0x168/0x4d8 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283 [<ffffffff8106bc56>] kthread+0xae/0xb6 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 179
CPU 1: hi: 186, btch: 31 usd: 213
active_anon:17298 inactive_anon:21061 isolated_anon:0
active_file:67491 inactive_file:93709 isolated_file:32
unevictable:1260 dirty:38914 writeback:49627 unstable:0
free:16338 slab_reclaimable:8198 slab_unreclaimable:9741
mapped:12981 shmem:1661 pagetables:2711 bounce:0
free_cma:0
Node 0 DMA free:8084kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:2040kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:57352kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:372680kB unevictable:4940kB isolated(anon):0kB isolated(file):116kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38628kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 85*4kB (UEM) 22*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8100kB
Node 0 DMA32: 13283*4kB (M) 16*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 57356kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163658 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1 Hardware name: System manufacturer P5KC/P5KC, BIOS 0502 05/24/2007 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffff880000000000 0000000000000046 0000000200000000 0000003000000000 ffff88007ff7ce00 Call Trace: [<ffffffff816090b3>] dump_stack+0x4e/0x7a [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854 [<ffffffff811063d7>] ? get_page_from_freelist+0x549/0x71d [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas] [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas] [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas] [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas] [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas] [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e [<ffffffff814964da>] scsi_request_fn+0x4da/0x507 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb [<ffffffff812dfa7d>] submit_bio+0x112/0x131 [<ffffffff8128274c>] rmw_work+0x112/0x162 [<ffffffff8125073f>] worker_loop+0x168/0x4d8 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283 [<ffffffff8106bc56>] kthread+0xae/0xb6 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 169
CPU 1: hi: 186, btch: 31 usd: 185
active_anon:17298 inactive_anon:21061 isolated_anon:0
active_file:67491 inactive_file:93277 isolated_file:32
unevictable:1260 dirty:38914 writeback:49627 unstable:0
free:16250 slab_reclaimable:8198 slab_unreclaimable:9762
mapped:12981 shmem:1661 pagetables:2711 bounce:0
free_cma:0
Node 0 DMA free:8080kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:1980kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56920kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:371128kB unevictable:4940kB isolated(anon):0kB isolated(file):128kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38712kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 86*4kB (UEM) 19*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8080kB
Node 0 DMA32: 13210*4kB (M) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56984kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163317 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1 Hardware name: System manufacturer P5KC/P5KC, BIOS 0502 05/24/2007 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffffffff00000029 ffffffff81e8fe40 0000000200000000 0000003000000000 ffff88007ff7ce00 Call Trace: [<ffffffff816090b3>] dump_stack+0x4e/0x7a [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas] [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas] [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas] [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas] [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas] [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e [<ffffffff814964da>] scsi_request_fn+0x4da/0x507 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb [<ffffffff812dfa7d>] submit_bio+0x112/0x131 [<ffffffff8128274c>] rmw_work+0x112/0x162 [<ffffffff8125073f>] worker_loop+0x168/0x4d8 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283 [<ffffffff8106bc56>] kthread+0xae/0xb6 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 169
CPU 1: hi: 186, btch: 31 usd: 185
active_anon:17298 inactive_anon:21061 isolated_anon:0
active_file:67491 inactive_file:93298 isolated_file:0
unevictable:1260 dirty:38914 writeback:49627 unstable:0
free:16250 slab_reclaimable:8198 slab_unreclaimable:9762
mapped:12981 shmem:1661 pagetables:2711 bounce:0
free_cma:0
Node 0 DMA free:8080kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:1980kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56920kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:371212kB unevictable:4940kB isolated(anon):0kB isolated(file):0kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38712kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 86*4kB (UEM) 19*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8080kB
Node 0 DMA32: 13210*4kB (M) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56984kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163317 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 8 lines · Marc MERLIN
My server died last night during a btrfs send/receive to a btrfs radi5 array
Here are the logs. Is this anything known or with a possible workaround?
Thanks,
Marcbtrfs-rmw-2: page allocation failure: order:1, mode:0x8020
This is an order 1 atomic allocation from the mvs driver, we really should not be depending on that to get IO done. A quick search and it looks like we're allocating MVS_SLOT_BUF_SZ (8192) bytes.
You could try bumping the lowmem reserves.
-chris
Quoted 9 lines · Chris Mason
Quoted 8 lines · Marc MERLIN
My server died last night during a btrfs send/receive to a btrfs radi5 array
Here are the logs. Is this anything known or with a possible workaround?
Thanks,
Marc
Quoted 7 lines
Quoted 1 line
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
This is an order 1 atomic allocation from the mvs driver, we really should not be depending on that to get IO done. A quick search and it looks like we're allocating
MVS_SLOT_BUF_SZ(8192) bytes.You could try bumping the lowmem reserves.
Thanks for the info.
So for now, I haveCONFIG_X86_RESERVE_LOW=64
This is the option we're talking about, right?
Should I double it?
For now, I have the copy running again, and it's been going for 8 hours without failure on the old kernel but of course that doesn't mean my 2TB copy will complete without hitting the bug again.
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 24 lines · Marc MERLIN
Quoted 10 lines · Chris Mason
Quoted 4 lines · Marc MERLIN
My server died last night during a btrfs send/receive to a btrfs radi5 array
Here are the logs. Is this anything known or with a possible
workaround?
Quoted 4 lines
Thanks,
MarcQuoted 7 lines
Quoted 1 line
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
This is an order 1 atomic allocation from the mvs driver, we really should not be depending on that to get IO done. A quick search and it looks like we're allocating
MVS_SLOT_BUF_SZ(8192) bytes.You could try bumping the lowmem reserves.
Thanks for the info.
So for now, I have
CONFIG_X86_RESERVE_LOW=64
Quoted 7 lines
This is the option we're talking about, right?
Should I double it?
For now, I have the copy running again, and it's been going for 8 hours without failure on the old kernel but of course that doesn't mean my 2TB copy will complete without hitting the bug again.
Sorry, I misspoke, you should bump /proc/sys/vm/min_free_kbytes. Honestly though, it¹s just a bug in the mvs driver. Atomic 8K allocations are doomed to fail eventually.
The driver should either busy loop until the allocation completes (really not a great choice), gracefully deal with the failure (looks tricky), or preallocate the space (like the rest of the block layer).
-chris
Quoted 9 lines · Chris Mason
Quoted 5 lines
Should I double it?
For now, I have the copy running again, and it's been going for 8 hours without failure on the old kernel but of course that doesn't mean my 2TB copy will complete without hitting the bug again.
Sorry, I misspoke, you should bump
/proc/sys/vm/min_free_kbytes. Honestly though, it¹s just a bug in the mvs driver. Atomic 8K allocations are doomed to fail eventually.
Gotcha
polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes 45056 polgara:/mnt/btrfs_backupcopy# echo 100000 > /proc/sys/vm/min_free_kbytes polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes 100000 polgara:/mnt/btrfs_backupcopy#
Quoted 3 lines
The driver should either busy loop until the allocation completes (really not a great choice), gracefully deal with the failure (looks tricky), or preallocate the space (like the rest of the block layer).
Gotcha. I'll report this to the folks maintaining the marvel driver.
So just to make sure I got you right, although the page allocation failure was shown in btrfs, it's really the underlying marvel driver at fault here, and there isn't really anything to change on the btrfs side, correct?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 26 lines · Marc MERLIN
Quoted 11 lines · Chris Mason
Quoted 4 lines
Should I double it?
For now, I have the copy running again, and it's been going for 8 hours without failure on the old kernel but of course that doesn't mean my
2TB
Quoted 1 line
copy will complete without hitting the bug again.
Sorry, I misspoke, you should bump
/proc/sys/vm/min_free_kbytes. Honestly
though, it¹s just a bug in the mvs driver. Atomic 8K allocations are doomed to fail eventually.Gotcha
polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes 45056 polgara:/mnt/btrfs_backupcopy# echo 100000 > /proc/sys/vm/min_free_kbytes polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes 100000 polgara:/mnt/btrfs_backupcopy#Quoted 4 lines
The driver should either busy loop until the allocation completes (really
not a great choice), gracefully deal with the failure (looks tricky), or preallocate the space (like the rest of the block layer).
Quoted 6 lines
Gotcha. I'll report this to the folks maintaining the marvel driver.
So just to make sure I got you right, although the page allocation failure was shown in btrfs, it's really the underlying marvel driver at fault here,
and there isn't really anything to change on the btrfs side, correct?
The process is a btrfs worker, and the IO was started by btrfs, but the allocation failure is all inside the mvs driver. There’s even the printk in there from mvs about the allocation failing.
The only reason it’s btrfs instead of a regular process is because for raid5/6 the rmw is farmed out to helper threads.
-chris
�淸�{.n�+돴윯돪�+%듚�깁負쪐w��{.n�+돴쪐{깰팳~穹왲^n뇊⊆쫧�곷h솳鈺�&｛�z요z받쀺+��+zf＂톒쉱�~넮녬i����묎�z_溫�j:+v돣�)山�m
Chris Mason posted on Thu, 20 Mar 2014 01:01:35 +0000 as excerpted:
Quoted 10 lines
Message-ID: <CF4FB5F3.2116%[email protected]>
Accept-Language: en-US
Content-Type: text/plain; charset=euc-kr
Content-Transfer-Encoding: base64
Content-Language: en-USQuoted 3 lines
Quoted 3 lines
Sorry, I misspoke, you should bump
/proc/sys/vm/min_free_kbytes. Honestly though, it©ös just a bug in the mvs driver. Atomic 8K allocations are doomed to fail eventually.
Quoted 7 lines
The process is a btrfs worker, and the IO was started by btrfs, but the allocation failure is all inside the mvs driver. There¡¯s even the printk in there from mvs about the allocation failing.
-chris
N§²æìržyúèØb²X¬¶Ç§vØ^)Þº{.nÇ+·¥{±nÚß²)í æèw* jg¬±š ¶Ý¢
j/êäz¹Þà2ÞšèÚ&¢)ß¡«a¶Úþø ®G«éh®æj:+všwèÙ¥
Chris, you might wish to take a look at this and/or have one of the FB techs familiar with your mail transport layers (and/or your mail client, and/or perhaps it's vger's list-serv bot) look at it. That list-sig came thru as garbage at least here, and your "it's" and "there's" appear to have strange apostrophes as well. As you can see I also included the headers I think might be relevant, plus the Message-ID.
The problem wasn't a big one here, but if it starts corrupting patches or something too...
I'd guess it has something to do with that content-type charset=euc-kr in the headers, which looks really strange combined with the accept-language and content-language both being en-US.
The other messages of yours (with an uncorrupted list sig) I checked had a content-type charset=iso-8859-1 header, which seems to be most common in English messages, anyway. Why this one had euc-kr I don't know.
However, it's also worth noting that the third-level quoting above, also from you, has an "it's", and when I checked /that/ message, the apostrophe was a super-script-1, as it was when quoted in Marc's reply, but that got changed to the copyright symbol (plus something else) in your quote, which at least as I'm posting the quote, is showing the same way.
Meanwhile, the apostrophe in the "there's" in your message's new content (which is thus first-level quote in this message) is different, an i followed by an over-line. I'd /guess/ that's what actually triggered the Korean (?) charset in your message, in ordered to handle that or perhaps some other character I missed, while your other messages are iso-8859-1, which thus changed the auto-inserted list-sig into garbage, since the list-bot presumably inserted it as the usual iso-8859-1 while your message claimed to be in eur-kr.
(It'll be interesting to see if my message, with both quotes, looks the same when I read it on the list via gmane, as it does when I send it, or if it's further garbled, and if my charset gets set to something exotic too. FWIW, my client, which should be visible in my headers, is pan, via gmane.org's list2news service.)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 22 lines · Duncan
Chris Mason posted on Thu, 20 Mar 2014 01:01:35 +0000 as excerpted:
Quoted 10 lines
Message-ID: <CF4FB5F3.2116%[email protected]>
Accept-Language: en-US
Content-Type: text/plain; charset=euc-kr
Content-Transfer-Encoding: base64
Content-Language: en-USQuoted 3 lines
Quoted 3 lines
Sorry, I misspoke, you should bump
/proc/sys/vm/min_free_kbytes. Honestly though, it©ös just a bug in the mvs driver. Atomic 8K allocations are doomed to fail eventually.Quoted 7 lines
The process is a btrfs worker, and the IO was started by btrfs, but the allocation failure is all inside the mvs driver. There¡¯s even the printk in there from mvs about the allocation failing.
-chris
N§²æìržyúèØb²X¬¶Ç§vØ^)Þº{.nÇ+·¥{±nÚß²)í æèw* jg¬±š ¶Ý¢
j/êäz¹Þà2ÞšèÚ&¢)ß¡«a¶Úþø ®G«éh®æj:+všwèÙ¥
Quoted 7 lines
Chris, you might wish to take a look at this and/or have one of the FB techs familiar with your mail transport layers (and/or your mail client, and/or perhaps it's vger's list-serv bot) look at it. That list-sig came thru as garbage at least here, and your "it's" and "there's" appear to have strange apostrophes as well. As you can see I also included the headers I think might be relevant, plus the Message-ID.
So the list-sig is a vger feature. It just tacks on the list sig regardless of the encoding of the message, which generates garbage some times.
The rest sounds like a facebook feature. I've got thunderbird configured to force saner encodings, hopefully it'll be more consistent now.
Thanks
-chris