Skip to contentSkip to conversation

order 7 allocations from xt_recent

DISCUSSION Dave Jones 3 participants 7 messages Jan 3, 2013

Subsystems (2)
lore
Share & tools
lore

Dave Jones

author
#0 lore raw

We had a report from a user that shows this code trying to do enormous allocations, which isn't going to work too well..

iptables: page allocation failure: order:7, mode:0xc0d0
Pid: 2822, comm: iptables Not tainted 3.6.10-2.fc17.x86_64 #1

Call Trace:
 [<ffffffff8113130b>] warn_alloc_failed+0xeb/0x150
 [<ffffffff81616576>] ? __alloc_pages_direct_compact+0x17e/0x190
 [<ffffffff81135196>] __alloc_pages_nodemask+0x736/0x990
 [<ffffffff811710e0>] alloc_pages_current+0xb0/0x120
 [<ffffffff8113022a>] __get_free_pages+0x2a/0x80
 [<ffffffff811786d9>] kmalloc_order_trace+0x39/0xb0
 [<ffffffff8117ae3a>] __kmalloc+0x16a/0x1a0
 [<ffffffff8118aa7c>] ? mem_cgroup_bad_page_check+0x1c/0x30
 [<ffffffff81134563>] ? get_page_from_freelist+0x453/0x950
 [<ffffffffa007696e>] recent_mt_check.isra.6+0x16e/0x2c0 [xt_recent]
 [<ffffffffa0076b4b>] recent_mt_check_v0+0x6b/0xa0 [xt_recent]
 [<ffffffff8153fdda>] xt_check_match+0xaa/0x1e0
 [<ffffffff8153f3ab>] ? xt_find_match+0x11b/0x130
 [<ffffffff8153f3ab>] ? xt_find_match+0x11b/0x130
 [<ffffffff8159257c>] check_match+0x3c/0x50
 [<ffffffff81593ccb>] translate_table+0x39b/0x5b0
 [<ffffffff815956f3>] do_ipt_set_ctl+0x133/0x200
 [<ffffffff8153e10b>] nf_setsockopt+0x6b/0x90
 [<ffffffff8161f236>] ? _raw_spin_lock_bh+0x16/0x40
 [<ffffffff8154e41f>] ip_setsockopt+0x8f/0xa0
 [<ffffffff8156f49d>] raw_setsockopt+0x1d/0x30
 [<ffffffff814fcf14>] sock_common_setsockopt+0x14/0x20
 [<ffffffff814fc23c>] sys_setsockopt+0x7c/0xe0
 [<ffffffff816270e9>] system_call_fastpath+0x16/0x1b

which looks like it's this..

        t = kzalloc(sizeof(*t) + sizeof(t->iphash[0]) * ip_list_hash_size,
                    GFP_KERNEL);

Which is initialised thus..

        ip_list_hash_size = 1 << fls(ip_list_tot);

And ip_list_tot is 10000 in this case. Hmm ?

Complete report and setup described in his bug report at https://bugzilla.redhat.com/show_bug.cgi?id=890715

Dave

#1 lore raw
Quoted 38 lines · Dave Jones

We had a report from a user that shows this code trying to do enormous allocations, which isn't going to work too well..

iptables: page allocation failure: order:7, mode:0xc0d0
Pid: 2822, comm: iptables Not tainted 3.6.10-2.fc17.x86_64 #1

Call Trace:
 [<ffffffff8113130b>] warn_alloc_failed+0xeb/0x150
 [<ffffffff81616576>] ? __alloc_pages_direct_compact+0x17e/0x190
 [<ffffffff81135196>] __alloc_pages_nodemask+0x736/0x990
 [<ffffffff811710e0>] alloc_pages_current+0xb0/0x120
 [<ffffffff8113022a>] __get_free_pages+0x2a/0x80
 [<ffffffff811786d9>] kmalloc_order_trace+0x39/0xb0
 [<ffffffff8117ae3a>] __kmalloc+0x16a/0x1a0
 [<ffffffff8118aa7c>] ? mem_cgroup_bad_page_check+0x1c/0x30
 [<ffffffff81134563>] ? get_page_from_freelist+0x453/0x950
 [<ffffffffa007696e>] recent_mt_check.isra.6+0x16e/0x2c0 [xt_recent]
 [<ffffffffa0076b4b>] recent_mt_check_v0+0x6b/0xa0 [xt_recent]
 [<ffffffff8153fdda>] xt_check_match+0xaa/0x1e0
 [<ffffffff8153f3ab>] ? xt_find_match+0x11b/0x130
 [<ffffffff8153f3ab>] ? xt_find_match+0x11b/0x130
 [<ffffffff8159257c>] check_match+0x3c/0x50
 [<ffffffff81593ccb>] translate_table+0x39b/0x5b0
 [<ffffffff815956f3>] do_ipt_set_ctl+0x133/0x200
 [<ffffffff8153e10b>] nf_setsockopt+0x6b/0x90
 [<ffffffff8161f236>] ? _raw_spin_lock_bh+0x16/0x40
 [<ffffffff8154e41f>] ip_setsockopt+0x8f/0xa0
 [<ffffffff8156f49d>] raw_setsockopt+0x1d/0x30
 [<ffffffff814fcf14>] sock_common_setsockopt+0x14/0x20
 [<ffffffff814fc23c>] sys_setsockopt+0x7c/0xe0
 [<ffffffff816270e9>] system_call_fastpath+0x16/0x1b

which looks like it's this..

        t = kzalloc(sizeof(*t) + sizeof(t->iphash[0]) * ip_list_hash_size,
                    GFP_KERNEL);

Which is initialised thus..

Quoted 9 lines
        ip_list_hash_size = 1 << fls(ip_list_tot);

And ip_list_tot is 10000 in this case. Hmm ?

Complete report and setup described in his bug report at https://bugzilla.redhat.com/show_bug.cgi?id=890715

Dave

Yes, we had a report and a patch :

http://comments.gmane.org/gmane.linux.network/248216

I'll send it in a more formal way.

Thanks

author
#2 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Dave Jones

We had a report from a user that shows this code trying to do enormous allocations, which isn't going to work too well..
...
Which is initialised thus..

        ip_list_hash_size = 1 << fls(ip_list_tot);

And ip_list_tot is 10000 in this case. Hmm ?

Quoted 7 lines
Quoted 1 line

Complete report and setup described in his bug report at https://bugzilla.redhat.com/show_bug.cgi?id=890715

Yes, we had a report and a patch :

http://comments.gmane.org/gmane.linux.network/248216

I'll send it in a more formal way.

Ah! Excellent.

That 'check size and vmalloc/kmalloc accordingly' thing seems to be a pattern that comes up time and time again. Is it worth maybe making a more generic version of that instead of open-coding it each time it comes up ?

thanks,

Dave

author
#3 lore raw
Quoted 17 lines · Dave Jones
Quoted 16 lines · Eric Dumazet
Quoted 10 lines · Dave Jones

We had a report from a user that shows this code trying to do enormous allocations, which isn't going to work too well..
...
Which is initialised thus..

        ip_list_hash_size = 1 << fls(ip_list_tot);

And ip_list_tot is 10000 in this case. Hmm ?

Complete report and setup described in his bug report at https://bugzilla.redhat.com/show_bug.cgi?id=890715

Yes, we had a report and a patch :

http://comments.gmane.org/gmane.linux.network/248216

Quoted 7 lines
Quoted 1 line

I'll send it in a more formal way.

Ah! Excellent.

That 'check size and vmalloc/kmalloc accordingly' thing seems to be a pattern that comes up time and time again. Is it worth maybe making a more generic version of that instead of open-coding it each time it comes up ?

Something else that I'm puzzled by.

In the report above, it failed to allocate 512kb, but..

Node 0 Normal: 2388*4kB 347*8kB 1029*16kB 3512*32kB 29*64kB 2*128kB 1*256kB 5*512kB 1*1024kB 0*2048kB 0*4096kB = 147128kB

                                                                            ^^^^^^^^^^^^^^^^

Shouldn't the allocator have been able to satisfy that anyway ?

Dave

#4 lore raw
Quoted 28 lines · Dave Jones
Quoted 17 lines · Dave Jones
Quoted 16 lines · Eric Dumazet
Quoted 10 lines · Dave Jones

We had a report from a user that shows this code trying to do enormous allocations, which isn't going to work too well..
...
Which is initialised thus..

        ip_list_hash_size = 1 << fls(ip_list_tot);

And ip_list_tot is 10000 in this case. Hmm ?

Complete report and setup described in his bug report at https://bugzilla.redhat.com/show_bug.cgi?id=890715

Yes, we had a report and a patch :

http://comments.gmane.org/gmane.linux.network/248216

Quoted 7 lines
Quoted 1 line

I'll send it in a more formal way.

Ah! Excellent.

That 'check size and vmalloc/kmalloc accordingly' thing seems to be a pattern that comes up time and time again. Is it worth maybe making a more generic version of that instead of open-coding it each time it comes up ?

Something else that I'm puzzled by.

Quoted 9 lines

In the report above, it failed to allocate 512kb, but..

Node 0 Normal: 2388*4kB 347*8kB 1029*16kB 3512*32kB 29*64kB 2*128kB 1*256kB 5*512kB 1*1024kB 0*2048kB 0*4096kB = 147128kB

                                                                            ^^^^^^^^^^^^^^^^

Shouldn't the allocator have been able to satisfy that anyway ?

Dave

Might be something related to the CONFIG_COMPACTION=y and lumpy reclaim removal ?

Anyway, we keep a fraction of memory for ATOMIC allocations.

#6 lore raw
In reply to Dave Jones
Quoted 3 lines · Dave Jones

That 'check size and vmalloc/kmalloc accordingly' thing seems to be a pattern that comes up time and time again. Is it worth maybe making a more generic version of that instead of open-coding it each time it comes up ?

We had numerous discussions and patch submissions in the past, and this went nowhere. I am not sure anybody wants to spend more cycles on this.

Its sad, but true.

#5 lore raw
In reply to Eric Dumazet

Am 03.01.2013 19:00, schrieb Eric Dumazet:

Quoted 34 lines · Dave Jones
Quoted 33 lines · Dave Jones
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · Dave Jones
Quoted 10 lines

We had a report from a user that shows this code trying to do enormous allocations, which isn't going to work too well..
...
Which is initialised thus..

        ip_list_hash_size = 1 << fls(ip_list_tot);

And ip_list_tot is 10000 in this case. Hmm ?

Complete report and setup described in his bug report at https://bugzilla.redhat.com/show_bug.cgi?id=890715

Yes, we had a report and a patch :

http://comments.gmane.org/gmane.linux.network/248216

Quoted 7 lines
Quoted 1 line

I'll send it in a more formal way.

Ah! Excellent.

That 'check size and vmalloc/kmalloc accordingly' thing seems to be a pattern that comes up time and time again. Is it worth maybe making a more generic version of that instead of open-coding it each time it comes up ?

Something else that I'm puzzled by.

In the report above, it failed to allocate 512kb, but..

Node 0 Normal: 2388*4kB 347*8kB 1029*16kB 3512*32kB 29*64kB 2*128kB 1*256kB 5*512kB 1*1024kB 0*2048kB 0*4096kB = 147128kB

                                                                            ^^^^^^^^^^^^^^^^
Quoted 7 lines
Quoted 2 lines

Shouldn't the allocator have been able to satisfy that anyway ?

Might be something related to the CONFIG_COMPACTION=y and lumpy reclaim removal ?

Anyway, we keep a fraction of memory for ATOMIC allocations

on the machine there is even "vm.min_free_kbytes" set to 128 MB however, something goes terrible wrong if cache-pages leads to stack-traces about failed memory allocation

vm.swappiness = 0
vm.overcommit_memory = 1
vm.overcommit_ratio = 60
vm.vfs_cache_pressure = 30
vm.dirty_background_ratio = 15
vm.dirty_ratio = 40
vm.dirty_expire_centisecs = 1500
vm.dirty_writeback_centisecs = 1500
vm.zone_reclaim_mode = 0
vm.min_free_kbytes = 131072