ipset memory usage
Hello,
The ipset modules seems to eat a lot of memory :
with a test set of :
- 150 000 networks in a nethash
- 10 000 hosts in a iphash
ipset sometimes fails with out of memory errors :
"kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to increase size"
The VmAllocUsed variable from /proc/meminfo goes from 6Mo to 160Mo after loading sets.
Is this normal ? Is there some way to reduce memory usage ?
Thanks
Quoted 10 lines · tedemo
The ipset modules seems to eat a lot of memory :
with a test set of :
- 150 000 networks in a nethash
- 10 000 hosts in a iphash
ipset sometimes fails with out of memory errors :
"kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to increase size"The VmAllocUsed variable from
/proc/meminfogoes from 6Mo to 160Mo after loading sets.
Check out the real size of the hashes by searching the kernel log:0 "rehashing of set <foo> triggered: hashsize grows from <n> to <m>".
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Hello,
Quoted 6 lines · Jozsef Kadlecsik
Check out the real size of the hashes by searching the kernel log:0 "rehashing of set <foo> triggered: hashsize grows from <n> to <m>".
Best regards,
Jozsef
The size for the nethash, reported in the log is arround 20mo the first time it I run :
ipset --restore < ip.sets
I deleted each set (using ipset -X on each other) and the recreated them, the size of the nethash went to 40Mo
A few reloads later its size is 90Mo.
NB: The set definition file has been created manualy to avoid hundred thousand calls to "ipset -A" (i.e. without calling ipset --save). Can this cause troubles ?
Hi,
Quoted 6 lines · tedemo
Quoted 2 lines
Check out the real size of the hashes by searching the kernel log:0 "rehashing of set <foo> triggered: hashsize grows from <n> to <m>".
The size for the nethash, reported in the log is arround 20mo the first time it I run :
ipset --restore < ip.sets
When the hash size is about 20 million entries, that alone requires ~80MB physical memory. If resizing is triggered, at least two times of that amount of memory is needed temporarily.
Quoted 4 lines
I deleted each set (using ipset -X on each other) and the recreated them, the size of the nethash went to 40Mo
A few reloads later its size is 90Mo.
Please write it exactly what you do and what happens. Do you create, fill up and delete the *same* sets with exactly the *same* content, in a loop?
Quoted 3 lines
NB: The set definition file has been created manualy to avoid hundred thousand calls to "ipset -A"
(i.e. without calling ipset --save). Can this cause troubles ?
No.
What is your kernel and ipset versions?
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Quoted 5 lines · Jozsef Kadlecsik
Please write it exactly what you do and what happens. Do you create, fill up and delete the *same* sets with exactly the *same* content, in a loop?
Yes I execute those commands in a loop on the same set (a nethash of 150 000 entries).
the commands in the loop are :
ipset -X neth
ipset --restore < ip.sets
The obtained results are :
iteration 1: neth size 25856860 VmallocUsed 108004 kB
iteration 2: neth size 25856860 VmallocUsed 108004 kB
iteration 3: neth size 38785290 VmallocUsed 158168 kB
iteration 6: neth size 25856860 VmallocUsed 108004 kB
iteration 9: neth size 58177935 VmallocUsed 235844 kB
This seems a lot for 0.15 million entries.
Are the 230Mo of Vmalloc'ed memory really used by the kernel ?
Quoted 4 lines
What is your kernel and ipset versions?
ipset v2.2.2 Protocol version 2
iptables v1.3.2
Linux 2.6.13 i686
Quoted 17 lines · tedemo
Quoted 3 lines
Please write it exactly what you do and what happens. Do you create, fill up and delete the *same* sets with exactly the *same* content, in a loop?
Yes I execute those commands in a loop on the same set (a nethash of 150 000 entries).
the commands in the loop are :
ipset -X neth
ipset --restore < ip.setsThe obtained results are :
iteration 1: neth size 25856860 VmallocUsed 108004 kB
iteration 2: neth size 25856860 VmallocUsed 108004 kB
iteration 3: neth size 38785290 VmallocUsed 158168 kB
iteration 6: neth size 25856860 VmallocUsed 108004 kB
iteration 9: neth size 58177935 VmallocUsed 235844 kB
Quoted 2 lines
This seems a lot for 0.15 million entries.
Are the 230Mo of Vmalloc'ed memory really used by the kernel ?
25-58 million hash entries isn't really needed to store 150.000 IP addresses/netblocks!
How do you create the set neth in ip.sets? What is the exact command?
Coul you send me the file ip.sets in private for testings?
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Quoted 8 lines · Jozsef Kadlecsik
25-58 million hash entries isn't really needed to store 150.000 IP addresses/netblocks!
How do you create the set neth in ip.sets? What is the exact command?
Coul you send me the file ip.sets in private for testings?
Unfortunatly no, because it seems that your mail server banned the one of my isp ...
Anyway, I could reproduce this with the set generated by the attached perl script.
Regards
print "-N neth nethash --hashsize 1024 --probes 2 --resize 50\n"; for ($i = 1; $i < 3; $i++) {
for ($j = 0; $j < 255; $j++) {
for ($k = 0; $k < 255; $k++) {
print "-A neth $i.$j.$k.0/24\n";
} } };
print "COMMIT\n";
Quoted 5 lines · Jozsef Kadlecsik
When the hash size is about 20 million entries, that alone requires ~80MB physical memory. If resizing is triggered, at least two times of that amount of memory is needed temporarily.
The problem was comming from the --probes parameter which was set to 2 ! Increasing its value to 10, reduced the hashsize up to a factor 100.
Thanks !
Quoted 6 lines · tedemo
Quoted 4 lines
When the hash size is about 20 million entries, that alone requires ~80MB physical memory. If resizing is triggered, at least two times of that amount of memory is needed temporarily.
The problem was comming from the --probes parameter which was set to 2 ! Increasing its value to 10, reduced the hashsize up to a factor 100.
Thus you trade memory for speed: lookups in the set will be ten times slower.
Still I don't get how does it come that the initial hash size was so huge.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary