memory usage in module geoip (probably)
Hello!
I've noticed high size of kmalloc-512/2048 on my vps after a one, two weeks of work. Example:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 83440 83411 99% 0.57K 5960 14 47680K kmalloc-512 21615 21604 99% 2.07K 1441 15 46112K kmalloc-2048 103776 103765 99% 0.31K 8648 12 34592K skbuff_head_cache
I've done a little investigation (trial by error method) to discover which process keep data in kmalloc. It looks those kmalloc are used by modules from xtables-addons-1.39. I'm using ipset6, geoip, sysrq, tarpit. I greped sources for string "kmalloc", kmalloc is used by sysrq and geoip (in some other modules too but i don't use them).
I'd like to ask is such usage of memory ok? Which module could take so much precious ram?:) (I suspect geoip). How much memory geoip can use in worse case (db with ipv4 and ipv6)? Can i freed this cache without reboot? (I tried flush rules, rmmod but without succes).
Thank you for any answers!
Marcin
P.S. Is any other method to check which process/module/etc uses kmallocs?
Quoted 6 lines · Marcin MirosÅ‚aw
Hello!
I've noticed high size of kmalloc-512/2048 on my vps after a one, two weeks of work. Example:OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 83440 83411 99% 0.57K 5960 14 47680K kmalloc-512 21615 21604 99% 2.07K 1441 15 46112K kmalloc-2048
Quoted 7 lines
103776 103765 99% 0.31K 8648 12 34592K
skbuff_head_cacheI've done a little investigation (trial by error method) to discover which process keep data in kmalloc. It looks those kmalloc are used by modules from xtables-addons-1.39. I'm using ipset6, geoip, sysrq, tarpit. I greped sources for string "kmalloc", kmalloc is used by sysrq and geoip (in some other modules too but i don't use them).
geoip uses vmalloc for its huge allocations, so that can't be it. The massive size of skbuff_head_cache would point towards there being a lot of dead skbs being held, which - in my nose - would smell of tarpit. Did you properly feed all packets that you tarpited also to -j CT --notrack?
Quoted 3 lines
I'd like to ask is such usage of memory ok? Which module could take so much precious ram?:) (I suspect geoip). How much memory geoip can use in worse case (db with ipv4 and ipv6)?
The on-disk files for geoip are loaded verbatim into the kernel, so that would be only ~6 MB at worst for geoip, not 40+.
W dniu 2012-01-20 18:05, Jan Engelhardt napisał(a):
Quoted 5 lines
geoip uses vmalloc for its huge allocations, so that can't be it. The massive size of
skbuff_head_cachewould point towards there being a lot of dead skbs being held, which - in my nose - would smell of tarpit. Did you properly feed all packets that you tarpited also to -j CT --notrack?
Surely i didn't use notrack;) I has problem how to use it, i'd like to tarpit packets at the end of INPUT at filter table, notrack i can use only in raw table. Have you got idea about kmallo-512 and kmalloc-2048? I'll try every single module and watch kmalloc usage. It will take a couple of days for each one. Maybe i'll isolate which module uses kmallocs. I suspect xtables because without them i didn't notice such situation.
Quoted 6 lines
Quoted 3 lines
I'd like to ask is such usage of memory ok? Which module could take so much precious ram?:) (I suspect geoip). How much memory geoip can use in worse case (db with ipv4 and ipv6)?
The on-disk files for geoip are loaded verbatim into the kernel, so that would be only ~6 MB at worst for geoip, not 40+.
Thank you for all information.
Marcin
Quoted 9 lines · Marcin Mirosław
W dniu 2012-01-20 18:05, Jan Engelhardt napisał(a):
Quoted 4 lines
geoip uses vmalloc for its huge allocations, so that can't be it. The massive size of
skbuff_head_cachewould point towards there being a lot of dead skbs being held, which - in my nose - would smell of tarpit. Did you properly feed all packets that you tarpited also to -j CT --notrack?Surely i didn't use notrack;) I has problem how to use it, i'd like to tarpit packets
at the end of INPUT at filter table, notrack i can use only in raw table.
TARPIT does not have much of a use for random ports without any services because it is specific to clients sending data. You should use DELUDE at the end of the chain, also because it does not keep any connections around like tarpit.
W dniu 2012-01-20 22:00, Jan Engelhardt pisze:
Quoted 4 lines
TARPIT does not have much of a use for random ports without any services because it is specific to clients sending data. You should use DELUDE at the end of the chain, also because it does not keep any connections around like tarpit.
Hello Jan.
Meseems TARPIT is what i need, i'd like to slow down (a little) bots looking for mssql and other MS specific soft.
I've made quick test, it looks when TARPIT is in chain then size of all three slabs (kmalloc-512/2048, skbuff_head_cache) slowly grow up. Without TARPIT its size is almost constant. It appears on host:
domU, FV, 3.1.8-hardened (i have such situation since some time, with older kernels too) , x86_64, xtables-addons-1.37 and 1.39.
Other host doesn't suffer such problem:
bare metal, 3.1.5-hardened, i686, xtables-addons-1.37
How can i track down what is the reason of such situation? Thank you.
W dniu 2012-01-22 21:38, Marcin Miros³aw pisze:
Quoted 2 lines
Other host doesn't suffer such problem:
bare metal, 3.1.5-hardened, i686, xtables-addons-1.37
I forgot mention, those kernel has patch taken from http://www.ssi.bg/~ja/#routes
Quoted 9 lines · Marcin MirosÅ‚aw
W dniu 2012-01-20 22:00, Jan Engelhardt pisze:
Quoted 4 lines
TARPIT does not have much of a use for random ports without any services because it is specific to clients sending data. You should use DELUDE at the end of the chain, also because it does not keep any connections around like tarpit.
Hello Jan.
Meseems TARPIT is what i need, i'd like to slow down (a little) bots looking for mssql and other MS specific soft.
Then you should use
-t raw -A PREROUTING -p tcp --dport ms-sql-s -j CT --notrack
if your service is not exposed to the internet.
Quoted 1 line
How can i track down what is the reason of such situation?
Also compare output of ss -at.
W dniu 20.01.2012 17:12, Marcin Mirosław pisze:
Quoted 14 lines
Hello!
I've noticed high size of kmalloc-512/2048 on my vps after a one, two weeks of work. Example:OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 83440 83411 99% 0.57K 5960 14 47680K kmalloc-512 21615 21604 99% 2.07K 1441 15 46112K kmalloc-2048 103776 103765 99% 0.31K 8648 12 34592K skbuff_head_cacheI've done a little investigation (trial by error method) to discover which process keep data in kmalloc. It looks those kmalloc are used by modules from xtables-addons-1.39. I'm using ipset6, geoip, sysrq, tarpit. I greped sources for string "kmalloc", kmalloc is used by sysrq and geoip (in some other modules too but i don't use them).
Quoted 4 lines
I'd like to ask is such usage of memory ok? Which module could take so much precious ram?:) (I suspect geoip). How much memory geoip can use in worse case (db with ipv4 and ipv6)? Can i freed this cache without reboot? (I tried flush rules, rmmod but without succes).
With xtables-addons-1.46 I don't have such issue, probably commit 4ff5a8fbf62b75b17194a5cd324c08b8fa0f08bf in TARPIT resolves my problem. Thanks:)