Performance issue due to constant "modprobes"
Hi, I am using a relatively low powered (embedded) platform and I have a significant performance problem due to slow "modprobe" performance.
I have my kernel compiled without modules. My modprobe takes a little under 1ms to execute. "iptables" appears to try and modprobe some 21 match/target modules. As a result, even "iptables -h" takes around 14ms to run. This is adding some substantial time to my firewall setup time (hacking out the modprobes reduces run time from the 14ms to near zero, ie it's 90+% of my runtime)
I have dug through the code a bit and the first thing I notice is that there is no --modprobe option actually parsed for, and the undocumented "-M" option doesn't appear to pass through to xtables.c? (I thought about simply lying about the modprobe binary name)
My next thought was to collect all the modprobes and run them with a single execution (modprobe -a). However, it's not clear to me whether it's important that the modprobe occurs in the middle of xtables.c / compatible_revision() ?
The final thought is whether it's possible to notice that a module is already loaded and skip the modprobe call altogether? (/proc/modules is not enough because the module could be built into the kernel)
Does someone have any ideas on how I can finesse these constant (and expensive in my case) modprobes each time we run the iptables command?
Thanks
Ed W
Quoted 6 lines · Ed W
Hi, I am using a relatively low powered (embedded) platform and I have a significant performance problem due to slow "modprobe" performance.
I have my kernel compiled without modules. My modprobe takes a little under 1ms to execute. "iptables" appears to try and modprobe some 21 match/target modules.
This is your kernel calling modprobe through ____call_usermodehelper in response to inquiries about modules' ABIs. This will also happen if you removed modprobing from iptables - the latter of which is only used to load ip_tables.ko itself, since that is the only thing that currently is not fully autoloadable by the kernel.
Quoted 4 lines
I have dug through the code a bit and the first thing I notice is that there is no --modprobe option actually parsed for, and the undocumented "-M" option doesn't appear to pass through to
xtables.c? (I thought about simply lying about the modprobe binary name)
ip6tables.c:do_command6, near case 'M'.
Quoted 4 lines
My next thought was to collect all the modprobes and run them with a single execution (modprobe -a). However, it's not clear to me whether it's important that the modprobe occurs in the middle of
xtables.c/compatible_revision()?
To determine which ABI/revision is available on the kernel side, and based upon the result, display what options the extensions can use when you invoke -h.
Quoted 3 lines
The final thought is whether it's possible to notice that a module is already loaded and skip the modprobe call altogether? (
/proc/modulesis not enough because the module could be built into the kernel)
If ip_tables.ko is already loaded, iptc_init will not fail (per the command above xtables_load_ko(xtables_modprobe_program, false), and thus, modprobe will not be called - at least not by iptables.
Quoted 20 lines · Ed W
Hi, I am using a relatively low powered (embedded) platform and I have a significant performance problem due to slow "modprobe" performance.
I have my kernel compiled without modules. My modprobe takes a little under 1ms to execute. "iptables" appears to try and modprobe some 21 match/target modules. As a result, even "iptables -h" takes around 14ms to run. This is adding some substantial time to my firewall setup time (hacking out the modprobes reduces run time from the 14ms to near zero, ie it's 90+% of my runtime)
I have dug through the code a bit and the first thing I notice is that there is no --modprobe option actually parsed for, and the undocumented "-M" option doesn't appear to pass through to
xtables.c? (I thought about simply lying about the modprobe binary name)My next thought was to collect all the modprobes and run them with a single execution (modprobe -a). However, it's not clear to me whether it's important that the modprobe occurs in the middle of
xtables.c/compatible_revision()?
Quoted 6 lines
The final thought is whether it's possible to notice that a module is already loaded and skip the modprobe call altogether? (
/proc/modulesis not enough because the module could be built into the kernel)Does someone have any ideas on how I can finesse these constant (and expensive in my case) modprobes each time we run the iptables command?
Could you try with an iptables built from iptables git master branch? I believe a recent change I submitted (delayed initialization of target/matches to prevent module autoloading) may actually fix your problem.
- Maciej
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Jan Engelhardt
Quoted 3 lines
I have my kernel compiled without modules. My modprobe takes a little under 1ms to execute. "iptables" appears to try and modprobe some 21 match/target modules.
This is your kernel calling modprobe through
I think not, or that you mean in a more roundabout fashion than I do?
Quoted 6 lines
Quoted 4 lines
I have dug through the code a bit and the first thing I notice is that there is no --modprobe option actually parsed for, and the undocumented "-M" option doesn't appear to pass through to
xtables.c? (I thought about simply lying about the modprobe binary name)ip6tables.c:do_command6, near case 'M'.
I see the code, but "iptables -M /dev/null -h" still runs /sbin/modprobe (lots)
Quoted 7 lines
Quoted 3 lines
The final thought is whether it's possible to notice that a module is already loaded and skip the modprobe call altogether? (
/proc/modulesis not enough because the module could be built into the kernel)If ip_tables.ko is already loaded,
iptc_initwill not fail (per the command abovextables_load_ko(xtables_modprobe_program, false), and thus, modprobe will not be called - at least not by iptables.
Sure, but everything derived from xtables.c calls modprobe?
I'm sorry if I'm misunderstanding "who" calls modprobe, but the point is that you can do a simple strace on "iptables -h" and without Maciej' patch you will see a boatload of modprobe's...
Even with his patch then I still seem to get a load of modprobe's where extra xtable matches are needed, and in my case (500Mhz processor), modprobe is slow and also unnecessary because (nearly) everything is compiled into the kernel
Avoiding modprobe would be very desirable in my case. Any ideas? (For the moment I have just patched out the xtables.c insmod function, but it's not very neat)
Thanks all
Ed W
Quoted 7 lines · Maciej Żenczykowski
Quoted 2 lines
Does someone have any ideas on how I can finesse these constant (and expensive in my case) modprobes each time we run the iptables command?
Could you try with an iptables built from iptables git master branch? I believe a recent change I submitted (delayed initialization of target/matches to prevent module autoloading) may actually fix your problem.
Thanks - very helpful!
It was easiest for me to patch my iptables with just your commit and my results are very promising:
Starting "shorewall"
- using busybox modprobe + released iptables = several minutes...
- module-init-tools + released iptables = 12s
- module-init-tools + your commit = 7.7s
- module-init-tools + patching out modprobe completely = 4.9s
So, whilst your patch has a huge positive benefit, I'm still seeing a substantial amount of cpu going to useless modprobing.
I don't see an immediate solution, *unless* there is some way to ask the kernel if some module is already compiled in? I don't immediately see that this is possible and google didn't turn anything up? I guess the various xtables modules could export something that allows them to be detected as loaded, but I sense that this is unlikely to be an acceptable patch unless others have shown that there is a performance problem?
Of the rest of my 4.9s, 97% of that is waiting for iptables and tc to do stuff. I need to profile further to see where the delays are though
Thanks for your commit above - extremely helpful - grateful if you might consider whether there is some way to avoid any modprobes at all? (Note that the -M option appears not to work in iptables at present?)
Thanks
Ed W
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Ed W
Starting "shorewall"
- using busybox modprobe + released iptables = several minutes...
- module-init-tools + released iptables = 12s
- module-init-tools + your commit = 7.7s
- module-init-tools + patching out modprobe completely = 4.9sSo, whilst your patch has a huge positive benefit, I'm still seeing a substantial amount of cpu going to useless modprobing.
Which is only natural, because the kernel loads stuff. If you have a problem with time (why so?) you should <*> the xtables modules, and just leave iptables in its origin state.
Quoted 2 lines
Of the rest of my 4.9s, 97% of that is waiting for iptables and tc to do stuff. I need to profile further to see where the delays are though
Just how often are you calling iptables, anyway?
Quoted 8 lines · Jan Engelhardt
Quoted 6 lines · Ed W
Starting "shorewall"
- using busybox modprobe + released iptables = several minutes...
- module-init-tools + released iptables = 12s
- module-init-tools + your commit = 7.7s
- module-init-tools + patching out modprobe completely = 4.9s
Quoted 6 lines
Quoted 2 lines
So, whilst your patch has a huge positive benefit, I'm still seeing a substantial amount of cpu going to useless modprobing.
Which is only natural, because the kernel loads stuff. If you have a problem with time (why so?) you should <*> the xtables modules, and just leave iptables in its origin state.
I'm sorry that I haven't managed to explain my situation clearly. However, I'm trying very hard to explain that I *have* got <*> in my kernel. There are no modules.
Stated another way. I don't have any kernel modules - everything is compiled into the kernel
Just to try another tack: iptables is calling modprobe even though there is *no* module to load (it's already compiled into the kernel)
So, to restate my original problem:
With a latest released busybox/iptables system it takes 1.5 secs to run a simple "iptables -h", ie no action actually performed and no modules on the system to even load.... Staggering...
(Part of that problem is that the busybox modprobe call is very slow. However, even switching to a "fatter" modprobe takes nearly 5ms to run "iptables -h" - that really starts to stack up...)
Apologies if I now over laboured the point, but do you see my issue?
Thanks for any thoughts on reducing the number of calls to modprobe (given that there are no modules to load on my system)
Ed W
Quoted 16 lines · Ed W
Quoted 8 lines · Jan Engelhardt
Quoted 6 lines · Ed W
Starting "shorewall"
- using busybox modprobe + released iptables = several minutes...
- module-init-tools + released iptables = 12s
- module-init-tools + your commit = 7.7s
- module-init-tools + patching out modprobe completely = 4.9sQuoted 6 lines
Quoted 2 lines
So, whilst your patch has a huge positive benefit, I'm still seeing a substantial amount of cpu going to useless modprobing.
Which is only natural, because the kernel loads stuff. If you have a problem with time (why so?) you should <*> the xtables modules, and just leave iptables in its origin state.
Quoted 6 lines
I'm sorry that I haven't managed to explain my situation clearly. However, I'm trying very hard to explain that I *have* got <*> in my kernel. There are no modules.
Stated another way. I don't have any kernel modules - everything is compiled into the kernel
In that case, iptables should never call modprobe.
You said iptables -h itself would take a long time due to modprobe, however, I can spot no invocation of it using strace -fe execve iptables -h when the desired codes are already loaded into the kernel one way or another.
Quoted 5 lines · Jan Engelhardt
You said
iptables -hitself would take a long time due to modprobe, however, I can spot no invocation of it usingstrace -fe execve iptables -hwhen the desired codes are already loaded into the kernel one way or another.
Maciej pointed out this is the case for git as of around 4 days ago, and in my last email I confirmed I saw the same. I don't expect you to observe what you say if you use last released 1.4.10?
However, if you use an xtables match then I would expect you to see a module loaded? (For example running up a fairly vanilla "shorewall" script manages to cause some 20 odd modules to load). My specific problem is that on my hardware, invoking modprobe takes some good few ms (whether something is loaded or not), all modules are compiled into the kernel, so this time is dead time. In the case of shorewall (which for sure is a beefy firewall script), this amounts to some several seconds of wasted time (since nothing is loaded)
I appear to be able to eliminate this delay by modifying /proc/sys/kernel/modprobe, but I not (apparently) by using "iptables -M blah"? No one else is commenting on this, so either it's resolved in current git, or ...?
No one has suggested anything, but basically, is there some way to avoid running modprobe when the kernel has all the xtables modules compiled in? ie can we detect that the code is already loaded and avoid trying to probe for it?
However, many thanks to Maciej - that commit massively improves the performance of the simple cases!
Cheers
Ed W
Quoted 10 lines · Ed W
Quoted 5 lines · Jan Engelhardt
You said
iptables -hitself would take a long time due to modprobe, however, I can spot no invocation of it usingstrace -fe execve iptables -hwhen the desired codes are already loaded into the kernel one way or another.Maciej pointed out this is the case for git as of around 4 days ago, and in my last email I confirmed I saw the same. I don't expect you to observe what you say if you use last released 1.4.10?
Maciej's patches do not change the kernel behavior, which is the only place from where kernel modules are loaded.
Quoted 2 lines
(For example running up a fairly vanilla "shorewall" script manages to cause some 20 odd modules to load).
Some scripts irresponsibly call iptables often, and thus modprobe gets invoked that many times. I have said earlier, look at strace -fe execve iptables....
Quoted 4 lines
I appear to be able to eliminate this delay by modifying
/proc/sys/kernel/modprobe, but I not (apparently) by using "iptables -M blah"? No one else is commenting on this, so either it's resolved in current git, or ...?
Because, as I have said earlier:
- iptables loads ip_tables.ko, and
- the kernel loads any remaining requested modules
The -M option only affects the first.
Quoted 4 lines
No one has suggested anything, but basically, is there some way to avoid running modprobe when the kernel has all the xtables modules compiled in? ie can we detect that the code is already loaded and avoid trying to probe for it?
This detection is precisely what the kernel has been doing _for years_. Changing /proc/sys/kernel/modprobe to a script that logs the call will show it.
echo '#!/bin/sh' >/tmp/modp
echo 'echo "$@" >>/tmp/modp.log' >>/tmp/modp
echo /tmp/modp >/proc/sys/kernel/modprobe
iptables-restore foo.txt
Quoted 7 lines · Jan Engelhardt
Quoted 5 lines · Ed W
Quoted 4 lines · Jan Engelhardt
You said
iptables -hitself would take a long time due to modprobe, however, I can spot no invocation of it usingstrace -fe execve iptables -hwhen the desired codes are already loaded into the kernel one way or another.
Quoted 5 lines
Quoted 3 lines
Maciej pointed out this is the case for git as of around 4 days ago, and in my last email I confirmed I saw the same. I don't expect you to observe what you say if you use last released 1.4.10?
Maciej's patches do not change the kernel behavior, which is the only place from where kernel modules are loaded.
I'm reasonably sure that we are talking cross purposes and the fault is no doubt mine.
However, to be clear - my problem is with xtables.c / xtables_insmod(). This function is called multiple times (in 1.4.10) even doing a simple "iptables -h" on a kernel where there are no modules. I don't quite understand what *causes* that code to be called, my point is simply that it's called repeatedly (and needlessly) on each invocation of "iptables -h"
Quoted 5 lines
Quoted 2 lines
(For example running up a fairly vanilla "shorewall" script manages to cause some 20 odd modules to load).
Some scripts irresponsibly call iptables often, and thus modprobe gets invoked that many times. I have said earlier, look at
strace -fe execve iptables....
Actually, having looked closer at shorewall it's "feature" is that it builds an iptables restore script and runs it once. Therefore I'm seeing around 2+ seconds of modprobe attempts during a single iptables restore command. This penalty occurs each time "iptables restore" is called (ie doesn't go away after first call)
Quoted 7 lines
Quoted 4 lines
I appear to be able to eliminate this delay by modifying
/proc/sys/kernel/modprobe, but I not (apparently) by using "iptables -M blah"? No one else is commenting on this, so either it's resolved in current git, or ...?Because, as I have said earlier:
- iptables loads ip_tables.ko, and
- the kernel loads any remaining requested modules
It may well be the case that it's the kernel which is running the code in xtables.c. However, my problem is therefore solveable because the kernel should know that the relevant module is compiled in and doesn't need loading.
Can you please clarify the chain of calling that leads to xtables.c / xtables_insmod() to be called and perhaps some ideas how the kernel modules could be modified to pass back some information that they are compiled in?
Quoted 7 lines
Quoted 4 lines
No one has suggested anything, but basically, is there some way to avoid running modprobe when the kernel has all the xtables modules compiled in? ie can we detect that the code is already loaded and avoid trying to probe for it?
This detection is precisely what the kernel has been doing
_foryears_. Changing/proc/sys/kernel/modprobeto a script that logs the call will show it.
OK, just to highlight the issue, I run iptables -h in my build chroot (32bit chroot on a 64bit host):
chroot:quad iptables-1.4.10 # iptables -h
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
iptables v1.4.10
...snip normal iptables help text...
Quoted 4 lines
echo '#!/bin/sh' >/tmp/modp
echo 'echo "$@" >>/tmp/modp.log' >>/tmp/modp
echo/tmp/modp>/proc/sys/kernel/modprobe
iptables-restore foo.txt
OK, with the addition of an "chmod +x" I run the above and get (deliberately used non existent foo.txt):
$ iptables-restore foo.txt Can't open foo.txt: No such file or directory
$ cat /tmp/modp.log ip_tables -q -q -- ipt_SET -q -- ipt_SET -q -- ipt_set -q -- ipt_set
OK, I'm slightly unsure why there are fewer calls in that last time I did exactly this, but nonetheless we see a bunch of modprobe calls. strace also this code being called.
I then apply Maciej's patch to delay loading and also patch out xtables.c / xtables_insmod(). Marciej's patch would mean that your test above wouldn't cause any module loading because it's now delayed until something uses it, so instead I run up shorewall which loads a fairly complete real set of rules using "iptables restore". Result is no log file is created
Therefore I believe that the delays are due only to the modprobe calls in xtables.c, and as they are slow and completely redundant (for me) I'm hoping to find a way to eliminate the call (for the case of compiled in kernel modules).
Hopefully this is now clearer?
Can anyone please help?
Ed W
Quoted 6 lines · Ed W
However, to be clear - my problem is with
xtables.c/xtables_insmod(). This function is called multiple times (in 1.4.10) even doing a simple "iptables -h" on a kernel where there are no modules. I don't quite understand what *causes* that code to be called, my point is simply that it's called repeatedly (and needlessly) on each invocation of "iptables -h"
Then I suggest to grab backtraces with gdb.
Quoted 2 lines
It may well be the case that it's the kernel which is running the code in
xtables.c.
No no no. Not unless you run a hack like Kernel Mode Linux, which I doubt you do. Or do you? I have no idea what grsec put into their stuff, or whether you have an authentic unpatched iptables program. It does not look standard at all.
Quoted 20 lines
chroot:quad iptables-1.4.10 # iptables -h
FATAL: Could not load/lib/modules/2.6.36.2-grsec2.2.1-vs2.3.0.36.38.4/modules.dep: No such file or directory
[repeated 21 times]
...snip normal iptables help text...Quoted 4 lines
echo '#!/bin/sh' >/tmp/modp
echo 'echo "$@" >>/tmp/modp.log' >>/tmp/modp
echo/tmp/modp>/proc/sys/kernel/modprobe
iptables-restore foo.txtOK, with the addition of an "chmod +x" I run the above and get (deliberately used non existent foo.txt):
$ iptables-restore foo.txt Can't open foo.txt: No such file or directory$ cat /tmp/modp.log ip_tables -q
Review
Quoted 4 lines
(quoted diff) +0
-4-q -- ipt_SET -q -- ipt_SET -q -- ipt_set -q -- ipt_set
The trailing -q is a hint towards the call from iptables, whereas the leading -q is kernel-invoked calls. Specifically, it seems that xt_set is indeed not built-in, but a module.
Quoted 3 lines
OK, I'm slightly unsure why there are fewer calls in that last time I did exactly this, but nonetheless we see a bunch of modprobe calls. strace also this code being called.
Please try with your real ruleset rather than foo.txt after clearing modp.log.
I should probably point out that iptables module autoloading behaviour is very different for a pre-my commit statically compiled, non-shared iptables binary
vs a post-my commit or non-statically compiled binary.
The modprobe issue would only show up if you had a statically compiled, non-shared iptables binary.
ie. one where all the extensions were already part of the iptables binary and would always get initialized at startup (even for iptables -h).
Basically the problem is when we're doing revision compatibility testing as part of an extensions init sequence we will modprobe from userspace.
I believe we always attempt a module load there instead of first checking for api existence, and modprobing if the API is missing.
The fix would probably be in xtables.c in compatible_revision() to change everything starting with:
xtables_load_ko(xtables_modprobe_program, true);
into a for loop:
something along the lines of
for (int load = 0; load <= 1; ++load) {
if (load) xtables_load_ko(...);
...
and replace "close(sockfd); return 0;" with "if (load) {
close(sockfd); return 0; }; continue;"
I'll try to write up and maybe even test a real patch tomorrow.
- Maciej
Quoted 8 lines
Quoted 3 lines
It may well be the case that it's the kernel which is running the code in
xtables.c.No no no. Not unless you run a hack like Kernel Mode Linux, which I doubt you do. Or do you? I have no idea what grsec put into their stuff, or whether you have an authentic unpatched iptables program. It does not look standard at all.
Nope, normal 2.6.37 kernel with grsec patches. iptables 1.4.10 is compiled under gentoo and largely vanilla other than I have added the commit from Maciej and my own hack to delete modprobing
I think grsec is relatively unexciting in this case - largely it's just a big "constifying" patch, plus some RBAC features. If you strip out the "constify" stuff (and chroot limitations), the remaining patch is reasonably small and sane.
Quoted 10 lines
Review
Quoted 6 lines
$ cat /tmp/modp.log ip_tables -q(quoted diff) +0
-4-q -- ipt_SET -q -- ipt_SET -q -- ipt_set -q -- ipt_setThe trailing -q is a hint towards the call from iptables, whereas the leading -q is kernel-invoked calls. Specifically, it seems that
xt_setis indeed not built-in, but a module.
Hmm, that's an interesting point.
So ipt_SET of course not build in to the kernel (2.6.37/38). Further I can't find any reference to it in a 2.6.38 kernel tree? I do however see something in the 2.6.39-rc3 tree (sanity check please - is this correct?)
This would seem to explain why the kernel autoloader was having a try to find some missing module? It's actually completely missing on this kernel?
However, I really don't understand the sequence of calls here that has caused apparently a "modprobe iptables" from xtables.c to trigger the kernel (?) to try and insert some module that it knows nothing about (only iptables seems to know about it)?
Hmm, not sure how to progress this now?
Ed W
Quoted 4 lines · Maciej Żenczykowski
I should probably point out that iptables module autoloading behaviour is very different for a pre-my commit statically compiled, non-shared iptables binary
vs a post-my commit or non-statically compiled binary.
Aha. I am using a static (well, --static --shared) iptables
Quoted 5 lines
The modprobe issue would only show up if you had a statically compiled, non-shared iptables binary.
ie. one where all the extensions were already part of the iptables binary and would always get initialized at startup (even for iptables -h).
Thanks - although my evidence suggests its simply the statically compiled bit which is enough to trigger this? My binary is both static and shared (is that the correct term?)
Quoted 10 lines
The fix would probably be in
xtables.cincompatible_revision()to change everything starting with:
xtables_load_ko(xtables_modprobe_program, true);
into a for loop:something along the lines of
for (int load = 0; load <= 1; ++load) {
if (load)xtables_load_ko(...);
Quoted 6 lines
...
and replace "close(sockfd); return 0;" with "if (load) {
close(sockfd); return 0; }; continue;"I'll try to write up and maybe even test a real patch tomorrow.
Thanks - although I'm not quite sure that I see that this changes anything? There is already a "static int load;" in, I think it's xtables_load_ko (?) which prevents the module loading more than once per call of iptables - I'm not sure I understand how your loop above changes that?
Additionally, as helpfully pointed out by Jan, a chunk of my problem is my static iptables apparently trying to probe a kernel module which isn't incorporated into my kernel version. I can't immediately see a solution to not uselessly probing for that (without patching iptables)?
Any ideas?
Thanks
Ed W
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Ed W
Quoted 10 lines
Review
Quoted 6 lines
$ cat /tmp/modp.log ip_tables -q(quoted diff) +0
-4-q -- ipt_SET -q -- ipt_SET -q -- ipt_set -q -- ipt_setThe trailing -q is a hint towards the call from iptables, whereas the leading -q is kernel-invoked calls. Specifically, it seems that
xt_setis indeed not built-in, but a module.
Quoted 5 lines
Hmm, that's an interesting point.
So
ipt_SETof course not build in to the kernel (2.6.37/38). Further I can't find any reference to it in a 2.6.38 kernel tree? I do however see something in the 2.6.39-rc3 tree (sanity check please - is this correct?)
Yes, ipset was first merged into the kernel for 2.6.39.
Quoted 2 lines
This would seem to explain why the kernel autoloader was having a try to find some missing module? It's actually completely missing on this kernel?
On your 37/38, yes.
Quoted 4 lines
However, I really don't understand the sequence of calls here that has caused apparently a "modprobe iptables" from
xtables.cto trigger the kernel (?) to try and insert some module that it knows nothing about (only iptables seems to know about it)?
You mentioned your program _is_ actually compiled with --enable-static, as such this applies:
Since libxt_SET and libxt_set are built into iptables-multi, their revision will be checked unconditionally (previous to Maciej's patches). Since the 37s and 38s kernel has no xt_SET/set, it will try probing. All looks normal. Including that your modprobe count still is just 5 and not 20+.
Quoted 14 lines · Ed W
Quoted 4 lines · Maciej Żenczykowski
I should probably point out that iptables module autoloading behaviour is very different for a pre-my commit statically compiled, non-shared iptables binary
vs a post-my commit or non-statically compiled binary.Aha. I am using a static (well, --static --shared) iptables
Quoted 5 lines
The modprobe issue would only show up if you had a statically compiled, non-shared iptables binary.
ie. one where all the extensions were already part of the iptables binary and would always get initialized at startup (even for iptables -h).
Quoted 3 lines
Thanks - although my evidence suggests its simply the statically compiled bit which is enough to trigger this? My binary is both static and shared (is that the correct term?)
In iptables, the options --enable-static and --enable-shared are semantically different from other projects.
--enable-static/--disable-static decides whether [not] to put all shipped plugins into the binary.
--enable-shared/--disable-shared selects whether the iptables-multi binary will attempt to dlopen extensions when they have not already been found inside the binary.
Which is really what people want. --enable-static is _not_ going to produce a fully-static executable (one that would yield "not a dynamic executable" when running ldd) in iptables.
Quoted 5 lines
Additionally, as helpfully pointed out by Jan, a chunk of my problem is my static iptables apparently trying to probe a kernel module which isn't incorporated into my kernel version. I can't immediately see a solution to not uselessly probing for that (without patching iptables)? Any ideas?
I would have said it could be the missing SET module being the cause for your modprobe time accumulation, but since you also use iptables-restore that possibility, too, is eliminated.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi
Quoted 2 lines
In iptables, the options --enable-static and --enable-shared are semantically different from other projects.
Thanks for confirming - iptables also helpfully spells exactly this out in the INSTALL doc (+1 for open source documentation!!)
Quoted 9 lines
Quoted 5 lines
Additionally, as helpfully pointed out by Jan, a chunk of my problem is my static iptables apparently trying to probe a kernel module which isn't incorporated into my kernel version. I can't immediately see a solution to not uselessly probing for that (without patching iptables)? Any ideas?
I would have said it could be the missing SET module being the cause for your modprobe time accumulation, but since you also use iptables-restore that possibility, too, is eliminated.
Yes, although these modules are being probed for even on a zero (missing) input to iptables-restore. However, that seems consistent with a v1.4.10 iptables --enable-static based binary? Presumably this just probes everything?
(To be clear my test in my previous email was NOT using your git commit to delay mod probing)
I will have to retest with your commit and without my hack to see exactly what is still being probed for
Thanks
Ed W
Quoted 7 lines · Ed W
Hi
Quoted 2 lines
In iptables, the options --enable-static and --enable-shared are semantically different from other projects.
Thanks for confirming - iptables also helpfully spells exactly this out in the INSTALL doc (+1 for open source documentation!!)
-1 to the user for not reading it ;-)
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Additionally, as helpfully pointed out by Jan, a chunk of my problem is my static iptables apparently trying to probe a kernel module which isn't incorporated into my kernel version. I can't immediately see a solution to not uselessly probing for that (without patching iptables)? Any ideas?
Quoted 8 lines
Quoted 3 lines
I would have said it could be the missing SET module being the cause for your modprobe time accumulation, but since you also use iptables-restore that possibility, too, is eliminated.
Yes, although these modules are being probed for even on a zero (missing) input to iptables-restore. However, that seems consistent with a v1.4.10 iptables --enable-static based binary? Presumably this just probes everything?
Yes, and that which does not exist in the kernel you pay with a modprobe call then. That would not only include SET, but also extensions long obsoleted, such as libipt_unclean's counterpart.
Quoted 2 lines
-1 to the user for not reading it ;-)
I guess my point wasn't clear - I HAD already read it. Just saying thanks for being very patient with me and assuming otherwise
Quoted 8 lines
Quoted 4 lines
Yes, although these modules are being probed for even on a zero (missing) input to iptables-restore. However, that seems consistent with a v1.4.10 iptables --enable-static based binary? Presumably this just probes everything?
Yes, and that which does not exist in the kernel you pay with a modprobe call then. That would not only include SET, but also extensions long obsoleted, such as libipt_unclean's counterpart.
Hmm, for the moment I'm happy to simply patch out all modprobe calls in xtables.c, but there may come a time when I need more flexibility. Does anyone care enough about this to consider a more clever solution?
The issue would be that someone might genuinely want to forward/backward port modules between kernel releases, however, perhaps it would be reasonable to offer a compile time option for use with --enable-static which limits compiled in modules to those which match a kernel version?
I can see lots of negatives here - does anyone have a better idea?
Thanks
Ed W
Quoted 5 lines
Hmm, for the moment I'm happy to simply patch out all modprobe calls in
xtables.c, but there may come a time when I need more flexibility. Does anyone care enough about this to consider a more clever solution?
I've been following this with great interest as I also intend to compile and use iptables in one of my embedded devices (603e/G2-based) and will, most probably, have similar problems to what you have experienced, so if there is a better solution to the (proposed) patches I'd like to know about it too!
Okay, so I've been doing some experimentation.
Based on this, my current belief is that ip6?tables will only attempt to ever load either ip_tables or ip6_tables.
Indeed some greps on my almost pristine iptables git master source tree show:
$ egrep -r xtables_insmod . ./xtables.c:int xtables_insmod(const char *modname, const char *modprobe, bool quiet) ./xtables.c: ret = xtables_insmod(afinfo->kmod, modprobe, quiet); ./include/xtables.h.in:extern int xtables_insmod(const char *, const char *, bool);
So xtables_insmod() is declared in xtables.h, defined in xtables.c and called from one spot in xtables.c (within xtables_load_ko), and afinfo->kmod will be either "ip_tables" or "ip6_tables" (depending on whether afinfo is for AF_INET or AF_INET6).
$ egrep -r xtables_load_ko . ./ip6tables-save.c: xtables_load_ko(xtables_modprobe_program, false); ./xtables.c:int xtables_load_ko(const char *modprobe, bool quiet) ./xtables.c: xtables_load_ko(xtables_modprobe_program, true); ./iptables.c: if (!*handle && xtables_load_ko(xtables_modprobe_program, false) != -1) ./ip6tables.c: if (!*handle && xtables_load_ko(xtables_modprobe_program, false) != -1) ./iptables-save.c: xtables_load_ko(xtables_modprobe_program, false); ./ip6tables-restore.c: xtables_load_ko(xtables_modprobe_program, false); ./include/xtables.h.in:extern int xtables_load_ko(const char *, bool); ./iptables-restore.c: xtables_load_ko(xtables_modprobe_program, false);
Which shows that ip6?tables{,-restore,-save} can all cause the load to happen, as well as the revision compatibility checking code in xtables.c.
Either way, the userspace iptables will only ever directly call modprobe ip_tables or modprobe ip6_tables.
This is consistent with what I'm seeing in tests.
This can be disabled with -M '' parameter.
ie. "iptables -M '' -j SET --help" does not result in the iptables userspace component loading any modules.
However, this does not prevent the kernel from loading modules. And the kernel will attempt to load modules for any matches that we attempt to access that it does not know about.
This should not be a problem with an iptables userspace once my patch has been applied, since the kernel won't load modules we (userspace) don't attempt to use, and once loaded they'll stay. Since we can assume that userspace doesn't attempt to use modules the kernel doesn't have, we should relatively quickly reach steady state were all modules userspace cares to use are already loaded, and no further load attempts are issued.
Basically I believe that an iptables userspace binary with my patch, called with -M '' should quickly settle into a mode were no further modprobes (userspace or kernel initiated) ever happen.
The only exception is if you attempt to use iptables extensions which aren't compiled for the kernel, but iptables-save won't do that, and iptables-restore would fail if it did that (since obviously the extension wouldn't exist and thus wouldn't be restorable). Hence this is not an exciting case (it's an error scenario which you shouldn't be triggering during normal operation).
Ed: could you verify this hypothesis? ie. that iptables with my patch, when called with -M '' doesn't trigger repeated modprobes of the same modules?
[I've verified that this is the behaviour I'm seeing, ie. every invocation of "iptables -M '' -j SET" triggers two loads of ipt_SET (which fails, since the kernel is to old), but only the first invocation of "iptables -M '' -j CONNMARK" triggers a single load of ipt_CONNMARK, afterwards further invocations don't trigger any module loads, since the first load succeeded]
-- Maciej
PS. Alternatively one could replace the modprobe binary with a script which does nothing for certain modules.
ie. something along the lines of:
cat <<EOF > /sbin/modprobe.skip
#!/bin/sh
[[ "$*" == "-q -- ipt_SET" ]] && exit 0
exec /sbin/modprobe "$@" ;;
EOF
chmod a+x /sbin/modprobe.skip
echo "/sbin/modprobe.skip" > /proc/sys/kernel/modprobe
-- Maciej Z.
Note that: -M '' is -M followed by a space and two single quotes.
Furthermore, note that with -M '', you will want to modprobe ip_tables or modprobe ip6_tables manually first at system startup (or build them into the kernel), since those modules don't autoload (hence why iptables tries to load them).
I wonder if there's an easy way iptables userspace could detect whether these modules are already loaded (or compiled into the kernel), and not even try to load them, if so...
- Maciej
Quoted 10 lines · Maciej Żenczykowski
Note that: -M '' is -M followed by a space and two single quotes.
Furthermore, note that with -M '', you will want to modprobe
ip_tablesor modprobeip6_tablesmanually first at system startup (or build them into the kernel), since those modules don't autoload (hence why iptables tries to load them).I wonder if there's an easy way iptables userspace could detect whether these modules are already loaded (or compiled into the kernel), and not even try to load them, if so...
Not with the socket interface, but it's on the plate for netlink-based Xtables.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 14.04.2011 09:19, schrieb Jan Engelhardt:
Quoted 9 lines · Maciej Żenczykowski
Quoted 8 lines
Note that: -M '' is -M followed by a space and two single quotes.
Furthermore, note that with -M '', you will want to modprobe
ip_tablesor modprobeip6_tablesmanually first at system startup (or build them into the kernel), since those modules don't autoload (hence why iptables tries to load them).
Quoted 6 lines
Quoted 3 lines
I wonder if there's an easy way iptables userspace could detect whether these modules are already loaded (or compiled into the kernel), and not even try to load them, if so...
Not with the socket interface, but it's on the plate for netlink-based Xtables.
We do have the /proc/net/ip_tables_{names,matches,targets} files which should be usable for this.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Maciej Żenczykowski
Note that: -M '' is -M followed by a space and two single quotes.
Furthermore, note that with -M '', you will want to modprobe
ip_tablesor modprobeip6_tablesmanually first at system startup (or build them into the kernel), since those modules don't autoload (hence why iptables tries to load them).I wonder if there's an easy way iptables userspace could detect whether these modules are already loaded (or compiled into the kernel), and not even try to load them, if so...
OK, using kernel 2.6.38 (previously on .37) iptables 1.4.10 patched with the delayed module loading commit, then I still get something like 20 attempts to "modprobe iptables -q" when I start up a near vanilla shorewall script (I just entered enough info that it boots up with a couple of basic zones).
If I just do an iptables restore, or a near equivalent "shorewall restore" then I get just a single modprobe iptables -q.
This suggests that the shorewall start tickles several iptables calls. Each call causing one modprobe
Now this seems to be coming from the iptables.c modprobe call. Annoyingly this didn't seem to be happening when I used kernel 2.6.37. It's timeconsuming to reload kernel changes to this embedded device, but I will check back and confirm this is a change in behaviour between kernels.
However, it seems unexpected that there are any calls from iptables since it does some kind of test before calling modprobe? I'm sure I didn't get any on .37??! Any insights on why I get even a single modprobe call given everything built in kernel and a static iptables binary?
Thanks
Ed W
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
You didn't actually reply to my email: you were asked to try -M ''.
Anyway, the 'maze' branch of git://github.com/zenczykowski/iptables.git
has the following commits:e39f367d905670e39e6f08d2b73c715a6d0b4bfb [upstream iptables master]
SET target revision 2 added590e3e8653057385ce503ac037783bbb9f5584d4 Don't load ip6?_tables
module when already loaded.8a4efee1bd7e6137a6b4faa08c49dbb9722af1c8 Add --ipv4/-4 and --ipv6/-6
support to ip6?tables{,-restore}.3436427a9757705ad471f1b77c56b9afd6b2e40c Move common parts of
libext{4,6}.a into libext.aa9e4c0909bec8424b6166cc1662958d66edc15e7 combine ip6?tables-multi
into xtables-multif75e8347925b9ffd5b01c677e86acc16c7554821 add xtables-multi{32,64} recognition
You may want to cherrypick 590e3e86... and try again.
Maciej
And the "Don't load ip6?_tables module when already loaded" patch is now available in upstream git...
That patch combined with my previous patch should solve all your woes. If they don't... let me know.
- Maciej
Quoted 5 lines · Maciej Żenczykowski
And the "Don't load ip6?_tables module when already loaded" patch is now available in upstream git...
That patch combined with my previous patch should solve all your woes. If they don't... let me know.
Yep, now I just see an attempt to modprobe "ip_set" (under limited circumstances), which seems reasonable given that I am on 2.6.38 and this isn't compiled in (nor is it a module)
That said, I'm also slightly baffled where it's getting probed since I patched out xtables.c/xtables_load_ko(). The modprobe call doesn't have a -q on it, so I suspect somehow it might be the kernel or something else calling it?
It's an acceptable casualty for the moment though...
Thanks for your interest and looking into this - big increase in performance!
Cheers
Ed W
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines
Yep, now I just see an attempt to modprobe "
ip_set" (under limited circumstances), which seems reasonable given that I am on 2.6.38 and this isn't compiled in (nor is it a module)That said, I'm also slightly baffled where it's getting probed since I patched out xtables.c/xtables_load_ko(). The modprobe call doesn't have a -q on it, so I suspect somehow it might be the kernel or something else calling it?
Yeah, that's getting modprobe'd by the kernel.
But that shouldn't happen (after both my patches are applied) unless you actually try to use it, so what are you doing to cause that to happen? ie. there should not be any attempted loads of ip_set without "iptables -m set" being called (or equivalent).
- Maciej
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
happen? ie. there should not be any attempted loads of
ip_setwithout "iptables -m set" being called (or equivalent).
Actually, are you sure the module is called 'ip_set'?
Kernel iptables match and target extensions are named xt_* or ipt_* or ip6t_*. If the module is called ip_set and not ipt_set, then maybe we're seeing some other dependency chain?
contains:
21 MODULE_LICENSE("GPL");
22 MODULE_AUTHOR("Jozsef Kadlecsik <[email protected]>"); 23 MODULE_DESCRIPTION("Xtables: IP set match and target module"); 24 MODULE_ALIAS("xt_SET");
25 MODULE_ALIAS("ipt_set");
26 MODULE_ALIAS("ip6t_set");
27 MODULE_ALIAS("ipt_SET");
28 MODULE_ALIAS("ip6t_SET");
(plus the name derived from the filename which is 'xt_set')
so I'm guessing your 'ip_set' was simply a typo and should have been 'ipt_set'.
- Maciej
Quoted 22 lines · Maciej Żenczykowski
Quoted 2 lines
happen? ie. there should not be any attempted loads of
ip_setwithout "iptables -m set" being called (or equivalent).Actually, are you sure the module is called '
ip_set'?
Kernel iptables match and target extensions are named xt_* or ipt_* or ip6t_*. If the module is calledip_setand notipt_set, then maybe we're seeing some other dependency chain?contains:
21
MODULE_LICENSE("GPL");
22MODULE_AUTHOR("Jozsef Kadlecsik <[email protected]>"); 23MODULE_DESCRIPTION("Xtables: IP set match and target module"); 24MODULE_ALIAS("xt_SET");
25MODULE_ALIAS("ipt_set");
26MODULE_ALIAS("ip6t_set");
27MODULE_ALIAS("ipt_SET");
28MODULE_ALIAS("ip6t_SET");
(plus the name derived from the filename which is 'xt_set')
Quoted 1 line
so I'm guessing your '
ip_set' was simply a typo and should have been 'ipt_set'.
The module "ip_set" is loaded in if the "ipset" program is installed and invoked on that system.
Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
Quoted 2 lines · Jozsef Kadlecsik
The module "
ip_set" is loaded in if the "ipset" program is installed and invoked on that system.
Aha, that is almost certainly the cause. As well as kernel, I had updated the software image to include ipset/conntrack and didn't connect the events
I see that I obviously haven't built my image correctly and included the ip_set module - apologies to the iptables folks - obviously user error here..
So, in conclusion, with the two patches from Maciej, there is now no modprobe activity in the event of static iptables and kernel with all modules built-in. Perfect
Thanks for everyone's interest in this - superb response
Ed W