Skip to contentSkip to conversation

Scaling Max IP address limitation

DISCUSSION David Jones 8 participants 19 messages Jun 24, 2007 – Jun 26, 2007 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,
I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. But I am hitting a max limit of 4000 IP address . Seems like there is a limiting variable in linux kernel (which one? ) that prevents from adding more IP addresses than 4096. What do I need to change in Linux kernel ( and then recompile ) to be able to add more IP addresses than 4K addresses per system? .. Thanks,
-d

#1 lore raw
Quoted 7 lines · On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <[email protected]> wrote:

Hi,
I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. But I am hitting a max limit of 4000 IP address . Seems like there is a limiting variable in linux kernel (which one? ) that prevents from adding more IP addresses than 4096. What do I need to change in Linux kernel ( and then recompile ) to be able to add more IP addresses than 4K addresses per system? ..

(cc netdev)

#2 lore raw

Hi,

Quoted 8 lines · On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <[email protected]> wrote:
Quoted 7 lines

I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. But I am hitting a max limit of 4000 IP address . Seems like there is a limiting variable in linux kernel (which one? ) that prevents from adding more IP addresses than 4096. What do I need to change in Linux kernel ( and then recompile ) to be able to add more IP addresses than 4K addresses per system? ..

We are adding tens of thousand IPv4 addresses using netlink interface without any problems. The maximum we added was 60K of secondary IPv4 addresses. It consumes some memory, however.

We have also added thousands of IPv6. I will try to test, if there is any limit for doing it.

Signature · 6 lines
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.
#3 lore raw
Quoted 14 lines · Robert Iakobashvili

Hi,

Quoted 10 lines · On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <[email protected]> wrote:
Quoted 3 lines

I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. But I am hitting a max limit of 4000 IP address . Seems like there

is a

Quoted 3 lines

limiting variable in linux kernel (which one? ) that prevents from adding more IP addresses than 4096. What do I need to change in Linux kernel ( and then recompile ) to be able to add more IP addresses

than

Quoted 1 line

4K addresses per system? ..

Quoted 7 lines

We are adding tens of thousand IPv4 addresses using netlink interface without any problems. The maximum we added was 60K of secondary IPv4 addresses. It consumes some memory, however.

We have also added thousands of IPv6. I will try to test, if there is any limit for doing it.

I am using the "ip add " command looping sequentially up until RTNETLINK starts refusing to add more IP addresses. I am using a simple shell script to do the trick. One quick fact : If I exhaust 4K addresses on one port , then I can not add more IP's ( v4/v6 alike ) on any port on the system. So seems like its a system wide limitation . Tried digging through the kernel source code but no luck so far. So definitely need pointers in this regard. How are you adding via Netlink interface ?
Thanks,
-d

#4 lore raw

David,

Quoted 11 lines · David Jones
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. But I am hitting a max limit of 4000 IP address . Seems like there

is a

Quoted 3 lines

limiting variable in linux kernel (which one? ) that prevents from adding more IP addresses than 4096. What do I need to change in Linux kernel ( and then recompile ) to be able to add more IP addresses

than 4K addresses per system? ..

I am using the "ip add " command looping sequentially up until RTNETLINK starts refusing to add more IP addresses.
How are you adding via Netlink interface ?

Yes.

OK. Now it looks that I am reproducing something.

Running curl-loader with 60K.conf (edit the name of interface) configuration:
#ulimit -n 80000
#curl-loader -f ./conf-examples/60K.conf -w
it adds successfully 60 000 secondary IPv4 addresses as seen by #ip addr | wc -l

When I tryed adding IPv6 addresses, using ipv6.conf with addresses range edited:
IP_ADDR_MIN= 2001:db8:fff5:1::1
IP_ADDR_MAX= 2001:db8:fff5:ffff::1

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory" and
#ip addr | wc-l is 8194.

8K addresses added and no more? It might be a memory issue. Y can dig into the code and look into the allocation process and limits on the kernel memory for IPv6.

The physical memory on my computer is 480 MB.
kernel is vanilla 2.6.20.7.

Try to see, what happens, when you increase the memory on your comp, if an option.

Signature · 6 lines
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.
#5 lore raw
Quoted 5 lines · Robert Iakobashvili

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory" and
#ip addr | wc-l is 8194.

I'd be surprised if it was 4096 on x86 and 8192 on x86_64...

Jan

#6 lore raw

Hi

Quoted 9 lines · Jan Engelhardt
Quoted 5 lines · Robert Iakobashvili

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory" and
#ip addr | wc-l is 8194.

I'd be surprised if it was 4096 on x86 and 8192 on x86_64...

Missed to mention: the CPU is Pentium-4.

Signature · 6 lines
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.
#7 lore raw
Quoted 8 lines · Robert Iakobashvili
Quoted 6 lines
Quoted 4 lines

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory" and
#ip addr | wc-l is 8194.

I'd be surprised if it was 4096 on x86 and 8192 on x86_64...

Missed to mention: the CPU is Pentium-4.

That's like saying you've got a SPARC. Or a MIPS. Or a PPC. (I can't infer from your answer whether that is running 32 or 64-bit kernel, because there are P4s with and without 64-bit extensions.)

Jan

#8 lore raw
Quoted 7 lines · Jan Engelhardt
Quoted 5 lines · Robert Iakobashvili
Quoted 5 lines
Quoted 4 lines

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory" and
#ip addr | wc-l is 8194.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I'd be surprised if it was 4096 on x86 and 8192 on x86_64...

Missed to mention: the CPU is Pentium-4.

That's like saying you've got a SPARC. Or a MIPS. Or a PPC. (I can't infer from your answer whether that is running 32 or 64-bit kernel, because there are P4s with and without 64-bit extensions.)

32/32

Quoted 3 lines

Jan
--

Signature · 6 lines
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.
#9 lore raw

Ok I have tried it on a Pentium-M ( 32 Bit ,) with 512 MB RAM and Core 2 Duo with 1Gig RAM ( running SMP kernel , 2 CPUS) with same results. Cant go more than ~4K addresses. I have tried them with vanilla and custom kernels all 2.6.19+ versions. Results are same on both systems , so thats the reason I am thinking that there is some limit in kernel source tree which I cant seem to find . Really appreciate your help in this regard. Thanks,
-d

Quoted 23 lines · Robert Iakobashvili
Quoted 7 lines · Jan Engelhardt
Quoted 5 lines · Robert Iakobashvili
Quoted 5 lines
Quoted 4 lines

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory" and
#ip addr | wc-l is 8194.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I'd be surprised if it was 4096 on x86 and 8192 on x86_64...

Missed to mention: the CPU is Pentium-4.

That's like saying you've got a SPARC. Or a MIPS. Or a PPC. (I can't infer from your answer whether that is running 32 or 64-bit kernel, because there are P4s with and without 64-bit extensions.)

32/32

Quoted 3 lines

Jan

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw

Hi David,

Quoted 12 lines · David Jones

Ok I have tried it on a Pentium-M ( 32 Bit ,) with 512 MB RAM and Core
2 Duo with 1Gig RAM ( running SMP kernel , 2 CPUS) with same results. Cant go more than ~4K addresses. I have tried them with vanilla and custom kernels all 2.6.19+ versions. Results are same on both systems , so thats the reason I am thinking that there is some limit in kernel

Quoted 5 lines
Quoted 5 lines · Robert Iakobashvili
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory" and
#ip addr | wc-l is 8194.

Looking into 2.6.20 sources:

IPv6 addresses are added via rtnetlink by:
inet6_rtm_newaddr ()
inet6_addr_add ()
ipv6_add_addr (),
where inet6_ifaddr structure is allocated by kzalloc ()

IPv4 addresses are added via rtnetlink by:
inet_rtm_newaddr () with further __inet_insert_ifa () inet_rtm_newaddr () allocates in_ifaddr structure by kzalloc ()

Structure inet6_ifaddr is ~85-95 bytes (depending on config options, arch, etc),
structure in_ifaddr is ~55-60 bytes (also depending). It looks like the case, when
size does matter.
:)

When adding secondary IPv4 addresses from /proc/slabinfo is seen, that slab "size-64" is used, whereas for IPv6 addresses "size-128" is in use.

There should be a way to make some fine tuning and, most probably, memory allocation for the right pool (in your case "size-128").
I do recall some echo to /proc/slabinfo, but there were some evil details.

Probably, some great people of the netdev list can advise you, how to increase memory resources in the right direction. Hopefully it explains the facts and directs you.

Take care,

Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

#12 lore raw
In reply to Andrew Morton
Quoted 7 lines · On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <[email protected]> wrote:
Quoted 6 lines

I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. But I am hitting a max limit of 4000 IP address . Seems like there is a limiting variable in linux kernel (which one? ) that prevents from adding more IP addresses than 4096. What do I need to change in Linux kernel ( and then recompile ) to be able to add more IP addresses than 4K addresses per system? ..

How are you doing this?

Could it be some IPv6 issue like scope?

Signature · 6 lines
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.
#13 lore raw
In reply to David Jones
Quoted 1 line · David Jones

Hi, I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. But I am hitting a max limit of 4000 IP address . Seems like there is a limiting variable in linux kernel (which one? ) that prevents from adding more IP addresses than 4096. What do I need to change in Linux kernel ( and then recompile ) to be able to add more IP addresses than 4K addresses per system? ..

Do you really need that many IP addresses? When somebody finally gets around to implementing REDIRECT support for ip6tables then you could just redirect them all to the same port on the local system. Then with a happy little getsockopt() you can find out the original IP address for use in whatever application you are running. That's likely to be a thousand times more efficient than binary searching through 5000+ mostly-sequential IP addresses per received packet.

<Unrelated wishful thinking>
I keep having hopeful dreams that one day netfilter will grow support for cross-protocol NAT (IE: NAT a TCPv4 connection over TCPv6 to the IPv6-only local web server, or vice versa). It would seem that would require a merged "xtables" program.

Having routing table operations, IPsec transformations, etc just be another step in the firewall rules would also be useful. It would be handy to be able to "-j ROUTE", then "-j IPSEC", then "-j ROUTE" again, to re-route the now-encapsulated IPsec packets to their proper destination. That would also eliminate the sort-of-hacky problems with destination network interface in the bridging code: "-j BRIDGE" might be another step in the process, and conceivably you could have independent bridge MAC tables too. You'd probably also want "-j BRIDGE_TEST" and "-j ROUTE_TEST" to compute the output network interface without actually modifying the addresses.

That would also appear to get rid of the need for all tables other than "filter" and all predefined chains other than "INPUT" and "OUTPUT". Default rules would be these:
nettables -A INPUT -j CONNTRACK
nettables -A INPUT -j LOCALMATCH
nettables -A INPUT --for-this-host -j ACCEPT
nettables -A INPUT -j OUTPUT
nettables -A OUTPUT -j ROUTE
nettables -A OUTPUT -j TRANSMIT

Forwarded packets would be sent right into the OUTPUT chain from the INPUT chain by appropriate rules. Instead of turning off ip_forwarding in /proc/sys, you could just change the "-j OUTPUT" in the INPUT chain to "-j ACCEPT", and it would be impossible to forward packets. I can't see any functionality that we have today which a mechanism like this wouldn't support, aside from the fact that it hands the admin a loaded nuclear missile aimed at their foot (Flushing the INPUT chain would basically be analogous to committing network suicide, although there exist other ways to do that with netfilter today. </Unrelated wishful thinking>

Cheers,
Kyle Moffett

#14 lore raw
Quoted 5 lines

<Unrelated wishful thinking>
I keep having hopeful dreams that one day netfilter will grow support for cross-protocol NAT (IE: NAT a TCPv4 connection over TCPv6 to the IPv6-only local web server, or vice versa). It would seem that would require a merged "xtables" program.

You don't actually need it (at least for easy cases like that),
because IPv6 defines IPv4 mapped IPv6 addresses of the form ::ffff:a.b.c.d. These will generate IPv4 packets for a.b.c.d, from a v6 socket.
Unless you're using v6only binding (a sysctl option), you can
connect to v6-only servers using a v4 network and a v4 address of the server. The peer address on those connections will show up as a v4 mapped address, and all the traffic will be v4, but the socket layer is all v6.

                                                                +-DLS
#15 lore raw
In reply to Kyle Moffett
Quoted 4 lines · Kyle Moffett

Do you really need that many IP addresses? When somebody finally gets around to implementing REDIRECT support for ip6tables then you could just redirect them all to the same port on the local system.

The way I see it, it's: "if someone gets around to implement *IPv6 NAT*" (which, if its designers were asked, is contrary to the idea of ipv6).

Quoted 11 lines

<Unrelated wishful thinking>
I keep having hopeful dreams that one day netfilter will grow support for cross-protocol NAT (IE: NAT a TCPv4 connection over TCPv6 to the IPv6-only local web server, or vice versa). It would seem that would require a merged "xtables" program.

Having routing table operations, IPsec transformations, etc just be another step in the firewall rules would also be useful. It would be handy to be able to "-j ROUTE", then "-j IPSEC", then "-j ROUTE" again, to re-route the now-encapsulated IPsec packets to their proper destination.

Absolutely...

Quoted 2 lines

That would also eliminate the sort-of-hacky problems with destination network interface in the bridging code:

Where's the hack? iptables operates on what it sees, and it sees br0. The physdev match is justified IMO.

Quoted 3 lines

"-j BRIDGE" might be
another step in the process, and conceivably you could have independent bridge MAC tables too.

Whether a packet goes out a bridge (was that the intention of -j BRIDGE?) is determined by the routing table, which, in most cases, is just a matter of destination address.

Quoted 11 lines

You'd probably also want "-j BRIDGE_TEST" and
"-j ROUTE_TEST" to compute the output network interface without actually modifying the addresses.

That would also appear to get rid of the need for all tables other than "filter" and all predefined chains other than "INPUT" and "OUTPUT". Default rules would be these:
nettables -A INPUT -j CONNTRACK
nettables -A INPUT -j LOCALMATCH
nettables -A INPUT --for-this-host -j ACCEPT
nettables -A INPUT -j OUTPUT

I'd prefer if Linux outputted its packets by default :)

Quoted 2 lines

nettables -A OUTPUT -j ROUTE
nettables -A OUTPUT -j TRANSMIT

pkttables it is!

But this idea may have its benefit: by not restricting rules to certain positions like currently, throughput could be achieved. "Evil packets" f.e. could be dropped really early. (Well, you could also drop them early _today_, but a DROP in the mangle table will everyone make their eyes roll :-)

Jan

#16 lore raw
Quoted 9 lines · Jan Engelhardt
Quoted 4 lines · Kyle Moffett

Do you really need that many IP addresses? When somebody finally gets around to implementing REDIRECT support for ip6tables then you could just redirect them all to the same port on the local system.

The way I see it, it's: "if someone gets around to implement *IPv6 NAT*" (which, if its designers were asked, is contrary to the idea of ipv6).

true, but back in the real world it's sometimes desriable to hid _chich_ specific machine somethign comes from. so I expect that implementation of NAT is going tohappen at some point before it's widely deployed.

David Lang

#17 lore raw
Quoted 6 lines · On Jun 24 2007 13:44, [email protected] wrote:
Quoted 6 lines · Kyle Moffett
Quoted 4 lines

Do you really need that many IP addresses? When somebody finally gets around to implementing REDIRECT support for ip6tables then you could just redirect them all to the same port on the local system.

Quoted 6 lines
Quoted 2 lines

The way I see it, it's: "if someone gets around to implement *IPv6 NAT*" (which, if its designers were asked, is contrary to the idea of ipv6).

true, but back in the real world it's sometimes desriable to hid _chich_ specific machine somethign comes from. so I expect that implementation of NAT is going tohappen at some point before it's widely deployed.

Client-transparent SOCKS5 proxy. It already exists today! ;-) (Not as performant as an in-kernel NAT, though.)

Jan

#18 lore raw
In reply to Jan Engelhardt
Quoted 4 lines · Jan Engelhardt
Quoted 1 line · Kyle Moffett

Do you really need that many IP addresses? When somebody finally gets around to implementing REDIRECT support for ip6tables then you could just redirect them all to the same port on the local system.

The way I see it, it's: "if someone gets around to implement *IPv6 NAT*" (which, if its designers were asked, is contrary to the idea of ipv6).

I totally agree. On the other hand, you need REDIRECT for things like transparent proxies which by definition aren't visible as anything other than a router or bridge.

Quoted 6 lines
Quoted 1 line

Having routing table operations, IPsec transformations, etc just be another step in the firewall rules would also be useful. It would be handy to be able to "-j ROUTE", then "-j IPSEC", then "-j ROUTE" again, to re-route the now-encapsulated IPsec packets to their proper destination.

Absolutely...

Quoted 1 line

That would also eliminate the sort-of-hacky problems with destination network interface in the bridging code:

Where's the hack? iptables operates on what it sees, and it sees br0. The physdev match is justified IMO.

The problem is this:
I want to be able to filter bridged network traffic *both* based on the IP layer *and* the physical device it's going to be routed out. Due to fundamental problems with a statically-ordered architecture, it's impossible to get both, see commit 68df071a201f06b08cdc07111c6d4af918e64edd (found here: http://lists.netfilter.org/pipermail/netfilter-devel/2006-December/026388.html). Basically if you want such cross-layer hooks, right now your *ONLY* choice is to use marks with 2 drawbacks: (1) There are a very small number of marks which must be carefully allocated by your firewall-setup script (2) Marks are inherently extremely fragile for passing data between layers.

Quoted 3 lines
Quoted 1 line

"-j BRIDGE" might be another step in the process, and conceivably you could have independent bridge MAC tables too.

Whether a packet goes out a bridge (was that the intention of -j BRIDGE?) is determined by the routing table, which, in most cases, is just a matter of destination address.

No, the intent of "-j BRIDGE" would be _after_ "-j ROUTE" and some kind of "-j ARP", to actually compute which physical port a given packet should go.

Quoted 9 lines
Quoted 7 lines

That would also appear to get rid of the need for all tables other than "filter" and all predefined chains other than "INPUT" and "OUTPUT". Default rules would be these:
nettables -A INPUT -j CONNTRACK
nettables -A INPUT -j LOCALMATCH
nettables -A INPUT --for-this-host -j ACCEPT
nettables -A INPUT -j OUTPUT

I'd prefer if Linux outputted its packets by default :)

Well the problem is this: Do you want the packet accepted locally or forwarded. If forwarded, how do you want it routed, and which physical port do you want it to go out? Without a statically-coded ordering for all those things there is no way to say what the "default" is. You would need some way to switch between iptables/ip6tables (for compatibility) and pkttables/nettables (for advanced functionality).

Quoted 1 line

But this idea may have its benefit: by not restricting rules to certain positions like currently, throughput could be achieved. "Evil packets" f.e. could be dropped really early. (Well, you could also drop them early _today_, but a DROP in the mangle table will everyone make their eyes roll

It does give you a million more ways to shoot yourself in the foot. Some things would have constraints like "output device must be set" (BRIDGE/ARP, for example). If you accidentally stuck non-constrained things in the wrong order you could get totally-non-IP-compliant behavior. On the other hand, it does give you many choices about IPsec before or after ROUTING (or after one routing step and before another), etc.

Cheers,
Kyle Moffett

#19 lore raw
In reply to Jan Engelhardt
Quoted 9 lines · Jan Engelhardt
Quoted 3 lines · Kyle Moffett

Do you really need that many IP addresses? When somebody finally gets around to implementing REDIRECT support for ip6tables then you could just redirect them all to the same port on the local system.

The way I see it, it's: "if someone gets around to implement *IPv6 NAT*" (which, if its designers were asked, is contrary to the idea of ipv6).

You don't necessarily need NAT for REDIRECT, so we might actually have an ip6tables REDIRECT some day. Check out the current TPROXY patches for an example how to do it without NAT in case you're interested.