Skip to contentSkip to conversation

Question about limited primary addresses

DISCUSSION Patrick Turley 2 participants 6 messages Sep 20, 2004 – Sep 21, 2004

lore
Share & tools
lore
#0 lore raw

I am trying to configure a Linux box with all possible VLANs (4094 of them), and a subnet on each VLAN. Creating the VLANs isn't a problem. But, when I try to use "ip addr add ..." commands to assign an IP address to each VLAN interface, I get to do about 280 of them before all the interfaces on the box become unresponsive. I've also tried to assign the same IP addresses all to eth0, and I get the same result.

The problem seems to hinge on the distinction between primary and secondary addresses.

If you're reading the LARTC list, then you've probably had the need (e.g., load testing) to assign thousands of IP addresses to a single interface. However, it's usually the case that one is assigning multiple addresses that all belong to the same subnet. In that case, the first such IP address is "primary" and all the additional addresses that belong to the same subnet are "secondary". My investigations tell me that "secondary" addresses are much lighter-weight. On the other hand, I don't seem to be able to assign more than about 280 "primary" addresses in the entire system before all the interfaces become unresponsive.

For additional information about "primary" vs. "secondary" addresses, take a look at:

http://www.linux-ip.net/html/linux-ip.html#tools-ip-address-add
http://www.linux-ip.net/gl/ip-cref/node33.html

Does anyone know why there appears to be a limit on the number of networks? Does anyone know where in the code this limit arises? Does anyone know a way to get around this limit?

#1 lore raw
Quoted 23 lines · Patrick Turley

I am trying to configure a Linux box with all possible VLANs (4094 of them), and a subnet on each VLAN. Creating the VLANs isn't a problem. But, when I try to use "ip addr add ..." commands to assign an IP address to each VLAN interface, I get to do about 280 of them before all the interfaces on the box become unresponsive. I've also tried to assign the same IP addresses all to eth0, and I get the same result.

The problem seems to hinge on the distinction between primary and secondary addresses.

If you're reading the LARTC list, then you've probably had the need (e.g., load testing) to assign thousands of IP addresses to a single interface. However, it's usually the case that one is assigning multiple addresses that all belong to the same subnet. In that case, the first such IP address is "primary" and all the additional addresses that belong to the same subnet are "secondary". My investigations tell me that "secondary" addresses are much lighter-weight. On the other hand, I don't seem to be able to assign more than about 280 "primary" addresses in the entire system before all the interfaces become unresponsive.

For additional information about "primary" vs. "secondary" addresses, take a look at:

Quoted 6 lines

http://www.linux-ip.net/html/linux-ip.html#tools-ip-address-add
http://www.linux-ip.net/gl/ip-cref/node33.html

Does anyone know why there appears to be a limit on the number of networks? Does anyone know where in the code this limit arises? Does anyone know a way to get around this limit?

i'm not aware of any limit. i believe 2.0/2.2 had an alias limit of 256 per interface, but i could be way off base... there's probably something going on with you particular machine (i.e. memory problem).

my test to prove there's not a limit at 4096:

$ uname -a
Linux wolfpack 2.4.22-1.2199.nptl.ipsec.pom #2 Wed Aug 11 13:55:37 EDT
2004 i686 i686 i386 GNU/Linux
$ sudo ip addr add 220.1.1.1/24 dev eth0
$ ip -o -4 addr sh eth0
2: eth0    inet 220.1.1.1/24 scope global eth0
$ bash test	# see the contents of "test" below
$ ip -4 addr sh eth0 | wc -l
   4354

box is still responsive on random sampling of IP's (17.247.1.1, 16.212.1.1, 10.218.1.1, etc)...

contents of "test" are:

----
i=1
function addIP() {
j=0
while [ $j -le 255 ]; do

    sudo ip addr add $1.$j.1.1/24 dev eth0
    let "j = $j + 1"

done
}

while [ $i -le 17 ]; do
addIP $i
let "i = $i + 1"
done
----

essentially--this script is adding IP's of 1.[0-255].1.1/24, 2.[0-255].1.1/24...up to 17.[0-255].1.1/24...for a total of 256*17 = 4352 IP's, all on different networks, all on eth0.

this is on my laptop i'm using to write this email (P4 2.2 GHz, 768 MB RAM, FC1)

if this is not a valid reproduction of what you're trying to do, let me know.

-j

Signature · 1 line
Jason Opperisano <[email protected]>
#2 lore raw
Quoted 8 lines · Jason Opperisano
Quoted 6 lines · Patrick Turley

I am trying to configure a Linux box with all possible VLANs (4094 of them), and a subnet on each VLAN. Creating the VLANs isn't a problem. But, when I try to use "ip addr add ..." commands to assign an IP address to each VLAN interface, I get to do about 280 of them before all the interfaces on the box become unresponsive. I've also tried to assign the same IP addresses all to eth0, and I get the same result.

.
.
.

Quoted 5 lines

i'm not aware of any limit. i believe 2.0/2.2 had an alias limit of 256 per interface, but i could be way off base... there's probably something going on with you particular machine (i.e. memory problem).

my test to prove there's not a limit at 4096:

.
.
.

Quoted 9 lines

contents of "test" are:

----
i=1
function addIP() {
j=0
while [ $j -le 255 ]; do

    sudo ip addr add $1.$j.1.1/24 dev eth0
    let "j = $j + 1"
Quoted 8 lines

done
}

while [ $i -le 17 ]; do
addIP $i
let "i = $i + 1"
done
----

First of all, thank you very much for the time you spent investigating this problem.

Your script works fine on my test computer. I was able to assign all 4000+ addresses with no problem.

I captured the commands emitted by your script and the commands emitted by our software and began the process of making them look more and more like each other until the essential difference became clear. I finally got to the point where the *only* difference was the IP addresses.

The addresses assigned by your script are (as you said) [1-17].[1-255].1.1/24.

The addresses I was assigning were 10.[1-2].[1-255].1/24.

So, the major difference seems to be that all my addresses were confined to the 10/8 subnet. It may be important to note that some of the addresses generated by your script *did* lie in that range, but the majority did not.

I adjusted your script very slightly to put all the addresses in the 10/8 subnet:

    i=1
    function addIP() {
      j=0
      while [ $j -le 255 ]; do
        sudo ip addr add 10.$1.$j.1/24 dev eth0
        let "j = $j + 1"
      done
    }

    while [ $i -le 2 ]; do
       addIP $i
       let "i = $i + 1"
    done

When I ran this version, I saw the very same failure I've been seeing all along.

Would it be possible for you to run the modified version of this script on your machine and see if you have any problem?

Thanks again for your time.

#3 lore raw
Quoted 19 lines · Patrick Turley

I adjusted your script very slightly to put all the addresses in the 10/8 subnet:

     i=1
     function addIP() {
       j=0
       while [ $j -le 255 ]; do
         sudo ip addr add 10.$1.$j.1/24 dev eth0
         let "j = $j + 1"
       done
     }

     while [ $i -le 2 ]; do
        addIP $i
        let "i = $i + 1"
     done
Quoted 7 lines

When I ran this version, I saw the very same failure I've been seeing all along.

Would it be possible for you to run the modified version of this script on your machine and see if you have any problem?

Thanks again for your time.

not trying to complicate things, but i had to test this on a diff machine (same HW, but running gentoo instead of FC1--can't blow up my laptop right this moment).

details:

# uname -a
Linux vmg2 2.4.26-gentoo-r9 #2 Fri Sep 3 07:13:35 EDT 2004 i686 Intel(R)
Pentium(R) 4 CPU 2.20GHz GenuineIntel GNU/Linux
# ip -4 add sh eth0 | wc -l
513
# ip -4 add sh eth0 | head
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    inet 10.1.0.1/24 scope global eth0
    inet 10.1.1.1/24 scope global eth0
    inet 10.1.2.1/24 scope global eth0
    inet 10.1.3.1/24 scope global eth0
    inet 10.1.4.1/24 scope global eth0
    inet 10.1.5.1/24 scope global eth0
    inet 10.1.6.1/24 scope global eth0
    inet 10.1.7.1/24 scope global eth0
    inet 10.1.8.1/24 scope global eth0
# ip -4 add sh eth0 | tail
    inet 10.2.246.1/24 scope global eth0
    inet 10.2.247.1/24 scope global eth0
    inet 10.2.248.1/24 scope global eth0
    inet 10.2.249.1/24 scope global eth0
    inet 10.2.250.1/24 scope global eth0
    inet 10.2.251.1/24 scope global eth0
    inet 10.2.252.1/24 scope global eth0
    inet 10.2.253.1/24 scope global eth0
    inet 10.2.254.1/24 scope global eth0
    inet 10.2.255.1/24 scope global eth0

from a machine assigned 10.1.1.100/16 and 10.2.1.100/16, i can ping:

10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1, 10.1.4.1, 10.2.254.1, 10.2.25.1, 10.1.25.1, 10.2.255.1

...on the test machine with all the 10.[1-2].[0-255].1/24 addresses.

we might be going about this all wrong...are you sure the problem is the linux box with all the IP's, and not something else? perhaps you're overflowing the MAC table of a connected switch (just a thought)?

-j

Signature · 1 line
Jason Opperisano <[email protected]>
#4 lore raw
Quoted 32 lines · Jason Opperisano

On Tue, 2004-09-21 at 14:51, Patrick Turley wrote:

# uname -a
Linux vmg2 2.4.26-gentoo-r9 #2 Fri Sep 3 07:13:35 EDT 2004 i686 Intel(R)
Pentium(R) 4 CPU 2.20GHz GenuineIntel GNU/Linux
# ip -4 add sh eth0 | wc -l
513
# ip -4 add sh eth0 | head
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    inet 10.1.0.1/24 scope global eth0
    inet 10.1.1.1/24 scope global eth0
    inet 10.1.2.1/24 scope global eth0
    inet 10.1.3.1/24 scope global eth0
    inet 10.1.4.1/24 scope global eth0
    inet 10.1.5.1/24 scope global eth0
    inet 10.1.6.1/24 scope global eth0
    inet 10.1.7.1/24 scope global eth0
    inet 10.1.8.1/24 scope global eth0
# ip -4 add sh eth0 | tail
    inet 10.2.246.1/24 scope global eth0
    inet 10.2.247.1/24 scope global eth0
    inet 10.2.248.1/24 scope global eth0
    inet 10.2.249.1/24 scope global eth0
    inet 10.2.250.1/24 scope global eth0
    inet 10.2.251.1/24 scope global eth0
    inet 10.2.252.1/24 scope global eth0
    inet 10.2.253.1/24 scope global eth0
    inet 10.2.254.1/24 scope global eth0
    inet 10.2.255.1/24 scope global eth0
Quoted 6 lines

from a machine assigned 10.1.1.100/16 and 10.2.1.100/16, i can ping:

10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1, 10.1.4.1, 10.2.254.1, 10.2.25.1, 10.1.25.1, 10.2.255.1

...on the test machine with all the 10.[1-2].[0-255].1/24 addresses.

I've found that ping is not a good test. Even with the networking broken, ping still works. Can you try to SSH/telnet/ftp/foo to the test machine?

#5 lore raw

I've discovered the source of this problem. Unfortunately, the cause is so embarassing that I'm not able to discuss it. I will now go commit sepuku. Thank you for your time and kind attention.

Quoted 44 lines · Patrick Turley
Quoted 32 lines · Jason Opperisano

On Tue, 2004-09-21 at 14:51, Patrick Turley wrote:

# uname -a
Linux vmg2 2.4.26-gentoo-r9 #2 Fri Sep 3 07:13:35 EDT 2004 i686 Intel(R)
Pentium(R) 4 CPU 2.20GHz GenuineIntel GNU/Linux
# ip -4 add sh eth0 | wc -l
513
# ip -4 add sh eth0 | head
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    inet 10.1.0.1/24 scope global eth0
    inet 10.1.1.1/24 scope global eth0
    inet 10.1.2.1/24 scope global eth0
    inet 10.1.3.1/24 scope global eth0
    inet 10.1.4.1/24 scope global eth0
    inet 10.1.5.1/24 scope global eth0
    inet 10.1.6.1/24 scope global eth0
    inet 10.1.7.1/24 scope global eth0
    inet 10.1.8.1/24 scope global eth0
# ip -4 add sh eth0 | tail
    inet 10.2.246.1/24 scope global eth0
    inet 10.2.247.1/24 scope global eth0
    inet 10.2.248.1/24 scope global eth0
    inet 10.2.249.1/24 scope global eth0
    inet 10.2.250.1/24 scope global eth0
    inet 10.2.251.1/24 scope global eth0
    inet 10.2.252.1/24 scope global eth0
    inet 10.2.253.1/24 scope global eth0
    inet 10.2.254.1/24 scope global eth0
    inet 10.2.255.1/24 scope global eth0
Quoted 6 lines

from a machine assigned 10.1.1.100/16 and 10.2.1.100/16, i can ping:

10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1, 10.1.4.1, 10.2.254.1, 10.2.25.1, 10.1.25.1, 10.2.255.1

...on the test machine with all the 10.[1-2].[0-255].1/24 addresses.

I've found that ping is not a good test. Even with the networking broken, ping still works. Can you try to SSH/telnet/ftp/foo to the test machine?