How do containers tie to multiple IP's on a NIC?

Hi,

In the containerless world, I often have multiple IPs assigned to a NIC. The scant documentation I can find on running containers only ever speaks of single IP assignment schemes. Can I have for example a box with a single NIC with 8 IPs assigned to it, where the host gets one IP, or perhaps alternately can see all 8 to run iptables across, but each of the containers can see only whichever IP or IPs are assigned to it?

If it can work this way, I'd appreciate any hints on the correct way to implement it. If it can't, I apologize for barking up the wrong tree, and will need to look at full virtualization methods which can.

Thanks,
Whit

What container userspace command are you using ? libvirt ? liblxc ? unshare --net ?

Thanks
-- Daniel

Which one do you recommend, considering what I'm trying to do with multiple IPs on a NIC? I haven't committed to one yet. Which utility do you expect future development will favor most? I'll be happy to use any tool which gets the job done, preferably one that has a future.

Thanks,
Whit

Well ... please don't consider what I will suggest as "preaching for its parish" ;)
(not sure it is a correct expression. It is a direct translation from French)

I would recommend to use the lxc tools, preferably the 0.7.1 version. These tools allow to do what you are expecting that is assign several Ip addresses to the same virtual nic.

They are available at:

http://lxc.sourceforge.net/download/lxc/lxc-0.7.1.tar.gz

an older version is certainly available on your distro.

As a quick start:

write a configuration file (eg. lxc.conf)

lxc.network.type=macvlan
lxc.network.link=eth0
lxc.network.flags=up
lxc.network.ipv4=1.2.3.4/24
lxc.network.ipv4=192.168.1.123/24
lxc.network.ipv4=10.0.0.23
lxc.network.ipv4=172.2.1.3

And then lxc-execute -n foo -f lxc.conf /bin/bash

In your shell you should have a new network with one interface and several IP addresses.

You can create much more complex configuration but I let you check if these tools fit your needs.

Thanks
-- Daniel

In English, "Preaching to the choir."

Will do.

Ah, then what I need to understand is the relationship of the virtual NIC to the real NIC. That is, some of what I set up is multi-purpose boxes, where the single machine functions as an iptables firewall, perhaps multi-homed to two ISPs, with 3 real NICs, one for the IP block assigned by each ISP, and one for the LAN - which might also have more than on IP on it. But these aren't just firewalls. They tend to serve a website or two, perhaps ftp, smtp, dns - spread over serveral of the IPs. They're also doing SNAT and DNAT for systems behind them.

It would make all sorts of sense to be adding containers to these systems, in terms of security, isolation, and the flexibility to easily migrate services to other servers. But unlike the more usual virtualization instance, where someone has a dozen different boxes and wants to consolidate them, I'm already fully consolidated. What I need to do is split things apart more, so they can go into containers, but still consolidated on boxes which continue to be multi-purpose, and where each single NIC may have over a dozen IPs assigned to it, but as a rule from within a single block per NIC.

I've seen discussions elsewhere (using Google to try to find hints for this) where people have given a machine two IPs on the same LAN by actually using two physical NICs (and then need to play STP tricks). My attitude is "Why use two pieces of hardware where one can do the job?"

Time for some trial-and-error with lxc tools.

Regards,
Whit

05.07.2010 07:08, Whit Blauvelt пишет:

What exactly are you trying to achieve? A transparent packet forwarding between containers and external networks?
_______________________________________________
Containers mailing list [email protected]
https://lists.linux-foundation.org/mailman/listinfo/containers

I'm trying to get the overview of what can be achieved, and how. Unless I've missed it, there's not much documentation on even moderately complex use of containers. Since the capabilities are rapidly advancing, maybe I'm just asking the question a few months too early? From the outside, as someone new to containers, it looks like a maze where there are a number of entrances, each of which may lead approximately to the goal, but some of which may be dead ends.

The examples I have found are along the lines of: Here's how to start a container, bridge it to a NIC which has a single IP assigned, ssh to it ... and the examples stop there. What I'd like to achieve is a setup where, say, a box with 6 IPs on an external network - on eth0 before bridging (or its alternatives) - can have 5 of those IPs each dedicated to different single container.

I'm not committed to a particular way of achieving that yet, just looking at the maze entrances wondering which to choose. Standard packet forwarding, routing and firewalling in Linux isn't what I'm asking about, I use that stuff frequently in complex ways, and I already run some things in simple chroots. I'm trying to learn how, on a test basis, to set up something like a production environment with lxc, where it involves multiple IPs, WAN or LAN, on each single NIC, behind some of which should be containers which effectively own individual IPs, publicly available.

It's probably looking harder to me than it is, because I haven't found a clear description of it yet.

Thanks,
Whit

Hi Whit,

may be this documents can help you:

http://lxc.sourceforge.net/doc/sigops/appcr.pdf

Thanks much Daniel. A clarifying and enjoyable read.

Whit

05.07.2010 22:07, Whit Blauvelt пишет:

I have a setup similar to that you describe. It's a bit hackish, but I like it's transparency and isolation capabilities (so I can leave CAP_NET_RAW and CAP_NET_ADMIN for containers). It looks like that:

For the host:

host # ip addr add 1.1.128.2/20 dev eth0
host # route add -net default gw 1.1.128.1

For lxc1:

host # cat /etc/lxc/lxc1/config | grep net
lxc.network.type = veth
lxc.network.veth.pair = lxc1_veth0
lxc.network.flags = up

host # arp -Ds 1.1.128.3 eth0 pub
host # arp -Ds 1.1.128.4 eth0 pub
host # arp -Ds 1.1.128.5 eth0 pub
host # arp -Ds 1.1.128.6 eth0 pub

host # sysctl -w net.ipv4.conf.lxc1_veth0.proxy_arp = 1

host # ip addr add 10.0.1.1/24 dev lxc1_veth0

host # route add -host 1.1.128.3 gw 10.0.1.2
host # route add -host 1.1.128.4 gw 10.0.1.2
host # route add -host 1.1.128.5 gw 10.0.1.2
host # route add -host 1.1.128.6 gw 10.0.1.2

lxc1 # ip addr add 10.0.1.2/24 dev lxc1_veth0

lxc1 # ip addr add 1.1.128.3/20 dev eth0
lxc1 # ip addr add 1.1.128.4/20 dev eth0
lxc1 # ip addr add 1.1.128.5/20 dev eth0
lxc1 # ip addr add 1.1.128.6/20 dev eth0

lxc1 # route add -net default gw 1.1.128.1

For lxc2:

host # cat /etc/lxc/lxc2/config | grep net
lxc.network.type = veth
lxc.network.veth.pair = lxc2_veth0
lxc.network.flags = up

host # arp -Ds 1.1.128.7 eth0 pub
host # sysctl -w net.ipv4.conf.lxc2_veth0.proxy_arp = 1 host # ip addr add 10.0.2.1/24 dev lxc2_veth0 host # route add -host 1.1.128.7 gw 10.0.2.2

lxc2 # ip addr add 10.0.2.2/24 dev eth0
lxc2 # ip addr add 1.1.128.7/20 dev eth0
lxc2 # route add -net default gw 1.1.128.1

Hope it helps. Btw, I use custom udev scripts to do the host part, and stock init scripts inside the containers to do the lxc part.
_______________________________________________
Containers mailing list [email protected]
https://lists.linux-foundation.org/mailman/listinfo/containers

06.07.2010 23:00, Pavel Labushev пишет:

Uh, it's eth0 instead of lxc1_veth0.
_______________________________________________
Containers mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/containers

The paper has probably already answered this but the sound bite answer is:

Each network namespace appears to userspace as separate instance of the network stack. Separate network device, separate forwarding tables, separate iptables rules etc. Network devices can be moved between network namespaces.

paired veth devices are interesting because you can put one end of a logical tunnel in each network namespace.

macvlan devices are interesting because you can create assign multiple mac addresses to a nic and have a different network device for each mac address.

You can use special tools like lxc to set these things up, but you can also just run commands inside the network namespace and setup the environment like you would normally.

Eric

These two statements could be section headers in a valuable article - or book chapter.

The problem with the standard VM concep: it takes the metaphor of separate computers too literally. The potential for lessening the materials expense and environmental cost of producing so much hardware and the electricity to power it - is huge. Conversion to containers instead of VMs can be a major economic win.

But only if the knowledge of how to do it becomes widespread.

Whit