introduction
Subsystems (10)
cover introduction
The following patches create a private "network namespace" for use within containers. This is intended for use with system containers like vserver, but might also be useful for restricting individual applications' access to the network stack.
These patches isolate traffic inside the network namespace. The network ressources, the incoming and the outgoing packets are identified to be related to a namespace.
It hides network resource not contained in the current namespace, but still allows administration of the network with normal commands like ifconfig.
It applies to the kernel version 2.6.17-rc6-mm1
It provides the following:
-------------------------
- when an application unshares its network namespace, it looses its view of all network devices by default. The administrator can choose to make any devices to become visible again. The container then gains a view to the device but without the ip address configured on it. It is up to the container administrator to use ifconfig or ip command to setup a new ip address. This ip address is only visible inside the container.
- the loopback is isolated inside the container and it is not possible to communicate between containers via the loopback.
- several containers can have an application bind to the same address:port without conflicting.
What is for ?
-------------
- security : an application can be bounded inside a container without interacting with the network used by another container
- consolidation : several instance of the same application can be ran in different container because the network namespace allows to bind to the same addr:port
What could be done ?
--------------------
- because the network ressources are related to a namespace, it is easy to identify them. That facilitate the implementation of the network migration
How to use ?
------------
- do unshare with the CLONE_NEWNET flag as root
- do echo eth0 > /sys/kernel/debug/net_ns/dev
- use ifconfig or ip command to set a new ip address
What is missing ?
-----------------
The routes are not yet isolated, that implies:
- binding to another container's address is allowed
- an outgoing packet which has an unset source address can potentially get another container's address
- an incoming packet can be routed to the wrong container if there are several containers listening to the same addr:port
--
RFC PATCH 1/6 Network namespace structure
This patch adds to the nsproxy the network namespace and a set of functions to unshare it. The network namespace structure should be filled later with the identified network ressources needed for more isolation.
Replace-Subject: [Network namespace] Network namespace structure
--
include/linux/init_task.h | 2 include/linux/net_ns.h | 59 ++++++++++++++++++++++++++++ include/linux/nsproxy.h | 2 include/linux/sched.h | 1 init/version.c | 8 +++ kernel/fork.c | 24 +++++++++-- kernel/nsproxy.c | 38 +++++++++++------- net/Kconfig | 9 ++++ net/Makefile | 1 net/net_ns.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++ 10 files changed, 222 insertions(+), 18 deletions(-)
include/linux/net_ns.h new file +59 -0
@@ -0,0 +1,59 @@ +#ifndef _LINUX_NET_NS_H +#define _LINUX_NET_NS_H + +#include <linux/kref.h> +#include <linux/sched.h> +#include <linux/nsproxy.h> + +struct net_namespace { +struct kref kref; +}; + +extern struct net_namespace init_net_ns; + +#ifdef CONFIG_NET_NS + +extern int unshare_network (unsigned long unshare_flags, +struct net_namespace **new_net); + +extern int copy_network (int flags,struct task_struct *tsk); + +static inline void get_net_ns (struct net_namespace *ns) +{ +kref_get (&ns->kref); +} + +void free_net_ns (struct kref *kref); + +static inline void put_net_ns (struct net_namespace *ns) +{ +kref_put (&ns->kref, free_net_ns); +} + +static inline void exit_network (struct task_struct *p) +{ +struct net_namespace *net_ns = p->nsproxy->net_ns; +if (net_ns) +put_net_ns (net_ns); +} +#else /* !CONFIG_NET_NS */ +static inline int unshare_network (unsigned long unshare_flags, +struct net_namespace **new_net) +{ +return -EINVAL ; +} +static inline int copy_network (int flags,struct task_struct *tsk) +{ +return 0 ; +} +static inline void get_net_ns (struct net_namespace *ns) {} +static inline void put_net_ns (struct net_namespace *ns) {} +static inline void exit_network (struct task_struct *p) {} +#endif /* CONFIG_NET_NS */ + +static inline struct net_namespace *net_ns (void ) +{ +return current->nsproxy->net_ns; +} + +#endif
net/net_ns.c new file +96 -0
96 lines not on this page · raw message
include/linux/nsproxy.h +2 -0
@@ -6,6 +6,7 @@struct namespace ;struct uts_namespace ; +struct net_namespace ;/* * A structure to contain pointers to all per-process
@@ -23,6 +24,7 @@ struct nsproxy {atomic_t count;spinlock_t nslock;struct uts_namespace *uts_ns; +struct net_namespace *net_ns;struct namespace *namespace; };extern struct nsproxy init_nsproxy;
kernel/nsproxy.c +25 -13
@@ -14,6 +14,7 @@#include <linux/nsproxy.h> #include <linux/namespace.h> #include <linux/utsname.h> +#include <linux/net_ns.h> static inline void get_nsproxy (struct nsproxy *ns) {
@@ -59,6 +60,8 @@ struct nsproxy *dup_namespaces(struct nsget_namespace (ns->namespace);if (ns->uts_ns)get_uts_ns (ns->uts_ns); +if (ns->net_ns) +get_net_ns (ns->net_ns); }return ns;
@@ -79,7 +82,7 @@ int copy_namespaces(int flags, struct taget_nsproxy (old_ns); -if (!(flags & (CLONE_NEWNS |CLONE_NEWUTS ))) +if (!(flags & (CLONE_NEWNS |CLONE_NEWUTS |CLONE_NEWNET )))return 0 ; new_ns =clone_namespaces (old_ns);
@@ -91,21 +94,28 @@ int copy_namespaces(int flags, struct ta tsk->nsproxy = new_ns; err =copy_namespace (flags, tsk); -if (err) { - tsk->nsproxy = old_ns; -put_nsproxy (new_ns); - goto out; - } +if (err) + goto bad_copy_namespace; err =copy_utsname (flags, tsk); -if (err) { -if (new_ns->namespace) -put_namespace (new_ns->namespace); - tsk->nsproxy = old_ns; -put_nsproxy (new_ns); - goto out; - } +if (err) + goto bad_copy_utsname; + err =copy_network (flags, tsk); +if (err) + goto bad_copy_network; + + goto out; + +bad_copy_network: +if (new_ns->uts_ns) +put_uts_ns (new_ns->uts_ns); +bad_copy_utsname: +if (new_ns->namespace) +put_namespace (new_ns->namespace); +bad_copy_namespace: + tsk->nsproxy = old_ns; +put_nsproxy (new_ns); out:put_nsproxy (old_ns);return err;
@@ -117,5 +127,7 @@ void free_nsproxy(struct nsproxy *ns)put_namespace (ns->namespace);if (ns->uts_ns)put_uts_ns (ns->uts_ns); +if (ns->net_ns) +put_net_ns (ns->net_ns);kfree (ns); }
include/linux/sched.h +1 -0
@@ -25,6 +25,7 @@#define CLONE_CHILD_SETTID 0x01000000/* set the TID in the child */ #define CLONE_STOPPED 0x02000000/* Start in stopped state */ #define CLONE_NEWUTS 0x04000000/* New utsname group? */ +#define CLONE_NEWNET 0x08000000/* New network namespace */ /* * Scheduling policies
net/Kconfig +9 -0
@@ -60,6 +60,15 @@ config INET Short answer: say Y. +config NET_NS + bool "Network namespaces" + depends on NET + default n + ---help--- + Support for network namespaces. This allows containers, i.e. + vservers, to use network namespaces to provide isolated + network for different servers. If unsure, say N. + if INET source "net/ipv4/Kconfig" source "net/ipv6/Kconfig"
net/Makefile +1 -0
@@ -50,3 +50,4 @@ obj-$(CONFIG_TIPC) += tipc/ ifeq ($(CONFIG_NET),y) obj-$(CONFIG_SYSCTL) += sysctl_net.o endif +obj-$(CONFIG_NET_NS) += net_ns.o
include/linux/init_task.h +2 -0
@@ -4,6 +4,7 @@#include <linux/file.h> #include <linux/rcupdate.h> #include <linux/utsname.h> +#include <linux/net_ns.h> #include <linux/interrupt.h> #define INIT_FDTABLE \
@@ -73,6 +74,7 @@ extern struct nsproxy init_nsproxy; .count = ATOMIC_INIT(1), \ .nslock = SPIN_LOCK_UNLOCKED, \ .uts_ns = &init_uts_ns, \ + .net_ns = &init_net_ns, \ .namespace = NULL, \ }
kernel/fork.c +19 -5
@@ -1592,13 +1592,15 @@ asmlinkage long sys_unshare(unsigned lonstruct sem_undo_list *new_ulist =NULL ;struct nsproxy *new_nsproxy =NULL , *old_nsproxy =NULL ;struct uts_namespace *uts, *new_uts =NULL ; +struct net_namespace *net, *new_net =NULL ;check_unshare_flags (&unshare_flags);/* Return -EINVAL for all unsupported flags */ err = -EINVAL ;if (unshare_flags & ~(CLONE_THREAD |CLONE_FS |CLONE_NEWNS |CLONE_SIGHAND | -CLONE_VM |CLONE_FILES |CLONE_SYSVSEM |CLONE_NEWUTS )) +CLONE_VM |CLONE_FILES |CLONE_SYSVSEM |CLONE_NEWUTS | +CLONE_NEWNET )) goto bad_unshare_out;if ((err =unshare_thread (unshare_flags)))
@@ -1617,18 +1619,20 @@ asmlinkage long sys_unshare(unsigned lon goto bad_unshare_cleanup_fd;if ((err =unshare_utsname (unshare_flags, &new_uts))) goto bad_unshare_cleanup_semundo; +if ((err =unshare_network (unshare_flags, &new_net))) + goto bad_unshare_cleanup_utsname; -if (new_ns || new_uts) { +if (new_ns || new_uts || new_net) { old_nsproxy = current->nsproxy; new_nsproxy =dup_namespaces (old_nsproxy);if (!new_nsproxy) { err = -ENOMEM ; - goto bad_unshare_cleanup_uts; + goto bad_unshare_cleanup_net; } }if (new_fs || new_ns || new_sigh || new_mm || new_fd || new_ulist || - new_uts) { + new_uts || new_net) {task_lock (current);
@@ -1676,13 +1680,23 @@ asmlinkage long sys_unshare(unsigned lon new_uts = uts; } +if (new_net) { + net = current->nsproxy->net_ns; + current->nsproxy->net_ns = new_net; + new_net = net; + } +task_unlock (current); }if (new_nsproxy)put_nsproxy (new_nsproxy); -bad_unshare_cleanup_uts: +bad_unshare_cleanup_net: +if (new_net) +put_net_ns (new_net); + +bad_unshare_cleanup_utsname:if (new_uts)put_uts_ns (new_uts);
init/version.c +8 -0
@@ -10,6 +10,7 @@#include <linux/module.h> #include <linux/uts.h> #include <linux/utsname.h> +#include <linux/net_ns.h> #include <linux/version.h> #include <linux/sched.h>
@@ -33,6 +34,13 @@ struct uts_namespace init_uts_ns = { };EXPORT_SYMBOL_GPL (init_uts_ns); +struct net_namespace init_net_ns = { + .kref = { + .refcount =ATOMIC_INIT (2 ), + }, +}; +EXPORT_SYMBOL_GPL (init_net_ns); +const char linux_banner[] ="Linux version " UTS_RELEASE " (" LINUX_COMPILE_BY "@" LINUX_COMPILE_HOST ") (" LINUX_COMPILER ") " UTS_VERSION "\n" ;
--
RFC PATCH 2/6 Network device sharing by view
In reply to [email protected]Adds to the network namespace a device list view. This view is emptied when the unshare is done. The view is filled/emptied by a set of function which can be called by an external module.
Replace-Subject: [Network namespace] Network device sharing by view
--
include/linux/net_ns.h | 2 include/linux/net_ns_dev.h | 32 +++++++ init/version.c | 4 net/core/Makefile | 2 net/core/net_ns_dev.c | 205 +++++++++++++++++++++++++++++++++++++++++++++ net/net_ns.c | 6 + 6 files changed, 250 insertions(+), 1 deletion(-)
include/linux/net_ns_dev.h new file +32 -0
@@ -0,0 +1,32 @@ +#ifndef _LINUX_NET_NS_DEV_H +#define _LINUX_NET_NS_DEV_H + +struct net_device ; + +struct net_ns_dev { +struct list_head list; +struct net_device *dev; +}; + +struct net_ns_dev_list { +struct list_head list; +rwlock_t lock; +}; + +extern int net_ns_dev_unregister (struct net_device *dev, +struct net_ns_dev_list *devlist); + +extern int net_ns_dev_register (struct net_device *dev, +struct net_ns_dev_list *devlist); + +extern struct net_device *net_ns_dev_find_by_name (const char *devname, +struct net_ns_dev_list *devlist); +extern int net_ns_dev_remove (const char *devname, +struct net_ns_dev_list *devlist); + +extern int net_ns_dev_add (const char *devname, +struct net_ns_dev_list *devlist); + +extern int free_net_ns_dev (struct net_ns_dev_list *devlist); + +#endif
include/linux/net_ns.h +2 -0
@@ -4,9 +4,11 @@#include <linux/kref.h> #include <linux/sched.h> #include <linux/nsproxy.h> +#include <linux/net_ns_dev.h> struct net_namespace {struct kref kref; +struct net_ns_dev_list dev_list; };extern struct net_namespace init_net_ns;
net/core/net_ns_dev.c new file +205 -0
205 lines not on this page · raw message
net/net_ns.c +6 -0
@@ -23,11 +23,16 @@struct net_namespace *clone_net_ns (struct net_namespace *old_ns) {struct net_namespace *new_ns; +struct net_ns_dev_list *new_dev_list; new_ns =kmalloc (sizeof (*new_ns),GFP_KERNEL );if (!new_ns)return NULL ; +kref_init (&new_ns->kref); + new_dev_list = &new_ns->dev_list; +INIT_LIST_HEAD (&new_dev_list->list); + new_dev_list->lock =RW_LOCK_UNLOCKED ;return new_ns; }
@@ -92,5 +97,6 @@ void free_net_ns(struct kref *kref)struct net_namespace *ns; ns = container_of(kref, struct net_namespace, kref); +free_net_ns_dev (&ns->dev_list);kfree (ns); }
net/core/Makefile +1 -1
@@ -7,7 +7,7 @@ obj-y := sock.o request_sock.o skbuff.o obj-$(CONFIG_SYSCTL) += sysctl_net_core.o -obj-y += dev.o ethtool.o dev_mcast.o dst.o \ +obj-y += dev.o net_ns_dev.o ethtool.o dev_mcast.o dst.o \ neighbour.o rtnetlink.o utils.o link_watch.o filter.o obj-$(CONFIG_XFRM) += flow.o
init/version.c +4 -0
@@ -38,6 +38,10 @@ struct net_namespace init_net_ns = { .kref = { .refcount = ATOMIC_INIT(2), }, + .dev_list = { + .lock = RW_LOCK_UNLOCKED, + .list = LIST_HEAD_INIT(init_net_ns.dev_list.list), + }, };EXPORT_SYMBOL_GPL (init_net_ns);
--
Re: [RFC] [patch 2/6] [Network namespace] Network device sharing by view
Quoted 9 lines · On Fri, 09 Jun 2006 23:02:04 +0200 [email protected] wrote:
net/core/net_ns_dev.c new file +9
-0+int net_ns_dev_add (const char *devname, +struct net_ns_dev_list *devlist) +{ +struct net_ns_dev *db; +struct net_device *dev; +int ret =0 ; + +read_lock (&dev_base_lock); +
net/core/net_ns_dev.c line 121
Quoted 10 lines
net/core/net_ns_dev.c new file +10
-0+for (dev = dev_base; dev; dev = dev->next) +if (!strncmp (dev->name, devname,IFNAMSIZ )) +break ; + +if (!dev) { + ret = -ENODEV ; + goto out; + } + + db =kmalloc (sizeof (*db),GFP_KERNEL );
sleep-in-spinlock. Please always test new code with all kernel debugging options enabled.
Re: [RFC] [patch 2/6] [Network namespace] Network device sharing by view
In reply to [email protected]net/core/net_ns_dev.c line 121
Quoted 12 lines · On Fri, Jun 09, 2006 at 11:02:04PM +0200, [email protected] wrote:
net/core/net_ns_dev.c new file +12
-0+read_lock (&dev_base_lock); + +for (dev = dev_base; dev; dev = dev->next) +if (!strncmp (dev->name, devname,IFNAMSIZ )) +break ; + +if (!dev) { + ret = -ENODEV ; + goto out; + } + + db =kmalloc (sizeof (*db),GFP_KERNEL );
deadlock.
Besides, holding references to net_device from userland is Not Good(tm).
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to [email protected]Hi Daniel,
It's good that you kicked off network namespace discussion. Although I wish you'd Cc'ed someone at OpenVZ so I could notice it earlier :).
Indeed, the first point to agree in this discussion is device list. In your patch, you essentially introduce a data structure parallel to the main device list, creating a "view" of this list.
I see a fundamental problem with this approach.
When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?
We have to conclude that each device should be visible only in one namespace. In this case, instead of introducing net_ns_dev and net_ns_dev_list structures, we can simply have a separate dev_base list head in each namespace. Moreover, separate device list in each namespace will be in line with making namespace isolation complete. Complete isolation will allow each namespace to set up own tun/tap devices, have own routes, netfilter tables, and so on.
My follow-up messages will contain the first set of patches with network namespaces implemented in the same way as network isolation in OpenVZ. This patchset introduces namespaces for device list and IPv4 FIB/routing. Two technical issues are omitted to make the patch idea clearer: device moving between namespaces, and selective routing cache flush + garbage collection.
If this patchset is agreeable, the next patchset will finalize integration with nsproxy, add namespaces to socket lookup code and neighbour cache, and introduce a simple device to pass traffic between namespaces. Then we will turn to less obvious matters including netlink messages, network statistics, representation of network information in proc and sysfs, tuning of parameters through sysctl, IPv6 and other protocols, and per-namespace netfilters.
Best regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 9 lines · Andrey Savochkin
Hi Daniel,
It's good that you kicked off network namespace discussion Although I.
wish you'd Cc'ed someone at OpenVZ so I could notice it earlier :) .Indeed, the first point to agree in this discussion is device list. In your patch, you essentially introduce a data structure parallel to the main device list, creating a "view" of this list.
Quoted 7 lines
I see a fundamental problem with this approach. When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?
this is something which isn't a fundamental problem at all, and IMHO there are at least three options here (probably more)
- check at 'bind' time if the binding would overlap and give the 'proper' error (as it happens right now on the host)
(this is how Linux-VServer currently handles the network isolation, and yes, it works quite fine :)
- allow arbitrary binds and 'tag' the packets according to some 'host' policy (e.g. iptables or tc) (this is how the Linux-VServer ngnet was designed)
- deliver packets to _all_ bound sockets/destinations (this is probably a more unusable but quite thinkable solution)
Quoted 2 lines
We have to conclude that each device should be visible only in one namespace.
I disagree here, especially some supervisor context or the host context should be able to 'see' and probably manipulate _all_ of the devices
Quoted 7 lines
In this case, instead of introducing
net_ns_devandnet_ns_dev_liststructures, we can simply have a separatedev_baselist head in each namespace. Moreover, separate device list in each namespace will be in line with making namespace isolation complete.Complete isolation will allow each namespace to set up own tun/tap devices, have own routes, netfilter tables, and so on.
tun/tap devices are quite possible with this approach too, I see no problem here ...
for iptables and routes, I'm worried about the required 'policy' to make them secure, i.e. how do you ensure that the packets 'leaving' guest X do not contain 'evil' packets and/or disrupt your host system?
Quoted 3 lines
My follow-up messages will contain the first set of patches with network namespaces implemented in the same way as network isolation in OpenVZ.
hmm, you probably mean 'network virtualization' here
Quoted 9 lines
This patchset introduces namespaces for device list and IPv4 FIB/routing. Two technical issues are omitted to make the patch idea
clearer: device moving between namespaces, and selective routing cache flush + garbage collection.If this patchset is agreeable, the next patchset will finalize integration with nsproxy, add namespaces to socket lookup code and neighbour cache, and introduce a simple device to pass traffic between namespaces.
passing traffic 'between' namespaces should happen via lo, no? what kind of 'device' is required there, and what overhead does it add to the networking?
TIA,
Herbert
Quoted 7 lines
Then we will turn to less obvious matters including netlink messages, network statistics, representation of network information in proc and sysfs, tuning of parameters through sysctl, IPv6 and other protocols, and per-namespace netfilters.
Best regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 14 lines · Herbert Poetzl
Quoted 12 lines · Andrey Savochkin
Hi Daniel,
It's good that you kicked off network namespace discussion Although I.
wish you'd Cc'ed someone at OpenVZ so I could notice it earlier :) .Indeed, the first point to agree in this discussion is device list. In your patch, you essentially introduce a data structure parallel to the main device list, creating a "view" of this list.
I see a fundamental problem with this approach. When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Quoted 7 lines
Quoted 3 lines
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?
this is something which isn't a fundamental problem at all, and IMHO there are at least three options here (probably more)
I agree that there are other implementations that can be used for containers. However when you think namespaces this is what you need.
For several reasons.
1) So you can use AF_PACKET safely.
This allows a network namespace to use DHCP and all of the other usual network autoconfiguration tools. 0.0.0.0:80 is just a special subset of that.
2) It means the existing network stack can be used without logic changes. All that is needed is a lookup of the appropriate context. This is very straight forward to audit.
3) Since all of the network stack is trivially available all of the advanced network stack features like iptables are easily available.
4) There is no retraining or other rules for user to learn. Because people understand what is going on it is more likely a setup will be secure. Most of the other implementations don't quite act like a normal network setup and the special rules can be hard to learn.
Quoted 5 lines
- check at 'bind' time if the binding would overlap and give the 'proper' error (as it happens right now on the host)
(this is how Linux-VServer currently handles the network isolation, and yes, it works quite fine :)
It works yes but it limits you to a subset of the network
stack. And has serious problems with concepts like INADDR_ANY.PF_PACKET is not an option.
Quoted 3 lines
- allow arbitrary binds and 'tag' the packets according to some 'host' policy (e.g. iptables or tc) (this is how the Linux-VServer ngnet was designed)
A little more general but very weird.
Quoted 10 lines
- deliver packets to _all_ bound sockets/destinations (this is probably a more unusable but quite thinkable solution)
Quoted 2 lines
We have to conclude that each device should be visible only in one namespace.
I disagree here, especially some supervisor context or the host context should be able to 'see' and probably manipulate _all_ of the devices
This part really is necessary. This does not preclude managing a network namespace from outside of the namespace.
Quoted 8 lines
Quoted 7 lines
In this case, instead of introducing
net_ns_devandnet_ns_dev_liststructures, we can simply have a separatedev_baselist head in each namespace. Moreover, separate device list in each namespace will be in line with making namespace isolation complete.Complete isolation will allow each namespace to set up own tun/tap devices, have own routes, netfilter tables, and so on.
Quoted 7 lines
tun/tap devices are quite possible with this approach too, I see no problem here ...
for iptables and routes, I'm worried about the required 'policy' to make them secure, i.e. how do you ensure that the packets 'leaving' guest X do not contain 'evil' packets and/or disrupt your host system?
In the traditional ways. When you control the router and/or the switch someone is directly connected to.
We don't need to reinvent the wheel if we do this properly.
Quoted 5 lines
Quoted 5 lines
This patchset introduces namespaces for device list and IPv4 FIB/routing. Two technical issues are omitted to make the patch idea
clearer: device moving between namespaces, and selective routing cache flush + garbage collection.
Quoted 8 lines
Quoted 4 lines
If this patchset is agreeable, the next patchset will finalize integration with nsproxy, add namespaces to socket lookup code and neighbour cache, and introduce a simple device to pass traffic between namespaces.
passing traffic 'between' namespaces should happen via lo, no? what kind of 'device' is required there, and what overhead does it add to the networking?
Definitely not. lo is a local loopback interface.
What is needed is a two headed device that is the cousin of lo. But with one network interface in each network namespace.
Note even connecting network namespaces is optional.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Herbert PoetzlHi Herbert,
Quoted 14 lines · Herbert Poetzl
Quoted 7 lines · Andrey Savochkin
I see a fundamental problem with this approach. When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?
this is something which isn't a fundamental problem at all, and IMHO there are at least three options here (probably more)
Quoted 5 lines
- check at 'bind' time if the binding would overlap and give the 'proper' error (as it happens right now on the host)
(this is how Linux-VServer currently handles the network isolation, and yes, it works quite fine :)
I'm not comfortable with this as a permanent mainstream solution. It means that network namespaces are actually not namespaces: you can't run some program (e.g., apache) with default configs in a new namespace without regards to who runs what in other namespaces.
In other words, name "0.0.0.0:80" creates a collision in your implementation, so socket "names" do not form isolated spaces.
Quoted 8 lines
- allow arbitrary binds and 'tag' the packets according to some 'host' policy (e.g. iptables or tc) (this is how the Linux-VServer ngnet was designed)
- deliver packets to _all_ bound sockets/destinations (this is probably a more unusable but quite thinkable solution)
Deliver TCP packets to all sockets?
How many connections do you expect to be established in this case?
Quoted 7 lines
Quoted 3 lines
We have to conclude that each device should be visible only in one namespace.
I disagree here, especially some supervisor context or the host context should be able to 'see' and probably manipulate _all_ of the devices
Right, manipulating all devices from some supervisor context is useful.
But this shouldn't necessarily be done by regular ip/ifconfig tools. Besides, it could be quite confusing if in ifconfig output in the supervisor context you see 325 "tun0" devices coming from different namespaces :)
So I'm all for network namespace management mechanisms not bound to existing tools/APIs.
Quoted 11 lines
Quoted 3 lines
Complete isolation will allow each namespace to set up own tun/tap devices, have own routes, netfilter tables, and so on.
tun/tap devices are quite possible with this approach too, I see no problem here ...
for iptables and routes, I'm worried about the required 'policy' to make them secure, i.e. how do you ensure that the packets 'leaving' guest X do not contain 'evil' packets and/or disrupt your host system?
Sorry, I don't get your point.
How do you ensure that packets leaving your neighbor's computer do not disrupt your system?
Quoted 8 lines
From my point of view, network namespaces are just neighbors.
Quoted 3 lines
My follow-up messages will contain the first set of patches with network namespaces implemented in the same way as network isolation in OpenVZ.
hmm, you probably mean 'network virtualization' here
I meant isolation between different network contexts/namespaces.
Quoted 6 lines
Quoted 6 lines
This patchset introduces namespaces for device list and IPv4 FIB/routing. Two technical issues are omitted to make the patch idea
clearer: device moving between namespaces, and selective routing cache flush + garbage collection.
Quoted 8 lines
Quoted 4 lines
If this patchset is agreeable, the next patchset will finalize integration with nsproxy, add namespaces to socket lookup code and neighbour cache, and introduce a simple device to pass traffic between namespaces.
passing traffic 'between' namespaces should happen via lo, no? what kind of 'device' is required there, and what overhead does it add to the networking?
OpenVZ provides 2 options.
1) A packet appears right inside some namespace, without any additional overhead. Usually this implies that either all packets from this device belong to this namespace, i.e. simple device->namespace assignment. However, there is nothing conceptually wrong with having namespace-aware device drivers or netfilter modules selecting namespaces for each incoming packet. It all depends on how you want packets go through various network layers, and how much network management abilities you want to have in non-root namespaces.
My point is that for network namespaces being real namespaces, decision making should be done somewhere before socket lookup.
2) Parent network namespace acts as a router forwarding packets to child namespaces. This scheme is the preferred one in OpenVZ for various reasons, most important being the simplicity of migration of network namespaces. In this case flexibility has the cost of going through packet handling layers two times.
Technically, this is implemented via a simple netdevice doing netif_rx in hard_xmit.
Regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 23 lines · Andrey Savochkin
Hi Herbert,
Quoted 14 lines · Herbert Poetzl
Quoted 7 lines · Andrey Savochkin
I see a fundamental problem with this approach. When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?
this is something which isn't a fundamental problem at all, and IMHO there are at least three options here (probably more)
Quoted 5 lines
- check at 'bind' time if the binding would overlap and give the 'proper' error (as it happens right now on the host)
(this is how Linux-VServer currently handles the network isolation, and yes, it works quite fine :)
Quoted 4 lines
I'm not comfortable with this as a permanent mainstream solution. It means that network namespaces are actually not namespaces: you can't run some program (e.g., apache) with default configs in a new namespace without regards to who runs what in other namespaces.
not at all, maybe you should take a closer look at the current Linux-VServer implementation, which is quite simple and _does_ allow guests to bind to IP_ANY quite fine, only the host (which has all priviledes) has to be careful with binding to 0.0.0.0 ...
Quoted 8 lines
In other words, name "0.0.0.0:80" creates a collision in your implementation, so socket "names" do not form isolated spaces.
Quoted 5 lines
- allow arbitrary binds and 'tag' the packets according to some 'host' policy (e.g. iptables or tc) (this is how the Linux-VServer ngnet was designed)
Quoted 6 lines
Quoted 3 lines
- deliver packets to _all_ bound sockets/destinations (this is probably a more unusable but quite thinkable solution)
Deliver TCP packets to all sockets?
How many connections do you expect to be established in this case?
well, roughly the same number of connections you'll get when you have two boxes with the same IP on the same subnet :)
in other words, if there are more than one guest with the same ip and port open, then we have some kind of misconfiguration (i.e. policy is required)
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
We have to conclude that each device should be visible only in one namespace.
I disagree here, especially some supervisor context or the host context should be able to 'see' and probably manipulate _all_ of the devices
Quoted 6 lines
Right, manipulating all devices from some supervisor context is useful.
But this shouldn't necessarily be done by regular ip/ifconfig tools. Besides, it could be quite confusing if in ifconfig output in the supervisor context you see 325 "tun0" devices coming from different namespaces :)
isolation would not provide more than _one_ tun0 interfaces, virtualization OTOH will ...
Quoted 2 lines
So I'm all for network namespace management mechanisms not bound to existing
tools/APIs.
well, I'm not against new APIs/tools, but I prefer to keep it simple, and elegant, which often includes reusing existing APIs and tools ...
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Complete isolation will allow each namespace to set up own tun/tap devices, have own routes, netfilter tables, and so on.
tun/tap devices are quite possible with this approach too, I see no problem here ...
Quoted 8 lines
Quoted 4 lines
for iptables and routes, I'm worried about the required 'policy' to make them secure, i.e. how do you ensure that the packets 'leaving' guest X do not contain 'evil' packets and/or disrupt your host system?
Sorry, I don't get your point.
How do you ensure that packets leaving your neighbor's computer do not disrupt your system?
by having a strong 'policy' on the router/switch which will (hopefully) reject everything sent in error or to disrupt/harm other boxes ...
Quoted 1 line
Quoted 1 line
From my point of view, network namespaces are just neighbors.
yes, but you _need_ policy there the same way you need it for resources, i.e. you cannot simply allow everyone to do everything with his network interface, especially if that interface is _shared_ with all others ...
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
My follow-up messages will contain the first set of patches with network namespaces implemented in the same way as network isolation in OpenVZ.
hmm, you probably mean 'network virtualization' here
I meant isolation between different network contexts/namespaces.
well, isolation is basically what we do in Linux-VServer by allowing to bind to certain IPs (or ranges) instead of binding _all_ available IPs ... this can be extended for routing and iptables as well, and does not require any 'virtualization' which would give each guest it's own set of interfaces, routes, iptables etc ... and it is usually more lightweight too ..
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
This patchset introduces namespaces for device list and IPv4 FIB/routing. Two technical issues are omitted to make the patch idea clearer: device moving between namespaces, and selective routing cache flush + garbage collection.
Quoted 8 lines
Quoted 4 lines
If this patchset is agreeable, the next patchset will finalize integration with nsproxy, add namespaces to socket lookup code and neighbour cache, and introduce a simple device to pass traffic between namespaces.
passing traffic 'between' namespaces should happen via lo, no? what kind of 'device' is required there, and what overhead does it add to the networking?
OpenVZ provides 2 options.
Quoted 11 lines
1) A packet appears right inside some namespace, without any additional overhead. Usually this implies that either all packets from this device belong to this namespace, i.e. simple
device->namespaceassignment. However, there is nothing conceptually wrong with having namespace-aware device drivers or netfilter modules selecting namespaces for each incoming packet. It all depends on how you want packets go through various network layers, and how much network management abilities you want to have in non-root namespaces. My point is that for network namespaces being real namespaces, decision making should be done somewhere before socket lookup.
well, I doubt that many providers will be able to put roughly hundred or more network interface cards into their machines, plus a proper switch to do the policy :)
Quoted 8 lines
2) Parent network namespace acts as a router forwarding packets to child namespaces. This scheme is the preferred one in OpenVZ for various reasons, most important being the simplicity of migration of network namespaces. In this case flexibility has the cost of going through packet handling layers two times.
Technically, this is implemented via a simple netdevice doing
netif_rxinhard_xmit.
which results in a duplicate stack traversal and kernel side policy to decide which goes where ... i.e. at least twice as much overhead than any isolation would have
best,
Herbert
Quoted 3 lines
Regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 7 lines · Herbert Poetzl
On Mon, Jun 26, 2006 at 06:08:03PM +0400, Andrey Savochkin wrote:
not at all, maybe you should take a closer look at the current Linux-VServer implementation, which is quite simple and _does_ allow guests to bind to
IP_ANYquite fine, only the host (which has all priviledes) has to be careful with binding to 0.0.0.0 ...
It works, and is a reasonable implementation. However the semantics change.
The real practical problem is that you loose power, and the ability to migrate applications. Not that this precludes you from loading a security module and doing what you do now.
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
We have to conclude that each device should be visible only in one namespace.
I disagree here, especially some supervisor context or the host context should be able to 'see' and probably manipulate _all_ of the devices
Right, manipulating all devices from some supervisor context is useful.
Quoted 7 lines
Quoted 4 lines
But this shouldn't necessarily be done by regular ip/ifconfig tools. Besides, it could be quite confusing if in ifconfig output in the supervisor context you see 325 "tun0" devices coming from different namespaces :)
isolation would not provide more than _one_ tun0 interfaces, virtualization OTOH will ...
Think layer 2 isolation not layer 3 isolation.
Quoted 6 lines
Quoted 2 lines
So I'm all for network namespace management mechanisms not bound to existing
tools/APIs.well, I'm not against new APIs/tools, but I prefer to keep it simple, and elegant, which often includes reusing existing APIs and tools ...
And knowledge. Except for the single IP per guest case filtering at BIND time starts show some surprising semantics.
Quoted 3 lines
by having a strong 'policy' on the router/switch which will (hopefully) reject everything sent in error or to disrupt/harm other boxes ...
And linux has a software router and switch capabilities so those can easily be used unmodified.
Quoted 6 lines
Quoted 1 line
Quoted 1 line
From my point of view, network namespaces are just neighbors.
yes, but you _need_ policy there the same way you need it for resources, i.e. you cannot simply allow everyone to do everything with his network interface, especially if that interface is _shared_ with all others ...
Agreed. And the network stack seems to have a perfectly good set of utilities to handle that already.
Quoted 8 lines
well, isolation is basically what we do in Linux-VServer by allowing to bind to certain IPs (or ranges) instead of binding _all_ available IPs ... this can be extended for routing and iptables as well, and does not require any 'virtualization' which would give each guest it's own set of interfaces, routes, iptables etc ... and it is usually more lightweight too ..
I disagree with the cost. Done properly we should have the cost of the existing networking stack plus the cost of an extra pointer dereference when we look at global variables.
This is layer 2 isolation. So we can use protocols like DHCP, unmodified.
In the normally accepted definition it isn't virtualization because we aren't emulating anything.
Quoted 28 lines
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
This patchset introduces namespaces for device list and IPv4 FIB/routing. Two technical issues are omitted to make the patch idea clearer: device moving between namespaces, and selective routing cache flush + garbage collection.
Quoted 8 lines
Quoted 4 lines
If this patchset is agreeable, the next patchset will finalize integration with nsproxy, add namespaces to socket lookup code and neighbour cache, and introduce a simple device to pass traffic between namespaces.
passing traffic 'between' namespaces should happen via lo, no? what kind of 'device' is required there, and what overhead does it add to the networking?
OpenVZ provides 2 options.
Quoted 11 lines
1) A packet appears right inside some namespace, without any additional overhead. Usually this implies that either all packets from this device belong to this namespace, i.e. simple
device->namespaceassignment. However, there is nothing conceptually wrong with having namespace-aware device drivers or netfilter modules selecting namespaces for each incoming packet. It all depends on how you want packets go through various network layers, and how much network management abilities you want to have in non-root namespaces. My point is that for network namespaces being real namespaces, decision making should be done somewhere before socket lookup.
Quoted 3 lines
well, I doubt that many providers will be able to put roughly hundred or more network interface cards into their machines, plus a proper switch to do the policy :)
Well switches exist. But yes because physical hardware
is limited this is a limited policy.
Quoted 6 lines
Quoted 5 lines
2) Parent network namespace acts as a router forwarding packets to child namespaces. This scheme is the preferred one in OpenVZ for various reasons, most important being the simplicity of migration of network namespaces. In this case flexibility has the cost of going through packet handling layers two times.
Quoted 6 lines
Quoted 2 lines
Technically, this is implemented via a simple netdevice doing
netif_rxinhard_xmit.which results in a duplicate stack traversal and kernel side policy to decide which goes where ... i.e. at least twice as much overhead than any isolation would have
Not twice because you don't traverse the entire network stack. Just up to the routing layer and then across, and there are optimization possibilities that should keep it down to a single traversal of the network stack.
Note: We are not encouraging saying that the linux-vserver implementation must die. Only that we are solving something with much larger scope.
If the first case does not at least pass packets as fast as the existing network stack I doubt we will be allowed to merge it. By making the nic drivers smarter we can have a single driver that creates multiple network interfaces simply by looking at the destination mac address, sort of like the bonding driver in reverse. That will trivially remove the extra network stack traversals if we don't want to apply before we let the packet out on the wire.
And there is not requirement that after the namespace is setup we leave any applications on the inside with CAP_NET_ADMIN so we don't need to worry about user space applications changing the network configurations if we don't want to.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Andrey SavochkinQuoted 1 line · Andrey Savochkin
Hi Daniel,
Hi Andrey,
Quoted 3 lines
It's good that you kicked off network namespace discussion. Although I wish you'd Cc'ed someone at OpenVZ so I could notice it earlier :).
Quoted 5 lines
When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?
Exact. But, the idea was to retrieve the namespace from the routes.
IMHO, I think there are roughly 2 network isolation implementation:
- make all network ressources private to the namespace
- keep a "flat" model where network ressources have a new identifier
which is the network namespace pointer. The idea is to move only some network informations private to the namespace (eg port range, stats, ...)
Daniel.
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 10 lines · Daniel Lezcano
Quoted 1 line · Andrey Savochkin
Hi Daniel,
Hi Andrey,
Quoted 2 lines
It's good that you kicked off network namespace discussion. Although I wish you'd Cc'ed someone at OpenVZ so I could notice it earlier :).
Quoted 7 lines
Quoted 5 lines
When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?Exact. But, the idea was to retrieve the namespace from the routes.
The problem is that if you start at the routes you have to do things at layer 3 and you can't do anything at layer 2. (i.e. You can't use DHCP). You loose a whole lot of flexibility and power when you make it a layer 3 only mechanism.
Quoted 7 lines
IMHO, I think there are roughly 2 network isolation implementation:
- make all network ressources private to the namespace
- keep a "flat" model where network ressources have a new identifier
which is the network namespace pointer. The idea is to move only some network informations private to the namespace (eg port range, stats, ...)
The problem is that you have to add a lot of new logic which is very hard to get right and has some really weird corner cases that are very hard to understand.
- That makes the patches hard to review.
- It makes it hard for the implementors to get it right.
- It means that there will be corner cases that the users don't understand.
- It is less flexible/powerful in what you can express?
I've been down that route it sucks. Anything more than the simple filter at bind time is asking for real trouble until you do the whole thing.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Daniel LezcanoDaniel,
Quoted 6 lines · Daniel Lezcano
Quoted 3 lines · Andrey Savochkin
It's good that you kicked off network namespace discussion. Although I wish you'd Cc'ed someone at OpenVZ so I could notice it earlier :).
[email protected] is fine
Quoted 8 lines
Quoted 6 lines
When a device presents an skb to the protocol layer, it needs to know to which namespace this skb belongs.
Otherwise you would never get rid of problems with bind: what to do if device eth1 is visible in namespace1, namespace2, and root namespace, and each namespace has a socket bound to 0.0.0.0:80?Exact. But, the idea was to retrieve the namespace from the routes.
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
Quoted 8 lines
IMHO, I think there are roughly 2 network isolation implementation:
- make all network ressources private to the namespace
- keep a "flat" model where network ressources have a new identifier
which is the network namespace pointer. The idea is to move only some network informations private to the namespace (eg port range, stats, ...)
Sorry, I don't get the second idea with only some information private to namespace.
How do you want TCP_INC_STATS macro look?
In my concept, it would be something like
#define TCP_INC_STATS(field) SNMP_INC_STATS(current_net_ns->tcp_stat, field) where tcp_stat is a TCP statistics array inside net_namespace.
Regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 5 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
I mean instead of having the route tables private to the namespace, the routes have the information to which namespace they are associated.
Quoted 9 lines
- keep a "flat" model where network ressources have a new identifier
Quoted 1 line
which is the network namespace pointer. The idea is to move only some network informations private to the namespace (eg port range, stats, ...)
Sorry, I don't get the second idea with only some information private to namespace.
How do you want
TCP_INC_STATSmacro look?
I was thinking in TCP_INC_STATS(net_ns, field) SNMP_INC_STATS(net_ns->tcp_stat, field)
Quoted 3 lines
In my concept, it would be something like
#defineTCP_INC_STATS(field)SNMP_INC_STATS(current_net_ns->tcp_stat, field)wheretcp_statis a TCP statistics array insidenet_namespace.
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 7 lines · Daniel Lezcano
Quoted 4 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
I mean instead of having the route tables private to the namespace, the routes have the information to which namespace they are associated.
Is this an implementation difference or is this a user visible difference? As an implementation difference this is sensible, as it is pretty insane to allocate hash tables at run time.
As a user visible difference that affects semantics of the operations this is not something we want to do.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 10 lines · Eric W. Biederman
Quoted 7 lines · Daniel Lezcano
Quoted 4 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
I mean instead of having the route tables private to the namespace, the routes have the information to which namespace they are associated.
Quoted 6 lines
Is this an implementation difference or is this a user visible difference? As an implementation difference this is sensible, as it is pretty insane to allocate hash tables at run time.
As a user visible difference that affects semantics of the operations this is not something we want to do.
well, I guess there are even more options here, for example I'd like to propose the following idea, which might be a viable solution for the policy/isolation problem, with the actual overhead on the setup part not the hot pathes for packet and connection handling
we could use the multiple routing tables to provide a single routing table for each guest, which could be used inside the guest to add arbitrary routes, but would allow to keep the 'main' policy on the host, by selecting the proper table based on IPs and guest tags
similar we could allow to have a separate iptables chain for each guest (or several chains), which are once again directed by the host system (applying the required prolicy) which can be managed and configured via normal iptable interfaces (both on the guest and host) but actually provide at least to layers
note: this does not work for hierarchical network contexts, but I do not see that the yet proposed implementations would do, so I do not think that is of concern here ...
best,
Herbert
Quoted 1 line
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 31 lines · Herbert Poetzl
Quoted 8 lines · Eric W. Biederman
Quoted 6 lines · Daniel Lezcano
Quoted 4 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
I mean instead of having the route tables private to the namespace, the
routes
Quoted 8 lines
Quoted 1 line
have the information to which namespace they are associated.
Is this an implementation difference or is this a user visible difference? As an implementation difference this is sensible, as it is pretty insane to allocate hash tables at run time.
As a user visible difference that affects semantics of the operations this is not something we want to do.
well, I guess there are even more options here, for example I'd like to propose the following idea, which might be a viable solution for the policy/isolation problem, with the actual overhead on the setup part not the hot pathes for packet and connection handling
we could use the multiple routing tables to provide a single routing table for each guest, which could be used inside the guest to add arbitrary routes, but would allow to keep the 'main' policy on the host, by selecting the proper table based on IPs and guest tags
Quoted 6 lines
similar we could allow to have a separate iptables chain for each guest (or several chains), which are once again directed by the host system (applying the required prolicy) which can be managed and configured via normal iptable interfaces (both on the guest and host) but actually provide at least to layers
I have real concerns about the complexity of the route you have described.
Quoted 4 lines
note: this does not work for hierarchical network contexts, but I do not see that the yet proposed implementations would do, so I do not think that is of concern here ...
Well we are hierarchical in the sense that a parent can have a different network namespace then a child. So recursive containers work fine. So this is like the uts namespace or the ipc namespace rather than like the pid namespace.
I really do not believe we have a hotpath issue, if this is implemented properly. Benchmarks of course need to be taken, to prove this.
There are only two places a sane implementation should show issues.
- When the access to a pointer goes through a pointer to find that global variable.
- When doing a lookup in a hash table we need to look at an additional field to verify a hash match. Because having a completely separate hash table is likely too expensive.
If that can be shown to really slow down packets on the hot path I am willing to consider other possibilities. Until then I think we are on path to the simplest and most powerful version of building a network namespace usable by containers.
The routing between network namespaces does have the potential to be more expensive than just a packet trivially coming off the wire into a socket. However that is fundamentally from a lack of hardware. If the rest works smarter filters in the drivers should enable to remove the cost.
Basically it is just a matter of:
if (dest_mac == my_mac1) it is for device 1.
If (dest_mac == my_mac2) it is for device 2.
etc.
At a small count of macs it is trivial to understand it will go fast for a larger count of macs it only works with a good data structure. We don't hit any extra cache lines of the packet, and the above test can be collapsed with other routing lookup tests.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 48 lines · Eric W. Biederman
Quoted 31 lines · Herbert Poetzl
Quoted 8 lines · Eric W. Biederman
Quoted 6 lines · Daniel Lezcano
Quoted 4 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
I mean instead of having the route tables private to the namespace, the
routes
Quoted 8 lines
Quoted 1 line
have the information to which namespace they are associated.
Is this an implementation difference or is this a user visible difference? As an implementation difference this is sensible, as it is pretty insane to allocate hash tables at run time.
As a user visible difference that affects semantics of the operations this is not something we want to do.
well, I guess there are even more options here, for example I'd like to propose the following idea, which might be a viable solution for the policy/isolation problem, with the actual overhead on the setup part not the hot pathes for packet and connection handling
we could use the multiple routing tables to provide a single routing table for each guest, which could be used inside the guest to add arbitrary routes, but would allow to keep the 'main' policy on the host, by selecting the proper table based on IPs and guest tags
Quoted 6 lines
similar we could allow to have a separate iptables chain for each guest (or several chains), which are once again directed by the host system (applying the required prolicy) which can be managed and configured via normal iptable interfaces (both on the guest and host) but actually provide at least to layers
I have real concerns about the complexity of the route you have described.
Quoted 4 lines
note: this does not work for hierarchical network contexts, but I do not see that the yet proposed implementations would do, so I do not think that is of concern here ...
Quoted 5 lines
Well we are hierarchical in the sense that a parent can have a different network namespace then a child. So recursive containers work fine. So this is like the uts namespace or the ipc namespace rather than like the pid namespace.
yes, but you will not be able to apply policy on the parent, restricting the child networking in a proper way without jumping through hoops ...
Quoted 3 lines
I really do not believe we have a hotpath issue, if this is implemented properly. Benchmarks of course need to be taken, to prove this.
I'm fine with proper testing and good numbers here but until then, I can only consider it a prototype
Quoted 11 lines
There are only two places a sane implementation should show issues.
- When the access to a pointer goes through a pointer to find that global variable.
- When doing a lookup in a hash table we need to look at an additional field to verify a hash match. Because having a completely separate hash table is likely too expensive.If that can be shown to really slow down packets on the hot path I am willing to consider other possibilities. Until then I think we are on path to the simplest and most powerful version of building a network namespace usable by containers.
keep in mind that you actually have three kinds of network traffic on a typical host/guest system:
- traffic between unit and outside
- host traffic should be quite minimal
- guest traffic will be quite high
- traffic between host and guest
probably minimal too (only for shared services)
- traffic between guests
can be as high (or even higher) than the outbound traffic, just think web guest and database guest
Quoted 3 lines
The routing between network namespaces does have the potential to be more expensive than just a packet trivially coming off the wire into a socket.
IMHO the routing between network namespaces should not require more than the current local traffic does (i.e. you should be able to achieve loopback speed within an insignificant tolerance) and not nearly the time required for on-wire stuff ...
Quoted 8 lines
However that is fundamentally from a lack of hardware. If the rest works smarter filters in the drivers should enable to remove the cost.
Basically it is just a matter of:
if (dest_mac==my_mac1) it is for device 1.
If (dest_mac==my_mac2) it is for device 2.
etc.
hmm, so you plan on providing a different MAC for each guest? how should that be handled from the user PoV? you cannot simply make up MACs as you go, and, depending on the network card, operation in promisc mode might be slower than for a given set (maybe only one) MAC, no?
Quoted 4 lines
At a small count of macs it is trivial to understand it will go fast for a larger count of macs it only works with a good data structure. We don't hit any extra cache lines of the packet, and the above test can be collapsed with other routing lookup tests.
well, I'm absolutely not against flexibility or full virtualization, but the proposed 'routing' on the host effectively doubles the time the packet spends in the network stack(s), so I can not believe that this approach would not add (significant) overhead to the hot path ...
best,
Herbert
Quoted 1 line
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 7 lines · Herbert Poetzl
Quoted 2 lines · Eric W. Biederman
Herbert Poetzl <[email protected]> writes:
yes, but you will not be able to apply policy on the parent, restricting the child networking in a proper way without jumping through hoops ...
? I don't understand where you are coming from. There is no restriction on where you can apply policy.
Quoted 6 lines
Quoted 3 lines
I really do not believe we have a hotpath issue, if this is implemented properly. Benchmarks of course need to be taken, to prove this.
I'm fine with proper testing and good numbers here but until then, I can only consider it a prototype
We are taking the first steps to get this all sorted out. I think what we have is more than a prototype but less then the final implementation. Call it the very first draft version.
Quoted 19 lines
Quoted 11 lines
There are only two places a sane implementation should show issues.
- When the access to a pointer goes through a pointer to find that global variable.
- When doing a lookup in a hash table we need to look at an additional field to verify a hash match. Because having a completely separate hash table is likely too expensive.If that can be shown to really slow down packets on the hot path I am willing to consider other possibilities. Until then I think we are on path to the simplest and most powerful version of building a network namespace usable by containers.
keep in mind that you actually have three kinds of network traffic on a typical host/guest system:
- traffic between unit and outside
- host traffic should be quite minimal
- guest traffic will be quite high
Quoted 7 lines
- traffic between host and guest
probably minimal too (only for shared services)- traffic between guests
can be as high (or even higher) than the outbound traffic, just think web guest and database guest
Interesting.
Quoted 9 lines
Quoted 3 lines
The routing between network namespaces does have the potential to be more expensive than just a packet trivially coming off the wire into a socket.
IMHO the routing between network namespaces should not require more than the current local traffic does (i.e. you should be able to achieve loopback speed within an insignificant tolerance) and not nearly the time required for on-wire stuff ...
That assumes on the wire stuff is noticeably slower. You can achieve over 1GB/s on some networks.
But I agree that the cost should resemble the current loopback device. I have seen nothing that suggests it is not.
Quoted 9 lines
Quoted 8 lines
However that is fundamentally from a lack of hardware. If the rest works smarter filters in the drivers should enable to remove the cost.
Basically it is just a matter of:
if (dest_mac==my_mac1) it is for device 1.
If (dest_mac==my_mac2) it is for device 2.
etc.
Quoted 6 lines
hmm, so you plan on providing a different MAC for each guest? how should that be handled from the user PoV? you cannot simply make up MACs as you go, and, depending on the network card, operation in promisc mode might be slower than for a given set (maybe only one) MAC, no?
The speed is a factor certainly. As for making up macs. There is a local assignment bit that you can set. With that set it is just a matter of using a decent random number generator. The kernel already does this is some places.
Quoted 11 lines
Quoted 4 lines
At a small count of macs it is trivial to understand it will go fast for a larger count of macs it only works with a good data structure. We don't hit any extra cache lines of the packet, and the above test can be collapsed with other routing lookup tests.
well, I'm absolutely not against flexibility or full virtualization, but the proposed 'routing' on the host effectively doubles the time the packet spends in the network stack(s), so I can not believe that this approach would not add (significant) overhead to the hot path ...
It might, but I am pretty certain it won't double
the cost, as you don't do 2 full network stack traversals. And even at a full doubling I doubt it will affect bandwith or latency very much. If it does we have a lot more to optimize in the network stack than just this code.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Herbert Poetzl
Quoted 1 line · Eric W. Biederman
Herbert Poetzl <[email protected]> writes:
yes, but you will not be able to apply policy on the parent, restricting the child networking in a proper way without jumping through hoops ...
? I don't understand where you are coming from. There is no restriction on where you can apply policy.
in a fully hierarchical system (not that I really think this is required here) you would be able to 'delegate' certain addresses (ranges) to a namespace (the child) below the current one (the parent) with the ability to limit/control the input/output (which is required for security)
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
I really do not believe we have a hotpath issue, if this is implemented properly. Benchmarks of course need to be taken, to prove this.
I'm fine with proper testing and good numbers here but until then, I can only consider it a prototype
We are taking the first steps to get this all sorted out. I think what we have is more than a prototype but less then the final implementation. Call it the very first draft version.
what we are desperately missing here is a proper way to testing this, maybe the network folks can come up with some test equipment/ideas here ...
Quoted 33 lines
Quoted 19 lines
Quoted 11 lines
There are only two places a sane implementation should show issues.
- When the access to a pointer goes through a pointer to find that global variable.
- When doing a lookup in a hash table we need to look at an additional field to verify a hash match. Because having a completely separate hash table is likely too expensive.If that can be shown to really slow down packets on the hot path I am willing to consider other possibilities. Until then I think we are on path to the simplest and most powerful version of building a network namespace usable by containers.
keep in mind that you actually have three kinds of network traffic on a typical host/guest system:
- traffic between unit and outside
- host traffic should be quite minimal
- guest traffic will be quite highQuoted 7 lines
- traffic between host and guest
probably minimal too (only for shared services)- traffic between guests
can be as high (or even higher) than the outbound traffic, just think web guest and database guestInteresting.
Quoted 4 lines
Quoted 3 lines
The routing between network namespaces does have the potential to be more expensive than just a packet trivially coming off the wire into a socket.
Quoted 8 lines
Quoted 5 lines
IMHO the routing between network namespaces should not require more than the current local traffic does (i.e. you should be able to achieve loopback speed within an insignificant tolerance) and not nearly the time required for on-wire stuff ...
That assumes on the wire stuff is noticeably slower. You can achieve over 1GB/s on some networks.
well, have you ever tried how much you can achieve over loopback :)
Quoted 20 lines
But I agree that the cost should resemble the current loopback device. I have seen nothing that suggests it is not.
Quoted 9 lines
Quoted 8 lines
However that is fundamentally from a lack of hardware. If the rest works smarter filters in the drivers should enable to remove the cost.
Basically it is just a matter of:
if (dest_mac==my_mac1) it is for device 1.
If (dest_mac==my_mac2) it is for device 2.
etc.Quoted 6 lines
hmm, so you plan on providing a different MAC for each guest? how should that be handled from the user PoV? you cannot simply make up MACs as you go, and, depending on the network card, operation in promisc mode might be slower than for a given set (maybe only one) MAC, no?
Quoted 2 lines
The speed is a factor certainly. As for making up macs. There is a local assignment bit that you can set.
well, local is fine, but you cannot utilize that on-wire which basically means that you would have either to 'map' the MAC on transmission (to the real one) which would basically make the approach useless, or to use addresses which are fine within a certain range of routers ...
Quoted 2 lines
With that set it is just a matter of using a decent random number generator. The kernel already does this is some places.
sure you can make up MACs, but you will never be able to use them 'outside' the box
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
At a small count of macs it is trivial to understand it will go fast for a larger count of macs it only works with a good data structure. We don't hit any extra cache lines of the packet, and the above test can be collapsed with other routing lookup tests.
well, I'm absolutely not against flexibility or full virtualization, but the proposed 'routing' on the host effectively doubles the time the packet spends in the network stack(s), so I can not believe that this approach would not add (significant) overhead to the hot path ...
Quoted 5 lines
It might, but I am pretty certain it won't double the cost, as you don't do 2 full network stack traversals.
And even at a full doubling I doubt it will affect bandwith or latency very much.
well, for loopback that would mean half the bandwidth and twice the latency, no?
Quoted 2 lines
If it does we have a lot more to optimize in the network stack than just this code.
why? duplicate stack traversal takes roughly twice the time, or am I wrong here? if so, please enlighten me ...
best,
Herbert
Quoted 1 line
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 117 lines · Herbert Poetzl
Quoted 5 lines · Eric W. Biederman
Quoted 2 lines · Herbert Poetzl
On Mon, Jun 26, 2006 at 01:35:15PM -0600, Eric W. Biederman wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Herbert Poetzl <[email protected]> writes:
yes, but you will not be able to apply policy on the parent, restricting the child networking in a proper way without jumping through hoops ...
? I don't understand where you are coming from. There is no restriction on where you can apply policy.
in a fully hierarchical system (not that I really think this is required here) you would be able to 'delegate' certain addresses (ranges) to a namespace (the child) below the current one (the parent) with the ability to limit/control the input/output (which is required for security)
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
I really do not believe we have a hotpath issue, if this is implemented properly. Benchmarks of course need to be taken, to prove this.
I'm fine with proper testing and good numbers here but until then, I can only consider it a prototype
We are taking the first steps to get this all sorted out. I think what we have is more than a prototype but less then the final implementation. Call it the very first draft version.
what we are desperately missing here is a proper way to testing this, maybe the network folks can come up with some test equipment/ideas here ...
Quoted 34 lines
Quoted 19 lines
Quoted 11 lines
There are only two places a sane implementation should show issues.
- When the access to a pointer goes through a pointer to find that global variable.
- When doing a lookup in a hash table we need to look at an additional field to verify a hash match. Because having a completely separate hash table is likely too expensive.If that can be shown to really slow down packets on the hot path I am willing to consider other possibilities. Until then I think we are on path to the simplest and most powerful version of building a network namespace usable by containers.
keep in mind that you actually have three kinds of network traffic on a typical host/guest system:
- traffic between unit and outside
- host traffic should be quite minimal
- guest traffic will be quite highQuoted 7 lines
- traffic between host and guest
probably minimal too (only for shared services)- traffic between guests
can be as high (or even higher) than the outbound traffic, just think web guest and database guestInteresting.
Quoted 4 lines
Quoted 3 lines
The routing between network namespaces does have the potential to be more expensive than just a packet trivially coming off the wire into a socket.
Quoted 8 lines
Quoted 5 lines
IMHO the routing between network namespaces should not require more than the current local traffic does (i.e. you should be able to achieve loopback speed within an insignificant tolerance) and not nearly the time required for on-wire stuff ...
That assumes on the wire stuff is noticeably slower. You can achieve over 1GB/s on some networks.
well, have you ever tried how much you can achieve over loopback :)
Quoted 20 lines
But I agree that the cost should resemble the current loopback device. I have seen nothing that suggests it is not.
Quoted 15 lines
Quoted 8 lines
However that is fundamentally from a lack of hardware. If the rest works smarter filters in the drivers should enable to remove the cost.
Basically it is just a matter of:
if (dest_mac==my_mac1) it is for device 1.
If (dest_mac==my_mac2) it is for device 2.
etc.hmm, so you plan on providing a different MAC for each guest? how should that be handled from the user PoV? you cannot simply make up MACs as you go, and, depending on the network card, operation in promisc mode might be slower than for a given set (maybe only one) MAC, no?
well, local is fine, but you cannot utilize that on-wire which basically means that you would have either to 'map' the MAC on transmission (to the
real one) which would basically make the approach
useless, or to use addresses which are fine within
a certain range of routers ...
Quoted 6 lines
Quoted 2 lines
With that set it is just a matter of using a decent random number generator. The kernel already does this is some places.
sure you can make up MACs, but you will never be able to use them 'outside' the box
I do it all the time with my mac-vlan patch and it works fine..so long as you pay minimal attention. Just let the user specify the MAC addr and they can manage the uniqueness as they wish....
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Herbert PoetzlQuoted 13 lines · Herbert Poetzl
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Herbert Poetzl
Quoted 1 line · Eric W. Biederman
Herbert Poetzl <[email protected]> writes:
yes, but you will not be able to apply policy on the parent, restricting the child networking in a proper way without jumping through hoops ...
? I don't understand where you are coming from. There is no restriction on where you can apply policy.
Quoted 6 lines
in a fully hierarchical system (not that I really think this is required here) you would be able to 'delegate' certain addresses (ranges) to a namespace (the child) below the current one (the parent) with the ability to limit/control the input/output (which is required for security)
All of that is possible with the current design. It is merely a matter of using the features the kernel currently has.
The trick is know that a child namespace only has a loopback by default, and has to have a network interface added from the parent to be able to talk to anything.
Quoted 16 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
I really do not believe we have a hotpath issue, if this is implemented properly. Benchmarks of course need to be taken, to prove this.
I'm fine with proper testing and good numbers here but until then, I can only consider it a prototype
We are taking the first steps to get this all sorted out. I think what we have is more than a prototype but less then the final implementation. Call it the very first draft version.
what we are desperately missing here is a proper way to testing this, maybe the network folks can come up with some test equipment/ideas here ...
Quoted 5 lines
Quoted 2 lines
That assumes on the wire stuff is noticeably slower. You can achieve over 1GB/s on some networks.
well, have you ever tried how much you can achieve over loopback :)
Not recently.
Quoted 6 lines
well, local is fine, but you cannot utilize that on-wire which basically means that you would have either to 'map' the MAC on transmission (to the real one) which would basically make the approach useless, or to use addresses which are fine within a certain range of routers ...
I believe on the wire is fine as well. Certainly I had no problems when I tested it. I do agree that it increases the chance for a mac address collision so should be handled carefully. But we are talking a number with almost as many random bits as a UUID.
As I recall the rule from the birthday paradox is something like: if you have N possibilities you get a 50% chance of collision when you have sqrt(N) items present. sqrt(2**40) = 2**20 ~= 1 Million.
So as long as you have a good random generator the odds of a collision are quite small until you have used a million local mac addresses.
Now while I can see some small chance of that happening on a very crowded local area network, using lots of logical servers the kernel arp cache and switch mac cache limits would start causing real problems long before you got that far. So you would need to start routing at which point the practical problem goes away.
Quoted 3 lines
well, for loopback that would mean half the bandwidth and twice the latency, no?
Not at all. The usual bottle neck is copying the data. The data only gets put in a skb once and then pointers to the skb are passed around. So you get copied in once and copied out once. We are certainly not going to add an extra copy to it.
For practical purposes the fast path through the network stack is a series of hash table looks.
That added to the fact that we don't make a full trip through the network stack on both sides (unless someone is running tcp dump) for example.
Quoted 6 lines
Quoted 2 lines
If it does we have a lot more to optimize in the network stack than just this code.
why? duplicate stack traversal takes roughly twice the time, or am I wrong here? if so, please enlighten me ...
The network stack is how we decide what goes where. Sending and receiving packets should always have hardware as the bottleneck, not software. So software should be a tiny percentage of the time it takes to send or receive any packet.
With packets limited to 1.5K and below things aren't always that clear cut but the ideal remains. But basically anything we can do besides remove the copy in and the copy out of the network stack we should do.
The copy in and the copy out are fundamentally hard to remove without modifying page tables which because of tlb invalidates can be even more expensive than a copy.
The best you can hope for on a loopback scenario is a single copy from one user space buffer to another skipping the intermediate kernel buffer. But that is tricky for an entirely different set of reasons.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Herbert PoetzlHerbert,
Quoted 8 lines · Herbert Poetzl
keep in mind that you actually have three kinds of network traffic on a typical host/guest system:
- traffic between unit and outside
- host traffic should be quite minimal
- guest traffic will be quite high
Quoted 7 lines
- traffic between host and guest
probably minimal too (only for shared services)- traffic between guests
can be as high (or even higher) than the outbound traffic, just think web guest and database guest
My experience with host-guest systems tells me the opposite:
outside traffic is a way higher than traffic between guests. People put web server and database in different guests not more frequent than they put them on separate physical server.
Unless people are building a really huge system when 1 server can't take the whole load, web and database live together and benefit from communications over UNIX sockets.
Guests are usually comprised of web-db pairs, and people place many such guests on a single computer.
Quoted 10 lines
Quoted 4 lines
The routing between network namespaces does have the potential to be more expensive than just a packet trivially coming off the wire into a socket.
IMHO the routing between network namespaces should not require more than the current local traffic does (i.e. you should be able to achieve loopback speed within an insignificant tolerance) and not nearly the time required for on-wire stuff ...
I'd like to caution about over-optimizing communications between different network namespaces.
Many optimizations of local traffic (such as high MTU) don't look so appealing when you start to think about live migration of namespaces.
Regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 19 lines · Andrey Savochkin
Herbert,
Quoted 8 lines · Herbert Poetzl
keep in mind that you actually have three kinds of network traffic on a typical host/guest system:
- traffic between unit and outside
- host traffic should be quite minimal
- guest traffic will be quite highQuoted 7 lines
- traffic between host and guest
probably minimal too (only for shared services)- traffic between guests
can be as high (or even higher) than the outbound traffic, just think web guest and database guest
Quoted 7 lines
My experience with host-guest systems tells me the opposite: outside traffic is a way higher than traffic between guests. People put web server and database in different guests not more frequent than they put them on separate physical server. Unless people are building a really huge system when 1 server can't take the whole load, web and database live together and benefit from communications over UNIX sockets.
well, that's probably because you (or your company) focuses on providers which simply (re)sell the entities to their customers, in which case it would be more expensive to put e.g. the database into a separate guest. but let me state here that this is not the only application for this technology
many folks use Linux-VServer for separating services (e.g. mail, web, database, ...) and here a _lot_ of traffic happens between guests (as it would on a normal linux system or within a single guest in your case)
Quoted 2 lines
Guests are usually comprised of web-db pairs, and people place many such guests on a single computer.
in case two guests cost more than one, yes, in case two guests allow for better isolation and easier maintainance without additional cost, no :)
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
The routing between network namespaces does have the potential to be more expensive than just a packet trivially coming off the wire into a socket.
IMHO the routing between network namespaces should not require more than the current local traffic does (i.e. you should be able to achieve loopback speed within an insignificant tolerance) and not nearly the time required for on-wire stuff ...
Quoted 4 lines
I'd like to caution about over-optimizing communications between different network namespaces. Many optimizations of local traffic (such as high MTU) don't look so appealing when you start to think about live migration of namespaces.
I think the 'optimization' (or to be precise: desire not to sacrifice local/loopback traffic for some use case as you describe it) does not interfere with live migration at all, we still will have 'local' and 'remote' traffic, and personally I doubt that the live migration is a feature for the masses ...
best,
Herbert
Quoted 2 lines
Regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Herbert,
Quoted 17 lines · Herbert Poetzl
Quoted 8 lines · Andrey Savochkin
Quoted 5 lines · Herbert Poetzl
- traffic between guests
can be as high (or even higher) than the
outbound traffic, just think web guest and
database guestQuoted 7 lines
My experience with host-guest systems tells me the opposite: outside traffic is a way higher than traffic between guests. People put web server and database in different guests not more frequent than they put them on separate physical server. Unless people are building a really huge system when 1 server can't take the whole load, web and database live together and benefit from communications over UNIX sockets.
Quoted 6 lines
well, that's probably because you (or your company) focuses on providers which simply (re)sell the entities to their customers, in which case it would be more expensive to put e.g. the database into a separate guest. but let me state here that this is not the only application for this technology
I'm just sharing my experience.
You have one experience, I have another, and your classification of traffic importance is not the universal one.
My point was that we shouldn't overestimate the use of INET sockets vs. UNIX ones in configurations where communications but not web/db operations play a big role in overall performance.
And indeed I've talked with many different people, from universities to large enterprises.
Quoted 1 line
[snip]
Quoted 11 lines
Quoted 4 lines
I'd like to caution about over-optimizing communications between different network namespaces. Many optimizations of local traffic (such as high MTU) don't look so appealing when you start to think about live migration of namespaces.
I think the 'optimization' (or to be precise: desire not to sacrifice local/loopback traffic for some use case as you describe it) does not interfere with live migration at all, we still will have 'local' and 'remote' traffic, and personally I doubt that the live migration is a feature for the masses ...
Why not for the masses?
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Herbert PoetzlQuoted 7 lines · Herbert Poetzl
Quoted 5 lines · Andrey Savochkin
I'd like to caution about over-optimizing communications between different network namespaces. Many optimizations of local traffic (such as high MTU) don't look so appealing when you start to think about live migration of namespaces.
Quoted 6 lines
I think the 'optimization' (or to be precise: desire not to sacrifice local/loopback traffic for some use case as you describe it) does not interfere with live migration at all, we still will have 'local' and 'remote' traffic, and personally I doubt that the live migration is a feature for the masses ...
Several things.
- The linux loopback device is not strongly optimized, it is a compatibility layer.
- Traffic between guests is an implementation detail.
There is nothing fundamental in our semantics that says the traffic has to be slow for any workload (except for the limuts imposed by using actual on the wire protocols). The lo shares the same problem.
Worry about this case now when it has clearly been shown that there are several possible ways to optimize this and get back any lost local performance is optimizing way too early.
Criticize the per namespace performance and all you want. That is pretty much a merge blocker. Unless we do worse than a 1-5% penalty the communication across namespaces is really a non-issue.
Even with your large communications flows between guests 1-5% is nothing.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Eric W. BiedermanQuoted 9 lines · Eric W. Biederman
Basically it is just a matter of:
if (dest_mac==my_mac1) it is for device 1.
If (dest_mac==my_mac2) it is for device 2.
etc.At a small count of macs it is trivial to understand it will go fast for a larger count of macs it only works with a good data structure. We don't hit any extra cache lines of the packet, and the above test can be collapsed with other routing lookup tests.
I think you should do this at the layer-2 level, well before you get to routing. That will make the virtual mac-vlan work with arbitrary protocols and appear very much like a regular ethernet interface. This approach worked well with .1q vlans, and with my version of the mac-vlan module.
Using the mac-vlan and source-based routing tables, I can give a unique 'interface' to each process and have each process able to bind to the same IP port, for instance. Using source-based routing (by binding to a local IP explicitly and adding a route table for that source IP), I can give unique default routes to each interface as well. Since we cannot have more than 256 routing tables, this approach is currently limitted to around 250 virtual interfaces, but that is still a substantial amount.
My mac-vlan patch, redirect-device patch, and other hackings are consolidated in this patch:
http://www.candelatech.com/oss/candela_2.6.16.patch
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 12 lines · Ben Greear
Quoted 9 lines · Eric W. Biederman
Basically it is just a matter of:
if (dest_mac==my_mac1) it is for device 1.
If (dest_mac==my_mac2) it is for device 2.
etc.At a small count of macs it is trivial to understand it will go fast for a larger count of macs it only works with a good data structure. We don't hit any extra cache lines of the packet, and the above test can be collapsed with other routing lookup tests.
Quoted 5 lines
I think you should do this at the layer-2 level, well before you get to routing. That will make the virtual mac-vlan work with arbitrary protocols and appear very much like a regular ethernet interface. This approach worked well with .1q vlans, and with my version of the mac-vlan module.
yes, that sounds good to me, any numbers how that affects networking in general (performance wise and memory wise, i.e. caches and hashes) ...
Quoted 8 lines
Using the mac-vlan and source-based routing tables, I can give a unique 'interface' to each process and have each process able to bind to the same IP port, for instance. Using source-based routing (by binding to a local IP explicitly and adding a route table for that source IP), I can give unique default routes to each interface as well. Since we cannot have more than 256 routing tables, this approach is currently limitted to around 250 virtual interfaces, but that is still a substantial amount.
an typically that would be sufficient IMHO, but of course, a more 'general' hash tag would be better in the long run ...
Quoted 4 lines
My mac-vlan patch, redirect-device patch, and other hackings are consolidated in this patch:
great! thanks!
best,
Herbert
Quoted 6 lines
Thanks,
BenSignature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 5 lines · Herbert Poetzl
On Mon, Jun 26, 2006 at 03:13:17PM -0700, Ben Greear wrote:
yes, that sounds good to me, any numbers how that affects networking in general (performance wise and memory wise, i.e. caches and hashes) ...
I'll run some tests later today. Based on my previous tests, I don't remember any significant overhead.
Quoted 10 lines
Quoted 8 lines
Using the mac-vlan and source-based routing tables, I can give a unique 'interface' to each process and have each process able to bind to the same IP port, for instance. Using source-based routing (by binding to a local IP explicitly and adding a route table for that source IP), I can give unique default routes to each interface as well. Since we cannot have more than 256 routing tables, this approach is currently limitted to around 250 virtual interfaces, but that is still a substantial amount.
Quoted 3 lines
an typically that would be sufficient IMHO, but of course, a more 'general' hash tag would be better in the long run ...
I'm willing to offer a bounty (hardware, beer, money, ...) if someone will 'fix' this so we can have 1000 or more routes....
Being able to select these routes at a more global level (without having to specifically bind to a local IP would be nice as well.)
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 5 lines · Ben Greear
Quoted 1 line · Herbert Poetzl
On Mon, Jun 26, 2006 at 03:13:17PM -0700, Ben Greear wrote:
Quoted 7 lines
Quoted 3 lines
yes, that sounds good to me, any numbers how that affects networking in general (performance wise and memory wise, i.e. caches and hashes) ...
I'll run some tests later today. Based on my previous tests, I don't remember any significant overhead.
Here's a quick benchmark using my redirect devices (RDD). Each RDD comes in a pair...when you tx on one, the pkt is rx'd on the peer. The idea is that it is exactly like two physical ethernet interfaces connected by a cross-over cable.
My test system is a 64-bit dual-core Intel system, 3.013 Ghz processor with 1GB RAM. Fairly standard stuff..it's one of the Shuttle XPC systems.
Kernel is 2.6.16.16 (64-bit).
Test setup is: rdd1 -- rdd2 [bridge] rdd3 -- rdd4
I am using my proprietary module for the bridge logic...and the default bridge should be at least this fast. I am injecting 1514 byte packets on rdd1 and rdd4 with pktgen (bi-directional flow). My pktgen is also receiving the pkts and gathering stats.
This setup sustains 1.7Gbps of generated and received traffic between rdd1 and rdd4.
Running only the [bridge] between two 10/100/1000 ports on an Intel PCI-E NIC will sustain about 870Mbps (bi-directional) on this system, so the virtual devices are quite efficient, as suspected.
I have not yet had time to benchmark the mac-vlans...hopefully later today.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 32 lines · Ben Greear
Quoted 10 lines · Ben Greear
Quoted 5 lines · Herbert Poetzl
On Mon, Jun 26, 2006 at 03:13:17PM -0700, Ben Greear wrote:
yes, that sounds good to me, any numbers how that affects networking in general (performance wise and memory wise, i.e. caches and hashes) ...
I'll run some tests later today. Based on my previous tests, I don't remember any significant overhead.
Here's a quick benchmark using my redirect devices (RDD). Each RDD comes in a pair...when you tx on one, the pkt is rx'd on the peer. The idea is that it is exactly like two physical ethernet interfaces connected by a cross-over cable.
My test system is a 64-bit dual-core Intel system, 3.013 Ghz processor with 1GB RAM. Fairly standard stuff..it's one of the Shuttle XPC systems. Kernel is 2.6.16.16 (64-bit).
Test setup is: rdd1 -- rdd2 [bridge] rdd3 -- rdd4
I am using my proprietary module for the bridge logic...and the default bridge should be at least this fast. I am injecting 1514 byte packets on rdd1 and rdd4 with pktgen (bi-directional flow). My pktgen is also receiving the pkts and gathering stats.
This setup sustains 1.7Gbps of generated and received traffic between rdd1 and rdd4.
Quoted 6 lines
Running only the [bridge] between two 10/100/1000 ports on an Intel PCI-E NIC will sustain about 870Mbps (bi-directional) on this system, so the virtual devices are quite efficient, as suspected.
I have not yet had time to benchmark the mac-vlans...hopefully later today.
hmm, maybe you could also benchmark loopback connections (and their throughput) on your system?
my (not so fancy) PIII, 32bit, 2.6.17.1 seems to do roughly 2Gbs on the loopback device (tested with dd and netcat)
best,
Herbert
Quoted 6 lines
Thanks,
BenSignature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Daniel LezcanoDaniel,
Quoted 8 lines · Daniel Lezcano
Quoted 5 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
I mean instead of having the route tables private to the namespace, the routes have the information to which namespace they are associated.
I think I understand what you're talking about: you want to make routing responsible for determining destination namespace ID in addition to route type (local, unicast etc), nexthop information, and so on. Right?
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 9 lines · Andrey Savochkin
Daniel,
Quoted 5 lines · Daniel Lezcano
Quoted 4 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
Quoted 6 lines
Quoted 1 line
I mean instead of having the route tables private to the namespace, the routes have the information to which namespace they are associated.
I think I understand what you're talking about: you want to make routing responsible for determining destination namespace ID in addition to route type (local, unicast etc), nexthop information, and so on. Right?
Yes.
Quoted 4 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 18 lines · Daniel Lezcano
Quoted 9 lines · Andrey Savochkin
Daniel,
Quoted 5 lines · Daniel Lezcano
Quoted 4 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
Quoted 7 lines
Quoted 2 lines
I mean instead of having the route tables private to the namespace, the routes have the information to which namespace they are associated.
I think I understand what you're talking about: you want to make routing responsible for determining destination namespace ID in addition to route type (local, unicast etc), nexthop information, and so on. Right?
Quoted 8 lines
Yes.
Quoted 4 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
Routing is everything.
For example, I want namespaces to have their private tunnel devices. It means that namespaces should be allowed have private routes of local type, private default routes, and so on...
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Quoted 8 lines
Quoted 1 line
Right. What is the advantage of having separate the routing tables ?
Routing is everything.
For example, I want namespaces to have their private tunnel devices. It means that namespaces should be allowed have private routes of local type, private default routes, and so on...
Ok, we are talking about the same things. We do it only in a different way:
* separate routing table : namespace | \--- route_tables | \---routes * tagged routing table : route_tables | \---routes | \---namespace
When using routes private to the namespace, globally the logic of the ip stack is not changed, it manipulates only differents variables. It is more clean than tagging the route for the reasons mentioned by Eric.
When using route tagging, the logic is changed because when doing lookup on the routes table which is global, the namespace is used to match the route and make it visible.
I use the second method, because I think it is more effecient and reduce the overhead. But the isolation is minimalist and only aims to avoid the application using ressources outside of the container (aka namespace) without taking care of the system. For example, I didn't take care of network devices, because as far as see I can't imagine an administrator wanting to change the network device name while there are hundred of containers running. Concerning tunnel devices for example, they should be created inside the container.
I think, private network ressources method is more elegant and involves more network ressources, but there is probably a significant overhead and some difficulties to have __lightweight__ container (aka application container), make nfs working well, etc... I did some tests with tbench and the loopback with the private namespace and there is roughly an overhead of 4 % without the isolation since with the tagging method there is 1 % with the isolation.
The network namespace aims the isolation for now, but the container based on the namespaces will probably need checkpoint/restart and migration ability. The migration is needed not only for servers but for HPC jobs too.
So I don't know what level of isolation/virtualization is really needed by users, what should be acceptable (strong isolation and overhead / weak isolation and efficiency). I don't know if people wanting strong isolation will not prefer Xen (cleary with much more overhead than your patches ;) )
Regards
-- Daniel
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 13 lines · Daniel Lezcano
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
Routing is everything.
For example, I want namespaces to have their private tunnel devices. It means that namespaces should be allowed have private routes of local type, private default routes, and so on...Ok, we are talking about the same things. We do it only in a different way:
Quoted 13 lines
* separate routing table : namespace | \--- route_tables | \---routes * tagged routing table : route_tables | \---routes | \---namespace
There is a third possibility, that falls in between these two if local communication is really the bottle neck.
We have the dst cache for caching routes and cache multiple transformations that happen on a packet.
With a little extra knowledge it is possible to have the separate routing tables but have special logic that recognizes the local tunnel device that connects namespaces and have it look into the next namespaces routes, and build up a complete stack of dst entries of where the packet needs to go.
I keep forgetting about that possibility. But as long as everything is done at the routing layer that should work.
Quoted 7 lines
I use the second method, because I think it is more effecient and reduce the overhead. But the isolation is minimalist and only aims to avoid the application using ressources outside of the container (aka namespace) without taking care of the system. For example, I didn't take care of network devices, because as far as see I can't imagine an administrator wanting to change the network device name while there are hundred of containers running. Concerning tunnel devices for example, they should be created inside the container.
Inside the containers I want all network devices named eth0!
Quoted 6 lines
I think, private network ressources method is more elegant and involves more network ressources, but there is probably a significant overhead and some difficulties to have
__lightweight__container (aka application container), make nfs working well, etc... I did some tests with tbench and the loopback with the private namespace and there is roughly an overhead of 4 % without the isolation since with the tagging method there is 1 % with the isolation.
The overhead went down?
Quoted 3 lines
The network namespace aims the isolation for now, but the container based on the namespaces will probably need checkpoint/restart and migration ability. The migration is needed not only for servers but for HPC jobs too.
Yes.
Quoted 4 lines
So I don't know what level of isolation/virtualization is really needed by users, what should be acceptable (strong isolation and overhead / weak isolation and efficiency). I don't know if people wanting strong isolation will not prefer Xen (cleary with much more overhead than your patches ;) )
We need a clean abstraction that optimizes well.
However local communication between containers is not what we should benchmark. That can always be improved later. So long as the performance is reasonable. What needs to be benchmarked is the overhead of namespaces when connected to physical networking devices and on their own local loopback, and comparing that to a kernel without namespace support.
If we don't hurt that core case we have an implementation we can merge. There are a lot of optimization opportunities for local communications and we can do that after we have a correct and accepted implementation. Anything else is optimizing too soon, and will just be muddying the waters.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 53 lines · Eric W. Biederman
Quoted 13 lines · Daniel Lezcano
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
Routing is everything. For example, I want namespaces to have their private tunnel devices. It means that namespaces should be allowed have private routes of local type, private default routes, and so on...
Ok, we are talking about the same things. We do it only in a different way:
Quoted 13 lines
* separate routing table : namespace | \--- route_tables | \---routes * tagged routing table : route_tables | \---routes | \---namespaceThere is a third possibility, that falls in between these two if local communication is really the bottle neck.
We have the dst cache for caching routes and cache multiple transformations that happen on a packet.
With a little extra knowledge it is possible to have the separate routing tables but have special logic that recognizes the local tunnel device that connects namespaces and have it look into the next namespaces routes, and build up a complete stack of dst entries of where the packet needs to go.
I keep forgetting about that possibility. But as long as everything is done at the routing layer that should work.
Quoted 8 lines
I use the second method, because I think it is more effecient and reduce the overhead. But the isolation is minimalist and only aims to avoid the application using ressources outside of the container (aka namespace) without taking care of the system. For example, I didn't take care of network devices, because as far as see I can't imagine an administrator wanting to change the network device name while there are hundred of containers running. Concerning tunnel devices for example, they should be created inside the container.
Quoted 1 line
Inside the containers I want all network devices named eth0!
huh? even if there are two of them? also tun?
I think you meant, you want to be able to have eth0 in _more_ than one guest where eth0 in a guest can also be/use/relate to eth1 on the host, right?
Quoted 10 lines
Quoted 8 lines
I think, private network ressources method is more elegant and involves more network ressources, but there is probably a significant overhead and some difficulties to have
__lightweight__container (aka application container), make nfs working well, etc... I did some tests with tbench and the loopback with the private namespace and there is roughly an overhead of 4 % without the isolation since with the tagging method there is 1 % with the isolation.The overhead went down?
yes, this might actually happen, because the guest has only to look at a certain subset of entries but this needs a lot more testing, especially with a lot of guests
Quoted 5 lines
Quoted 4 lines
The network namespace aims the isolation for now, but the container based on the namespaces will probably need checkpoint/restart and migration ability. The migration is needed not only for servers but for HPC jobs too.
Quoted 7 lines
Yes.
Quoted 5 lines
So I don't know what level of isolation/virtualization is really needed by users, what should be acceptable (strong isolation and overhead / weak isolation and efficiency). I don't know if people wanting strong isolation will not prefer Xen (cleary with much more overhead than your patches ;) )
well, Xen claims something below 2% IIRC, and would be clearly the better choice if you want strict separation with the complete functionality, especially with hardware support
Quoted 8 lines
We need a clean abstraction that optimizes well.
However local communication between containers is not what we should benchmark. That can always be improved later. So long as the performance is reasonable. What needs to be benchmarked is the overhead of namespaces when connected to physical networking devices and on their own local loopback, and comparing that to a kernel without namespace support.
well, for me (obviously advocating the lightweight case) it seems improtant that the following conditions are met:
- loopback traffic inside a guest is insignificantly slower than on a normal system
- loopback traffic on the host is insignificantly slower than on a normal system
- inter guest traffic is faster than on-wire traffic, and should be withing a small tolerance of the loopback case (as it really isn't different)
- network (on-wire) traffic should be as fast as without the namespace (i.e. within 1% or so, better not really measurable)
- all this should be true in a setup with a significant number of guests, when only one guest is active, but all other guests are ready/configured
- all this should scale well with a few hundred guests
Quoted 5 lines
If we don't hurt that core case we have an implementation we can merge. There are a lot of optimization opportunities for local communications and we can do that after we have a correct and accepted implementation. Anything else is optimizing too soon, and will just be muddying the waters.
what I fear is that once something is in, the kernel will just become slower (as it already did in some areas) and nobody will care/be-able to fix that later on ...
best,
Herbert
Quoted 1 line
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 9 lines · Herbert Poetzl
Quoted 2 lines · Eric W. Biederman
Inside the containers I want all network devices named eth0!
huh? even if there are two of them? also tun?
I think you meant, you want to be able to have eth0 in _more_ than one guest where eth0 in a guest can also be/use/relate to eth1 on the host, right?
Right I want to have an eth0 in each guest where eth0 is it's own network device and need have no relationship to eth0 on the host.
Quoted 26 lines
Quoted 8 lines
We need a clean abstraction that optimizes well.
However local communication between containers is not what we should benchmark. That can always be improved later. So long as the performance is reasonable. What needs to be benchmarked is the overhead of namespaces when connected to physical networking devices and on their own local loopback, and comparing that to a kernel without namespace support.
well, for me (obviously advocating the lightweight case) it seems improtant that the following conditions are met:
- loopback traffic inside a guest is insignificantly slower than on a normal system
- loopback traffic on the host is insignificantly slower than on a normal system
- inter guest traffic is faster than on-wire traffic, and should be withing a small tolerance of the loopback case (as it really isn't different)
- network (on-wire) traffic should be as fast as without the namespace (i.e. within 1% or so, better not really measurable)
Quoted 5 lines
- all this should be true in a setup with a significant number of guests, when only one guest is active, but all other guests are ready/configured
- all this should scale well with a few hundred guests
Ultimately I agree. However. Only host performance should be a merge blocker. Allowing us to go back and reclaim the few percentage points we lost later.
Quoted 9 lines
Quoted 5 lines
If we don't hurt that core case we have an implementation we can merge. There are a lot of optimization opportunities for local communications and we can do that after we have a correct and accepted implementation. Anything else is optimizing too soon, and will just be muddying the waters.
what I fear is that once something is in, the kernel will just become slower (as it already did in some areas) and nobody will care/be-able to fix that later on ...
If nobody cares it doesn't matter.
If no one can fix it that is a problem. Which is why we need high standards and clean code, not early optimizations.
But on that front each step of the way must be justified on it's own merits. Not because it will give us some holy grail.
The way to keep the inter guest performance from degrading is to measure it an complain. But the linux network stack is too big to get in one pass.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 9 lines · Eric W. Biederman
Quoted 6 lines · Herbert Poetzl
Quoted 1 line · Eric W. Biederman
Inside the containers I want all network devices named eth0!
huh? even if there are two of them? also tun?
Quoted 8 lines
Quoted 3 lines
I think you meant, you want to be able to have eth0 in _more_ than one guest where eth0 in a guest can also be/use/relate to eth1 on the host, right?
Right I want to have an eth0 in each guest where eth0 is it's own network device and need have no relationship to eth0 on the host.
How does that help anything? Do you envision programs that make special decisions on whether the interface is called eth0 v/s eth151?
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 15 lines · Ben Greear
Quoted 14 lines · Eric W. Biederman
Quoted 9 lines · Herbert Poetzl
Quoted 1 line · Eric W. Biederman
Inside the containers I want all network devices named eth0!
huh? even if there are two of them? also tun?
I think you meant, you want to be able to have eth0 in _more_ than one guest where eth0 in a guest can also be/use/relate to eth1 on the host, right?
Quoted 7 lines
Quoted 3 lines
Right I want to have an eth0 in each guest where eth0 is it's own network device and need have no relationship to eth0 on the host.
How does that help anything? Do you envision programs that make special decisions on whether the interface is called eth0 v/s eth151?
well, those poor folks who do not have ethernet devices for networking :)
seriously, what I think Eric meant was that it might be nice (especially for migration purposes) to keep the device namespace completely virtualized and not just isolated ...
I'm fine with that, as long as it does not add overhead or complicate handling, and as far as I can tell, it should not do that ...
best,
Herbert
Quoted 6 lines
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 4 lines · Herbert Poetzl
seriously, what I think Eric meant was that it might be nice (especially for migration purposes) to keep the device namespace completely virtualized and not just isolated ...
It might be nice, but it is probably unneeded for an initial implementation. In practice, a cluster doing checkpoint/restart/migration will already have a system in place for assigning unique IPs or other identifiers to each container. It could just as easily make sure to assign unique network device names to containers.
The issues really only come into play when you have an unstructured set of machines and you want to migrate between them without having prepared them with any kind of unique net device names beforehand.
It may look weird, but do application really *need* to see eth0 rather than eth858354?
-- Dave
Re: [patch 2/6] [Network namespace] Network device sharing by view
Hello!
Quoted 2 lines
It may look weird, but do application really *need* to see eth0 rather than eth858354?
Applications do not care, humans do. :-)
What's about applications they just need to see exactly the same device after migration. Not only name, but f.e. also its ifindex. If you do not create a separate namespace for netdevices, you will inevitably end up with some strange hack sort of VPIDs to translate (or to partition) ifindices or to tell that "ping -I eth858354 xxx" is too coimplicated application to survive migration.
Alexey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 5 lines · Alexey Kuznetsov
Hello!
Quoted 2 lines
It may look weird, but do application really *need* to see eth0 rather than eth858354?
Quoted 8 lines
Applications do not care, humans do. :-)
What's about applications they just need to see exactly the same device after migration. Not only name, but f.e. also its ifindex. If you do not create a separate namespace for netdevices, you will inevitably end up with some strange hack sort of VPIDs to translate (or to partition) ifindices or to tell that "ping -I eth858354 xxx" is too coimplicated application to survive migration.
Actually there are applications with peculiar licensing practices that do look at devices like eth0 to verify you have the appropriate mac, and do really weird things if you don't have an eth0.
Plus there are other cases where it can be simpler to hard code things if it is allowable. (The human factor) Otherwise your configuration must be done through hotplug scripts.
But yes there are misguided applications that care.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 21 lines · Eric W. Biederman
Quoted 5 lines · Alexey Kuznetsov
Hello!
Quoted 2 lines
It may look weird, but do application really *need* to see eth0 rather than eth858354?
Quoted 8 lines
Applications do not care, humans do. :-)
What's about applications they just need to see exactly the same device after migration. Not only name, but f.e. also its ifindex. If you do not create a separate namespace for netdevices, you will inevitably end up with some strange hack sort of VPIDs to translate (or to partition) ifindices or to tell that "ping -I eth858354 xxx" is too coimplicated application to survive migration.
Actually there are applications with peculiar licensing practices that do look at devices like eth0 to verify you have the appropriate mac, and do really weird things if you don't have an eth0.
Quoted 5 lines
Plus there are other cases where it can be simpler to hard code things if it is allowable. (The human factor) Otherwise your configuration must be done through hotplug scripts.
But yes there are misguided applications that care.
last time I pointed to such 'misguided' apps which made assumptions that are not necessarily true inside a virtual environment (e.g. pstree, initpid) the general? position was that those apps should be fixed instead adding a 'workaround'
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
best,
Herbert
Quoted 1 line
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 6 lines · Herbert Poetzl
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
Anyways, interesting discussion.
cheers,
jamal
Re: [patch 2/6] [Network namespace] Network device sharing by view
Hi Jamal,
Quoted 10 lines · jamal
Quoted 6 lines · Herbert Poetzl
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Quoted 6 lines
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
Seeing guestXX-eth0 interfaces by standard tools has certain attractive sides. But it creates a lot of undesired side effects.
For example, ntpd queries all network devices by the same ioctls as ifconfig, and creates separate sockets bound to IP addresses of each device, which is certainly not desired with namespaces.
Or more subtle question: do you want hotplug events to be generated when guest0-eth0 interface comes up in the root namespace, and standard scripts to try to set some IP address on this interface?..
In my opinion, the downside of this scheme overweights possible advantages, and I'm personally quite happy with running commands with switched namespace, like
vzctl exec guest0 ip addr list
vzctl exec guest0 ip link set eth0 up
and so on.
Best regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Andrey,
Quoted 9 lines · Andrey Savochkin
Hi Jamal,
Quoted 1 line · jamal
Seeing guestXX-eth0 interfaces by standard tools has certain attractive sides. But it creates a lot of undesired side effects.
I apologize because i butted into the discussion without perhaps reading the full thread.
Quoted 4 lines
For example, ntpd queries all network devices by the same ioctls as ifconfig, and creates separate sockets bound to IP addresses of each device, which is certainly not desired with namespaces.
Ok, so the problem is that ntp in this case runs on the host side as opposed to the guest? This would explain why Eric is reacting vehemently to the suggestion.
Quoted 4 lines
Or more subtle question: do you want hotplug events to be generated when guest0-eth0 interface comes up in the root namespace, and standard scripts to try to set some IP address on this interface?..
yes, thats what i was thinking. Even go further and actually create guestxx-eth0 on the host (which results in creating eth0 on the guest) and other things.
Quoted 6 lines
In my opinion, the downside of this scheme overweights possible advantages, and I'm personally quite happy with running commands with switched namespace, like
vzctl exec guest0 ip addr list
vzctl exec guest0 ip link set eth0 up
and so on.
Ok, above may be good enough and doesnt require any state it seems on the host side.
I got motivated when the word "migration" was mentioned. I understood it to be meaning that a guest may become inoperative for some reason and that its info will be transfered to another guest which may be local or even remote. In such a case, clearly one would need a protocol and the state of all guests sitting at the host. Maybe i am over-reaching.
cheers,
jamal
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 9 lines · jamal
Quoted 4 lines · Andrey Savochkin
Seeing guestXX-eth0 interfaces by standard tools has certain attractive sides. But it creates a lot of undesired side effects.
I apologize because i butted into the discussion without perhaps reading the full thread.
Your comments are quite welcome
Quoted 7 lines
Quoted 5 lines
For example, ntpd queries all network devices by the same ioctls as ifconfig, and creates separate sockets bound to IP addresses of each device, which is certainly not desired with namespaces.
Ok, so the problem is that ntp in this case runs on the host side as
yes
Quoted 2 lines
opposed to the guest? This would explain why Eric is reacting vehemently to the suggestion.
:)
And I actually do not want to distinguish host and guest sides much. They are namespaces in the first place.
Parent namespace may have some capabilities to manipulate its child namespaces, like donate its own device to one of its children.
But it comes secondary to having namespace isolation borders. In particular, because most cases of cross-namespace interaction lead to failures of formal security models and inability to migrate namespaces between computers.
Quoted 9 lines
Quoted 5 lines
Or more subtle question: do you want hotplug events to be generated when guest0-eth0 interface comes up in the root namespace, and standard scripts to try to set some IP address on this interface?..
yes, thats what i was thinking. Even go further and actually create guestxx-eth0 on the host (which results in creating eth0 on the guest) and other things.
This actually goes in the opposite direction to what I keep in mind. I want to offload as much as possible of network administration work to guests. Delegation of management is one of the motivating factors behind covering not only sockets but devices, routes, and so on by the namespace patches.
Quoted 8 lines
Quoted 7 lines
In my opinion, the downside of this scheme overweights possible advantages, and I'm personally quite happy with running commands with switched namespace, like
vzctl exec guest0 ip addr list
vzctl exec guest0 ip link set eth0 up
and so on.
Quoted 7 lines
Ok, above may be good enough and doesnt require any state it seems on the host side.
I got motivated when the word "migration" was mentioned. I understood it to be meaning that a guest may become inoperative for some reason and that its info will be transfered to another guest which may be local or even remote. In such a case, clearly one would need a protocol and the state of all guests sitting at the host. Maybe i am over-reaching.
Migration will work inside the kernel, so it has full access to whatever state information it needs.
Best regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to jamalQuoted 9 lines · jamal
Andrey,
Quoted 6 lines · Andrey Savochkin
Hi Jamal,
Quoted 1 line · jamal
Quoted 6 lines
Quoted 3 lines
Seeing guestXX-eth0 interfaces by standard tools has certain attractive sides. But it creates a lot of undesired side effects.
I apologize because i butted into the discussion without perhaps reading the full thread.
This thread is serving as an educational vehicle, and the more people from outside of our little biased group that begin to understand what we are about the better.
Quoted 8 lines
Quoted 4 lines
For example, ntpd queries all network devices by the same ioctls as ifconfig, and creates separate sockets bound to IP addresses of each device, which is certainly not desired with namespaces.
Ok, so the problem is that ntp in this case runs on the host side as opposed to the guest? This would explain why Eric is reacting vehemently to the suggestion.
Yes, that would be one problem case.
Quoted 16 lines
Quoted 4 lines
Or more subtle question: do you want hotplug events to be generated when guest0-eth0 interface comes up in the root namespace, and standard scripts to try to set some IP address on this interface?..
yes, thats what i was thinking. Even go further and actually create guestxx-eth0 on the host (which results in creating eth0 on the guest) and other things.
Quoted 6 lines
In my opinion, the downside of this scheme overweights possible advantages, and I'm personally quite happy with running commands with switched namespace, like
vzctl exec guest0 ip addr list
vzctl exec guest0 ip link set eth0 up
and so on.
Quoted 7 lines
Ok, above may be good enough and doesnt require any state it seems on the host side.
I got motivated when the word "migration" was mentioned. I understood it to be meaning that a guest may become inoperative for some reason and that its info will be transfered to another guest which may be local or even remote. In such a case, clearly one would need a protocol and the state of all guests sitting at the host. Maybe i am over-reaching.
Not really. Network namespaces while useful in their own right lay the foundation for some more interesting applications. Application migration between machines in particular.
The biggest fundamental problem in migration is after checkpointing your application you can not acquire the resources you need on the new machine because of name conflicts.
So for those of us concerned with migration a question we ask is can we successfully import resource names that another machine has assigned without consulting us.
The context for all of this goes to other discussion that we have been having since January. Breaking all of this into small pieces that can be merged and tested a little at a time is a challenge.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Andrey SavochkinQuoted 19 lines · Andrey Savochkin
Hi Jamal,
Quoted 9 lines · jamal
Quoted 5 lines · Herbert Poetzl
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Quoted 6 lines
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
Quoted 2 lines
Seeing guestXX-eth0 interfaces by standard tools has certain attractive sides. But it creates a lot of undesired side effects.
which all can be avoided by not using the host context for that, but a special 'all seeing' context (as we have in Linux-VServer) which can see (and probably manipulate) those interfaces from the 'admin' PoV without entering the guest context
Quoted 3 lines
For example, ntpd queries all network devices by the same ioctls as ifconfig, and creates separate sockets bound to IP addresses of each device, which is certainly not desired with namespaces.
applications scanning the interfaces at startup are broken by design and should probably be fixed instead of worked around ...
Quoted 3 lines
Or more subtle question: do you want hotplug events to be generated when guest0-eth0 interface comes up in the root namespace, and standard scripts to try to set some IP address on this interface?..
why not, would it do any harm when the hotplug scripts on the host would take the appropriate actions (i.e. do the required config for the guest) for special guest interfaces?
but now that you mention it, what about hotplug events inside the guest?
Quoted 5 lines
In my opinion, the downside of this scheme overweights possible advantages, and I'm personally quite happy with running commands with switched namespace, like
vzctl exec guest0 ip addr list
vzctl exec guest0 ip link set eth0 up
I do not consider this the best solution, especially from the security PoV. don't forget you basically enter the guest and execute arbitrary programs (which might have been compromised) to do a setup task you actually want to happen on the host
best,
Herbert
Quoted 5 lines
and so on.
Best regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to jamalQuoted 9 lines · jamal
Quoted 6 lines · Herbert Poetzl
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Quoted 8 lines
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
Anyways, interesting discussion.
Please no.
We really want the fundamental rule that a network device is tied to a single namespace, and that a socket is tied to a single namespace. If those two conditions are met we don't have to tag packets with a namespace identifier.
We only have to modify hash table lookups in the networking code to look at a namespace tag in addition to the rest because that is less expensive than allocating new hash tables.
Currently with a network device only being usable in one network namespace we have the situation where we can fairly safely give a guest CAP_NET_ADMIN without problems.
In addition currently nothing in the implementation knows about the hierarchical structure of how the network namespace will be used. To allow ifconfig guest0-eth0 to work would require understanding the hierarchical structure and places serious questions on how safe we can make CAP_NET_ADMIN.
Now I am open to radically different designs if they allow the implementation cost to be lower and they have clean semantics, and don't wind up being an ugly unmaintainable wart on the linux networking stack. The only route I could imagine such a thing coming from is something like tagging flows, in some netfiler like way. Which might allow ifconfig guest-eth0 from the host without problems. But I have not seen such a design.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 7 lines · Eric W. Biederman
Quoted 7 lines
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
Quoted 10 lines
Please no.
[...]
Now I am open to radically different designs if they allow the implementation cost to be lower and they have clean semantics, and don't wind up being an ugly unmaintainable wart on the linux networking stack. The only route I could imagine such a thing coming from is something like tagging flows, in some netfiler like way. Which might allow ifconfig guest-eth0 from the host without problems. But I have not seen such a design.
Right. New tools to support new features would probably be tidier, anyway.
Sam.
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to jamalQuoted 8 lines · jamal
Quoted 7 lines
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Quoted 6 lines
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
That particular convention only works if you have network namespaces and UTS namespaces tightly bound. We plan to have them separate - so for that to work, each network namespace could have an arbitrary "prefix" that determines what the interface name will look like from the outside when combined. We'd have to be careful about length limits.
And guest0-eth0 doesn't necessarily make sense; it's not really an ethernet interface, more like a tun or something.
So, an equally good convention might be to use sequential prefixes on the host, like "tun", "dummy", or a new prefix - then a property of that is what the name of the interface is perceived to be to those who are in the corresponding network namespace.
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
Sam.
strict isolation of net interfaces
Quoted 24 lines · Sam Vilain
Quoted 7 lines · jamal
Quoted 7 lines
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Quoted 6 lines
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
That particular convention only works if you have network namespaces and UTS namespaces tightly bound. We plan to have them separate - so for that to work, each network namespace could have an arbitrary "prefix" that determines what the interface name will look like from the outside when combined. We'd have to be careful about length limits.
And guest0-eth0 doesn't necessarily make sense; it's not really an ethernet interface, more like a tun or something.
Quoted 7 lines
So, an equally good convention might be to use sequential prefixes on the host, like "tun", "dummy", or a new prefix - then a property of that is what the name of the interface is perceived to be to those who are in the corresponding network namespace.
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
we could work on virtualizing the net interfaces in the host, map them to eth0 or something in the guest and let the guest handle upper network layers ?
lo0 would just be exposed relying on skbuff tagging to discriminate traffic between guests.
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------
| | | |
|-> l0 <-------+-> lo0 ... | lo0 | lo0
| | | |
|-> bar0 <--------+-> eth0 | |
| | | |
|-> foo0 <--------+-----------+-----------+-> eth0
| | | |
`-> foo0:1 <-------+-----------+-> eth0 |
| | |is that clear ? stupid ? reinventing the wheel ?
thanks,
C.
Re: strict isolation of net interfaces
Quoted 34 lines · Cedric Le Goater
Quoted 24 lines · Sam Vilain
Quoted 7 lines · jamal
Quoted 7 lines
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Quoted 6 lines
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
That particular convention only works if you have network namespaces and UTS namespaces tightly bound. We plan to have them separate - so for that to work, each network namespace could have an arbitrary "prefix" that determines what the interface name will look like from the outside when combined. We'd have to be careful about length limits.
And guest0-eth0 doesn't necessarily make sense; it's not really an ethernet interface, more like a tun or something.
Quoted 7 lines
So, an equally good convention might be to use sequential prefixes on the host, like "tun", "dummy", or a new prefix - then a property of that is what the name of the interface is perceived to be to those who are in the corresponding network namespace.
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
Quoted 5 lines
we could work on virtualizing the net interfaces in the host, map them to eth0 or something in the guest and let the guest handle upper network layers ?
lo0 would just be exposed relying on skbuff tagging to discriminate traffic between guests.
This seems to me the preferable way. We create a full virtual net device for each new container, and fully virtualize the device namespace.
Quoted 13 lines
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------| | | | |-> l0 <-------+-> lo0 ... | lo0 | lo0 | | | | |-> bar0 <--------+-> eth0 | | | | | | |-> foo0 <--------+-----------+-----------+-> eth0 | | | | `-> foo0:1 <-------+-----------+-> eth0 | | | |
Quoted 1 line
is that clear ? stupid ? reinventing the wheel ?
The last one in your diagram confuses me - why foo0:1? I would have thought it'd be
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------
| | | | |-> l0 <-------+-> lo0 ... | lo0 | lo0 | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |
I think we should avoid using device aliases, as trying to do something like giving eth0:1 to guest1 and eth0:2 to guest2 while hiding eth0:1 from guest2 requires some uglier code (as I recall) than working with full devices. In other words, if a namespace can see eth0, and eth0:2 exists, it should always see eth0:2.
So conceptually using a full virtual net device per container certainly seems cleaner to me, and it seems like it should be simpler by way of statistics gathering etc, but are there actually any real gains? Or is the support for multiple IPs per device actually enough?
Herbert, is this basically how ngnet is supposed to work?
-serge
Re: strict isolation of net interfaces
Quoted 16 lines · Serge E. Hallyn
The last one in your diagram confuses me - why foo0:1? I would have thought it'd be
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------| | | | |-> l0 <-------+-> lo0 ... | lo0 | lo0 | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |
Quoted 8 lines
[...]
So conceptually using a full virtual net device per container certainly seems cleaner to me, and it seems like it should be simpler by way of statistics gathering etc, but are there actually any real gains? Or is the support for multiple IPs per device actually enough?
Why special case loopback?
Why not:
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------
| | | | |-> lo | | | | | | | |-> vlo0 <---------+-> lo | | | | | | |-> vlo1 <---------+-----------+-----------+-> lo | | | | |-> vlo2 <--------+-----------+-> lo | | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |
Sam.
Re: strict isolation of net interfaces
Sam, Serge, Cedric,
Quoted 32 lines · Sam Vilain
Quoted 16 lines · Serge E. Hallyn
The last one in your diagram confuses me - why foo0:1? I would have thought it'd be
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------| | | | |-> l0 <-------+-> lo0 ... | lo0 | lo0 | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |Quoted 8 lines
[...]
So conceptually using a full virtual net device per container certainly seems cleaner to me, and it seems like it should be simpler by way of statistics gathering etc, but are there actually any real gains? Or is the support for multiple IPs per device actually enough?
Why special case loopback?
Why not:
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------
Quoted 16 lines
| | | | |-> lo | | | | | | | |-> vlo0 <---------+-> lo | | | | | | |-> vlo1 <---------+-----------+-----------+-> lo | | | | |-> vlo2 <--------+-----------+-> lo | | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |
I still can't completely understand your direction of thoughts. Could you elaborate on IP address assignment in your diagram, please? For example, guest0 wants 127.0.0.1 and 192.168.0.1 addresses on its lo interface, and 10.1.1.1 on its eth0 interface.
Does this diagram assume any local IP addresses on v* interfaces in the "host"?
And the second question.
Are vlo0, veth0, etc. devices supposed to have hard_xmit routines?
Best regards
Andrey
Re: strict isolation of net interfaces
Quoted 24 lines · Andrey Savochkin
Quoted 6 lines
Why special case loopback?
Why not:
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------Quoted 17 lines
| | | | |-> lo | | | | | | | |-> vlo0 <---------+-> lo | | | | | | |-> vlo1 <---------+-----------+-----------+-> lo | | | | |-> vlo2 <--------+-----------+-> lo | | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |
Quoted 7 lines
I still can't completely understand your direction of thoughts. Could you elaborate on IP address assignment in your diagram, please? For example, guest0 wants 127.0.0.1 and 192.168.0.1 addresses on its lo interface, and 10.1.1.1 on its eth0 interface.
Does this diagram assume any local IP addresses on v* interfaces in the "host"?
Well, Eric already pointed out some pretty good reasons why this thread should die.
The idea is that each "lo" interface would have the same set of addresses. Which would make routing on the host confusing. Yet another reason to kill this idea. Let's just make better tools instead.
Sam.
Quoted 3 lines
And the second question.
Are vlo0, veth0, etc. devices supposed to havehard_xmitroutines?
Re: strict isolation of net interfaces
In reply to Andrey SavochkinQuoted 10 lines · Andrey Savochkin
I still can't completely understand your direction of thoughts. Could you elaborate on IP address assignment in your diagram, please? For example, guest0 wants 127.0.0.1 and 192.168.0.1 addresses on its lo interface, and 10.1.1.1 on its eth0 interface.
Does this diagram assume any local IP addresses on v* interfaces in the "host"?And the second question.
Are vlo0, veth0, etc. devices supposed to havehard_xmitroutines?
Andrey,
some people are interested by a network full isolation/virtualization like you did with the layer 2 isolation and some other people are interested by a light network isolation done at the layer 3. This one is intended to implement "application container" aka "lightweight container".
In the case of a layer 3 isolation, the network interface is not totally isolated and the debate here is to find a way to have something intuitive to manage the network devices.
IHMO, all the discussion we had convinced me of the needs to have the possibility to choose between a layer 2 or a layer 3 isolation.
If it is ok for you, we can collaborate to merge the two solutions in one. I will focus on layer 3 isolation and you on the layer 2.
Regards
- Daniel
Re: strict isolation of net interfaces
Quoted 3 lines · Daniel Lezcano
If it is ok for you, we can collaborate to merge the two solutions in one. I will focus on layer 3 isolation and you on the layer 2.
So, you're writing a LSM module or adapting the BSD Jail LSM, right? :)
Sam.
Re: strict isolation of net interfaces
Quoted 9 lines · Sam Vilain
Quoted 2 lines · Daniel Lezcano
If it is ok for you, we can collaborate to merge the two solutions in one. I will focus on layer 3 isolation and you on the layer 2.
So, you're writing a LSM module or adapting the BSD Jail LSM, right? :)
Sam.
No. I am adapting a prototype of network application container we did.
-- Daniel
Re: strict isolation of net interfaces
In reply to Serge E. HallynQuoted 3 lines · Serge E. Hallyn
The last one in your diagram confuses me - why foo0:1? I would have thought it'd be
just thinking aloud. I thought that any kind/type of interface could be mapped from host to guest.
Quoted 27 lines
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------| | | | |-> l0 <-------+-> lo0 ... | lo0 | lo0 | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |I think we should avoid using device aliases, as trying to do something like giving eth0:1 to guest1 and eth0:2 to guest2 while hiding eth0:1 from guest2 requires some uglier code (as I recall) than working with full devices. In other words, if a namespace can see eth0, and eth0:2 exists, it should always see eth0:2.
So conceptually using a full virtual net device per container certainly seems cleaner to me, and it seems like it should be simpler by way of statistics gathering etc, but are there actually any real gains? Or is the support for multiple IPs per device actually enough?
Herbert, is this basically how ngnet is supposed to work?
Re: strict isolation of net interfaces
Quoted 35 lines · Cedric Le Goater
Quoted 3 lines · Serge E. Hallyn
The last one in your diagram confuses me - why foo0:1? I would have thought it'd be
just thinking aloud. I thought that any kind/type of interface could be mapped from host to guest.
Quoted 27 lines
host | guest 0 | guest 1 | guest2
----------------------+-----------+-----------+--------------| | | | |-> l0 <-------+-> lo0 ... | lo0 | lo0 | | | | |-> eth0 | | | | | | | |-> veth0 <--------+-> eth0 | | | | | | |-> veth1 <--------+-----------+-----------+-> eth0 | | | | |-> veth2 <-------+-----------+-> eth0 |I think we should avoid using device aliases, as trying to do something like giving eth0:1 to guest1 and eth0:2 to guest2 while hiding eth0:1 from guest2 requires some uglier code (as I recall) than working with full devices. In other words, if a namespace can see eth0, and eth0:2 exists, it should always see eth0:2.
So conceptually using a full virtual net device per container certainly seems cleaner to me, and it seems like it should be simpler by way of statistics gathering etc, but are there actually any real gains? Or is the support for multiple IPs per device actually enough?
Herbert, is this basically how ngnet is supposed to work?
hard to tell, we have at least three ngnet prototypes and basically all variants are covered there, from separate interfaces which map to real ones to perfect isolation of addresses assigned to global interfaces
IMHO the 'virtual' interface per guest is fine, as the overhead and consumed resources are non critical and it will definitely simplify handling for the guest side
I'd really appreciate if we could find a solution which allows both, isolation and virtualization, and if the bridge scenario is as fast as a direct mapping, I'm perfectly fine with a big bridge + ebtables to handle security issues
best,
Herbert
Re: strict isolation of net interfaces
In reply to Serge E. HallynQuoted 5 lines · Serge E. Hallyn
Quoted 3 lines · Cedric Le Goater
we could work on virtualizing the net interfaces in the host, map them to eth0 or something in the guest and let the guest handle upper network layers ?
Quoted 7 lines
Quoted 2 lines
lo0 would just be exposed relying on skbuff tagging to discriminate traffic between guests.
This seems to me the preferable way. We create a full virtual net device for each new container, and fully virtualize the device namespace.
I have a few questions about all the network isolation stuff:
* What level of isolation is wanted for the network ? network devices
? IPv4/IPv6 ? TCP/UDP ?
* How is handled the incoming packets from the network ? I mean what
will be mecanism to dispatch the packet to the right virtual device ?
* How to handle the SO_BINDTODEVICE socket option ?
* Has the virtual device a different MAC address ? How to manage it
with the real MAC address on the system ? How to manage ARP, ICMP, multicasting and IP ?
It seems for me, IMHO that will require a lot of translation and browsing table. It will probably add a very significant overhead.
* How to handle NFS access mounted outside of the container ?
* How to handle ICMP_REDIRECT ?
Regards
Re: strict isolation of net interfaces
Quoted 6 lines · Daniel Lezcano
Quoted 5 lines · Serge E. Hallyn
Quoted 3 lines · Cedric Le Goater
we could work on virtualizing the net interfaces in the host, map them to eth0 or something in the guest and let the guest handle upper network layers ?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
lo0 would just be exposed relying on skbuff tagging to discriminate traffic between guests.
This seems to me the preferable way. We create a full virtual net device for each new container, and fully virtualize the device namespace.
I have a few questions about all the network isolation stuff:
So far I have seen two viable possibilities on the table, neither of them involve multiple names for a network device.
layer 3 (filtering the allowed ip addresses at bind time roughly the current vserver).
- implementable as a security hook.
- Benefit no measurable performance impact.
- Downside not many things we can do.
layer 2 (What appears to applications a separate instance of the network stack).
- Implementable as a namespace.
- Each network namespace would have dedicated network devices.
- Benefit extremely flexible.
- Downside since at least the slow path must examine the packet it has the possibility of slowing down the networking stack.
For me the important characteristics.
- Allows for application migration, when we take our ip address with us. In particular it allows for importation of addresses assignments mad on other machines.
- No measurable impact on the existing networking when the code is compiled in.
- Clean predictable semantics.
This whole debate on network devices show up in multiple network namespaces is just silly. The only reason for wanting that appears to be better management. We have deeper issues like can we do a reasonable implementation without a network device showing up in multiple namespaces.
I think the reason the debate exists at all is that it is a very approachable topic, as opposed to the fundamentals here.
If we can get layer 2 level isolation working without measurable overhead with one namespace per device it may be worth revisiting things. Until then it is a side issue at best.
Eric
Re: strict isolation of net interfaces
Quoted 11 lines · Eric W. Biederman
Quoted 8 lines · Daniel Lezcano
Quoted 6 lines · Serge E. Hallyn
Quoted 3 lines · Cedric Le Goater
we could work on virtualizing the net interfaces in the host, map them to eth0 or something in the guest and let the guest handle upper network layers ?
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
lo0 would just be exposed relying on skbuff tagging to discriminate traffic between guests.
This seems to me the preferable way. We create a full virtual net device for each new container, and fully virtualize the device namespace.
I have a few questions about all the network isolation stuff:
It seems these questions are not important.
Quoted 8 lines
So far I have seen two viable possibilities on the table, neither of them involve multiple names for a network device.
layer 3 (filtering the allowed ip addresses at bind time roughly the current vserver).
- implementable as a security hook.
- Benefit no measurable performance impact.
- Downside not many things we can do.
What things ? Can you develop please ? Can you give some examples ?
Quoted 3 lines
layer 2 (What appears to applications a separate instance of the network stack).
- Implementable as a namespace.
what about accessing a NFS mounted outside the container ?
Quoted 2 lines
- Each network namespace would have dedicated network devices.
- Benefit extremely flexible.
For what ? For who ? Do you have examples ?
Quoted 2 lines
- Downside since at least the slow path must examine the packet it has the possibility of slowing down the networking stack.
What is/are the slow path(s) you identified ?
Quoted 4 lines
For me the important characteristics.
- Allows for application migration, when we take our ip address with us. In particular it allows for importation of addresses assignments mad on other machines.
Ok for the two methods no ?
Quoted 2 lines
- No measurable impact on the existing networking when the code is compiled in.
You contradict ...
Quoted 1 line
- Clean predictable semantics.
What that means ? Can you explain, please ?
Quoted 2 lines
This whole debate on network devices show up in multiple network namespaces is just silly.
The debate is not on the network device show up. The debate is can we have a network isolation ___usable for everybody___ not only for the beauty of having namespaces and for a system container like.
I am not against the network device virtualization or against the namespaces. I am just asking if the namespace is the solution for all the network isolation. Should we nest layer 2 and layer 3 vitualization into namespaces or separate them in order to have the flexibility to choose isolation/performance.
Quoted 3 lines
The only reason for wanting that appears to be better management. We have deeper issues like can we do a reasonable implementation without a network device showing up in multiple namespaces.
Again, I am not against having the network device virtualization. It is a good idea.
Quoted 6 lines
I think the reason the debate exists at all is that it is a very approachable topic, as opposed to the fundamentals here.
If we can get layer 2 level isolation working without measurable overhead with one namespace per device it may be worth revisiting things. Until then it is a side issue at best.
I agree, so where are the answers of the questions I asked in my previous email ? You said you did some implementation of network isolation with and without namespaces, so you should be able to answer...
-- Daniel
Re: strict isolation of net interfaces
Quoted 15 lines · Daniel Lezcano
Quoted 9 lines · Eric W. Biederman
Quoted 7 lines · Daniel Lezcano
Quoted 5 lines · Serge E. Hallyn
Quoted 2 lines · Cedric Le Goater
we could work on virtualizing the net interfaces in the host, map them to eth0 or something in the guest and let the guest handle upper network layers
?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
lo0 would just be exposed relying on skbuff tagging to discriminate traffic between guests.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
This seems to me the preferable way. We create a full virtual net device for each new container, and fully virtualize the device namespace.
I have a few questions about all the network isolation stuff:
It seems these questions are not important.
I'm just trying to get us back to a productive topic.
Quoted 9 lines
Quoted 7 lines
So far I have seen two viable possibilities on the table, neither of them involve multiple names for a network device. layer 3 (filtering the allowed ip addresses at bind time roughly the current vserver).
- implementable as a security hook.
- Benefit no measurable performance impact.
- Downside not many things we can do.What things ? Can you develop please ? Can you give some examples ?
DHCP, tcpdump,.. Probably a bad way of phrasing it. But there is a lot more that we can do using a pure layer 2 approach.
Quoted 5 lines
Quoted 3 lines
layer 2 (What appears to applications a separate instance of the network stack).
- Implementable as a namespace.what about accessing a NFS mounted outside the container ?
As I replied earlier it isn't a problem. If you get to it through the filesystem namespace it uses the network namespace it was mounted with for it's connection.
Quoted 4 lines
Quoted 2 lines
- Each network namespace would have dedicated network devices.
- Benefit extremely flexible.For what ? For who ? Do you have examples ?
See above.
Quoted 4 lines
Quoted 2 lines
- Downside since at least the slow path must examine the packet it has the possibility of slowing down the networking stack.
What is/are the slow path(s) you identified ?
Grr. I put that badly. Basically at least on the slow path you need to look at a per network namespace data structure. The extra pointer indirection could slow things down. The point is that we may be able to have a fast path that is exactly the same as the rest of the network stack.
If the obvious approach does not work my gut the feeling the network stack fast path will give us an implementation without overhead.
Quoted 6 lines
Quoted 4 lines
For me the important characteristics.
- Allows for application migration, when we take our ip address with us. In particular it allows for importation of addresses assignments mad on other machines.Ok for the two methods no ?
So far.
Quoted 4 lines
Quoted 2 lines
- No measurable impact on the existing networking when the code is compiled in.
You contradict ...
How so? As far as I can tell this is a basic requirement to get merged.
Quoted 10 lines
Quoted 1 line
- Clean predictable semantics.
What that means ? Can you explain, please ?
Quoted 2 lines
This whole debate on network devices show up in multiple network namespaces is just silly.
The debate is not on the network device show up. The debate is can we have a network isolation
___usableforeverybody___not only for the beauty of having namespaces and for a system container like.
This subthread talking about devices showing up in multiple namespaces seemed
very much exactly on how network devices show up.
Quoted 4 lines
I am not against the network device virtualization or against the namespaces. I am just asking if the namespace is the solution for all the network isolation. Should we nest layer 2 and layer 3 vitualization into namespaces or separate them in order to have the flexibility to choose isolation/performance.
I believe I addressed Herbert Poetzl's concerns earlier. To me the question is can we implement an acceptable layer 2 solution, that distrubutions and other people who do not need isolation would have no problem compiling in by default.
The joy of namespaces is that if you don't want it you don't have to use it. Layer 2 can do everything and is likely usable by everyone iff the performance is acceptable.
Quoted 13 lines
Quoted 3 lines
The only reason for wanting that appears to be better management. We have deeper issues like can we do a reasonable implementation without a network device showing up in multiple namespaces.
Again, I am not against having the network device virtualization. It is a good idea.
Quoted 5 lines
I think the reason the debate exists at all is that it is a very approachable topic, as opposed to the fundamentals here.
If we can get layer 2 level isolation working without measurable overhead with one namespace per device it may be worth revisiting things. Until then it is a side issue at best.
Quoted 3 lines
I agree, so where are the answers of the questions I asked in my previous email ? You said you did some implementation of network isolation with and without namespaces, so you should be able to answer...
Sorry. More than anything those questions looked retorical and aimed at disarming some of the silliness. I will go back and try and answer those. Fundamentally when we have one namespace that includes network devices, network sockets, and all of the data structures necessary to use them (routing tables and the like) and we have a tunnel device that can connect namespaces the answers are trivial and I though obvious.
Eric
Re: strict isolation of net interfaces
In reply to Eric W. BiedermanQuoted 2 lines · Eric W. Biederman
This whole debate on network devices show up in multiple network namespaces is just silly. The only reason for wanting that appears to be better management.
A damned good reason. Clearly we want the parent namespace to be able to control what the child can do. So whatever interface a child gets, the parent should be able to somehow address. Simple iptables rules controlling traffic between it's own netdevice and the one it hands it's children seem a good option.
Quoted 2 lines
We have deeper issues like can we do a reasonable implementation without a network device showing up in multiple namespaces.
Isn't that the same issue?
Quoted 3 lines
If we can get layer 2 level isolation working without measurable overhead with one namespace per device it may be worth revisiting things. Until then it is a side issue at best.
Ok, and in the meantime we can all use the network part of the bsdjail lsm? :)
-serge
Re: strict isolation of net interfaces
Quoted 6 lines · Serge E. Hallyn
Quoted 2 lines · Eric W. Biederman
This whole debate on network devices show up in multiple network namespaces is just silly. The only reason for wanting that appears to be better
management.
A damned good reason.
Better management is a good reason. But constructing the management in a way that hampers the implementation and confuses existing applications is a problem.
Things are much easier if namespaces are completely independent.
Among other things the semantics are clear and obvious.
Quoted 5 lines
Clearly we want the parent namespace to be able
to control what the child can do. So whatever interface a child gets, the parent should be able to somehow address. Simple iptables rules controlling traffic between it's own netdevice and the one it hands it's children seem a good option.
That or we setup the child and then drop CAP_NET_ADMIN.
Quoted 4 lines
Quoted 2 lines
We have deeper issues like can we do a reasonable implementation without a network device showing up in multiple namespaces.
Isn't that the same issue?
I guess I was thinking from the performance and cleanliness point of view.
Quoted 6 lines
Quoted 3 lines
If we can get layer 2 level isolation working without measurable overhead with one namespace per device it may be worth revisiting things. Until then it is a side issue at best.
Ok, and in the meantime we can all use the network part of the bsdjail lsm? :)
If necessary. But mostly we concentrate on the fundamentals and figure out what it takes to take the level 2 stuff working.
Eric
Re: strict isolation of net interfaces
In reply to Daniel LezcanoQuoted 12 lines · Daniel Lezcano
Quoted 10 lines · Serge E. Hallyn
Quoted 5 lines · Cedric Le Goater
we could work on virtualizing the net interfaces in the host, map them to eth0 or something in the guest and let the guest handle upper network layers ?
lo0 would just be exposed relying on skbuff tagging to discriminate traffic between guests.
This seems to me the preferable way. We create a full virtual net device for each new container, and fully virtualize the device namespace.
Answers with respect to how I see layer 2 isolation, with network devices and sockets as well as the associated routing information given per namespace.
Quoted 7 lines
I have a few questions about all the network isolation stuff:
* What level of isolation is wanted for the network ? network devices ?
IPv4/IPv6 ? TCP/UDP ?* How is handled the incoming packets from the network ? I mean what will be
mecanism to dispatch the packet to the right virtual device ?
Wrong question. A better question is to ask how do you know which namespace a packet is in.
Answer: By looking at which device or socket it just came from.
How do you get a packet into a non-default namespace? Either you move a real network interface into that namespace. Or you use a tunnel device that shows up as two network interfaces in two different namespaces.
Then you route, or bridge packets between the two. Trivial.
Quoted 1 line
* How to handle the
SO_BINDTODEVICEsocket option ?
Just like we do now.
Quoted 1 line
* Has the virtual device a different MAC address ?
All network devices are abstractions of the hardware so they are all sort of virtual. My implementation of a tunnel device has a mac address so I can use it with ethernet bridging but that isn't a hard requirement. And yes the mac address is different because you can't do layer 2 switching if everyone has the same mac address.
But there is no special ``virtual'' device.
Quoted 1 line
How to manage it with the real MAC address on the system ?
Manage?
Quoted 1 line
How to manage ARP, ICMP, multicasting and IP ?
Like you always do. It would be a terrible implementation if we had to change that logic. There is a little bit of that where we need to detect which network namespace we are going to because the answers can differ but that is pretty straight forward.
Quoted 2 lines
It seems for me, IMHO that will require a lot of translation and browsing table. It will probably add a very significant overhead.
Then look at:git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/linux-2.6-ns.git#proof-of-concept
or the OpenVZ implementation.
It isn't serious overhead.
Quoted 1 line
* How to handle NFS access mounted outside of the container ?
The socket should remember it's network namespace. It works fine.
Quoted 1 line
* How to handle
ICMP_REDIRECT?
Just like we always do?
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Sam VilainQuoted 11 lines · Sam Vilain
Quoted 6 lines · jamal
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
That particular convention only works if you have network namespaces and UTS namespaces tightly bound.
that would be one approach. Another less sophisticated approach is to have no binding whatsoever, rather some translation table to map two unrelated devices.
Quoted 8 lines
We plan to have them separate - so for
that to work, each network namespace could have an arbitrary "prefix" that determines what the interface name will look like from the outside when combined. We'd have to be careful about length limits.And guest0-eth0 doesn't necessarily make sense; it's not really an ethernet interface, more like a tun or something.
it wouldnt quiet fit as a tun device. More like a mirror side of the guest eth0 created on the host side
i.e a sort of passthrough device with one side visible on the host (send from guest0-eth0 is received on eth0 in the guest and vice-versa).
Note this is radically different from what i have heard Andrey and co talk about and i dont wanna disturb any shit because there seems to be some agreement. But if you address me i respond because it is very interesting a topic;->
Quoted 7 lines
So, an equally good convention might be to use sequential prefixes on the host, like "tun", "dummy", or a new prefix - then a property of that is what the name of the interface is perceived to be to those who are in the corresponding network namespace.
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
on the host ip addr and the one seen on the guest side are the same. Except one is seen (on the host) on guest0-eth0 and another is seen on eth0 (on guest).
Anyways, ignore what i am saying if it is disrupting the discussion.
cheers,
jamal
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 31 lines · jamal
Quoted 11 lines · Sam Vilain
Quoted 6 lines · jamal
Makes sense for the host side to have naming convention tied to the guest. Example as a prefix: guest0-eth0. Would it not be interesting to have the host also manage these interfaces via standard tools like ip or ifconfig etc? i.e if i admin up guest0-eth0, then the user in guest0 will see its eth0 going up.
That particular convention only works if you have network namespaces and UTS namespaces tightly bound.
that would be one approach. Another less sophisticated approach is to have no binding whatsoever, rather some translation table to map two unrelated devices.
Quoted 8 lines
We plan to have them separate - so for
that to work, each network namespace could have an arbitrary "prefix" that determines what the interface name will look like from the outside when combined. We'd have to be careful about length limits.And guest0-eth0 doesn't necessarily make sense; it's not really an ethernet interface, more like a tun or something.
it wouldnt quiet fit as a tun device. More like a mirror side of the guest eth0 created on the host side
i.e a sort of passthrough device with one side visible on the host (send from guest0-eth0 is received on eth0 in the guest and vice-versa).
Quoted 4 lines
Note this is radically different from what i have heard Andrey and co talk about and i dont wanna disturb any shit because there seems to be some agreement. But if you address me i respond because it is very interesting a topic;->
thing is, we have several things we should care about and some of them 'look' or 'sound' similar, although they are not really ... I'll try to clarify
first, we want to have 'per guest' interfaces, which do not clash with any interfaces on the host or in other guests
then, we want to 'connect' them, implicitely or explicetly with 'other' interfaces or devices inside other guests or on the host, here we have the following cases (some are a little special):
- lo interface, guest and host private (by default)
- tap/tun interfaces, again host/guest private
- tun like interfaces between host and guests
- tun like interfaces between guests
- 'normal' interfaces mapped into guests
on the traffic side we have the following cases:
- local traffic on the host
- local traffic on the guest
- local traffic between host and guest
- local traffic between guests
- routed traffic from guest via host
- bridged traffic from guest via host
special cases here would be tun/tap traffic inside a guest, but that can be considered local too
Quoted 11 lines
Quoted 7 lines
So, an equally good convention might be to use sequential prefixes on the host, like "tun", "dummy", or a new prefix - then a property of that is what the name of the interface is perceived to be to those who are in the corresponding network namespace.
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
on the host ip addr and the one seen on the guest side are the same. Except one is seen (on the host) on guest0-eth0 and another is seen on eth0 (on guest).
this depends on the way the interfaces are handled and how they actually work, means:
if the interfaces _solely_ work via routing or bridging, then the 'host' end has to exist and be visible similar to 'normal' interfaces
if the traffic is (magically) mapped from guest interfaces to real (outside) host interfaces, we might want the same view as the guest has (i.e. basically a 'copy' which is not real)
Quoted 1 line
Anyways, ignore what i am saying if it is disrupting the discussion.
IMHO input is always welcome .. helps the folks to do better thinking :)
Quoted 5 lines
cheers,
jamal
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to jamalHi Jamal,
Quoted 1 line · jamal
On Fri, 2006-30-06 at 09:07 +1200, Sam Vilain wrote:
[snip]
Quoted 14 lines
Quoted 8 lines
We plan to have them separate - so for
that to work, each network namespace could have an arbitrary "prefix" that determines what the interface name will look like from the outside when combined. We'd have to be careful about length limits.And guest0-eth0 doesn't necessarily make sense; it's not really an ethernet interface, more like a tun or something.
it wouldnt quiet fit as a tun device. More like a mirror side of the guest eth0 created on the host side
i.e a sort of passthrough device with one side visible on the host (send from guest0-eth0 is received on eth0 in the guest and vice-versa).
Quoted 4 lines
Note this is radically different from what i have heard Andrey and co talk about and i dont wanna disturb any shit because there seems to be some agreement. But if you address me i respond because it is very interesting a topic;->
I do not have anything against guest-eth0 - eth0 pairs _if_ they are set up by the host administrators explicitly for some purpose.
For example, if these guest-eth0 and eth0 devices stay as pure virtual ones, i.e. they don't have any physical NIC, host administrator may route traffic to guestXX-eth0 interfaces to pass it to the guests.
However, I oppose the idea of automatic mirroring of _all_ devices appearing inside some namespaces ("guests") to another namespace (the "host"). This clearly goes against the concept of namespaces as independent realms, and creates a lot of problems with applications running in the host, hotplug scripts and so on.
Quoted 6 lines
Quoted 6 lines
So, an equally good convention might be to use sequential prefixes on the host, like "tun", "dummy", or a new prefix - then a property of that is what the name of the interface is perceived to be to those who are in the corresponding network namespace.
Quoted 6 lines
Quoted 2 lines
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
on the host ip addr and the one seen on the guest side are the same. Except one is seen (on the host) on guest0-eth0 and another is seen on eth0 (on guest).
Then what to do if the host system has 10.0.0.1 as a private address on eth3, and then interfaces guest1-tun0 and guest2-tun0 both get address 10.0.0.1 when each guest has added 10.0.0.1 to their tun0 device?
Regards,
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
Hi Andrey,
BTW - I was just looking at openvz, very impressive. To the other folks, I am not putting down any of your approaches - just havent had time to study them. Andrey, this is the same thing you guys have been working on for a few years now, you just changed the name, correct?
Ok, since you guys are encouraging me to speak, here goes ;-> Hopefully this addresses the other email from Herbert et al.
Quoted 4 lines · Andrey Savochkin
Hi Jamal,
Quoted 1 line · jamal
On Fri, 2006-30-06 at 09:07 +1200, Sam Vilain wrote:
Quoted 9 lines
[snip]
I do not have anything against guest-eth0 - eth0 pairs _if_ they are set up by the host administrators explicitly for some purpose.
For example, if these guest-eth0 and eth0 devices stay as pure virtual ones, i.e. they don't have any physical NIC, host administrator may route traffic to guestXX-eth0 interfaces to pass it to the guests.
Well there will be purely virtual of course. Something along the lines for openvz:
// create the guest
[host-node]# vzctl create 101 --ostemplate fedora-core-5-minimal // create guest101::eth0, seems to only create config to boot up with [host-node]# vzctl create 101 --netdev eth0
// bootup guest101
[host-node]# vzctl start 101
As soon as bootup of guest101 happens, creating guest101::eth0 should activate creation of the host side netdevice. This could be triggered for example by the netlink event message seen on host whic- which is a result of creating guest101::eth0 Which means control sits purely in user space.
at that point if i do ifconfig on host i see g101-eth0 on guest101 i see just name eth0.
My earlier suggestion was that instead of:
host-node]# vzctl set 101 --ipadd 10.1.2.3
you do:
host-node]# ip addr add g101-eth0 10.1.2.3/32 you should still use vzctl to save config for next bootup
Quoted 6 lines
However, I oppose the idea of automatic mirroring of _all_ devices appearing inside some namespaces ("guests") to another namespace (the "host"). This clearly goes against the concept of namespaces as independent realms, and creates a lot of problems with applications running in the host, hotplug scripts and so on.
I was thinking that the host side is the master i.e you can peek at namespaces in the guest from the host.
Also note that having the pass through device allows for guests to be connected via standard linux schemes in the host side (bridge, point routes, tc redirect etc); so you dont need a speacial device to hook them together.
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
on the host ip addr and the one seen on the guest side are the same. Except one is seen (on the host) on guest0-eth0 and another is seen on eth0 (on guest).
Then what to do if the host system has 10.0.0.1 as a private address on eth3, and then interfaces guest1-tun0 and guest2-tun0 both get address 10.0.0.1 when each guest has added 10.0.0.1 to their tun0 device?
Yes, that would be a conflict that needs to be resolved. If you look at ip addresses as also belonging to namespaces, then it should work, no? i am assuming a tag at the ifa table level.
cheers,
jamal
Re: [patch 2/6] [Network namespace] Network device sharing by view
Jamal,
Quoted 2 lines · jamal
BTW - I was just looking at openvz, very impressive. To the other folks,
Thanks!
Quoted 3 lines
I am not putting down any of your approaches - just havent had time to study them. Andrey, this is the same thing you guys have been working on for a few years now, you just changed the name, correct?
The relations are more complicated than just the change of name, but yes, OpenVZ represents the result of our work for a few years.
Quoted 4 lines
Ok, since you guys are encouraging me to speak, here goes ;-> Hopefully this addresses the other email from Herbert et al.
[snip]
Quoted 11 lines
// create the guest
[host-node]# vzctl create 101 --ostemplate fedora-core-5-minimal // createguest101::eth0, seems to only create config to boot up with [host-node]# vzctl create 101 --netdev eth0
// bootup guest101
[host-node]# vzctl start 101As soon as bootup of guest101 happens, creating
guest101::eth0should activate creation of the host side netdevice. This could be triggered for example by the netlink event message seen on host whic- which is a result of creatingguest101::eth0Which means control sits purely in user space.
I'd like to clarify you idea: whether this host-side device is a real device capable of receiving and transmitting packets (by moving them between namespaces), or it's a fake device creating only a view of other namespace's devices?
[snip]
Quoted 9 lines
Quoted 6 lines
However, I oppose the idea of automatic mirroring of _all_ devices appearing inside some namespaces ("guests") to another namespace (the "host"). This clearly goes against the concept of namespaces as independent realms, and creates a lot of problems with applications running in the host, hotplug scripts and so on.
I was thinking that the host side is the master i.e you can peek at namespaces in the guest from the host.
"Host(master)-guest" relations is a valid and useful scheme. However, I'm thinking about broader application of network namespaces, when they can form an arbitrary tree and may not be in "host-guest" relations.
Quoted 4 lines
Also note that having the pass through device allows for guests to be connected via standard linux schemes in the host side (bridge, point routes, tc redirect etc); so you dont need a speacial device to hook them together.
What do you mean under pass through device?
Do you mean using guest1-tun0 as a backdoor to talk to the guest?
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
on the host ip addr and the one seen on the guest side are the same. Except one is seen (on the host) on guest0-eth0 and another is seen on eth0 (on guest).
Quoted 8 lines
Quoted 4 lines
Then what to do if the host system has 10.0.0.1 as a private address on eth3, and then interfaces guest1-tun0 and guest2-tun0 both get address 10.0.0.1 when each guest has added 10.0.0.1 to their tun0 device?
Yes, that would be a conflict that needs to be resolved. If you look at ip addresses as also belonging to namespaces, then it should work, no? i am assuming a tag at the ifa table level.
I'm not sure, it's complicated.
You wouldn't want automatic local routes to be added for IP addresses on the host-side interfaces, right?
Do you expect these IP addresses to act as local addresses in other places, like answering to arp requests about these IP on all physical devices?
But anyway, you'll have conflicts on the application level.
Many programs like ntpd, bind, and others fetch the device list using the same ioctls as ifconfig, and make (un)intelligent decisions basing on what they see.
Mirroring may have some advantages if I am both host and guest administrator. But if I create a namespace for my friend Joe to play with IPv6 and sit tunnels, why should I face inconveniences because of what he does there?
Best regards
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to jamalQuoted 7 lines · jamal
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Then the pragmatic question becomes how to correlate what you see from `ip addr list' to guests.
on the host ip addr and the one seen on the guest side are the same. Except one is seen (on the host) on guest0-eth0 and another is seen on eth0 (on guest).
Quoted 7 lines
Quoted 3 lines
Then what to do if the host system has 10.0.0.1 as a private address on eth3, and then interfaces guest1-tun0 and guest2-tun0 both get address 10.0.0.1 when each guest has added 10.0.0.1 to their tun0 device?
Yes, that would be a conflict that needs to be resolved. If you look at ip addresses as also belonging to namespaces, then it should work, no? i am assuming a tag at the ifa table level.
Yes. The conception is that everything belongs to the namespace, so it looks like you have multiple instances of the network stack.
Which means through existing interfaces it would be a real problem if a network device showed up in more than one network stack as that would confuse things.
Basically the reading and configuration through existing interfaces is expected to be in the namespace as well which is where the difficulty shows up.
When you get serious about splitting up roots powers this becomes a real advantage. Because you might want to have one person responsible for what would normally be eth0 and another person responsible for eth1.
Anyway Jamal can you see the problem the aliases present to the implementation?
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 3 lines · Eric W. Biederman
Anyway Jamal can you see the problem the aliases present to the implementation?
I think more than anything i may have a different view of things and no code ;-> And you are trying to restore order in the discussion - so my wild ideas dont help. If you guys have a meeting somewhere like this coming OLS I will come over and disrupt your meeting ;-> I actually have attempted to implement things like virtual routers but you guys seem way ahead of anywhere i was.
cheers,
jamal
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 7 lines · jamal
Quoted 2 lines · Eric W. Biederman
Anyway Jamal can you see the problem the aliases present to the
implementation?
Quoted 1 line
Quoted 6 lines
I think more than anything i may have a different view of things and no code ;-> And you are trying to restore order in the discussion - so my wild ideas dont help. If you guys have a meeting somewhere like this coming OLS I will come over and disrupt your meeting ;-> I actually have attempted to implement things like virtual routers but you guys seem way ahead of anywhere i was.
Currently I think we have both a talk, and a BOFH at OLS plus probably a little time at kernel summit.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Herbert PoetzlQuoted 5 lines · Herbert Poetzl
last time I pointed to such 'misguided' apps which made assumptions that are not necessarily true inside a virtual environment (e.g. pstree, initpid) the general? position was that those apps should be fixed instead adding a 'workaround'
I agree that if it was solely misguided apps. There would be no justification.
One of the standard applications interfaces we support is renaming a network interface. So supporting those misguided apps is a actually a side effect of supporting one the standard operations on a network interface.
Another way of looking at it is that the names of networking devices like the names of devices in /dev are a user space policy (today). In the configuration of the networking stack historically we had those identifiers hard coded. It isn't until just recently that user space has been able to cope with dynamically added/removed network devices.
As for initpid and friends. In the context where you are simply isolating pids and not doing a full pid namespaces it was felt that changing the few user space applications that care was easier and probably worth doing anyway.
Quoted 5 lines
note: personally I'm absolutely not against virtualizing the device names so that each guest can have a separate name space for devices, but there should be a way to 'see' _and_ 'identify' the interfaces from outside (i.e. host or spectator context)
Yep. Basically that interface comes when we fix the sysfs support, to support per namespace reporting.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Dave HansenQuoted 13 lines · Dave Hansen
Quoted 4 lines · Herbert Poetzl
seriously, what I think Eric meant was that it might be nice (especially for migration purposes) to keep the device namespace completely virtualized and not just isolated ...
It might be nice, but it is probably unneeded for an initial implementation. In practice, a cluster doing checkpoint/restart/migration will already have a system in place for assigning unique IPs or other identifiers to each container. It could just as easily make sure to assign unique network device names to containers.
Quoted 6 lines
The issues really only come into play when you have an unstructured set of machines and you want to migrate between them without having prepared them with any kind of unique net device names beforehand.
It may look weird, but do application really *need* to see eth0 rather than eth858354?
Actually there is a very practical reason we don't need to preserve device names across a migration event between machines, is the only sane thing to do is to generate a hotplug event that says you have removed the old interface and added a new interface.
My expectation is that during migration you will wind up with add and remove events for all of your hardware devices. But most applications because they do not access hardware devices directly will not care.
I haven't looked closely but I suspect this is one area where a container style approach will be noticeably different from a Xen or Vmware style approach.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Herbert PoetzlQuoted 9 lines · Herbert Poetzl
- loopback traffic inside a guest is insignificantly slower than on a normal system
- loopback traffic on the host is insignificantly slower than on a normal system
- inter guest traffic is faster than on-wire traffic, and should be withing a small tolerance of the loopback case (as it really isn't different)
I do not follow what are you people arguing about?
Intra-guest, guest-guest and host-guest paths have _no_ differences from host-host loopback. Only the device is different:
* virtual loopback for intra-guest
* virtual interface for guest-guest and host-guest
But the work is exactly the same, only the place where packets looped back is different. How could this be issue to break a lance over? :-)
Alexey
PS. The only thing, which I can imagine is "optimized" out ip_route_input() in the case of loopback. But this optimization was an obvious design mistake (mine, sorry) and apparently will die together with removal of current deficiences of routing cache. Actually, it is one of deficiences.
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Daniel LezcanoDaniel,
Quoted 7 lines · Daniel Lezcano
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
Quoted 7 lines
Quoted 5 lines
Routing is everything.
For example, I want namespaces to have their private tunnel devices. It means that namespaces should be allowed have private routes of local type, private default routes, and so on...Ok, we are talking about the same things. We do it only in a different way:
We are not talking about the same things.
It isn't a technical thing whether route lookup is performed before or after namespace change.
It is a fundamental question determining functionality of network namespaces. We are talking about the capabilities namespaces provide.
Your proposal essentially denies namespaces to have their own tunnel or other devices. There is no point in having a device inside a namespace if the namespace owner can't route all or some specific outgoing packets through that device. You don't allow system administrators to completely delegate management of network configuration to namespace owners.
Andrey
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Daniel LezcanoQuoted 6 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
it is impossible to have bridged networking, tun/tap and many other features without it. I even doubt that it is possible to introduce private netfilter rules w/o virtualization of routing.
The question is do we want to have fully featured namespaces which allow to create isolated virtual environments with semantics and behaviour of standalone linux box or do we want to introduce some hacks with new rules/restrictions to meet ones goals only?
From my POV, fully virtualized namespaces are the future. It is what makes virtualization solution usable (w/o apps modifications), provides all the features and doesn't require much efforts from people to be used.
Thanks,
Kirill
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 10 lines · Kirill Korotaev
Quoted 6 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
it is impossible to have bridged networking, tun/tap and many other features without it. I even doubt that it is possible to introduce private netfilter rules w/o virtualization of routing.
why? iptables work quite fine on a typical linux system when you 'delegate' certain functionality to certain chains (i.e. doesn't require access to _all_ of them)
Quoted 4 lines
The question is do we want to have fully featured namespaces which allow to create isolated virtual environments with semantics and behaviour of standalone linux box or do we want to introduce some hacks with new rules/restrictions to meet ones goals only?
well, soemtimes 'hacks' are not only simpler but also a much better solution for a given problem than the straight forward approach ...
for example, you won't have multiple routing tables in a kernel where this feature is disabled, no? so why should it affect a guest, or require modified apps inside a guest when we would decide to provide only a single routing table?
Quoted 1 line
From my POV, fully virtualized namespaces are the future.
the future is already there, it's called Xen or UML, or QEMU :)
Quoted 3 lines
It is what makes virtualization solution usable (w/o apps modifications), provides all the features and doesn't require much efforts from people to be used.
and what if they want to use virtualization inside their guests? where do you draw the line?
best,
Herbert
Quoted 2 lines
Thanks,
Kirill
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 17 lines · Herbert Poetzl
Quoted 10 lines · Kirill Korotaev
Quoted 6 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
it is impossible to have bridged networking, tun/tap and many other features without it. I even doubt that it is possible to introduce private netfilter rules w/o virtualization of routing.
why? iptables work quite fine on a typical linux system when you 'delegate' certain functionality to certain chains (i.e. doesn't require access to _all_ of them)
Quoted 8 lines
Quoted 4 lines
The question is do we want to have fully featured namespaces which allow to create isolated virtual environments with semantics and behaviour of standalone linux box or do we want to introduce some hacks with new rules/restrictions to meet ones goals only?
well, soemtimes 'hacks' are not only simpler but also a much better solution for a given problem than the straight forward approach ...
Well I would like to see a hack that qualifies. I watched the linux-vserver irc channel for a while and almost every network problem was caused by the change in semantics vserver provides.
In this case when you allow a guest more than one IP your hack while easy to maintain becomes much more complex. Especially as you address each case people care about one at a time.
In one shot this goes the entire way. Given how many people miss that you do the work at layer 2 than at layer 3 I would not call this the straight forward approach. The straight forward implementation yes, but not the straight forward approach.
Quoted 9 lines
for example, you won't have multiple routing tables in a kernel where this feature is disabled, no? so why should it affect a guest, or require modified apps inside a guest when we would decide to provide only a single routing table?
Quoted 1 line
From my POV, fully virtualized namespaces are the future.
the future is already there, it's called Xen or UML, or QEMU :)
Yep. And now we need it to run fast.
Quoted 6 lines
Quoted 3 lines
It is what makes virtualization solution usable (w/o apps modifications), provides all the features and doesn't require much efforts from people to be used.
and what if they want to use virtualization inside their guests? where do you draw the line?
The implementation doesn't have any problems with guests inside of guests.
The only reason to restrict guests inside of guests is because the we aren't certain which permissions make sense.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 28 lines · Eric W. Biederman
Quoted 17 lines · Herbert Poetzl
Quoted 10 lines · Kirill Korotaev
Quoted 6 lines
Quoted 3 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
it is impossible to have bridged networking, tun/tap and many other features without it. I even doubt that it is possible to introduce private netfilter rules w/o virtualization of routing.
why? iptables work quite fine on a typical linux system when you 'delegate' certain functionality to certain chains (i.e. doesn't require access to _all_ of them)
Quoted 8 lines
Quoted 4 lines
The question is do we want to have fully featured namespaces which allow to create isolated virtual environments with semantics and behaviour of standalone linux box or do we want to introduce some hacks with new rules/restrictions to meet ones goals only?
well, soemtimes 'hacks' are not only simpler but also a much better solution for a given problem than the straight forward approach ...
Quoted 5 lines
Well I would like to see a hack that qualifies.
I watched the linux-vserver irc channel for a while and almost every network problem was caused by the change in semantics vserver provides.
the problem here is not the change in semantics compared to a real linux system (as there basically is none) but compared to _other_ technologies like UML or QEMU, which add the need for bridging and additional interfaces, while Linux-VServer only focuses on the IP layer ...
Quoted 2 lines
In this case when you allow a guest more than one IP your hack while easy to maintain becomes much more complex.
why? a set of IPs is quite similar to a single IP (which is actually a subset), so no real change there, only IP_ANY means something different for a guest ...
Quoted 1 line
Especially as you address each case people care about one at a time.
hmm?
Quoted 4 lines
In one shot this goes the entire way. Given how many people miss that you do the work at layer 2 than at layer 3 I would not call this the straight forward approach. The straight forward implementation yes, but not the straight forward approach.
seems I lost you here ...
Quoted 11 lines
Quoted 9 lines
for example, you won't have multiple routing tables in a kernel where this feature is disabled, no? so why should it affect a guest, or require modified apps inside a guest when we would decide to provide only a single routing table?
Quoted 1 line
From my POV, fully virtualized namespaces are the future.
the future is already there, it's called Xen or UML, or QEMU :)
Yep. And now we need it to run fast.
hmm, maybe you should try to optimize linux for Xen then, as I'm sure it will provide the optimal virtualization and has all the features folks are looking for (regarding virtualization)
I thought we are trying to figure a light-weight subset of isolation and virtualization technologies and methods which make sense to have in mainline ...
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
It is what makes virtualization solution usable (w/o apps modifications), provides all the features and doesn't require much efforts from people to be used.
Quoted 8 lines
Quoted 2 lines
and what if they want to use virtualization inside their guests? where do you draw the line?
The implementation doesn't have any problems with guests inside of guests.
The only reason to restrict guests inside of guests is because the we aren't certain which permissions make sense.
well, we have not even touched the permission issues yet
best,
Herbert
Quoted 1 line
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 7 lines · Herbert Poetzl
Quoted 5 lines · Eric W. Biederman
Herbert Poetzl <[email protected]> writes:
I watched the linux-vserver irc channel for a while and almost every network problem was caused by the change in semantics vserver provides.
Quoted 5 lines
the problem here is not the change in semantics compared to a real linux system (as there basically is none) but compared to _other_ technologies like UML or QEMU, which add the need for bridging and additional interfaces, while Linux-VServer only focuses on the IP layer ...
Not being able to bind to INADDR_ANY is a huge semantic change. Unless things have changed recently you get that change when you have two IP addresses in Linux-Vserver.
Talking to the outsider world through the loop back interface is a noticeable semantics change.
Having to be careful of who uses INADDR_ANY on the host when you have guests is essentially a semantics change.
Being able to talk to the outside world with a server bound only to the loopback IP is a weird semantic change.
And I suspect I missed something, it is weird peculiar and I don't care to remember all of the exceptions.
Have a few more network interfaces for a layer 2 solution is fundamental. Believing without proof and after arguments to the contrary that you have not contradicted that a layer 2 solution is inherently slower is non-productive. Arguing that a layer 2 only solution most prove itself on guest to guest communication is also non-productive.
So just to sink one additional nail in the coffin of the silly guest to guest communication issue. For any two guests where fast communication between them is really important I can run an additional interface pair that requires no routing or bridging. Given that the implementation of the tunnel device is essentially the same as the loopback interface and that I make only one trip through the network stack there will be no performance overhead. Similarly for any critical guest communication to the outside world I can give the guest a real network adapter.
That said I don't think those things will be necessary and that if they are it is an optimization opportunity to make various bits of the network stack faster.
Bridging or routing between guests is an exercise in simplicity and control not a requirement.
Quoted 6 lines
Quoted 2 lines
In this case when you allow a guest more than one IP your hack while easy to maintain becomes much more complex.
why? a set of IPs is quite similar to a single IP (which is actually a subset), so no real change there, only
IP_ANYmeans something different for a guest ...
Which simply filtering at bind time makes impossible.
With a guest with 4 IPs
10.0.0.1 192.168.0.1 172.16.0.1 127.0.0.1
How do you make INADDR_ANY work with just filtering at bind time?
The host has at least the additional IPs.
10.0.0.2 192.168.0.2 172.16.0.2 127.0.0.1
Herbert I suspect we are talking about completely different implementations otherwise I can't possibly see how we have such different perceptions of their capabilities.
I am talking precisely about filter IP addresses at connect or bind time that a guest can use. Which as I recall is what vserver implements. If you are thinking of your ngnet implementation that would explain things.
Quoted 3 lines
Quoted 1 line
Especially as you address each case people care about one at a time.
hmm?
Multiple IPs, IPv6, additional protocols, firewalls. etc.
Quoted 20 lines
Quoted 4 lines
In one shot this goes the entire way. Given how many people miss that you do the work at layer 2 than at layer 3 I would not call this the straight forward approach. The straight forward implementation yes, but not the straight forward approach.
seems I lost you here ...
Quoted 11 lines
Quoted 9 lines
for example, you won't have multiple routing tables in a kernel where this feature is disabled, no? so why should it affect a guest, or require modified apps inside a guest when we would decide to provide only a single routing table?
Quoted 1 line
From my POV, fully virtualized namespaces are the future.
the future is already there, it's called Xen or UML, or QEMU :)
Yep. And now we need it to run fast.
Quoted 8 lines
hmm, maybe you should try to optimize linux for Xen then, as I'm sure it will provide the optimal virtualization and has all the features folks are looking for (regarding virtualization)
I thought we are trying to figure a light-weight subset of isolation and virtualization technologies and methods which make sense to have in mainline ...
And you presume doing things at layer 2 is more expensive than layer 3.
Quoted 1 line
From what I have seen of layer 3 solutions it is a
bloody maintenance nightmare, and an inflexible mess.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It is what makes virtualization solution usable (w/o apps modifications), provides all the features and doesn't require much efforts from people to be used.
and what if they want to use virtualization inside their guests? where do you draw the line?
Quoted 7 lines
Quoted 5 lines
The implementation doesn't have any problems with guests inside of guests.
The only reason to restrict guests inside of guests is because the we aren't certain which permissions make sense.
well, we have not even touched the permission issues yet
Agreed, permissions have not discussed but the point is that is the only reason to keep from nesting the networking stack the way I have described it.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 7 lines · Eric W. Biederman
Have a few more network interfaces for a layer 2 solution is fundamental. Believing without proof and after arguments to the contrary that you have not contradicted that a layer 2 solution is inherently slower is non-productive. Arguing that a layer 2 only solution most prove itself on guest to guest communication is also non-productive.
Yes, it does break what some people consider to be a sanity condition when you don't have loopback anymore within a guest. I once experimented with using 127.* addresses for per-guest loopback devices with vserver to fix this, but that couldn't work without fixing glibc to not make assumptions deep in the bowels of the resolver. I logged a fault with gnu.org and you can guess where it went :-).
I don't think it's just the performance issue, though. Consider also that if you only have one set of interfaces to manage, the overall configuration of the network stack is simpler. `ip addr list' on the host shows all the addresses on the system, you only have one routing table to manage, one set of iptables, etc.
That being said, perhaps if each guest got its own interface, and from some suitably privileged context you could see them all, perhaps it would be nicer and maybe just as fast. Perhaps then *devices* could get their own routing namespaces, and routing namespaces could get iptables namespaces, or something like that, to give the most options.
Quoted 4 lines
With a guest with 4 IPs
10.0.0.1 192.168.0.1 172.16.0.1 127.0.0.1
How do you makeINADDR_ANYwork with just filtering at bind time?
It used to just bind to the first one. Don't know if it still does.
Sam.
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 9 lines · Sam Vilain
Quoted 7 lines · Eric W. Biederman
Have a few more network interfaces for a layer 2 solution is fundamental. Believing without proof and after arguments to the contrary that you have not contradicted that a layer 2 solution is inherently slower is non-productive. Arguing that a layer 2 only solution most prove itself on guest to guest communication is also non-productive.
Quoted 6 lines
Yes, it does break what some people consider to be a sanity condition when you don't have loopback anymore within a guest. I once experimented with using 127.* addresses for per-guest loopback devices with vserver to fix this, but that couldn't work without fixing glibc to not make assumptions deep in the bowels of the resolver. I logged a fault with gnu.org and you can guess where it went :-).
this is what the lo* patches address, by providing the required loopback isolation and providing lo inside a guest (i.e. it looks and feels like a normal system, except that you cannot modify the interfaces from inside)
Quoted 12 lines
I don't think it's just the performance issue, though. Consider also that if you only have one set of interfaces to manage, the overall configuration of the network stack is simpler. `ip addr list' on the host shows all the addresses on the system, you only have one routing table to manage, one set of iptables, etc.
That being said, perhaps if each guest got its own interface, and from some suitably privileged context you could see them all, perhaps it would be nicer and maybe just as fast. Perhaps then *devices* could get their own routing namespaces, and routing namespaces could get iptables namespaces, or something like that, to give the most options.
Quoted 6 lines
Quoted 4 lines
With a guest with 4 IPs
10.0.0.1 192.168.0.1 172.16.0.1 127.0.0.1
How do you makeINADDR_ANYwork with just filtering at bind time?It used to just bind to the first one. Don't know if it still does.
no, it _alway_ binds to INADDR_ANY and checks against other sockets (in the same context) comparing the lists of assigned IPs (the subset)
so all checks happen at bind/connect time and always against the set of IPs, only exception is a performance optimization we do for single IP guests (where INADDR_ANY gets rewritten to the single IP)
best,
Herbert
Quoted 1 line
Sam.
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 17 lines · Herbert Poetzl
Quoted 9 lines · Sam Vilain
Quoted 7 lines · Eric W. Biederman
Have a few more network interfaces for a layer 2 solution is fundamental. Believing without proof and after arguments to the contrary that you have not contradicted that a layer 2 solution is inherently slower is non-productive. Arguing that a layer 2 only solution most prove itself on guest to guest communication is also non-productive.
Quoted 6 lines
Yes, it does break what some people consider to be a sanity condition when you don't have loopback anymore within a guest. I once experimented with using 127.* addresses for per-guest loopback devices with vserver to fix this, but that couldn't work without fixing glibc to not make assumptions deep in the bowels of the resolver. I logged a fault with gnu.org and you can guess where it went :-).
Quoted 5 lines
this is what the lo* patches address, by providing the required loopback isolation and providing lo inside a guest (i.e. it looks and feels like a normal system, except that you cannot modify the interfaces from inside)
Ok. This is new. How do you talk between guests now?
Before those patches it was through IP addresses on the loopback interface as I recall.
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
With a guest with 4 IPs
10.0.0.1 192.168.0.1 172.16.0.1 127.0.0.1
How do you makeINADDR_ANYwork with just filtering at bind time?It used to just bind to the first one. Don't know if it still does.
no, it _alway_ binds to
INADDR_ANYand checks against other sockets (in the same context) comparing the lists of assigned IPs (the subset)
Quoted 5 lines
so all checks happen at bind/connect time and always against the set of IPs, only exception is a performance optimization we do for single IP guests (where
INADDR_ANYgets rewritten to the single IP)
What is the mechanism there?
My rough extrapolation says this mechanism causes problems when migrating between machines. In particular it sounds like only one process can bind to *:80, even if it is only allowed to accept connections from a subset of those IPs.
So if on another machine I bound something to *:80 and only allowed to use a different set of IPs and then attempted to migrate it, the migration would fail because I could not restart the application, with all of it's layer 3 resources.
To be clear I assume when I migrate I always take my IP address or addresses with me.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 26 lines · Eric W. Biederman
Quoted 18 lines · Herbert Poetzl
Quoted 9 lines · Sam Vilain
Quoted 7 lines · Eric W. Biederman
Have a few more network interfaces for a layer 2 solution is fundamental. Believing without proof and after arguments to the contrary that you have not contradicted that a layer 2 solution is inherently slower is non-productive. Arguing that a layer 2 only solution most prove itself on guest to guest communication is also non-productive.
Quoted 7 lines
Yes, it does break what some people consider to be a sanity condition when you don't have loopback anymore within a guest. I once experimented with using 127.* addresses for per-guest loopback devices with vserver to fix this, but that couldn't work without fixing glibc to not make assumptions deep in the bowels of the resolver. I logged a fault with gnu.org and you can guess where it went :-).
Quoted 5 lines
this is what the lo* patches address, by providing the required loopback isolation and providing lo inside a guest (i.e. it looks and feels like a normal system, except that you cannot modify the interfaces from inside)
Quoted 4 lines
Ok. This is new. How do you talk between guests now?
Before those patches it was through IP addresses on the loopback interface as I recall.
no, that was probably your assumption, the IPs are assigned (in a perfectly normal way) to the interfaces (e.g. eth0 carries some IPs for guest A and B, eth1 carries others for guest C). the way the linux network stack works, local addresses (i.e. those of A,B and C) will automatically communicate via loopback (as they are local) while outbound traffic will use the proper interface (nothing is changed here)
the difference in the lo patches is, that we allow to use the 'localhost' ip range (127.x.x.x) by isolating traffic (in this range) on the loopback interface (which typically allows to have 127.0.0.1 and lo visible inside a guest)
Quoted 17 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
With a guest with 4 IPs
10.0.0.1 192.168.0.1 172.16.0.1 127.0.0.1
How do you makeINADDR_ANYwork with just filtering at bind time?It used to just bind to the first one. Don't know if it still does.
no, it _alway_ binds to
INADDR_ANYand checks against other sockets (in the same context) comparing the lists of assigned IPs (the subset)Quoted 5 lines
so all checks happen at bind/connect time and always against the set of IPs, only exception is a performance optimization we do for single IP guests (where
INADDR_ANYgets rewritten to the single IP)
Quoted 4 lines
What is the mechanism there?
My rough extrapolation says this mechanism causes problems when migrating between machines.
that might be, as we do not consider migration such important as other folks do :)
Quoted 3 lines
In particular it sounds like only one process can bind to *:80, even if it is only allowed to accept connections from a subset of those IPs.
no, guest A,B and C can all bind to *:80 and coexist quite fine, given that they do not have any IP in the intersection of their subsets (which is checked at bind time)
Quoted 4 lines
So if on another machine I bound something to *:80 and only allowed to use a different set of IPs and then attempted to migrate it, the migration would fail because I could not restart the application, with all of it's layer 3 resources.
actually I do not see why, unless the destination has a conflict on the ip subset, in which case you would end up with a migrated, but not working guest :)
Quoted 2 lines
To be clear I assume when I migrate I always take my IP address or addresses with me.
that's fine, the only requirement would be that the host has a superset of the IP addresses used by the guests ...
HTC,
Herbert
Quoted 1 line
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Eric W. BiedermanHi !
Eric W. Biederman wrote:
[ ... ]
Quoted 10 lines
So just to sink one additional nail in the coffin of the silly guest to guest communication issue. For any two guests where fast communication between them is really important I can run an additional interface pair that requires no routing or bridging. Given that the implementation of the tunnel device is essentially the same as the loopback interface and that I make only one trip through the network stack there will be no performance overhead. Similarly for any critical guest communication to the outside world I can give the guest a real network adapter.
Quoted 3 lines
That said I don't think those things will be necessary and that if they are it is an optimization opportunity to make various bits of the network stack faster.
just one comment on the 'guest to guest communication' topic :
guest to guest communication is an important factor in consolidation scenarios, where containers are packed on one server. This for maintenance issues or priority issues on a HPC cluster for example. This case of container migration is problably the most interesting and the performance should be more than acceptable. May be not a top priority for the moment.
thanks,
C.
Re: [patch 2/6] [Network namespace] Network device sharing by view
In reply to Eric W. BiedermanQuoted 15 lines · Eric W. Biederman
Quoted 7 lines · Herbert Poetzl
Quoted 5 lines · Eric W. Biederman
Herbert Poetzl <[email protected]> writes:
I watched the linux-vserver irc channel for a while and almost every network problem was caused by the change in semantics vserver provides.
Quoted 5 lines
the problem here is not the change in semantics compared to a real linux system (as there basically is none) but compared to _other_ technologies like UML or QEMU, which add the need for bridging and additional interfaces, while Linux-VServer only focuses on the IP layer ...
Quoted 3 lines
Not being able to bind to
INADDR_ANYis a huge semantic change. Unless things have changed recently you get that change when you have two IP addresses in Linux-Vserver.
not at all, you probably looked at a different code, binding to INADDR_ANY actually _is_ the default inside a guest, the only difference here is that INADDR_ANY maps to a subset of _all_ available IPs ...
Quoted 2 lines
Talking to the outsider world through the loop back interface is a noticeable semantics change.
this does not happen either, as I said several times, networking happens as on a _normal_ linux system, local traffic uses loop back, while outbound traffic uses the appropriate network interfaces
Quoted 2 lines
Having to be careful of who uses
INADDR_ANYon the host when you have guests is essentially a semantics change.
this 'semantic' change is intentional, and it would be quite easy to change that (by putting the host in a network context too) but as the mechanism is isolation the host, similar to the chroot() semantic for filesystems, sees _all_ the interfaces and IPs and therefore can also bind to all of them ...
Quoted 3 lines
Being able to talk to the outside world with a server bound only to the loopback IP is a weird semantic change.
that does not happen either ...
IMHO you should have a closer look (or ask more questions) before making false assumptions
Quoted 2 lines
And I suspect I missed something, it is weird peculiar and I don't care to remember all of the exceptions.
there are no real exceptions, we have a legacy mapping which basically 'remaps' localhost to the first assigned IP (to make guest local traffic secure without messing with the network stack) but this can be avoided completely
Quoted 4 lines
Have a few more network interfaces for a layer 2 solution is fundamental. Believing without proof and after arguments to the contrary that you have not contradicted that a layer 2 solution is inherently slower is non-productive.
assuming that it will not be slower, although it will now pass two network stacks and the bridging code is non-productive too, let's see how it goes but do not ignore the overhead just because it might simplify the implementation ...
Quoted 10 lines
Arguing that a layer 2 only solution most prove itself on guest to guest communication is also non-productive.
So just to sink one additional nail in the coffin of the silly guest to guest communication issue. For any two guests where fast communication between them is really important I can run an additional interface pair that requires no routing or bridging. Given that the implementation of the tunnel device is essentially the same as the loopback interface and that I make only one trip through the network stack there will be no performance overhead.
that is a good argument and I think I'm perfectly fine with this, given that the implementation allows that (i.e. the network stack can handle two interfaces with the same IP assigned and will choose the local interface over the remote one when the traffic will be between guests)
Quoted 2 lines
Similarly for any critical guest communication to the outside world I can give the guest a real network adapter.
with a single MAC assigned, that is, I presume?
Quoted 20 lines
That said I don't think those things will be necessary and that if they are it is an optimization opportunity to make various bits of the network stack faster.
Bridging or routing between guests is an exercise in simplicity and control not a requirement.
Quoted 6 lines
Quoted 2 lines
In this case when you allow a guest more than one IP your hack while easy to maintain becomes much more complex.
why? a set of IPs is quite similar to a single IP (which is actually a subset), so no real change there, only
IP_ANYmeans something different for a guest ...Which simply filtering at bind time makes impossible.
With a guest with 4 IPs
10.0.0.1 192.168.0.1 172.16.0.1 127.0.0.1
How do you makeINADDR_ANYwork with just filtering at bind time?
Quoted 6 lines
The host has at least the additional IPs.
10.0.0.2 192.168.0.2 172.16.0.2 127.0.0.1Herbert I suspect we are talking about completely different implementations otherwise I can't possibly see how we have such different perceptions of their capabilities.
guess that's what this discussion is about, finding out the various aspects how isolation and/or vitrtualization can be accomplished and what features we consider common/useful enough for mainline ... for me that is still in the brainstorming phase, although several 'working prototypes' already exist. IMHO the next step is to collect a set of representative use cases and test them with each implementation, regarding performance, usability and practicability
Quoted 4 lines
I am talking precisely about filter IP addresses at connect or bind time that a guest can use. Which as I recall is what vserver implements. If you are thinking of your ngnet implementation that would explain things.
I'm thinking of all the various implementations and 'prototypes' we did and tested, I agree this might be confusing ...
Quoted 31 lines
Quoted 3 lines
Quoted 1 line
Especially as you address each case people care about one at a time.
hmm?
Multiple IPs, IPv6, additional protocols, firewalls. etc.
Quoted 25 lines
Quoted 4 lines
In one shot this goes the entire way. Given how many people miss that you do the work at layer 2 than at layer 3 I would not call this the straight forward approach. The straight forward implementation yes, but not the straight forward approach.
seems I lost you here ...
Quoted 11 lines
Quoted 9 lines
for example, you won't have multiple routing tables in a kernel where this feature is disabled, no? so why should it affect a guest, or require modified apps inside a guest when we would decide to provide only a single routing table?
Quoted 1 line
From my POV, fully virtualized namespaces are the future.
the future is already there, it's called Xen or UML, or QEMU :)
Yep. And now we need it to run fast.
hmm, maybe you should try to optimize linux for Xen then, as I'm sure it will provide the optimal virtualization and has all the features folks are looking for (regarding virtualization)
Quoted 6 lines
Quoted 3 lines
I thought we are trying to figure a light-weight subset of isolation and virtualization technologies and methods which make sense to have in mainline ...
And you presume doing things at layer 2 is more expensive than layer 3.
not necessarily, but I _know_ that the overhead added at layer 3 is unmeasureable, and it still needs to be proven that this is true for a layer 2 solution (which I'd actually prefer, because it solves the protocol _and_ setup issues)
Quoted 2 lines
Quoted 1 line
From what I have seen of layer 3 solutions it is a
bloody maintenance nightmare, and an inflexible mess.
that is your opinion, I really doubt that you will have less maintenance when you apply policy to the guests ...
example here (just to clarify):
- let's assume we have eth0 on the host and in guest A and B, with the following setup:
eth0(H) 192.168.0.1/24
eth0(A) 10.0.1.1/16 10.0.1.2/16
eth0(B) 10.0.2.1/16
- now what keeps guest B from jsut assigning 10.0.2.2/16 to eth0? you need some kind of mechanism to prevent that, and/or to block the packets using inappropriate IPs
* in the first case, i.e. you prevent assigning certain IPs inside a guest, you get a semantic change in the behaviour compared to a normal system, but there is no additional overhead on the communication
* in the second case, you have to maintain the policy mechanism and keep it in sync with the guest configuration (somehow), and of course you have to verify every communication
- OTOH, if you do not care about collisions basically assuming the point "that's like a hub on a network, if there are two guests with the same ip, it will be trouble, but that's okay" then this becomes a real issue for providers with potentially 'evil' customers
best,
Herbert
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It is what makes virtualization solution usable (w/o apps modifications), provides all the features and doesn't require much efforts from people to be used.
and what if they want to use virtualization inside their guests? where do you draw the line?
Quoted 7 lines
Quoted 5 lines
The implementation doesn't have any problems with guests inside of guests.
The only reason to restrict guests inside of guests is because the we aren't certain which permissions make sense.
well, we have not even touched the permission issues yet
Agreed, permissions have not discussed but the point is that is the only reason to keep from nesting the networking stack the way I have described it.
Eric
Re: [patch 2/6] [Network namespace] Network device sharing by view
Quoted 10 lines · Herbert Poetzl
Quoted 4 lines
Have a few more network interfaces for a layer 2 solution is fundamental. Believing without proof and after arguments to the contrary that you have not contradicted that a layer 2 solution is inherently slower is non-productive.
assuming that it will not be slower, although it will now pass two network stacks and the bridging code is non-productive too, let's see how it goes but do not ignore the overhead just because it might simplify the implementation ...
Sure. Mostly I have set it aside because the overhead is not horrible and it is a very specific case that can be heavily optimized if the core infrastructure is solid.
Quoted 11 lines
Quoted 10 lines
Arguing that a layer 2 only solution most prove itself on guest to guest communication is also non-productive.
So just to sink one additional nail in the coffin of the silly guest to guest communication issue. For any two guests where fast communication between them is really important I can run an additional interface pair that requires no routing or bridging. Given that the implementation of the tunnel device is essentially the same as the loopback interface and that I make only one trip through the network stack there will be no performance overhead.
Quoted 6 lines
that is a good argument and I think I'm perfectly fine with this, given that the implementation allows that (i.e. the network stack can handle two interfaces with the same IP assigned and will choose the local interface over the remote one when the traffic will be between guests)
Yep. That exists today. The network stack prefers routes as specific as possible.
Quoted 4 lines
Quoted 2 lines
Similarly for any critical guest communication to the outside world I can give the guest a real network adapter.
with a single MAC assigned, that is, I presume?
Yes.
Quoted 11 lines
guess that's what this discussion is about, finding out the various aspects how isolation and/or vitrtualization can be accomplished and what features we consider common/useful enough for mainline ... for me that is still in the brainstorming phase, although several 'working prototypes' already exist. IMHO the next step is to collect a set of representative use cases and test them with each implementation, regarding performance, usability and practicability
I am fairly strongly convinced a layer 2 solution will do fine. So for me it is a matter of proving that and ensuring a good implementation.
Quoted 5 lines
not necessarily, but I _know_ that the overhead added at layer 3 is unmeasureable, and it still needs to be proven that this is true for a layer 2 solution (which I'd actually prefer, because it solves the protocol _and_ setup issues)
That is a good perspective. Layer 3 is free, is layer 2 also free? Unless the cache miss penalty is a killer layer 2 should come very close. Of course VJ recently gave some evidence that packet processing is dominated by cache misses.
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
From what I have seen of layer 3 solutions it is a
bloody maintenance nightmare, and an inflexible mess.
that is your opinion, I really doubt that you will have less maintenance when you apply policy to the guests ...
Yes and mostly of the layer 3 things that I implemented. At a moderately fundamental level I see layer 3 implementations being a special case that is a tangent from the rest of the networking code. So I don't see a real synthesis with what the rest of the networking stack is doing. Plus all of the limitations that come with a layer 3 implementation.
Quoted 25 lines
example here (just to clarify):
- let's assume we have eth0 on the host and in guest A and B, with the following setup:
eth0(H) 192.168.0.1/24 eth0(A) 10.0.1.1/16 10.0.1.2/16 eth0(B) 10.0.2.1/16- now what keeps guest B from jsut assigning 10.0.2.2/16 to eth0? you need some kind of mechanism to prevent that, and/or to block the packets using inappropriate IPs
* in the first case, i.e. you prevent assigning certain IPs inside a guest, you get a semantic change in the behaviour compared to a normal system, but there is no additional overhead on the communication
* in the second case, you have to maintain the policy mechanism and keep it in sync with the guest configuration (somehow), and of course you have to verify every communication
Quoted 6 lines
- OTOH, if you do not care about collisions basically assuming the point "that's like a hub on a network, if there are two guests with the same ip, it will be trouble, but that's okay" then this becomes a real issue for providers with potentially 'evil' customers
So linux when serving as a router has strong filter capabilities.
So we can either use the strong network filtering linux already has making work for the host administrator who has poorly behaved customers. Or we can simply not give those poorly behaved guests CAP_NET_ADMIN, and assign the IP address at guest startup before dropping the capability. At which point the guest cannot misbehave.
Eric
Routing tables (Re: [patch 2/6] [Network namespace] Network device sharing by view)
In reply to Daniel LezcanoQuoted 18 lines
Quoted 9 lines · Andrey Savochkin
Daniel,
Quoted 5 lines · Daniel Lezcano
Quoted 4 lines
Then you lose the ability for each namespace to have its own routing entries. Which implies that you'll have difficulties with devices that should exist and be visible in one namespace only (like tunnels), as they require IP addresses and route.
Quoted 7 lines
Quoted 2 lines
I mean instead of having the route tables private to the namespace, the routes have the information to which namespace they are associated.
I think I understand what you're talking about: you want to make routing responsible for determining destination namespace ID in addition to route type (local, unicast etc), nexthop information, and so on. Right?
Quoted 8 lines
Yes.
Quoted 4 lines
My point is that if you make namespace tagging at routing time, and your packets are being routed only once, you lose the ability to have separate routing tables in each namespace.
Right. What is the advantage of having separate the routing tables ?
One application may be following. Consider firewall
(isp1) (isp2)
I I
+----------- red0------------- red1 ---------+
| + + |
| | red routing deamon (BGP) | |
| | | |
| | red routing tables | |
| | | |
| +----------tun(?)-----------------+ |
| |
| +----------tun(?)-----------------+ |
| | | |
| | green routing tables | |
I mana0 | | |
| | green routing deamon (ospf) | |
| + + |
+--------- green0 ---------- green1 ----------+
I I
That may allow running different routing deamon on red and green side. That is possible if they manage different routing tables on kernel. They not need communigate together, when route between them is static.
/ Kari Hurtta
Quoted 5 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
RFC PATCH 3/6 Network devices isolation
In reply to [email protected]The dev list view is filled and used from here. The dev_base_list has been replaced to the dev list view and devices can be accessed only if the view has the device in its list. All calls from the userspace, ioctls, netlinks and procfs, will use the network devices view instead of the global network device list.
Replace-Subject: [Network namespace] Network devices isolation
--
net/core/dev.c | 147 ++++++++++++++++++++++++++++++++++++++------------- net/core/rtnetlink.c | 21 +++++-- 2 files changed, 126 insertions(+), 42 deletions(-)
net/core/dev.c +110 -37
@@ -115,6 +115,7 @@#include <net/iw_handler.h> #include <asm/current.h> #include <linux/audit.h> +#include <linux/net_ns.h> #include <linux/dmaengine.h> /*
@@ -474,13 +475,16 @@struct net_device *__dev_get_by_name (const char *name) { -struct hlist_node *p; +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db; +struct net_device *dev; -hlist_for_each (p,dev_name_hash (name)) { -struct net_device *dev - = hlist_entry(p, struct net_device, name_hlist); +list_for_each (l, list) { + db = list_entry(l, struct net_ns_dev, list); + dev = db->dev;if (!strncmp (dev->name, name,IFNAMSIZ )) -return dev; +return dev; }return NULL ; }
@@ -498,13 +502,14 @@struct net_device *dev_get_by_name (const char *name) { +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list);struct net_device *dev; -read_lock (&dev_base_lock); +read_lock (&dev_list->lock); dev =__dev_get_by_name (name);if (dev)dev_hold (dev); -read_unlock (&dev_base_lock); +read_unlock (&dev_list->lock);return dev; }
@@ -521,11 +526,14 @@struct net_device *__dev_get_by_index (int ifindex) { -struct hlist_node *p; +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db; +struct net_device *dev; -hlist_for_each (p,dev_index_hash (ifindex)) { -struct net_device *dev - = hlist_entry(p, struct net_device, index_hlist); +list_for_each (l, list) { + db = list_entry(l, struct net_ns_dev, list); + dev = db->dev;if (dev->ifindex == ifindex)return dev; }
@@ -545,13 +553,14 @@struct net_device *dev_get_by_index (int ifindex) { +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list);struct net_device *dev; -read_lock (&dev_base_lock); +read_lock (&dev_list->lock); dev =__dev_get_by_index (ifindex);if (dev)dev_hold (dev); -read_unlock (&dev_base_lock); +read_unlock (&dev_list->lock);return dev; }
@@ -571,14 +580,24 @@struct net_device *dev_getbyhwaddr (unsigned short type,char *ha) { -struct net_device *dev; +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db; +struct net_device *dev =NULL ;ASSERT_RTNL (); -for (dev = dev_base; dev; dev = dev->next) +read_lock (&dev_list->lock); +list_for_each (l, list) { + db = list_entry(l, struct net_ns_dev, list); + dev = db->dev;if (dev->type == type && !memcmp (dev->dev_addr, ha, dev->addr_len)) -break ; + goto out; + } + dev =NULL ; +out: +read_unlock (&dev_list->lock);return dev; }
@@ -586,15 +605,25 @@struct net_device *dev_getfirstbyhwtype (unsigned short type) { +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db;struct net_device *dev;rtnl_lock (); -for (dev = dev_base; dev; dev = dev->next) { + +read_lock (&dev_list->lock); +list_for_each (l, list) { + db = list_entry(l, struct net_ns_dev, list); + dev = db->dev;if (dev->type == type) {dev_hold (dev); -break ; + goto out; } } + dev =NULL ; +out: +read_unlock (&dev_list->lock);rtnl_unlock ();return dev; }
@@ -614,16 +643,23 @@struct net_device *dev_get_by_flags (unsigned short if_flags,unsigned short mask) { +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db;struct net_device *dev; -read_lock (&dev_base_lock); -for (dev = dev_base; dev !=NULL ; dev = dev->next) { +read_lock (&dev_list->lock); +list_for_each (l, list) { + db = list_entry(l, struct net_ns_dev, list); + dev = db->dev;if (((dev->flags ^ if_flags) & mask) ==0 ) {dev_hold (dev); -break ; + goto out; } } -read_unlock (&dev_base_lock); + dev =NULL ; +out: +read_unlock (&dev_list->lock);return dev; }
@@ -1942,6 +1978,9 @@static int dev_ifconf (char __user *arg) {struct ifconf ifc; +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db;struct net_device *dev;char __user *pos;int len;
@@ -1963,8 +2002,14 @@*/ total =0 ; -for (dev = dev_base; dev; dev = dev->next) { + +list_for_each (l, list) { + + db = list_entry(l, struct net_ns_dev, list); + dev = db->dev; +for (i =0 ; i <NPROTO ; i++) { +if (gifconf_list[i]) {int done;if (!pos)
@@ -1995,40 +2040,63 @@* This is invoked by the /proc filesystem handler to display a device * in detail. */ -static __inline__struct net_device *dev_get_idx (loff_t pos) +static __inline__struct net_ns_dev *dev_get_idx (loff_t pos) { -struct net_device *dev; -loff_t i; - -for (i =0 , dev = dev_base; dev && i < pos; ++i, dev = dev->next); - -return i == pos ? dev :NULL ; +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db; + +loff_t i =0 ; + +list_for_each (l, list) { + db = list_entry(l, struct net_ns_dev, list); +if (i == pos) +return db; + i++; + }; +return NULL ; }void *dev_seq_start (struct seq_file *seq,loff_t *pos) { -read_lock (&dev_base_lock); +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); + +read_lock (&dev_list->lock);return *pos ?dev_get_idx (*pos -1 ) :SEQ_START_TOKEN ; }void *dev_seq_next (struct seq_file *seq,void *v,loff_t *pos) { +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct net_ns_dev *db =NULL ; +struct list_head *next; + ++*pos; -return v ==SEQ_START_TOKEN ? dev_base : ((struct net_device *)v)->next; + +if (v ==SEQ_START_TOKEN ) + next = dev_list->list.next; +else + next = ((struct net_ns_dev *)v)->list.next; +if (next && next != &dev_list->list) + db = list_entry(next, struct net_ns_dev, list); +return db; }void dev_seq_stop (struct seq_file *seq,void *v) { -read_unlock (&dev_base_lock); +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +read_unlock (&dev_list->lock); } -static void dev_seq_printf_stats (struct seq_file *seq,struct net_device *dev) +static void dev_seq_printf_stats (struct seq_file *seq,struct net_ns_dev *db) { +struct net_device *dev = db->dev; +if (dev->get_stats) {struct net_device_stats *stats = dev->get_stats (dev); seq_printf(seq, "%6s:%8lu %7lu %4lu %4lu %4lu %5lu %10lu %9lu " - "%8lu %7lu %4lu %4lu %4lu %5lu %7lu %10lu\n", + "%8lu %7lu %4lu %4lu %4lu %5lu %7lu %10lu\n", dev->name, stats->rx_bytes, stats->rx_packets, stats->rx_errors, stats->rx_dropped + stats->rx_missed_errors,
@@ -2402,7 +2470,7 @@*/ static int dev_ifsioc (struct ifreq *ifr,unsigned int cmd) { -int err; +int err =0 ;struct net_device *dev =__dev_get_by_name (ifr->ifr_name);if (!dev)
@@ -2509,7 +2577,6 @@/* * Unknown or private ioctl */ -default : if ((cmd >= SIOCDEVPRIVATE && cmd <= SIOCDEVPRIVATE + 15) ||
@@ -2847,6 +2914,10 @@ } } + ret =net_ns_dev_register (dev, &(net_ns ()->dev_list)); +if (ret) + goto out_err; +/* Fix illegal SG+CSUM combinations. */ if ((dev->features & NETIF_F_SG) && !(dev->features & (NETIF_F_IP_CSUM |
@@ -3218,6 +3289,8 @@return -ENODEV ; } +net_ns_dev_unregister (dev, &(net_ns ()->dev_list)); + dev->reg_state =NETREG_UNREGISTERING ;synchronize_net ();
net/core/rtnetlink.c +16 -5
@@ -55,6 +55,7 @@#include <linux/wireless.h> #include <net/iw_handler.h> #endif /* CONFIG_NET_WIRELESS_RTNETLINK */ +#include <linux/net_ns.h> static DEFINE_MUTEX (rtnl_mutex);
@@ -315,21 +316,31 @@static int rtnetlink_dump_ifinfo (struct sk_buff *skb,struct netlink_callback *cb) { -int idx; +int idx =0 ;int s_idx = cb->args[0 ]; + +struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list); +struct list_head *l, *list = &dev_list->list; +struct net_ns_dev *db;struct net_device *dev; -read_lock (&dev_base_lock); -for (dev=dev_base, idx=0 ; dev; dev = dev->next, idx++) { -if (idx < s_idx) +read_lock (&dev_list->lock); +list_for_each (l, list) { + +if (idx++ < s_idx)continue ; + + db = list_entry(l, struct net_ns_dev, list); + dev = db->dev; +if (rtnetlink_fill_ifinfo (skb, dev,RTM_NEWLINK ,NETLINK_CB (cb->skb).pid, cb->nlh->nlmsg_seq,0 ,NLM_F_MULTI ) <=0 )break ; } -read_unlock (&dev_base_lock); +read_unlock (&dev_list->lock); + cb->args[0 ] = idx;return skb->len;
--
Re: [RFC] [patch 3/6] [Network namespace] Network devices isolation
net/core/dev.c line 513
Quoted 13 lines · On Fri, Jun 09, 2006 at 11:02:05PM +0200, [email protected] wrote:
struct
net_device*dev_get_by_name(const char *name) {net/core/dev.c +3
-2+struct net_ns_dev_list *dev_list = &(net_ns ()->dev_list);struct net_device *dev; -read_lock (&dev_base_lock); +read_lock (&dev_list->lock); dev =__dev_get_by_name (name);if (dev)dev_hold (dev); -read_unlock (&dev_base_lock); +read_unlock (&dev_list->lock);return dev;
And what would stop renames done via different lists from creating a conflict? Incidentally, WTF protects the device name while we are doing that lookup?
While we are at it, what are you going to do with sysfs? ls /sys/class/net and watch the fun...
RFC PATCH 4/6 Network inet devices isolation
In reply to [email protected]The network isolation relies on the fact that an application can not use IP addresses not belonging to the container in which it's running. This patch isolates the inet device level by adding a structure namespace pointer in the structure in_ifaddr. When an ip address is set inside a network namespace, the structure in_ifaddr is filled with the current namespace pointer. There is a special case with loopback address which belongs to all the namespaces and its particularity is to have the network namespace pointer set to NULL. This patch isolates the ifconfig, ip addr commands, so when an IP address is set, this one it is not visible by another network namespaces.
Replace-Subject: [Network namespace] Network inet devices isolation
--
include/linux/inetdevice.h | 1 + net/ipv4/devinet.c | 28 +++++++++++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-)
include/linux/inetdevice.h +1 -0
@@ -99,6 +99,7 @@unsigned char ifa_flags;unsigned char ifa_prefixlen;char ifa_label[IFNAMSIZ ]; +struct net_namespace *ifa_net_ns; };extern int register_inetaddr_notifier (struct notifier_block *nb);
net/ipv4/devinet.c +27 -1
@@ -54,6 +54,7 @@#include <linux/notifier.h> #include <linux/inetdevice.h> #include <linux/igmp.h> +#include <linux/net_ns.h> #ifdef CONFIG_SYSCTL #include <linux/sysctl.h> #endif
@@ -257,6 +258,7 @@if (!(ifa->ifa_flags &IFA_F_SECONDARY ) || ifa1->ifa_mask != ifa->ifa_mask || + ifa->ifa_net_ns !=net_ns () || !inet_ifa_match (ifa1->ifa_address, ifa)) { ifap1 = &ifa->ifa_next; prev_prom = ifa;
@@ -317,6 +319,8 @@if (destroy) {inet_free_ifa (ifa1); +put_net_ns (ifa1->ifa_net_ns); +if (!in_dev->ifa_list)inetdev_destroy (in_dev); }
@@ -343,6 +347,7 @@ ifa->ifa_scope <= ifa1->ifa_scope) last_primary = &ifa1->ifa_next;if (ifa1->ifa_mask == ifa->ifa_mask && + ifa1->ifa_net_ns == ifa->ifa_net_ns &&inet_ifa_match (ifa1->ifa_address, ifa)) {if (ifa1->ifa_local == ifa->ifa_local) {inet_free_ifa (ifa);
@@ -437,6 +442,8 @@for (ifap = &in_dev->ifa_list; (ifa = *ifap) !=NULL ; ifap = &ifa->ifa_next) { +if (ifa->ifa_net_ns !=net_ns ()) +continue ; if ((rta[IFA_LOCAL - 1] && memcmp(RTA_DATA(rta[IFA_LOCAL - 1]), &ifa->ifa_local, 4)) ||
@@ -497,6 +504,9 @@ ifa->ifa_scope = ifm->ifa_scope;in_dev_hold (in_dev); ifa->ifa_dev = in_dev; + ifa->ifa_net_ns =net_ns (); +get_net_ns (net_ns ()); +if (rta[IFA_LABEL -1 ])rtattr_strlcpy (ifa->ifa_label, rta[IFA_LABEL -1 ],IFNAMSIZ );else
@@ -631,10 +641,15 @@for (ifap = &in_dev->ifa_list; (ifa = *ifap) !=NULL ; ifap = &ifa->ifa_next)if (!strcmp (ifr.ifr_name, ifa->ifa_label)) -break ; +if (!ifa->ifa_net_ns || + ifa->ifa_net_ns ==net_ns ()) +break ; } } +if (ifa && ifa->ifa_net_ns && ifa->ifa_net_ns !=net_ns ()) + goto done; + ret = -EADDRNOTAVAIL ;if (!ifa && cmd !=SIOCSIFADDR && cmd !=SIOCSIFFLAGS ) goto done;
@@ -678,6 +693,12 @@ ret = -ENOBUFS ;if ((ifa =inet_alloc_ifa ()) ==NULL )break ; +if (!LOOPBACK (sin->sin_addr.s_addr)) { + ifa->ifa_net_ns =net_ns (); +get_net_ns (net_ns ()); + }else + ifa->ifa_net_ns =NULL ; +if (colon)memcpy (ifa->ifa_label, ifr.ifr_name,IFNAMSIZ );else
@@ -782,6 +803,8 @@ goto out;for (; ifa; ifa = ifa->ifa_next) { +if (ifa->ifa_net_ns && ifa->ifa_net_ns !=net_ns ()) +continue ;if (!buf) { done +=sizeof (ifr);continue ;
@@ -1012,6 +1035,7 @@ ifa->ifa_address =htonl (INADDR_LOOPBACK ); ifa->ifa_prefixlen =8 ; ifa->ifa_mask =inet_make_mask (8 ); + ifa->ifa_net_ns =NULL ;in_dev_hold (in_dev); ifa->ifa_dev = in_dev; ifa->ifa_scope =RT_SCOPE_HOST ;
@@ -1110,6 +1134,8 @@for (ifa = in_dev->ifa_list, ip_idx =0 ; ifa; ifa = ifa->ifa_next, ip_idx++) { +if (ifa->ifa_net_ns && ifa->ifa_net_ns !=net_ns ()) +continue ;if (ip_idx < s_ip_idx)continue ; if (inet_fill_ifaddr(skb, ifa, NETLINK_CB(cb->skb).pid,
--
RFC PATCH 5/6 ipv4 isolation
In reply to [email protected]This patch partially isolates ipv4 by adding the network namespace structure in the structure sock, bind bucket and skbuf. When a socket is created, the pointer to the network namespace is stored in the struct sock and the socket belongs to the namespace by this way. That allows to identify sockets related to a namespace for lookup and procfs.
The lookup is extended with a network namespace pointer, in order to identify listen points binded to the same port. That allows to have several applications binded to INADDR_ANY:port in different network namespace without conflicting. The bind is checked against port and network namespace.
When an outgoing packet has the loopback destination addres, the skbuff is filled with the network namespace. So the loopback packets never go outside the namespace. This approach facilitate the migration of loopback because identification is done by network namespace and not by address. The loopback has been benchmarked by tbench and the overhead is roughly 1.5 %
Replace-Subject: [Network namespace] ipv4 isolation
--
include/linux/skbuff.h | 2 ++ include/net/inet_hashtables.h | 34 ++++++++++++++++++++++++---------- include/net/inet_timewait_sock.h | 1 + include/net/sock.h | 4 ++++ net/dccp/ipv4.c | 7 ++++--- net/ipv4/af_inet.c | 2 ++ net/ipv4/inet_connection_sock.c | 3 ++- net/ipv4/inet_diag.c | 3 ++- net/ipv4/inet_hashtables.c | 6 +++++- net/ipv4/inet_timewait_sock.c | 1 + net/ipv4/ip_output.c | 4 ++++ net/ipv4/tcp_ipv4.c | 25 ++++++++++++++++--------- net/ipv4/udp.c | 7 +++++-- 13 files changed, 72 insertions(+), 27 deletions(-)
include/linux/skbuff.h +2 -0
@@ -27,6 +27,7 @@#include <linux/poll.h> #include <linux/net.h> #include <linux/textsearch.h> +#include <linux/net_ns.h> #include <net/checksum.h> #include <linux/dmaengine.h>
@@ -301,6 +302,7 @@ *data, *tail, *end; +struct net_namespace *net_ns; };#ifdef __KERNEL__
include/net/inet_hashtables.h +24 -10
@@ -23,6 +23,8 @@#include <linux/spinlock.h> #include <linux/types.h> #include <linux/wait.h> +#include <linux/in.h> +#include <linux/net_ns.h> #include <net/inet_connection_sock.h> #include <net/inet_sock.h>
@@ -78,6 +80,7 @@signed short fastreuse;struct hlist_node node;struct hlist_head owners; +struct net_namespace *net_ns; };#define inet_bind_bucket_for_each (tb, node, head) \
@@ -274,13 +277,15 @@extern struct sock *__inet_lookup_listener (const struct hlist_head *head,const u32 daddr,const unsigned short hnum, -const int dif); +const int dif, +const struct net_namespace *net_ns);/* Optimize the common listener case. */ static inline struct sock *inet_lookup_listener (struct inet_hashinfo *hashinfo,const u32 daddr, -const unsigned short hnum,const int dif) +const unsigned short hnum,const int dif, +const struct net_namespace *net_ns) {struct sock *sk =NULL ;const struct hlist_head *head;
@@ -294,8 +299,9 @@ (!inet->rcv_saddr || inet->rcv_saddr == daddr) && (sk->sk_family ==PF_INET || !ipv6_only_sock (sk)) && !sk->sk_bound_dev_if) - goto sherry_cache; - sk =__inet_lookup_listener (head, daddr, hnum, dif); + if (sk->sk_net_ns == net_ns && LOOPBACK(daddr)) + goto sherry_cache; + sk =__inet_lookup_listener (head, daddr, hnum, dif, net_ns); }if (sk) { sherry_cache:
@@ -358,7 +364,8 @@ __inet_lookup_established(struct inet_hashinfo *hashinfo, const u32 saddr, const u16 sport, const u32 daddr, const u16 hnum, - const int dif) + const int dif, + const struct net_namespace *net_ns) {INET_ADDR_COOKIE (acookie ,saddr ,daddr ) const __u32 ports =INET_COMBINED_PORTS (sport, hnum);
@@ -373,12 +380,16 @@prefetch (head->chain.first);read_lock (&head->lock);sk_for_each (sk, node, &head->chain) { +if (sk->sk_net_ns != net_ns &&LOOPBACK (daddr)) +continue ;if (INET_MATCH (sk, hash, acookie, saddr, daddr, ports, dif)) goto hit;/* You sunk my battleship! */ }/* Must check for a TIME_WAIT'er before going to listener hash. */ sk_for_each (sk, node, &(head + hashinfo->ehash_size)->chain) { +if (sk->sk_net_ns != net_ns &&LOOPBACK (daddr)) +continue ;if (INET_TW_MATCH (sk, hash, acookie, saddr, daddr, ports, dif)) goto hit; }
@@ -394,22 +405,25 @@static inline struct sock *__inet_lookup (struct inet_hashinfo *hashinfo,const u32 saddr,const u16 sport,const u32 daddr,const u16 hnum, -const int dif) +const int dif, +const struct net_namespace *net_ns) {struct sock *sk =__inet_lookup_established (hashinfo, saddr, sport, daddr, - hnum, dif); -return sk ? :inet_lookup_listener (hashinfo, daddr, hnum, dif); + hnum, dif, net_ns); +return sk ? :inet_lookup_listener (hashinfo, daddr, hnum, dif, net_ns); }static inline struct sock *inet_lookup (struct inet_hashinfo *hashinfo,const u32 saddr,const u16 sport,const u32 daddr,const u16 dport, -const int dif) +const int dif, +const struct net_namespace *net_ns) {struct sock *sk;local_bh_disable (); - sk =__inet_lookup (hashinfo, saddr, sport, daddr,ntohs (dport), dif); + sk =__inet_lookup (hashinfo, saddr, sport, daddr,ntohs (dport), + dif, net_ns);local_bh_enable ();return sk;
include/net/inet_timewait_sock.h +1 -0
@@ -115,6 +115,7 @@#define tw_refcnt __tw_common.skc_refcnt#define tw_hash __tw_common.skc_hash#define tw_prot __tw_common.skc_prot +#define tw_net_ns __tw_common.skc_net_nsvolatile unsigned char tw_substate;/* 3 bits hole, try to pack */ unsigned char tw_rcv_wscale;
include/net/sock.h +4 -0
@@ -47,6 +47,7 @@#include <linux/netdevice.h> #include <linux/skbuff.h> /* struct sk_buff */ #include <linux/security.h> +#include <linux/net_ns.h> #include <linux/filter.h>
@@ -94,6 +95,7 @@* @skc_refcnt: reference count * @skc_hash: hash value used with various protocol lookup tables * @skc_prot: protocol handlers inside a network family +* @skc_net_ns: network namespace owning the socket * * This is the minimal network layer representation of sockets, the header * for struct sock and struct inet_timewait_sock.
@@ -108,6 +110,7 @@atomic_t skc_refcnt;unsigned int skc_hash;struct proto *skc_prot; +struct net_namespace *skc_net_ns; };/**
@@ -183,6 +186,7 @@#define sk_refcnt __sk_common.skc_refcnt#define sk_hash __sk_common.skc_hash#define sk_prot __sk_common.skc_prot +#define sk_net_ns __sk_common.skc_net_ns unsigned char sk_shutdown : 2, sk_no_check : 2, sk_userlocks : 4;
net/dccp/ipv4.c +4 -3
@@ -308,7 +308,8 @@ } sk =inet_lookup (&dccp_hashinfo, iph->daddr, dh->dccph_dport, - iph->saddr, dh->dccph_sport,inet_iif (skb)); + iph->saddr, dh->dccph_sport,inet_iif (skb), + skb->net_ns);if (sk ==NULL ) {ICMP_INC_STATS_BH (ICMP_MIB_INERRORS );return ;
@@ -610,7 +611,7 @@ nsk =__inet_lookup_established (&dccp_hashinfo, iph->saddr, dh->dccph_sport, iph->daddr,ntohs (dh->dccph_dport), -inet_iif (skb)); +inet_iif (skb), skb->net_ns);if (nsk !=NULL ) {if (nsk->sk_state !=DCCP_TIME_WAIT ) {bh_lock_sock (nsk);
@@ -924,7 +925,7 @@ sk =__inet_lookup (&dccp_hashinfo, skb->nh.iph->saddr, dh->dccph_sport, skb->nh.iph->daddr,ntohs (dh->dccph_dport), -inet_iif (skb)); +inet_iif (skb), skb->net_ns);/* * Step 2:
net/ipv4/af_inet.c +2 -0
@@ -325,6 +325,7 @@ sk->sk_family =PF_INET ; sk->sk_protocol = protocol; sk->sk_backlog_rcv = sk->sk_prot->backlog_rcv; + sk->sk_net_ns =net_ns (); inet->uc_ttl =-1 ; inet->mc_loop =1 ;
@@ -616,6 +617,7 @@sock_graft (sk2, newsock); + sk2->sk_net_ns =net_ns (); newsock->state =SS_CONNECTED ; err =0 ;release_sock (sk2);
net/ipv4/inet_connection_sock.c +2 -1
@@ -116,7 +116,7 @@ head = &hashinfo->bhash[inet_bhashfn (snum, hashinfo->bhash_size)];spin_lock (&head->lock);inet_bind_bucket_for_each (tb, node, &head->chain) -if (tb->port == snum) +if (tb->port == snum && tb->net_ns ==net_ns ()) goto tb_found; } tb =NULL ;
@@ -146,6 +146,7 @@ }else if (tb->fastreuse && (!sk->sk_reuse || sk->sk_state ==TCP_LISTEN )) tb->fastreuse =0 ; + tb->net_ns =net_ns (); success:if (!inet_csk (sk)->icsk_bind_hash)inet_bind_hash (sk, tb, snum);
net/ipv4/inet_diag.c +2 -1
@@ -241,7 +241,8 @@if (req->idiag_family ==AF_INET ) { sk =inet_lookup (hashinfo, req->id.idiag_dst[0 ], req->id.idiag_dport, req->id.idiag_src[0 ], - req->id.idiag_sport, req->id.idiag_if); + req->id.idiag_sport, req->id.idiag_if, + in_skb->net_ns); }#if defined(CONFIG_IPV6 ) || defined (CONFIG_IPV6_MODULE ) else if (req->idiag_family == AF_INET6) {
net/ipv4/inet_hashtables.c +5 -1
@@ -126,7 +126,8 @@* wildcarded during the search since they can never be otherwise. */ struct sock *__inet_lookup_listener (const struct hlist_head *head,const u32 daddr, -const unsigned short hnum,const int dif) +const unsigned short hnum,const int dif, +const struct net_namespace *net_ns) {struct sock *result =NULL , *sk;const struct hlist_node *node;
@@ -139,6 +140,9 @@const __u32 rcv_saddr = inet->rcv_saddr;int score = sk->sk_family ==PF_INET ?1 :0 ; +if (sk->sk_net_ns != net_ns &&LOOPBACK (daddr)) +continue ; +if (rcv_saddr) {if (rcv_saddr != daddr)continue ;
net/ipv4/inet_timewait_sock.c +1 -0
@@ -110,6 +110,7 @@ tw->tw_hash = sk->sk_hash; tw->tw_ipv6only =0 ; tw->tw_prot = sk->sk_prot_creator; + tw->tw_net_ns = sk->sk_net_ns;atomic_set (&tw->tw_refcnt,1 );inet_twsk_dead_node_init (tw);__module_get (tw->tw_prot->owner);
net/ipv4/ip_output.c +4 -0
@@ -284,6 +284,10 @@ skb->dev = dev; skb->protocol =htons (ETH_P_IP ); +if ((skb->nh.iph->protocol ==IPPROTO_TCP || + skb->nh.iph->protocol ==IPPROTO_UDP ) && +LOOPBACK (skb->nh.iph->daddr)) + skb->net_ns = skb->sk->sk_net_ns; return NF_HOOK_COND(PF_INET, NF_IP_POST_ROUTING, skb, NULL, dev, ip_finish_output,
net/ipv4/tcp_ipv4.c +16 -9
@@ -349,7 +349,7 @@ } sk =inet_lookup (&tcp_hashinfo, iph->daddr, th->dest, iph->saddr, - th->source,inet_iif (skb)); + th->source,inet_iif (skb), skb->net_ns);if (!sk) {ICMP_INC_STATS_BH (ICMP_MIB_INERRORS );return ;
@@ -933,7 +933,8 @@ nsk =__inet_lookup_established (&tcp_hashinfo, skb->nh.iph->saddr, th->source, skb->nh.iph->daddr, -ntohs (th->dest),inet_iif (skb)); +ntohs (th->dest),inet_iif (skb), + skb->net_ns);if (nsk) {if (nsk->sk_state !=TCP_TIME_WAIT ) {
@@ -1071,7 +1072,7 @@ sk =__inet_lookup (&tcp_hashinfo, skb->nh.iph->saddr, th->source, skb->nh.iph->daddr,ntohs (th->dest), -inet_iif (skb)); +inet_iif (skb), skb->net_ns);if (!sk) goto no_tcp_socket;
@@ -1149,7 +1150,8 @@struct sock *sk2 =inet_lookup_listener (&tcp_hashinfo, skb->nh.iph->daddr,ntohs (th->dest), -inet_iif (skb)); +inet_iif (skb), + skb->net_ns);if (sk2) {inet_twsk_deschedule ((struct inet_timewait_sock *)sk, &tcp_death_row);
@@ -1395,7 +1397,8 @@ } get_sk:sk_for_each_from (sk, node) { -if (sk->sk_family == st->family) { +if (sk->sk_family == st->family && + sk->sk_net_ns ==net_ns ()) { cur = sk; goto out; }
@@ -1446,7 +1449,8 @@read_lock (&tcp_hashinfo.ehash[st->bucket].lock);sk_for_each (sk, node, &tcp_hashinfo.ehash[st->bucket].chain) { -if (sk->sk_family != st->family) { +if (sk->sk_family != st->family || + sk->sk_net_ns !=net_ns ()) {continue ; } rc = sk;
@@ -1455,7 +1459,8 @@ st->state =TCP_SEQ_STATE_TIME_WAIT ;inet_twsk_for_each (tw, node, &tcp_hashinfo.ehash[st->bucket + tcp_hashinfo.ehash_size].chain) { -if (tw->tw_family != st->family) { +if (tw->tw_family != st->family || + tw->tw_net_ns !=net_ns ()) {continue ; } rc = tw;
@@ -1481,7 +1486,8 @@ tw = cur; tw =tw_next (tw); get_tw: -while (tw && tw->tw_family != st->family) { +while (tw && (tw->tw_family != st->family || + tw->tw_net_ns !=net_ns ())) { tw =tw_next (tw); }if (tw) {
@@ -1505,7 +1511,8 @@ sk =sk_next (sk);sk_for_each_from (sk, node) { -if (sk->sk_family == st->family) +if (sk->sk_family == st->family && + sk->sk_net_ns ==net_ns ()) goto found; }
net/ipv4/udp.c +5 -2
@@ -184,6 +184,7 @@ (!inet2->rcv_saddr || !inet->rcv_saddr || inet2->rcv_saddr == inet->rcv_saddr) && + sk2->sk_net_ns == sk->sk_net_ns && (!sk2->sk_reuse || !sk->sk_reuse)) goto fail; }
@@ -1404,7 +1405,8 @@for (state->bucket =0 ; state->bucket <UDP_HTABLE_SIZE ; ++state->bucket) {struct hlist_node *node;sk_for_each (sk, node, &udp_hash[state->bucket]) { -if (sk->sk_family == state->family) +if (sk->sk_family == state->family && + sk->sk_net_ns ==net_ns ()) goto found; } }
@@ -1421,7 +1423,8 @@ sk =sk_next (sk); try_again: ; - }while (sk && sk->sk_family != state->family); + }while (sk && (sk->sk_family != state->family || + sk->sk_net_ns !=net_ns ()));if (!sk && ++state->bucket <UDP_HTABLE_SIZE ) { sk =sk_head (&udp_hash[state->bucket]);
--
Re: [RFC] [patch 5/6] [Network namespace] ipv4 isolation
Quoted 6 lines · On Fri, 9 Jun 2006, [email protected] wrote:
When an outgoing packet has the loopback destination addres, the skbuff is filled with the network namespace. So the loopback packets never go outside the namespace. This approach facilitate the migration of loopback because identification is done by network namespace and not by address. The loopback has been benchmarked by tbench and the overhead is roughly 1.5 %
I think you'll need to make it so this code has zero impact when not configured.
- James
Signature · 2 lines
James Morris <[email protected]>
Re: [RFC] [patch 5/6] [Network namespace] ipv4 isolation
Quoted 11 lines · James Morris
Quoted 6 lines · On Fri, 9 Jun 2006, [email protected] wrote:
When an outgoing packet has the loopback destination addres, the skbuff is filled with the network namespace. So the loopback packets never go outside the namespace. This approach facilitate the migration of loopback because identification is done by network namespace and not by address. The loopback has been benchmarked by tbench and the overhead is roughly 1.5 %
Quoted 1 line
I think you'll need to make it so this code has zero impact when not configured.
Indeed, and over stuff other than loopback too. I'll not so humbly suggest :) netperf TCP_STREAM and TCP_RR figures _with_ CPU utilization/service demand measures.
rick jones
Re: [RFC] [patch 5/6] [Network namespace] ipv4 isolation
Quoted 4 lines · Rick Jones
Quoted 2 lines
I think you'll need to make it so this code has zero impact when not configured.
Indeed, and over stuff other than loopback too. I'll not so humbly suggest :)
Yes, I meant the whole lot.
- James
Signature · 2 lines
James Morris <[email protected]>
RFC PATCH 6/6 Network namespace debugfs
In reply to [email protected]This patch is for testing purpose. It allows to read which network devices are accessible and to add a network device to the view. This RFC hack is purely for discussing the best way to do that.
After unsharing with CLONE_NEWNET flag:
--------------------------------------
To see which devices are accessible:
cat /sys/kernel/debug/net_ns/dev
To add a device:
echo eth1 > /sys/kernel/debug/net_ns/dev
This functionnality is intended to be implemented in an higher level container configuration.
Replace-Subject: [Network namespace] Network namespace debugfs
--
fs/debugfs/Makefile | 2 fs/debugfs/net_ns.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++ net/Kconfig | 4 + 3 files changed, 146 insertions(+), 1 deletion(-)
fs/debugfs/Makefile +1 -1
@@ -1,4 +1,4 @@ debugfs-objs := inode.o file.o obj-$(CONFIG_DEBUG_FS) += debugfs.o - +obj-$(CONFIG_NET_NS_DEBUG) += net_ns.o
fs/debugfs/net_ns.c new file +141 -0
141 lines not on this page · raw message
net/Kconfig +4 -0
@@ -69,6 +69,10 @@ config NET_NS vservers, to use network namespaces to provide isolated network for different servers. If unsure, say N. +config NET_NS_DEBUG + bool "Debug fs for network namespace" + depends on DEBUG_FS && NET_NS + if INET source "net/ipv4/Kconfig" source "net/ipv6/Kconfig"
--
[email protected] writes in gmane.linux.network,gmane.linux.kernel:
Quoted 15 lines
The following patches create a private "network namespace" for use within containers. This is intended for use with system containers like vserver, but might also be useful for restricting individual applications' access to the network stack.
These patches isolate traffic inside the network namespace. The network ressources, the incoming and the outgoing packets are identified to be related to a namespace.
It hides network resource not contained in the current namespace, but still allows administration of the network with normal commands like ifconfig.
It applies to the kernel version 2.6.17-rc6-mm1
Quoted 9 lines
It provides the following:
-------------------------
- when an application unshares its network namespace, it looses its view of all network devices by default. The administrator can choose to make any devices to become visible again. The container then gains a view to the device but without the ip address configured on it. It is up to the container administrator to use ifconfig or ip command to setup a new ip address. This ip address is only visible inside the container.
Do other namespaces work differently ?
When namespace is unshared, it has initially the same resources (for example compare to CLONE_NEWNS)
Quoted 6 lines
- the loopback is isolated inside the container and it is not possible to communicate between containers via the loopback.
- several containers can have an application bind to the same address:port without conflicting.
That of course be problem, if initially unshared namespace shares same resources.
/ Kari Hurtta
My apologies for not looking at this earlier I had an email hickup so I'm having to recreate the context from email archives, and you didn't copy me.
Have you seen my previous work in this direction?
I know I had a much much more complete implementation. The only part I had not completed was iptables support and that was about a days more work.
Quoted 9 lines
The following patches create a private "network namespace" for use within containers. This is intended for use with system containers like vserver, but might also be useful for restricting individual applications' access to the network stack.
These patches isolate traffic inside the network namespace. The network ressources, the incoming and the outgoing packets are identified to be related to a namespace.
Quoted 5 lines
It hides network resource not contained in the current namespace, but still allows administration of the network with normal commands like ifconfig.
It applies to the kernel version 2.6.17-rc6-mm1
A couple of comments.
Quoted 9 lines
------------
- do unshare with theCLONE_NEWNETflag as root
- do echo eth0 >/sys/kernel/debug/net_ns/dev
- use ifconfig or ip command to set a new ip addressWhat is missing ?
-----------------
The routes are not yet isolated, that implies:
Quoted 7 lines
- binding to another container's address is allowed
- an outgoing packet which has an unset source address can potentially get another container's address
- an incoming packet can be routed to the wrong container if there are several containers listening to the same addr:port
I haven't looked at the patches in enough detail to see how the network isolation is being done exactly. But some of these comments and some of the other pieces I have seen don't give me warm fuzzies.
In particular I did not see a provision for multiple instance of the loopback device.
As a general rule network sockets and network devices should be attached to the network namespaces, which basically preserves all of the existing network stack logic.
Basically this means that the only operations that get more expensive are reads of global variables, which take a necessary extra indirection.
As a general rule I found that it usually makes sense to add an additional namespace field to hash tables so they can still use the boot time memory allocator. Although if you already have a network device as part of your hash table key that isn't necessary for the network stack.
Eric
Quoted 5 lines · Eric W. Biederman
Have you seen my previous work in this direction?
I know I had a much much more complete implementation. The only part I had not completed was iptables support and that was about a days more work.
No, I didn't see your work, is it possible to send me a pointer on it or to have a patchset of your code ?
Quoted 9 lines · Daniel Lezcano
Quoted 4 lines · Eric W. Biederman
Have you seen my previous work in this direction?
I know I had a much much more complete implementation. The only part I had not completed was iptables support and that was about a days more work.No, I didn't see your work, is it possible to send me a pointer on it or to have a patchset of your code ?
It is in my git tree up on kernel.org. I think it is in my proof-of-concept branch.
The individual commits tell a tangled tale that is definitely unacceptable for an upstream merge but the actual result is interesting.
If that isn't enough to find it, tell me and I will track down the details. It has been a couple months since I posted it during the design discussion and I'm way overdue for being in bed, tonight.
Eric
Quoted 11 lines · On Fri, Jun 09, 2006 at 11:02:02PM +0200, [email protected] wrote:
What is missing ?
-----------------
The routes are not yet isolated, that implies:- binding to another container's address is allowed
- an outgoing packet which has an unset source address can potentially get another container's address
- an incoming packet can be routed to the wrong container if there are several containers listening to the same addr:port
- renaming an interface in one "namespace" affects everyone.
Quoted 2 lines · Al Viro
On Fri, Jun 09, 2006 at 11:02:02PM +0200, [email protected] wrote:
- renaming an interface in one "namespace" affects everyone.
Exact. If we ensure the interface can't be renamed if used in different namespace, is it really a problem ?
Quoted 6 lines · Daniel Lezcano
Quoted 2 lines · Al Viro
On Fri, Jun 09, 2006 at 11:02:02PM +0200, [email protected] wrote:
- renaming an interface in one "namespace" affects everyone.Exact. If we ensure the interface can't be renamed if used in different namespace, is it really a problem ?
You _still_ have a single namespace; look in /sys/class/net and you'll see.
Quoted 11 lines · Al Viro
Quoted 6 lines · Daniel Lezcano
Quoted 2 lines · Al Viro
On Fri, Jun 09, 2006 at 11:02:02PM +0200, [email protected] wrote:
- renaming an interface in one "namespace" affects everyone.Exact. If we ensure the interface can't be renamed if used in different namespace, is it really a problem ?
You _still_ have a single namespace; look in
/sys/class/netand you'll see.
Yes, that's right. The network devices namespaces are not yet implemented. There are potentially some conflicts with /proc and sysfs but we will address them in a future.
BTW, do you have some ideas on how handle these conflicts ?
Quoted 11 lines · [email protected] wrote:
What is missing ?
-----------------
The routes are not yet isolated, that implies:- binding to another container's address is allowed
- an outgoing packet which has an unset source address can potentially get another container's address
- an incoming packet can be routed to the wrong container if there are several containers listening to the same addr:port
Does that mean that identification of containers for incoming packets is done by IP address through routing (just had a quick look at the patches, if I missed something obvious please just point me to it)? How is code that uses global data without verifying its presence (and visibility in the container) at initialization time going to be handled? Netfilter and I think the TC action code contain some examples for this.