Skip to contentSkip to conversation

Contribution - Bug fixes and contributions to UML

uml-devel Anton Ivanov (antivano) 2 participants 5 messages Feb 28, 2014 – Mar 6, 2014 User-mode Linux (UML)

lore
Share & tools
lore
#0 lore raw

Hi Richard, Hi Jeff, hi list,

On behalf of Cisco systems, I am authorized to make a offer a set bug fixes as well as contribute several additional features and performance improvements to UML. All of these have been used internally for a couple of years and will ship as parts of product(s) in the near future. Some of these improve performance by up to 8 times on use cases which are of interest to us and are likely to be of interest to the community.

As the full patchset is now in the 100k+ zone, so I am going to do only the announcement now and submit the patches one by one after that over the next 1-2 weeks.

We will submit separately bug fixes for:

1. Critical memory corruption on startup observed on heavily loaded machines (especially when multiple UMLs run simultaneously).
2. Fix(es) for incorrect handling of error conditions when UML is run under expect and conX=fd: is used to communicate with another process. The same error may be observed on internal UML IPCs too leading to immediate crash.

I will also file bugs for both vs Debian UML package so that patches for both can go in ASAP.

In addition to the bug fixes, the new features include:

1. Several transports. All can do up to multi-gigabit throughput on some scenarios. We are contributing their counterparts to qemu/kvm as well.

1.1. Direct connection of UML to overlay networks/L2 VPNs using L2TPv3.

This has a number of advantages compared to the existing UML "multicast" and qemu "socket" transports.

* Standard compliant - RFC 3931 updated recently by RFC 5641
* Supported on most network equipment
* Allowing to move virtual switching off-host to an NPU or high
performance physical switch
* Allowing to mix virtual and physical switching (well supported on
modern Linuxes and other OSes)
* Well researched security profile as well as established
interactions with IPSEC allowing to extend virtual networks outside the datacenter to remote physical devices and/or VMs.

1.2. Raw transport which allows both bi-directional communication with any network device which looks like Ethernet as well as in-span listening at speeds in the multi-gigabit range.

1.3. We intend to contribute other key overlay transports like GRE, etc as well. The ones we are contributing at this point are the ones which we have used most extensively and have had the most testing (~ 1.5-2 years).

2. New high res timer subsystem

Adding these new network transports to UML revealed a key issue - it cannot meter or shape any traffic correctly as its internal timer system is way off. Personally, I consider it a bug, however there is no "easy" fix here. The only way to fix it is a new timer driver. Unfortunately, it does not fix uml userspace - timers there remain off. It does fix all kernel timer functionality - traffic shaping (both qdisc and iptables traffic limits).

As a side effect, this provides performance improvements for tcp and other protocols which rely on kernel high res timers for their state machines.

We have further scalability contributions lined up which improve network and IO performance between 1.5 and 8 times (depending on use case), allow hundreds of virtual interfaces per UML without performance penalties, allow to run several hundreds (if not thousands) of UMLs per machine, etc. All in all, it can no go where no virtualization and no virtual networking has gone before.

However, I would prefer to take it one step at a time and get through these first (even these are quite a lot for one "sitting").

Best Regards,

Anton.
------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#1 lore raw

Am 28.02.2014 09:27, schrieb Anton Ivanov (antivano):

Quoted 74 lines

Hi Richard, Hi Jeff, hi list,

On behalf of Cisco systems, I am authorized to make a offer a set bug fixes as well as contribute several additional features and performance improvements to UML. All of these have been used internally for a couple of years and will ship as parts of product(s) in the near future. Some of these improve performance by up to 8 times on use cases which are of interest to us and are likely to be of interest to the community.

As the full patchset is now in the 100k+ zone, so I am going to do only the announcement now and submit the patches one by one after that over the next 1-2 weeks.

We will submit separately bug fixes for:

1. Critical memory corruption on startup observed on heavily loaded machines (especially when multiple UMLs run simultaneously).
2. Fix(es) for incorrect handling of error conditions when UML is run under expect and conX=fd: is used to communicate with another process. The same error may be observed on internal UML IPCs too leading to immediate crash.

I will also file bugs for both vs Debian UML package so that patches for both can go in ASAP.

In addition to the bug fixes, the new features include:

1. Several transports. All can do up to multi-gigabit throughput on some scenarios. We are contributing their counterparts to qemu/kvm as well.

1.1. Direct connection of UML to overlay networks/L2 VPNs using L2TPv3.

This has a number of advantages compared to the existing UML "multicast" and qemu "socket" transports.

* Standard compliant - RFC 3931 updated recently by RFC 5641
* Supported on most network equipment
* Allowing to move virtual switching off-host to an NPU or high
performance physical switch
* Allowing to mix virtual and physical switching (well supported on
modern Linuxes and other OSes)
* Well researched security profile as well as established
interactions with IPSEC allowing to extend virtual networks outside the datacenter to remote physical devices and/or VMs.

1.2. Raw transport which allows both bi-directional communication with any network device which looks like Ethernet as well as in-span listening at speeds in the multi-gigabit range.

1.3. We intend to contribute other key overlay transports like GRE, etc as well. The ones we are contributing at this point are the ones which we have used most extensively and have had the most testing (~ 1.5-2 years).

2. New high res timer subsystem

Adding these new network transports to UML revealed a key issue - it cannot meter or shape any traffic correctly as its internal timer system is way off. Personally, I consider it a bug, however there is no "easy" fix here. The only way to fix it is a new timer driver. Unfortunately, it does not fix uml userspace - timers there remain off. It does fix all kernel timer functionality - traffic shaping (both qdisc and iptables traffic limits).

As a side effect, this provides performance improvements for tcp and other protocols which rely on kernel high res timers for their state machines.

We have further scalability contributions lined up which improve network and IO performance between 1.5 and 8 times (depending on use case), allow hundreds of virtual interfaces per UML without performance penalties, allow to run several hundreds (if not thousands) of UMLs per machine, etc. All in all, it can no go where no virtualization and no virtual networking has gone before.

Quoted 2 lines

However, I would prefer to take it one step at a time and get through these first (even these are quite a lot for one "sitting").

Sounds awesome!

Please send the patches as soon as possible.
I'm eager to test and merge them.

Thanks,
//richard

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#2 lore raw

Bugfixes.

I need to pull actual changesets for the drivers, etc properly and verify that they build so those will be coming next week. You will be getting them one by one.

1. Memory corruption.

The reverse case of this race (you need to msync, before you do non-mmap fileops) is well known and textbook. This is the first and only time I have seen this one (fsync before mmap). I have not heard it mentioned either. It is however fairly easy to reproduce. If you run 200+ UML on a system ~0.2-0.5% will always die at startup with a memory corruption warning. While this does not happen every time (0.2-0.5% and only on startup) it is very reproducible for systems running lots of UMLs.

Once this fix went in we stopped seeing that one. Observed on 3.2, 3.3 and 3.8, fix tested on 3.2, 3.3, 3.4 and 3.8.

2. SIGPIPE.

Linux actually produces SIGPIPE ane EPIPE not only on missing reader. It will produce it under some circumstances on a stalled reader. Discovered when running UML under expect and/or trying to use fds and other virtual serials to do management transactions.

While I have not seen it on UML internal pipes I would not be surprised if you can reproduce it there too (f.e. if ubd thread is too slow). So SIGPIPE needs to be disabled. From there on, for most drivers have correct error handling for this.

Observed on 3.2, 3.3 and 3.8, fix tested on 3.2, 3.3, 3.4 and 3.8.

A.

Quoted 77 lines · Richard Weinberger

Am 28.02.2014 09:27, schrieb Anton Ivanov (antivano):

Quoted 74 lines

Hi Richard, Hi Jeff, hi list,

On behalf of Cisco systems, I am authorized to make a offer a set bug fixes as well as contribute several additional features and performance improvements to UML. All of these have been used internally for a couple of years and will ship as parts of product(s) in the near future. Some of these improve performance by up to 8 times on use cases which are of interest to us and are likely to be of interest to the community.

As the full patchset is now in the 100k+ zone, so I am going to do only the announcement now and submit the patches one by one after that over the next 1-2 weeks.

We will submit separately bug fixes for:

1. Critical memory corruption on startup observed on heavily loaded machines (especially when multiple UMLs run simultaneously).
2. Fix(es) for incorrect handling of error conditions when UML is run under expect and conX=fd: is used to communicate with another process. The same error may be observed on internal UML IPCs too leading to immediate crash.

I will also file bugs for both vs Debian UML package so that patches for both can go in ASAP.

In addition to the bug fixes, the new features include:

1. Several transports. All can do up to multi-gigabit throughput on some scenarios. We are contributing their counterparts to qemu/kvm as well.

1.1. Direct connection of UML to overlay networks/L2 VPNs using L2TPv3.

This has a number of advantages compared to the existing UML "multicast" and qemu "socket" transports.

* Standard compliant - RFC 3931 updated recently by RFC 5641
* Supported on most network equipment
* Allowing to move virtual switching off-host to an NPU or high
performance physical switch
* Allowing to mix virtual and physical switching (well supported on
modern Linuxes and other OSes)
* Well researched security profile as well as established
interactions with IPSEC allowing to extend virtual networks outside the datacenter to remote physical devices and/or VMs.

1.2. Raw transport which allows both bi-directional communication with any network device which looks like Ethernet as well as in-span listening at speeds in the multi-gigabit range.

1.3. We intend to contribute other key overlay transports like GRE, etc as well. The ones we are contributing at this point are the ones which we have used most extensively and have had the most testing (~ 1.5-2 years).

2. New high res timer subsystem

Adding these new network transports to UML revealed a key issue - it cannot meter or shape any traffic correctly as its internal timer system is way off. Personally, I consider it a bug, however there is no "easy" fix here. The only way to fix it is a new timer driver. Unfortunately, it does not fix uml userspace - timers there remain off. It does fix all kernel timer functionality - traffic shaping (both qdisc and iptables traffic limits).

As a side effect, this provides performance improvements for tcp and other protocols which rely on kernel high res timers for their state machines.

We have further scalability contributions lined up which improve network and IO performance between 1.5 and 8 times (depending on use case), allow hundreds of virtual interfaces per UML without performance penalties, allow to run several hundreds (if not thousands) of UMLs per machine, etc. All in all, it can no go where no virtualization and no virtual networking has gone before.

Quoted 2 lines

However, I would prefer to take it one step at a time and get through these first (even these are quite a lot for one "sitting").

Quoted 7 lines

Sounds awesome!

Please send the patches as soon as possible.
I'm eager to test and merge them.

Thanks,
//richard

arch/um/os-Linux/main.c +1 -0
index 7a86dd5..048166d 100644
@@ -149,6 +149,7 @@ int __init main(int argc, char **argv, char **envp)
 #endif
 
 	do_uml_initcalls();
+	change_sig(SIGPIPE, 0);
 	ret = linux_main(argc, argv);
 
 	/*
arch/um/include/shared/os.h +1 -0
index b314cf7..e984b2c 100644
@@ -144,6 +144,7 @@ extern int os_write_file(int fd, const void *buf, int count);
 extern int os_file_size(const char *file, unsigned long long *size_out);
 extern int os_file_modtime(const char *file, unsigned long *modtime);
 extern int os_pipe(int *fd, int stream, int close_on_exec);
+extern int os_fsync(int fd);
 extern int os_set_fd_async(int fd);
 extern int os_clear_fd_async(int fd);
 extern int os_set_fd_block(int fd, int blocking);
arch/um/kernel/physmem.c +8 -6
index f116db1..ea606ee 100644
@@ -87,6 +87,14 @@ void __init setup_physmem(unsigned long start, unsigned long reserve_end,
 
 	physmem_fd = create_mem_file(len + highmem);
 
+	/*
+	 * Special kludge - This page will be mapped in to userspace processes
+	 * from physmem_fd, so it needs to be written out there.
+	 */
+	os_seek_file(physmem_fd, __pa(&__syscall_stub_start));
+	os_write_file(physmem_fd, &__syscall_stub_start, PAGE_SIZE);
+        os_fsync(physmem_fd);
+
 	offset = uml_reserved - uml_physmem;
 	err = os_map_memory((void *) uml_reserved, physmem_fd, offset,
 			    len - offset, 1, 1, 1);
@@ -97,12 +105,6 @@ void __init setup_physmem(unsigned long start, unsigned long reserve_end,
 		exit(1);
 	}
 
-	/*
-	 * Special kludge - This page will be mapped in to userspace processes
-	 * from physmem_fd, so it needs to be written out there.
-	 */
-	os_seek_file(physmem_fd, __pa(&__syscall_stub_start));
-	os_write_file(physmem_fd, &__syscall_stub_start, PAGE_SIZE);
 
 	bootmap_size = init_bootmem(pfn, pfn + delta);
 	free_bootmem(__pa(reserve_end) + bootmap_size,
arch/um/os-Linux/file.c +4 -0
index b049a63..d4985be99 100644
@@ -33,6 +33,10 @@ static void copy_stat(struct uml_stat *dst, const struct stat64 *src)
 	});
 }
 
+int os_fsync(int fd) {
+   fsync(fd);
+}
+
 int os_stat_fd(const int fd, struct uml_stat *ubuf)
 {
 	struct stat64 sbuf;

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#3 lore raw

HI Richard, hi list.

I will reformat these (as well as the new drivers), clean them up to the current coding standard and resubmit so it will be easier to review them and consider form merging.

Apology for not coming back to this earlier - I was busy with the qemu counterparts of the transport patches (I will also apply the "lessons learned" from there to these).

A.

On 28/02/14 08:54, Anton Ivanov (antivano) wrote:

Bugfixes.

I need to pull actual changesets for the drivers, etc properly and verify that they build so those will be coming next week. You will be getting them one by one.

1. Memory corruption.

The reverse case of this race (you need to msync, before you do non-mmap fileops) is well known and textbook. This is the first and only time I have seen this one (fsync before mmap). I have not heard it mentioned either. It is however fairly easy to reproduce. If you run 200+ UML on a system ~0.2-0.5% will always die at startup with a memory corruption warning. While this does not happen every time (0.2-0.5% and only on startup) it is very reproducible for systems running lots of UMLs.

Once this fix went in we stopped seeing that one. Observed on 3.2, 3.3 and 3.8, fix tested on 3.2, 3.3, 3.4 and 3.8.

2. SIGPIPE.

Linux actually produces SIGPIPE ane EPIPE not only on missing reader. It will produce it under some circumstances on a stalled reader. Discovered when running UML under expect and/or trying to use fds and other virtual serials to do management transactions.

While I have not seen it on UML internal pipes I would not be surprised if you can reproduce it there too (f.e. if ubd thread is too slow). So SIGPIPE needs to be disabled. From there on, for most drivers have correct error handling for this.

Observed on 3.2, 3.3 and 3.8, fix tested on 3.2, 3.3, 3.4 and 3.8.

A.

On 28/02/14 08:33, Richard Weinberger wrote:

Am 28.02.2014 09:27, schrieb Anton Ivanov (antivano):

Hi Richard, Hi Jeff, hi list,

On behalf of Cisco systems, I am authorized to make a offer a set bug fixes as well as contribute several additional features and performance improvements to UML. All of these have been used internally for a couple of years and will ship as parts of product(s) in the near future. Some of these improve performance by up to 8 times on use cases which are of interest to us and are likely to be of interest to the community.

As the full patchset is now in the 100k+ zone, so I am going to do only the announcement now and submit the patches one by one after that over the next 1-2 weeks.

We will submit separately bug fixes for:

1. Critical memory corruption on startup observed on heavily loaded machines (especially when multiple UMLs run simultaneously).
2. Fix(es) for incorrect handling of error conditions when UML is run under expect and conX=fd: is used to communicate with another process. The same error may be observed on internal UML IPCs too leading to immediate crash.

I will also file bugs for both vs Debian UML package so that patches for both can go in ASAP.

In addition to the bug fixes, the new features include:

1. Several transports. All can do up to multi-gigabit throughput on some scenarios. We are contributing their counterparts to qemu/kvm as well.

1.1. Direct connection of UML to overlay networks/L2 VPNs using L2TPv3.

This has a number of advantages compared to the existing UML "multicast" and qemu "socket" transports.

* Standard compliant - RFC 3931 updated recently by RFC 5641
* Supported on most network equipment
* Allowing to move virtual switching off-host to an NPU or high
performance physical switch
* Allowing to mix virtual and physical switching (well supported on
modern Linuxes and other OSes)
* Well researched security profile as well as established
interactions with IPSEC allowing to extend virtual networks outside the datacenter to remote physical devices and/or VMs.

1.2. Raw transport which allows both bi-directional communication with any network device which looks like Ethernet as well as in-span listening at speeds in the multi-gigabit range.

1.3. We intend to contribute other key overlay transports like GRE, etc as well. The ones we are contributing at this point are the ones which we have used most extensively and have had the most testing (~ 1.5-2 years).

2. New high res timer subsystem

Adding these new network transports to UML revealed a key issue - it cannot meter or shape any traffic correctly as its internal timer system is way off. Personally, I consider it a bug, however there is no "easy" fix here. The only way to fix it is a new timer driver. Unfortunately, it does not fix uml userspace - timers there remain off. It does fix all kernel timer functionality - traffic shaping (both qdisc and iptables traffic limits).

As a side effect, this provides performance improvements for tcp and other protocols which rely on kernel high res timers for their state machines.

We have further scalability contributions lined up which improve network and IO performance between 1.5 and 8 times (depending on use case), allow hundreds of virtual interfaces per UML without performance penalties, allow to run several hundreds (if not thousands) of UMLs per machine, etc. All in all, it can no go where no virtualization and no virtual networking has gone before.

However, I would prefer to take it one step at a time and get through these first (even these are quite a lot for one "sitting").

Sounds awesome!

Please send the patches as soon as possible.
I'm eager to test and merge them.

Thanks,
//richard

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

_______________________________________________
User-mode-linux-devel mailing list [email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce. With Perforce, you get hassle-free workflows. Merge that actually works. Faster operations. Version large binaries. Built-in WAN optimization and the freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk

_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#4 lore raw
In reply to Richard Weinberger

Hi all,

First of all, if this does not apply cleanly somewhere, accept my apologies. The patchsets have grown organically for the last 1.5 years and I am having trouble disentangling some of them into distinct features. Feel free to bounce, I will be happy to fix it.

This is taken vs 3.3.8. The choice of kernel is because this is the production Openwrt AA kernel which what we use for development. It applies with minimal fuss from 3.2 to 3.8 (I have not tested it further).

Attached are the l2tpv3 and the raw transports.

Principles of operation:

UML original network infrastructure allocates a SKB at a time, reads into it using a driver read function and if the read is successful passes the skb to if_rx. I

This will not work for multi-packet receive unless you copy. Without multipacket receive it is difficult to go beyond 1.5Gbit for L2TPv3 and 1Gbit for raw. With multipacket you are looking at double that (at least). As an alternative to multipacket rx raw can also use packet mmap. While this incurs an extra copy the savings from fewer syscalls allow it to get fairly reasonable performance. Multipacket tx is not used for now.

To avoid the copy we pre-allocate a vector of skbs, a matching vector of supporting messages for recvmsg/sendmsg and matching iovs.

If the encapsulation has a distinct header (l2tpv3, gre, etc) we use readmsg (or readmmsg) to split that header from the packet payload on rx. If there is no header (raw) - we read directly. Same on xmit. So the structures for the former and the latter differ somewhat - the former uses a pointer to a two buffer iov (one fo header, one for payload), the latter a pointer to a single buffer iov in the msg_hdr structures. Otherwise, they are similar.

In order to avoid invoking kernel side functions directly the 3 kernel side bits we need to build these vectors are isolated in net_extra_kern.c

The userspace bits including wrappers around recvmsg, sendmsg and recvmmsg are in net_extra_user.c The same file contains all the infra to build the vectors.

When using vector io we no longer need drop_skb too. If there is no skb alloced (buffer length set to zero or buffer is NULL), recvmsg and recvmmsg will drop the packet for us.

Quoted 1 line
Quoted 1 line

From there on, the transports use this common infra.

The arguments for l2tpv3 are:

eth1=l2tpv3,,src,srcport,dst,dstport,rxcookie,txcookie,rxsession,txsession,unused,mode

mode is a bitmask (described in the uml_l2tpv3.h include file).

1           /* on for v6, off for v4 */
2           /* on for udp, off for raw ip */
4           /* cookie present */
8           /* on for 64 bit cookie */
16          /* draft keyed ip - no counter */

src, srcport - source, format according to the above

dst, dstport - destination (optional, if null will listen for first packet)

txcookie, rxcookie, txsession, rxsession - same as for ip l2tpv3 arguments

unused - older mode spec, not used now

mode - current (bitmask) mode spec

Examples:

eth1=l2tpv3,,192.168.64.1,,192.168.128.1,,0xdeadbeefdeadbeef,0xbeefdeadbeefdead,0xffffffff,0xffffffff,,c

This will configure raw ip v4 EoGRE tunnel from 64.1 to 128.1. The config is not very user friendly, but it is a bit difficult to make it so using current option parsing routines.

Raw is much simpler:

eth1=raw,,ifname

ifname is the name of the interface to bind to.

Note - the driver does not bring up the interface or set its mode. For most use cases you have to do an ifconfig ethX up promisc before binding to it for this to work.

A.

Quoted 85 lines · Richard Weinberger

Am 28.02.2014 09:27, schrieb Anton Ivanov (antivano):

Quoted 74 lines

Hi Richard, Hi Jeff, hi list,

On behalf of Cisco systems, I am authorized to make a offer a set bug fixes as well as contribute several additional features and performance improvements to UML. All of these have been used internally for a couple of years and will ship as parts of product(s) in the near future. Some of these improve performance by up to 8 times on use cases which are of interest to us and are likely to be of interest to the community.

As the full patchset is now in the 100k+ zone, so I am going to do only the announcement now and submit the patches one by one after that over the next 1-2 weeks.

We will submit separately bug fixes for:

1. Critical memory corruption on startup observed on heavily loaded machines (especially when multiple UMLs run simultaneously).
2. Fix(es) for incorrect handling of error conditions when UML is run under expect and conX=fd: is used to communicate with another process. The same error may be observed on internal UML IPCs too leading to immediate crash.

I will also file bugs for both vs Debian UML package so that patches for both can go in ASAP.

In addition to the bug fixes, the new features include:

1. Several transports. All can do up to multi-gigabit throughput on some scenarios. We are contributing their counterparts to qemu/kvm as well.

1.1. Direct connection of UML to overlay networks/L2 VPNs using L2TPv3.

This has a number of advantages compared to the existing UML "multicast" and qemu "socket" transports.

* Standard compliant - RFC 3931 updated recently by RFC 5641
* Supported on most network equipment
* Allowing to move virtual switching off-host to an NPU or high
performance physical switch
* Allowing to mix virtual and physical switching (well supported on
modern Linuxes and other OSes)
* Well researched security profile as well as established
interactions with IPSEC allowing to extend virtual networks outside the datacenter to remote physical devices and/or VMs.

1.2. Raw transport which allows both bi-directional communication with any network device which looks like Ethernet as well as in-span listening at speeds in the multi-gigabit range.

1.3. We intend to contribute other key overlay transports like GRE, etc as well. The ones we are contributing at this point are the ones which we have used most extensively and have had the most testing (~ 1.5-2 years).

2. New high res timer subsystem

Adding these new network transports to UML revealed a key issue - it cannot meter or shape any traffic correctly as its internal timer system is way off. Personally, I consider it a bug, however there is no "easy" fix here. The only way to fix it is a new timer driver. Unfortunately, it does not fix uml userspace - timers there remain off. It does fix all kernel timer functionality - traffic shaping (both qdisc and iptables traffic limits).

As a side effect, this provides performance improvements for tcp and other protocols which rely on kernel high res timers for their state machines.

We have further scalability contributions lined up which improve network and IO performance between 1.5 and 8 times (depending on use case), allow hundreds of virtual interfaces per UML without performance penalties, allow to run several hundreds (if not thousands) of UMLs per machine, etc. All in all, it can no go where no virtualization and no virtual networking has gone before.

Quoted 2 lines

However, I would prefer to take it one step at a time and get through these first (even these are quite a lot for one "sitting").

Sounds awesome!

Please send the patches as soon as possible.
I'm eager to test and merge them.

Thanks,
//richard

Quoted 11 lines

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

arch/um/Kconfig.net +16 -0
index 3160b1a..386dc12 100644
@@ -108,6 +108,22 @@ config UML_NET_DAEMON
         more than one without conflict.  If you don't need UML networking,
         say N.
 
+config UML_NET_RAW
+	bool "Raw transport"
+	depends on UML_NET
+	help
+	This User-Mode Linux network transport binds a VM ethX interface
+	to a host interface using raw sockets. The host interface is 
+	expected to have Ethernet framing
+
+config UML_NET_L2TPV3
+	bool "L2TPv3 transport"
+	depends on UML_NET
+	help
+	This User-Mode Linux network transport binds an IP address (v4 or v6)
+	and listens for incoming L2TPV3 frames. Frames are decapsulated
+	and presented to the VM. 
+
 config UML_NET_VDE
 	bool "VDE transport"
 	depends on UML_NET
arch/um/drivers/Makefile +8 -2
index e7582e1..ae30b83 100644
@@ -1,6 +1,7 @@
 # 
 # Copyright (C) 2000, 2002, 2003 Jeff Dike ([email protected])
 # Licensed under the GPL
+# Copyright (C) 2012 - 2014 Cisco Systems
 #
 
 # pcap is broken in 2.5 because kbuild doesn't allow pcap.a to be linked
@@ -9,8 +10,10 @@
 slip-objs := slip_kern.o slip_user.o
 slirp-objs := slirp_kern.o slirp_user.o
 daemon-objs := daemon_kern.o daemon_user.o
+uml_raw-objs := uml_raw_kern.o uml_raw_user.o
+uml_l2tpv3-objs := uml_l2tpv3_kern.o uml_l2tpv3_user.o
 umcast-objs := umcast_kern.o umcast_user.o
-net-objs := net_kern.o net_user.o
+net-objs := net_kern.o net_user.o net_extra_user.o net_extra_kern.o
 mconsole-objs := mconsole_kern.o mconsole_user.o
 hostaudio-objs := hostaudio_kern.o
 ubd-objs := ubd_kern.o ubd_user.o
@@ -43,6 +46,8 @@ obj-$(CONFIG_STDERR_CONSOLE) += stderr_console.o
 obj-$(CONFIG_UML_NET_SLIP) += slip.o slip_common.o
 obj-$(CONFIG_UML_NET_SLIRP) += slirp.o slip_common.o
 obj-$(CONFIG_UML_NET_DAEMON) += daemon.o 
+obj-$(CONFIG_UML_NET_RAW) += uml_raw.o 
+obj-$(CONFIG_UML_NET_L2TPV3) += uml_l2tpv3.o 
 obj-$(CONFIG_UML_NET_VDE) += vde.o
 obj-$(CONFIG_UML_NET_MCAST) += umcast.o
 obj-$(CONFIG_UML_NET_PCAP) += pcap.o
@@ -56,12 +61,13 @@ obj-$(CONFIG_PORT_CHAN) += port.o
 obj-$(CONFIG_PTY_CHAN) += pty.o
 obj-$(CONFIG_TTY_CHAN) += tty.o 
 obj-$(CONFIG_XTERM_CHAN) += xterm.o xterm_kern.o
+obj-$(CONFIG_UNIX_CHAN) += chan_unix.o
 obj-$(CONFIG_UML_WATCHDOG) += harddog.o
 obj-$(CONFIG_BLK_DEV_COW_COMMON) += cow_user.o
 obj-$(CONFIG_UML_RANDOM) += random.o
 
 # pcap_user.o must be added explicitly.
-USER_OBJS := fd.o null.o pty.o tty.o xterm.o slip_common.o pcap_user.o vde_user.o
+USER_OBJS := fd.o null.o pty.o tty.o xterm.o slip_common.o pcap_user.o vde_user.o chan_unix.o
 CFLAGS_null.o = -DDEV_NULL=$(DEV_NULL_PATH)
 
 include arch/um/scripts/Makefile.rules
arch/um/drivers/net_extra_kern.c new file +62 -0

62 lines not on this page · raw message

arch/um/drivers/net_extra_user.c new file +291 -0

291 lines not on this page · raw message

arch/um/drivers/net_kern.c +36 -17

104 lines not on this page · raw message

arch/um/drivers/net_user.c +3 -0
index 05090c3..bbfaf47 100644
@@ -1,4 +1,5 @@
 /*
+ * Copyright (C) 2012 - 2014 Cisco Systems
  * Copyright (C) 2001 - 2007 Jeff Dike (jdike@{addtoit,linux.intel}.com)
  * Licensed under the GPL
  */
@@ -120,6 +121,7 @@ int net_recvfrom(int fd, void *buf, int len)
 	return n;
 }
 
+
 int net_write(int fd, void *buf, int len)
 {
 	int n;
@@ -164,6 +166,7 @@ int net_sendto(int fd, void *buf, int len, void *to, int sock_len)
 	return n;
 }
 
+
 struct change_pre_exec_data {
 	int close_me;
 	int stdout;
arch/um/drivers/uml_l2tpv3.h new file +109 -0

109 lines not on this page · raw message

arch/um/drivers/uml_l2tpv3_kern.c new file +354 -0

354 lines not on this page · raw message

arch/um/drivers/uml_l2tpv3_user.c new file +391 -0

391 lines not on this page · raw message

arch/um/drivers/uml_raw.h new file +49 -0

49 lines not on this page · raw message

arch/um/drivers/uml_raw_kern.c new file +213 -0

213 lines not on this page · raw message

arch/um/drivers/uml_raw_user.c new file +151 -0

151 lines not on this page · raw message

arch/um/include/shared/net_kern.h +8 -0
index 5c367f2..6f4dd8e 100644
@@ -1,4 +1,5 @@
 /*
+ * Copyright (C) 2012 - 2014 Cisco Systems
  * Copyright (C) 2002 2007 Jeff Dike (jdike@{addtoit,linux.intel}.com)
  * Licensed under the GPL
  */
@@ -21,6 +22,8 @@ struct uml_net {
 	unsigned char mac[ETH_ALEN];
 };
 
+#define UML_NET_USE_SKB_READ 1
+
 struct uml_net_private {
 	struct list_head list;
 	spinlock_t lock;
@@ -29,6 +32,7 @@ struct uml_net_private {
 
 	struct work_struct work;
 	int fd;
+	unsigned int options;
 	unsigned char mac[ETH_ALEN];
 	int max_packet;
 	unsigned short (*protocol)(struct sk_buff *);
@@ -36,6 +40,7 @@ struct uml_net_private {
 	void (*close)(int, void *);
 	void (*remove)(void *);
 	int (*read)(int, struct sk_buff *skb, struct uml_net_private *);
+	struct sk_buff * (*skb_read)(struct uml_net_private *);
 	int (*write)(int, struct sk_buff *skb, struct uml_net_private *);
 
 	void (*add_address)(unsigned char *, unsigned char *, void *);
@@ -46,7 +51,9 @@ struct uml_net_private {
 struct net_kern_info {
 	void (*init)(struct net_device *, void *);
 	unsigned short (*protocol)(struct sk_buff *);
+	unsigned int options;
 	int (*read)(int, struct sk_buff *skb, struct uml_net_private *);
+	struct sk_buff * (*skb_read)(struct uml_net_private *);
 	int (*write)(int, struct sk_buff *skb, struct uml_net_private *);
 };
 
@@ -66,5 +73,6 @@ extern int tap_setup_common(char *str, char *type, char **dev_name,
 			    char **mac_out, char **gate_addr);
 extern void register_transport(struct transport *new);
 extern unsigned short eth_protocol(struct sk_buff *skb);
+extern struct sk_buff *my_build_skb(void * head, void *data, unsigned int frag_size);
 
 #endif
arch/um/include/shared/net_user.h +25 -0
index 3dabbe1..52f087a 100644
@@ -1,4 +1,5 @@
 /*
+ * Copyright (C) 2012 - 2014 Cisco Systems
  * Copyright (C) 2002 - 2007 Jeff Dike (jdike@{addtoit,linux.intel}.com)
  * Licensed under the GPL
  */
@@ -38,10 +39,15 @@ extern void tap_check_ips(char *gate_addr, unsigned char *eth_addr);
 extern void read_output(int fd, char *output_out, int len);
 
 extern int net_read(int fd, void *buf, int len);
+extern int net_readv(int fd, void *iov, int iovcnt);
 extern int net_recvfrom(int fd, void *buf, int len);
+extern int net_recvfrom2(int fd, void *buf, int len, void *src_addr, int *addrlen);
 extern int net_write(int fd, void *buf, int len);
+extern int net_writev(int fd, void *iov, int iovcnt);
 extern int net_send(int fd, void *buf, int len);
 extern int net_sendto(int fd, void *buf, int len, void *to, int sock_len);
+extern int net_sendmessage(int fd, void *msg, int flags);
+extern int net_recvmessage(int fd, void *msg, int flags);
 
 extern void open_addr(unsigned char *addr, unsigned char *netmask, void *arg);
 extern void close_addr(unsigned char *addr, unsigned char *netmask, void *arg);
@@ -50,4 +56,23 @@ extern char *split_if_spec(char *str, ...);
 
 extern int dev_netmask(void *d, void *m);
 
+
+/* net kern extra */
+extern void uml_net_destroy_skb(void * skb);
+extern void * uml_net_build_skb (void * dev);
+extern void * uml_net_skb_data (void * skb);
+
+extern void add_skbuffs(void * msgvec, void ** skbvec, int size, int skb_size, int offset);
+extern void add_header_buffers(void * msgvec, int size, int header_size);
+extern void * build_mmsg_vector(int size, int iovsize);
+extern void rebuild_skbuf_vector(void ** skbvec, int size, void * dev);
+extern void * build_skbuf_vector(int size, void * dev);
+extern int net_recvmmsg(int fd, void *msgvec, unsigned int vlen,
+                    unsigned int flags, struct timespec *timeout);
+extern int net_sendmmsg(int fd, void *msgvec, unsigned int vlen,
+                    unsigned int flags);
+extern void repair_mmsg (void *msgvec, int iovsize, int header_size);
+extern void destroy_skb_vector(void ** vector, int size);
+extern void destroy_mmsg_vector(void * mmsgvector, int size, int free_iov_base);
+
 #endif
arch/um/include/shared/os.h +3 -0
index 89b686c1..d2a4f4b 100644
@@ -1,4 +1,5 @@
 /*
+ * Copyright (C) 2012 - 2014 Cisco Systems
  * Copyright (C) 2002 - 2007 Jeff Dike (jdike@{addtoit,linux.intel}.com)
  * Licensed under the GPL
  */
@@ -274,6 +275,7 @@ extern void halt_skas(void);
 extern void reboot_skas(void);
 
 /* irq.c */
+
 extern int os_waiting_for_events(struct irq_fd *active_fds);
 extern int os_create_pollfd(int fd, int events, void *tmp_pfd, int size_tmpfds);
 extern void os_free_irq_by_cb(int (*test)(struct irq_fd *, void *), void *arg,
@@ -299,4 +301,5 @@ extern int get_pty(void);
 /* sys-$ARCH/task_size.c */
 extern unsigned long os_get_top_address(void);
 
+
 #endif

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel