Isolating two network processes on same machine
Hi,
We need to test a peer-to-peer network application that is supposed to be scalable. To that end, we have a FreeBSD box with dummynet and a small cluster of Linux test machines. The box act as the gateway for the test machines and delay incoming packets for a while before throwing them back to the cluster to simulate latency on the Internet.
By letting the test machines think they run on separate subnets, we have been able to fool them into forwarding their packets to the FreeBSD gateway even though everyone is connected to the same switch. This is working fine.
The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with
ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255 ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255
and start two instances on the same machine with the two IP addresses, then they communicate directly with each other instead of going through the FreeBSD gateway. Can anyone see a way to solve this problem?
(I've CC'ed the other guys in my group.)
Signature · 2 lines
Ole Laursen http://www.cs.aau.dk/~olau/
Quoted 17 lines · Ole Laursen
Hi,
We need to test a peer-to-peer network application that is supposed to be scalable. To that end, we have a FreeBSD box with dummynet and a small cluster of Linux test machines. The box act as the gateway for the test machines and delay incoming packets for a while before throwing them back to the cluster to simulate latency on the Internet.
By letting the test machines think they run on separate subnets, we have been able to fool them into forwarding their packets to the FreeBSD gateway even though everyone is connected to the same switch. This is working fine.
The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with
Quoted 8 lines
ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255 ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255
and start two instances on the same machine with the two IP addresses, then they communicate directly with each other instead of going through the FreeBSD gateway. Can anyone see a way to solve this problem?
I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway.... But there is a bug somewhere that doesn't allow a netmask of anything but 0 in the last byte.
So, just add a host route....
route add -host 10.0.1.2 gw server
Quoted 7 lines
(I've CC'ed the other guys in my group.)
Signature · 3 lines
Ole Laursen http://www.cs.aau.dk/~olau/ -
FYI, probably nobody will admit to it being a bug, but it's another example of policy spreading throughout the kernel. If I set the netmask to 0.0.0.0 or 255.255.255.255, and anything in-between, it should let me....
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.
98.36% of all statistics are fiction.
Quoted 5 lines · linux-os
I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway.... But there is a bug somewhere that doesn't allow a netmask of anything but 0 in the last byte.
Really? That would be a horrible bug. How about some references?
Lee
<<< No Message Collected >>>
Quoted 16 lines · linux-os
Quoted 11 lines · Lee Revell
Quoted 5 lines · linux-os
I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway.... But there is a bug somewhere that doesn't allow a netmask of anything but 0 in the last byte.
Really? That would be a horrible bug. How about some references?
Lee
Huh? Try it........
Quoted 8 lines
Script started on Wed 24 Nov 2004 12:03:43 PM EST
# ifconfig eth1 1.2.3.4 netmask 255.255.255.7 SIOCSIFNETMASK: Invalid argument # ifconfig eth1 1.2.3.4 netmask 255.255.255.0 # exitScript done on Wed 24 Nov 2004 12:04:42 PM EST
Well I guess you can set it to:
eth1 Link encap:Ethernet HWaddr 00:10:5A:27:7B:16
inet addr:1.3.4.5 Bcast:1.255.255.255 Mask:255.255.255.252
^^^
inet6 addr: fe80::210:5aff:fe27:7b16/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:68120 errors:0 dropped:0 overruns:0 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:3
collisions:0 txqueuelen:1000
RX bytes:4383953 (4.1 Mb) TX bytes:378 (378.0 b)
Interrupt:22 Base address:0xbc00That should ne able to isolate the two machines on the same wire.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.
98.36% of all statistics are fiction.
Quoted 3 lines · linux-os
I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway....
Yeah, but that part of it is actually working as long as our processes are running on different machines. The problem is that on the same machine e.g. with this configuration
Quoted 2 lines
Quoted 2 lines
ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255 ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255
then the kernel somehow shortcircuits the routing table and doesn't forward the packets to the default gateway, even though the two addresses are on different subnets. It probably somehow knows that it possesses both IPs itself, and then skip any further routing.
So basically, our problem is that the kernel is being too clever. If we could just dumb it down or trick it somehow...
Thanks for your input,
Signature · 2 lines
Ole Laursen http://www.cs.aau.dk/~olau/
Quoted 4 lines
The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with
The easiest solution is probably to have the FreeBSD box DNAT the linux boxes so they don't know they are talking to themselves. Then you only need to use 1 ip address per linux box.
Ross
Quoted 7 lines · Ross Biro
Quoted 3 lines
The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with
The easiest solution is probably to have the FreeBSD box DNAT the linux boxes so they don't know they are talking to themselves. Then you only need to use 1 ip address per linux box.
Thanks, DNAT seems to be a good solution.
I think we will let the Linux boxes use DNAT to send the packets to the FreeBSD box and then let that use DNAT to send them back again. This way we won't have to change our test program, which would be a bit complicated because the addresses of the peers is an integral part of the design.
Though it would have been simpler if the kernel supported blindly forwarding a packet to another host without messing with the IP destination address, but that does not seem to be the case. It could have saved us from the double DNAT.
Thanks again,
Signature · 2 lines
Ole Laursen http://www.cs.aau.dk/~olau/