Skip to contentSkip to conversation

Isolating two network processes on same machine

DISCUSSION Ole Laursen 4 participants 8 messages Nov 24, 2004 – Nov 27, 2004

lore
Share & tools
lore
#0 lore raw

Hi,

We need to test a peer-to-peer network application that is supposed to be scalable. To that end, we have a FreeBSD box with dummynet and a small cluster of Linux test machines. The box act as the gateway for the test machines and delay incoming packets for a while before throwing them back to the cluster to simulate latency on the Internet.

By letting the test machines think they run on separate subnets, we have been able to fool them into forwarding their packets to the FreeBSD gateway even though everyone is connected to the same switch. This is working fine.

The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with

ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255 ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255

and start two instances on the same machine with the two IP addresses, then they communicate directly with each other instead of going through the FreeBSD gateway. Can anyone see a way to solve this problem?

(I've CC'ed the other guys in my group.)

Signature · 2 lines
Ole Laursen
http://www.cs.aau.dk/~olau/
#1 lore raw
Quoted 17 lines · Ole Laursen

Hi,

We need to test a peer-to-peer network application that is supposed to be scalable. To that end, we have a FreeBSD box with dummynet and a small cluster of Linux test machines. The box act as the gateway for the test machines and delay incoming packets for a while before throwing them back to the cluster to simulate latency on the Internet.

By letting the test machines think they run on separate subnets, we have been able to fool them into forwarding their packets to the FreeBSD gateway even though everyone is connected to the same switch. This is working fine.

The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with

Quoted 8 lines

ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255 ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255

and start two instances on the same machine with the two IP addresses, then they communicate directly with each other instead of going through the FreeBSD gateway. Can anyone see a way to solve this problem?

I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway.... But there is a bug somewhere that doesn't allow a netmask of anything but 0 in the last byte.

So, just add a host route....

route add -host 10.0.1.2 gw server

Quoted 7 lines

(I've CC'ed the other guys in my group.)

Signature · 3 lines
Ole Laursen
http://www.cs.aau.dk/~olau/
-

FYI, probably nobody will admit to it being a bug, but it's another example of policy spreading throughout the kernel. If I set the netmask to 0.0.0.0 or 255.255.255.255, and anything in-between, it should let me....

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.
98.36% of all statistics are fiction.

#2 lore raw
Quoted 5 lines · linux-os

I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway.... But there is a bug somewhere that doesn't allow a netmask of anything but 0 in the last byte.

Really? That would be a horrible bug. How about some references?

Lee

#4 lore raw
Quoted 16 lines · linux-os
Quoted 11 lines · Lee Revell
Quoted 5 lines · linux-os

I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway.... But there is a bug somewhere that doesn't allow a netmask of anything but 0 in the last byte.

Really? That would be a horrible bug. How about some references?

Lee

Huh? Try it........

Quoted 8 lines

Script started on Wed 24 Nov 2004 12:03:43 PM EST

# ifconfig eth1 1.2.3.4 netmask 255.255.255.7
SIOCSIFNETMASK: Invalid argument
# ifconfig eth1 1.2.3.4 netmask 255.255.255.0
# exit

Script done on Wed 24 Nov 2004 12:04:42 PM EST

Well I guess you can set it to:

eth1 Link encap:Ethernet HWaddr 00:10:5A:27:7B:16

          inet addr:1.3.4.5  Bcast:1.255.255.255  Mask:255.255.255.252
                                                                   ^^^
          inet6 addr: fe80::210:5aff:fe27:7b16/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:68120 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:3
          collisions:0 txqueuelen:1000
          RX bytes:4383953 (4.1 Mb)  TX bytes:378 (378.0 b)
          Interrupt:22 Base address:0xbc00

That should ne able to isolate the two machines on the same wire.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by John Ashcroft.
98.36% of all statistics are fiction.

#5 lore raw
In reply to linux-os
Quoted 3 lines · linux-os

I was going to say, set the netmask small enough so that both machines are on different networks and set default routes to your gateway....

Yeah, but that part of it is actually working as long as our processes are running on different machines. The problem is that on the same machine e.g. with this configuration

Quoted 2 lines
Quoted 2 lines

ifconfig eth0:0 10.0.0.2 netmask 255.255.255.0 broadcast 10.0.0.255 ifconfig eth0:1 10.0.1.2 netmask 255.255.255.0 broadcast 10.0.1.255

then the kernel somehow shortcircuits the routing table and doesn't forward the packets to the default gateway, even though the two addresses are on different subnets. It probably somehow knows that it possesses both IPs itself, and then skip any further routing.

So basically, our problem is that the kernel is being too clever. If we could just dumb it down or trick it somehow...

Thanks for your input,

Signature · 2 lines
Ole Laursen
http://www.cs.aau.dk/~olau/
#6 lore raw
In reply to Ole Laursen
Quoted 4 lines

The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with

The easiest solution is probably to have the FreeBSD box DNAT the linux boxes so they don't know they are talking to themselves. Then you only need to use 1 ip address per linux box.

Ross

#7 lore raw
Quoted 7 lines · Ross Biro
Quoted 3 lines

The problem is that we need to run several instances of our network application on the same test machine since we have too few machines. But when we create two IP addresses on the same machine with

The easiest solution is probably to have the FreeBSD box DNAT the linux boxes so they don't know they are talking to themselves. Then you only need to use 1 ip address per linux box.

Thanks, DNAT seems to be a good solution.

I think we will let the Linux boxes use DNAT to send the packets to the FreeBSD box and then let that use DNAT to send them back again. This way we won't have to change our test program, which would be a bit complicated because the addresses of the peers is an integral part of the design.

Though it would have been simpler if the kernel supported blindly forwarding a packet to another host without messing with the IP destination address, but that does not seem to be the case. It could have saved us from the double DNAT.

Thanks again,

Signature · 2 lines
Ole Laursen
http://www.cs.aau.dk/~olau/