Skip to contentSkip to conversation

Load Balance/Redundant Internet Connections

LARTC William L. Thomson Jr. 2 participants 4 messages Apr 28, 2002 – Jun 5, 2002

lore
Share & tools
lore
#0 lore raw

I know this is a question that has been asked a million times over but here goes.

I have two SDSL lines that I would like to load balance and/or provided redundant internet access.

Now since these lines come from two different ISPs I have two blocks of IPs, one block for each line.

Now I have servers that I would like to be able to provided public access using two different public IP's. Two public IP for each server, one from each ISP block, that will be port address translated to a single internal IP.

This brings about many different problems, one which lies in the way I do DNS. At the moment if you talk to NS1 you will get public IPs for each server using ISP1's SDSL line and public IP block. If you talk to NS2 you will get public IPs for each server using ISP2's SDSL line and public IP block.

Now I have tried to read and experiment with all the examples on the net I could find including the following links which are the main ones I came across several times.

http://www.samag.com/documents/s24/sam0201h/0201h.htm
http://www.linuxvirtualserver.org/~julian/nano.txt
http://www.linuxvirtualserver.org/~julian/dgd-usage.txt

Now with each of theses I run into a similar issue. That I will get to in a moment.

I also tried buying a cheap solution, NexLand's ISBPro800, which in short does not work. NexLand is aware of the problems I have experienced, and have been working on a new firmware release for over two months now. I am not sure if they will get it to work or not.

The only other choices of solutions I can buy are out of my price range at the moment. One is $5000, and the other starts at $7500.00. Here are links in order of pricing,
http://www.sysmaster.com/products.htm
http://www.fatpipeinc.com/

Now it is rumored that the second runs a modified flavor of Linux.

So far the closest I have ever been able to get using Linux as a load balancing/redundant gateway router has some issues when it comes to routes existing in the route cache table.

If you establish a connection to a server using a public IP from ISP1, and then go and establish a connection to the same server using a public IP from ISP2, you cannot until after the routing cache has been flushed.

I tried playing around with the garbage collection and other params that are supposed to control the frequency that the routing cache is flushed. I was not able to reach a happy medium.

I was talking with a engineer buddy of mine from Cisco and he said I needed proper aging to take place in my route cache. So if a new connection is attempted and an older different route exists in the routing cache, the older one will be dropped/removed to allow the new one in and traffic to flow. Makes sense, but I have no clue how to implement this, if it can be done outside of the kernel itself.

So the bottom line is I can get it to work, providing that each and ever requests made to the server uses the same public IP from one or the other ISP. If there is a need to switch to the other line/ip then it will have to occur once the route has been cleared out of my routers cache, and possible the requesting machines cache as well.

I am starting to think that this cannot be done properly with the current state of the linux kernel's multipath routing. Do I have to buy a >$5000.00 unit to do this?

It seems that there are allot of people currently, and the more people switch to broadband I could see it only increasing, trying to do the same thing I am. So there really needs to be a solution for this. I will document a working solution if and when I come across one.

Being that a couple companies are providing products that can do what I am attempting to, it seems this is something that the linux kernel hackers should look into adding or improving.

If I had the proper knowledge of C/C++ and the linux kernel I would work on a solution myself. I am trying to envision a shell script or program outside of the kernel that could help out. But where I run into problems is for my program to be aware of each and every connection request and making a decision about the state of or content in the route cache without depleting the machine of it's resources.

Is there anybody out there that has been able to get this to work? If so, how? What I am doing wrong?

I can provide examples and anything needed.

FYI I am currently using the Linux Router release of the 2.2 kernel, which is the linux router 2.9.8beta and is based on kernel 2.2.19. I think.

I also attempted it with a newer kernel, 2.4.9-31 from Redhat, that was recompiled by me. Both lead me back to the same problem with the route cache.

I am getting desperate and about ready to drop $5000.00, although I will not do it without kicking and screaming.

Please help, anybody, I will try to make it up to you, so it is worth your time.

Thank you in advance.

Signature · 13 lines
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#1 lore raw

Hello,

Quoted 3 lines · William L. Thomson Jr.

If you establish a connection to a server using a public IP from ISP1, and then go and establish a connection to the same server using a public IP from ISP2, you cannot until after the routing cache has been flushed.

This sounds wrong. The connections from different pub IPs
differ, so they use different cache entries. But it is true when NAT is used. Then the INTIP->REMOTESERVER is cached.

Quoted 3 lines

I tried playing around with the garbage collection and other params that are supposed to control the frequency that the routing cache is flushed. I was not able to reach a happy medium.

Can this help?:

echo 5 > /proc/sys/net/ipv4/route/gc_timeout
echo 2 > /proc/sys/net/ipv4/route/gc_interval

Quoted 1 line

Thank you in advance.

Regards

--
Julian Anastasov <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to William L. Thomson Jr.

Ok,
I have removed the Nexland ISBPro800, and put our linux router back
into use. My initial testing was not to bad. Not exactly what I was looking for but it seems to be the closest I have gotten so far.

Quoted 7 lines · Julian Anastasov

Hello,

Quoted 3 lines · William L. Thomson Jr.

I tried playing around with the garbage collection and other params that are supposed to control the frequency that the routing cache is flushed. I was not able to reach a happy medium.

Quoted 7 lines

Can this help?:

echo 5 > /proc/sys/net/ipv4/route/gc_timeout
echo 2 > /proc/sys/net/ipv4/route/gc_interval

--
Julian Anastasov <[email protected]>

Thanks Julian, this setting seems to improve performance a little. I doubt to many people will be bouncing back and forth between the IP's from different ISP. It will have more to do with what the DNS lookup's return.

DNS not really being a topic for this list, but directly related to the outcome of what I am attempting to do. So please forgive me asking these questions here. If you know of a better place please let me know.

Anyway, if anyone reading this could take a few seconds and click on the following links. Both of my SDSL lines are up and all servers and services should be available if everything is correct in the linux router.

Please feel free to email the results of your using the following links to me. Also please click beyond the initial link just to make sure further lookups are the same.

Links
http://www.obsidian-studios.com
ftp://ftp.obsidian-studios.com
http://www.elvisplace.com
http://www.suzakstrading.com

The others are mail and DB related and will have to be tested out by clients. These are available to the general public and I am curious to know if the general public has any problems.

Once again for clarification I am using two SDSL lines as a single gateway for my internal singles. Hopefully this will allow me to use both lines. Possible providing redundant service. If I am lucky a simple form of load balancing, although by load balancing I mean just being able to use both lines as one. Having the linux router determine the use of each line and balancing that out is beyond my expectations at the moment. But would be nice.

So at the time I have this router in place, and if the above links do not work, it is due to the router's config.

Thank you very much for anyone taking to time to click and provide feedback.

Also I am open to comments or suggestions from anyone about my config.

I do plan to share the results with others so they will not be faced with the same problems. Much less have to buy a Nexland unit or other piece of hardware that may or may not work.

Signature · 13 lines
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#3 lore raw
In reply to William L. Thomson Jr.

Well after many months, a failed attempt at solving my problem with a Nexland ISBPro800, I finally now am using both of my SDSL lines together.

Bliss. :)

This was not an easy process, and I ended up going through just about every possible scenario until finding the one that worked.

I would not have found the one tha worked had it not been for the assistance of

Julian Anastasov

He responded to many direct emails, and was very patient and cooperative.

His patches are a must in a situation like this.
http://www.linuxvirtualserver.org/~julian/#routes

Also stick to the examples shown in the nano-how-to
http://www.linuxvirtualserver.org/~julian/nano.txt

What I had problems with and what is a must is that the Linux router must perform some sort of NAT.

In my situation it was a matter of using

ipmasqadm portfw

to do DNAT on my public IP's

and a

ipchains -A forward -s w.x.y.z -j MASQ

One the internal network.

I am willing to answer questions and help others out. That's just my way of giving back what I have received. Someone help's me, I help someone else.

So others heading down this path, it is possible, but there is only one way at the present moment to go about it.

Have a nice day, and route on.

Signature · 13 lines
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/