Skip to contentSkip to conversation

Two links and a SMTP sever.

DISCUSSION Ulysses Almeida 3 participants 9 messages Jan 18, 2005 – Feb 7, 2005

lore
Share & tools
lore
#0 lore raw

Hi folks,

I have an unsolved problem with iptables + iproute2 that I can't
figure out how to solve it.

I explaint it on the attachment (i sent this attachment, because I
don't thin it will cause any trouble, since it's text/plain, if does, say and I won't send again). I wrote this as a mini-HowTo, and I pretend to publish this, but I need to solve the problem before doing this ;)

The strange thing. I have almost the same scenario on other host,
What differs is the fact that on the other host I have 4 Internet links, not only two, and everything works as I expected. I tried the same solution on another two hosts, and on both I had this releated problem.

If some one can help-me track this problem, I thank in advance.

Regars.

Signature · 1 line
 - Ulysses Almeida

I Have the following scenario:

- One server with two links:
* First link has dynamic IP (cheap one)
* Second links has fixed IP (expensive one)

- Link(1) is used by my intranet users, they use Internet services by this link. To do this, I just setuped Link(1) as my default gw. And made my server a linux router.

    # echo 1 > /proc/sys/net/ipv4/ip_forward
    # iptables -t nat -A POSTROUTING -s <intranet_net> -o <eth_link(1)> -j MASQUERADE 
    # ip route add default via <link(1)_gw> dev <eth_link(1)>

- Link(2) is used by some incomming requests from internet to this server. This services are HTTP and SMTP services. To use second link properly, I setuped iproute2

    # echo 100 link2 >> /etc/iproute2/rt_tables
    # ip route add <intranet_net> dev <eth_intranet> src <server_intranet_ip> table link2
    # ip route add <link(2)_net> dev <eth_link(2)> src <server_link(2)_ip> table link2
    # ip route add default via <link(2)_gw> dev <eth_link(2)> table link2
    # ip rule add from <server_link(2)_ip> lookup link2

______

             |      |	          __________  
             | S    | Link(1)    /	    \ 
             | E  S |<--------->| 	     |
  ________   | R  M |       	|  INTERNET  |
 |INTRANET|->| V  T | Link(2)   | 	     |
 '--------'  | E  P |<--------->| 	     |
             | R    |	         \__________/
             |______|		       	

Almost every thing works at this point. All my intranet users, are
using Link(1) to navigate on Internet, use MSN/ICQ/Jabber, and etc.
When someone on Internet want's to access my WEB page, they use
server_link(2)_ip to get into. The same when they want to accesse their e-mails. If some MTA on internet, needs to deliver some e-mail to my smtp server, it also uses the server_link(2)_ip. And it's work fine.

There's only one problem. If my MTA wants to deliver an e-mail, it
tries it by Link(1), what's reasonable, because that's the server default gateway. But it's not good, when the link(1) (dynamic IP) is a black listed one

To solve this problem, let's play with iptables and iproute2 a little more.

Every outgoing packet, with destination port set to 25, I will mark.

    # iptables -t mangle -A OUTPUT -p tcp --dport 25 -j MARK --set-mark 0x02

Now I can say to iproute2 use the right routing table.

    # ip rule add fwmark 0x02 lookup link2

Still not working, if i log outgoing packts, with destination port 25,
I see SMTP packets going through eth_link(2), but with server_link(1)_ip. Let's handle those packets a little more.

    # iptables -t nat -I POSTROUTING -o <eth_link(2)> -j SNAT --to-source <server_link(2)_ip>

Yes, now I'm sure every outgoing trafic through eth_link(2) is going
with the correct IP.

== UNSOLVED PROBLEM ==

But, argh, it still not working. I tried to track connections:

  # iptables -t mangle -m tcp -I OUTPUT -p tcp --dport 25 -j LOG --log-level 3 --log-prefix 'mangle OUTPUT: '
  # iptables -t nat -m tcp -I POSTROUTING -p tcp --dport 25 -j LOG --log-level 3 --log-prefix 'nat POSTROUTING: 
  # iptables -t mangle -m tcp -I PREROUTING -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'mangle PREROUTING: '
  # iptables -t nat -m tcp -I PREROUTING -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'nat PREROUTING: '
  # iptables -t mangle -m tcp -I INPUT -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'mangle INPUT: '
  # iptables -t filter -m tcp -I INPUT -p tcp --sport 25 -j LOG --log-level 3 --log-prefix 'filter INPUT: '

To simulate, I started a telnet some_smtp_server 25. I could see:

- packet on output mangle -> postrouting nat.
- pachet on prerouting mangle.

After packet appears on mangle's prerouting, it disappers? I can't
understando why.
How can I debug this?
Any one knows what can causinig it?

I'm using:

# uname -r
2.6.5-63255U10_3cl    (I think is something near 2.6.9, with conectiva patchs)
# iptables -V
iptables v1.2.11
# ip -V  
ip utility, iproute2-ss020116
#1 lore raw
Quoted 1 line · Ulysses Almeida

Hi folks,

Hi Ulysses!

Quoted 3 lines

I have an unsolved problem with iptables + iproute2 that I can't
figure out how to solve it.

(After looking attached file)
Did you enable NAT of local connection in your kernel ?
To be honest, I don't know anything about Lartc. And can't tell you how it should behave (my guess is it should pickup the right IP, of course)

However, the (ugly) SNAT work-around should work with NAT of local connection enabled.

Quoted 4 lines

I explaint it on the attachment (i sent this attachment, because I
don't thin it will cause any trouble, since it's text/plain, if does, say and I won't send again).

Please put it in the body so we can comment some part.

Quoted 7 lines

If some one can help-me track this problem, I thank in advance.

Regars.

--
- Ulysses Almeida

HTH,

Samuel

#2 lore raw

Thanks for the answer....

Quoted 9 lines · Samuel Jean

On Tue, January 18, 2005 8:39 am, Ulysses Almeida said:

(After looking attached file)
Did you enable NAT of local connection in your kernel ?
To be honest, I don't know anything about Lartc. And can't tell you how it should behave (my guess is it should pickup the right IP, of course)

However, the (ugly) SNAT work-around should work with NAT of local connection enabled.

Hmm. the NAT/SNAT is working fine, when I recive the SMTP answer
(LOG on PREROUTING and INPUT of filter, nat and mangle) the packet arrives with the correct ip on dst addr. But I can only see it, on mangle PREROUTING, but can't on other hooks. I don't know if it is related with some sanity check or checksum....

That's another doubt, dos nat PREROUTING called with syn+ack
packets, or only with sym packets?

I have these modules loaded:

ipt_LOG                 5760  7 
ipt_mark                1792  0 
ipt_MARK                2176  1 
iptable_mangle          2944  1 
ipt_state               2048  8 
iptable_filter          2944  1 
ipt_MASQUERADE          3840  4 
iptable_nat            23980  2 ipt_MASQUERADE
ip_conntrack           33072  3 ipt_state,ipt_MASQUERADE,iptable_nat
ip_tables              17920  8

ipt_LOG,ipt_mark,ipt_MARK,iptable_mangle,ipt_state,iptable_filter,ipt_MASQUERADE,iptable_nat

Quoted 8 lines
Quoted 5 lines

I explaint it on the attachment (i sent this attachment, because I
don't thin it will cause any trouble, since it's text/plain, if does, say and I won't send again).

Please put it in the body so we can comment some part.

Ok, I used to use MUA that shows-me the text/plain attachments as the e-mail body (like mutt). But now I see it won't happen all the time. My mistake, sorry.

Signature · 1 line
 - Ulysses Almeida
#3 lore raw
Quoted 8 lines · Ulysses Almeida

Hmm. the NAT/SNAT is working fine, when I recive the SMTP answer
(LOG on PREROUTING and INPUT of filter, nat and mangle) the packet arrives with the correct ip on dst addr. But I can only see it, on mangle PREROUTING, but can't on other hooks. I don't know if it is related with some sanity check or checksum....

That's another doubt, dos nat PREROUTING called with syn+ack
packets, or only with sym packets?

the nat table is special, it only sees the first packet of a connection.

All other subsequent packet of this connection get verdict via the conntrack. (bypass NAT chains)

Quoted 5 lines

--
- Ulysses Almeida

Samuel

#4 lore raw

How can I debug, what happens after mangle PREROUTING. Can I do it
only for packets with src port set to 25? to get a "clear" output?

Quoted 9 lines · Samuel Jean
Quoted 2 lines

That's another doubt, dos nat PREROUTING called with syn+ack
packets, or only with sym packets?

the nat table is special, it only sees the first packet of a connection.

All other subsequent packet of this connection get verdict via the conntrack. (bypass NAT chains)

So this conntrack entry is created while my syn packet is going
out.... Can something wrong happening here, and conntrack gives the wrong veridict?

Quoted 4 lines

Samuel

Signature · 1 line
 - Ulysses Almeida
#5 lore raw
Quoted 2 lines · Ulysses Almeida

How can I debug, what happens after mangle PREROUTING. Can I do it
only for packets with src port set to 25? to get a "clear" output?

If you are willing to debug, use TRACE target from pom. (at least 2 patches are needed).

Quoted 9 lines
Quoted 3 lines · Samuel Jean

All other subsequent packet of this connection get verdict via the conntrack. (bypass NAT chains)

So this conntrack entry is created while my syn packet is going
out.... Can something wrong happening here, and conntrack gives the wrong veridict?

As I said, did you enable this ? :

CONFIG_IP_NF_NAT_LOCAL

Cheers,

Samuel

#6 lore raw
Quoted 5 lines · Samuel Jean

As I said, did you enable this ? :

CONFIG_IP_NF_NAT_LOCAL

Yes CONFIG_IP_NF_NAT_LOCAL=y

I did put some -j LOG on target host. Tried a telnet to cited host
on port 25. On
target's host everything looks right. I get the packets from the right ip (link(2)). But it still disapper after mangles PREROUTING. Maybe i'm missing something on second routing table.... I'll try to find out.

It's strange, because I have other host, with this really same
solution, and everything works fine.

Signature · 1 line
 - Ulysses Almeida
#7 lore raw
In reply to Ulysses Almeida
Quoted 2 lines · Ulysses Almeida

That's another doubt, dos nat PREROUTING called with syn+ack
packets, or only with sym packets?

nat PREROUTING sees NEW packets, without regard to TCP flags.

-j
--
"Oh, people can come up with statistics to prove anything, Kent. 14%
of people know that."
--The Simpsons

#8 lore raw
In reply to Ulysses Almeida

Some time ago, I wrote this e-mail below. A lot of people tried to
help-me (thanks guys), but I couldn't find the problem. A week ago, with help from someone on IRC I solved my problem (simple one). And I finished my MiniHowTo.
http://munky.maluco.com.br/lar_with_smtp.html
Maybe it won't tell you something new. But it's shorter do read
instead the whole LARTC documentation.
This miniHowTo shows how to use iproute2 with iptables mark rule. I
think it's nice. If someone has something to say about it (in private), I would thanks.

PS: The problem was just Reverse Path Filtering enabled.

Quoted 26 lines · Ulysses Almeida

Hi folks,

I have an unsolved problem with iptables + iproute2 that I can't
figure out how to solve it.

I explaint it on the attachment (i sent this attachment, because I
don't thin it will cause any trouble, since it's text/plain, if does, say and I won't send again). I wrote this as a mini-HowTo, and I pretend to publish this, but I need to solve the problem before doing this ;)

The strange thing. I have almost the same scenario on other host,
What differs is the fact that on the other host I have 4 Internet links, not only two, and everything works as I expected. I tried the same solution on another two hosts, and on both I had this releated problem.

If some one can help-me track this problem, I thank in advance.

Regars.

--
- Ulysses Almeida

Signature · 1 line
 - Ulysses Almeida