Skip to contentSkip to conversation

NAT not working on one of 3 firewalls

DISCUSSION George Garvey 2 participants 6 messages Jun 24, 2002 – Jun 25, 2002 1 message not in the archive

lore
Share & tools
lore

George Garvey

author
#0 lore raw

I'm using almost the same iptables scripts on 3 systems. 2 will do NAT, 1 won't.

The 2 that do both have 3 NICs, 1 connected to a switch for a LAN, 2 connected to DSL lines. The one that doesn't has 2 NICs, 1 for a LAN, 1 for a DSL.

The DSL I'm having problems with is connected through a router that translates a single IP to 5 IPs, only one of which I'm using. I'm told by the ISP that I can use any 2 of the 5 on the internet. At least, that's my understanding.

This system also has a GRE tunnel. I've turned of IPSEC until I get the nat worked out.

I've attached a dump from iptables. I did a search/replace to change the internet IPs to a unique identifier in the dump. If you need the IPs, I'll supply them.

The LAN works fine. Internet with the firewall works fine.

If I ping an internet IP from the LAN, I'm pretty sure it goes out to the internet with the source IP still the LAN IP, without translation.

I have no clue what I'm doing wrong. Any assistance will be appreciated.

XX.XXX.XXX.XXX is the IP on eth1 of the firewall. YY.YYY.YYY.YYY is the IP at the other end of the GRE tunnel.

Chain INPUT (policy ACCEPT 5 packets, 372 bytes)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0          
    2   152 ACCEPT     all  --  eth0   *       192.168.2.0/24       0.0.0.0/0          
    4   208 ACCEPT     all  --  eth1   *       XX.XXX.XXX.XXX/29    XX.XXX.XXX.XXX/29  
    0     0 log-and-rej-in  all  --  eth1   *       192.168.2.0/24       0.0.0.0/0          
    0     0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     
  745 66748 ACCEPT     all  --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     state RELATED,ESTABLISHED 
    0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     state NEW,RELATED,ESTABLISHED multiport dports 113,123,209 
    0     0 ACCEPT     udp  --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     state NEW,RELATED,ESTABLISHED multiport dports 123,500,50,51 
    0     0 ACCEPT     47   --  eth1   *       YY.YYY.YYY.YYY         XX.XXX.XXX.XXX     
    0     0 ACCEPT     all  --  withvan *       192.168.3.1          192.168.2.0/24     
    0     0 ACCEPT     all  --  withvan *       192.168.1.12         XX.XXX.XXX.XXX     
  715 47612 ACCEPT     all  --  withvan *       192.168.1.0/24       192.168.2.2        
    0     0 ACCEPT     all  --  withvan *       192.168.1.0/24       192.168.3.2        
    0     0 log-and-rej-in  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  eth0   withvan  0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     all  --  withvan eth0    0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     all  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0          state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0          
    0     0 log-and-rej-fwd  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 65 packets, 12174 bytes)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     all  --  *      eth0    XX.XXX.XXX.XXX       192.168.2.0/24     
   32  5342 ACCEPT     all  --  *      eth0    192.168.2.2          192.168.2.0/24     
    0     0 ACCEPT     icmp --  *      eth1    XX.XXX.XXX.XXX       0.0.0.0/0          
    0     0 ACCEPT     tcp  --  *      eth1    0.0.0.0/0            0.0.0.0/0          tcp spt:209 
    0     0 ACCEPT     all  --  *      withvan  192.168.3.2          192.168.1.0/24     
  425 46996 ACCEPT     all  --  *      withvan  192.168.2.2          192.168.1.0/24     
    0     0 ACCEPT     all  --  *      withvan  192.168.2.2          192.168.3.1        
    0     0 log-and-rej-out  all  --  *      eth1    0.0.0.0/0            192.168.2.0/24     
  449 58860 ACCEPT     all  --  *      eth1    XX.XXX.XXX.XXX       0.0.0.0/0          
    0     0 log-and-rej-out  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain log-and-rej-fwd (1 references)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0          LOG flags 0 level 7 prefix `REJECT-FWD: ' 
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0          reject-with icmp-port-unreachable

Chain log-and-rej-in (2 references)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0          LOG flags 0 level 7 prefix `REJECT-INP: ' 
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0          reject-with icmp-port-unreachable

Chain log-and-rej-out (2 references)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0          LOG flags 0 level 7 prefix `REJECT-OUT: ' 
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0          reject-with icmp-port-unreachable

MANGLE

Chain PREROUTING (policy ACCEPT 2026 packets, 166K bytes)
pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 1988 packets, 163K bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 25 packets, 1872 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 16420 packets, 2797K bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 16445 packets, 2799K bytes)
pkts bytes target prot opt in out source destination

FILTER

Chain INPUT (policy ACCEPT 5 packets, 372 bytes)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0          
    2   152 ACCEPT     all  --  eth0   *       192.168.2.0/24       0.0.0.0/0          
    4   208 ACCEPT     all  --  eth1   *       XX.XXX.XXX.XXX/29    XX.XXX.XXX.XXX/29  
    0     0 log-and-rej-in  all  --  eth1   *       192.168.2.0/24       0.0.0.0/0          
    0     0 ACCEPT     icmp --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     
  745 66748 ACCEPT     all  --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     state RELATED,ESTABLISHED 
    0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     state NEW,RELATED,ESTABLISHED multiport dports 113,123,209 
    0     0 ACCEPT     udp  --  eth1   *       0.0.0.0/0            XX.XXX.XXX.XXX     state NEW,RELATED,ESTABLISHED multiport dports 123,500,50,51 
    0     0 ACCEPT     47   --  eth1   *       YY.YYY.YYY.YYY         XX.XXX.XXX.XXX     
    0     0 ACCEPT     all  --  withvan *       192.168.3.1          192.168.2.0/24     
    0     0 ACCEPT     all  --  withvan *       192.168.1.12         XX.XXX.XXX.XXX     
  715 47612 ACCEPT     all  --  withvan *       192.168.1.0/24       192.168.2.2        
    0     0 ACCEPT     all  --  withvan *       192.168.1.0/24       192.168.3.2        
    0     0 log-and-rej-in  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  eth0   withvan  0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     all  --  withvan eth0    0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     all  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0          state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0          
    0     0 log-and-rej-fwd  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 65 packets, 12174 bytes)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     all  --  *      eth0    XX.XXX.XXX.XXX       192.168.2.0/24     
   32  5342 ACCEPT     all  --  *      eth0    192.168.2.2          192.168.2.0/24     
    0     0 ACCEPT     icmp --  *      eth1    XX.XXX.XXX.XXX       0.0.0.0/0          
    0     0 ACCEPT     tcp  --  *      eth1    0.0.0.0/0            0.0.0.0/0          tcp spt:209 
    0     0 ACCEPT     all  --  *      withvan  192.168.3.2          192.168.1.0/24     
  425 46996 ACCEPT     all  --  *      withvan  192.168.2.2          192.168.1.0/24     
    0     0 ACCEPT     all  --  *      withvan  192.168.2.2          192.168.3.1        
    0     0 log-and-rej-out  all  --  *      eth1    0.0.0.0/0            192.168.2.0/24     
  449 58860 ACCEPT     all  --  *      eth1    XX.XXX.XXX.XXX       0.0.0.0/0          
    0     0 log-and-rej-out  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain log-and-rej-fwd (1 references)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0          LOG flags 0 level 7 prefix `REJECT-FWD: ' 
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0          reject-with icmp-port-unreachable

Chain log-and-rej-in (2 references)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0          LOG flags 0 level 7 prefix `REJECT-INP: ' 
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0          reject-with icmp-port-unreachable

Chain log-and-rej-out (2 references)

 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0          LOG flags 0 level 7 prefix `REJECT-OUT: ' 
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0          reject-with icmp-port-unreachable

NAT

Chain PREROUTING (policy ACCEPT 26 packets, 1872 bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 16 packets, 1439 bytes)

 pkts bytes target     prot opt in     out     source               destination         
   16  1184 SNAT       all  --  *      eth1    0.0.0.0/0            0.0.0.0/0          to:XX.XXX.XXX.XXX

Chain OUTPUT (policy ACCEPT 43 packets, 3409 bytes)
pkts bytes target prot opt in out source destination

#1 lore raw
Quoted 4 lines · George Garvey

The DSL I'm having problems with is connected through a router that translates a single IP to 5 IPs, only one of which I'm using. I'm told by the ISP that I can use any 2 of the 5 on the internet. At least, that's my understanding.

Um, what are the other three for, then ? If you can only use 2 IPs, why has
the ISP given you any more ?

Quoted 2 lines

This system also has a GRE tunnel. I've turned of IPSEC until I get the nat worked out.

Good idea :-)

Quoted 2 lines

If I ping an internet IP from the LAN, I'm pretty sure it goes out to the internet with the source IP still the LAN IP, without translation.

I'm not so sure about that (why do you think that's what's happening ?).

If you look at your log entry for the nat POSTROUTING table:

Quoted 4 lines

Chain POSTROUTING (policy ACCEPT 16 packets, 1439 bytes)
pkts bytes target prot opt in out source
destination 16 1184 SNAT all -- * eth1 0.0.0.0/0
0.0.0.0/0 to:XX.XXX.XXX.XXX

(Sorry about the way my email client has re-wrapped it...)

You can see that 16 packets / 1184 bytes have matched this rule, which means that they've been SNATted to XX.XXX.XXX.XXX

Also, I see that your FORWARD rules are logging no packets through them. What does your routing table look like ?

Antony.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 10 lines · George Garvey
Quoted 3 lines · Antony Stone

Um, what are the other three for, then ? If you can only use 2 IPs, why
has the ISP given you any more ?

Good question. I don't know. I have a lot of trouble getting
information from them. I assume they're doing some kind of address translation on them. They said I can use all 5. They said that 2 of the 5 would be visible to the net. I had to buy a block of 5 IPs to get the DSL for some reason I don't understand. I only wanted 1 IP.

Okay. Let's assume that's not important right now...

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

If I ping an internet IP from the LAN, I'm pretty sure it goes out to the internet with the source IP still the LAN IP, without translation.

I'm not so sure about that (why do you think that's what's happening ?).

I'm probably wrong. I noticed that, too. I ran iptables with a lot
more logging, and never say the source being changed. But I may have missed it. The machine is an old 586 and drops fast logs a lot.

How were you doing the logging ? Was it in the POSTROUTING chain, after the
rule which would change the address ? If the LOG line was any earlier than
that, then you would still see the original source address...

Quoted 11 lines

This is what ip says:

66.123.115.208/29 dev eth1  proto kernel  scope link  src 66.123.115.210
192.168.3.0/24 dev withvan  proto kernel  scope link  src 192.168.3.2

192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.2 192.168.1.0/24 via 192.168.3.2 dev withvan
default via 66.123.115.209 dev eth1

I have trouble reading that (you probably don't ;), so here's the ip commands if that's better:

Quoted 11 lines

+ /sbin/ip link set dev eth0 up mtu 1500
+ /sbin/ip address add 192.168.2.2/24 broadcast 192.168.2.255 dev eth0
+ /sbin/ip link set dev eth1 up
+ /sbin/ip address add 66.123.115.210/29 broadcast 66.123.115.215 dev eth1
+ /sbin/ip link set dev lo up
+ /sbin/ip address add 127.0.0.1/8 broadcast + dev lo
+ /sbin/ip route add default via 66.123.115.209 dev eth1
+ /sbin/ip tunnel add withvan mode gre remote 63.193.79.19 local 66.123.115.210 ttl 255 + /sbin/ip link set withvan up
+ /sbin/ip address add 192.168.3.2/24 broadcast + dev withvan
+ /sbin/ip route add 192.168.1.0/24 via 192.168.3.2 dev withvan

I see (from the bit I've chopped out of your ip output) that you still have the IPsec stuff in there - I'll assume for the time being that that's not interfering with things in any way ?

However, I do not recognise what the "withvan" device is doing. I assume
it's the GRE stuff that you're trying to debug here, so if I asked you to get
rid of it, that wouldn't help solve the problem ? Maybe someone else on the
list has more experience of GRE stuff than me, so can offer some advice here ?

Any chance you can put another machine running ethereal or similar on the eth1 interface and see what's really coming out of the box ?

Antony.

author
#4 lore raw
Quoted 5 lines · Antony Stone

On Monday 24 June 2002 8:42 pm, George Garvey wrote:

How were you doing the logging ? Was it in the POSTROUTING chain, after the
rule which would change the address ? If the LOG line was any earlier than
that, then you would still see the original source address...

It was before the NAT rule. I also logged the OUTPUT chain. But since
it has come up, I'll get better information. I was just logging to try and get a better understanding of how iptables worked, since I was having problems.

Quoted 3 lines

I see (from the bit I've chopped out of your ip output) that you still have the IPsec stuff in there - I'll assume for the time being that that's not interfering with things in any way ?

That's from the kernel itself. This is FreeS/WAN. I've not run their
script to activate it, on either side of the tunnel.

Quoted 4 lines

However, I do not recognise what the "withvan" device is doing. I assume
it's the GRE stuff that you're trying to debug here, so if I asked you to get
rid of it, that wouldn't help solve the problem ? Maybe someone else on the
list has more experience of GRE stuff than me, so can offer some advice here ?

Yes, that is the tunnel. I'm at the other end of the tunnel from the
box, so I need it to access the box in question, or need to go where it is physically. With some work I could do this with only ssh and remove the tunnel completely. I'll try that.
I'm not trying to debug the GRE tunnel. Its working fine. I'm trying
to find out why packets from other computers on the LAN (eth0 on the firewall in question) whose gateway is this firewall, send packets to the internet and don't get replies. I don't even see reply packets coming in to the firewall computer itself being logged. That's why I came up with the theory that the NAT line wasn't working, and they were going out with their source still set to the 192.168.2 address.
As I said, I'll put some specific logging in just for this, so
perhaps the box won't drop logging information, and see what happens.

Quoted 2 lines

Any chance you can put another machine running ethereal or similar on the eth1 interface and see what's really coming out of the box ?

Yes. It will take me a bit of time, but I'll do it.

I was really, really, wrong, fortunately. The outgoing NAT is
working. I tried, on a computer connected by LAN, "telnet 152.2.210.81 ftp". It hung. I did "cat /proc/net/ip_conntrack|grep 152.2.21.18":

tcp 6 55 SYN_RECV src=192.168.2.3 dst=152.2.210.81 sport=32807 dport=21 src=152.2.210.81 dst=66.123.115.210 sport=21 dport=32807 use=1

So, I guess it is the input chain that's giving me the problems. Does that seem right?

#5 lore raw
Quoted 13 lines · George Garvey
Quoted 2 lines · Antony Stone

Any chance you can put another machine running ethereal or similar on the eth1 interface and see what's really coming out of the box ?

Yes. It will take me a bit of time, but I'll do it.

I was really, really, wrong, fortunately. The outgoing NAT is
working. I tried, on a computer connected by LAN, "telnet 152.2.210.81 ftp". It hung. I did "cat /proc/net/ip_conntrack|grep 152.2.21.18":

Quoted 5 lines

tcp 6 55 SYN_RECV src=192.168.2.3 dst=152.2.210.81 sport=32807
dport=21 src=152.2.210.81 dst=66.123.115.210 sport=21 dport=32807 use=1

So, I guess it is the input chain that's giving me the problems. Does that seem right?

No, and this comment makes me wonder if you've made a common mistake in changing from ipchains to iptables ?

In ipchains, packets going from one side of the firewall to the other had to go through the input, forward and output chains; packets going into the firewall just went through the input chain.

However, in iptables, the input chain is *only* for packets going in to the firewall itself; packets going through it go through the forward chain, but not the input or output chains.

If that wasn't your understanding of how packets go through iptables, it might be worth you going back to your original ruleset and see if you have all the rules you need for packets going through the machine, in your forward chain, because that's the only one they'll go through (as well as prerouting and postrouting nat chains, of course).

The conntrack entry you've shown above simply tells us that an initial packet was received from 192.168.2.3 going to 152.2.210.81 port 21 (ftp control
port). It doesn't actually confirm whether that packet left the firewall or
not, or if it did, how it tried to get to the outside world. It does
confirm that no reply came back (syn_recv means first syn packet seen, second syn-ack packet expected), but we still don't know why.

I'd still like to see an ethereal output from the eth1 cable, I'm afraid, because that's a sure way to know what packets actually go out to the Internet.

Antony.

author
#6 lore raw

I used the following rules, very early in the script:

$IPTABLES -t mangle -I PREROUTING -j LOG --log-prefix="PreMangle " --log-level debug $IPTABLES -t nat -I PREROUTING -j LOG --log-prefix="PreNat " --log-level debug $IPTABLES -t mangle -I FORWARD -j LOG --log-prefix="FwdMangle " --log-level debug $IPTABLES -I FORWARD -j LOG --log-prefix="Forward " --log-level debug

Again, I used telnet to connect to sunsite's FTP port:

Jun 24 16:34:06 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=152.2.210.81 DST=66.123.115.210 LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=21 DPT=32811 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
^^^^^^^^^^^^^^
Jun 24 16:34:31 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:20:6f:0f:5f:40:08:00 SRC=66.123.115.209 DST=66.123.115.215 LEN=52 TOS=0x00 PREC=0x00 TTL=60 ID=33105 PROTO=UDP SPT=520 DPT=520 LEN=32 
Jun 24 16:34:31 salesns kernel: PreNat IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:20:6f:0f:5f:40:08:00 SRC=66.123.115.209 DST=66.123.115.215 LEN=52 TOS=0x00 PREC=0x00 TTL=60 ID=33105 PROTO=UDP SPT=520 DPT=520 LEN=32 
Jun 24 16:34:33 salesns kernel: PreMangle IN=eth1 OUT= MAC=01:00:5e:00:00:09:00:20:6f:0f:5f:40:08:00 SRC=66.123.115.209 DST=224.0.0.9 LEN=52 TOS=0x00 PREC=0x00 TTL=60 ID=33106 PROTO=UDP SPT=520 DPT=520 LEN=32 
Jun 24 16:34:33 salesns kernel: PreNat IN=eth1 OUT= MAC=01:00:5e:00:00:09:00:20:6f:0f:5f:40:08:00 SRC=66.123.115.209 DST=224.0.0.9 LEN=52 TOS=0x00 PREC=0x00 TTL=60 ID=33106 PROTO=UDP SPT=520 DPT=520 LEN=32 
Jun 24 16:34:34 salesns kernel: PreMangle IN=eth0 OUT= MAC=00:50:ba:37:d8:5e:00:50:ba:8f:e1:7e:08:00 SRC=192.168.2.4 DST=192.168.2.2 LEN=76 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=123 DPT=123 LEN=56 
Jun 24 16:34:55 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=152.2.210.81 DST=66.123.115.210 LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=21 DPT=32811 WINDOW=5840 RES=0x00 ACK SYN URGP=0

^^^^^^^^^^^^^^

The packets are coming in. They don't seem to be going through the forward chain, but I don't see anything stopping them looking at the rules.

I took out the rule that accepts the packets the router mutters to itself (which bother me, because I just don't know how the kernel interprets them), and got:

Jun 25 03:13:57 salesns kernel: PreMangle IN=eth0 OUT= MAC=00:50:ba:37:d8:5e:00:50:ba:37:d8:3e:08:00 SRC=192.168.2.3 DST=152.2.210.81 LEN=60 TOS=0x10 PREC=0x00 TTL=128 ID=45095 DF PROTO=TCP SPT=32813 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 
Jun 25 03:13:57 salesns kernel: PreNat IN=eth0 OUT= MAC=00:50:ba:37:d8:5e:00:50:ba:37:d8:3e:08:00 SRC=192.168.2.3 DST=152.2.210.81 LEN=60 TOS=0x10 PREC=0x00 TTL=128 ID=45095 DF PROTO=TCP SPT=32813 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 
Jun 25 03:13:57 salesns kernel: FwdMangle IN=eth0 OUT=eth1 SRC=192.168.2.3 DST=152.2.210.81 LEN=60 TOS=0x10 PREC=0x00 TTL=127 ID=45095 DF PROTO=TCP SPT=32813 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 
Jun 25 03:13:57 salesns kernel: Forward IN=eth0 OUT=eth1 SRC=192.168.2.3 DST=152.2.210.81 LEN=60 TOS=0x10 PREC=0x00 TTL=127 ID=45095 DF PROTO=TCP SPT=32813 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 
^^^^^^^^^^^^^^^^^
Jun 25 03:13:57 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=76 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:13:57 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:4c:00:00:40:00:fb:2f:3a:61:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:34:4d:91:40:00:7f:06:29:cc:c0:a8 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=127 ID=19857 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:13:57 salesns kernel: FwdMangle IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19857 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:13:57 salesns kernel: Forward IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19857 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:13:57 salesns kernel: PreMangle IN=eth0 OUT= MAC=00:50:ba:37:d8:5e:00:50:ba:37:d8:3e:08:00 SRC=192.168.2.3 DST=192.168.1.3 LEN=84 TOS=0x10 PREC=0x00 TTL=128 ID=49185 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:13:57 salesns kernel: FwdMangle IN=eth0 OUT=withvan SRC=192.168.2.3 DST=192.168.1.3 LEN=84 TOS=0x10 PREC=0x00 TTL=127 ID=49185 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:13:57 salesns kernel: Forward IN=eth0 OUT=withvan SRC=192.168.2.3 DST=192.168.1.3 LEN=84 TOS=0x10 PREC=0x00 TTL=127 ID=49185 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:13:57 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=152.2.210.81 DST=66.123.115.210 LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=21 DPT=32813 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
^^^^^^^^^^^^^^^^^
Jun 25 03:13:57 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=76 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:13:57 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:4c:00:00:40:00:fb:2f:3a:61:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:34:4d:92:40:00:7f:06:29:cb:c0:a8 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=127 ID=19858 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:13:57 salesns kernel: FwdMangle IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19858 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:13:57 salesns kernel: Forward IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19858 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:13:59 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:20:6f:0f:5f:40:08:00 SRC=66.123.115.209 DST=66.123.115.215 LEN=52 TOS=0x00 PREC=0x00 TTL=60 ID=35746 PROTO=UDP SPT=520 DPT=520 LEN=32 
Jun 25 03:13:59 salesns kernel: PreNat IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:20:6f:0f:5f:40:08:00 SRC=66.123.115.209 DST=66.123.115.215 LEN=52 TOS=0x00 PREC=0x00 TTL=60 ID=35746 PROTO=UDP SPT=520 DPT=520 LEN=32 
Jun 25 03:13:59 salesns kernel: REJECT-INP: IN=eth1 OUT= MAC=ff:ff:ff:ff:ff:ff:00:20:6f:0f:5f:40:08:00 SRC=66.123.115.209 DST=66.123.115.215 LEN=52 TOS=0x00 PREC=0x00 TTL=60 ID=35746 PROTO=UDP SPT=520 DPT=520 LEN=32 
Jun 25 03:14:00 salesns kernel: PreMangle IN=eth0 OUT= MAC=00:50:ba:37:d8:5e:00:50:ba:37:d8:3e:08:00 SRC=192.168.2.3 DST=152.2.210.81 LEN=60 TOS=0x10 PREC=0x00 TTL=128 ID=45096 DF PROTO=TCP SPT=32813 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 
Jun 25 03:14:00 salesns kernel: FwdMangle IN=eth0 OUT=eth1 SRC=192.168.2.3 DST=152.2.210.81 LEN=60 TOS=0x10 PREC=0x00 TTL=127 ID=45096 DF PROTO=TCP SPT=32813 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 
Jun 25 03:14:00 salesns kernel: Forward IN=eth0 OUT=eth1 SRC=192.168.2.3 DST=152.2.210.81 LEN=60 TOS=0x10 PREC=0x00 TTL=127 ID=45096 DF PROTO=TCP SPT=32813 DPT=21 WINDOW=5840 RES=0x00 SYN URGP=0 
^^^^^^^^^^^
Jun 25 03:14:00 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=108 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:14:00 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:6c:00:00:40:00:fb:2f:3a:41:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:54:4d:93:40:00:7f:06:29:aa:c0:a8 SRC=192.168.1.3 DST=192.168.2.3 LEN=84 TOS=0x10 PREC=0x00 TTL=127 ID=19859 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: FwdMangle IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=84 TOS=0x10 PREC=0x00 TTL=126 ID=19859 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: Forward IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=84 TOS=0x10 PREC=0x00 TTL=126 ID=19859 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: PreMangle IN=eth0 OUT= MAC=00:50:ba:37:d8:5e:00:50:ba:37:d8:3e:08:00 SRC=192.168.2.3 DST=192.168.1.3 LEN=100 TOS=0x10 PREC=0x00 TTL=128 ID=49186 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: FwdMangle IN=eth0 OUT=withvan SRC=192.168.2.3 DST=192.168.1.3 LEN=100 TOS=0x10 PREC=0x00 TTL=127 ID=49186 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: Forward IN=eth0 OUT=withvan SRC=192.168.2.3 DST=192.168.1.3 LEN=100 TOS=0x10 PREC=0x00 TTL=127 ID=49186 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: PreMangle IN=eth0 OUT= MAC=00:50:ba:37:d8:5e:00:50:ba:37:d8:3e:08:00 SRC=192.168.2.3 DST=192.168.1.3 LEN=84 TOS=0x10 PREC=0x00 TTL=128 ID=49187 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: FwdMangle IN=eth0 OUT=withvan SRC=192.168.2.3 DST=192.168.1.3 LEN=84 TOS=0x10 PREC=0x00 TTL=127 ID=49187 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: Forward IN=eth0 OUT=withvan SRC=192.168.2.3 DST=192.168.1.3 LEN=84 TOS=0x10 PREC=0x00 TTL=127 ID=49187 DF PROTO=TCP SPT=22 DPT=34265 WINDOW=5792 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:00 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=152.2.210.81 DST=66.123.115.210 LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=21 DPT=32813 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
^^^^^^^^^^^^^^
Jun 25 03:14:00 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=76 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:14:00 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:4c:00:00:40:00:fb:2f:3a:61:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:34:4d:94:40:00:7f:06:29:c9:c0:a8 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=127 ID=19860 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:14:00 salesns kernel: FwdMangle IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19860 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:14:00 salesns kernel: Forward IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19860 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:14:00 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=76 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:14:00 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:4c:00:00:40:00:fb:2f:3a:61:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:34:4d:95:40:00:7f:06:29:c8:c0:a8 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=127 ID=19861 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:14:00 salesns kernel: FwdMangle IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19861 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:14:00 salesns kernel: Forward IN=withvan OUT=eth0 SRC=192.168.1.3 DST=192.168.2.3 LEN=52 TOS=0x10 PREC=0x00 TTL=126 ID=19861 DF PROTO=TCP SPT=34265 DPT=22 WINDOW=45568 RES=0x00 ACK URGP=0 
Jun 25 03:14:01 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=108 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:14:01 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:6c:00:00:40:00:fb:2f:3a:41:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:54:fc:7a:40:00:7f:06:7a:c3:c0:a8 SRC=192.168.1.3 DST=192.168.2.2 LEN=84 TOS=0x10 PREC=0x00 TTL=127 ID=64634 DF PROTO=TCP SPT=34819 DPT=22 WINDOW=14112 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:01 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=108 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:14:01 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:6c:00:00:40:00:fb:2f:3a:41:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:54:fc:7b:40:00:7f:06:7a:c2:c0:a8 SRC=192.168.1.3 DST=192.168.2.2 LEN=84 TOS=0x10 PREC=0x00 TTL=127 ID=64635 DF PROTO=TCP SPT=34819 DPT=22 WINDOW=14112 RES=0x00 ACK PSH URGP=0 
Jun 25 03:14:01 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=76 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:14:01 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:4c:00:00:40:00:fb:2f:3a:61:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:34:fc:7c:40:00:7f:06:7a:e1:c0:a8 SRC=192.168.1.3 DST=192.168.2.2 LEN=52 TOS=0x10 PREC=0x00 TTL=127 ID=64636 DF PROTO=TCP SPT=34819 DPT=22 WINDOW=14112 RES=0x00 ACK URGP=0 
Jun 25 03:14:01 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=63.193.79.19 DST=66.123.115.210 LEN=76 TOS=0x00 PREC=0x00 TTL=251 ID=0 DF PROTO=47 
Jun 25 03:14:01 salesns kernel: PreMangle IN=withvan OUT= MAC=45:00:00:4c:00:00:40:00:fb:2f:3a:61:3f:c1:4f:13:42:7b:73:d2:00:00:08:00:45:10:00:34:fc:7d:40:00:7f:06:7a:e0:c0:a8 SRC=192.168.1.3 DST=192.168.2.2 LEN=52 TOS=0x10 PREC=0x00 TTL=127 ID=64637 DF PROTO=TCP SPT=34819 DPT=22 WINDOW=14112 RES=0x00 ACK URGP=0 
Jun 25 03:14:01 salesns kernel: PreMangle IN=eth1 OUT= MAC=ff:fe:24:4b:c6:13:00:20:6f:0f:5f:40:08:00 SRC=152.2.210.81 DST=66.123.115.210 LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=21 DPT=32813 WINDOW=5840 RES=0x00 ACK SYN URGP=0

^^^^^^^^^^^^^^^^^^

I built the disk drive for the computer I'm having trouble with by copying if from another system that works, and modifying the setup tables, and removing services that aren't used. On the original system:

Jun 25 03:25:32 ns kernel: PreMangle IN=eth2 OUT= MAC=00:10:5a:60:3f:7f:00:10:67:00:b5:58:08:00 SRC=152.2.210.81 DST=63.193.79.19 LEN=44 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=21 DPT=34823 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
Jun 25 03:25:32 ns kernel: FwdMangle IN=eth2 OUT=eth0 SRC=152.2.210.81 DST=192.168.1.3 LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=21 DPT=34823 WINDOW=5840 RES=0x00 ACK SYN URGP=0 
Jun 25 03:25:32 ns kernel: Forward IN=eth2 OUT=eth0 SRC=152.2.210.81 DST=192.168.1.3 LEN=44 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=21 DPT=34823 WINDOW=5840 RES=0x00 ACK SYN URGP=0

These have identical kernels, identical scripts, different NICs, different names, different connections to the same ISP. The ns kernel has a name in our DNS on the internet, the salesns kernel does not (except at the ISP). The ns kernel is our primary name and mail server, and the other end of the tunnel between ns and salesns. It also handles most of the NAT for the 196.168.1 LAN.

My theories are stupid, because I don't completely understand what netfilter is doing, but right now it looks like the packet never gets to the forward chain, and I don't know what is stopping it. Am I wrong that mangle is the first step of the forward chain? Is there a way to see what conntrack is doing?

Is there a way I can find out what happens to that packet after it goes through mangle PREROUTING?