Skip to contentSkip to conversation

: FTP connection tracking problem

netfilter Jose Luis Domingo Lopez 2 participants 3 messages Feb 12, 2002 – Feb 21, 2002

lore
Share & tools
lore
#0 lore raw

Hi kernel developers:

[I'm am not subscribed to netfilter-devel, so please Cc: linux-kernel]

I am experiencing some problems with FTP connection tracking in recent 2.4.x kernels. In short, it seems ip_conntrack_ftp misses some legitimate packets belonging to already stablished connections, and sometimes FTP sessions don't work. Of course, this is NATing outgoing traffic, including FTP. A more complete explanation follows.

A Linux machine (tried 2.4.17 and 2.4.18-pre9) acting as a router/NAT inconditionally does SNAT on all traffic leaving some interface:
iptables -t nat -A POSTROUTING --out-interface eth2 --jump SNAT \
--to-source $PUBLIC_ADDRESS

As we don't want to leave the box open to external port scan, but we do want to let the machine connect to the Internet to grab security updates from time to time, we allow all outgoing traffic through "eth2" but just allow related traffic in the opposite direction:
iptables -t filter -A OUTPUT --jump ACCEPT

iptables -t filter -A INPUT --in-interface eth2 --match state \
    --state RELATED,ESTABLISHED --jump ACCEPT
iptables -t filter -A INPUT --in-interface eth2 --match state \
    --state NEW,INVALID --jump DROP

So, as we manually (well, our script :) install all required netfilter modules into memory, we expected most protocols to work, including FTP (ip_conntrack, ip_conntrack_ftp, ip_nat_ftp among others are in memory). And in fact, most of the time every "supported" protocol work fine. However, some combinations of FTP client and remote FTP server fail.

Starting with iptables counters reset, we try to connect to some remote FTP server, and the connection isn't fully established. With no other network traffic going on we see packets matching the DROP rule. Logging those packets to syslog (via --jump LOG) shows they are packets from the FTP connection. This happens both with active and passive transfer modes.

We tried to ACCEPT INVALID traffic, with no luck: packets continue to be dropped, so to FTP connection tracking they appear as NEW instead of somewhat related to the ongoing FTP session.

The strange part is that this will show up with just some FTP clients and/or remote FTP servers. For example, text-mode web browser "links" doesn't work in any case. On the other hand, text-mode browser "lynx" works nice in the same places where "links" fails. "wget", "ftp" and "ncftp" show various degrees of success with several combinations of active/passive transfer mode and remote FTP servers (tried with ftp.kernel.org, ftp.at.kernel.org, ftp.mozilla.org, ftp.rediris.es, ftp.sourceforge.net).

For example, "ftp.rediris.es" won't even show the greeting message with "links", but will with for example "ncftp". However, trying to download some file from the server, stalls as soon as some kilobytes of data have been received (in the order of 8 to 10 KB).

If we configure INPUT to not discard packets, and just ACCEPT all that comes in solves the proble. ECN is not compiled in nor activated. The rest of network-related kernel tunnable parameters have default values.

Searching the Internet for information took me to a messages on netfilter-devel mailing list, namely:
http://lists.samba.org/pipermail/netfilter-devel/2001-October/002433.html

It contains a patch for a problem very similar to what I am experiencing, and seems to be related to the fact that some FTP clients end their FTP commands with just \r instead of \r\n, and viceversa. A "ethereal" packet capture in a "clean" (no firewall rules) state with the FTP clients described above show that all of them follow RFC959 and terminate FTP commands with "\r\n".

I compiled the same kernel (2.4.18-pre9 in this case) with the patch applied, and repeated some of the test described above with some degree of success. Some combinations of FTP client and remote FTP servers now work OK, but some other combinations continue failing. As previously said, in absence of (DROP) firewall rules, everything works fine. We checked the possibility of our ISP messing the packets up, but some tests limited to our LAN show the same problems.

NOTE: "links" author says in the manual page:
BUGS
Can't connect to some FTP servers (Novell, NT). Connection stays in "Request sent" state.
So in this particular case, the application could also have problems.

I am willing to perform as much tests as you need to help discover what is going on, so if I missed something important, please ask.

Keep up the good work !

Signature · 4 lines
José Luis Domingo López
Linux Registered User #189436     Debian Linux Woody (P166 64 MB RAM)
 
jdomingo AT internautas DOT   org  => Spam at your own risk
#1 lore raw
Quoted 8 lines · Jose Luis Domingo Lopez

The strange part is that this will show up with just some FTP clients and/or remote FTP servers. For example, text-mode web browser "links" doesn't work in any case. On the other hand, text-mode browser "lynx" works nice in the same places where "links" fails. "wget", "ftp" and "ncftp" show various degrees of success with several combinations of active/passive transfer mode and remote FTP servers (tried with ftp.kernel.org, ftp.at.kernel.org, ftp.mozilla.org, ftp.rediris.es, ftp.sourceforge.net).

FTP connection tracking generally works very well. It's widely deployed and in most scenarios it is working fine.

The "links" browser is violating the RFC's by not waiting for server responses after sending PORT commands to the server. This is a problem with the browser, and it should be fixed on the browser side. I doubt it works well with other ftp masquerading/nat solutions.

I'm a heavy user of wget / ftp / mirror / ncftp, and I've never experienced any of the problems you describe. Please give me exact ftp server hostnames where I can try to reproduce your problem.

Quoted 4 lines

For example, "ftp.rediris.es" won't even show the greeting message with "links", but will with for example "ncftp". However, trying to download some file from the server, stalls as soon as some kilobytes of data have been received (in the order of 8 to 10 KB).

As I said, forget about links. Use ftp or ncftp and you can be sure they comply to the RFC's.

Some other notes:

- greeting messages are part of the control connection which is not really touched by conntrack.

- I can not imagine how the ftp conntrack helper should cause stalls in file transfers. Are you 100% sure this is caused by netfilter? Do you see packet drops/ delays when comparing incoming/outgoing packets on your firewall? What happens without ip_conntrack_ftp? Stalls could be network problems in general or bandwith throttling on the server.

- I have just successfully downloaded about 50MB from ftp.rediris.es through a netfilter conntrack+nat box, I get constant transfer rates of about 700kbits/sec on a 1mbit internet uplink.

Quoted 3 lines

If we configure INPUT to not discard packets, and just ACCEPT all that comes in solves the proble. ECN is not compiled in nor activated. The rest of network-related kernel tunnable parameters have default values.

how are your input rules related to forwarded/NAT'ed traffic through the firewall? INPUT rules are for the local machine only. Or are you running some ftp/http proxy like squid on the local box?

Quoted 2 lines

I am willing to perform as much tests as you need to help discover what is going on, so if I missed something important, please ask.

- tcpdump/ethereral on incoming+outgoing interface. see if there are really packet drops / ack storms / ...
- enable ftp conntrack debugging (in ip_conntrack_ftp.c and ip_nat_ftp.c there is something like

#if 0

#define DEBUGP(format, args...) printk(KERN_DEBUG __FILE__ ":" __FUNCTION__ \
                                        ":" format, ## args)

#else
#define DEBUGP(format, args...)
#endif

just change the "if 0" to "if 1", recompile and reinstall your modules.

you should get detailed debug logs in your syslog for every ftp connection

- what does /proc/net/ip_conntrack say about expectations
- what if you only use ip_conntrack_ftp without nat

Quoted 7 lines

José Luis Domingo López
Linux Registered User #189436 Debian Linux Woody (P166 64 MB RAM)

jdomingo AT internautas DOT org => Spam at your own risk

Signature · 5 lines
Live long and prosper
- Harald Welte / [email protected]               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M+ 
V-- PS++ PE-- Y++ PGP++ t+ 5-- !X !R tv-- b+++ !DI !D G+ e* h--- r++ y+(*)
#2 lore raw
In reply to Jose Luis Domingo Lopez
Quoted 11 lines · Harald Welte
Quoted 8 lines · Jose Luis Domingo Lopez

The strange part is that this will show up with just some FTP clients and/or remote FTP servers. For example, text-mode web browser "links" doesn't work in any case. On the other hand, text-mode browser "lynx" works nice in the same places where "links" fails. "wget", "ftp" and "ncftp" show various degrees of success with several combinations of active/passive transfer mode and remote FTP servers (tried with ftp.kernel.org, ftp.at.kernel.org, ftp.mozilla.org, ftp.rediris.es, ftp.sourceforge.net).

The NAT/router box that seemed to have problems with FTP connection tracking went into production a week ago. Since them it has been working OK, at least there are no reports froms people on the internal network complaining about their file downloads stalling or failing.

However, most of the users don't use the net heavily. I use the net a lot every day, and have had no big problems so far, except from some (random) combinations of browser and remote FTP server that make directory listing not complete (although file downloads work OK).

So it seems the problem I was having was somewhere outside the linux kernel inside the router. Sorry for any inconvenience :-)

Signature · 4 lines
José Luis Domingo López
Linux Registered User #189436     Debian Linux Woody (P166 64 MB RAM)
 
jdomingo AT internautas DOT   org  => Spam at your own risk