Skip to contentSkip to conversation

Why does ipv6 enabled interfere with ipv4 SNAT?

DISCUSSION Whit Blauvelt 5 participants 14 messages Mar 25, 2008 – Apr 2, 2008 netfilter

lore
Share & tools
lore
#0 lore raw

I've just been setting up a replacement for an old server happily running Netfilter with SNAT for years. The new one - using Ubuntu Server 7.10 - was refusing to SNAT or MASQ, despite running the same long-proven Netfilter rules. After checking that everything was getting set right in /proc/sys/net/ipv4/conf (it was), I for some reason took a look in /proc/sys/net/ipv6/conf. (Ubuntu insists on running ipv6 by default.) There was an anomoly: The new machine has 6 NICs, and while ipv4/conf had eth0 through eth5, ipv6/conf only had them through eth3 - just the first 4. As it happens the Internet-facing NICs for this box are eth4 and eth5.

At first I looked for a way to populate the ipv6 proc area. But not finding that, and really having no use for ipv6 anyway - stupid to leave stuff on on a server when it's not being used right? - I finally went to /etc/modprobe.d/arch/i386 and added a line, "alias net-pf-10 off", which on rebooting blocks the ipv6 kernel module from loading. Now, with no other change, SNAT works as well as ever.

But I'd still like to understand what the principle here is. Are there two independent ipv6-related bugs - one that prevents its /proc space being populated beyond 4 NICs, and a totally different one that breaks ipv4 Netfilter SNAT? Or was the first bug what broke SNAT? What is ipv4 Netfilter doing being vulnerable to ipv6 problems anyway? I know there's Netfilter stuff for ipv6; is it somehow required to invoke that somehow if ipv6 is enabled in order for the ipv4 Netfilter code to work right with SNAT?

Finally, is the Ubuntu Server project being boneheaded to even have ipv6 enabled by default, if it's still this far from being ready for prime time? Servers should be rock-solid at basic functions like SNAT out-of-the-box, IMHO.

Thanks for any explanations. As I say, things work fine once I figure out what to disable. But frankly it took me half the day to track down what the problem was.

Regards,
Whit

#1 lore raw
Quoted 4 lines · Whit Blauvelt

I for some reason took a look in
/proc/sys/net/ipv6/conf. (Ubuntu insists on running ipv6 by default.) There was an anomoly: The new machine has 6 NICs, and while ipv4/conf had eth0 through eth5, ipv6/conf only had them through eth3 - just the first 4.

Just what _do_ you actually have in /proc/sys/net/ipv6/conf? Just 4 entries seems a bit spartanic, since there are also the "all" and "default" entries:

# ls /proc/sys/net/ipv6/conf/
all default lo rtl0 sis0 tun0 vmnet1

NAT-out device is sis0. Even if I add in a number of dummies, all remains normal:

all      dummy0  dummy2  dummy4  dummy6  dummy8  lo    sis0  vmnet1
default  dummy1  dummy3  dummy5  dummy7  dummy9  rtl0  tun0

It still works with opensuse plus 2.6.23. Well, I suggest you try a stock kernel.

Quoted 2 lines

Finally, is the Ubuntu Server project being boneheaded to even have ipv6 enabled by default, if it's still this far from being ready for prime time?

I would not say IPv6 was not ready.

Quoted 2 lines

Servers should be rock-solid at basic functions like SNAT out-of-the-box, IMHO.

#2 lore raw
Quoted 3 lines · Jan Engelhardt

Just what _do_ you actually have in /proc/sys/net/ipv6/conf? Just 4 entries seems a bit spartanic, since there are also the "all" and "default" entries:

I was mentioning where the mismatch was. Yeah the other stuff's there.

/proc/sys/net/ipv6/conf ends up with only:

all default eth0 eth1 eth2 eth3 lo

while /proc/sys/net/ipv4/conf ends up (correctly) with:

all default eth0 eth1 eth2 eth3 eth4 eth5 lo

Now, I don't know just which process is (not) doing the populating there, but that's consistently where it ends up with ipv6 enabled and 6 NICs in the box.

Quoted 8 lines
# ls /proc/sys/net/ipv6/conf/
all default lo rtl0 sis0 tun0 vmnet1

NAT-out device is sis0. Even if I add in a number of dummies, all remains normal:

all      dummy0  dummy2  dummy4  dummy6  dummy8  lo    sis0  vmnet1
default  dummy1  dummy3  dummy5  dummy7  dummy9  rtl0  tun0

Not sure what you're thinking there. My problem wasn't with there being too many devices, but with two of the devices I actually have not being represented - and with Netfilter not doing _ipv4_ SNAT on account of something with _ipv6_. Why should Netfilter ipv4 code even _care_ about what's right or not with ipv6? Do you have any knowledge about interdependency there?

Quoted 2 lines

It still works with opensuse plus 2.6.23. Well, I suggest you try a stock kernel.

It is a stock kernel, if by that you mean a stock distro kernel - Ubuntu's latest, 2.6.22-14-server.

Quoted 1 line

I would not say IPv6 was not ready.

I have no idea if the failure to fully populate the ipv6 proc eth? devices was Debian-specific, Ubuntu-specific, a shortcoming of the ifupdown suite both use, or a bug in the kernel itself. But it seems wrong on the face of it for ipv4 Netfilter SNAT code to depend on ipv6 in any way. Yet something about ipv6 on Ubuntu - possibly the failure to set NIC devices beyond eth3 up properly in the ipv6 /proc space - breaks Netfilter ipv4 snat. I'm still hoping to understand why, in part because there's obviously a serious bug _somewhere_, and it would be nice to report it to the right place.

Best,
Whit

#3 lore raw
Quoted 5 lines · Whit Blauvelt

Not sure what you're thinking there. My problem wasn't with there being too many devices, but with two of the devices I actually have not being represented - and with Netfilter not doing _ipv4_ SNAT on account of something with _ipv6_.

How does it break? Do the counters increase in the nat table at all? Do the chain and/or rule counters increase if you add the same rule without action? (I.e.:

	-t nat -A POSTROUTING -o eth4 -m whatever
	-t nat -A POSTROUTING -o eth4 -m whatever -j SNAT --to xyz
Quoted 5 lines
Quoted 2 lines

It still works with opensuse plus 2.6.23. Well, I suggest you try a stock kernel.

It is a stock kernel, if by that you mean a stock distro kernel - Ubuntu's latest, 2.6.22-14-server.

Stock vanilla kernel.org kernel.

#4 lore raw
Quoted 6 lines · Jan Engelhardt

How does it break? Do the counters increase in the nat table at all? Do the chain and/or rule counters increase if you add the same rule without action? (I.e.:

	-t nat -A POSTROUTING -o eth4 -m whatever
	-t nat -A POSTROUTING -o eth4 -m whatever -j SNAT --to xyz

No the counters don't increase. Again, ipv4 Netfilter SNAT does not work if ipv6 is enabled on the system. It works perfectly if ipv6 is disabled - no other changes. Do you have any theory about that, at all? It may well be downstream of the failure to handle /proc assignments correctly for > 4 NICs on the ipv6 side - understandably most people aren't network admins on my level running boxes with > 4 NICs so that wouldn't bite that often. That wouldn't make it not a bug, though.

Quoted 4 lines
Quoted 2 lines

It is a stock kernel, if by that you mean a stock distro kernel - Ubuntu's latest, 2.6.22-14-server.

Stock vanilla kernel.org kernel.

Why would I want to do that? If you read my orginal post closely, I have no need for ipv6. It's fine with me to run without it - which works perfectly. But what I want to do is understand where the fine bug is ... perhaps to report it to those responsible. Now, it looks pretty obviously like there's a serious bug in Netfilter here, because how else could anything about the state of the ipv6 configuration affect the success of ipv4 Netfilter SNAT?

Now, if you can explain, theoretically, why I'm wrong about that, why the state of the ipv6 configuration should be critical to ipv4 Netfilter SNAT operation, I'm quite curious. But if you have no idea why the ipv4 Netfilter SNAT rule fails if-and-only-if ipv6 operation is also enabled on the system, then please let's see if someone who knows the innards of this stuff better than either of us can help illuminate the puzzle.

Whit

#5 lore raw
In reply to Whit Blauvelt
Quoted 9 lines · Whit Blauvelt

I was mentioning where the mismatch was. Yeah the other stuff's there.

/proc/sys/net/ipv6/conf ends up with only:

all default eth0 eth1 eth2 eth3 lo

while /proc/sys/net/ipv4/conf ends up (correctly) with:

all default eth0 eth1 eth2 eth3 eth4 eth5 lo

If you have got ethernet devices with MTU smaller than 1500, those won't support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

Quoted 7 lines

Not sure what you're thinking there. My problem wasn't with there being too many devices, but with two of the devices I actually have not being represented - and with Netfilter not doing _ipv4_ SNAT on account of something with _ipv6_. Why should Netfilter ipv4 code even _care_ about what's right or not with ipv6? Do you have any knowledge about interdependency there?

Quoted 5 lines
Quoted 2 lines

It still works with opensuse plus 2.6.23. Well, I suggest you try a stock kernel.

It is a stock kernel, if by that you mean a stock distro kernel - Ubuntu's latest, 2.6.22-14-server.

Please check which modules are loaded in. Probably IPv4 NAT related modules are not auto-loaded. Also, is there any error message when you issue 'iptables -t nat ...' commands?

Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#6 lore raw
Quoted 10 lines · Jozsef Kadlecsik
Quoted 7 lines

/proc/sys/net/ipv6/conf ends up with only:

all default eth0 eth1 eth2 eth3 lo

while /proc/sys/net/ipv4/conf ends up (correctly) with:

all default eth0 eth1 eth2 eth3 eth4 eth5 lo

If you have got ethernet devices with MTU smaller than 1500, those won't support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

Interesting. But there is no difference between the MTU settings of eth2, eth3, eth4 and eth5 - in fact all four ports are on the same physical card. All the eth? devices are running at default of 1500. I've checked.

Quoted 3 lines

Please check which modules are loaded in. Probably IPv4 NAT related modules are not auto-loaded. Also, is there any error message when you issue 'iptables -t nat ...' commands?

Josef, the set of iptables modules is precisely the same whether or not ipv6 is enabled on the system (yes, I've looked). The only difference is whether the ipv6 module is loaded. Blocking the ipv6 module from loading at boot both prevents (of course) the /proc/sys/net/ipv6 space from being populated. It also fully restores SNAT functionality.

There are no error messages from "iptables -t nat" commands. The nat rules show up precisely the same in the resulting tables - they just don't count any packets traversing them if ipv6 is also enabled. I haven't done packet inspection to see what's happening, but the problem remains:

With _everything else_ unchanged, simply letting the ipv6 module load at boot results in:

1) eth4 and eth5 not showing up as they should for ipv6 in /proc/sys/net/ipv6/conf

and (whether related or a separate bug)

2) ipv4 Netfilter SNAT (in POSTROUTING in this case) becoming a black hole

Boxes behind the firewall can still ping the firewall. The firewall can still ping the world. But just with the ipv6 module loaded, the boxes behind the firewall cannot ping the world. Clearly ipv6 is not properly provisioning its space. But why should ipv6's failing have any effect at all on ipv4 functionality?

For me, not loading the ipv6 module is a totally fine workaround. It's useless cruft in my context anyway. But even though my problem is fixed for now, it will be a problem for other people (maybe only those with > 4 ethernet devices?) when the ipv6 transition finally happens. I remember in the early 90s when we were expecting it by about 1998.

So in the spirit of open source development, I'm looking for for someone with better knowledge of the underlying theory to help identify where the bug here is. Then the responsible developers can have a chance to fix it before a lot of other sysadmins also end up wasting many hours because of the non-obviousness of having to look at ipv6 stuff to find why ipv4 SNAT becomes a black hole.

Regards,
Whit

#7 lore raw
Quoted 34 lines · Whit Blauvelt

Josef, the set of iptables modules is precisely the same whether or not ipv6 is enabled on the system (yes, I've looked). The only difference is whether the ipv6 module is loaded. Blocking the ipv6 module from loading at boot both prevents (of course) the /proc/sys/net/ipv6 space from being populated. It also fully restores SNAT functionality.

There are no error messages from "iptables -t nat" commands. The nat rules show up precisely the same in the resulting tables - they just don't count any packets traversing them if ipv6 is also enabled. I haven't done packet inspection to see what's happening, but the problem remains:

With _everything else_ unchanged, simply letting the ipv6 module load at boot results in:

1) eth4 and eth5 not showing up as they should for ipv6 in /proc/sys/net/ipv6/conf

and (whether related or a separate bug)

2) ipv4 Netfilter SNAT (in POSTROUTING in this case) becoming a black hole

Boxes behind the firewall can still ping the firewall. The firewall can still ping the world. But just with the ipv6 module loaded, the boxes behind the firewall cannot ping the world. Clearly ipv6 is not properly provisioning its space. But why should ipv6's failing have any effect at all on ipv4 functionality?

For me, not loading the ipv6 module is a totally fine workaround. It's useless cruft in my context anyway. But even though my problem is fixed for now, it will be a problem for other people (maybe only those with > 4 ethernet devices?) when the ipv6 transition finally happens. I remember in the early 90s when we were expecting it by about 1998.

Quoted 6 lines

So in the spirit of open source development, I'm looking for for someone with better knowledge of the underlying theory to help identify where the bug here is. Then the responsible developers can have a chance to fix it before a lot of other sysadmins also end up wasting many hours because of the non-obviousness of having to look at ipv6 stuff to find why ipv4 SNAT becomes a black hole.

Please post the list of modules loaded and the output of /proc/net/nf_conntrack.

#10 lore raw
In reply to Whit Blauvelt
Quoted 6 lines · Whit Blauvelt
Quoted 2 lines

If you have got ethernet devices with MTU smaller than 1500, those won't support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

Interesting. But there is no difference between the MTU settings of eth2, eth3, eth4 and eth5 - in fact all four ports are on the same physical card. All the eth? devices are running at default of 1500. I've checked.

OK. That was a possibility which could have explained the difference in the interfaces.

Quoted 6 lines
Quoted 3 lines

Please check which modules are loaded in. Probably IPv4 NAT related modules are not auto-loaded. Also, is there any error message when you issue 'iptables -t nat ...' commands?

Josef, the set of iptables modules is precisely the same whether or not ipv6 is enabled on the system (yes, I've looked).

But which netfilter modules are loaded in? Again, there can be some auto-loading problem which we cannot figure out without the list of the modules. And as Patrick suggested: do you see the connections in /proc/net/nf_conntrack? Or you have got /proc/net/ip_conntrack instead?

Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#12 lore raw
In reply to Jozsef Kadlecsik
Quoted 2 lines · Jozsef Kadlecsik

If you have got ethernet devices with MTU smaller than 1500, those won't support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

Don't you mean smaller than 1280 ?

#13 lore raw
Quoted 6 lines · Pascal Hambourg
Quoted 3 lines · Jozsef Kadlecsik

If you have got ethernet devices with MTU smaller than 1500, those won't support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

Don't you mean smaller than 1280 ?

You are right, the exact boundary is 1280.

Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
#8 lore raw
In reply to Patrick McHardy
Quoted 2 lines · Patrick McHardy

Please post the list of modules loaded and the output of /proc/net/nf_conntrack.

First here is the list by the system in question, working once the ipv6 module is blocked from loading at boot. Next is the list from a system with identical hardware and near-identical configuration (same firewall rules), but with ipv6 loading - and which also has only 4 of the 6 NICs showing up in the ipv6 proc conf space, and also has NAT (in this case DNAT is what I tested) failing - also where the NICs on the Internet side of things are those coincidentally not showing up with proc ipv6 conf settings.

As to the output of /proc/net/nf_conntrack, you just want to see anything, or under specific load? I'm not going to just publicly post the raw data - although both systems have some there - since IPs can identify my client and their clients, which would violate confidentiality.

Okay, the fixed system:

Module                  Size  Used by
drbd                  208136  2 
cn                      9632  1 drbd
parport_pc             37668  0 
lp                     12452  0 
parport                37448  2 parport_pc,lp
loop                   19076  0 
sg                     36380  0 
sr_mod                 17700  0 
cdrom                  37408  1 sr_mod
ata_generic             8580  0 
usbhid                 29664  0 
hid                    28928  1 usbhid
pcspkr                  4224  0 
psmouse                39952  0 
serio_raw               8068  0 
shpchp                 34580  0 
pci_hotplug            32576  1 shpchp
evdev                  11136  0 
ipt_TOS                 3200  16 
ipt_REJECT              5760  2 
xt_state                3456  372 
nf_nat_ftp              4352  0 
nf_conntrack_ftp       11136  1 nf_nat_ftp
xt_limit                3584  3 
xt_tcpudp               4224  616 
ipt_LOG                 7552  2 
iptable_mangle          3840  1 
iptable_nat             8708  1 
nf_nat                 20012  2 nf_nat_ftp,iptable_nat
nf_conntrack_ipv4      19724  374 iptable_nat
nf_conntrack           65160  6 xt_state,nf_nat_ftp,nf_conntrack_ftp,iptable_nat,nf_nat,nf_conntrack_ipv4
nfnetlink               6936  3 nf_nat,nf_conntrack_ipv4,nf_conntrack
iptable_filter          3968  1 
ip_tables              13924  3 iptable_mangle,iptable_nat,iptable_filter
x_tables               16260  8 ipt_TOS,ipt_REJECT,xt_state,xt_limit,xt_tcpudp,ipt_LOG,iptable_nat,ip_tables
ext3                  133640  4 
jbd                    60456  1 ext3
mbcache                 9732  1 ext3
ata_piix               17540  0 
libata                125296  2 ata_generic,ata_piix
ehci_hcd               36748  0 
bnx2                  157208  0 
e1000                 126656  0 
uhci_hcd               26640  0 
usbcore               138760  4 usbhid,ehci_hcd,uhci_hcd
cciss                  61700  7 
scsi_mod              146828  4 sg,sr_mod,libata,cciss
dm_mirror              24320  0 
dm_snapshot            18980  0 
dm_mod                 58816  10 dm_mirror,dm_snapshot
thermal                14344  0 
processor              32072  1 thermal
fan                     5764  0 
fuse                   47124  1 
apparmor               40600  0 
commoncap               8320  1 apparmor

Here's the list from a nearly identical sytem that's still got the ipv6 module loading, and that's also failing at both populating the proc ipv6 space fully (same thing - just four of the 6 NICs) and also failing at NAT (in this case DNAT was what I tried):

Module                  Size  Used by
ipt_TOS                 3200  16 
ipt_REJECT              5760  2 
nf_nat_ftp              4352  0 
nf_conntrack_ftp       11136  1 nf_nat_ftp
xt_limit                3584  3 
xt_state                3456  92 
xt_tcpudp               4224  266 
ipt_LOG                 7552  2 
iptable_mangle          3840  1 
iptable_nat             8708  1 
nf_nat                 20012  2 nf_nat_ftp,iptable_nat
nf_conntrack_ipv4      19724  94 iptable_nat
nf_conntrack           65160  6 nf_nat_ftp,nf_conntrack_ftp,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nfnetlink               6936  3 nf_nat,nf_conntrack_ipv4,nf_conntrack
iptable_filter          3968  1 
ip_tables              13924  3 iptable_mangle,iptable_nat,iptable_filter
x_tables               16260  8 ipt_TOS,ipt_REJECT,xt_limit,xt_state,xt_tcpudp,ipt_LOG,iptable_nat,ip_tables
drbd                  208136  1 
cn                      9632  1 drbd
ipv6                  278916  30 
parport_pc             37668  0 
af_packet              24840  2 
lp                     12452  0 
parport                37448  2 parport_pc,lp
loop                   19076  0 
serio_raw               8068  0 
pcspkr                  4224  0 
psmouse                39952  0 
shpchp                 34580  0 
pci_hotplug            32576  1 shpchp
evdev                  11136  0 
sg                     36380  0 
sr_mod                 17700  0 
cdrom                  37408  1 sr_mod
usbhid                 29664  0 
hid                    28928  1 usbhid
ata_piix               17540  0 
ext3                  133640  2 
jbd                    60456  1 ext3
mbcache                 9732  1 ext3
ehci_hcd               36748  0 
ata_generic             8580  0 
libata                125296  2 ata_piix,ata_generic
uhci_hcd               26640  0 
usbcore               138760  4 usbhid,ehci_hcd,uhci_hcd
e1000                 126656  0 
bnx2                  157208  0 
cciss                  61700  6 
scsi_mod              146828  4 sg,sr_mod,libata,cciss
dm_mirror              24320  0 
dm_snapshot            18980  0 
dm_mod                 58816  10 dm_mirror,dm_snapshot
thermal                14344  0 
processor              32072  1 thermal
fan                     5764  0 
fuse                   47124  1 
apparmor               40600  0 
commoncap               8320  1 apparmor

- Whit

#11 lore raw
In reply to Jozsef Kadlecsik
Quoted 4 lines · Jozsef Kadlecsik

But which netfilter modules are loaded in? Again, there can be some auto-loading problem which we cannot figure out without the list of the modules. And as Patrick suggested: do you see the connections in /proc/net/nf_conntrack? Or you have got /proc/net/ip_conntrack instead?

I've just posted the modules list. What should I be looking for between /proc/net/nf_conntrack vs /proc/net/ip_conntrack? Anything that matches the internal or external IP? What should be there so I can check for anomoly, since I don't want to just put raw data to the list.

Thanks,
Whit

#9 lore raw
In reply to Whit Blauvelt
Quoted 5 lines · Whit Blauvelt
Quoted 2 lines · Patrick McHardy

Please post the list of modules loaded and the output of /proc/net/nf_conntrack.

Quoted 7 lines

First here is the list by the system in question, working once the ipv6 module is blocked from loading at boot. Next is the list from a system with identical hardware and near-identical configuration (same firewall rules), but with ipv6 loading - and which also has only 4 of the 6 NICs showing up in the ipv6 proc conf space, and also has NAT (in this case DNAT is what I tested) failing - also where the NICs on the Internet side of things are those coincidentally not showing up with proc ipv6 conf settings.

The firewall rules appear to be different between the two systems, the first one has a lot more references to the IPv4 conntrack module.

Quoted 4 lines

As to the output of /proc/net/nf_conntrack, you just want to see anything, or under specific load? I'm not going to just publicly post the raw data - although both systems have some there - since IPs can identify my client and their clients, which would violate confidentiality.

I'm mainly interested in one or more of the conntrack entries that should get NATed but don't. One entry should be enough, feel free to replace IPs as long as similar IPs still are similar.

Quoted 9 lines

Okay, the fixed system:

Module Size Used by
...

iptable_nat             8708  1
nf_nat                 20012  2 nf_nat_ftp,iptable_nat
nf_conntrack_ipv4      19724  374 iptable_nat
Quoted 7 lines

Here's the list from a nearly identical sytem that's still got the ipv6 module loading, and that's also failing at both populating the proc ipv6 space fully (same thing - just four of the 6 NICs) and also failing at NAT (in this case DNAT was what I tried):

iptable_nat 8708 1 nf_nat 20012 2 nf_nat_ftp,iptable_nat
nf_conntrack_ipv4 19724 94 iptable_nat

Could you figure out whats causing the different amount of references to nf_conntrack_ipv4? "-m state" rules, "-m conntrack" etc. take references, maybe something fails during load?