IPv6 forwarding to TAP-interface with ip6tables fails

Hello, list.

I wonder if the following problem is a bug in netfilter or I'm just doing something wrong, but it wasn't always like that. Problem started probably with some kernel update on router machine, since I can't trace any network layout updates since the point where it still worked, and complete downgrade is quite a bad solution.

What I'm trying to do is to connect with ssh and/or ping6 from 2001:470:1f0b:11de::13 to 2001:470:1f0b:11de::21.
And it works at first (and worked fine before), but stops after some time, which doesn't seem constant or relative to amount of data being forwarded.

"tcpdump -i lan" shows a lot of tcp retransmission packets or icmp6 (in case of ping6), but "tcpdump -i vde" shows none of them (while showing all of them while the connection is still working).

I've tried setting -j LOG to FORWARD chain of ip6tables and it always (both while the connection works and not) shows lines like this:

  Dec 19 14:09:48 kern.warn<4> kernel[-]@damnation: IN=lan OUT=vde
    SRC=2001:0470:1f0b:11de:0000:0000:0000:0013
    DST=2001:0470:1f0b:11de:0000:0000:0000:0021
    LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6
    TYPE=128 CODE=0 ID=29524 SEQ=8 

So, I assume the packet is lost somewhere in netfilter, but I've no idea how to tell why it's getting discarded like that.

Strangely enough (for me), when I type "ping6 2001:0470:1f0b:11de::21" on router machine it gives "Destination unreachable: Address unreachable" for the first packet and then it works for the rest of them from both router and remote machine in "lan" network (whole forwarding thing starts working).
When I stop this ping, situation repeats after some time.

IPv4 forwarding works fine over the same link, but it's NAT there.

Note that aforementioned "vde" is a tun/tap interface of Virtual Distributed Ethernet link, but tcpdump should show the packet on tap interface (vde) regardless of other software, monitoring it, so it's not a VDE-related bug, right?
That's the first question that's bothering me.

It worked a few months ago but I haven't used it actively since then, and now it doesn't. Same IPs, but different software versions (see below).

Is something wrong with the setup?
Are there any settings that can induce aforementioned behavior I should check, apart from the ones, included in this mail?

And if not...
Is there any way to debug the problem futher, apart of inserting (enabling) debug code in the kernel?
Does it look like a bug in netfilter forwarding, should I post it to bugzilla?

Thanks in advance for any answers / suggestions.

ip addr:
2: lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

      link/ether 00:22:15:3a:ef:7d brd ff:ff:ff:ff:ff:ff
      inet 192.168.0.10/29 scope global lan
      inet6 2001:470:1f0b:11de::10/125 scope global 
         valid_lft forever preferred_lft forever
      inet6 2001:470:1f0b:11de::1/125 scope global 
         valid_lft forever preferred_lft forever
      inet6 fe80::222:15ff:fe3a:ef7d/64 scope link 
         valid_lft forever preferred_lft forever
  7: vde: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
      link/ether b2:f5:e7:68:fd:9e brd ff:ff:ff:ff:ff:ff
      inet 192.168.0.20/28 scope global vde
      inet6 2001:470:1f0b:11de::20/125 scope global 
         valid_lft forever preferred_lft forever

ip -6 route:
2001:470:1f0b:11de::/125 dev lan proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 4294967295

  2001:470:1f0b:11de::10/125 dev lan  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
  2001:470:1f0b:11de::20/125 dev vde  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
  fe80::/64 dev lan  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
  fe80::/64 dev vde  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295

ip6tables has following rules (at the start of FORWARD chain):
-I FORWARD -i vde -j ACCEPT
-I FORWARD -i lan -j ACCEPT

Software:
Linux damnation 2.6.32-gentoo-fg.mf_master #1 SMP Sun Dec 13 22:35:05
YEKT 2009 x86_64 Intel(R) Xeon(R) CPU L5420 @ 2.50GHz GenuineIntel GNU/Linux (happened with .31 as well, will try official .32.2 today)
iptables-1.4.5
vde-2.2.3 (shouldn't be relevant?)

IPv6 forwarding is enabled in sysctl.

Mike Kazantsev // fraggod.net

Hello,

[...]

Have you tried to track the packet in the mangle/POSTROUTING chain too, either with the LOG target in that chain or with the TRACE target in the raw/PREROUTING chain ?

This indicates that the neighbour discovery for that address failed, either because the ICMPv6 neighbour solicitation request could not be sent or the ICMPv6 neighbour advertisement reply was not received. If neighbour discovery fails, packets cannot be sent or forwarded to that destination.

[...]

[...]

[...]

Regardless of your problem, 2001:470:1f0b:11de::10 and 2001:470:1f0b:11de::20 are the prefix addresses of 2001:470:1f0b:11de::10/25 and 2001:470:1f0b:11de::20/25 and have a special status : they are anycast addresses meaning "any router connected to that prefix". So I guess that they should not be assigned to an interface.

ICMPv6 packets involved in neighbour discovery go through the INPUT and OUTPUT chains. What do these chains contain ? Connection tracking of these ICMPv6 types changed in kernel 2.6.29.

You wrote in the subject "with ip6tables". Do you mean that it works without ip6tables, without ip6_tables loaded or without ip6table_filter loaded, or with all chains empty and with ACCEPT default policy ?

Didn't know about it, very handy way to check what's happening. Thanks. Pity I've missed it before.

As for the results - it seems that everything is passing through netfilter just fine, more about it below.

Yes, that seem to be the cause of a problem, but then I don't seem to understand why router doesn't send these requests.

I've set TRACE to packets with 2001:470:1f0b:11de::22 as a destination in raw.PREROUTING and -j LOG to INPUT and OUTPUT chains of filter table.

On VM start tcpdump (with "ip6" filter) shows these:
IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 IP6 :: > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::22, length 24 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28

Then I start the simple ping6 from lan network to vde (VM):
IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 1, length 64

TRACE: raw:PREROUTING:policy:2 IN=lan OUT= MAC=00:22:15:3a:ef:7d:00:17:c4:10:7f:4f:86:dd
SRC=2001:0470:1f0b:11de:0000:0000:0000:0013 DST=2001:0470:1f0b:11de:0000:0000:0000:0022
LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
TRACE: mangle:PREROUTING:policy:1 IN=lan OUT= MAC=00:22:15:3a:ef:7d:00:17:c4:10:7f:4f:86:dd
SRC=2001:0470:1f0b:11de:0000:0000:0000:0013 DST=2001:0470:1f0b:11de:0000:0000:0000:0022
LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
TRACE: mangle:FORWARD:policy:1 IN=lan OUT=vde
SRC=2001:0470:1f0b:11de:0000:0000:0000:0013 DST=2001:0470:1f0b:11de:0000:0000:0000:0022
LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
TRACE: filter:FORWARD:policy:1 IN=lan OUT=vde
SRC=2001:0470:1f0b:11de:0000:0000:0000:0013 DST=2001:0470:1f0b:11de:0000:0000:0000:0022
LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1
TRACE: mangle:POSTROUTING:policy:1 IN= OUT=vde
SRC=2001:0470:1f0b:11de:0000:0000:0000:0013 DST=2001:0470:1f0b:11de:0000:0000:0000:0022
LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=29452 SEQ=1

IP6 2001:470:1f0b:11de::22 > ff02::1:ff00:21: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::21, length 32

IN=vde OUT= MAC=33:33:ff:00:00:21:00:16:3e:16:38:41:86:dd
SRC=2001:0470:1f0b:11de:0000:0000:0000:0022 DST=ff02:0000:0000:0000:0000:0001:ff00:0021
LEN=72 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=135 CODE=0

IP6 2001:470:1f0b:11de::21 > 2001:470:1f0b:11de::22: ICMP6, neighbor advertisement, tgt is 2001:470:1f0b:11de::21, length 32

IN= OUT=vde
SRC=2001:0470:1f0b:11de:0000:0000:0000:0021 DST=2001:0470:1f0b:11de:0000:0000:0000:0022
LEN=72 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=136 CODE=0

IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 1, length 64

...and so on until tcpdump just stops, while traces show the same thing.

On lan interface, unlike vde, neighbor requests seem to be passed quite frequently:

...
IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 5, length 64
IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 5, length 64
IP6 fe80::222:15ff:fe3a:ef7d > 2001:470:1f0b:11de::13: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::13, length 32 IP6 2001:470:1f0b:11de::13 > fe80::222:15ff:fe3a:ef7d: ICMP6, neighbor advertisement, tgt is 2001:470:1f0b:11de::13, length 24 IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 6, length 64
IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 6, length 64
...

And when I type "ping6 2001:470:1f0b:11de::22" on router itself I can see neighbor solicitation in tcpdump and pings from lan start working again, although not for long (if I ping on router):

[complete silence]
IP6 2001:470:1f0b:11de::21 > ff02::1:ff00:22: ICMP6, neighbor solicitation, who has 2001:470:1f0b:11de::22, length 32 IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::21: ICMP6, neighbor advertisement, tgt is 2001:470:1f0b:11de::22, length 32 IP6 2001:470:1f0b:11de::13 > 2001:470:1f0b:11de::22: ICMP6, echo request, seq 135, length 64
IP6 2001:470:1f0b:11de::22 > 2001:470:1f0b:11de::13: ICMP6, echo reply, seq 135, length 64
...

So I guess that pinpoints the problem as "router doesn't send neighbor solicitation requests along with forwarded packets", thanks to your suggestion.

I have to admit that "neighbor solicitation", although basic IPv6 mechanism, is a obscure to me, since I haven't done necessary research on the topic before setting the network up, and that's quite a stupid mistake I'm going to fix right now ;)

Guess that gives me something to search for and debug further, so I'll probably be able to either fix it or come up with another, more specific, question.
Thanks for your help.

While that worked before, I agree that it's a bad practice, and I've fixed it now, although it doesn't seem to affect this particular problem indeed.

For the above tests I've changed ...::10 to 11, ...::20 to 21 and ...::21 to 22.

These (as well as the rest of ICMPv6) were unrestricted in all chains, but for the sake of this test (dumps/traces posted above) I've set everything to ACCEPT explicitly.

My mistake, sorry, ip6tables tool probably have nothing to do with it, it's just that the whole forwarding concept is closely associated with iptables in my mind ;)

ACCEPT everywhere doesn't affect the test, and above results acquired with just that setup.

Mike Kazantsev // fraggod.net

...

Just looking at these lines as I pasted them actually helped: ICMP6 neighbor solicitation request was sent from 22 to 21, while the packets were originated from 21.
Made me wondering about what kind of backwards magic is at work here...

...

And here's the problem: no link-local address on the interface. "ip addr flush dev vde" in init.d script setting up interface did that, that was the mistake.

With this address added (or rather preserved), router is able to send neighbor solicitation requests from it and everything works fine.

Anyway, strange thing is that, apparently, VM on the other side is able to send these requests without link-local address, using global one instead:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
inet6 2001:470:1f0b:11de::22/125 scope global
valid_lft forever preferred_lft forever

Guess older kernel (2.6.30.4) there is the cause of it, and that's also why it worked before with pretty much the same setup.

Thanks again to Pascal for pointing me in the right direction.

Mike Kazantsev // fraggod.net

Good catch. I didn't notice it.

[...]

I observed that when sending a locally generated packet, Linux uses the source address (which is one one of its own addresses) of the packet as the source address in the neighbour solicitation, whereas when forwarding a packet it uses the link-local address attached to the output interface. So there may be trouble when forwarding a packet and the output interface has no link-local address. The tcpdump trace in your previous message shows a neighbour solicitation packet with unspecified source address :

This may explain the different results with pings from the LAN and from the router, and also why VM does not have the problem even though it has no link-local address.