Skip to contentSkip to conversation

Success routing mark'ed packets - but still confused why it didn't work the first time....

DISCUSSION Eric B. 2 participants 6 messages Apr 8, 2008 – Apr 10, 2008

lore
Share & tools
lore

Eric B.

author
#0 lore raw

Okay - after an unbelievable number of hours struggling with this, I have finally managed to get my server working; responding to packets using 2 different gateways based on 2 different virtual ips. However, I don't understand why my original concept wasn't working.

Step 0) Create a default route in the main routing table for all "regular" packets
Step 1) Create an new iproute2 routing table with a default gw and an associated ip rule that redirects all "mark"ed packets to that table/route. Step 2) Marking the packets:
Originally, I tried mark'ing packets in the mangle Prerouting table destined for my machine's virtual ip. I was under the impression that any response to those packets would maintain the mark and hence be routed through the appropriate routing table created in step 1. However, this didn't work. And without any way of "seeing" whether the outgoing packets were marked, I couldn't tell why they weren't being routed properly (BTW - is there a way to "see" the mark on the packet in the log?)

My solution was to use the mangle Output table to mark all the outgoing packets with their source being the virtual ip. Once I did that, success. My outgoing packets are properly redirected out the appropriate gateways.

However my question now is the root of my confusion. If a packet is mark'ed in the Preroute mangle table, is that mark not supposed to be maintained throughout the life of the packet, including the machine's response to that packet? If not, there is a lot of documentation that indicates as much, including several howtos and guides that seem to indicate it. For example,
http://linux-ip.net/html/linux-ip.html#adv-multi-internet-outbound indicates to set the mark in the mangle preroute table, and then to add an ip rule. However, as much as I tried, that soln just didn't work.

Can anyhow help clear this up a little? When/where/why does that mark get cleared/reset?

Thanks,

Eric

#1 lore raw
Quoted 5 lines · Eric B.

Step 2) Marking the packets:
Originally, I tried mark'ing packets in the mangle Prerouting table destined for my machine's virtual ip. I was under the impression that any response to those packets would maintain the mark and hence be routed through the appropriate routing table created in step 1. However, this didn't work.

No, use CONNMARK as shown in http://dev.computergmbh.de/NF-Cookbook.tx

Quoted 3 lines

And without any way of "seeing" whether the outgoing packets were marked, I couldn't tell why they weren't being routed properly (BTW - is there a way to "see" the mark on the packet in the log?)

LOGMARK :p

Quoted 3 lines

My solution was to use the mangle Output table to mark all the outgoing packets with their source being the virtual ip. Once I did that, success. My outgoing packets are properly redirected out the appropriate gateways.

Make sure you do not set a mark in an input/prerouting chain so that it accidentally hits the routing rule because that would mean incoming packets are diverted before they reach the machine.

Quoted 3 lines

However my question now is the root of my confusion. If a packet is mark'ed in the Preroute mangle table, is that mark not supposed to be maintained throughout the life of the packet,

The mark is maintained during the life of the packet (until you change it of course).

Quoted 2 lines

including the machine's response to that
packet?

Response is a new packet. Only the ctmark (connection mark) can "survive" here.

author
#2 lore raw

"Jan Engelhardt" <[email protected]> wrote in message news:[email protected]...

Quoted 8 lines · Eric B.
Quoted 4 lines

And without any way of "seeing" whether the outgoing packets were marked, I
couldn't tell why they weren't being routed properly (BTW - is there a way to "see" the mark on the packet in the log?)

LOGMARK :p

Ah yes - back to the fun part about having to work on a fixed disto - ie:
RHEL4. :)

Quoted 8 lines
Quoted 3 lines

My solution was to use the mangle Output table to mark all the outgoing packets with their source being the virtual ip. Once I did that, success. My outgoing packets are properly redirected out the appropriate gateways.

Make sure you do not set a mark in an input/prerouting chain so that it accidentally hits the routing rule because that would mean incoming packets are diverted before they reach the machine.

But isn't that exactly what you are doing in your NF-Cookbook.txt? You have:
-t mangle -A PREROUTING -j CONNMARK --restore-mark
Isn't that setting the mark in the prerouting table? Or do you do it that way since the local routing table is the first table, and any packets destined for this machine will be accepted anyhow?

Quoted 5 lines
Quoted 4 lines

However my question now is the root of my confusion. If a packet is mark'ed
in the Preroute mangle table, is that mark not supposed to be maintained throughout the life of the packet,

Quoted 8 lines

The mark is maintained during the life of the packet (until you change it of course).

Quoted 2 lines

including the machine's response to that
packet?

Response is a new packet. Only the ctmark (connection mark) can "survive" here.

Do the ip rules based on the fwmark work on the individual packet's mark value or the conntrack mark, or both?

Thanks for the clarification!

Eric

author
#3 lore raw

"Eric B." <[email protected]> wrote in message
news:[email protected]...

Quoted 5 lines

"Jan Engelhardt" <[email protected]> wrote in message news:[email protected]...

Do the ip rules based on the fwmark work on the individual packet's mark value or the conntrack mark, or both?

If both (as what it seems), why do you bother with the --restore-mark in the PREROUTING chain, if the connection is already marked anyhow? And if not, then why do you not have a --restore-mark in both the FORWARD and OUTPUT chains?

Thanks,

Eric

#4 lore raw
In reply to Eric B.
Quoted 10 lines · Eric B.
Quoted 5 lines

Make sure you do not set a mark in an input/prerouting chain so that it accidentally hits the routing rule because that would mean incoming packets are diverted before they reach the machine.

But isn't that exactly what you are doing in your NF-Cookbook.txt? You have:
-t mangle -A PREROUTING -j CONNMARK --restore-mark
Isn't that setting the mark in the prerouting table?

Yes...

Quoted 3 lines

Or do you do it that
way since the local routing table is the first table, and any packets destined for this machine will be accepted anyhow?

On the very first packet, the incoming interface (or some other kind of parameter perhaps) is recorded by setting a connection mark. This connection mark is then copied to all packets in a given direction so as to route them into the interface they first one came from, for example. Because the connection mark is only set once, it can be assured that all packets will get the same mark.

Quoted 11 lines
Quoted 8 lines

The mark is maintained during the life of the packet (until you change it of course).

Quoted 2 lines

including the machine's response to that
packet?

Response is a new packet. Only the ctmark (connection mark) can "survive" here.

Do the ip rules based on the fwmark work on the individual packet's mark value or the conntrack mark, or both?

routing rules work on packet mark ("nfmark" "fwmark"), whch is why --restore-mark is needed.

author
#5 lore raw

"Jan Engelhardt" <[email protected]> wrote in message news:[email protected]...

Quoted 5 lines
Quoted 2 lines

Do the ip rules based on the fwmark work on the individual packet's mark value or the conntrack mark, or both?

routing rules work on packet mark ("nfmark" "fwmark"), whch is why --restore-mark is needed.

But then, do you not need a --restore-mark in both the FORWARD and OUTPUT chains?

-t mangle -A FORWARD -m connmark --mark 0 -m conntrack --ctstate NEW -j prefout

	-t mangle -A OUTPUT -m connmark --mark 0 -m conntrack --ctstate NEW -j 

prefout

Or are you simply marking the connection and expecting the system to randomly choose whether to output on ppp0 or ppp1?

Thanks,

Eric