Skip to contentSkip to conversation

VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

PATCH In review hanhuihui 4 participants 11 messages Apr 3, 2025 – Apr 15, 2025

Subsystems (3)
lore
Share & tools
lore

hanhuihui

author
#0 lore raw

Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.

Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0

ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1

ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srv

ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997

In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.

Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?

#1 lore raw
Quoted 23 lines · hanhuihui

Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.

Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0

ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1

ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srv

ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997

Quoted 8 lines

In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.

Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?

Can you try replacing the first two rules with:

ip rule add from 11.47.3.130 l3mdev prio 0

And see if it helps?

It's not exactly equivalent to your two rules, but it says "if source address is 11.47.3.130 and flow is associated with a L3 master device, then direct the FIB lookup to the table associated with the L3 master device"

The commit you referenced added the index of the L3 master device to the flow structure, but I don't believe we have an explicit way to match on it using FIB rules. It would be useful to add a new keyword (e.g., l3mdevif) and then your rules can become:

ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997

But it requires kernel changes.

author
#2 lore raw
Quoted 29 lines · Ido Schimmel
Quoted 28 lines · hanhuihui

Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.

Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0

ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1

ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srv

ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997

In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.

Quoted 8 lines
Quoted 3 lines

Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?

Can you try replacing the first two rules with:

ip rule add from 11.47.3.130 l3mdev prio 0

This does not work in scenarios where the routing table specified by the oif/iif is not in the l3mdev-table.

Quoted 12 lines

And see if it helps?

It's not exactly equivalent to your two rules, but it says "if source address is 11.47.3.130 and flow is associated with a L3 master device, then direct the FIB lookup to the table associated with the L3 master device"

The commit you referenced added the index of the L3 master device to the flow structure, but I don't believe we have an explicit way to match on it using FIB rules. It would be useful to add a new keyword (e.g., l3mdevif) and then your rules can become:

Quoted 4 lines

ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997

But it requires kernel changes.

Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. However, in this patch, traffic from the VRF resets fl->flowi_oif in l3mdev_update_flow. As a result, the 'rule->oifindex != fl->flowi_oif' condition in fib_rule_match cannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; reset to avoid oif match in fib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after the patch is installed, or is the usage of the oif/iif rule incorrectly used?
Any reply would be greatly appreciated.

Thanks!

maintainer
#3 lore raw
Quoted 8 lines · hanhuihui

Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. However, in this patch, traffic from the VRF resets fl->flowi_oif in l3mdev_update_flow. As a result, the 'rule->oifindex != fl->flowi_oif' condition in fib_rule_match cannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; reset to avoid oif match in fib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after the patch is installed, or is the usage of the oif/iif rule incorrectly used?
Any reply would be greatly appreciated.

oif/iif rules per VRF does not scale; the l3mdev rule was added to address that problem.

#4 lore raw
In reply to hanhuihui
Quoted 34 lines · hanhuihui
Quoted 33 lines · Ido Schimmel
Quoted 31 lines · hanhuihui

Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.

Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0

ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1

ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srv

ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997

In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.

Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?

Quoted 6 lines
Quoted 4 lines

Can you try replacing the first two rules with:

ip rule add from 11.47.3.130 l3mdev prio 0

This does not work in scenarios where the routing table specified by the oif/iif is not in the l3mdev-table.

You mean when "oif" / "iif" points to a VRF and "table" specifies a table different than the one associated with the VRF? Then, yes, it won't work, but in your example "vrf-srv-1" is associated with table "10".

Quoted 18 lines
Quoted 13 lines

And see if it helps?

It's not exactly equivalent to your two rules, but it says "if source address is 11.47.3.130 and flow is associated with a L3 master device, then direct the FIB lookup to the table associated with the L3 master device"

The commit you referenced added the index of the L3 master device to the flow structure, but I don't believe we have an explicit way to match on it using FIB rules. It would be useful to add a new keyword (e.g., l3mdevif) and then your rules can become:

Quoted 4 lines

ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997

But it requires kernel changes.

Quoted 7 lines

Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. However, in this patch, traffic from the VRF resets fl->flowi_oif in l3mdev_update_flow. As a result, the 'rule->oifindex != fl->flowi_oif' condition in fib_rule_match cannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; reset to avoid oif match in fib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after the patch is installed, or is the usage of the oif/iif rule incorrectly used?
Any reply would be greatly appreciated.

Before 40867d74c374 you couldn't match with FIB rules on oif/iif being a VRF slave because these fields in the flow structure were reset to the index of the VRF device in l3mdev_update_flow().

I will try to check if we can interpret FIB rules with "oif" / "iif" pointing to a VRF as matching on "fl->flowi_l3mdev" rather than "fl->flowi_oif" / "fl->flowi_iif".

author
#5 lore raw
Quoted 7 lines · On Tue, 8 Apr 2025 22:54:33 +0300 Ido Schimmel wrote:

Before 40867d74c374 you couldn't match with FIB rules on oif/iif being a VRF slave because these fields in the flow structure were reset to the index of the VRF device in l3mdev_update_flow().

I will try to check if we can interpret FIB rules with "oif" / "iif" pointing to a VRF as matching on "fl->flowi_l3mdev" rather than "fl->flowi_oif" / "fl->flowi_iif".

Thank you for your reply. Followed by a simple patch to restore the previous behavior, hopefully enlightening you.

author
#6 lore raw

PATCH 1/1 resume oif rule match l3mdev in fib_lookup

In reply to Ido Schimmel

flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device in fib_lookup. Let's get back to previous behavior.

Fixes 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")
---
 net/core/fib_rules.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/fib_rules.c +1 -1
index 4bc64d9..3c2a2db 100644
@@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops,
 		goto out;
 
 	oifindex = READ_ONCE(rule->oifindex);
-	if (oifindex && (oifindex != fl->flowi_oif))
+	if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))
 		goto out;
 
 	if ((rule->mark ^ fl->flowi_mark) & rule->mark_mask)
Signature · 1 line
2.27.0
maintainer
#7 lore raw

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

Quoted 6 lines · On Sat, 12 Apr 2025 21:19:10 +0800 hanhuihui wrote:

flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device in fib_lookup. Let's get back to previous behavior.

Fixes 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")

If you'd like to go ahead with this approach you need to at the very least adjust the fib_rule_tests.sh selftest..

#8 lore raw

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

In reply to hanhuihui
Quoted 15 lines · hanhuihui

flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device in fib_lookup. Let's get back to previous behavior.

Fixes 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")
---
 net/core/fib_rules.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/fib_rules.c +0 -0
index 4bc64d9..3c2a2db 100644

net/core/fib_rules.c static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops, · line 271

Quoted 5 lines
net/core/fib_rules.c +1 -1
@@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops,
 		goto out;
 
 	oifindex = READ_ONCE(rule->oifindex);
-	if (oifindex && (oifindex != fl->flowi_oif))
+	if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))

This will prevent us from matching on the output device when the device is enslaved to a VRF. We should try to match on L3 domain only if the FIB rule matches on a VRF device. I will try to send a fix today (wasn't feeling well in the last few days).

Quoted 6 lines
net/core/fib_rules.c +0 -0
 		goto out;
 
 	if ((rule->mark ^ fl->flowi_mark) & rule->mark_mask)
Signature · 1 line
2.27.0
#9 lore raw

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

Quoted 18 lines · Ido Schimmel
Quoted 17 lines · hanhuihui

flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device in fib_lookup. Let's get back to previous behavior.

Fixes 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")
---
 net/core/fib_rules.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/fib_rules.c +0 -0
index 4bc64d9..3c2a2db 100644
@@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops,
 		goto out;
 
Quoted 8 lines

net/core/fib_rules.c static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops, · line 271

Quoted 3 lines
net/core/fib_rules.c +1 -1
 	oifindex = READ_ONCE(rule->oifindex);
-	if (oifindex && (oifindex != fl->flowi_oif))
+	if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))

This will prevent us from matching on the output device when the device is enslaved to a VRF. We should try to match on L3 domain only if the FIB rule matches on a VRF device. I will try to send a fix today (wasn't feeling well in the last few days).

Posted a fix:
https://lore.kernel.org/netdev/[email protected]/

Can you please test it and tag it if it fixes your issue?

Thanks

author Tested-by
#10 lore raw

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

Quoted 1 line · Ido Schimmel

On Mon, Apr 14, 2025 at 10:17:35AM +0300, Ido Schimmel wrote:

..

Quoted 9 lines
Quoted 4 lines

This will prevent us from matching on the output device when the device is enslaved to a VRF. We should try to match on L3 domain only if the FIB rule matches on a VRF device. I will try to send a fix today (wasn't feeling well in the last few days).

Posted a fix:
https://lore.kernel.org/netdev/[email protected]/

Can you please test it and tag it if it fixes your issue?

The patch successfully resolves the VRF oif rule matching issue. The FIB lookup now correctly identifies the L3 domain when using VRFs. Thank you for the quick fix.