VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374
Subsystems (3)
Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.
Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0
ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1
ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srv
ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997
In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.
Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?
Quoted 23 lines · hanhuihui
Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed40867d74c374("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srvip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997
Quoted 8 lines
In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?
Can you try replacing the first two rules with:
ip rule add from 11.47.3.130 l3mdev prio 0
And see if it helps?
It's not exactly equivalent to your two rules, but it says "if source address is 11.47.3.130 and flow is associated with a L3 master device, then direct the FIB lookup to the table associated with the L3 master device"
The commit you referenced added the index of the L3 master device to the flow structure, but I don't believe we have an explicit way to match on it using FIB rules. It would be useful to add a new keyword (e.g., l3mdevif) and then your rules can become:
ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997
But it requires kernel changes.
Quoted 29 lines · Ido Schimmel
Quoted 28 lines · hanhuihui
Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed40867d74c374("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srvip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997
In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.
Quoted 8 lines
Quoted 3 lines
Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?Can you try replacing the first two rules with:
ip rule add from 11.47.3.130 l3mdev prio 0
This does not work in scenarios where the routing table specified by the oif/iif is not in the l3mdev-table.
Quoted 12 lines
And see if it helps?
It's not exactly equivalent to your two rules, but it says "if source address is 11.47.3.130 and flow is associated with a L3 master device, then direct the FIB lookup to the table associated with the L3 master device"
The commit you referenced added the index of the L3 master device to the flow structure, but I don't believe we have an explicit way to match on it using FIB rules. It would be useful to add a new keyword (e.g., l3mdevif) and then your rules can become:
Quoted 4 lines
ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997
But it requires kernel changes.
Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. However, in this patch, traffic from the VRF resets fl->flowi_oif in l3mdev_update_flow. As a result, the 'rule->oifindex != fl->flowi_oif' condition in fib_rule_match cannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; reset to avoid oif match in fib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after the patch is installed, or is the usage of the oif/iif rule incorrectly used?
Any reply would be greatly appreciated.
Thanks!
David Ahern
maintainerQuoted 8 lines · hanhuihui
Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. However, in this patch, traffic from the VRF resets
fl->flowi_oifinl3mdev_update_flow. As a result, the 'rule->oifindex!=fl->flowi_oif' condition infib_rule_matchcannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; reset to avoid oif match infib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after the patch is installed, or is the usage of the oif/iif rule incorrectly used?
Any reply would be greatly appreciated.
oif/iif rules per VRF does not scale; the l3mdev rule was added to address that problem.
Quoted 34 lines · hanhuihui
Quoted 33 lines · Ido Schimmel
Quoted 31 lines · hanhuihui
Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed40867d74c374("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule. Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srvip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997
In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?
Quoted 6 lines
Quoted 4 lines
Can you try replacing the first two rules with:
ip rule add from 11.47.3.130 l3mdev prio 0
This does not work in scenarios where the routing table specified by the oif/iif is not in the l3mdev-table.
You mean when "oif" / "iif" points to a VRF and "table" specifies a table different than the one associated with the VRF? Then, yes, it won't work, but in your example "vrf-srv-1" is associated with table "10".
Quoted 18 lines
Quoted 13 lines
And see if it helps?
It's not exactly equivalent to your two rules, but it says "if source address is 11.47.3.130 and flow is associated with a L3 master device, then direct the FIB lookup to the table associated with the L3 master device"
The commit you referenced added the index of the L3 master device to the flow structure, but I don't believe we have an explicit way to match on it using FIB rules. It would be useful to add a new keyword (e.g., l3mdevif) and then your rules can become:
Quoted 4 lines
ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0 ip rule add from 11.47.3.130 table 20 prio 997
But it requires kernel changes.
Quoted 7 lines
Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. However, in this patch, traffic from the VRF resets
fl->flowi_oifinl3mdev_update_flow. As a result, the 'rule->oifindex!=fl->flowi_oif' condition infib_rule_matchcannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; reset to avoid oif match infib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after the patch is installed, or is the usage of the oif/iif rule incorrectly used?
Any reply would be greatly appreciated.
Before 40867d74c374 you couldn't match with FIB rules on oif/iif being a VRF slave because these fields in the flow structure were reset to the index of the VRF device in l3mdev_update_flow().
I will try to check if we can interpret FIB rules with "oif" / "iif" pointing to a VRF as matching on "fl->flowi_l3mdev" rather than "fl->flowi_oif" / "fl->flowi_iif".
Quoted 7 lines · On Tue, 8 Apr 2025 22:54:33 +0300 Ido Schimmel wrote:
Before
40867d74c374you couldn't match with FIB rules on oif/iif being a VRF slave because these fields in the flow structure were reset to the index of the VRF device inl3mdev_update_flow().I will try to check if we can interpret FIB rules with "oif" / "iif" pointing to a VRF as matching on "
fl->flowi_l3mdev" rather than "fl->flowi_oif" / "fl->flowi_iif".
Thank you for your reply. Followed by a simple patch to restore the previous behavior, hopefully enlightening you.
PATCH 1/1 resume oif rule match l3mdev in fib_lookup
In reply to Ido Schimmelflowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device in fib_lookup. Let's get back to previous behavior.
--- net/core/fib_rules.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/fib_rules.c +1 -1
@@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops, goto out; oifindex =READ_ONCE (rule->oifindex); -if (oifindex && (oifindex != fl->flowi_oif)) +if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif))) goto out;if ((rule->mark ^ fl->flowi_mark) & rule->mark_mask)
Signature · 1 line
2.27.0
Jakub Kicinski
maintainerRe: [PATCH] resume oif rule match l3mdev in fib_lookup
Quoted 6 lines · On Sat, 12 Apr 2025 21:19:10 +0800 hanhuihui wrote:
flowi_oifwill be reset ifflowi_oifset to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device infib_lookup. Let's get back to previous behavior.Fixes 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")Signed-off-by hanhuihui [email protected]
If you'd like to go ahead with this approach you need to at the very least adjust the fib_rule_tests.sh selftest..
Re: [PATCH] resume oif rule match l3mdev in fib_lookup
In reply to hanhuihuiQuoted 15 lines · hanhuihui
flowi_oifwill be reset ifflowi_oifset to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device infib_lookup. Let's get back to previous behavior.Fixes 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")Signed-off-by hanhuihui [email protected]--- net/core/fib_rules.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)net/core/fib_rules.c +0
-0
net/core/fib_rules.c static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops, · line 271
Quoted 5 lines
net/core/fib_rules.c +1
-1@@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops, goto out; oifindex =READ_ONCE (rule->oifindex); -if (oifindex && (oifindex != fl->flowi_oif)) +if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))
This will prevent us from matching on the output device when the device is enslaved to a VRF. We should try to match on L3 domain only if the FIB rule matches on a VRF device. I will try to send a fix today (wasn't feeling well in the last few days).
Quoted 6 lines
net/core/fib_rules.c +0
-0goto out;if ((rule->mark ^ fl->flowi_mark) & rule->mark_mask)Signature · 1 line
2.27.0
Re: [PATCH] resume oif rule match l3mdev in fib_lookup
Quoted 18 lines · Ido Schimmel
Quoted 17 lines · hanhuihui
flowi_oifwill be reset ifflowi_oifset to a l3mdev (e.g., VRF) device. This causes the oif rule to fail to match the l3mdev device infib_lookup. Let's get back to previous behavior.Fixes 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")Signed-off-by hanhuihui [email protected]--- net/core/fib_rules.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)net/core/fib_rules.c +0
-0@@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops, goto out;
Quoted 8 lines
net/core/fib_rules.c static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops, · line 271
Quoted 3 lines
net/core/fib_rules.c +1
-1oifindex =READ_ONCE (rule->oifindex); -if (oifindex && (oifindex != fl->flowi_oif)) +if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))This will prevent us from matching on the output device when the device is enslaved to a VRF. We should try to match on L3 domain only if the FIB rule matches on a VRF device. I will try to send a fix today (wasn't feeling well in the last few days).
Posted a fix:
https://lore.kernel.org/netdev/[email protected]/
Can you please test it and tag it if it fixes your issue?
Thanks
hanhuihui
Tested-byRe: [PATCH] resume oif rule match l3mdev in fib_lookup
Quoted 1 line · Ido Schimmel
On Mon, Apr 14, 2025 at 10:17:35AM +0300, Ido Schimmel wrote:
..
Quoted 9 lines
Quoted 4 lines
This will prevent us from matching on the output device when the device is enslaved to a VRF. We should try to match on L3 domain only if the FIB rule matches on a VRF device. I will try to send a fix today (wasn't feeling well in the last few days).
Posted a fix:
https://lore.kernel.org/netdev/[email protected]/Can you please test it and tag it if it fixes your issue?
The patch successfully resolves the VRF oif rule matching issue. The FIB lookup now correctly identifies the L3 domain when using VRFs. Thank you for the quick fix.