Skip to contentSkip to conversation

"dst cache overflow" messages and crash

LARTC Frédéric Massot 3 participants 8 messages Feb 21, 2007 – Apr 6, 2007

Subsystems (3)
lore
Share & tools
lore
#0 lore raw
Quoted 2 lines · Ivan Vladimirov
Quoted 1 line · Frédéric Massot

Hi,

[...]

Quoted 13 lines
Quoted 13 lines

gc_thresh:4096
gc_timeout:300
max_delay:10
max_size:65536
min_adv_mss:256
min_delay:2
min_pmtu:552
mtu_expires:600
redirect_load:5
redirect_number:9
redirect_silence:5120
secret_interval:600

Quoted 8 lines
Quoted 6 lines

I can increase the maximum size of the cache, but that will do nothing but delay the crash.

Can you help me?

Regards.

max_sizee536
is to low increase size to 256k

Hi,

Thank you for your answer.

This change prevents the server crach or does nothing but delay it?

Regards.

Signature · 8 lines
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:[email protected]   |
=============Þbian=GNU/Linux=_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#1 lore raw
Quoted 26 lines · Frédéric Massot
Quoted 2 lines · Ivan Vladimirov
Quoted 1 line · Frédéric Massot

Hi,

[...]

Quoted 13 lines
Quoted 13 lines

gc_thresh:4096
gc_timeout:300
max_delay:10
max_size:65536
min_adv_mss:256
min_delay:2
min_pmtu:552
mtu_expires:600
redirect_load:5
redirect_number:9
redirect_silence:5120
secret_interval:600

Quoted 8 lines
Quoted 6 lines

I can increase the maximum size of the cache, but that will do nothing but delay the crash.

Can you help me?

Regards.

max_sizee536
is to low increase size to 256k

Quoted 7 lines

Hi,

Thank you for your answer.

This change prevents the server crach or does nothing but delay it?

Regards.

This change prevents server crash but you need careful tweaking of this parameter
use slabtop to see how much new objects in dstcache you have over 5min Also value of this parameter depends on type of traffic you have and number of destination routes which server have to cache.

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#2 lore raw
In reply to Frédéric Massot
Quoted 30 lines · Ivan Vladimirov
Quoted 29 lines · Frédéric Massot

Hi,

I regularly have errors (kernel: dst cache overflow) and crash of a firewall under Linux 2.6.17 and the route patch from Julian Anastasov.

With rtstat I see that the route cache size increases regularly without never decreasing.

I have this parameters:

fw:/proc/sys/net/ipv4/route# grep . *
error_burst:1250
error_cost:250
gc_elasticity:15
gc_interval:60
gc_min_interval:0
gc_min_interval_ms:500
gc_thresh:4096
gc_timeout:300
max_delay:10
max_size:65536
min_adv_mss:256
min_delay:2
min_pmtu:552
mtu_expires:600
redirect_load:5
redirect_number:9
redirect_silence:5120
secret_interval:600
Quoted 8 lines
Quoted 6 lines

I can increase the maximum size of the cache, but that will do nothing but delay the crash.

Can you help me?

Regards.

max_sizee536
is to low increase size to 256k

Hi,

I supervised the system during a few weeks with slabtop and rtstat.

What I could see, it is that the ip_dst_cache cache grow without never being cleaned by the garbage collector.

At the end of a few days, the traffic is slowed down and the customer restart the firewall. When the cache reaches its maximum value there is the error message (kernel: dst cache overflow) and the traffic is really disturbed.

I use the Linux kernel 2.6.17 and the route patch from Julian Anastasov.

- The bug comes from the kernel or the patch?

- Do you know if this bug were corrected in the new versions of the kernel?

Regards.

Signature · 9 lines
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:[email protected]   |
=============Þbian=GNU/Linux=
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#3 lore raw
In reply to Frédéric Massot

Patches from Julian Anastasov works only as he mentioned for 2.4 series of kernels
His patches are untested for 2.6 kernels
2.6.17 kernel das not suppose to have this bug cos it was fixed earlier in 2.6.16
My advise is to switch from 2.6.17 to 2.6.16.29 or lather and avoid patches from Julian
there are other ways to perform same task without patches.

Quoted 52 lines · Frédéric Massot
Quoted 30 lines · Ivan Vladimirov
Quoted 29 lines · Frédéric Massot

Hi,

I regularly have errors (kernel: dst cache overflow) and crash of a firewall under Linux 2.6.17 and the route patch from Julian Anastasov.

With rtstat I see that the route cache size increases regularly without never decreasing.

I have this parameters:

fw:/proc/sys/net/ipv4/route# grep . *
error_burst:1250
error_cost:250
gc_elasticity:15
gc_interval:60
gc_min_interval:0
gc_min_interval_ms:500
gc_thresh:4096
gc_timeout:300
max_delay:10
max_size:65536
min_adv_mss:256
min_delay:2
min_pmtu:552
mtu_expires:600
redirect_load:5
redirect_number:9
redirect_silence:5120
secret_interval:600
Quoted 8 lines
Quoted 6 lines

I can increase the maximum size of the cache, but that will do nothing but delay the crash.

Can you help me?

Regards.

max_sizee536
is to low increase size to 256k

Hi,

I supervised the system during a few weeks with slabtop and rtstat.

What I could see, it is that the ip_dst_cache cache grow without never being cleaned by the garbage collector.

At the end of a few days, the traffic is slowed down and the customer restart the firewall. When the cache reaches its maximum value there is the error message (kernel: dst cache overflow) and the traffic is really disturbed.

Quoted 8 lines

I use the Linux kernel 2.6.17 and the route patch from Julian Anastasov.

- The bug comes from the kernel or the patch?

- Do you know if this bug were corrected in the new versions of the kernel?

Regards.

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#4 lore raw
In reply to Frédéric Massot

Hello,

Quoted 2 lines · On Thu, 22 Mar 2007, [ISO-8859-15] Frédéric Massot wrote:
Quoted 2 lines
Quoted 2 lines

I regularly have errors (kernel: dst cache overflow) and crash of a firewall under Linux 2.6.17 and the route patch from Julian Anastasov.

I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
BRIDGE_NETFILTER enabled/used?

Quoted 5 lines

I use the Linux kernel 2.6.17 and the route patch from Julian Anastasov.

- The bug comes from the kernel or the patch?

- Do you know if this bug were corrected in the new versions of the kernel?

It should be the patch that triggers the problem.

Regards

--
Julian Anastasov <[email protected]>
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#5 lore raw
In reply to Frédéric Massot
Quoted 9 lines · Julian Anastasov

Hello,

Quoted 2 lines · On Thu, 22 Mar 2007, [ISO-8859-15] Frédéric Massot wrote:
Quoted 2 lines
Quoted 2 lines

I regularly have errors (kernel: dst cache overflow) and crash of a firewall under Linux 2.6.17 and the route patch from Julian Anastasov.

I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
BRIDGE_NETFILTER enabled/used?

Hi,

Thank you for your answer.

- IP_ROUTE_MULTIPATH_CACHED is not set
- BRIDGE_NETFILTER is set, but I do not use it.

Regards.

Signature · 8 lines
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:[email protected]   |
=============Þbian=GNU/Linux=_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#6 lore raw
In reply to Frédéric Massot

Hello,

Quoted 8 lines · Frédéric Massot
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I regularly have errors (kernel: dst cache overflow) and crash of a firewall under Linux 2.6.17 and the route patch from Julian Anastasov.

I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
BRIDGE_NETFILTER enabled/used?

- IP_ROUTE_MULTIPATH_CACHED is not set
- BRIDGE_NETFILTER is set, but I do not use it.

ok, then can you try the attached patch, it solves dst cache
problem for another user, may be it will help you too. This patch can be used with or without routes patches. It makes sure we don't leak dst entry in bridge-netfilter. If the patch does not help let me know and we can add some printks to catch the problem.

Regards

--
Julian Anastasov <[email protected]>

	Bridge-Netfilter should be gentle when calling ip_route_input,
	skb->dst can be already set.
net/bridge/br_netfilter.c renamed from linux/net/bridge/br_netfilter.c +4 -0
diff -urp v2.6.20/linux/net/bridge/br_netfilter.c linux/net/bridge/br_netfilter.c
@@ -269,6 +269,10 @@ static int br_nf_pre_routing_finish(stru
 	struct nf_bridge_info *nf_bridge = skb->nf_bridge;
 	int err;
 
+	/* Old skb->dst is not expected, it is lost in all cases */
+	dst_release(skb->dst);
+	skb->dst = NULL;
+
 	if (nf_bridge->mask & BRNF_PKT_TYPE) {
 		skb->pkt_type = PACKET_OTHERHOST;
 		nf_bridge->mask ^= BRNF_PKT_TYPE;

_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#7 lore raw
In reply to Frédéric Massot
Quoted 8 lines · Julian Anastasov

Hello,

Quoted 4 lines · Frédéric Massot
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I regularly have errors (kernel: dst cache overflow) and crash of a firewall under Linux 2.6.17 and the route patch from Julian Anastasov.

I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
BRIDGE_NETFILTER enabled/used?

Quoted 8 lines
Quoted 2 lines

- IP_ROUTE_MULTIPATH_CACHED is not set
- BRIDGE_NETFILTER is set, but I do not use it.

ok, then can you try the attached patch, it solves dst cache
problem for another user, may be it will help you too. This patch can be used with or without routes patches. It makes sure we don't leak dst entry in bridge-netfilter. If the patch does not help let me know and we can add some printks to catch the problem.

Hi,

Thank you for your answer, as your patch comes from the kernel 2.6.20, I installed this version of the kernel with the patch (routes-2.6.20-14.diff).

That made a little more than one week that I supervise and it cache is well cleaned regularly. All seems to be good. :o)

Regards.

Signature · 8 lines
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:[email protected]   |
=============Þbian=GNU/Linux=_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc