Skip to contentSkip to conversation

Random packets loss under x86_64 - routing?

DISCUSSION Peter Kruse 2 participants 5 messages Jan 14, 2005 – Jan 24, 2005

lore
Share & tools
lore
#0 lore raw

kernel: 2.4.28 smp x86_64

Hello,

We experience a problem in our amd64 beowulf clusters and could need some help.
When ping'ing other machines in a cluster on the same subnet, it fails for some machines. But only right after boot and after a day or so of idle time. After some time (a few minutes) the ping packets go through.

Other things we observed:

1. it is not always the same machines that fail
2. if it fails then no packets are sent or received (checked with tcpdump on sending and target host) although all hosts are up.
3. There is no difference if using a 64bit or 32bit ping
4. It does not depend on the network adapter or other hardware, we have machines with different NICs connected to different switches with the same problem.
5. It does however only happen on amd64 (biarch) systems and not on pure i386 systems so it looks like related to the kernel.
6. I have to reboot to reproduce the problem, it's not enough to unload and load the network module.
7. It only happens with ping, not with ssh.

The ping always succeeds when running with the "-r" switch, that bypasses "the normal routing tables and send directly to a host on an attached interface". This makes us think that it indeed it is related to routing - but how?

I can provide an strace output if you think that could help. What else can I do to gather more information?

Please cc to me, as I'm not subscribed, thanks.

Peter

Signature · 3 lines
Peter Kruse <[email protected]>, Chief Software Architect
Q-Leap Networks GmbH
phone: +497071-703171, mobile: +49172-6340044
#1 lore raw
Quoted 31 lines · Peter Kruse

kernel: 2.4.28 smp x86_64

Hello,

We experience a problem in our amd64 beowulf clusters and could need some help.
When ping'ing other machines in a cluster on the same subnet, it fails for some machines. But only right after boot and after a day or so of idle time. After some time (a few minutes) the ping packets go through.

Other things we observed:

1. it is not always the same machines that fail
2. if it fails then no packets are sent or received (checked with tcpdump on sending and target host) although all hosts are up.
3. There is no difference if using a 64bit or 32bit ping
4. It does not depend on the network adapter or other hardware, we have machines with different NICs connected to different switches with the same problem.
5. It does however only happen on amd64 (biarch) systems and not on pure i386 systems so it looks like related to the kernel.
6. I have to reboot to reproduce the problem, it's not enough to unload and load the network module.
7. It only happens with ping, not with ssh.

The ping always succeeds when running with the "-r" switch, that bypasses "the normal routing tables and send directly to a host on an attached interface". This makes us think that it indeed it is related to routing - but how?

Quoted 7 lines

I can provide an strace output if you think that could help. What else can I do to gather more information?

Please cc to me, as I'm not subscribed, thanks.

Peter

When they 'disappear', use arp -d hostname to delete the entry from the ARP tables. Then see if you can ping it. It is possible that the destination machine got re-routed and the new router's HW address wasn't updated in the ARP tables. If this is the case, I don't know hot to 'fix' it, but it's a new data-point. When you have dynamic routing, there needs to be some way to update the ARP tables even though they eventually expire.

The fact that ping -r works seems to show that the ARP table has stale entries in it.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#2 lore raw

Hello,

thanks for your reply

Quoted 9 lines · linux-os

When they 'disappear', use arp -d hostname to delete the entry from the ARP tables. Then see if you can ping it. It is possible that the destination machine got re-routed and the new router's HW address wasn't updated in the ARP tables. If this is the case, I don't know hot to 'fix' it, but it's a new data-point. When you have dynamic routing, there needs to be some way to update the ARP tables even though they eventually expire.

There is no router between sender and destination host, they are on the same subnet and connected on the same switch.

Quoted 3 lines

The fact that ping -r works seems to show that the ARP table has stale entries in it.

Even directly after reboot when the arp table is empty?

Peter

Signature · 3 lines
Peter Kruse <[email protected]>, Chief Software Architect
Q-Leap Networks GmbH
phone: +497071-703171, mobile: +49172-6340044
#3 lore raw
Quoted 15 lines · Peter Kruse

Hello,

thanks for your reply

Quoted 9 lines · linux-os

When they 'disappear', use arp -d hostname to delete the entry from the ARP tables. Then see if you can ping it. It is possible that the destination machine got re-routed and the new router's HW address wasn't updated in the ARP tables. If this is the case, I don't know hot to 'fix' it, but it's a new data-point. When you have dynamic routing, there needs to be some way to update the ARP tables even though they eventually expire.

Quoted 3 lines

There is no router between sender and destination host, they are on the same subnet and connected on the same switch.

I suggest that you may __think__ that there is no router.... But for instance, I can't talk to my own printer here because of some configuration changes made by the "Net Naz^M^M^M Wizards" here. Same network, same wire. It gets "redirected". Basically, everything on this wire is proxy-arped by the default-route machine. There are duplicate packets on the wire and redirections everywhere.

You can look at your ARP table with:

cat /proc/net/arp

Quoted 3 lines
Quoted 3 lines

The fact that ping -r works seems to show that the ARP table has stale entries in it.

The ping -r working shows that there were is either a bad ARP table entry or too small a netmask so the device isn't really on your network.

Quoted 5 lines

Even directly after reboot when the arp table is empty?

Peter

Just check it out. You'd be surprised what you may find. Look at the ARP-table entry. Try to ping something that doesn't respond. Look at the table again. That will tell you what's happening. It's likely that there is an ARP table entry from some 'router' that has been set to proxy-ARP whatever it sees on the wire.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#4 lore raw
In reply to linux-os
Quoted 5 lines · linux-os

The fact that ping -r works seems to show that the ARP table has stale entries in it.

The problem is gone with 2.6.10. We can live with this solution.

Thanks,

Peter

Signature · 3 lines
Peter Kruse <[email protected]>, Chief Software Architect
Q-Leap Networks GmbH
phone: +497071-703171, mobile: +49172-6340044