get_hz() with CONFIG_HIGH_RES_TIMERS

Hi Stephen,

there seems to be something wrong with the timer values as processed by at least the "ss" and "ip" commands when CONFIG_HIGH_RES_TIMERS is on.

$ zgrep -e HZ= -e HIGH_RES /proc/config.gz
CONFIG_HIGH_RES_TIMERS=y
CONFIG_HZ=250

For instance, the "ip route rtt <time>" and rto values returned by PROC_NET_TCP=/proc/net/tcp ss -i look incorrect:

$ PROC_NET_TCP=/proc/net/tcp ss -i
State       Recv-Q Send-Q  Local Address:Port      Peer Address:Port
ESTAB       0      0           127.0.0.1:35466        127.0.0.1:10198    rto:5.5e-08
[...]

$ ss -i
State       Recv-Q Send-Q  Local Address:Port      Peer Address:Port
ESTAB       0      0           127.0.0.1:35466        127.0.0.1:10198
        bic wscale:7,7 rto:220 rtt:23.5/19 send 11.2Mbps rcv_space:32792

$ ip route show
10.95.131.111 via 10.95.128.1 dev eth0  rtt 5ms

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.95.131.111   10.95.128.1     255.255.255.255 UGH       0 0     5000000 eth0

I think this is because get_hz gets the HZ value from the fourth field in /proc/net/psched. That's 1e9 in my case, because that's meant to be the frequency of CLOCK_MONOTONIC and when CONFIG_HIGH_RES_TIMERS is on, the CLOCK_MONOTONIC resolution is 1ns. So get_hz returns 1,000,000,000 instead of 250 in my case.

I don't know how to get the right value, clock_getres return 1ns as well.

That's with 2.6.24.2, iproute-20080108 from debian (but the GIT head code looks the same).

The neighbor table time of last use information is returned in the incorrect unit. Kernel to user space ABI's need to use USER_HZ (or milliseconds), otherwise the application has to try and discover the real system HZ value which is problematic. Linux has standardized on keeping USER_HZ consistent (100hz) even when kernel is running internally at some other value.

This change is small, but it breaks the ABI for older version of iproute2 utilities. But these utilities are already broken since they are looking at the psched_hz values which are completely different. So let's just go ahead and fix both kernel and user space. Older utilities will just print wrong values.

Signed-off-by Stephen Hemminger <[email protected]>

From: Stephen Hemminger <[email protected]>
Date: Wed, 21 May 2008 10:40:19 -0700

Quoted 5 lines

This change is small, but it breaks the ABI for older version of iproute2 utilities. But these utilities are already broken since they are looking at the psched_hz values which are completely different. So let's just go ahead and fix both kernel and user space. Older utilities will just print wrong values.

Can we add a new attribute or something like that?

* David Miller <[email protected]> 2008-05-21 13:35

Quoted 9 lines

From: Stephen Hemminger <[email protected]>
Date: Wed, 21 May 2008 10:40:19 -0700

Quoted 5 lines

This change is small, but it breaks the ABI for older version of iproute2 utilities. But these utilities are already broken since they are looking at the psched_hz values which are completely different. So let's just go ahead and fix both kernel and user space. Older utilities will just print wrong values.

Quoted 1 line

Can we add a new attribute or something like that?

We could do but I agree with Stephen to just fix it the way he proposes. The value we're talking is only useful in a debugging or statistical context. We're not changing the format of the attribute at all, not even the unit, srictly speaking.

Hi Stephen,

there seems to be something wrong with the timer values as processed by at least the "ss" and "ip" commands when CONFIG_HIGH_RES_TIMERS is on.

$ zgrep -e HZ= -e HIGH_RES /proc/config.gz
CONFIG_HIGH_RES_TIMERS=y
CONFIG_HZ=250

For instance, the "ip route rtt <time>" and rto values returned by PROC_NET_TCP=/proc/net/tcp ss -i look incorrect:

$ PROC_NET_TCP=/proc/net/tcp ss -i
State       Recv-Q Send-Q  Local Address:Port      Peer Address:Port
ESTAB       0      0           127.0.0.1:35466        127.0.0.1:10198    rto:5.5e-08
[...]

$ ss -i
State       Recv-Q Send-Q  Local Address:Port      Peer Address:Port
ESTAB       0      0           127.0.0.1:35466        127.0.0.1:10198
        bic wscale:7,7 rto:220 rtt:23.5/19 send 11.2Mbps rcv_space:32792

$ ip route show
10.95.131.111 via 10.95.128.1 dev eth0  rtt 5ms

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.95.131.111   10.95.128.1     255.255.255.255 UGH       0 0     5000000 eth0

I think this is because get_hz gets the HZ value from the fourth field in /proc/net/psched. That's 1e9 in my case, because that's meant to be the frequency of CLOCK_MONOTONIC and when CONFIG_HIGH_RES_TIMERS is on, the CLOCK_MONOTONIC resolution is 1ns. So get_hz returns 1,000,000,000 instead of 250 in my case.

I don't know how to get the right value, clock_getres return 1ns as well.

That's with 2.6.24.2, iproute-20080108 from debian (but the GIT head code looks the same).

Best regards,
Stéphane

Quoted 42 lines · On Wed, 21 May 2008 17:38:15 +0100 Stephane Chazelas…

Hi Stephen,

there seems to be something wrong with the timer values as processed by at least the "ss" and "ip" commands when CONFIG_HIGH_RES_TIMERS is on.

$ zgrep -e HZ= -e HIGH_RES /proc/config.gz
CONFIG_HIGH_RES_TIMERS=y
CONFIG_HZ=250

For instance, the "ip route rtt <time>" and rto values returned by PROC_NET_TCP=/proc/net/tcp ss -i look incorrect:

$ PROC_NET_TCP=/proc/net/tcp ss -i
State       Recv-Q Send-Q  Local Address:Port      Peer Address:Port
ESTAB       0      0           127.0.0.1:35466        127.0.0.1:10198    rto:5.5e-08
[...]

$ ss -i
State       Recv-Q Send-Q  Local Address:Port      Peer Address:Port
ESTAB       0      0           127.0.0.1:35466        127.0.0.1:10198
        bic wscale:7,7 rto:220 rtt:23.5/19 send 11.2Mbps rcv_space:32792

$ ip route show
10.95.131.111 via 10.95.128.1 dev eth0  rtt 5ms

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.95.131.111   10.95.128.1     255.255.255.255 UGH       0 0     5000000 eth0

I think this is because get_hz gets the HZ value from the fourth field in /proc/net/psched. That's 1e9 in my case, because that's meant to be the frequency of CLOCK_MONOTONIC and when CONFIG_HIGH_RES_TIMERS is on, the CLOCK_MONOTONIC resolution is 1ns. So get_hz returns 1,000,000,000 instead of 250 in my case.

I don't know how to get the right value, clock_getres return 1ns as well.

Quoted 5 lines

That's with 2.6.24.2, iproute-20080108 from debian (but the GIT head code looks the same).

Best regards,
Stéphane

Several places in iproute utils confuse psched and user hz values. I am looking into it.

There is an even bigger mess up. The API for route metrics has several values encoded in jiffies. This is a problem because there is no good way to find the internal kernel value of HZ. So all kernel/user ABI's are supposed to use an absolute value (like milliseconds) or clock_t which user USER_HZ.

The problem is that these values are now hardcoded into people's systems so anyone using the 'ip route' options: rttvar, rtomin, or rtt are broken. They might be lucky now (but I doubt it).

I propose doing the right thing and fixing kernel and iproute to always use milliseconds for these values. To maintain compatibility, the new metric values will be renumbered. So old kernels don't misinterpret the new values.

The problem is that these values are now hardcoded into people's systems so anyone using the 'ip route' options: rttvar, rtomin, or rtt are broken. They might be lucky now (but I doubt it).

On Wed, May 21, 2008 at 11:43:54AM -0700, Stephen Hemminger wrote:
[...]

Quoted 3 lines

The problem is that these values are now hardcoded into people's systems so anyone using the 'ip route' options: rttvar, rtomin, or rtt are broken. They might be lucky now (but I doubt it).

[...]

Hi Stephen, all

a slightly related question:

it seems that the "rtt" parameter provided in "ip route ... rtt <value>" is not taken into account for the retransmission of SYNs while it is for the retransmissions of SYN+ACKs, why would that be (2.6.24.2)?

Also, it seems we can't lower the initial RTO below the RFC 1122 default of 3 seconds. 3 seconds may be appropriate for a host for which we don't know how many hops, links, satellites are needed to reach it, but what about local/corporate networks where it's possible to administratively know the rtt so that it can be hardcoded in the routing table.

For instance, on the office wireless network, I know the average rtt is below the ms. Some SYNs may be lost, but they can't be delayed more than a few hundred ms. So, I may want to specify in the route to that network, the initial and maximum rto, so that a down host can be detected in less than a second.

The delay before the first retransmission is 3 seconds at the moment. That value is often more than what some applications are ready to wait for (applications that are meant to be run locally for instance). So, it's a shame, because the application will timeout on the connect even before the first retransmission, so the SYN retransmission mechanism is useless in that case.

Or is it because there's a risk of congesting the internet if people misuse that? Note that applications can always reattempt a connect to work around that (for SYNs to be sent more often).

It would be nice if what the "rtt" exactly is could be clarified. For instance, if I understand correctly, by default, the initial rtt is 0 and the rttvar 3s, which results in a rto of 3s. That "rtt" is the smoothed rtt, right? (I think the "route" man page from net-tools is incorrect about that, BTW.), but then when setting those variables per route, it's the RTT that can't be made lower than 3s, while rttvar can be as low as rto_min (200ms by default). It's all very confusing (well, I'm very confused ;).

regards,
Stéphane

Quoted 6 lines

Also, it seems we can't lower the initial RTO below the RFC 1122 default of 3 seconds. 3 seconds may be appropriate for a host for which we don't know how many hops, links, satellites are needed to reach it, but what about local/corporate networks where it's possible to administratively know the rtt so that it can be hardcoded in the routing table.

Violating RFC's is not really that useful.

If you have a network dropping SYN packets regularly than there are worse problems.

Relying on TCP to overcome wireless network problems is not a good idea.