Improving accuracy of PHC readings

I think there might be a way how we could significantly improve accuracy of synchronization between the system clock and a PTP hardware clock, at least with some network drivers.

Currently, the PTP_SYS_OFFSET ioctl reads the system clock, reads the PHC using the gettime64 function of the driver, and reads the system clock again. The ioctl can repeat this to provide multiple readings to the user space.

phc2sys (or another program synchronizing the system clock to the PHC) assumes the PHC timestamps were captured in the middle between the two closest system clock timestamps.

The trouble is that gettime64 typically reads multiple (2-3) registers and the timestamp is latched on the first one, so the assumption about middle point is wrong. There is an asymmetry, even if the delays on the PCIe bus are perfectly symmetric.

A solution to this would be a new driver function that wraps the latching register read with readings of the system clock and return three timestamps instead of one. For example:

        ktime_get_real_ts64(&sys_ts1);
	IXGBE_READ_REG(hw, IXGBE_SYSTIMR);
	ktime_get_real_ts64(&sys_ts2);
	phc_ts.tv_nsec = IXGBE_READ_REG(hw, IXGBE_SYSTIML);
	phc_ts.tv_sec = IXGBE_READ_REG(hw, IXGBE_SYSTIMH);

The extra timestamp doesn't fit the API of the PTP_SYS_OFFSET ioctl, so it would need to shift the timestamp it returns by the missing intervals (assuming the frequency offset between the PHC and system clock is small), or a new ioctl could be introduced that would return all timestamps in an array looking like this:

	[sys, phc, sys, sys, phc, sys, ...]

This should significantly improve the accuracy of the synchronization, reduce the uncertainty in the readings to less than a half or third, and also reduce the jitter as there are fewer register reads sensitive to the PCIe delay.

What do you think?

Miroslav Lichvar

Right! I feel like this is obvious now that you said it, so I'm surprised no one thought of it before...

I think the new ioctl is probably the better solution.

Nice! I think this is good. I'd love to see some data to back it up, but it makes sense to me.

Thanks,
Jake

+1

I tried a quick hack with an X550 and I219. The delay dropped from about 2940 ns to 1040 ns on the first port of the X550, from 1920 ns to 660 ns on the second port of the X550, and from 2500 ns to 1300 ns on the I219.

The I219 supports the SYS_OFFSET_PRECISE ioctl (cross timestamping), which we can use for comparison. The difference between the offsets calculated using the two ioctls was about 500-600 ns before and now it is about -50--150 ns.

I was not able to find any information on how accurate cross timestamping on this HW is actually supposed to be, so I'm wondering which of the two is closer to the truth.

Here is an output from phc2sys with the I219:

Before:

phc offset       -59 s2 freq     +40 delay   2527
phc offset        19 s2 freq    +101 delay   2526
phc offset       -23 s2 freq     +64 delay   2522
phc offset        46 s2 freq    +126 delay   2535
phc offset       -32 s2 freq     +62 delay   2530
phc offset       -10 s2 freq     +75 delay   2526
phc offset       102 s2 freq    +184 delay   2523

After:

phc offset        17 s2 freq    +105 delay   1298
phc offset        47 s2 freq    +140 delay   1299
phc offset       -42 s2 freq     +65 delay   1293
phc offset        -6 s2 freq     +88 delay   1299
phc offset        34 s2 freq    +127 delay   1300
phc offset       -14 s2 freq     +89 delay   1301
phc offset       -86 s2 freq     +13 delay   1296
phc offset       -21 s2 freq     +52 delay   1298

Miroslav Lichvar

Makes sense...

How about a new ioctl with number of trials as input and single offset as output?

Then it would be up to the driver to implement the device-specific loop.

Thanks,
Richard

The difference between the system timestamps is important as it gives an upper bound on the error in the offset, so I think the output should be at least a pair of offset and delay.

The question is from which triplet should be the offset and delay calculated. The one with the minimum delay is a good choice, but it's not the only option. For instance, an average or median from all triplets that have delay smaller than the minimum + 30 nanoseconds may give a more stable offset.

This is not that different from an NTP client filtering measurements made over network. I'm not sure if we should try to solve it in the kernel or drivers. My preference would be to give the user space all the data and process it there.

If the increased size of the array is an issue, we can reduce the maximum number of readings.

Does that make sense?

Miroslav Lichvar