Skip to contentSkip to conversation

Drivers: hv: util: Implement Time Synchronization using host time sample

PATCH Olaf Hering 4 participants 8 messages Sep 23, 2014 – Sep 26, 2014 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 6 lines · Thomas Shao

In current hyper-v time sync service,it only gets the initial clock time from the host. It didn't process the following time samples. This change introduced a module parameter called host_time_sync. If it is set to true, the guest will periodically sychronize it's time with the host clock using host time sample. By default it is disabled, because we still recommend user to configure NTP for time synchronization.

Is that new knob a temporary thing to aid testing and development? Because once it sneaks in it becomes some sort of ABIT.
So if the outcome of all that is that 'sync with host every once in a while' will be the default, the new knob is not required.

Olaf

#2 lore raw
Quoted 20 lines

-----Original Message-----
From: Olaf Hering [mailto:[email protected]]
Sent: Tuesday, September 23, 2014 3:56 PM
To: Thomas Shao
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; KY Srinivasan; Haiyang Zhang
Subject: Re: [PATCH] Drivers: hv: util: Implement Time Synchronization using host time sample

Quoted 6 lines · Thomas Shao

In current hyper-v time sync service,it only gets the initial clock time from the host. It didn't process the following time samples. This change introduced a module parameter called host_time_sync. If it is set to true, the guest will periodically sychronize it's time with the host clock using host time sample. By default it is disabled, because we still recommend user to configure NTP for time synchronization.

Quoted 5 lines

Is that new knob a temporary thing to aid testing and development? Because once it sneaks in it becomes some sort of ABIT.
So if the outcome of all that is that 'sync with host every once in a while' will be the default, the new knob is not required.

Hi Olaf,
If the guest is configured with other time sync source, like NTP, we don't want the guest to periodically sync with host. This feature is useful when the guest didn't have Internet connection to setup NTP. So the new knob is to allow users to only turn on the host-guest sync when they need.

Regards
Thomas

Quoted 1 line

Olaf

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#3 lore raw
In reply to Olaf Hering
Quoted 8 lines · Olaf Hering
Quoted 6 lines · Thomas Shao

In current hyper-v time sync service,it only gets the initial clock time from the host. It didn't process the following time samples. This change introduced a module parameter called host_time_sync. If it is set to true, the guest will periodically sychronize it's time with the host clock using host time sample. By default it is disabled, because we still recommend user to configure NTP for time synchronization.

You [Microsoft?] do? Can you link to public sources where is this stated please? I don't see any mention of doing this on
http://technet.microsoft.com/en-us/library/dn720239.aspx .. The only official Microsoft information with respect to Hyper-V guest time synchronisation I've seen has been for Windows guests but perhaps I've looked in the wrong places.

The reason I ask is because regular ntpd is not enough to discipline a Linux Hyper-V guest's clock. So much drift can occur under load that ntpd can't bring the clock under sync. For now, I've been using Chrony which has a higher tolerance for correcting drifting clocks. I'm not the only one seeing this either (see http://serverfault.com/a/488528/203726
and
http://serverfault.com/questions/523389/linux-clock-loses-10-minutes-every-week
).

It would be good to something official about this issue as it is painful when happens.

Signature · 1 line
Sitsofe | http://sucs.org/~sits/
#4 lore raw
Quoted 40 lines

-----Original Message-----
From: Sitsofe Wheeler [mailto:[email protected]]
Sent: Thursday, September 25, 2014 2:08 PM
To: Olaf Hering
Cc: Thomas Shao; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; KY Srinivasan; Haiyang Zhang
Subject: Re: [PATCH] Drivers: hv: util: Implement Time Synchronization using host time sample

Quoted 8 lines · Olaf Hering
Quoted 6 lines · Thomas Shao

In current hyper-v time sync service,it only gets the initial clock time from the host. It didn't process the following time samples. This change introduced a module parameter called host_time_sync. If it is set to true, the guest will periodically sychronize it's time with the host clock using host time sample. By default it is disabled, because we still recommend user to configure NTP for time

synchronization.

You [Microsoft?] do? Can you link to public sources where is this stated please? I don't see any mention of doing this on
http://technet.microsoft.com/en-us/library/dn720239.aspx .. The only official Microsoft information with respect to Hyper-V guest time synchronisation I've seen has been for Windows guests but perhaps I've looked in the wrong places.

The reason I ask is because regular ntpd is not enough to discipline a Linux Hyper-V guest's clock. So much drift can occur under load that ntpd can't bring the clock under sync. For now, I've been using Chrony which has a higher tolerance for correcting drifting clocks. I'm not the only one seeing this either (see http://serverfault.com/a/488528/203726
and
http://serverfault.com/questions/523389/linux-clock-loses-10-minutes-
every-week
).

Quoted 3 lines

It would be good to something official about this issue as it is painful when happens.

Hi Sitsofe,

As far as I know, currently the document about time sync in Hyper-V is only for Windows. Below is a doc from MSDN:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx
We are working on the document about Time Sync on Hyper-V to cover both Linux and Windows.

Quoted 2 lines

--
Sitsofe | http://sucs.org/~sits/

#5 lore raw

Re: Time keeps on slipping... on Hyper-V

Quoted 9 lines · Thomas Shao
Quoted 9 lines
Quoted 4 lines · Thomas Shao

with the host clock using host time sample. By default it is disabled, because we still recommend user to configure NTP for time

-----Original Message-----
From: Sitsofe Wheeler [mailto:[email protected]]
Sent: Thursday, September 25, 2014 2:08 PM

Quoted 5 lines
Quoted 1 line

You [Microsoft?] do? Can you link to public sources where is this stated

As far as I know, currently the document about time sync in Hyper-V is only for Windows. Below is a doc from MSDN:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx

That post is problematic and not quite enough because it doesn't say "we recommend you always configure an external time source for a guest" but rather gives a more complicated statement saying "because guest time drifts we have a time synchronisation service that will sort things out. If you happen to be running a domain controller we recommend you configure regular synchronisation to an external time source in addition". This leads to questions like: if my (Linux) guest's sole purpose is to run a web server am I in the domain controller situation (domain controllers have to re-serve time so perhaps they are special)? It also doesn't cover the "clock drift is so bad ntpd can't correct it" issue.

Have you seen this document on TechNet -
http://blogs.technet.com/b/enterprise_admin/archive/2010/04/04/linux-on-hyper-v-timesync.aspx
"With the new Integration Services, Linux virtual machines can!

TimeSync means that you don’t have to hassle with configur[ing] network-based time sync for your Linux VMs on Hyper-V…they can get the correct time from the host!"

This document gives simpler advice looks quite compelling at first glance.

Quoted 2 lines

We are working on the document about Time Sync on Hyper-V to cover both Linux and Windows.

The sooner you can get such an official document out the better. All sources of guest time synchronisation information (including Microsoft blogs and non-Microsoft Q&A sites) need updating so they say the same (correct!) thing in a clear way.

For the meantime can you state what you believe the recommend practice is for Linux Hyper-V guests with regards to time synchronisation?

Signature · 1 line
Sitsofe | http://sucs.org/~sits/
#6 lore raw

RE: Time keeps on slipping... on Hyper-V

Quoted 57 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Sitsofe Wheeler
Sent: Thursday, September 25, 2014 9:28 PM
To: Thomas Shao
Cc: Olaf Hering; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; KY Srinivasan; Haiyang Zhang
Subject: Re: Time keeps on slipping... on Hyper-V

Quoted 10 lines · Thomas Shao
Quoted 10 lines
Quoted 5 lines · Thomas Shao

with the host clock using host time sample. By default it is disabled, because we still recommend user to configure NTP for time

-----Original Message-----
From: Sitsofe Wheeler [mailto:[email protected]]
Sent: Thursday, September 25, 2014 2:08 PM

Quoted 7 lines
Quoted 2 lines

You [Microsoft?] do? Can you link to public sources where is this stated

As far as I know, currently the document about time sync in Hyper-V is only for Windows. Below is a doc from MSDN:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchro
nization-in-hyper-v.aspx

That post is problematic and not quite enough because it doesn't say "we recommend you always configure an external time source for a guest" but rather gives a more complicated statement saying "because guest time drifts we have a time synchronisation service that will sort things out.
If you happen to be running a domain controller we recommend you configure regular synchronisation to an external time source in addition". This leads to questions like: if my (Linux) guest's sole purpose is to run a web server am I in the domain controller situation (domain controllers have to re-serve time so perhaps they are special)?
It also doesn't cover the "clock drift is so bad ntpd can't correct it" issue.

Have you seen this document on TechNet -
http://blogs.technet.com/b/enterprise_admin/archive/2010/04/04/linux-on-hy
per-v-timesync.aspx
"With the new Integration Services, Linux virtual machines can!

TimeSync means that you don’t have to hassle with configur[ing] network-based time sync for your Linux VMs on Hyper-V…they can get the correct time from the host!"

This document gives simpler advice looks quite compelling at first glance.

Quoted 2 lines

We are working on the document about Time Sync on Hyper-V to cover both Linux and Windows.

Quoted 8 lines

The sooner you can get such an official document out the better. All sources of guest time synchronisation information (including Microsoft blogs and non-Microsoft Q&A sites) need updating so they say the same (correct!) thing in a clear way.

For the meantime can you state what you believe the recommend practice is for Linux Hyper-V guests with regards to time synchronisation?

We still recommend user to configure NTP in the guest VM. With the new time sync feature in this patch, you could have one more option to enable the guest-host sync, if the NTP didn't work in the environment. For example the guest VM didn't have network connection.

Quoted 7 lines

--
Sitsofe | http://sucs.org/~sits/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#7 lore raw

Re: Time keeps on slipping... on Hyper-V

Quoted 3 lines

We still recommend user to configure NTP in the guest VM. With the new time sync feature in this patch, you could have one more option to enable the guest-host sync, if the NTP didn't work in the environment. For example the guest VM didn't have network connection.

Microsoft used to use host time-samples in their older drivers but this was dropped (when I don't know).
Now we must use NTP to correct hyperv_clocksource which suffers from the usual problems associated with virtual environment cpu loading.

Host time-samples in conjunction with an effective clock stability algorithm with slews rather than steps should be the default. NTP is a workaround and should not be the primary solution.

I have seen a lots of posts for RHEL CENTOS Linux that have a FAST.hyper_clocksource confirmed by myself on CENTOS 6.5 and 7.0 (also confirmed by Olaf on SLES).

If this patch steps the clock once in a while without any form of slewing then it has the potential to break things as files and logs travel BACK in time.

Apologies Thomas could you send me the patch via email as lkml.org does not have it in the archives.

Regards Mike

#8 lore raw

RE: Time keeps on slipping... on Hyper-V

Quoted 29 lines

-----Original Message-----
From: Mike Surcouf [mailto:[email protected]]
Sent: Friday, September 26, 2014 5:24 PM
To: Thomas Shao
Cc: Sitsofe Wheeler; Olaf Hering; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; KY Srinivasan; Haiyang Zhang
Subject: Re: Time keeps on slipping... on Hyper-V

Quoted 2 lines

We still recommend user to configure NTP in the guest VM. With the new time sync feature in this patch, you could have one more option to enable

the guest-host sync, if the NTP didn't work in the environment.

Quoted 1 line

For example the guest VM didn't have network connection.

Microsoft used to use host time-samples in their older drivers but this was dropped (when I don't know).
Now we must use NTP to correct hyperv_clocksource which suffers from the usual problems associated with virtual environment cpu loading.

Host time-samples in conjunction with an effective clock stability algorithm with slews rather than steps should be the default.
NTP is a workaround and should not be the primary solution.

I have seen a lots of posts for RHEL CENTOS Linux that have a FAST.hyper_clocksource confirmed by myself on CENTOS 6.5 and 7.0 (also confirmed by Olaf on SLES).

Quoted 6 lines

If this patch steps the clock once in a while without any form of slewing then it has the potential to break things as files and logs travel BACK in time.

Apologies Thomas could you send me the patch via email as lkml.org does not have it in the archives.

Sure. I'll send the package again. It's strange that this patch didn't in the archives. And in my patch, it will slew the time, if the time drift is less than 1 s. It only steps the clock if the drift is larger than 1 s.

Quoted 1 line

Regards Mike

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�