Rework alarm timer infrastrucure - take 2

plain text document attachment (clock-hpet)
Linux operates the HPET timer in legacy replacement mode, which means that the periodic interrupt of the CMOS RTC is not delivered (qemu won't be able to use /dev/rtc). Add support for HPET (/dev/hpet) as a replacement for the RTC; the periodic interrupt is delivered via SIGIO and is handled in the same way as the RTC timer.

Signed-off-by Luca Tettamanti <[email protected]>

Quoted 7 lines · Luca Tettamanti

plain text document attachment (clock-hpet)
Linux operates the HPET timer in legacy replacement mode, which means that the periodic interrupt of the CMOS RTC is not delivered (qemu won't be able to use /dev/rtc). Add support for HPET (/dev/hpet) as a replacement for the RTC; the periodic interrupt is delivered via SIGIO and is handled in the same way as the RTC timer.

Quoted 1 line

Signed-off-by Luca Tettamanti <[email protected]>

I must be missing something silly here.. should I be able to open more than one instance of qemu with -clock hpet? Because upon invoking a second instance of qemu HPET_IE_ON fails.

Quoted 8 lines · Matthew Kent

Quoted 7 lines · Luca Tettamanti

plain text document attachment (clock-hpet)
Linux operates the HPET timer in legacy replacement mode, which means that the periodic interrupt of the CMOS RTC is not delivered (qemu won't be able to use /dev/rtc). Add support for HPET (/dev/hpet) as a replacement for the RTC; the periodic interrupt is delivered via SIGIO and is handled in the same way as the RTC timer.

Quoted 5 lines

Quoted 1 line

Signed-off-by Luca Tettamanti <[email protected]>

I must be missing something silly here.. should I be able to open more than one instance of qemu with -clock hpet? Because upon invoking a second instance of qemu HPET_IE_ON fails.

It depends on your hardware. Theoretically it's possible, but I've yet to see a motherboard with more than one periodic timer.

Quoted 15 lines · Luca

Quoted 13 lines · Matthew Kent

Quoted 8 lines · Luca Tettamanti

plain text document attachment (clock-hpet)
Linux operates the HPET timer in legacy replacement mode, which means that the periodic interrupt of the CMOS RTC is not delivered (qemu won't be able to use /dev/rtc). Add support for HPET (/dev/hpet) as a replacement for the RTC; the periodic interrupt is delivered via SIGIO and is handled in the same way as the RTC timer.

Signed-off-by Luca Tettamanti <[email protected]>

I must be missing something silly here.. should I be able to open more than one instance of qemu with -clock hpet? Because upon invoking a second instance of qemu HPET_IE_ON fails.

Quoted 2 lines

It depends on your hardware. Theoretically it's possible, but I've yet to see a motherboard with more than one periodic timer.

Ah thank you, after re-reading the docs I think I better understand this.

Quoted 19 lines · Matthew Kent

Quoted 17 lines · Luca

Quoted 15 lines · Matthew Kent

Quoted 9 lines · Luca Tettamanti

plain text document attachment (clock-hpet)
Linux operates the HPET timer in legacy replacement mode, which means that the periodic interrupt of the CMOS RTC is not delivered (qemu won't be able to use /dev/rtc). Add support for HPET (/dev/hpet) as a replacement for the RTC; the periodic interrupt is delivered via SIGIO and is handled in the same way as the RTC timer.

Signed-off-by Luca Tettamanti <[email protected]>

I must be missing something silly here.. should I be able to open more than one instance of qemu with -clock hpet? Because upon invoking a second instance of qemu HPET_IE_ON fails.

Quoted 6 lines

Quoted 3 lines

It depends on your hardware. Theoretically it's possible, but I've yet to see a motherboard with more than one periodic timer.

Ah thank you, after re-reading the docs I think I better understand this.

In a risk of being off-topic, maybe you can help me try the hpet support. When I try the hpet Documentation demo I get

# ./hpet poll /dev/hpet 1 1000
-hpet: executing poll
hpet_poll: info.hi_flags 0x0
hpet_poll, HPET_IE_ON failed

while I have

$ dmesg|grep -i HPET
ACPI: HPET 7D5B6AE0, 0038 (r1 A M I  OEMHPET   5000708 MSFT       97)
ACPI: HPET id: 0x8086a301 base: 0xfed00000
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 64-bit timers, 14318180 Hz
hpet_resources: 0xfed00000 is busy
Time: hpet clocksource has been installed.

Any idea what I am misconfiguring?

Maybe the kernel is using the timer, so userspace can't. Just a guess.

Quoted 1 line · Avi Kivity

Maybe the kernel is using the timer, so userspace can't. Just a guess.

HPET has multiple timers (variable, but typically 2 or 4). The kernel only uses timer 0. It's possible someone else in user space is using it though. Try lsof /dev/hpet

$ dmesg |grep -i hpet
ACPI: HPET 7D5B6AE0, 0038 (r1 A M I  OEMHPET   5000708 MSFT       97)
ACPI: HPET id: 0x8086a301 base: 0xfed00000
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 64-bit timers, 14318180 Hz
hpet_resources: 0xfed00000 is busy

Quoted 6 lines

$ dmesg |grep -i hpet
ACPI: HPET 7D5B6AE0, 0038 (r1 A M I  OEMHPET   5000708 MSFT       97)
ACPI: HPET id: 0x8086a301 base: 0xfed00000
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 64-bit timers, 14318180 Hz
hpet_resources: 0xfed00000 is busy

What kernel version was that? There was a bug that caused this pre .22

Quoted 7 lines · Andi Kleen

Quoted 6 lines

$ dmesg |grep -i hpet
ACPI: HPET 7D5B6AE0, 0038 (r1 A M I  OEMHPET   5000708 MSFT       97)
ACPI: HPET id: 0x8086a301 base: 0xfed00000
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 64-bit timers, 14318180 Hz
hpet_resources: 0xfed00000 is busy

Quoted 2 lines

What kernel version was that? There was a bug that caused this pre .22

I have vanilla 2.6.22.3 on that machine.

Quoted 11 lines · Dan Kenigsberg

Quoted 9 lines · Andi Kleen

Quoted 6 lines

$ dmesg |grep -i hpet
ACPI: HPET 7D5B6AE0, 0038 (r1 A M I  OEMHPET   5000708 MSFT       97)
ACPI: HPET id: 0x8086a301 base: 0xfed00000
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 64-bit timers, 14318180 Hz
hpet_resources: 0xfed00000 is busy

What kernel version was that? There was a bug that caused this pre .22

Quoted 1 line

I have vanilla 2.6.22.3 on that machine.

Try:
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

do you see HPET listed twice?

Quoted 8 lines · Matthew Kent

Quoted 7 lines · Luca Tettamanti

plain text document attachment (clock-hpet)
Linux operates the HPET timer in legacy replacement mode, which means that the periodic interrupt of the CMOS RTC is not delivered (qemu won't be able to use /dev/rtc). Add support for HPET (/dev/hpet) as a replacement for the RTC; the periodic interrupt is delivered via SIGIO and is handled in the same way as the RTC timer.

Quoted 5 lines

Quoted 1 line

Signed-off-by Luca Tettamanti <[email protected]>

I must be missing something silly here.. should I be able to open more than one instance of qemu with -clock hpet? Because upon invoking a second instance of qemu HPET_IE_ON fails.

It depends on your hardware. Theoretically it's possible, but I've yet to see a motherboard with more than one periodic timer.

"dmesg | grep hpet" should tell you something like:

hpet0: 3 64-bit timers, 14318180 Hz

Luca

Hello,
in reply to this mail I will send a serie of 4 patches that cleans up and expands the alarm timer handling in QEMU. Patches have been rebased on QEMU CVS.

Patch 1 is mostly a cleanup of the existing code; instead of having multiple #ifdefs to handle different timers scattered all over the code I've created a modular infrastructure where each timer type is self-contained and generic code is more readable. The resulting code is functionally equivalent to the old one.

Patch 2 implements the "-clock" command line option proposed by Daniel Berrange and Avi Kivity. By default QEMU tries RTC and then falls back to unix timer; user can override the order of the timer through this options. Syntax is pretty
simple: -clock timer1,timer2,etc. (QEMU will pick the first one that works).

Patch 3 adds support for HPET under Linux (which is basically my old patch). As suggested HPET takes precedence over other timers, but of course this can be overridden.

Patch 4 introduces "dynticks" timer source; patch is mostly based on the work Dan Kenigsberg. dynticks is now the default alarm timer.

Luca
--

there's a typo line 1432 on vl.c after applying all 4 patches (missing ';')

beyond this small typo, I managed to include this in a win32 qemu build. is there a specific practical test to see the improvement in a linux guest when running on a windows host ?

Hello,
in reply to this mail I will send a serie of 4 patches that cleans up and expands the alarm timer handling in QEMU. Patches have been rebased on QEMU CVS.

Patch 1 is mostly a cleanup of the existing code; instead of having multiple #ifdefs to handle different timers scattered all over the code I've created a modular infrastructure where each timer type is self-contained and generic code is more readable. The resulting code is functionally equivalent to the old one.

Patch 2 implements the "-clock" command line option proposed by Daniel Berrange and Avi Kivity. By default QEMU tries RTC and then falls back to unix timer; user can override the order of the timer through this options. Syntax is pretty
simple: -clock timer1,timer2,etc. (QEMU will pick the first one that works).

Patch 3 adds support for HPET under Linux (which is basically my old patch). As suggested HPET takes precedence over other timers, but of course this can be overridden.

Patch 4 introduces "dynticks" timer source; patch is mostly based on the work Dan Kenigsberg. dynticks is now the default alarm timer.

Luca

Quoted 2 lines

Hello,
in reply to this mail I will send a serie of 4 patches that cleans up

and

Quoted 1 line

expands the alarm timer handling in QEMU. Patches have been rebased

QEMU

Quoted 3 lines

CVS.

Patch 1 is mostly a cleanup of the existing code; instead of having

multiple

Quoted 1 line

#ifdefs to handle different timers scattered all over the code I've

created a

Quoted 1 line

modular infrastructure where each timer type is self-contained and

generic code

Quoted 1 line

is more readable. The resulting code is functionally equivalent to

old one.

Quoted 2 lines

Patch 2 implements the "-clock" command line option proposed by

Berrange

Quoted 1 line

and Avi Kivity. By default QEMU tries RTC and then falls back to unix

timer;

Quoted 1 line

user can override the order of the timer through this options. Syntax

is pretty

Quoted 1 line

simple: -clock timer1,timer2,etc. (QEMU will pick the first one that

works).

Quoted 2 lines

Patch 3 adds support for HPET under Linux (which is basically my old

patch). As

Quoted 1 line

suggested HPET takes precedence over other timers, but of course this

can be

Quoted 3 lines

overridden.

Patch 4 introduces "dynticks" timer source; patch is mostly based on

the work

Quoted 1 line

Dan Kenigsberg. dynticks is now the default alarm timer.

Why do you guard dynticks with #ifdef? Is there any reason why you wouldn't want to use dynticks?

Regards,

Anthony Liguori

Quoted 1 line

Luca

-----------------------------------------------------------------------

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Quoted 2 lines

Hello,
in reply to this mail I will send a serie of 4 patches that cleans up

and

Quoted 1 line

expands the alarm timer handling in QEMU. Patches have been rebased

QEMU

Quoted 3 lines

CVS.

Patch 1 is mostly a cleanup of the existing code; instead of having

multiple

Quoted 1 line

#ifdefs to handle different timers scattered all over the code I've

created a

Quoted 1 line

modular infrastructure where each timer type is self-contained and

generic code

Quoted 1 line

is more readable. The resulting code is functionally equivalent to

old one.

Quoted 2 lines

Patch 2 implements the "-clock" command line option proposed by

Berrange

Quoted 1 line

and Avi Kivity. By default QEMU tries RTC and then falls back to unix

timer;

Quoted 1 line

user can override the order of the timer through this options. Syntax

is pretty

Quoted 1 line

simple: -clock timer1,timer2,etc. (QEMU will pick the first one that

works).

Quoted 2 lines

Patch 3 adds support for HPET under Linux (which is basically my old

patch). As

Quoted 1 line

suggested HPET takes precedence over other timers, but of course this

can be

Quoted 3 lines

overridden.

Patch 4 introduces "dynticks" timer source; patch is mostly based on

the work

Quoted 1 line

Dan Kenigsberg. dynticks is now the default alarm timer.

Why do you guard dynticks with #ifdef? Is there any reason why you wouldn't want to use dynticks?

Regards,

Anthony Liguori

Quoted 1 line

Luca

-----------------------------------------------------------------------

-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
kvm-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Quoted 4 lines

Quoted 2 lines

Hello,
in reply to this mail I will send a serie of 4 patches that cleans up

and

Quoted 1 line

expands the alarm timer handling in QEMU. Patches have been rebased

on

Quoted 10 lines

QEMU

Quoted 3 lines

CVS.

Patch 1 is mostly a cleanup of the existing code; instead of having

multiple

Quoted 1 line

#ifdefs to handle different timers scattered all over the code I've

created a

Quoted 1 line

modular infrastructure where each timer type is self-contained and

generic code

Quoted 1 line

is more readable. The resulting code is functionally equivalent to

the

Quoted 3 lines

old one.

Quoted 2 lines

Patch 2 implements the "-clock" command line option proposed by

Daniel

Quoted 15 lines

Berrange

Quoted 1 line

and Avi Kivity. By default QEMU tries RTC and then falls back to unix

timer;

Quoted 1 line

user can override the order of the timer through this options. Syntax

is pretty

Quoted 1 line

simple: -clock timer1,timer2,etc. (QEMU will pick the first one that

works).

Quoted 2 lines

Patch 3 adds support for HPET under Linux (which is basically my old

patch). As

Quoted 1 line

suggested HPET takes precedence over other timers, but of course this

can be

Quoted 3 lines

overridden.

Patch 4 introduces "dynticks" timer source; patch is mostly based on

Quoted 5 lines

the work

Quoted 1 line

Dan Kenigsberg. dynticks is now the default alarm timer.

Why do you guard dynticks with #ifdef? Is there any reason why you wouldn't want to use dynticks?

I think too that it's should be the default.
There is no regression in performance nor behaviour with this option.

We didn't test qemu dyn-tick with kernels that don't have high-res-timer+dyn-tick.

In this case the dyn-tick minimum res will be 1msec. I believe it should work ok since this is the case without any dyn-tick.

Quoted 2 lines

In this case the dyn-tick minimum res will be 1msec. I believe it should work ok since this is the case without any dyn-tick.

Actually minimum resolution depends on host HZ setting, but - yes - essentially you have the same behaviour of the "unix" timer, plus the overhead of reprogramming the timer.

Add support for dynamic ticks.

If the the dynticks alarm timer is used qemu does not attempt to generate SIGALRM at a constant rate. Rather, the system timer is set to generate SIGALRM only when it is needed. Dynticks timer reduces the number of SIGALRMs sent to idle dynamic-ticked guests.
Original patch from Dan Kenigsberg <dank-atKUWr5tajBWk0Htik3J/[email protected]>

Signed-off-by Luca Tettamanti <[email protected]>

---
 vl.c |  178 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 170 insertions(+), 8 deletions(-)

Index: qemu/vl.c
===================================================================

vl.c +2 -0

@@ -784,12 +784,31 @@
 
 struct qemu_alarm_timer {
     char const *name;
+    unsigned int flags;
 
     int (*start)(struct qemu_alarm_timer *t);
     void (*stop)(struct qemu_alarm_timer *t);
+    void (*rearm)(struct qemu_alarm_timer *t);
     void *priv;

vl.c +160 -8

+    if (!alarm_has_dynticks(t))
+        return;
+
+    t->rearm(t);
+}
+
+/* TODO: MIN_TIMER_REARM_US should be optimized */
+#define MIN_TIMER_REARM_US 250
+
 static struct qemu_alarm_timer *alarm_timer;
 
 #ifdef _WIN32

@@ -802,12 +821,17 @@
 
 static int win32_start_timer(struct qemu_alarm_timer *t);
 static void win32_stop_timer(struct qemu_alarm_timer *t);
+static void win32_rearm_timer(struct qemu_alarm_timer *t);
 
 #else
 
 static int unix_start_timer(struct qemu_alarm_timer *t);
 static void unix_stop_timer(struct qemu_alarm_timer *t);
 
+static int dynticks_start_timer(struct qemu_alarm_timer *t);
+static void dynticks_stop_timer(struct qemu_alarm_timer *t);
+static void dynticks_rearm_timer(struct qemu_alarm_timer *t);
+
 #ifdef __linux__
 
 static int hpet_start_timer(struct qemu_alarm_timer *t);

@@ -816,21 +840,23 @@
 static int rtc_start_timer(struct qemu_alarm_timer *t);
 static void rtc_stop_timer(struct qemu_alarm_timer *t);
 
-#endif
+#endif /* __linux__ */
 
 #endif /* _WIN32 */
 
 static struct qemu_alarm_timer alarm_timers[] = {
+#ifndef _WIN32
+    {"dynticks", ALARM_FLAG_DYNTICKS, dynticks_start_timer, dynticks_stop_timer, dynticks_rearm_timer, NULL},
 #ifdef __linux__
     /* HPET - if available - is preferred */
-    {"hpet", hpet_start_timer, hpet_stop_timer, NULL},
+    {"hpet", 0, hpet_start_timer, hpet_stop_timer, NULL, NULL},
     /* ...otherwise try RTC */
-    {"rtc", rtc_start_timer, rtc_stop_timer, NULL},
+    {"rtc", 0, rtc_start_timer, rtc_stop_timer, NULL, NULL},
 #endif
-#ifndef _WIN32
-    {"unix", unix_start_timer, unix_stop_timer, NULL},
+    {"unix", 0, unix_start_timer, unix_stop_timer, NULL, NULL},
 #else
-    {"win32", win32_start_timer, win32_stop_timer, &alarm_win32_data},
+    {"dynticks", ALARM_FLAG_DYNTICKS, win32_start_timer, win32_stop_timer, win32_rearm_timer, &alarm_win32_data},
+    {"win32", 0, win32_start_timer, win32_stop_timer, NULL, &alarm_win32_data},
 #endif
     {NULL, }
 };

@@ -949,6 +975,8 @@
         }
         pt = &t->next;
     }
+
+    qemu_rearm_alarm_timer(alarm_timer);
 }
 
 /* modify the current timer so that it will be fired when current_time

@@ -1008,6 +1036,7 @@
         /* run the callback (the timer list can be modified) */
         ts->cb(ts->opaque);
     }
+    qemu_rearm_alarm_timer(alarm_timer);
 }
 
 int64_t qemu_get_clock(QEMUClock *clock)

@@ -1115,7 +1144,8 @@
         last_clock = ti;
     }
 #endif
-    if (qemu_timer_expired(active_timers[QEMU_TIMER_VIRTUAL],
+    if (alarm_has_dynticks(alarm_timer) ||
+        qemu_timer_expired(active_timers[QEMU_TIMER_VIRTUAL],
                            qemu_get_clock(vm_clock)) ||
         qemu_timer_expired(active_timers[QEMU_TIMER_REALTIME],
                            qemu_get_clock(rt_clock))) {

@@ -1136,6 +1166,27 @@
     }
 }
 
+static uint64_t qemu_next_deadline(void) {
+    uint64_t nearest_delta_us = ULLONG_MAX;
+    uint64_t vmdelta_us;
+
+    if (active_timers[QEMU_TIMER_REALTIME])
+        nearest_delta_us = (active_timers[QEMU_TIMER_REALTIME]->expire_time - qemu_get_clock(rt_clock))*1000;
+
+    if (active_timers[QEMU_TIMER_VIRTUAL]) {
+        /* round up */
+        vmdelta_us = (active_timers[QEMU_TIMER_VIRTUAL]->expire_time - qemu_get_clock(vm_clock)+999)/1000;
+        if (vmdelta_us < nearest_delta_us)
+            nearest_delta_us = vmdelta_us;
+    }
+
+    /* Avoid arming the timer to negative, zero, or too low values */
+    if (nearest_delta_us <= MIN_TIMER_REARM_US)
+        nearest_delta_us = MIN_TIMER_REARM_US;
+
+    return nearest_delta_us;
+}
+
 #ifndef _WIN32
 
 #if defined(__linux__)

@@ -1243,6 +1294,80 @@
 
 #endif /* !defined(__linux__) */
 
+static int dynticks_start_timer(struct qemu_alarm_timer *t)
+{
+    struct sigevent ev;
+    timer_t host_timer;
+    struct sigaction act;
+
+    sigfillset(&act.sa_mask);
+    act.sa_flags = 0;
+#if defined(TARGET_I386) && defined(USE_CODE_COPY)
+    act.sa_flags |= SA_ONSTACK;
+#endif
+    act.sa_handler = host_alarm_handler;
+
+    sigaction(SIGALRM, &act, NULL);
+
+    ev.sigev_value.sival_int = 0;
+    ev.sigev_notify = SIGEV_SIGNAL;
+    ev.sigev_signo = SIGALRM;
+
+    if (timer_create(CLOCK_REALTIME, &ev, &host_timer)) {
+        perror("timer_create");
+
+        /* disable dynticks */
+        fprintf(stderr, "Dynamic Ticks disabled\n");
+
+        return -1;
+    }
+
+    t->priv = (void *)host_timer;
+
+    return 0;
+}
+
+static void dynticks_stop_timer(struct qemu_alarm_timer *t)
+{
+    timer_t host_timer = (timer_t)t->priv;
+
+    timer_delete(host_timer);
+}
+
+static void dynticks_rearm_timer(struct qemu_alarm_timer *t)
+{
+    timer_t host_timer = (timer_t)t->priv;
+    struct itimerspec timeout;
+    int64_t nearest_delta_us = INT64_MAX;
+    int64_t current_us;
+
+    if (!active_timers[QEMU_TIMER_REALTIME] &&
+                !active_timers[QEMU_TIMER_VIRTUAL])
+            return;
+
+    nearest_delta_us = qemu_next_deadline();
+
+    /* check whether a timer is already running */
+    if (timer_gettime(host_timer, &timeout)) {
+        perror("gettime");
+        fprintf(stderr, "Internal timer error: aborting\n");
+        exit(1);
+    }
+    current_us = timeout.it_value.tv_sec * 1000000 + timeout.it_value.tv_nsec/1000;
+    if (current_us && current_us <= nearest_delta_us)
+        return;
+
+    timeout.it_interval.tv_sec = 0;
+    timeout.it_interval.tv_nsec = 0; /* 0 for one-shot timer */
+    timeout.it_value.tv_sec =  nearest_delta_us / 1000000;
+    timeout.it_value.tv_nsec = (nearest_delta_us % 1000000) * 1000;
+    if (timer_settime(host_timer, 0 /* RELATIVE */, &timeout, NULL)) {
+        perror("settime");
+        fprintf(stderr, "Internal timer error: aborting\n");
+        exit(1);
+    }
+}
+
 static int unix_start_timer(struct qemu_alarm_timer *t)
 {
     struct sigaction act;

@@ -1288,6 +1413,7 @@
 {
     TIMECAPS tc;
     struct qemu_alarm_win32 *data = t->priv;
+    UINT flags;
 
     data->host_alarm = CreateEvent(NULL, FALSE, FALSE, NULL);
     if (!data->host_alarm) {

@@ -1304,11 +1430,17 @@
 
     timeBeginPeriod(data->period);
 
+    flags = TIME_CALLBACK_FUNCTION;
+    if (alarm_has_dynticks(t))
+        flags |= TIME_ONESHOT;
+    else
+        flags |= TIME_PERIODIC;
+
     data->timerId = timeSetEvent(1,         // interval (ms)
                         data->period,       // resolution
                         host_alarm_handler, // function
                         (DWORD)t,           // parameter
-                        TIME_PERIODIC | TIME_CALLBACK_FUNCTION);
+                        flags);
 
     if (!data->timerId) {
         perror("Failed to initialize win32 alarm timer");

@@ -1333,6 +1465,35 @@
     CloseHandle(data->host_alarm);
 }
 
+static void win32_rearm_timer(struct qemu_alarm_timer *t)
+{
+    struct qemu_alarm_win32 *data = t->priv;
+    uint64_t nearest_delta_us;
+
+    if (!active_timers[QEMU_TIMER_REALTIME] &&
+                !active_timers[QEMU_TIMER_VIRTUAL])
+            return;
+
+    nearest_delta_us = qemu_next_deadline();
+    nearest_delta_us /= 1000;
+
+    timeKillEvent(data->timerId);
+
+    data->timerId = timeSetEvent(1,
+                        data->period,
+                        host_alarm_handler,
+                        (DWORD)t,
+                        TIME_ONESHOT | TIME_PERIODIC);
+
+    if (!data->timerId) {
+        perror("Failed to re-arm win32 alarm timer");
+
+        timeEndPeriod(data->period);
+        CloseHandle(data->host_alarm);
+        exit(1);
+    }
+}
+
 #endif /* _WIN32 */
 
 static void init_timer_alarm(void)

@@ -6491,6 +6652,7 @@
         cpu_enable_ticks();
         vm_running = 1;
         vm_state_notify(1);
+        qemu_rearm_alarm_timer(alarm_timer);
     }
 }
 

Luca

I think this is a really nice and important patch set. Just a couple things:

Quoted 1 line · Luca Tettamanti

Quoted 1 line

In this case the dyn-tick minimum res will be 1msec. I believe it

should

Quoted 5 lines

Quoted 1 line

work ok since this is the case without any dyn-tick.

Actually minimum resolution depends on host HZ setting, but - yes - essentially you have the same behaviour of the "unix" timer, plus the overhead of reprogramming the timer.

Is this significant? At a high guest HZ, this is could be quite a lot of additional syscalls right?

I think this is a really nice and important patch set. Just a couple things:

Quoted 3 lines · Luca Tettamanti

Quoted 3 lines

In this case the dyn-tick minimum res will be 1msec. I believe it should work ok since this is the case without any dyn-tick.

Quoted 4 lines

Actually minimum resolution depends on host HZ setting, but - yes - essentially you have the same behaviour of the "unix" timer, plus the overhead of reprogramming the timer.

Is this significant? At a high guest HZ, this is could be quite a lot of additional syscalls right?

Quoted 8 lines

Quoted 7 lines

Quoted 3 lines

In this case the dyn-tick minimum res will be 1msec. I believe it should work ok since this is the case without any dyn-tick.

Actually minimum resolution depends on host HZ setting, but - yes - essentially you have the same behaviour of the "unix" timer, plus the overhead of reprogramming the timer.

Quoted 3 lines

Is this significant? At a high guest HZ, this is could be quite a lot of additional syscalls right?

At HZ=1000, this adds a small multiple of 1000 syscalls, which is a fairly small overhead.

Quoted 9 lines · Avi Kivity

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

In this case the dyn-tick minimum res will be 1msec. I believe it should work ok since this is the case without any dyn-tick.

Actually minimum resolution depends on host HZ setting, but - yes - essentially you have the same behaviour of the "unix" timer, plus the overhead of reprogramming the timer.

Quoted 6 lines

Quoted 4 lines

Is this significant? At a high guest HZ, this is could be quite a lot of additional syscalls right?

At HZ=1000, this adds a small multiple of 1000 syscalls, which is a fairly small overhead.

Small, but maybe measurable.

That overhead could be removed if the dyn-tick code were to predictively set the host timer into a repeating mode when guests do actually require a regular tick.

I'm thinking when it detects it needed a tick a small number of times in a row, with the same interval, it could set the host timer to trigger repeatedly at that interval. Then it wouldn't need to reprogram if that stayed the same (except maybe to correct for drift?)

If a tick then _wasn't_ required for that interval (i.e. it was required for less, more, or not at all), then it would have to reprogram the host timer. If it really mattered, it wouldn't have to reprogram the host timer when the next required tick is further in the future or not at all; it would simply be a redundant SIGALRM. In weird cases that's worthwhile, but I suspect it generally isn't.

Yes, good thinking, but this should only be done if it actually impacts something. Reducing overhead from 0.1% to 0.05% is not worthwhile if it introduces extra complexity.

Quoted 4 lines

Yes, good thinking, but this should only be done if it actually impacts something. Reducing overhead from 0.1% to 0.05% is not worthwhile if it introduces extra complexity.

If the overhead is that small, why are we touching this code in the first place?

Quoted 1 line

Quoted 1 line

Yes, good thinking, but this should only be done if it actually

impacts

Quoted 1 line

Quoted 1 line

something. Reducing overhead from 0.1% to 0.05% is not worthwhile

it

Quoted 4 lines

Quoted 2 lines

introduces extra complexity.

Quoted 1 line

If the overhead is that small, why are we touching this code in the

first

Quoted 2 lines

place?

Accuracy is much more important from my point of view. Also, the reduction in the number of signals delivered when the guest uses 100Hz is significant.

Quoted 6 lines · Paul Brook

Quoted 4 lines

Yes, good thinking, but this should only be done if it actually impacts something. Reducing overhead from 0.1% to 0.05% is not worthwhile if it introduces extra complexity.

Quoted 2 lines

If the overhead is that small, why are we touching this code in the first place?

Accuracy is much more important from my point of view. Also, the reduction in the number of signals delivered when the guest uses 100Hz is significant.

Quoted 3 lines

Yes, good thinking, but this should only be done if it actually impacts something. Reducing overhead from 0.1% to 0.05% is not worthwhile if it introduces extra complexity.

If the overhead is that small, why are we touching this code in the first place?

[mkent@localhost ~]$ time ./hackbench 50
x86_64 host : real 0m10.845s
x86_64 host, bound to 1 cpu : real 0m21.884s

i386 guest+unix clock       : real 0m49.206s
i386 guest+hpet clock       : real 0m48.292s
i386 guest+dynticks clock   : real 0m28.835s

Results are repeatable and verfied with a stopwatch because I didn't believe them at first :)

Quoted 1 line · Paul Brook

Quoted 1 line

Yes, good thinking, but this should only be done if it actually

impacts

Quoted 1 line

Quoted 1 line

something. Reducing overhead from 0.1% to 0.05% is not worthwhile

if it

Quoted 3 lines

Quoted 1 line

introduces extra complexity.

If the overhead is that small, why are we touching this code in the

first

Quoted 1 line

place?

Insightful.

A benchmark result was posted which is rather interesting:

Quoted 7 lines

[mkent@localhost ~]$ time ./hackbench 50
x86_64 host : real 0m10.845s
x86_64 host, bound to 1 cpu : real 0m21.884s

i386 guest+unix clock       : real 0m49.206s
i386 guest+hpet clock       : real 0m48.292s
i386 guest+dynticks clock   : real 0m28.835s

Quoted 2 lines

Results are repeatable and verfied with a stopwatch because I didn't believe them at first :)

I am surprised if 1000 redundant SIGALRMs per second is really causing 70% overhead in normal qemu execution, except on a rather old or slow machine where signal delivery is very slow.

It would be good to understand the cause of that benchmark result.

Quoted 7 lines · Paul Brook

Quoted 4 lines

Yes, good thinking, but this should only be done if it actually impacts something. Reducing overhead from 0.1% to 0.05% is not worthwhile if it introduces extra complexity.

If the overhead is that small, why are we touching this code in the first place?

Insightful.

A benchmark result was posted which is rather interesting:

Quoted 7 lines

[mkent@localhost ~]$ time ./hackbench 50
x86_64 host : real 0m10.845s
x86_64 host, bound to 1 cpu : real 0m21.884s

i386 guest+unix clock       : real 0m49.206s
i386 guest+hpet clock       : real 0m48.292s
i386 guest+dynticks clock   : real 0m28.835s

Quoted 3 lines

Results are repeatable and verfied with a stopwatch because I didn't believe them at first :)

I am surprised if 1000 redundant SIGALRMs per second is really causing 70% overhead in normal qemu execution, except on a rather old or slow machine where signal delivery is very slow.

vl.c line 1171

Quoted 4 lines · Luca Tettamanti

vl.c +3 -0

+static uint64_t qemu_next_deadline(void) {
+    uint64_t nearest_delta_us = ULLONG_MAX;
+    uint64_t vmdelta_us;

Hum, I introduced a bug here... those vars should be signed.

On the overhead introduced: how do you measure it?

Quoted 7 lines · Luca

Quoted 4 lines · Luca Tettamanti

vl.c +3 -0

+static uint64_t qemu_next_deadline(void) {
+    uint64_t nearest_delta_us = ULLONG_MAX;
+    uint64_t vmdelta_us;

Quoted 5 lines

Hum, I introduced a bug here... those vars should be signed.

On the overhead introduced: how do you measure it?

Run a 100Hz guest, measure cpu usage using something accurate like cyclesoak, with and without dynticks, with and without kvm.

Il Sun, Aug 19, 2007 at 10:31:26PM +0300, Avi Kivity ha scritto:

Quoted 14 lines

Quoted 12 lines · Luca

Review

Quoted 4 lines · Luca Tettamanti

(quoted diff) +3 -0

+static uint64_t qemu_next_deadline(void) {
+    uint64_t nearest_delta_us = ULLONG_MAX;
+    uint64_t vmdelta_us;

Hum, I introduced a bug here... those vars should be signed.

On the overhead introduced: how do you measure it?

Quoted 2 lines

Run a 100Hz guest, measure cpu usage using something accurate like cyclesoak, with and without dynticks, with and without kvm.

Ok, here I've measured the CPU usage on the host when running an idle guest.

Quoted 21 lines · Luca Tettamanti

Il Sun, Aug 19, 2007 at 10:31:26PM +0300, Avi Kivity ha scritto:

Quoted 8 lines

Quoted 7 lines · Luca

Quoted 4 lines · Luca Tettamanti

(quoted diff) +3 -0

+static uint64_t qemu_next_deadline(void) {
+    uint64_t nearest_delta_us = ULLONG_MAX;
+    uint64_t vmdelta_us;

Quoted 8 lines

Quoted 5 lines

Hum, I introduced a bug here... those vars should be signed.

On the overhead introduced: how do you measure it?

Run a 100Hz guest, measure cpu usage using something accurate like cyclesoak, with and without dynticks, with and without kvm.

Ok, here I've measured the CPU usage on the host when running an idle guest.

The upshot is this - if you have used any standard utility (iostat, top - basically anything /proc/stat based) the accounting has a fair chance of being inaccurate. If cyclesoak is what you have used then the results should be better, but still i would be worried about them.

Quoted 5 lines

Quoted 2 lines

Run a 100Hz guest, measure cpu usage using something accurate like cyclesoak, with and without dynticks, with and without kvm.

Ok, here I've measured the CPU usage on the host when running an idle guest.

At 100Hz

QEMU
hpet 4.8%
dynticks 5.1%

Note: I've taken the mean over a period of 20 secs, but the difference between hpet and dynticks is well inside the variability of the test.

KVM
hpet 2.2%
dynticks 1.0%

Hum... here the numbers jumps a bit, but dynticks is always below hpet.

At 1000Hz:

QEMU
hpet 5.5%
dynticks 11.7%

KVM
hpet 3.4%
dynticks 7.3%

No surprises here, you can see the additional 1k syscalls per second.

Quoted 10 lines · Luca Tettamanti

At 1000Hz:

QEMU
hpet 5.5%
dynticks 11.7%

KVM
hpet 3.4%
dynticks 7.3%

Quoted 1 line

No surprises here, you can see the additional 1k syscalls per second.

This is very surprising to me. The 6.2% difference for the qemu case translates to 62ms per second, or 62us per tick at 1000Hz. That's more than a hundred simple syscalls on modern processors. We shouldn't have to issue a hundred syscalls per guest clock tick.

The difference with kvm is smaller (just 3.9%), which is not easily explained as the time for the extra syscalls should be about the same. My guess is that guest behavior is different; with dynticks the guest does about twice as much work as with hpet.

Can you verify this by running

strace -c -p pgrep qemu & sleep 10; pkill strace

for all 4 cases, and posting the results?

Avi Kivity ha scritto:

Quoted 17 lines

Quoted 11 lines · Luca Tettamanti

At 1000Hz:

QEMU
hpet 5.5%
dynticks 11.7%

KVM
hpet 3.4%
dynticks 7.3%

No surprises here, you can see the additional 1k syscalls per second.

This is very surprising to me. The 6.2% difference for the qemu case translates to 62ms per second, or 62us per tick at 1000Hz. That's more than a hundred simple syscalls on modern processors. We shouldn't have to issue a hundred syscalls per guest clock tick.

I've also tried APC which was suggested by malc[1] and:
- readings are far more stable
- the gap between dynticks and non-dynticks seems not significant

Luca
[1] copy_to_user inside spinlock is a big no-no ;)

Actually I'm having troubles with cyclesoak (probably it's calibration), numbers are not very stable across multiple runs...

I've also tried APC which was suggested by malc[1] and:
- readings are far more stable
- the gap between dynticks and non-dynticks seems not significant

Quoted 6 lines

Can you verify this by running

strace -c -p pgrep qemu & sleep 10; pkill strace

for all 4 cases, and posting the results?

Plain QEMU:

With dynticks:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 57.97    0.000469           0     13795           clock_gettime
 32.88    0.000266           0      1350           gettimeofday
  7.42    0.000060           0      1423      1072 sigreturn
  1.73    0.000014           0      5049           timer_gettime
  0.00    0.000000           0      1683      1072 select
  0.00    0.000000           0      2978           timer_settime

------ ----------- ----------- --------- --------- ----------------
100.00 0.000809 26278 2144 total

HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 87.48    0.010459           1     10381     10050 select
  8.45    0.001010           0     40736           clock_gettime
  2.73    0.000326           0     10049           gettimeofday
  1.35    0.000161           0     10086     10064 sigreturn

------ ----------- ----------- --------- --------- ----------------
100.00 0.011956 71252 20114 total

Unix (SIGALRM):

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 90.36    0.011663           1     10291      9959 select
  7.38    0.000953           0     40355           clock_gettime
  2.05    0.000264           0      9960           gettimeofday
  0.21    0.000027           0      9985      9969 sigreturn

------ ----------- ----------- --------- --------- ----------------
100.00 0.012907 70591 19928 total

And KVM:

dynticks:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 78.90    0.004001           1      6681      5088 rt_sigtimedwait
 10.87    0.000551           0     27901           clock_gettime
  4.93    0.000250           0      7622           timer_settime
  4.30    0.000218           0     10078           timer_gettime
  0.39    0.000020           0      3863           gettimeofday
  0.35    0.000018           0      6054           ioctl
  0.26    0.000013           0      4196           select
  0.00    0.000000           0      1593           rt_sigaction

------ ----------- ----------- --------- --------- ----------------
100.00 0.005071 67988 5088 total

HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 90.20    0.011029           0     32437     22244 rt_sigtimedwait
  4.46    0.000545           0     44164           clock_gettime
  2.59    0.000317           0     12128           gettimeofday
  1.50    0.000184           0     10193           rt_sigaction
  1.10    0.000134           0     12461           select
  0.15    0.000018           0      6060           ioctl

------ ----------- ----------- --------- --------- ----------------
100.00 0.012227 117443 22244 total

Unix:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 83.29    0.012522           0     31652     21709 rt_sigtimedwait
  6.91    0.001039           0     43125           clock_gettime
  3.50    0.000526           0      6042           ioctl
  2.74    0.000412           0      9943           rt_sigaction
  1.98    0.000298           0     12183           select
  1.58    0.000238           0     11850           gettimeofday

------ ----------- ----------- --------- --------- ----------------
100.00 0.015035 114795 21709 total

The guest is an idle kernel with HZ=1000.

Quoted 3 lines · Luca Tettamanti

Actually I'm having troubles with cyclesoak (probably it's calibration), numbers are not very stable across multiple runs...

I've had good results with cyclesoak; maybe you need to run it in runlevel 3 so the load generated by moving the mouse or breathing doesn't affect meaurements.

Quoted 2 lines

The guest is an idle kernel with HZ=1000.

Can you double check this? The dyntick results show that this is either a 100Hz kernel, or that there is a serious bug in dynticks.

Il Wed, Aug 22, 2007 at 08:02:07AM +0300, Avi Kivity ha scritto:

Quoted 7 lines

Quoted 4 lines · Luca Tettamanti

Actually I'm having troubles with cyclesoak (probably it's calibration), numbers are not very stable across multiple runs...

Quoted 4 lines

I've had good results with cyclesoak; maybe you need to run it in runlevel 3 so the load generated by moving the mouse or breathing doesn't affect meaurements.

This is what I did, I tested with -no-grapich in text console.

Quoted 6 lines

Quoted 3 lines

The guest is an idle kernel with HZ=1000.

Can you double check this? The dyntick results show that this is either a 100Hz kernel, or that there is a serious bug in dynticks.

Ops I sent the wrong files, sorry.

This is QEMU, with dynticks and HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 52.10    0.002966           0     96840           clock_gettime
 19.50    0.001110           0     37050           timer_gettime
 10.66    0.000607           0     20086           timer_settime
 10.40    0.000592           0      8985      2539 sigreturn
  4.94    0.000281           0      8361      2485 select
  2.41    0.000137           0      8362           gettimeofday

------ ----------- ----------- --------- --------- ----------------
100.00 0.005693 179684 5024 total

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 93.37    0.025541           3     10194     10193 select
  4.82    0.001319           0     33259           clock_gettime
  1.10    0.000301           0     10195           gettimeofday
  0.71    0.000195           0     10196     10194 sigreturn

------ ----------- ----------- --------- --------- ----------------
100.00 0.027356 63844 20387 total

And this KVM:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 42.66    0.002885           0     45527        24 ioctl
 25.62    0.001733           0     89305           clock_gettime
 13.12    0.000887           0     34894           timer_gettime
  7.97    0.000539           0     18016           timer_settime
  4.70    0.000318           0     12224      7270 rt_sigtimedwait
  2.79    0.000189           0      7271           select
  1.86    0.000126           0      7271           gettimeofday
  1.27    0.000086           0      4954           rt_sigaction

------ ----------- ----------- --------- --------- ----------------
100.00 0.006763 219462 7294 total

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

 49.41    0.004606           0     59900        27 ioctl
 24.14    0.002250           0     31252     21082 rt_sigtimedwait
  9.65    0.000900           0     51856           clock_gettime
  8.44    0.000787           0     17819           select
  4.42    0.000412           0     17819           gettimeofday
  3.94    0.000367           0     10170           rt_sigaction

------ ----------- ----------- --------- --------- ----------------
100.00 0.009322 188816 21109 total

Quoted 9 lines · Luca Tettamanti

Il Wed, Aug 22, 2007 at 08:02:07AM +0300, Avi Kivity ha scritto:

Quoted 7 lines

Quoted 4 lines · Luca Tettamanti

Actually I'm having troubles with cyclesoak (probably it's calibration), numbers are not very stable across multiple runs...

Quoted 6 lines

Quoted 4 lines

I've had good results with cyclesoak; maybe you need to run it in runlevel 3 so the load generated by moving the mouse or breathing doesn't affect meaurements.

This is what I did, I tested with -no-grapich in text console.

Okay. Maybe cpu frequency scaling confused it then. Or something else?

Quoted 13 lines

Quoted 6 lines

Quoted 3 lines

The guest is an idle kernel with HZ=1000.

Can you double check this? The dyntick results show that this is either a 100Hz kernel, or that there is a serious bug in dynticks.

Ops I sent the wrong files, sorry.

This is QEMU, with dynticks and HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

Quoted 9 lines

 52.10    0.002966           0     96840           clock_gettime
 19.50    0.001110           0     37050           timer_gettime
 10.66    0.000607           0     20086           timer_settime
 10.40    0.000592           0      8985      2539 sigreturn
  4.94    0.000281           0      8361      2485 select
  2.41    0.000137           0      8362           gettimeofday

------ ----------- ----------- --------- --------- ----------------
100.00 0.005693 179684 5024 total

This looks like 250 Hz?

And a huge number of settime calls?

Quoted 14 lines

Quoted 4 lines

This is QEMU, with dynticks and HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

Quoted 10 lines

 52.10    0.002966           0     96840           clock_gettime
 19.50    0.001110           0     37050           timer_gettime
 10.66    0.000607           0     20086           timer_settime
 10.40    0.000592           0      8985      2539 sigreturn
  4.94    0.000281           0      8361      2485 select
  2.41    0.000137           0      8362           gettimeofday

------ ----------- ----------- --------- --------- ----------------
100.00 0.005693 179684 5024 total

Quoted 2 lines

This looks like 250 Hz?

Nope:

# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

and I'm reading it from /proc/config.gz on the guest.

Quoted 26 lines · Luca

Quoted 14 lines

Quoted 4 lines

This is QEMU, with dynticks and HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

Quoted 10 lines

 52.10    0.002966           0     96840           clock_gettime
 19.50    0.001110           0     37050           timer_gettime
 10.66    0.000607           0     20086           timer_settime
 10.40    0.000592           0      8985      2539 sigreturn
  4.94    0.000281           0      8361      2485 select
  2.41    0.000137           0      8362           gettimeofday

------ ----------- ----------- --------- --------- ----------------
100.00 0.005693 179684 5024 total

Quoted 2 lines

This looks like 250 Hz?

Nope:

# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

Quoted 2 lines

and I'm reading it from /proc/config.gz on the guest.

Yeah, thought so -- so dyntick is broken at present.

Or maybe your host kernel can't support such a high rate.

I see a lot of sub ms timer_settime(). Many of them are the result of ->expire_time being less than the current qemu_get_clock().

Quoted 27 lines · Avi Kivity

Quoted 26 lines · Luca

Quoted 14 lines

Quoted 4 lines

This is QEMU, with dynticks and HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

Quoted 10 lines

 52.10    0.002966           0     96840           clock_gettime
 19.50    0.001110           0     37050           timer_gettime
 10.66    0.000607           0     20086           timer_settime
 10.40    0.000592           0      8985      2539 sigreturn
  4.94    0.000281           0      8361      2485 select
  2.41    0.000137           0      8362           gettimeofday

------ ----------- ----------- --------- --------- ----------------
100.00 0.005693 179684 5024 total

Quoted 2 lines

This looks like 250 Hz?

Nope:

# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

Quoted 4 lines

Quoted 2 lines

and I'm reading it from /proc/config.gz on the guest.

Yeah, thought so -- so dyntick is broken at present.

I see a lot of sub ms timer_settime(). Many of them are the result of ->expire_time being less than the current qemu_get_clock(). This results into 250us timer due to MIN_TIMER_REARM_US; this happens only for the REALTIME timer. Other sub-ms timers are generated by the VIRTUAL timer.

This first issue is easily fixed; if expire_time < current time then the timer has expired and hasn't been reprogrammed (and thus can be ignored).
VIRTUAL just becomes more accurate with dyntics, before multiple timers were batched together.

Quoted 1 line

Or maybe your host kernel can't support such a high rate.

I don't know... a simple printf tells me that the signal handler is called about 1050 times per second, which sounds about right.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

This is QEMU, with dynticks and HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- ---------

---

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

52.10 0.002966 0 96840

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

19.50 0.001110 0 37050

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

10.66 0.000607 0 20086

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

 10.40    0.000592           0      8985      2539 sigreturn
  4.94    0.000281           0      8361      2485 select
  2.41    0.000137           0      8362           gettimeofday

------ ----------- ----------- --------- ---------

---

Quoted 34 lines

Quoted 15 lines

Quoted 15 lines

Quoted 5 lines

Quoted 3 lines

100.00 0.005693 179684 5024 total

This looks like 250 Hz?

Nope:

# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

Quoted 4 lines

Quoted 2 lines

and I'm reading it from /proc/config.gz on the guest.

Yeah, thought so -- so dyntick is broken at present.

I see a lot of sub ms timer_settime(). Many of them are the result of ->expire_time being less than the current qemu_get_clock(). This results into 250us timer due to MIN_TIMER_REARM_US; this happens only for the REALTIME timer. Other sub-ms timers are generated by the VIRTUAL timer.

This first issue is easily fixed; if expire_time < current time then the timer has expired and hasn't been reprogrammed (and thus can be ignored).
VIRTUAL just becomes more accurate with dyntics, before multiple timers were batched together.

Quoted 1 line

Or maybe your host kernel can't support such a high rate.

Quoted 2 lines

I don't know... a simple printf tells me that the signal handler is called about 1050 times per second, which sounds about right.

...unless strace is attached. ptrace()'ing the process really screw up the timing with dynticks; HPET is also affected but the performance hit is not as severe.

Luca

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

This is QEMU, with dynticks and HPET:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- ---------

-------------

Quoted 2 lines

---

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

52.10 0.002966 0 96840

clock_gettime

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

19.50 0.001110 0 37050

timer_gettime

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

10.66 0.000607 0 20086

timer_settime

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

 10.40    0.000592           0      8985      2539 sigreturn
  4.94    0.000281           0      8361      2485 select
  2.41    0.000137           0      8362           gettimeofday

------ ----------- ----------- --------- ---------

-------------

Quoted 35 lines

---

Quoted 34 lines

Quoted 15 lines

Quoted 15 lines

Quoted 5 lines

Quoted 3 lines

100.00 0.005693 179684 5024 total

This looks like 250 Hz?

Nope:

# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

Quoted 4 lines

Quoted 2 lines

and I'm reading it from /proc/config.gz on the guest.

Yeah, thought so -- so dyntick is broken at present.

I see a lot of sub ms timer_settime(). Many of them are the result of ->expire_time being less than the current qemu_get_clock(). This results into 250us timer due to MIN_TIMER_REARM_US; this happens only for the REALTIME timer. Other sub-ms timers are generated by the VIRTUAL timer.

This first issue is easily fixed; if expire_time < current time then the timer has expired and hasn't been reprogrammed (and thus can be ignored).
VIRTUAL just becomes more accurate with dyntics, before multiple timers were batched together.

Quoted 1 line

Or maybe your host kernel can't support such a high rate.

Quoted 7 lines

Quoted 2 lines

I don't know... a simple printf tells me that the signal handler is called about 1050 times per second, which sounds about right.

...unless strace is attached. ptrace()'ing the process really screw up the timing with dynticks; HPET is also affected but the performance hit is not as severe.

I didn't figure out how you use both hpet and dyn-tick together.

Hpet has periodic timer while dyn-tick is one shot timer each time. Is ther a chance that both are working and that's the source of our problems?

and I'm reading it from /proc/config.gz on the guest.

Quoted 1 line

And a huge number of settime calls?

Yes, maybe some QEMU timer is using an interval < 1ms? Dan do you any any idea of what's going on?