Skip to contentSkip to conversation

NMI watchdog logic on X86-64

DISCUSSION Stephane Eranian 1 participant 2 messages Dec 13, 2005 – Dec 19, 2005 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Hello,

In the context of the perfmon2 interface, I am trying to understand the NMI watchdog logic on X86-64 (I think this is identical on IA-32).

I am looking at the case where the NMI watchdog uses the Local APIC and the performance counters (AMD or P4). The code of reference is in 2.6.15.rc5-git1 arch/x86_64/kernel/nmi.c.

It is not clear to me what is going in the case of an SMP machine.

I would like to use reserve_lapic_nmi() and release_lapic_nmi() from perfmon2 to start/stop the NMI watchdog while we have active monitoring (i.e., get the performance counters back). The perfctr subsystem is coded similarly.

The call reserve_lapic_nmi() invokes enable_lapic_nmi_watchdog() which ends up in setup_apic_nmi_watchdog(). That function is executed only on the calling processor. In SMP it means one CPU does the NMI watchdog.

In the case of the LAPIC APIC, this means that the call to release_lapic_nmi() MUST be done on the same CPU as the one used for reserve_lapic_nmi() otherwise the NMI watchdog setup for the performance counters will not be quiesced.

I would like to make sure that the logic of the code is that ONLY one processor does the NMI watchdog timer?

In that case, I think it would be good to document or enforce the fact that when Local APIC is used, calls to reserve and release must be made on the same processor.

Am I missing something here?

Signature · 1 line
-Stephane

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

Eric,

Quoted 5 lines · Eric W. Biederman
Quoted 2 lines
Quoted 2 lines

I would like to make sure that the logic of the code is that ONLY one processor does the NMI watchdog timer?

That much I answer. The logic is that each processor does the NMI watchdog.

I checked again and there is something that looks broken to me. I agree with you that the NMI watchdog must be per-CPU. In fact it is setup per local APIC in smpboot.c. Yet what is confusing is that the owner variable lapic_nmi_owner is shared with spinlock. So you either setup NMI on all CPUs or none.

That also implies that the reserve_lapic_nmi() function must be called only once. If called for each CPU, it will fail for the second caller because it will see the lapic_nmi_owner already set to RESERVED. But if called only once, then the the disable logic in reserve_lapic_nmi() is wrong:

int reserve_lapic_nmi(void)

{
        unsigned int old_owner;

        spin_lock(&lapic_nmi_owner_lock);
        old_owner = lapic_nmi_owner;
        lapic_nmi_owner |= LAPIC_NMI_RESERVED;
        spin_unlock(&lapic_nmi_owner_lock);
        if (old_owner & LAPIC_NMI_RESERVED)
                return -EBUSY;
        if (old_owner & LAPIC_NMI_WATCHDOG)
                disable_lapic_nmi_watchdog();
        return 0;
}

If reserve_lapic_nmi() is called only once, then it will call disable_lapic_nmi_watchdog() on ONLY one CPU even though it was setup on all CPUs in smpboot.c. The function disable_lapic_nmi_watchdog() should be called on all CPUs (once nmi_active, nmi_watchdog are moved out of it).

It appears to me that the lapic_nmi_owner variable somehow needs to become a per-cpu variable. Then the reserve_lapic_nmi() would have to be called on CPUs as necessary.

Signature · 1 line
-Stephane