Problem with percpu values when bringing up second CPU?
Hi,
I just tracked down a bug I was having to a change where I changed one of my Xen event channel variables to a percpu variable, relating to masking an event channel.
The symptom was that shortly after bringing up the second CPU, the first CPU's timer events stopped arriving, apparently because they had become masked.
The event channels masks are declared as:
#define NR_EVENT_CHANNEL_LONGS (NR_EVENT_CHANNELS/BITS_PER_LONG) static DEFINE_PER_CPU(unsigned long,
cpu_evtchn_mask[NR_EVENT_CHANNEL_LONGS]) =
{[0 ... NR_EVENT_CHANNEL_LONGS-1] = ~0ul }; /* everything masked by default */My theory about what's happening is that when the second CPU comes up, it allocates separate percpu areas for each CPU, but it is somehow failing to accurately copy CPU 0's percpu data over; either it isn't copying it all (ie, using the initialized values rather than the current values), or failing to copy the values in an interrupt-atomic way.
Does this sound plausible?
When I convert this back to an ad-hoc percpu variable (an array indexed by cpu number), it goes back to working. Also, if I boot with maxcpus=1 it also works with percpu data.
Also, because we don't have large pages under Xen, it always allocates percpu as 4k pages:
PERCPU: Allocated 21 4k pages, static data 82080 bytes
Thanks,
J
Quoted 14 lines · Jeremy Fitzhardinge
Hi,
I just tracked down a bug I was having to a change where I changed one of my Xen event channel variables to a percpu variable, relating to masking an event channel.
The symptom was that shortly after bringing up the second CPU, the first CPU's timer events stopped arriving, apparently because they had become masked.
The event channels masks are declared as:
#define
NR_EVENT_CHANNEL_LONGS(NR_EVENT_CHANNELS/BITS_PER_LONG) static DEFINE_PER_CPU(unsigned long,
Quoted 7 lines
cpu_evtchn_mask[NR_EVENT_CHANNEL_LONGS]) = {[0 ... NR_EVENT_CHANNEL_LONGS-1] = ~0ul }; /* everything masked by default */My theory about what's happening is that when the second CPU comes up, it allocates separate percpu areas for each CPU, but it is somehow failing to accurately copy CPU 0's percpu data over
If you touch the per-cpu vars before setup_per_cpu_areas(), you will hit the master copy.
Is that possible?
Rusty.
Quoted 25 lines · Rusty Russell
Quoted 14 lines · Jeremy Fitzhardinge
Hi,
I just tracked down a bug I was having to a change where I changed one of my Xen event channel variables to a percpu variable, relating to masking an event channel.
The symptom was that shortly after bringing up the second CPU, the first CPU's timer events stopped arriving, apparently because they had become masked.
The event channels masks are declared as:
#define
NR_EVENT_CHANNEL_LONGS(NR_EVENT_CHANNELS/BITS_PER_LONG) static DEFINE_PER_CPU(unsigned long,Quoted 8 lines
cpu_evtchn_mask[NR_EVENT_CHANNEL_LONGS]) = {[0 ... NR_EVENT_CHANNEL_LONGS-1] = ~0ul }; /* everything masked by default */My theory about what's happening is that when the second CPU comes up, it allocates separate percpu areas for each CPU, but it is somehow failing to accurately copy CPU 0's percpu data over
Quoted 5 lines
If you touch the per-cpu vars before
setup_per_cpu_areas(), you will hit the master copy.Is that possible?
Likely, I think. It depends on whether interrupts can happen before
that point. But hitting the master copy should be OK.
However, the problem I'm seeing happens when the second CPU starts. I was working on the assumption that that's when the transfer from master to allocated copy happens, but it looks like I'm mistaken.
Maybe I'm barking up the wrong tree, but the problem does bisect to a simple conversion to percpu...
J
Hello,
Quoted 7 lines · Jeremy Fitzhardinge
I just tracked down a bug I was having to a change where I changed one of my Xen event channel variables to a percpu variable, relating to masking an event channel.
The symptom was that shortly after bringing up the second CPU, the first CPU's timer events stopped arriving, apparently because they had become masked.
Hmmmm...
Quoted 7 lines
The event channels masks are declared as:
#define
NR_EVENT_CHANNEL_LONGS(NR_EVENT_CHANNELS/BITS_PER_LONG) static DEFINE_PER_CPU(unsigned long,cpu_evtchn_mask[NR_EVENT_CHANNEL_LONGS]) = {[0 ... NR_EVENT_CHANNEL_LONGS-1] = ~0ul }; /* everything masked by default */
Quoted 7 lines
My theory about what's happening is that when the second CPU comes up, it allocates separate percpu areas for each CPU, but it is somehow failing to accurately copy CPU 0's percpu data over; either it isn't copying it all (ie, using the initialized values rather than the current values), or failing to copy the values in an interrupt-atomic way.
Does this sound plausible?
Percpu areas aren't setup when the first cpu comes up. They're allocated and copied from the master copy during early init when only the boot cpu is running.
Quoted 3 lines
When I convert this back to an ad-hoc percpu variable (an array indexed by cpu number), it goes back to working. Also, if I boot with maxcpus=1 it also works with percpu data.
Hmmm... strange. Can you try to print out the values along the boot process and see when things go wrong?
Quoted 4 lines
Also, because we don't have large pages under Xen, it always allocates percpu as 4k pages:
PERCPU: Allocated 21 4k pages, static data 82080 bytes
I don't think the choice of first chunk allocator would cause any difference.
Thanks.
Signature · 1 line
tejun