Skip to contentSkip to conversation

i386 spinlock fairness: bizarre test results

DISCUSSION Robert Hancock 3 participants 3 messages Oct 11, 2005 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 7 lines · Chuck Ebbert

After seeing Kirill's message about spinlocks I decided to do my own
testing with the userspace program below; the results were very strange.

When using the 'mov' instruction to do the unlock I was able to reproduce
hogging of the spinlock by a single CPU even on Pentium II under some conditions, while using 'xchg' always allowed the other CPU to get the lock:

This might not necessarily be a win in all situations. If two CPUs A and
B are trying to get into a spinlock-protected critical section to do 5
operations, it may well be more efficient for them to do AAAAABBBBB as opposed to ABABABABAB, as the second situation may result in cache lines bouncing between the two CPUs each time, etc.

I don't know that making spinlocks "fairer" is really very worthwhile. If some spinlocks are so heavily contented that fairness becomes an issue, it would be better to find a way to reduce that contention.

Signature · 3 lines
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
#2 lore raw
Quoted 12 lines · Robert Hancock
Quoted 8 lines · Chuck Ebbert

After seeing Kirill's message about spinlocks I decided to do my own
testing with the userspace program below; the results were very strange.

When using the 'mov' instruction to do the unlock I was able to
reproduce
hogging of the spinlock by a single CPU even on Pentium II under some conditions, while using 'xchg' always allowed the other CPU to get the lock:

Quoted 6 lines

This might not necessarily be a win in all situations. If two CPUs A and
B are trying to get into a spinlock-protected critical section to do 5
operations, it may well be more efficient for them to do AAAAABBBBB as opposed to ABABABABAB, as the second situation may result in cache lines bouncing between the two CPUs each time, etc.

I don't know that making spinlocks "fairer" is really very worthwhile. If some spinlocks are so heavily contented that fairness becomes an issue, it would be better to find a way to reduce that contention.

You're right that it wouldn't be an issue on a system with relatively few cpu's since that amount of contention would cripple the system. Though with 100's of cpu's you could get contention hotspots with some spin locks being concurrently accessed by some subset of the cpu's for periods of time.

The real issue is scalability or how gracefully does a system degrade when it starts to hit its contention limits. It's not a good thing when a system appears to run fine and then catastrophically hangs when it bumps across its critical limit. It's better when a system exhibit's some sort of linear degradation. The former exhibits bistable behavior which requires a drastic, probably impossible, reduction in work load to regain normal performance. Reboots are the normal course of correction. The linearly degrading systems just require moderation of the workload to move back into acceptable performance.

Anyway, if you want to build a scalable system, it makes sense to build it out of scalable components. Right?

Joe Seigh

#3 lore raw
Quoted 35 lines · Joe Seigh
Quoted 13 lines · Robert Hancock
Quoted 8 lines · Chuck Ebbert

After seeing Kirill's message about spinlocks I decided to do my own
testing with the userspace program below; the results were very strange.

When using the 'mov' instruction to do the unlock I was able to
reproduce
hogging of the spinlock by a single CPU even on Pentium II under some conditions, while using 'xchg' always allowed the other CPU to get the lock:

Quoted 4 lines

This might not necessarily be a win in all situations. If two CPUs A and B are trying to get into a spinlock-protected critical section to do 5 operations, it may well be more efficient for them to do AAAAABBBBB as opposed to ABABABABAB, as the second situation may result in cache lines bouncing between the two CPUs each time, etc.

I don't know that making spinlocks "fairer" is really very worthwhile. If some spinlocks are so heavily contented that fairness becomes an issue, it would be better to find a way to reduce that contention.

You're right that it wouldn't be an issue on a system with relatively few cpu's since that amount of contention would cripple the system. Though with 100's of cpu's you could get contention hotspots with some spin locks being concurrently accessed by some subset of the cpu's for periods of time.

The real issue is scalability or how gracefully does a system degrade when it starts to hit its contention limits. It's not a good thing when a system appears to run fine and then catastrophically hangs when it bumps across its critical limit. It's better when a system exhibit's some sort of linear degradation. The former exhibits bistable behavior which requires a drastic, probably impossible, reduction in work load to regain normal performance. Reboots are the normal course of correction. The linearly degrading systems just require moderation of the workload to move back into acceptable performance.

Quoted 3 lines

Anyway, if you want to build a scalable system, it makes sense to build it out of scalable components. Right?

Joe, I totally agree. That type of non-linear performance is sometimes called a "jackpot condition," and you see it in various algorithms. The comment that depending on fairness is poor design is correct, but because loads may be vastly higher than the original program design, it's sometimes not obvious that there could be high contention.

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me