ipmi_si_intf: use usleep_range() instead of busy looping
PATCH 1/1 ipmi_si_intf: use usleep_range() instead of busy looping
ipmi_thread() uses back-to-back schedule() to poll for command completion which, on some machines, can push up CPU consumption and heavily tax the scheduler locks leading to noticeable overall performance degradation.
This patch replaces schedule() with usleep_range(100, 200). This allows the sensor readings to finish resonably fast and the cpu consumption of the kthread is kept under several percents of a core.
--- drivers/char/ipmi/ipmi_si_intf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/char/ipmi/ipmi_si_intf.c +1 -1
@@ -1010,7 +1010,7 @@ static int ipmi_thread(void *data)if (smi_result ==SI_SM_CALL_WITHOUT_DELAY ) ;/* do nothing */ else if (smi_result ==SI_SM_CALL_WITH_DELAY && busy_wait) -schedule (); +usleep_range (100 ,200 );else if (smi_result ==SI_SM_IDLE ) {if (atomic_read (&smi_info->need_watch)) {schedule_timeout_interruptible (100 );
Quoted 8 lines · Tejun Heo
ipmi_thread()uses back-to-backschedule()to poll for command completion which, on some machines, can push up CPU consumption and heavily tax the scheduler locks leading to noticeable overall performance degradation.This patch replaces
schedule()withusleep_range(100, 200). This allows the sensor readings to finish resonably fast and the cpu consumption of the kthread is kept under several percents of a core.
The IPMI thread was not really designed for sensor reading, it was designed so that firmware updates would happen in a reasonable time on systems without an interrupt on the IPMI interface. This change will degrade performance for that function. IIRC correctly the people who did the patch tried this and it slowed things down too much.
I'm also a little confused because the CPU in question shouldn't be doing anything else if the schedule() immediately returns here, so it's not wasting CPU that could be used on another process. Or is it lock contention that is causing an issue on other CPUs?
IMHO, this whole thing is stupid; if you design hardware with stupid interfaces (byte at a time, no interrupts) you should expect to get bad performance. But I can't control what the hardware vendors do. This whole thing is a carefully tuned compromise.
So I can't really take this as-is.
-corey
Quoted 19 lines
Signed-off-by Tejun Heo <[email protected]>--- drivers/char/ipmi/ipmi_si_intf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/char/ipmi/ipmi_si_intf.c +1
-1@@ -1010,7 +1010,7 @@ static int ipmi_thread(void *data)if (smi_result ==SI_SM_CALL_WITHOUT_DELAY ) ;/* do nothing */ else if (smi_result ==SI_SM_CALL_WITH_DELAY && busy_wait) -schedule (); +usleep_range (100 ,200 );else if (smi_result ==SI_SM_IDLE ) {if (atomic_read (&smi_info->need_watch)) {schedule_timeout_interruptible (100 );
Hello, Corey.
Quoted 4 lines · Corey Minyard
I'm also a little confused because the CPU in question shouldn't be doing anything else if the
schedule()immediately returns here, so it's not wasting CPU that could be used on another process. Or is it lock contention that is causing an issue on other CPUs?
Yeah, pretty pronounced too and it also keeps the CPU busy which makes the load balancer deprioritize that CPU. Busy looping is never free.
Quoted 5 lines
IMHO, this whole thing is stupid; if you design hardware with stupid interfaces (byte at a time, no interrupts) you should expect to get bad performance. But I can't control what the hardware vendors do. This whole thing is a carefully tuned compromise.
I'm really not sure "carefully tuned" is applicable on indefinite busy looping.
Quoted 1 line
So I can't really take this as-is.
We can go for shorter timeouts for sure but I don't think this sort of busy looping is acceptable. Is your position that this must be a busy loop?
Thanks.
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Hello, Corey.
Quoted 4 lines · Corey Minyard
I'm also a little confused because the CPU in question shouldn't be doing anything else if the
schedule()immediately returns here, so it's not wasting CPU that could be used on another process. Or is it lock contention that is causing an issue on other CPUs?Yeah, pretty pronounced too and it also keeps the CPU busy which makes the load balancer deprioritize that CPU. Busy looping is never free.
Quoted 8 lines
Quoted 5 lines
IMHO, this whole thing is stupid; if you design hardware with stupid interfaces (byte at a time, no interrupts) you should expect to get bad performance. But I can't control what the hardware vendors do. This whole thing is a carefully tuned compromise.
I'm really not sure "carefully tuned" is applicable on indefinite busy looping.
Well, yeah, but other things were tried and this was the only thing we could find that worked. That was before the kind of SMP stuff we have now, though.
Quoted 6 lines
Quoted 2 lines
So I can't really take this as-is.
We can go for shorter timeouts for sure but I don't think this sort of busy looping is acceptable. Is your position that this must be a busy loop?
Well, no. I want something that provides as high a throughput as possible and doesn't cause scheduling issues. But that may not be possible. Screwing up the scheduler is a lot worse than slow IPMI firmware updates.
How short can the timeouts be and avoid issues?
Thanks,
-corey
Quoted 5 lines
Thanks.
Signature · 1 line
tejun
Hello,
Quoted 6 lines · Corey Minyard
Quoted 2 lines
I'm really not sure "carefully tuned" is applicable on indefinite busy looping.
Well, yeah, but other things were tried and this was the only thing we could find that worked. That was before the kind of SMP stuff we have now, though.
I see.
Quoted 10 lines
Quoted 3 lines
We can go for shorter timeouts for sure but I don't think this sort of busy looping is acceptable. Is your position that this must be a busy loop?
Well, no. I want something that provides as high a throughput as possible and doesn't cause scheduling issues. But that may not be possible. Screwing up the scheduler is a lot worse than slow IPMI firmware updates.
How short can the timeouts be and avoid issues?
We first tried msleep(1) and that was too slow even for sensor reading making it take longer than 50s. With the 100us-200us sleep, it got down to ~5s which was good enough for our use case and the cpu / scheduler impact was still mostly negligible. I can't tell for sure without testing but going significantly below 100us is likely to become visible pretty quickly.
We can also take a hybrid approach where we busy poll w/ 1us udelay upto, say, fifty times and then switch to sleeping poll.
Are there some tests which can be used to verify the cases which may get impacted by these changes?
Thanks.
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Hello,
Quoted 9 lines
Quoted 3 lines
We can go for shorter timeouts for sure but I don't think this sort of busy looping is acceptable. Is your position that this must be a busy loop?
Well, no. I want something that provides as high a throughput as possible and doesn't cause scheduling issues. But that may not be possible. Screwing up the scheduler is a lot worse than slow IPMI firmware updates.
Quoted 8 lines
Quoted 1 line
How short can the timeouts be and avoid issues?
We first tried msleep(1) and that was too slow even for sensor reading making it take longer than 50s. With the 100us-200us sleep, it got down to ~5s which was good enough for our use case and the cpu / scheduler impact was still mostly negligible. I can't tell for sure without testing but going significantly below 100us is likely to become visible pretty quickly.
What was the time to read the sensors before you did the change? It depends a lot on the system, so I can't really guess.
Quoted 3 lines
We can also take a hybrid approach where we busy poll w/ 1us udelay upto, say, fifty times and then switch to sleeping poll.
I'm pretty sure we didn't try that in the original work, but I'm not sure that would work. Most of the initial spinning would be pointless.
I would guess that you would decrease the delay and the performance would improve linearly until you hit a certain point, and then decreasing the delay wouldn't make a big difference. That's the point you want to use, I think.
What might actually be best would be for the driver to measure the time it takes the BMC to respond and try to set the timeout based on that information. BMCs vary a lot, a constant probably won't work.
And I was just saying that I wasn't expecting any big changes in the IPMI driver any more...
Quoted 3 lines
Are there some tests which can be used to verify the cases which may get impacted by these changes?
Unfortunately not. The original people at Dell that did the work don't work there any more, I don't think.
I mostly use qemu now for testing, but this is not something you can really simulate on qemu very well. Can you do an IPMI firmware update on your system? That would be the easiest way to measure.
Thanks,
-corey
Quoted 5 lines
Thanks.
Signature · 1 line
tejun
Quoted 21 lines · Tejun Heo
Hello,
Quoted 6 lines · Corey Minyard
Quoted 2 lines
I'm really not sure "carefully tuned" is applicable on indefinite busy looping.
Well, yeah, but other things were tried and this was the only thing we could find that worked. That was before the kind of SMP stuff we have now, though.
I see.
Quoted 9 lines
Quoted 3 lines
We can go for shorter timeouts for sure but I don't think this sort of busy looping is acceptable. Is your position that this must be a busy loop?
Well, no. I want something that provides as high a throughput as possible and doesn't cause scheduling issues. But that may not be possible. Screwing up the scheduler is a lot worse than slow IPMI firmware updates.
Quoted 8 lines
Quoted 1 line
How short can the timeouts be and avoid issues?
We first tried msleep(1) and that was too slow even for sensor reading making it take longer than 50s. With the 100us-200us sleep, it got down to ~5s which was good enough for our use case and the cpu / scheduler impact was still mostly negligible. I can't tell for sure without testing but going significantly below 100us is likely to become visible pretty quickly.
I spent some time looking at this. Without the patch, I measured around 3.5ms to send/receive a get device ID message and uses 100% of the CPU on a core.
I measured your patch, it slowed it down to around 10.5ms per message, which is not good. Though it did just use a few percent of the core.
I wrote some code to auto-adjust the timer. It settled on a delay around 35us, which gave 4.7ms per message, which is probably acceptable, and used around 40% of the CPU. If I use any timeout (even a 0-10us range) it won't go below 4ms per message.
The process is running at nice 19 priority, so it won't have a significant effect on other processes from a priority point of view. It may still be hitting the scheduler locks pretty hard, though. But I played with things quite a bit and the behavior or the management controller is too erratic to set a really good timeout. Maybe other ones are better, don't know.
One other option we have is that the driver has something called "maintenance mode". If it detect that you have reset the management controller or are issuing firmware commands, it will modify timeout behavior. It can also be activated manually. I could also make it switch to just calling schedule instead of delaying when in that mode.
The right thing it do is complain bitterly to vendors who build hardware that has to be polled. But besides that, I'm thinking the maintenance mode is the thing to do. It will also change behavior if you reset the management controller, but only for 30 seconds or so. Does that work?
-corey
Quoted 11 lines
We can also take a hybrid approach where we busy poll w/ 1us udelay upto, say, fifty times and then switch to sleeping poll.
Are there some tests which can be used to verify the cases which may get impacted by these changes?
Thanks.
Signature · 1 line
tejun
Hello, Corey.
Quoted 8 lines · Corey Minyard
I spent some time looking at this. Without the patch, I measured around 3.5ms to send/receive a get device ID message and uses 100% of the CPU on a core.
I measured your patch, it slowed it down to around 10.5ms per message, which is not good. Though it did just use a few percent of the core.
Quoted 5 lines
I wrote some code to auto-adjust the timer. It settled on a delay around 35us, which gave 4.7ms per message, which is probably acceptable, and used around 40% of the CPU. If I use any timeout (even a 0-10us range) it won't go below 4ms per message.
Out of curiosity, what makes 4.7ms okay and 10.5ms not? At least for the use case we have in the fleet (sensor reading mostly), the less disruptive the better as long as things don't timeout and fail.
Quoted 4 lines
The process is running at nice 19 priority, so it won't have a significant effect on other processes from a priority point of view. It may still be hitting the scheduler locks pretty hard, though. But I played with things quite a bit
And power. Imagine multi six digit number of machines burning a full core just because of this busy loop to read temperature sensors some msecs faster.
Quoted 11 lines
and the behavior or the management controller is too erratic to set a really good timeout. Maybe other ones are better, don't know.
One other option we have is that the driver has something called "maintenance mode". If it detect that you have reset the management controller or are issuing firmware commands, it will modify timeout behavior. It can also be activated manually. I could also make it switch to just calling schedule instead of delaying when in that mode.
Yeah, whatever which makes the common-case behavior avoid busy looping would work.
Quoted 2 lines
The right thing it do is complain bitterly to vendors who build hardware that has to be polled. But besides that,
For sure, but there already are a lot of machines with this thing and it'll take multiple years for them to retire so...
Quoted 4 lines
I'm thinking the maintenance mode is the thing to do. It will also change behavior if you reset the management controller, but only for 30 seconds or so. Does that work?
Yeah, sounds good to me.
Thnaks.
Signature · 1 line
tejun
Quoted 17 lines · Tejun Heo
Hello, Corey.
Quoted 8 lines · Corey Minyard
I spent some time looking at this. Without the patch, I measured around 3.5ms to send/receive a get device ID message and uses 100% of the CPU on a core.
I measured your patch, it slowed it down to around 10.5ms per message, which is not good. Though it did just use a few percent of the core.
Quoted 5 lines
I wrote some code to auto-adjust the timer. It settled on a delay around 35us, which gave 4.7ms per message, which is probably acceptable, and used around 40% of the CPU. If I use any timeout (even a 0-10us range) it won't go below 4ms per message.
Quoted 3 lines
Out of curiosity, what makes 4.7ms okay and 10.5ms not? At least for the use case we have in the fleet (sensor reading mostly), the less disruptive the better as long as things don't timeout and fail.
Well, when you are loading firmware and it takes 10 minutes at max speed, taking 20-30 minutes is a lot worse. It's not reading sensors, which would be fine, it's tranferring large chunks of data.
Quoted 22 lines
Quoted 5 lines
The process is running at nice 19 priority, so it won't have a significant effect on other processes from a priority point of view. It may still be hitting the scheduler locks pretty hard, though. But I played with things quite a bit
And power. Imagine multi six digit number of machines burning a full core just because of this busy loop to read temperature sensors some msecs faster.
Quoted 11 lines
and the behavior or the management controller is too erratic to set a really good timeout. Maybe other ones are better, don't know.
One other option we have is that the driver has something called "maintenance mode". If it detect that you have reset the management controller or are issuing firmware commands, it will modify timeout behavior. It can also be activated manually. I could also make it switch to just calling schedule instead of delaying when in that mode.
Quoted 2 lines
Yeah, whatever which makes the common-case behavior avoid busy looping would work.
Ok, it's queued in linux-next now (and has been for a few days). I'll get it into the next kernel release (and I just noticed a spelling error and fixed it).
-corey
Quoted 18 lines
Quoted 3 lines
The right thing it do is complain bitterly to vendors who build hardware that has to be polled. But besides that,
For sure, but there already are a lot of machines with this thing and it'll take multiple years for them to retire so...
Quoted 4 lines
I'm thinking the maintenance mode is the thing to do. It will also change behavior if you reset the management controller, but only for 30 seconds or so. Does that work?
Yeah, sounds good to me.
Thnaks.
Signature · 1 line
tejun
Quoted 6 lines · Corey Minyard
Quoted 2 lines
Yeah, whatever which makes the common-case behavior avoid busy looping would work.
Ok, it's queued in linux-next now (and has been for a few days). I'll get it into the next kernel release (and I just noticed a spelling error and fixed it).
Looks great to me. Thanks a lot for working on the issue.
Signature · 1 line
tejun
Quoted 10 lines · Corey Minyard
Quoted 8 lines · Tejun Heo
ipmi_thread()uses back-to-backschedule()to poll for command completion which, on some machines, can push up CPU consumption and heavily tax the scheduler locks leading to noticeable overall performance degradation.This patch replaces
schedule()withusleep_range(100, 200). This allows the sensor readings to finish resonably fast and the cpu consumption of the kthread is kept under several percents of a core.
Quoted 6 lines
The IPMI thread was not really designed for sensor reading, it was designed so that firmware updates would happen in a reasonable time on systems without an interrupt on the IPMI interface. This change will degrade performance for that function. IIRC correctly the people who did the patch tried this and it slowed things down too much.
Also, can you point me to the exact patch? I'm kinda curious what kind of timning they used.
Thanks.
Signature · 1 line
tejun
Quoted 18 lines · Tejun Heo
Quoted 10 lines · Corey Minyard
Quoted 8 lines · Tejun Heo
ipmi_thread()uses back-to-backschedule()to poll for command completion which, on some machines, can push up CPU consumption and heavily tax the scheduler locks leading to noticeable overall performance degradation.This patch replaces
schedule()withusleep_range(100, 200). This allows the sensor readings to finish resonably fast and the cpu consumption of the kthread is kept under several percents of a core.Quoted 6 lines
The IPMI thread was not really designed for sensor reading, it was designed so that firmware updates would happen in a reasonable time on systems without an interrupt on the IPMI interface. This change will degrade performance for that function. IIRC correctly the people who did the patch tried this and it slowed things down too much.
Quoted 2 lines
Also, can you point me to the exact patch? I'm kinda curious what kind of timning they used.
I believe the change was 33979734cd35ae "IPMI: use schedule in kthread" The original change that added the kthread was a9a2c44ff0a1350 "ipmi: add timer thread".
I mis-remembered this, we switched from doing a udelay() to schedule(), but that udelay was 1us, so that's probably not helpful information.
-corey
Quoted 11 lines
Thanks.
Signature · 7 lines
tejun _______________________________________________ Openipmi-developer mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/openipmi-developer
Hello, Corey.
Quoted 7 lines · Corey Minyard
I believe the change was
33979734cd35ae"IPMI: use schedule in kthread" The original change that added the kthread wasa9a2c44ff0a1350"ipmi: add timer thread".I mis-remembered this, we switched from doing a
udelay()toschedule(), but that udelay was 1us, so that's probably not helpful information.
I see, so it went from non-yielding busy looping to an yiedling one. And, yeah, udelay(1) isn't much of a data point.
Thanks.
Signature · 1 line
tejun