sched: Do not re-read h_load_next during hierarchical load calculation
PATCH 1/1 sched: Do not re-read h_load_next during hierarchical load calculation
A NULL pointer dereference bug was reported on a distribution kernel but the same issue should be present on mainline kernel. It occured on s390 but should not be arch-specific. A partial oops looks like
[775277.408564] Unable to handle kernel pointer dereference in virtual kernel address space ... [775277.408759] Call Trace: [775277.408763] ([<0002c11c56899c61>] 0x2c11c56899c61) [775277.408766] [<0000000000177bb4>] try_to_wake_up+0xfc/0x450 [775277.408773] [<000003ff81ede872>] vhost_poll_wakeup+0x3a/0x50 [vhost] [775277.408777] [<0000000000194ae4>] __wake_up_common+0xbc/0x178 [775277.408779] [<0000000000194f86>] __wake_up_common_lock+0x9e/0x160 [775277.408780] [<00000000001950de>] __wake_up_sync_key+0x4e/0x60 [775277.408785] [<00000000005d911e>] sock_def_readable+0x5e/0x98
The bug hits any time between 1 hour to 3 days. The dereference occurs in update_cfs_rq_h_load when accumulating h_load. The problem is that cfq_rq->h_load_next is not protected by any locking and can be updated by parallel calls to task_h_load. Depending on the compiler, code may be generated that re-reads cfq_rq->h_load_next after the check for NULL and then oops when reading se->avg.load_avg. The dissassembly showed that it was possible to reread h_load_next after the check for NULL.
While this does not appear to be an issue for later compilers, it's still an accident if the correct code is generated. Full locking in this path would have high overhead so this patch uses READ_ONCE to read h_load_next only once and check for NULL before dereferencing. It was confirmed that there were no further oops after 10 days of testing.
--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
@@ -7726,7 +7726,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) cfs_rq->last_h_load_update = now; } -while ((se = cfs_rq->h_load_next) !=NULL ) { +while ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg,cfs_rq_load_avg (cfs_rq) +1 );
Hi,
Quoted 23 lines · Mel Gorman
A NULL pointer dereference bug was reported on a distribution kernel but the same issue should be present on mainline kernel. It occured on s390 but should not be arch-specific. A partial oops looks like
[775277.408564] Unable to handle kernel pointer dereference in virtual kernel address space ... [775277.408759] Call Trace: [775277.408763] ([<0002c11c56899c61>] 0x2c11c56899c61) [775277.408766] [<0000000000177bb4>] try_to_wake_up+0xfc/0x450 [775277.408773] [<000003ff81ede872>] vhost_poll_wakeup+0x3a/0x50 [vhost] [775277.408777] [<0000000000194ae4>] __wake_up_common+0xbc/0x178 [775277.408779] [<0000000000194f86>] __wake_up_common_lock+0x9e/0x160 [775277.408780] [<00000000001950de>] __wake_up_sync_key+0x4e/0x60 [775277.408785] [<00000000005d911e>] sock_def_readable+0x5e/0x98The bug hits any time between 1 hour to 3 days. The dereference occurs in
update_cfs_rq_h_loadwhen accumulatingh_load. The problem is thatcfq_rq->h_load_nextis not protected by any locking and can be updated by parallel calls totask_h_load. Depending on the compiler, code may be generated that re-readscfq_rq->h_load_nextafter the check for NULL and then oops when readingse->avg.load_avg. The dissassembly showed that it was possible to rereadh_load_nextafter the check for NULL.
Quoted 6 lines
While this does not appear to be an issue for later compilers, it's still an accident if the correct code is generated. Full locking in this path would have high overhead so this patch uses
READ_ONCEto readh_load_nextonly once and check for NULL before dereferencing. It was confirmed that there were no further oops after 10 days of testing.
Does that also want a
?
Other than that, the change looks sane to me.
Quoted 19 lines
Signed-off-by Mel Gorman <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +1
-1@@ -7726,7 +7726,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) cfs_rq->last_h_load_update = now; } -while ((se = cfs_rq->h_load_next) !=NULL ) { +while ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg,cfs_rq_load_avg (cfs_rq) +1 );
Quoted 35 lines · Valentin Schneider
Hi,
Quoted 23 lines · Mel Gorman
A NULL pointer dereference bug was reported on a distribution kernel but the same issue should be present on mainline kernel. It occured on s390 but should not be arch-specific. A partial oops looks like
[775277.408564] Unable to handle kernel pointer dereference in virtual kernel address space ... [775277.408759] Call Trace: [775277.408763] ([<0002c11c56899c61>] 0x2c11c56899c61) [775277.408766] [<0000000000177bb4>] try_to_wake_up+0xfc/0x450 [775277.408773] [<000003ff81ede872>] vhost_poll_wakeup+0x3a/0x50 [vhost] [775277.408777] [<0000000000194ae4>] __wake_up_common+0xbc/0x178 [775277.408779] [<0000000000194f86>] __wake_up_common_lock+0x9e/0x160 [775277.408780] [<00000000001950de>] __wake_up_sync_key+0x4e/0x60 [775277.408785] [<00000000005d911e>] sock_def_readable+0x5e/0x98The bug hits any time between 1 hour to 3 days. The dereference occurs in
update_cfs_rq_h_loadwhen accumulatingh_load. The problem is thatcfq_rq->h_load_nextis not protected by any locking and can be updated by parallel calls totask_h_load. Depending on the compiler, code may be generated that re-readscfq_rq->h_load_nextafter the check for NULL and then oops when readingse->avg.load_avg. The dissassembly showed that it was possible to rereadh_load_nextafter the check for NULL.Quoted 6 lines
While this does not appear to be an issue for later compilers, it's still an accident if the correct code is generated. Full locking in this path would have high overhead so this patch uses
READ_ONCEto readh_load_nextonly once and check for NULL before dereferencing. It was confirmed that there were no further oops after 10 days of testing.Does that also want a
Quoted 9 lines
Fixes 685207963be9 ("sched: Move h_load calculation to task_h_load()")?
Other than that, the change looks sane to me.
Reviewed-by Valentin Schneider <[email protected]>
Thanks. I'll add the Fixes because it does look like the old code would have been ok. Thanks for the review!
On 19/03/2019 12:30, Mel Gorman wrote:
[...]
Quoted 8 lines
Quoted 8 lines
Does that also want a
Fixes 685207963be9 ("sched: Move h_load calculation to task_h_load()")?
Quoted 8 lines
Quoted 4 lines
Other than that, the change looks sane to me.
Reviewed-by Valentin Schneider <[email protected]>Thanks. I'll add the Fixes because it does look like the old code would have been ok. Thanks for the review!
Well it's not worth much because I completely skipped that we *didn't* already have a WRITE_ONCE(), which you do need as Peter mentioned. But I do think stable will be happy to get this one :)
Peter Zijlstra
maintainerQuoted 15 lines · Mel Gorman
A NULL pointer dereference bug was reported on a distribution kernel but the same issue should be present on mainline kernel. It occured on s390 but should not be arch-specific. A partial oops looks like
[775277.408564] Unable to handle kernel pointer dereference in virtual kernel address space ... [775277.408759] Call Trace: [775277.408763] ([<0002c11c56899c61>] 0x2c11c56899c61) [775277.408766] [<0000000000177bb4>] try_to_wake_up+0xfc/0x450 [775277.408773] [<000003ff81ede872>] vhost_poll_wakeup+0x3a/0x50 [vhost] [775277.408777] [<0000000000194ae4>] __wake_up_common+0xbc/0x178 [775277.408779] [<0000000000194f86>] __wake_up_common_lock+0x9e/0x160 [775277.408780] [<00000000001950de>] __wake_up_sync_key+0x4e/0x60 [775277.408785] [<00000000005d911e>] sock_def_readable+0x5e/0x98
Quoted 4 lines
The bug hits any time between 1 hour to 3 days. The dereference occurs in
update_cfs_rq_h_loadwhen accumulatingh_load. The problem is thatcfq_rq->h_load_nextis not protected by any locking and can be updated by parallel calls totask_h_load.
Hurpmh, right.
Quoted 21 lines
Depending on the compiler, code may be
generated that re-readscfq_rq->h_load_nextafter the check for NULL and then oops when readingse->avg.load_avg. The dissassembly showed that it was possible to rereadh_load_nextafter the check for NULL.While this does not appear to be an issue for later compilers, it's still an accident if the correct code is generated. Full locking in this path would have high overhead so this patch uses
READ_ONCEto readh_load_nextonly once and check for NULL before dereferencing. It was confirmed that there were no further oops after 10 days of testing.Signed-off-by Mel Gorman <[email protected]>--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) · line 7732
Quoted 8 lines
kernel/sched/fair.c +1
-1@@ -7726,7 +7726,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) cfs_rq->last_h_load_update = now; } -while ((se = cfs_rq->h_load_next) !=NULL ) { +while ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg,cfs_rq_load_avg (cfs_rq) +1 );
Where there is a READ_ONCE there should also be a corresponding WRITE_ONCE(). Otherwise the compiler can still screw us over by doing store-tearing.
So something like the below. But looking at this, we probably also want ONCE treatment on cfs_rq->h_load itself, but that's another patch.
And I think we can do something with cfs_rq->last_h_load_update.
---
kernel/sched/fair.c +3 -3
@@ -7784,10 +7784,10 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)if (cfs_rq->last_h_load_update == now)return ; - cfs_rq->h_load_next =NULL ; +WRITE_ONCE (cfs_rq->h_load_next,NULL );for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); - cfs_rq->h_load_next = se; +WRITE_ONCE (cfs_rq->h_load_next, se);if (cfs_rq->last_h_load_update == now)break ; }
@@ -7797,7 +7797,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) cfs_rq->last_h_load_update = now; } -while ((se = cfs_rq->h_load_next) !=NULL ) { +while ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg,cfs_rq_load_avg (cfs_rq) +1 );
Quoted 22 lines · Peter Zijlstra
Quoted 9 lines
--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/fair.c +0
-0kernel/sched/fair.c static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) · line 7732
Quoted 8 lines
kernel/sched/fair.c +1
-1@@ -7726,7 +7726,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) cfs_rq->last_h_load_update = now; } -while ((se = cfs_rq->h_load_next) !=NULL ) { +while ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg,cfs_rq_load_avg (cfs_rq) +1 );Where there is a
READ_ONCEthere should also be a correspondingWRITE_ONCE(). Otherwise the compiler can still screw us over by doing store-tearing.
Quoted 5 lines
So something like the below. But looking at this, we probably also want ONCE treatment on
cfs_rq->h_loaditself, but that's another patch.And I think we can do something with
cfs_rq->last_h_load_update.
Ok, I hadn't taken into account the possibility of store tearing but you're right, it is a possibility depending on the architecture and your patch is better. While there are some potential issues around h_load, I had treated it as data that is approximate and only considered the potential NULL dereference to be relevant.
I'll send a v2 shortly.