relay: handle alloc_percpu returning NULL in relay_open
PATCH 1/1 relay: handle alloc_percpu returning NULL in relay_open
alloc_percpu() may return NULL, which means chan->buf may be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:
BUG: Unable to handle kernel data access at 0x7dae0000
Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600
Call Trace: [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable) [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600 [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0 [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80 [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300 [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130 [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0 [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
Check if alloc_percpu returns NULL. Because we can readily catch and handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.
--
There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.
I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the relay_channels_mutex, so it blocks all future relay activity.
--- kernel/relay.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/relay.c +7 -1
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,if (!chan)return NULL ; - chan->buf = alloc_percpu(struct rchan_buf *); + chan->buf = alloc_percpu_gfp(struct rchan_buf *, +GFP_KERNEL | __GFP_NOWARN); +if (!chan->buf) { +kfree (chan); +return NULL ; + } + chan->version =RELAYFS_CHANNEL_VERSION ; chan->n_subbufs = n_subbufs; chan->subbuf_size = subbuf_size;
Signature · 1 line
2.20.1
Quoted 35 lines · Daniel Axtens
alloc_percpu()may return NULL, which meanschan->bufmay be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:BUG: Unable to handle kernel data access at 0x7dae0000Faulting instruction address:
0xc0000000003f3fec
...
NIP [c0000000003f3fec]relay_open+0x29c/0x600
LR [c0000000003f3fc0]relay_open+0x270/0x600Call Trace: [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable) [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600 [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0 [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80 [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300 [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130 [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0 [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68Check if
alloc_percpureturns NULL. Because we can readily catch and handle this situation, switch toalloc_cpu_gfpand pass in__GFP_NOWARN.This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.
Fixes 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Cc Akash Goel <[email protected]>Cc Andrew Donnellan <[email protected]> # syzkaller-ppc64Cc [email protected] # v4.10+Signed-off-by Daniel Axtens <[email protected]>
...
Quoted 9 lines
kernel/relay.c +0
-1@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,if (!chan)return NULL ; - chan->buf = alloc_percpu(struct rchan_buf *);
kernel/relay.c struct rchan *relay_open(const char *base_filename, · line 592
Quoted 10 lines
kernel/relay.c +7
-0+ chan->buf = alloc_percpu_gfp(struct rchan_buf *, +GFP_KERNEL | __GFP_NOWARN); +if (!chan->buf) { +kfree (chan); +return NULL ; + } + chan->version =RELAYFS_CHANNEL_VERSION ; chan->n_subbufs = n_subbufs; chan->subbuf_size = subbuf_size;
This looks right to me. The kfree + direct return is correct, there's nothing else that needs tear down in this function.
I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per cpu, and if that fails the system is pretty sick, so a warning could be helpful. There's also logic in the percpu allocator to limit the number of warnings printed. But see what others think.
cheers
Andrew Donnellan
Reviewed-byQuoted 23 lines · Michael Ellerman
Quoted 9 lines
kernel/relay.c +0
-1@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,if (!chan)return NULL ; - chan->buf = alloc_percpu(struct rchan_buf *);kernel/relay.c struct rchan *relay_open(const char *base_filename, · line 592
Quoted 10 lines
kernel/relay.c +7
-0+ chan->buf = alloc_percpu_gfp(struct rchan_buf *, +GFP_KERNEL | __GFP_NOWARN); +if (!chan->buf) { +kfree (chan); +return NULL ; + } + chan->version =RELAYFS_CHANNEL_VERSION ; chan->n_subbufs = n_subbufs; chan->subbuf_size = subbuf_size;This looks right to me. The kfree + direct return is correct, there's nothing else that needs tear down in this function.
Quoted 4 lines
I think I'm 50/50 on the
__GFP_NOWARN. We're only asking for 8 bytes per cpu, and if that fails the system is pretty sick, so a warning could be helpful. There's also logic in the percpu allocator to limit the number of warnings printed. But see what others think.
mpe was wondering why we didn't see a message printed from the percpu allocator - the answer appears to be that we hit this case when the process is killed while the percpu allocator is waiting for pcpu_alloc_mutex, in which case it bails out without printing a warning.
It looks to me like that case doesn't warrant a warning message, while a failing allocation for other reasons should probably get a warning.
But whatever, otherwise this patch looks good to me. I've told our powerpc syzbot to test it.
Signature · 2 lines
Andrew Donnellan OzLabs, ADL Canberra [email protected] IBM Australia Limited
Quoted 11 lines · Daniel Axtens
--
There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.
I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the
relay_channels_mutex, so it blocks all future relay activity.
CVE-2019-19462 has been assigned.
Regards,
Daniel
Quoted 25 lines
--- kernel/relay.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)kernel/relay.c +7
-1@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,if (!chan)return NULL ; - chan->buf = alloc_percpu(struct rchan_buf *); + chan->buf = alloc_percpu_gfp(struct rchan_buf *, +GFP_KERNEL | __GFP_NOWARN); +if (!chan->buf) { +kfree (chan); +return NULL ; + } + chan->version =RELAYFS_CHANNEL_VERSION ; chan->n_subbufs = n_subbufs; chan->subbuf_size = subbuf_size;Signature · 1 line
2.20.1
Quoted 26 lines · Daniel Axtens
alloc_percpu()may return NULL, which meanschan->bufmay be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:BUG: Unable to handle kernel data access at 0x7dae0000Faulting instruction address:
0xc0000000003f3fec
...
NIP [c0000000003f3fec]relay_open+0x29c/0x600
LR [c0000000003f3fc0]relay_open+0x270/0x600Call Trace: [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable) [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600 [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0 [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80 [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300 [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130 [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0 [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68Check if
alloc_percpureturns NULL. Because we can readily catch and handle this situation, switch toalloc_cpu_gfpand pass in__GFP_NOWARN.This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.
Quoted 10 lines
Fixes 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Cc Akash Goel <[email protected]>Cc Andrew Donnellan <[email protected]> # syzkaller-ppc64Cc [email protected] # v4.10+Signed-off-by Daniel Axtens <[email protected]>
So there is a CVE now, but it appears that the patch went nowhere. Are there any plans to actually apply it ?
Thanks,
Guenter
Quoted 37 lines
--
There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.
I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the
relay_channels_mutex, so it blocks all future relay activity.--- kernel/relay.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)kernel/relay.c +7
-1@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,if (!chan)return NULL ; - chan->buf = alloc_percpu(struct rchan_buf *); + chan->buf = alloc_percpu_gfp(struct rchan_buf *, +GFP_KERNEL | __GFP_NOWARN); +if (!chan->buf) { +kfree (chan); +return NULL ; + } + chan->version =RELAYFS_CHANNEL_VERSION ; chan->n_subbufs = n_subbufs; chan->subbuf_size = subbuf_size;Signature · 1 line
2.20.1
Quoted 38 lines · Guenter Roeck
Quoted 26 lines · Daniel Axtens
alloc_percpu()may return NULL, which meanschan->bufmay be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:BUG: Unable to handle kernel data access at 0x7dae0000Faulting instruction address:
0xc0000000003f3fec
...
NIP [c0000000003f3fec]relay_open+0x29c/0x600
LR [c0000000003f3fc0]relay_open+0x270/0x600Call Trace: [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable) [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600 [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0 [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80 [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300 [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130 [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0 [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68Check if
alloc_percpureturns NULL. Because we can readily catch and handle this situation, switch toalloc_cpu_gfpand pass in__GFP_NOWARN.This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.
Quoted 10 lines
Fixes 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Reported-by [email protected]Cc Akash Goel <[email protected]>Cc Andrew Donnellan <[email protected]> # syzkaller-ppc64Cc [email protected] # v4.10+Signed-off-by Daniel Axtens <[email protected]>
Quoted 2 lines
So there is a CVE now, but it appears that the patch went nowhere. Are there any plans to actually apply it ?
I sent a v2 that addresses some review comments, I guess if anything is applied it will be that.
Daniel
Quoted 41 lines
Thanks,
GuenterQuoted 37 lines
--
There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.
I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the
relay_channels_mutex, so it blocks all future relay activity.--- kernel/relay.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)kernel/relay.c +7
-1@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,if (!chan)return NULL ; - chan->buf = alloc_percpu(struct rchan_buf *); + chan->buf = alloc_percpu_gfp(struct rchan_buf *, +GFP_KERNEL | __GFP_NOWARN); +if (!chan->buf) { +kfree (chan); +return NULL ; + } + chan->version =RELAYFS_CHANNEL_VERSION ; chan->n_subbufs = n_subbufs; chan->subbuf_size = subbuf_size;Signature · 1 line
2.20.1