Skip to contentSkip to conversation

relay: handle alloc_percpu returning NULL in relay_open

PATCH Superseded Daniel Axtens 4 participants 6 messages Nov 29, 2019 – Dec 24, 2019 Relay

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 relay: handle alloc_percpu returning NULL in relay_open

alloc_percpu() may return NULL, which means chan->buf may be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:

BUG: Unable to handle kernel data access at 0x7dae0000

Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600

Call Trace:
[c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
[c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
[c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
[c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
[c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
[c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
[c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
[c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68

Check if alloc_percpu returns NULL. Because we can readily catch and handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.

This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.

Fixes 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Cc Akash Goel <[email protected]>
Cc Andrew Donnellan <[email protected]> # syzkaller-ppc64
Cc [email protected] # v4.10+

--

There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.

I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the relay_channels_mutex, so it blocks all future relay activity.

---
 kernel/relay.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/relay.c +7 -1
index ade14fb7ce2e..a376cc6b54ec 100644
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
 	if (!chan)
 		return NULL;
 
-	chan->buf = alloc_percpu(struct rchan_buf *);
+	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!chan->buf) {
+		kfree(chan);
+		return NULL;
+	}
+
 	chan->version = RELAYFS_CHANNEL_VERSION;
 	chan->n_subbufs = n_subbufs;
 	chan->subbuf_size = subbuf_size;
Signature · 1 line
2.20.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 35 lines · Daniel Axtens

alloc_percpu() may return NULL, which means chan->buf may be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:

BUG: Unable to handle kernel data access at 0x7dae0000

Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600

Call Trace:
[c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
[c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
[c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
[c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
[c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
[c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
[c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
[c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68

Check if alloc_percpu returns NULL. Because we can readily catch and handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.

This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.

Fixes 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Cc Akash Goel <[email protected]>
Cc Andrew Donnellan <[email protected]> # syzkaller-ppc64
Cc [email protected] # v4.10+

...

Quoted 9 lines
kernel/relay.c +0 -1
index ade14fb7ce2e..a376cc6b54ec 100644
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
 	if (!chan)
 		return NULL;
 
-	chan->buf = alloc_percpu(struct rchan_buf *);

kernel/relay.c struct rchan *relay_open(const char *base_filename, · line 592

Quoted 10 lines
kernel/relay.c +7 -0
+	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!chan->buf) {
+		kfree(chan);
+		return NULL;
+	}
+
 	chan->version = RELAYFS_CHANNEL_VERSION;
 	chan->n_subbufs = n_subbufs;
 	chan->subbuf_size = subbuf_size;

This looks right to me. The kfree + direct return is correct, there's nothing else that needs tear down in this function.

I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per cpu, and if that fails the system is pretty sick, so a warning could be helpful. There's also logic in the percpu allocator to limit the number of warnings printed. But see what others think.

cheers

Reviewed-by
#2 lore raw
Quoted 23 lines · Michael Ellerman
Quoted 9 lines
kernel/relay.c +0 -1
index ade14fb7ce2e..a376cc6b54ec 100644
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
  	if (!chan)
  		return NULL;
 
-	chan->buf = alloc_percpu(struct rchan_buf *);

kernel/relay.c struct rchan *relay_open(const char *base_filename, · line 592

Quoted 10 lines
kernel/relay.c +7 -0
+	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!chan->buf) {
+		kfree(chan);
+		return NULL;
+	}
+
  	chan->version = RELAYFS_CHANNEL_VERSION;
  	chan->n_subbufs = n_subbufs;
  	chan->subbuf_size = subbuf_size;

This looks right to me. The kfree + direct return is correct, there's nothing else that needs tear down in this function.

Quoted 4 lines

I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per cpu, and if that fails the system is pretty sick, so a warning could be helpful. There's also logic in the percpu allocator to limit the number of warnings printed. But see what others think.

mpe was wondering why we didn't see a message printed from the percpu allocator - the answer appears to be that we hit this case when the process is killed while the percpu allocator is waiting for pcpu_alloc_mutex, in which case it bails out without printing a warning.

It looks to me like that case doesn't warrant a warning message, while a failing allocation for other reasons should probably get a warning.

But whatever, otherwise this patch looks good to me. I've told our powerpc syzbot to test it.

Signature · 2 lines
Andrew Donnellan              OzLabs, ADL Canberra
[email protected]             IBM Australia Limited
#3 lore raw
In reply to Daniel Axtens
Quoted 11 lines · Daniel Axtens

--

There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.

I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the relay_channels_mutex, so it blocks all future relay activity.

CVE-2019-19462 has been assigned.

Regards,
Daniel

Quoted 25 lines
---
 kernel/relay.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/relay.c +7 -1
index ade14fb7ce2e..a376cc6b54ec 100644
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
 	if (!chan)
 		return NULL;
 
-	chan->buf = alloc_percpu(struct rchan_buf *);
+	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!chan->buf) {
+		kfree(chan);
+		return NULL;
+	}
+
 	chan->version = RELAYFS_CHANNEL_VERSION;
 	chan->n_subbufs = n_subbufs;
 	chan->subbuf_size = subbuf_size;
Signature · 1 line
2.20.1
#4 lore raw
In reply to Daniel Axtens
Quoted 26 lines · Daniel Axtens

alloc_percpu() may return NULL, which means chan->buf may be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:

BUG: Unable to handle kernel data access at 0x7dae0000

Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600

Call Trace:
[c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
[c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
[c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
[c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
[c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
[c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
[c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
[c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68

Check if alloc_percpu returns NULL. Because we can readily catch and handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.

This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.

Quoted 10 lines
Fixes 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Cc Akash Goel <[email protected]>
Cc Andrew Donnellan <[email protected]> # syzkaller-ppc64
Cc [email protected] # v4.10+

So there is a CVE now, but it appears that the patch went nowhere. Are there any plans to actually apply it ?

Thanks,
Guenter

Quoted 37 lines

--

There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.

I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the relay_channels_mutex, so it blocks all future relay activity.

---
 kernel/relay.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/relay.c +7 -1
index ade14fb7ce2e..a376cc6b54ec 100644
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
 	if (!chan)
 		return NULL;
 
-	chan->buf = alloc_percpu(struct rchan_buf *);
+	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!chan->buf) {
+		kfree(chan);
+		return NULL;
+	}
+
 	chan->version = RELAYFS_CHANNEL_VERSION;
 	chan->n_subbufs = n_subbufs;
 	chan->subbuf_size = subbuf_size;
Signature · 1 line
2.20.1
#5 lore raw
Quoted 38 lines · Guenter Roeck
Quoted 26 lines · Daniel Axtens

alloc_percpu() may return NULL, which means chan->buf may be set to NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we dereference an invalid pointer:

BUG: Unable to handle kernel data access at 0x7dae0000

Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600

Call Trace:
[c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
[c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
[c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
[c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
[c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
[c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
[c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
[c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68

Check if alloc_percpu returns NULL. Because we can readily catch and handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.

This was found by syzkaller both on x86 and powerpc, and the reproducer it found on powerpc is capable of hitting the issue as an unprivileged user.

Quoted 10 lines
Fixes 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Reported-by [email protected]
Cc Akash Goel <[email protected]>
Cc Andrew Donnellan <[email protected]> # syzkaller-ppc64
Cc [email protected] # v4.10+
Quoted 2 lines

So there is a CVE now, but it appears that the patch went nowhere. Are there any plans to actually apply it ?

I sent a v2 that addresses some review comments, I guess if anything is applied it will be that.

Daniel

Quoted 41 lines

Thanks,
Guenter

Quoted 37 lines

--

There's a syz reproducer on the powerpc syzbot that eventually hits the bug, but it can take up to an hour or so before it keels over on a kernel with all the syzkaller debugging on, and even longer on a production kernel. I have been able to reproduce it once on a stock Ubuntu 5.0 ppc64le kernel.

I will ask MITRE for a CVE - while only the process doing the syscall gets killed, it gets killed while holding the relay_channels_mutex, so it blocks all future relay activity.

---
 kernel/relay.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/relay.c +7 -1
index ade14fb7ce2e..a376cc6b54ec 100644
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
 	if (!chan)
 		return NULL;
 
-	chan->buf = alloc_percpu(struct rchan_buf *);
+	chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+				     GFP_KERNEL | __GFP_NOWARN);
+	if (!chan->buf) {
+		kfree(chan);
+		return NULL;
+	}
+
 	chan->version = RELAYFS_CHANNEL_VERSION;
 	chan->n_subbufs = n_subbufs;
 	chan->subbuf_size = subbuf_size;
Signature · 1 line
2.20.1