Skip to contentSkip to conversation

fs/select.c: batch user writes in do_sys_poll

PATCH Superseded Daniel Axtens 4 participants 7 messages Aug 13, 2020 VFS

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs/select.c: batch user writes in do_sys_poll

When returning results to userspace, do_sys_poll repeatedly calls put_user() - once per fd that it's watching.

This means that on architectures that support some form of kernel-to-userspace access protection, we end up enabling and disabling access once for each file descripter we're watching. This is inefficent and we can improve things by batching the accesses together.

To make sure there's not too much happening in the window when user accesses are permitted, we don't walk the linked list with accesses on. This leads to some slightly messy code in the loop, unfortunately.

Unscientific benchmarking with the poll2_threads microbenchmark from will-it-scale, run as ./poll2_threads -t 1 -s 15:

- Bare-metal Power9 with KUAP: ~48.8% speed-up
- VM on amd64 laptop with SMAP: ~25.5% speed-up

---
 fs/select.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
fs/select.c +12 -2
index 7aef49552d4c..f58976da9e63 100644
@@ -1015,9 +1015,19 @@ static int do_sys_poll(struct pollfd __user *ufds, unsigned int nfds,
 		struct pollfd *fds = walk->entries;
 		int j;
 
+		if (!user_write_access_begin(ufds, (sizeof(struct pollfd) *
+						    walk->len)))
+			goto out_fds;
+
 		for (j = 0; j < walk->len; j++, ufds++)
-			if (__put_user(fds[j].revents, &ufds->revents))
-				goto out_fds;
+			unsafe_put_user(fds[j].revents, &ufds->revents, loop_fault);
+
+		user_write_access_end();
+		continue;
+
+loop_fault:
+		user_write_access_end();
+		goto out_fds;
   	}
 
 	err = fdcount;
Signature · 1 line
2.25.1
#1 lore raw
Quoted 12 lines · Daniel Axtens

When returning results to userspace, do_sys_poll repeatedly calls put_user() - once per fd that it's watching.

This means that on architectures that support some form of kernel-to-userspace access protection, we end up enabling and disabling access once for each file descripter we're watching. This is inefficent and we can improve things by batching the accesses together.

To make sure there's not too much happening in the window when user accesses are permitted, we don't walk the linked list with accesses on. This leads to some slightly messy code in the loop, unfortunately.

Quoted 7 lines

Unscientific benchmarking with the poll2_threads microbenchmark from will-it-scale, run as ./poll2_threads -t 1 -s 15:

- Bare-metal Power9 with KUAP: ~48.8% speed-up
- VM on amd64 laptop with SMAP: ~25.5% speed-up

Seem like this could simply use a copy_to_user to further simplify things?

Also please don't pointlessly add overly long lines.

#2 lore raw
Quoted 16 lines · Christoph Hellwig
Quoted 15 lines · Daniel Axtens

When returning results to userspace, do_sys_poll repeatedly calls put_user() - once per fd that it's watching.

This means that on architectures that support some form of kernel-to-userspace access protection, we end up enabling and disabling access once for each file descripter we're watching. This is inefficent and we can improve things by batching the accesses together.

To make sure there's not too much happening in the window when user accesses are permitted, we don't walk the linked list with accesses on. This leads to some slightly messy code in the loop, unfortunately.

Unscientific benchmarking with the poll2_threads microbenchmark from will-it-scale, run as ./poll2_threads -t 1 -s 15:

Quoted 7 lines
Quoted 4 lines

- Bare-metal Power9 with KUAP: ~48.8% speed-up
- VM on amd64 laptop with SMAP: ~25.5% speed-up

Seem like this could simply use a copy_to_user to further simplify things?

I'll benchmark it and find out.

Quoted 1 line

Also please don't pointlessly add overly long lines.

Weird, I ran the commit through checkpatch and it didn't pick it up. I'll check the next version more carefully.

Regards,
Daniel

#3 lore raw

Hi,

Quoted 4 lines
Quoted 2 lines

Seem like this could simply use a copy_to_user to further simplify things?

I'll benchmark it and find out.

I tried this:

        for (walk = head; walk; walk = walk->next) {

- struct pollfd *fds = walk->entries;
- int j;
-
- for (j = 0; j < walk->len; j++, ufds++)
- if (__put_user(fds[j].revents, &ufds->revents))
- goto out_fds;
+ if (copy_to_user(ufds, walk->entries,
+ sizeof(struct pollfd) * walk->len))
+ goto out_fds;

+               ufds += walk->len;
        }

With that approach, the poll2 microbenchmark (which polls 128 fds) is about as fast as v1.

However, the poll1 microbenchmark, which polls just 1 fd, regresses a touch (<1% - ~2%) compared to the current code, although it's largely within the noise. Thoughts?

Kind regards,
Daniel

Quoted 7 lines
Quoted 1 line

Also please don't pointlessly add overly long lines.

Weird, I ran the commit through checkpatch and it didn't pick it up. I'll check the next version more carefully.

Regards,
Daniel

#4 lore raw

From: Daniel Axtens

Quoted 22 lines

Sent: 13 August 2020 12:37

Quoted 4 lines
Quoted 2 lines

Seem like this could simply use a copy_to_user to further simplify things?

I'll benchmark it and find out.

I tried this:

        for (walk = head; walk; walk = walk->next) {
(quoted diff) +4 -6
-               struct pollfd *fds = walk->entries;
-               int j;
-
-               for (j = 0; j < walk->len; j++, ufds++)
-                       if (__put_user(fds[j].revents, &ufds->revents))
-                               goto out_fds;
+               if (copy_to_user(ufds, walk->entries,
+                                sizeof(struct pollfd) * walk->len))
+                       goto out_fds;
+               ufds += walk->len;
        }
Quoted 6 lines

With that approach, the poll2 microbenchmark (which polls 128 fds) is about as fast as v1.

However, the poll1 microbenchmark, which polls just 1 fd, regresses a touch (<1% - ~2%) compared to the current code, although it's largely within the noise. Thoughts?

Is that with or without 'user copy hardening'?
Or use __copy_to_user() to skip all that 'crap'.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

maintainer
#5 lore raw
In reply to Daniel Axtens
Quoted 22 lines · Daniel Axtens

Hi,

Quoted 4 lines
Quoted 2 lines

Seem like this could simply use a copy_to_user to further simplify things?

I'll benchmark it and find out.

I tried this:

        for (walk = head; walk; walk = walk->next) {
(quoted diff) +4 -6
-               struct pollfd *fds = walk->entries;
-               int j;
-
-               for (j = 0; j < walk->len; j++, ufds++)
-                       if (__put_user(fds[j].revents, &ufds->revents))
-                               goto out_fds;
+               if (copy_to_user(ufds, walk->entries,
+                                sizeof(struct pollfd) * walk->len))
+                       goto out_fds;
+               ufds += walk->len;
        }
Quoted 6 lines

With that approach, the poll2 microbenchmark (which polls 128 fds) is about as fast as v1.

However, the poll1 microbenchmark, which polls just 1 fd, regresses a touch (<1% - ~2%) compared to the current code, although it's largely within the noise. Thoughts?

I'd go with copy_to_user() here; post such variant and I'll throw it into -next after -rc1.

#6 lore raw
In reply to Christoph Hellwig

From: Christoph Hellwig

Quoted 18 lines

Sent: 13 August 2020 08:32

Quoted 15 lines · Daniel Axtens

When returning results to userspace, do_sys_poll repeatedly calls put_user() - once per fd that it's watching.

This means that on architectures that support some form of kernel-to-userspace access protection, we end up enabling and disabling access once for each file descripter we're watching. This is inefficent and we can improve things by batching the accesses together.

To make sure there's not too much happening in the window when user accesses are permitted, we don't walk the linked list with accesses on. This leads to some slightly messy code in the loop, unfortunately.

Unscientific benchmarking with the poll2_threads microbenchmark from will-it-scale, run as ./poll2_threads -t 1 -s 15:

Quoted 7 lines
Quoted 4 lines

- Bare-metal Power9 with KUAP: ~48.8% speed-up
- VM on amd64 laptop with SMAP: ~25.5% speed-up

Seem like this could simply use a copy_to_user to further simplify things?

That would copy out 8 bytes/fd instead of 2.
So a slight change for 32bit kernels.
However the 'user copy hardening' checks that copy_to_user() does probably make a measurable difference.

Quoted 1 line

Also please don't pointlessly add overly long lines.

Shorten the error label?

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)