Skip to contentSkip to conversation

epoll: trim epitem by one cache line on x86_64

PATCH In review Eric Wong 2 participants 5 messages Mar 4, 2013 – Mar 9, 2013 VFS

lore
Share & tools
lore

Eric Wong

author
#0 lore raw

PATCH 1/1 epoll: trim epitem by one cache line on x86_64

It is common for epoll users to have thousands of epitems, so saving a cache line on every allocation leads to large memory savings.

Since epitem allocations are cache-aligned, reducing sizeof(struct epitem) from 136 bytes to 128 bytes will allow it to squeeze under a cache line boundary on x86_64.

Quoted 1 line

From /sys/kernel/slab/eventpoll_epi, I see the following changes on my

x86_64 Core2 Duo (which has 64-byte cache alignment):

	object_size  :  192 => 128
	objs_per_slab:   21 =>  32

I have no access to other 64-bit machines, so I am limiting this to x86_64-only with EPOLL_PACKED instead of __attribute__((packed))

Cc Davide Libenzi <[email protected]>
Cc Al Viro <[email protected]>
Cc Andrew Morton <[email protected]>
---
 fs/eventpoll.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/eventpoll.c +1 -1
index cfc4b16..06f3d0e 100644
@@ -107,7 +107,7 @@
 struct epoll_filefd {
 	struct file *file;
 	int fd;
-};
+} EPOLL_PACKED;
 
 /*
  * Structure used to track possible nested calls, for too deep recursions
Signature · 1 line
Eric Wong
#1 lore raw
Quoted 21 lines · On Mon, 4 Mar 2013 11:29:41 +0000 Eric Wong <[email protected]> wrote:

It is common for epoll users to have thousands of epitems, so saving a cache line on every allocation leads to large memory savings.

Since epitem allocations are cache-aligned, reducing sizeof(struct epitem) from 136 bytes to 128 bytes will allow it to squeeze under a cache line boundary on x86_64.

Quoted 1 line

From /sys/kernel/slab/eventpoll_epi, I see the following changes on my

x86_64 Core2 Duo (which has 64-byte cache alignment):

	object_size  :  192 => 128
	objs_per_slab:   21 =>  32

I have no access to other 64-bit machines, so I am limiting this to x86_64-only with EPOLL_PACKED instead of __attribute__((packed))

...

fs/eventpoll.c line 113

Quoted 8 lines
fs/eventpoll.c +1 -1
@@ -107,7 +107,7 @@
 struct epoll_filefd {
 	struct file *file;
 	int fd;
-};
+} EPOLL_PACKED;
 
 /*
  * Structure used to track possible nested calls, for too deep recursions

Yes, I see the same numbers on my gcc, x86_64 allmodconfig.

It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.

I don't see much additional room to be saved. We could probably remove epitem.nwait, but that wouldn't actually save anything because nwait nestles with ffd.fd.

I tested your patch on powerpc and it reduced sizeof(epitem) from 136 to 128 for that arch as well, so I suggest we run with

fs/eventpoll.c renamed from fs/eventpoll.c~epoll-trim-epitem-by-one-cache-line-on-x86_64-fix +1 -1
@@ -105,7 +105,7 @@
 struct epoll_filefd {
 	struct file *file;
 	int fd;
-} EPOLL_PACKED;
+} __packed;
 
 /*
  * Structure used to track possible nested calls, for too deep recursions

_

author Acked-by
#2 lore raw
Quoted 3 lines · Andrew Morton

It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.

Thanks for looking at this patch. I'll send a patch with a comment about keeping epitem size in check. Also, would adding (with comments):

	BUILD_BUG_ON(sizeof(struct epitem) > 128);

...be too heavy-handed? I used that in my testing. I'll check for:
sizeof(void *) <= 8 too; in case 128-bit machines appear...

Quoted 3 lines

I don't see much additional room to be saved. We could probably remove epitem.nwait, but that wouldn't actually save anything because nwait nestles with ffd.fd.

If we remove nwait, we can move epoll_event up and have event.events tucked in there. I have more and more depending on epoll, so I'll be around to comment on future epoll changes as they come up.

fs/eventpoll.c line 108

Quoted 11 lines

I tested your patch on powerpc and it reduced sizeof(epitem) from 136 to 128 for that arch as well, so I suggest we run with

fs/eventpoll.c renamed from fs/eventpoll.c~epoll-trim-epitem-by-one-cache-line-on-x86_64-fix +1 -1
@@ -105,7 +105,7 @@
 struct epoll_filefd {
 	struct file *file;
 	int fd;
-} EPOLL_PACKED;
+} __packed;

Thanks for testing on ppc. Looks good to me. For what it's worth:

#3 lore raw
Quoted 5 lines · On Thu, 7 Mar 2013 10:32:40 +0000 Eric Wong <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.

Quoted 7 lines

Thanks for looking at this patch. I'll send a patch with a comment about keeping epitem size in check. Also, would adding (with comments):

	BUILD_BUG_ON(sizeof(struct epitem) > 128);

...be too heavy-handed? I used that in my testing. I'll check for:
sizeof(void *) <= 8 too; in case 128-bit machines appear...

I guess such a check might avoid accidents in the future. If it becomes a problem, we can always delete it.

author
#4 lore raw

PATCH 1/1 epoll: comment + BUILD_BUG_ON to prevent epitem bloat

This will prevent us from accidentally introducing a memory bloat regression here in the future.

Cc Andrew Morton <[email protected]>
Cc Davide Libenzi <[email protected]>,
Cc Al Viro <[email protected]>

---

Quoted 10 lines · Andrew Morton
Quoted 8 lines · On Thu, 7 Mar 2013 10:32:40 +0000 Eric Wong <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.

Thanks for looking at this patch. I'll send a patch with a comment about keeping epitem size in check. Also, would adding (with comments):

Quoted 7 lines
Quoted 4 lines
	BUILD_BUG_ON(sizeof(struct epitem) > 128);

...be too heavy-handed? I used that in my testing. I'll check for:
sizeof(void *) <= 8 too; in case 128-bit machines appear...

I guess such a check might avoid accidents in the future. If it becomes a problem, we can always delete it.

 fs/eventpoll.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/eventpoll.c +8 -0
index 9fec183..55028da 100644
@@ -128,6 +128,8 @@ struct nested_calls {
 /*
  * Each file descriptor added to the eventpoll interface will
  * have an entry of this type linked to the "rbr" RB tree.
+ * Avoid increasing the size of this struct, there can be many thousands
+ * of these on a server and we do not want this to take another cache line.
  */
 struct epitem {
 	/* RB tree node used to link this structure to the eventpoll RB tree */
@@ -1964,6 +1966,12 @@ static int __init eventpoll_init(void)
 	/* Initialize the structure used to perform file's f_op->poll() calls */
 	ep_nested_calls_init(&poll_readywalk_ncalls);
 
+	/*
+	 * We can have many thousands of epitems, so prevent this from
+	 * using an extra cache line on 64-bit (and smaller) CPUs
+	 */
+	BUILD_BUG_ON(sizeof(void *) <= 8 && sizeof(struct epitem) > 128);
+
 	/* Allocates slab cache used to allocate "struct epitem" items */
 	epi_cache = kmem_cache_create("eventpoll_epi", sizeof(struct epitem),
 			0, SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL);
Signature · 1 line
Eric Wong