epoll: trim epitem by one cache line on x86_64
PATCH 1/1 epoll: trim epitem by one cache line on x86_64
It is common for epoll users to have thousands of epitems, so saving a cache line on every allocation leads to large memory savings.
Since epitem allocations are cache-aligned, reducing sizeof(struct epitem) from 136 bytes to 128 bytes will allow it to squeeze under a cache line boundary on x86_64.
Quoted 1 line
From
/sys/kernel/slab/eventpoll_epi, I see the following changes on my
x86_64 Core2 Duo (which has 64-byte cache alignment):
object_size : 192 => 128 objs_per_slab: 21 => 32
I have no access to other 64-bit machines, so I am limiting this to x86_64-only with EPOLL_PACKED instead of __attribute__((packed))
--- fs/eventpoll.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/eventpoll.c +1 -1
@@ -107,7 +107,7 @@struct epoll_filefd {struct file *file;int fd; -}; +}EPOLL_PACKED ;/* * Structure used to track possible nested calls, for too deep recursions
Signature · 1 line
Eric Wong
Quoted 21 lines · On Mon, 4 Mar 2013 11:29:41 +0000 Eric Wong <[email protected]> wrote:
It is common for epoll users to have thousands of epitems, so saving a cache line on every allocation leads to large memory savings.
Since epitem allocations are cache-aligned, reducing sizeof(struct epitem) from 136 bytes to 128 bytes will allow it to squeeze under a cache line boundary on
x86_64.Quoted 1 line
From
/sys/kernel/slab/eventpoll_epi, I see the following changes on my
x86_64Core2 Duo (which has 64-byte cache alignment):object_size : 192 => 128 objs_per_slab: 21 => 32I have no access to other 64-bit machines, so I am limiting this to x86_64-only with
EPOLL_PACKEDinstead of __attribute__((packed))...
fs/eventpoll.c line 113
Quoted 8 lines
fs/eventpoll.c +1
-1@@ -107,7 +107,7 @@struct epoll_filefd {struct file *file;int fd; -}; +}EPOLL_PACKED ;/* * Structure used to track possible nested calls, for too deep recursions
Yes, I see the same numbers on my gcc, x86_64 allmodconfig.
It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.
I don't see much additional room to be saved. We could probably remove epitem.nwait, but that wouldn't actually save anything because nwait nestles with ffd.fd.
I tested your patch on powerpc and it reduced sizeof(epitem) from 136 to 128 for that arch as well, so I suggest we run with
fs/eventpoll.c renamed from fs/eventpoll.c~epoll-trim-epitem-by-one-cache-line-on-x86_64-fix +1 -1
@@ -105,7 +105,7 @@struct epoll_filefd {struct file *file;int fd; -}EPOLL_PACKED ; +} __packed;/* * Structure used to track possible nested calls, for too deep recursions
_
Eric Wong
Acked-byQuoted 3 lines · Andrew Morton
It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.
Thanks for looking at this patch. I'll send a patch with a comment about keeping epitem size in check. Also, would adding (with comments):
BUILD_BUG_ON(sizeof(struct epitem) > 128);
...be too heavy-handed? I used that in my testing. I'll check for:sizeof(void *) <= 8 too; in case 128-bit machines appear...
Quoted 3 lines
I don't see much additional room to be saved. We could probably remove epitem.nwait, but that wouldn't actually save anything because nwait nestles with ffd.fd.
If we remove nwait, we can move epoll_event up and have event.events tucked in there. I have more and more depending on epoll, so I'll be around to comment on future epoll changes as they come up.
fs/eventpoll.c line 108
Quoted 11 lines
I tested your patch on powerpc and it reduced sizeof(epitem) from 136 to 128 for that arch as well, so I suggest we run with
fs/eventpoll.c renamed from fs/eventpoll.c~epoll-trim-epitem-by-one-cache-line-on-x86_64-fix +1
-1@@ -105,7 +105,7 @@struct epoll_filefd {struct file *file;int fd; -}EPOLL_PACKED ; +} __packed;
Thanks for testing on ppc. Looks good to me. For what it's worth:
Quoted 5 lines · On Thu, 7 Mar 2013 10:32:40 +0000 Eric Wong <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.
Quoted 7 lines
Thanks for looking at this patch. I'll send a patch with a comment about keeping epitem size in check. Also, would adding (with comments):
BUILD_BUG_ON(sizeof(struct epitem) > 128);...be too heavy-handed? I used that in my testing. I'll check for:
sizeof(void *)<= 8 too; in case 128-bit machines appear...
I guess such a check might avoid accidents in the future. If it becomes a problem, we can always delete it.
PATCH 1/1 epoll: comment + BUILD_BUG_ON to prevent epitem bloat
This will prevent us from accidentally introducing a memory bloat regression here in the future.
---
Quoted 10 lines · Andrew Morton
Quoted 8 lines · On Thu, 7 Mar 2013 10:32:40 +0000 Eric Wong <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
It's going to be hard to maintain this - someone will change something sometime and break it. I suppose we could add a runtime check if we cared enough. Adding a big fat comment to struct epitem might help.
Thanks for looking at this patch. I'll send a patch with a comment about keeping epitem size in check. Also, would adding (with comments):
Quoted 7 lines
Quoted 4 lines
BUILD_BUG_ON(sizeof(struct epitem) > 128);...be too heavy-handed? I used that in my testing. I'll check for:
sizeof(void *)<= 8 too; in case 128-bit machines appear...I guess such a check might avoid accidents in the future. If it becomes a problem, we can always delete it.
fs/eventpoll.c | 8 ++++++++ 1 file changed, 8 insertions(+)
fs/eventpoll.c +8 -0
@@ -128,6 +128,8 @@ struct nested_calls {/* * Each file descriptor added to the eventpoll interface will * have an entry of this type linked to the "rbr" RB tree. +* Avoid increasing the size of this struct, there can be many thousands +* of these on a server and we do not want this to take another cache line. */ struct epitem {/* RB tree node used to link this structure to the eventpoll RB tree */
@@ -1964,6 +1966,12 @@ static int __init eventpoll_init(void)/* Initialize the structure used to perform file's f_op->poll() calls */ ep_nested_calls_init (&poll_readywalk_ncalls); +/* +* We can have many thousands of epitems, so prevent this from +* using an extra cache line on 64-bit (and smaller) CPUs +*/ +BUILD_BUG_ON (sizeof (void *) <=8 &&sizeof (struct epitem ) >128 ); +/* Allocates slab cache used to allocate "struct epitem" items */ epi_cache =kmem_cache_create ("eventpoll_epi" ,sizeof (struct epitem ),0 ,SLAB_HWCACHE_ALIGN |SLAB_PANIC ,NULL );
Signature · 1 line
Eric Wong