all your slabs are belong to ram ?

DISCUSSION [email protected] 3 participants 4 messages Oct 6, 2015

Share & tools

[email protected]

author

2015-10-06 14:24 UTC #0 lore raw

Dear List, nice to meet you :)

First post, straight to the point:
I'm wrestling from few weeks with a problem: roundcube (a webmail client) takes too long to open a dovecot (pop/imap server) mailbox with many emails (files). So, when such user has more than 10K emails, it takes around 1 minute to open the mailbox.
Meanwhile, I/O %util goes to 100% and bottlenecks the whole system.

I've tried memcache(d) integration with roundcube but it doesn't eliminate the problem.

OS is Debian Wheezy 32-bit, 16GB of ECC RAM and storage is a simple hardware raid-1 with a couple of sata2 hard disks. I've just used mkfs.xfs (with no tuning) and no options while mounting (in fstab).

It looks like the problem is the slow access to dentries and inodes, so I've set vfs_cache_pressure to 1, forced buffering with few "find /var/mail > /dev/null"
and have it running like this from around 4 days.
Didn't help: slabs still gets flushed and opening folders is slow as before.

Current slabtop usage shows:
235352K used by xfs_inode
and
49708K used by dentry
while I would expect to have at least 1 GB of xfs_inode and at least 200MB of dentry.

So I'm asking you:
1. is there a way to force dentries and inodes to stay in ram ?
2. can I perhaps move dentries and inodes to a dedicated SSD ?

I'm open to all possibilities, perhaps increase RAM ? Upgrade to Debian Jessie and 64 bit ?

Let me know if I can provide more info.

Thank you very much!
Mike
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Darrick J. Wong

2015-10-06 14:50 UTC #1 lore raw

Quoted 23 lines · On Tue, Oct 06, 2015 at 04:24:43PM +0200, [email protected] wrote:

Dear List, nice to meet you :)
First post, straight to the point:
I'm wrestling from few weeks with a problem: roundcube (a webmail client) takes too long to open a dovecot (pop/imap server) mailbox with many emails (files). So, when such user has more than 10K emails, it takes around 1 minute to open the mailbox.
Meanwhile, I/O %util goes to 100% and bottlenecks the whole system.
I've tried memcache(d) integration with roundcube but it doesn't eliminate the problem.
OS is Debian Wheezy 32-bit, 16GB of ECC RAM and storage is a simple hardware raid-1 with a couple of sata2 hard disks. I've just used mkfs.xfs (with no tuning) and no options while mounting (in fstab).
It looks like the problem is the slow access to dentries and inodes, so I've set vfs_cache_pressure to 1, forced buffering with few "find /var/mail > /dev/null"
and have it running like this from around 4 days.
Didn't help: slabs still gets flushed and opening folders is slow as before.
Current slabtop usage shows:
235352K used by xfs_inode
and
49708K used by dentry
while I would expect to have at least 1 GB of xfs_inode and at least 200MB of dentry.

Quoted 6 lines

So I'm asking you:
1. is there a way to force dentries and inodes to stay in ram ?
2. can I perhaps move dentries and inodes to a dedicated SSD ?
I'm open to all possibilities, perhaps increase RAM ? Upgrade to Debian Jessie and 64 bit ?

ISTR that kernel data such as slabs cannot live in highmem, which means that dentries and slab cannot live in highmem. A 32bit kernel sets up ~900M of low memory and ~15G of highmem, which is probably why the kernel has to evict things and why you see such problems.

A 64bit kernel sets up all the memory as lowmem, so the kernel can use all the memory for stuff like that. I'd give that a try first.

--D

Quoted 10 lines

Let me know if I can provide more info.
Thank you very much!
Mike
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Johannes Truschnigg

2015-10-06 16:16 UTC #2 lore raw

Am Di, 6.10.2015, 16:50 schrieb Darrick J. Wong:

Quoted 16 lines · On Tue, Oct 06, 2015 at 04:24:43PM +0200, [email protected] wrote:

Quoted 7 lines
[...]
So I'm asking you:
1. is there a way to force dentries and inodes to stay in ram ?
2. can I perhaps move dentries and inodes to a dedicated SSD ?
I'm open to all possibilities, perhaps increase RAM ? Upgrade to Debian Jessie and 64 bit ?
ISTR that kernel data such as slabs cannot live in highmem, which means that
dentries and slab cannot live in highmem. A 32bit kernel sets up ~900M of low
memory and ~15G of highmem, which is probably why the kernel has to evict things and why you see such problems.

Quoted 3 lines

A 64bit kernel sets up all the memory as lowmem, so the kernel can use all the
memory for stuff like that. I'd give that a try first.

A few years back, we solved pretty much that exact same problem by switching the kernel to amd64, with all of userspace remaining i386. You should definitely try this.

Signature · 21 lines

Mit freundlichen Grüßen
Johannes Truschnigg
Senior System Administrator
--
mailto:[email protected] (in dringenden Fällen bitte an
[email protected])

Geizhals(R) - Preisvergleich Internet Services AG
Obere Donaustrasse 63/2
A-1020 Wien
Tel: +43 1 5811609/87
Fax: +43 1 5811609/55
http://geizhals.at => Preisvergleich für Österreich
http://geizhals.de => Preisvergleich für Deutschland
http://geizhals.eu => Preisvergleich EU-weit
Handelsgericht Wien | FN 197241K | Firmensitz Wien

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

[email protected]

author

2015-10-06 18:34 UTC #3 lore raw

Thank you very much, will do!
(and report back the results)

Regards and thanks for supporting,
Mike

Quoted 17 lines · On Tue, 6 Oct 2015 18:16:46 +0200 "Johannes Truschnigg"…

Am Di, 6.10.2015, 16:50 schrieb Darrick J. Wong:
Quoted 16 lines · On Tue, Oct 06, 2015 at 04:24:43PM +0200, [email protected] wrote:
Quoted 7 lines
[...]
So I'm asking you:
1. is there a way to force dentries and inodes to stay in ram ?
2. can I perhaps move dentries and inodes to a dedicated SSD ?
I'm open to all possibilities, perhaps increase RAM ? Upgrade to Debian Jessie and 64 bit ?
ISTR that kernel data such as slabs cannot live in highmem, which means that
dentries and slab cannot live in highmem. A 32bit kernel sets up ~900M of low
memory and ~15G of highmem, which is probably why the kernel has to evict things and why you see such problems.

Quoted 30 lines

Quoted 3 lines
A 64bit kernel sets up all the memory as lowmem, so the kernel can use all the
memory for stuff like that. I'd give that a try first.
A few years back, we solved pretty much that exact same problem by switching the kernel to amd64, with all of userspace remaining i386. You should definitely try this.
Signature · 21 lines
Mit freundlichen Grüßen
Johannes Truschnigg
Senior System Administrator
--
mailto:[email protected] (in dringenden Fällen bitte an
[email protected])

Geizhals(R) - Preisvergleich Internet Services AG
Obere Donaustrasse 63/2
A-1020 Wien
Tel: +43 1 5811609/87
Fax: +43 1 5811609/55
http://geizhals.at => Preisvergleich für Österreich
http://geizhals.de => Preisvergleich für Deutschland
http://geizhals.eu => Preisvergleich EU-weit
Handelsgericht Wien | FN 197241K | Firmensitz Wien

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs