SSD slowdown with 3.3.X?
All,
Has anyone reported slowness using SSD's on kernel 3.3.X compiled as 32-bit PAE with 16 Gigs of memory (I know I need to update to 64-bit already, will do with Fedora 17)?
I am see terrible r/w to an SSD using 3.3.2. When I boot the same machine and SSD back in 3.2.15 all is fine.
Joe Ceklosky
Quoted 8 lines · Joe Ceklosky
All,
Has anyone reported slowness using SSD's on kernel 3.3.X compiled as 32-bit PAE with 16 Gigs of memory (I know I need to update to 64-bit already, will do with Fedora 17)?
I am see terrible r/w to an SSD using 3.3.2. When I boot the same machine and SSD back in 3.2.15 all is fine.
Double check which IO-scheduler the kernel is choosing. For SSDs, it is normally "noop", but I noticed "cfq" being chosen instead for some reason.
find /sys -name scheduler|grep '/sd[a-z]/'|xargs cat
-ml
Quoted 7 lines · Mark Lord
Quoted 6 lines · Joe Ceklosky
All,
Has anyone reported slowness using SSD's on kernel 3.3.X compiled as 32-bit PAE with 16 Gigs of memory (I know I need to update to 64-bit already, will do with Fedora 17)?
Quoted 7 lines
Quoted 2 lines
I am see terrible r/w to an SSD using 3.3.2. When I boot the same machine and SSD back in 3.2.15 all is fine.
Double check which IO-scheduler the kernel is choosing. For SSDs, it is normally "noop", but I noticed "cfq" being chosen instead for some reason.
The default I/O scheduler is the default I/O scheduler. Drivers may override this (some high-end PCIe SSD drivers do this, and I think s390 block drivers do as well), but in general the default is left alone.
Cheers,
Jeff
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Joe Ceklosky
Mark,
Thanks for the info, but nothing like that shows up:
Quoted 8 lines
[jceklosk@neptune tmp]$ cat c-3.2.15
noop deadline [cfq]
noop deadline [cfq][jceklosk@neptune tmp]$ cat c-3.3.2
noop deadline [cfq]
noop deadline [cfq]
Well, the stuff you posted (above) shows that cfq is being used instead of noop. For SSDs, noop is the more natural choice, and used to be the default in the kernel for a while. I wonder when that changed?
You can change it (after boot) by just echo'ing "noop" into those same sysfs entries.
Cheers
Quoted 17 lines · Mark Lord
Quoted 6 lines · Joe Ceklosky
Mark,
Thanks for the info, but nothing like that shows up:
Quoted 8 lines
[jceklosk@neptune tmp]$ cat c-3.2.15
noop deadline [cfq]
noop deadline [cfq][jceklosk@neptune tmp]$ cat c-3.3.2
noop deadline [cfq]
noop deadline [cfq]
Quoted 4 lines
Well, the stuff you posted (above) shows that cfq is being used instead of noop. For SSDs, noop is the more natural choice, and used to be the default in the kernel for a while. I wonder when that changed?
Looking into the block layer now, I see that "cfq" at some point became "SSD aware", which is probably when the default io scheduler for SSDs changed back to cfq from noop.
Not 100% sure, but that's how it appears now.
I still have my systems set it to noop when an SSD is detected.
Quoted 8 lines · Mark Lord
Quoted 7 lines · Mark Lord
Quoted 6 lines · Joe Ceklosky
Mark,
Thanks for the info, but nothing like that shows up:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[jceklosk@neptune tmp]$ cat c-3.2.15
noop deadline [cfq]
noop deadline [cfq][jceklosk@neptune tmp]$ cat c-3.3.2
noop deadline [cfq]
noop deadline [cfq]
Probably not relevant in this case but maybe worth mentioning to get the word out:
"As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS."
http://www.xfs.org/index.php/XFS_FAQ
Signature · 1 line
Stan
Quoted 7 lines
Probably not relevant in this case but maybe worth mentioning to get the word out:
"As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS."
Not that it's terribly relevant to btrfs, but do you have a better citation for that than a very recent one-line wiki change that only cites the user's own anecdote?
Quoted 11 lines · cwillu
Quoted 7 lines
Probably not relevant in this case but maybe worth mentioning to get the word out:
"As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS."
Not that it's terribly relevant to btrfs, but do you have a better citation for that than a very recent one-line wiki change that only cites the user's own anecdote?
Apologies for the rather weak citation. It was simply easier to quote that wiki entry.
How about something directly from Dave's fingers:
http://www.spinics.net/lists/xfs/msg10824.html
The many issues with CFQ+XFS didn't start with 3.2.12, but long before that.
Signature · 1 line
Stan
Quoted 13 lines · Stan Hoeppner
Quoted 11 lines · cwillu
Quoted 7 lines
Probably not relevant in this case but maybe worth mentioning to get the word out:
"As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS."
Not that it's terribly relevant to btrfs, but do you have a better citation for that than a very recent one-line wiki change that only cites the user's own anecdote?
Quoted 7 lines
Apologies for the rather weak citation. It was simply easier to quote that wiki entry.
How about something directly from Dave's fingers:
http://www.spinics.net/lists/xfs/msg10824.htmlThe many issues with CFQ+XFS didn't start with 3.2.12, but long before that.
Thanks for the link. That's handy to know.
The problems there are for XFS+RAID vs. CFQ, not XFS by itself. Enterprise servers will normally have RAID under XFS, but not all smaller systems.
Cheers
Quoted 20 lines · Mark Lord
Quoted 19 lines · Stan Hoeppner
Quoted 11 lines · cwillu
Quoted 7 lines
Probably not relevant in this case but maybe worth mentioning to get the word out:
"As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS."
Not that it's terribly relevant to btrfs, but do you have a better citation for that than a very recent one-line wiki change that only cites the user's own anecdote?
Apologies for the rather weak citation. It was simply easier to quote that wiki entry.
How about something directly from Dave's fingers:
http://www.spinics.net/lists/xfs/msg10824.html
Quoted 8 lines
Quoted 1 line
The many issues with CFQ+XFS didn't start with 3.2.12, but long before that.
Thanks for the link. That's handy to know.
The problems there are for XFS+RAID vs. CFQ, not XFS by itself. Enterprise servers will normally have RAID under XFS, but not all smaller systems.
While it's true there are single disk XFS filesystems in the wild--I have one--I'd have to make an educated guess that the vast majority of XFS filesystems reside atop SAN, HBA, or md based RAID. For any hardware RAID solution with write cache, noop is recommended allowing the hardware scheduler to order the writes, since they're sitting in his cache. For md based RAID I believe most are getting best results with deadline.
I can't quote any numbers as I don't believe anyone has done such a poll or research on this. So it's best guess only.
Signature · 1 line
Stan
Quoted 18 lines · Mark Lord
Quoted 17 lines · Mark Lord
Quoted 14 lines · Joe Ceklosky
Mark,
Thanks for the info, but nothing like that shows up:
[jceklosk@neptune tmp]$ cat c-3.2.15
noop deadline [cfq]
noop deadline [cfq][jceklosk@neptune tmp]$ cat c-3.3.2
noop deadline [cfq]
noop deadline [cfq]
Quoted 8 lines
Quoted 4 lines
Well, the stuff you posted (above) shows that cfq is being used instead of noop. For SSDs, noop is the more natural choice, and used to be the default in the kernel for a while. I wonder when that changed?
Looking into the block layer now, I see that "cfq" at some point became "SSD aware", which is probably when the default io scheduler for SSDs changed back to cfq from noop.
The block layer never changed the I/O scheduler based on whether or not the underlying storage was an SSD. Maybe your particular distro did that for you? I can't say for sure, and it really doesn't matter as the original report, here, is that the SAME CONFIGURATION (cfq old and new) now regresses in performance. We should concentrate on fixing *that*.
Cheers,
Jeff