Skip to contentSkip to conversation

Debugging system freezes on filesystem writes

DISCUSSION Marcus Sundman 4 participants 34 messages Oct 28, 2012 – Sep 16, 2013

lore
Share & tools
lore
#0 lore raw

Hi,

I have a big problem with the system freezing and would appreciate any help on debugging this and pinpointing where exactly the problem is, so it could be fixed.

So, whenever I write to the disk the system comes to a crawl or freezes altogether. This happens even when the writing processes are running on nice '19' and ionice 'idle'. (E.g. a 10 second compile could freeze the system for several minutes, rendering the computer pretty much unusable for anything interesting.)

Here you can see a 20 second gap even in superhigh priority:

# nice -n -20 ionice -c1 iostat -t -m -d -x 1 > http://pastebin.com/j5qnh2VV

I'm currently running 3.5.0-17-lowlatency on the ZenBook UX31E, using the NOOP I/O scheduler on the SanDisk SSD U100. The chipset seems to be Intel QS67. I've had this same problem on 3.2.0 generic and lowlatency kernels.

The syslog says nothing relevant before/during/after these freezes.

Any ideas on finding the culprit?

Regards,
Marcus

#1 lore raw

On Mon 29-10-12 00:39:46, Marcus Sundman wrote:
Hello,

Quoted 10 lines

I have a big problem with the system freezing and would appreciate any help on debugging this and pinpointing where exactly the problem is, so it could be fixed.

So, whenever I write to the disk the system comes to a crawl or freezes altogether. This happens even when the writing processes are running on nice '19' and ionice 'idle'. (E.g. a 10 second compile could freeze the system for several minutes, rendering the computer pretty much unusable for anything interesting.)

Quoted 7 lines

Here you can see a 20 second gap even in superhigh priority:

# nice -n -20 ionice -c1 iostat -t -m -d -x 1 > http://pastebin.com/j5qnh2VV

I'm currently running 3.5.0-17-lowlatency on the ZenBook UX31E, using the NOOP I/O scheduler on the SanDisk SSD U100. The chipset seems to be Intel QS67. I've had this same problem on 3.2.0 generic and lowlatency kernels.

These are Ubuntu kernels. Any chance to reproduce the issue with vanilla
kernels - i.e. kernels without any Ubuntu patches? Also when you speak of system freezing - can you e.g. type to terminal while the system is frozen? Or is it just that running commands freezes? And how much free memory do you have while the system is frozen? Finally, can you trigger the freeze by something simpler than compilation - e.g. does
dd if=/dev/zero of=/tmp bs=1M
trigger the freeze as well?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#2 lore raw
Quoted 16 lines · Jan Kara

On Mon 29-10-12 00:39:46, Marcus Sundman wrote:
Hello,

Quoted 13 lines

I have a big problem with the system freezing and would appreciate any help on debugging this and pinpointing where exactly the problem is, so it could be fixed.

So, whenever I write to the disk the system comes to a crawl or freezes altogether. This happens even when the writing processes are running on nice '19' and ionice 'idle'. (E.g. a 10 second compile could freeze the system for several minutes, rendering the computer pretty much unusable for anything interesting.)

Here you can see a 20 second gap even in superhigh priority:

# nice -n -20 ionice -c1 iostat -t -m -d -x 1 > http://pastebin.com/j5qnh2VV
Quoted 6 lines
Quoted 4 lines

I'm currently running 3.5.0-17-lowlatency on the ZenBook UX31E, using the NOOP I/O scheduler on the SanDisk SSD U100. The chipset seems to be Intel QS67. I've had this same problem on 3.2.0 generic and lowlatency kernels.

These are Ubuntu kernels. Any chance to reproduce the issue with vanilla
kernels - i.e. kernels without any Ubuntu patches?

I'm afraid it's going to take a week to compile a kernel with this freezing going on, but I suppose I could get another computer to do the compiling. Or should I install some pre-compiled version? If so, which one?

Quoted 3 lines

Also when you speak of
system freezing - can you e.g. type to terminal while the system is frozen? Or is it just that running commands freezes?

Typing usually doesn't work very well. It works for a word or two and then stops working for a while and if I continue to type then when it resumes only the last few characters appears. Typing in the console is a bit better than in a terminal in X (not counting the several minutes it can take to switch to the console (Ctrl-Alt-F1)).

Quoted 2 lines

And how much free memory do
you have while the system is frozen?

It varies. Or it depends on how you look at it, usually my RAM is full, but mostly by "buffers" (whatever that is in practice). My swap is close to zero, because I keep swappiness at 1, or else the freezing gets totally out of control. And I've disabled journaling, because journaling also makes it much, much worse. (Using ext4, btw.)

Quoted 4 lines

Finally, can you trigger the freeze by
something simpler than compilation - e.g. does
dd if=/dev/zero of=/tmp bs=1M
trigger the freeze as well?

Yes, that command sure does trigger the freezes. However, if there's nothing else going on then that command doesn't make the system freeze totally (at least not immediately), but if I do some other filesystem activity (e.g., ls) at the same time then the freezing starts.

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Regards,
Marcus

#3 lore raw
Quoted 21 lines · Marcus Sundman
Quoted 20 lines · Jan Kara

On Mon 29-10-12 00:39:46, Marcus Sundman wrote:
Hello,

Quoted 17 lines

I have a big problem with the system freezing and would appreciate any help on debugging this and pinpointing where exactly the problem is, so it could be fixed.

So, whenever I write to the disk the system comes to a crawl or freezes altogether. This happens even when the writing processes are running on nice '19' and ionice 'idle'. (E.g. a 10 second compile could freeze the system for several minutes, rendering the computer pretty much unusable for anything interesting.)

Here you can see a 20 second gap even in superhigh priority:

# nice -n -20 ionice -c1 iostat -t -m -d -x 1 > http://pastebin.com/j5qnh2VV

I'm currently running 3.5.0-17-lowlatency on the ZenBook UX31E, using the NOOP I/O scheduler on the SanDisk SSD U100. The chipset seems to be Intel QS67. I've had this same problem on 3.2.0 generic and lowlatency kernels.

Quoted 7 lines
Quoted 2 lines

These are Ubuntu kernels. Any chance to reproduce the issue with vanilla
kernels - i.e. kernels without any Ubuntu patches?

I'm afraid it's going to take a week to compile a kernel with this freezing going on, but I suppose I could get another computer to do the compiling. Or should I install some pre-compiled version? If so, which one?

You can install anything precompiled. It's just that I want to rule out
some Ubuntu specific patches...

Quoted 9 lines
Quoted 3 lines

Also when you speak of
system freezing - can you e.g. type to terminal while the system is frozen? Or is it just that running commands freezes?

Typing usually doesn't work very well. It works for a word or two and then stops working for a while and if I continue to type then when it resumes only the last few characters appears. Typing in the console is a bit better than in a terminal in X (not counting the several minutes it can take to switch to the console (Ctrl-Alt-F1)).

I see.

Quoted 5 lines

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the freeze and send them for comparison. Maybe it will show us what is the system doing.

Also you can try doing:
echo never >/sys/kernel/mm/transparent_hugepage/enabled
and see whether it changes anything.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#4 lore raw
Quoted 22 lines · Jan Kara
Quoted 21 lines · Marcus Sundman
Quoted 20 lines · Jan Kara

On Mon 29-10-12 00:39:46, Marcus Sundman wrote:
Hello,

Quoted 17 lines

I have a big problem with the system freezing and would appreciate any help on debugging this and pinpointing where exactly the problem is, so it could be fixed.

So, whenever I write to the disk the system comes to a crawl or freezes altogether. This happens even when the writing processes are running on nice '19' and ionice 'idle'. (E.g. a 10 second compile could freeze the system for several minutes, rendering the computer pretty much unusable for anything interesting.)

Here you can see a 20 second gap even in superhigh priority:

# nice -n -20 ionice -c1 iostat -t -m -d -x 1 > http://pastebin.com/j5qnh2VV

I'm currently running 3.5.0-17-lowlatency on the ZenBook UX31E, using the NOOP I/O scheduler on the SanDisk SSD U100. The chipset seems to be Intel QS67. I've had this same problem on 3.2.0 generic and lowlatency kernels.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

These are Ubuntu kernels. Any chance to reproduce the issue with vanilla
kernels - i.e. kernels without any Ubuntu patches?

I'm afraid it's going to take a week to compile a kernel with this freezing going on, but I suppose I could get another computer to do the compiling. Or should I install some pre-compiled version? If so, which one?

You can install anything precompiled. It's just that I want to rule out
some Ubuntu specific patches...

OK, I tried it with a vanilla 3.6.6 -- "uname -a" says "Linux hal 3.6.6-030606-generic #201211050512 SMP Mon Nov 5 10:12:53 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux"

Quoted 15 lines
Quoted 8 lines
Quoted 3 lines

Also when you speak of
system freezing - can you e.g. type to terminal while the system is frozen? Or is it just that running commands freezes?

Typing usually doesn't work very well. It works for a word or two and then stops working for a while and if I continue to type then when it resumes only the last few characters appears. Typing in the console is a bit better than in a terminal in X (not counting the several minutes it can take to switch to the console (Ctrl-Alt-F1)).

I see.

Quoted 5 lines

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Quoted 4 lines

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the freeze and send them for comparison. Maybe it will show us what is the system doing.

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Quoted 3 lines

Also you can try doing:
echo never >/sys/kernel/mm/transparent_hugepage/enabled
and see whether it changes anything.

It's already set to 'nerver'. I think I configured this in the very beginning when trying to do something about these freezes.

I also have these in sysctl:
vm.swappiness=1
vm.vfs_cache_pressure=50
vm.dirty_ratio = 15
vm.dirty_background_ratio = 8

And /sys/block/sda/device/queue_depth is 1.

Regards,
Marcus

#5 lore raw
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Quoted 8 lines
Quoted 4 lines

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the freeze and send them for comparison. Maybe it will show us what is the system doing.

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Here are some more vmstats:
http://sundman.iki.fi/vmstats.tar.gz

They are from running this:
while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep 10; done

There were lots and lots of freezes for almost 20 mins from 14:37:45 onwards, pretty much constantly, but at 14:56:50 the freezes suddenly stopped and everything went back to how it should be.

Thanks,
Marcus

#6 lore raw
Quoted 21 lines · Marcus Sundman
Quoted 13 lines · Marcus Sundman
Quoted 11 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the
freeze and send them for comparison. Maybe it will show us what is the system doing.

Quoted 3 lines

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Here are some more vmstats:
http://sundman.iki.fi/vmstats.tar.gz

Quoted 7 lines

They are from running this:
while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep 10; done

There were lots and lots of freezes for almost 20 mins from 14:37:45 onwards, pretty much constantly, but at 14:56:50 the freezes suddenly stopped and everything went back to how it should be.

I was looking into the data but they didn't show anything problematic.
The machine seems to be writing a lot but there's always some free memory, even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't showing much IO going on but vmstats show there is about 1 GB written during the freeze. It is not a huge amount given the time span but it certainly gives a few MB/s of write load.

There's surprisingly high number of allocations going on but that may be due to the IO activity. So let's try something else: Can you switch to console and when the hang happens press Alt-Sysrq-w (or you can just do "echo w >/proc/sysrq-trigger" if the machine is live enough to do that). Then send me the output from dmesg. Thanks!

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#7 lore raw
Quoted 27 lines · Jan Kara
Quoted 19 lines · Marcus Sundman
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Quoted 8 lines
Quoted 5 lines

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the
freeze and send them for comparison. Maybe it will show us what is the system doing.

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Here are some more vmstats:
http://sundman.iki.fi/vmstats.tar.gz

Quoted 7 lines

They are from running this:
while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep 10; done

There were lots and lots of freezes for almost 20 mins from 14:37:45 onwards, pretty much constantly, but at 14:56:50 the freezes suddenly stopped and everything went back to how it should be.

Quoted 6 lines

I was looking into the data but they didn't show anything problematic.
The machine seems to be writing a lot but there's always some free memory, even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't showing much IO going on but vmstats show there is about 1 GB written during the freeze. It is not a huge amount given the time span but it certainly gives a few MB/s of write load.

I didn't watch iotop during this particular freeze. I'll try to keep an eye on iotop in the future. Is there some particular options I should run iotop with, or is a "nice -n -20 iotop -od3" fine?

Quoted 5 lines

There's surprisingly high number of allocations going on but that may be due to the IO activity. So let's try something else: Can you switch to console and when the hang happens press Alt-Sysrq-w (or you can just do "echo w >/proc/sysrq-trigger" if the machine is live enough to do that). Then send me the output from dmesg. Thanks!

Sure! Here are two:
http://sundman.iki.fi/dmesg-1.txt
http://sundman.iki.fi/dmesg-2.txt

Best regards,
Marcus

#8 lore raw
Quoted 35 lines · Marcus Sundman
Quoted 27 lines · Jan Kara
Quoted 19 lines · Marcus Sundman
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Quoted 8 lines
Quoted 5 lines

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the
freeze and send them for comparison. Maybe it will show us what is the system doing.

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Here are some more vmstats:
http://sundman.iki.fi/vmstats.tar.gz

Quoted 7 lines

They are from running this:
while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep 10; done

There were lots and lots of freezes for almost 20 mins from 14:37:45 onwards, pretty much constantly, but at 14:56:50 the freezes suddenly stopped and everything went back to how it should be.

Quoted 6 lines

I was looking into the data but they didn't show anything problematic.
The machine seems to be writing a lot but there's always some free memory, even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't showing much IO going on but vmstats show there is about 1 GB written during the freeze. It is not a huge amount given the time span but it certainly gives a few MB/s of write load.

Quoted 3 lines

I didn't watch iotop during this particular freeze. I'll try to keep an eye on iotop in the future. Is there some particular options I should run iotop with, or is a "nice -n -20 iotop -od3" fine?

I'm not really familiar with iotop :). Usually I use iostat...

Quoted 9 lines
Quoted 5 lines

There's surprisingly high number of allocations going on but that may be due to the IO activity. So let's try something else: Can you switch to console and when the hang happens press Alt-Sysrq-w (or you can just do "echo w >/proc/sysrq-trigger" if the machine is live enough to do that). Then send me the output from dmesg. Thanks!

Sure! Here are two:
http://sundman.iki.fi/dmesg-1.txt
http://sundman.iki.fi/dmesg-2.txt

Thanks for those and sorry for the delay (I was busy with other stuff).
I had a look into those traces and I have to say I'm not much wiser. In the first dump there is just kswapd waiting for IO. In the second dump there are more processes waiting for IO (mostly for reads - nautilus, thunderbird, opera, ...) but nothing really surprising. So I'm lost what could cause the hangs you observe. Recalling you wrote even simple programs like top hang, maybe it is some CPU scheduling issue? Can you boot with noautogroup kernel option?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#9 lore raw
Quoted 35 lines · Jan Kara
Quoted 34 lines · Marcus Sundman
Quoted 33 lines · Jan Kara
Quoted 19 lines · Marcus Sundman
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Quoted 8 lines
Quoted 5 lines

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the
freeze and send them for comparison. Maybe it will show us what is the system doing.

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Here are some more vmstats:
http://sundman.iki.fi/vmstats.tar.gz

Quoted 7 lines

They are from running this:
while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep 10; done

There were lots and lots of freezes for almost 20 mins from 14:37:45 onwards, pretty much constantly, but at 14:56:50 the freezes suddenly stopped and everything went back to how it should be.

I was looking into the data but they didn't show anything problematic.
The machine seems to be writing a lot but there's always some free memory, even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't showing much IO going on but vmstats show there is about 1 GB written during the freeze. It is not a huge amount given the time span but it certainly gives a few MB/s of write load.

Quoted 4 lines
Quoted 3 lines

I didn't watch iotop during this particular freeze. I'll try to keep an eye on iotop in the future. Is there some particular options I should run iotop with, or is a "nice -n -20 iotop -od3" fine?

I'm not really familiar with iotop :). Usually I use iostat...

OK, which options for iostat should I use then? :)

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

There's surprisingly high number of allocations going on but that may be due to the IO activity. So let's try something else: Can you switch to console and when the hang happens press Alt-Sysrq-w (or you can just do "echo w >/proc/sysrq-trigger" if the machine is live enough to do that). Then send me the output from dmesg. Thanks!

Sure! Here are two:
http://sundman.iki.fi/dmesg-1.txt
http://sundman.iki.fi/dmesg-2.txt

Quoted 6 lines

Thanks for those and sorry for the delay (I was busy with other stuff).
I had a look into those traces and I have to say I'm not much wiser. In the first dump there is just kswapd waiting for IO. In the second dump there are more processes waiting for IO (mostly for reads - nautilus, thunderbird, opera, ...) but nothing really surprising. So I'm lost what could cause the hangs you observe.

Yes, mostly it's difficult to trigger the sysrq thingy, because by the time I manage to switch to the console or running that echo to proc in a terminal the worst is already over.

Quoted 3 lines

Recalling you wrote even simple programs
like top hang, maybe it is some CPU scheduling issue? Can you boot with noautogroup kernel option?

Sure. I've been running with noautogroup for almost a week now, but no big change one way or the other. (E.g., it's still impossible to listen to music, because the songs will start skipping/looping several times during each song even if there isn't any big "hang" happening. And uncompressing a 100 MB archive (with nice '19' and ionice 'idle') is still, after a while, followed by a couple of minutes of superhigh I/O wait causing everything to become really slow.)

- Marcus

#10 lore raw
Quoted 36 lines · Marcus Sundman
Quoted 35 lines · Jan Kara
Quoted 34 lines · Marcus Sundman
Quoted 33 lines · Jan Kara
Quoted 19 lines · Marcus Sundman
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Quoted 8 lines
Quoted 5 lines

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the
freeze and send them for comparison. Maybe it will show us what is the system doing.

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Here are some more vmstats:
http://sundman.iki.fi/vmstats.tar.gz

Quoted 7 lines

They are from running this:
while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep 10; done

There were lots and lots of freezes for almost 20 mins from 14:37:45 onwards, pretty much constantly, but at 14:56:50 the freezes suddenly stopped and everything went back to how it should be.

I was looking into the data but they didn't show anything problematic.
The machine seems to be writing a lot but there's always some free memory, even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't showing much IO going on but vmstats show there is about 1 GB written during the freeze. It is not a huge amount given the time span but it certainly gives a few MB/s of write load.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

I didn't watch iotop during this particular freeze. I'll try to keep an eye on iotop in the future. Is there some particular options I should run iotop with, or is a "nice -n -20 iotop -od3" fine?

I'm not really familiar with iotop :). Usually I use iostat...

OK, which options for iostat should I use then? :)

I'm back from vacation. Sorry for the delay. You can use
iostat -x 1

Quoted 15 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

There's surprisingly high number of allocations going on but that may be due to the IO activity. So let's try something else: Can you switch to console and when the hang happens press Alt-Sysrq-w (or you can just do "echo w >/proc/sysrq-trigger" if the machine is live enough to do that). Then send me the output from dmesg. Thanks!

Sure! Here are two:
http://sundman.iki.fi/dmesg-1.txt
http://sundman.iki.fi/dmesg-2.txt

Quoted 6 lines

Thanks for those and sorry for the delay (I was busy with other stuff).
I had a look into those traces and I have to say I'm not much wiser. In the first dump there is just kswapd waiting for IO. In the second dump there are more processes waiting for IO (mostly for reads - nautilus, thunderbird, opera, ...) but nothing really surprising. So I'm lost what could cause the hangs you observe.

Quoted 3 lines

Yes, mostly it's difficult to trigger the sysrq thingy, because by the time I manage to switch to the console or running that echo to proc in a terminal the worst is already over.

I see. Maybe you could have something like
while true; do echo w >/proc/sysrq-trigger; sleep 10; done
running in the background?

Quoted 4 lines
Quoted 3 lines

Recalling you wrote even simple programs
like top hang, maybe it is some CPU scheduling issue? Can you boot with noautogroup kernel option?

Quoted 8 lines

Sure. I've been running with noautogroup for almost a week now, but no big change one way or the other. (E.g., it's still impossible to listen to music, because the songs will start skipping/looping several times during each song even if there isn't any big "hang" happening. And uncompressing a 100 MB archive (with nice '19' and ionice 'idle') is still, after a while, followed by a couple of minutes of superhigh I/O wait causing everything to become really slow.)

Hum, I'm starting to wander what's so special about your system that you
see these hangs while noone else seems to be hitting them. Your kernel is a standard one from Ubuntu so tons of people run it. Your HW doesn't seem to be too special either.

BTW the fact that you ionice 'tar' doesn't change anything because all the writes are done in the context of kernel flusher thread (tar just writes data into cache). But still it shouldn't lock the machine up. What might be interesting test though is running:
dd if=/dev/zero of=file bs=1M count=200 oflags=direct

Does this trigger any hangs?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw
Quoted 37 lines · Jan Kara
Quoted 36 lines · Marcus Sundman
Quoted 35 lines · Jan Kara
Quoted 34 lines · Marcus Sundman
Quoted 33 lines · Jan Kara
Quoted 19 lines · Marcus Sundman
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

Also, and this might be important, according to iotop there is almost no disk writing going on during the freeze. (Occasionally there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least when an iotop running on nice -20 hasn't frozen completely, which it does during the more severe freezes.

Quoted 8 lines
Quoted 5 lines

OK, it seems as if your machine has some problems with memory
allocations. Can you capture /proc/vmstat before the freeze and after the
freeze and send them for comparison. Maybe it will show us what is the system doing.

t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt

Here are some more vmstats:
http://sundman.iki.fi/vmstats.tar.gz

Quoted 7 lines

They are from running this:
while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep 10; done

There were lots and lots of freezes for almost 20 mins from 14:37:45 onwards, pretty much constantly, but at 14:56:50 the freezes suddenly stopped and everything went back to how it should be.

I was looking into the data but they didn't show anything problematic.
The machine seems to be writing a lot but there's always some free memory, even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't showing much IO going on but vmstats show there is about 1 GB written during the freeze. It is not a huge amount given the time span but it certainly gives a few MB/s of write load.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines

I didn't watch iotop during this particular freeze. I'll try to keep an eye on iotop in the future. Is there some particular options I should run iotop with, or is a "nice -n -20 iotop -od3" fine?

I'm not really familiar with iotop :). Usually I use iostat...

OK, which options for iostat should I use then? :)

I'm back from vacation. Sorry for the delay. You can use
iostat -x 1

Just when you got back I started my pre-vacation work stress and am now ending my post-vacation work-stress.. :)

That iostat -x 1 shows %util as 100 and w_await at 2,000 - 70,000.. like so:

avg-cpu: %user %nice %system %iowait %steal %idle

           9.05    0.00    1.51   66.33    0.00   23.12

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 1.00 0.00 184.00 368.00 137.08 62199.00 0.00 62199.00 1000.00 100.00

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 8 lines
Quoted 5 lines

There's surprisingly high number of allocations going on but that may be due to the IO activity. So let's try something else: Can you switch to console and when the hang happens press Alt-Sysrq-w (or you can just do "echo w >/proc/sysrq-trigger" if the machine is live enough to do that). Then send me the output from dmesg. Thanks!

Sure! Here are two:
http://sundman.iki.fi/dmesg-1.txt
http://sundman.iki.fi/dmesg-2.txt

Thanks for those and sorry for the delay (I was busy with other stuff).
I had a look into those traces and I have to say I'm not much wiser. In the first dump there is just kswapd waiting for IO. In the second dump there are more processes waiting for IO (mostly for reads - nautilus, thunderbird, opera, ...) but nothing really surprising. So I'm lost what could cause the hangs you observe.

Quoted 6 lines
Quoted 3 lines

Yes, mostly it's difficult to trigger the sysrq thingy, because by the time I manage to switch to the console or running that echo to proc in a terminal the worst is already over.

I see. Maybe you could have something like
while true; do echo w >/proc/sysrq-trigger; sleep 10; done
running in the background?

Sure, but I suspect it'll take until the worst is over before it manages to load and execute that "echo w".

Quoted 16 lines
Quoted 11 lines
Quoted 3 lines

Recalling you wrote even simple programs
like top hang, maybe it is some CPU scheduling issue? Can you boot with noautogroup kernel option?

Sure. I've been running with noautogroup for almost a week now, but no big change one way or the other. (E.g., it's still impossible to listen to music, because the songs will start skipping/looping several times during each song even if there isn't any big "hang" happening. And uncompressing a 100 MB archive (with nice '19' and ionice 'idle') is still, after a while, followed by a couple of minutes of superhigh I/O wait causing everything to become really slow.)

Hum, I'm starting to wander what's so special about your system that you
see these hangs while noone else seems to be hitting them. Your kernel is a standard one from Ubuntu so tons of people run it. Your HW doesn't seem to be too special either.

Quoted 7 lines

BTW the fact that you ionice 'tar' doesn't change anything because all the writes are done in the context of kernel flusher thread (tar just writes data into cache). But still it shouldn't lock the machine up. What might be interesting test though is running:
dd if=/dev/zero of=file bs=1M count=200 oflags=direct

Does this trigger any hangs?

Yes, sure. If I run nothing else then it's not so severe, but the system is still quite unusable during the time it runs that dd.

Also, the speeds are closer to an Amiga500-era floppy drive than to an SSD from 2012 which this is:

$ dd if=/dev/zero of=iotest-file bs=1M count=200 oflag=direct
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 171.701 s, 1.2 MB/s
$

Regards,
Marcus

#12 lore raw
Quoted 6 lines · Marcus Sundman
Quoted 3 lines · Jan Kara

I see. Maybe you could have something like
while true; do echo w >/proc/sysrq-trigger; sleep 10; done
running in the background?

Sure, but I suspect it'll take until the worst is over before it manages to load and execute that "echo w".

Here is a big run of sysrq-triggering, all while I was uncompressing a big rar file causing the whole system to be utterly unusable.
NB: Even with realtime I/O-priority the sysrq couldn't be triggered between 12:41:54 and 12:42:49, as you can see from the dmesg-3.txt file.

$ sudo ionice -c 1 su
# ionice
realtime: prio 4
# while true; do sleep 10; echo w >/proc/sysrq-trigger; done
^C
# tail -n +1700 /var/log/syslog >dmesg-3.txt

http://sundman.iki.fi/dmesg-3.txt

Regards,
Marcus

#13 lore raw
Quoted 15 lines · Marcus Sundman
Quoted 7 lines · Marcus Sundman
Quoted 3 lines · Jan Kara

I see. Maybe you could have something like
while true; do echo w >/proc/sysrq-trigger; sleep 10; done
running in the background?

Sure, but I suspect it'll take until the worst is over before it manages to load and execute that "echo w".

Here is a big run of sysrq-triggering, all while I was uncompressing a big rar file causing the whole system to be utterly unusable.
NB: Even with realtime I/O-priority the sysrq couldn't be triggered between 12:41:54 and 12:42:49, as you can see from the dmesg-3.txt file.

Quoted 8 lines
$ sudo ionice -c 1 su
# ionice
realtime: prio 4
# while true; do sleep 10; echo w >/proc/sysrq-trigger; done
^C
# tail -n +1700 /var/log/syslog >dmesg-3.txt

http://sundman.iki.fi/dmesg-3.txt

Thanks for the traces. I was looking at them and we seem to be always
waiting for IO. There don't seem to be that much CPU load either.

I'm actually starting to suspect the SSD in your laptop. The svctm field from iostat output shows it takes 1 second on average to complete an IO request. That is awfully slow given one request has ~180 KB of data on average. Ah, one more idea - can you post your /proc/mounts please?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#14 lore raw
Quoted 21 lines · Jan Kara
Quoted 20 lines · Marcus Sundman
Quoted 6 lines · Marcus Sundman
Quoted 3 lines · Jan Kara

I see. Maybe you could have something like
while true; do echo w >/proc/sysrq-trigger; sleep 10; done
running in the background?

Sure, but I suspect it'll take until the worst is over before it manages to load and execute that "echo w".

Here is a big run of sysrq-triggering, all while I was uncompressing a big rar file causing the whole system to be utterly unusable.
NB: Even with realtime I/O-priority the sysrq couldn't be triggered between 12:41:54 and 12:42:49, as you can see from the dmesg-3.txt file.

$ sudo ionice -c 1 su
# ionice
realtime: prio 4
# while true; do sleep 10; echo w >/proc/sysrq-trigger; done
^C
# tail -n +1700 /var/log/syslog >dmesg-3.txt
Quoted 5 lines
Quoted 1 line

http://sundman.iki.fi/dmesg-3.txt

Thanks for the traces. I was looking at them and we seem to be always
waiting for IO. There don't seem to be that much CPU load either.

I'm actually starting to suspect the SSD in your laptop.

I've suspected the driver, because I don't remember it being slow in windows before I wiped it and installed ubuntu on it. (I didn't do very much with it in windows, though. I just downloaded the ubuntu image, but AFAICR it had no problem saving the image at the speed of my internet connection (which is normally around 6 MiB/s), but nowadays I have to strype my downloads to less than 1 MiB/s for it to not lock up so much.)

Quoted 4 lines

The svctm field
from iostat output shows it takes 1 second on average to complete an IO request. That is awfully slow given one request has ~180 KB of data on average. Ah, one more idea - can you post your /proc/mounts please?

Sure:

Quoted 9 lines
$ cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=1964816k,nr_inodes=491204,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=789652k,mode=755 0 0
/dev/disk/by-uuid/5bfa7a58-2d35-4758-954e-4deafb09b892 / ext4 rw,noatime,discard,errors=remount-ro 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
Quoted 8 lines
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
/dev/sda6 /home ext4 rw,noatime,discard 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
gvfsd-fuse /run/user/marcus/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=100 0 0

Both / and /home are on the same SSD and suffer from the same problem. (I think the swap does as well, but I have my swappiness set very low to minimize swapping.)

Regards,
Marcus

#15 lore raw
Quoted 10 lines · Marcus Sundman
Quoted 10 lines
$ cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=1964816k,nr_inodes=491204,mode=755 0 0
devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=789652k,mode=755 0 0
/dev/disk/by-uuid/5bfa7a58-2d35-4758-954e-4deafb09b892 / ext4
rw,noatime,discard,errors=remount-ro 0 0
              ^^^^^^^
Quoted 8 lines
Quoted 8 lines

none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0 /dev/sda6 /home ext4 rw,noatime,discard 0 0

                                   ^^^^^^^

I'd say that's your problem....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#16 lore raw
Quoted 3 lines · Dave Chinner
Quoted 1 line
Quoted 1 line

/dev/sda6 /home ext4 rw,noatime,discard 0 0

                                   ^^^^^^^

I'd say that's your problem....

Looks like the Sandisk U100 is a good SSD for me to put on my personal "avoid" list:

http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/

There are a number of SSD's which do not implement "trim" efficiently, so these days, the recommended way to use trim is to run the "fstrim" command out of crontab.

There are some high performance flash devices (especially PCIe attached flash devices, where the TRIM command doesn't necessarily mean waiting for the entire contents of the Native Command Queue to drain) where using the discard mount option makes sense for best performance, but for most SATA drives (especially the really cheap-sh*t ones), I don't recommend it. If it weren't for the fact that for these devices exist and the discard option is especially useful for them, I probably would have removed the discard option from ext4.

- Ted

#33 lore raw
In reply to Marcus Sundman
Quoted 24 lines · Marcus Sundman

Sure:

Quoted 14 lines
$ cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=1964816k,nr_inodes=491204,mode=755 0 0
devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,relatime,size=789652k,mode=755 0 0
/dev/disk/by-uuid/5bfa7a58-2d35-4758-954e-4deafb09b892 / ext4
rw,noatime,discard,errors=remount-ro 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
Quoted 8 lines
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs
rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
/dev/sda6 /home ext4 rw,noatime,discard 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc
rw,nosuid,nodev,noexec,relatime 0 0
gvfsd-fuse /run/user/marcus/gvfs fuse.gvfsd-fuse
rw,nosuid,nodev,relatime,user_id=1000,group_id=100 0 0
Quoted 3 lines

Both / and /home are on the same SSD and suffer from the same problem. (I think the swap does as well, but I have my swappiness set very low to minimize swapping.)

Yeah, my suspicion is confirmed. Try removing the 'discard' option. See
Ted's email for more details.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#17 lore raw
In reply to Theodore Ts'o
Quoted 4 lines · Theodore Ts'o
Quoted 3 lines · Dave Chinner
Quoted 1 line
Quoted 1 line

/dev/sda6 /home ext4 rw,noatime,discard 0 0

                                    ^^^^^^^

I'd say that's your problem....

Quoted 8 lines

Looks like the Sandisk U100 is a good SSD for me to put on my personal "avoid" list:

http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/

There are a number of SSD's which do not implement "trim" efficiently, so these days, the recommended way to use trim is to run the "fstrim" command out of crontab.

OK. Removing 'discard' made it much better (the 60-600 second freezes are now 1-50 second freezes), but it's still at least an order of magnitude worse than a normal HD. When writing, that is -- reading is very fast (when there's no writing going on).

So, after reading up a bit on this trimming I'm thinking maybe my filesystem's block sizes don't match up with my SSD's blocks (or whatever its write unit is called). Then writing a FS block would always write to multiple SSD blocks, causing multiple read-erase-write sequences, right? So how can I check this, and how can I make the FS blocks match the SSD blocks?

Best regards,
Marcus

#18 lore raw
Quoted 7 lines · Marcus Sundman

So, after reading up a bit on this trimming I'm thinking maybe my filesystem's block sizes don't match up with my SSD's blocks (or whatever its write unit is called). Then writing a FS block would always write to multiple SSD blocks, causing multiple read-erase-write sequences, right? So how can I check this, and how can I make the FS blocks match the SSD blocks?

The erase block size for SSD's is typically in the area of 2MB (that's megabytes), with a page size of typically 4k, 8k, or 16k. So that means that erases take place with a granularity of 2 megabytes, and writes take place in chunks of the page size. It's up to the Flash Translation Layer to take the writes and map them to the NAND flash in an efficient way. This is the difference between high quality SSD's and really crappy SSD's. One of the best ways of measuring how good your SSD is to do random 4k write test, and see how well it handles a random write workload. I'm guessing you have a SSD which is really terrible at this.

In general you don't need to worry about alignment for most SSD's (eMMC/SD devices are a different story) since historically, Windows systems had partition offset by 63 (512 byte) sectors, which is the worst possible alignment. So SSD's in general can handle misaligned writes without any problems, or otherwise on Windows XP systems, their performance would be really crappy. SD card or eMMC devices don't deal with this well, so you need to worry about aligning your partitions appropriately. If your SSD is sensitive to partition alrignment, then it truly is a really crappy SSD. My suggestion to you is that the next time you buy an SSD, take a look at the reviews at web sites such as Anandtech, and in particular take a look at the 4k random write benchmark numbers and see how they compare with the competition.

As far as what you should do with your current SSD, if it's really that bad, I'm not sure I'd trust precious data on it, and I'd seriously consider simply getting a new SSD if budget allows this. Intel has historically does a really good job with their QA. They spent several months qual'ing the Sandforce controller, and so they were late to the market as a result, and their SSDs are generally a bit more expensive. However, the agreement they signed with Sandforce meant that the reliability/performance fixes in the Sandforce firmware which were the result of Intel's extended QA period would remain exclusive to Intel SSD's for some period of time and hence wouldn't be available to their competition. Guess which manufacturer's SSDs I generally tend to buy? :-)

Regards,

- Ted

#19 lore raw
In reply to Marcus Sundman
Quoted 19 lines · Marcus Sundman
Quoted 4 lines · Theodore Ts'o
Quoted 3 lines · Dave Chinner
Quoted 1 line
Quoted 1 line

/dev/sda6 /home ext4 rw,noatime,discard 0 0

                                   ^^^^^^^

I'd say that's your problem....

Quoted 8 lines

Looks like the Sandisk U100 is a good SSD for me to put on my personal "avoid" list:

http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/

There are a number of SSD's which do not implement "trim" efficiently, so these days, the recommended way to use trim is to run the "fstrim" command out of crontab.

OK. Removing 'discard' made it much better (the 60-600 second freezes are now 1-50 second freezes), but it's still at least an order of magnitude worse than a normal HD. When writing, that is -- reading is very fast (when there's no writing going on).

Quoted 6 lines

So, after reading up a bit on this trimming I'm thinking maybe my filesystem's block sizes don't match up with my SSD's blocks (or whatever its write unit is called). Then writing a FS block would always write to multiple SSD blocks, causing multiple read-erase-write sequences, right? So how can I check this, and how can I make the FS blocks match the SSD blocks?

As Ted wrote, alignment isn't usually a problem with SSDs. And even if it
was, it would be at most a factor 2 slow down and we don't seem to be at that fine grained level :)

At this point you might try mounting the fs with nobarrier mount option (I know you tried that before but without discard the difference could be more visible), switching IO scheduler to CFQ (for crappy SSDs it actually isn't a bad choice), and we'll see how much we can squeeze out of your drive...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#21 lore raw
In reply to Marcus Sundman
Quoted 46 lines · Marcus Sundman
Quoted 25 lines · Jan Kara
Quoted 18 lines · Marcus Sundman
Quoted 4 lines · Theodore Ts'o
Quoted 3 lines · Dave Chinner
Quoted 1 line
Quoted 1 line

/dev/sda6 /home ext4 rw,noatime,discard 0 0

                                   ^^^^^^^

I'd say that's your problem....

Quoted 8 lines

Looks like the Sandisk U100 is a good SSD for me to put on my personal "avoid" list:

http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/

There are a number of SSD's which do not implement "trim" efficiently, so these days, the recommended way to use trim is to run the "fstrim" command out of crontab.

OK. Removing 'discard' made it much better (the 60-600 second freezes are now 1-50 second freezes), but it's still at least an order of magnitude worse than a normal HD. When writing, that is -- reading is very fast (when there's no writing going on).

Quoted 6 lines

So, after reading up a bit on this trimming I'm thinking maybe my filesystem's block sizes don't match up with my SSD's blocks (or whatever its write unit is called). Then writing a FS block would always write to multiple SSD blocks, causing multiple read-erase-write sequences, right? So how can I check this, and how can I make the FS blocks match the SSD blocks?

Quoted 8 lines

As Ted wrote, alignment isn't usually a problem with SSDs. And even if it
was, it would be at most a factor 2 slow down and we don't seem to be at that fine grained level :)

At this point you might try mounting the fs with nobarrier mount option (I know you tried that before but without discard the difference could be more visible), switching IO scheduler to CFQ (for crappy SSDs it actually isn't a bad choice), and we'll see how much we can squeeze out of your drive...

I repartitioned the drive and reinstalled ubuntu and after that it gladly wrote over 100 MB/s to the SSD without any hangs. However, after a couple of months I noticed it had degraded considerably, and it keeps degrading. Now it's slowly becoming completely unusable again, with write speeds of the magnitude 1 MB/s and dropping.

As far as I can tell I have not made any relevant changes. Also, the amount of free space hasn't changed considerably, but it seems that the longer it's been since I reformatted the drive the more free space is required for it to perform well.

Quoted 6 lines

So, maybe the cause is fragmentation? I tried running e4defrag and then fstrim, but it didn't really help (well, maybe a little bit, but after a couple of days it was back in unusable-land). Also, "e4defrag -c" gives a fragmenation score of less than 5, so...

Any ideas?

So now you run without 'discard' mount option, right? My guess then would
be that the FTL layer on your SSD is just crappy and as the erase blocks get more fragmented as the filesystem is used it cannot keep up. But it's easy to put blame on someone else :)

You can check whether this is a problem of Linux or your SSD by writing a large file (few GB or more) like 'dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct'. What is the throughput? If it is bad, check output of 'filefrag -v testfile'. If the extents are reasonably large (1 MB and more), then the problem is in your SSD firmware. Not much we can do about it in that case...

If it really is SSD's firmware, maybe you could try f2fs or similar flash oriented filesystem which should put lower load on the disk's FTL.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#20 lore raw
In reply to Jan Kara
Quoted 25 lines · Jan Kara
Quoted 18 lines · Marcus Sundman
Quoted 4 lines · Theodore Ts'o
Quoted 3 lines · Dave Chinner
Quoted 1 line
Quoted 1 line

/dev/sda6 /home ext4 rw,noatime,discard 0 0

                                    ^^^^^^^

I'd say that's your problem....

Quoted 8 lines

Looks like the Sandisk U100 is a good SSD for me to put on my personal "avoid" list:

http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/

There are a number of SSD's which do not implement "trim" efficiently, so these days, the recommended way to use trim is to run the "fstrim" command out of crontab.

OK. Removing 'discard' made it much better (the 60-600 second freezes are now 1-50 second freezes), but it's still at least an order of magnitude worse than a normal HD. When writing, that is -- reading is very fast (when there's no writing going on).

Quoted 6 lines

So, after reading up a bit on this trimming I'm thinking maybe my filesystem's block sizes don't match up with my SSD's blocks (or whatever its write unit is called). Then writing a FS block would always write to multiple SSD blocks, causing multiple read-erase-write sequences, right? So how can I check this, and how can I make the FS blocks match the SSD blocks?

Quoted 8 lines

As Ted wrote, alignment isn't usually a problem with SSDs. And even if it
was, it would be at most a factor 2 slow down and we don't seem to be at that fine grained level :)

At this point you might try mounting the fs with nobarrier mount option (I know you tried that before but without discard the difference could be more visible), switching IO scheduler to CFQ (for crappy SSDs it actually isn't a bad choice), and we'll see how much we can squeeze out of your drive...

I repartitioned the drive and reinstalled ubuntu and after that it gladly wrote over 100 MB/s to the SSD without any hangs. However, after a couple of months I noticed it had degraded considerably, and it keeps degrading. Now it's slowly becoming completely unusable again, with write speeds of the magnitude 1 MB/s and dropping.

As far as I can tell I have not made any relevant changes. Also, the amount of free space hasn't changed considerably, but it seems that the longer it's been since I reformatted the drive the more free space is required for it to perform well.

So, maybe the cause is fragmentation? I tried running e4defrag and then fstrim, but it didn't really help (well, maybe a little bit, but after a couple of days it was back in unusable-land). Also, "e4defrag -c" gives a fragmenation score of less than 5, so...

Any ideas?

Best regards,
Marcus

#22 lore raw
In reply to Jan Kara
Quoted 64 lines · Jan Kara
Quoted 45 lines · Marcus Sundman
Quoted 25 lines · Jan Kara
Quoted 18 lines · Marcus Sundman
Quoted 4 lines · Theodore Ts'o
Quoted 3 lines · Dave Chinner
Quoted 1 line
Quoted 1 line

/dev/sda6 /home ext4 rw,noatime,discard 0 0

                                    ^^^^^^^

I'd say that's your problem....

Quoted 8 lines

Looks like the Sandisk U100 is a good SSD for me to put on my personal "avoid" list:

http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/

There are a number of SSD's which do not implement "trim" efficiently, so these days, the recommended way to use trim is to run the "fstrim" command out of crontab.

OK. Removing 'discard' made it much better (the 60-600 second freezes are now 1-50 second freezes), but it's still at least an order of magnitude worse than a normal HD. When writing, that is -- reading is very fast (when there's no writing going on).

Quoted 6 lines

So, after reading up a bit on this trimming I'm thinking maybe my filesystem's block sizes don't match up with my SSD's blocks (or whatever its write unit is called). Then writing a FS block would always write to multiple SSD blocks, causing multiple read-erase-write sequences, right? So how can I check this, and how can I make the FS blocks match the SSD blocks?

Quoted 8 lines

As Ted wrote, alignment isn't usually a problem with SSDs. And even if it
was, it would be at most a factor 2 slow down and we don't seem to be at that fine grained level :)

At this point you might try mounting the fs with nobarrier mount option (I know you tried that before but without discard the difference could be more visible), switching IO scheduler to CFQ (for crappy SSDs it actually isn't a bad choice), and we'll see how much we can squeeze out of your drive...

I repartitioned the drive and reinstalled ubuntu and after that it gladly wrote over 100 MB/s to the SSD without any hangs. However, after a couple of months I noticed it had degraded considerably, and it keeps degrading. Now it's slowly becoming completely unusable again, with write speeds of the magnitude 1 MB/s and dropping.

As far as I can tell I have not made any relevant changes. Also, the amount of free space hasn't changed considerably, but it seems that the longer it's been since I reformatted the drive the more free space is required for it to perform well.

Quoted 6 lines

So, maybe the cause is fragmentation? I tried running e4defrag and then fstrim, but it didn't really help (well, maybe a little bit, but after a couple of days it was back in unusable-land). Also, "e4defrag -c" gives a fragmenation score of less than 5, so...

Any ideas?

So now you run without 'discard' mount option, right? My guess then would
be that the FTL layer on your SSD is just crappy and as the erase blocks get more fragmented as the filesystem is used it cannot keep up. But it's easy to put blame on someone else :)

You can check whether this is a problem of Linux or your SSD by writing a large file (few GB or more) like 'dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct'. What is the throughput? If it is bad, check output of 'filefrag -v testfile'. If the extents are reasonably large (1 MB and more), then the problem is in your SSD firmware. Not much we can do about it in that case...

Quoted 2 lines

If it really is SSD's firmware, maybe you could try f2fs or similar flash oriented filesystem which should put lower load on the disk's FTL.

----8<---------------------------

$ grep LABEL /etc/fstab
LABEL=system    /        ext4    errors=remount-ro,nobarrier,noatime 0 1
LABEL=home    /home        ext4    defaults,nobarrier,noatime 0    2
$ df -h|grep home
/dev/sda3       104G   98G  5.1G  96% /home
$ sync && time dd if=/dev/zero of=testfile bs=1M count=2048 oflag=direct && time sync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 404.571 s, 5.3 MB/s

real 6m44.575s
user 0m0.000s
sys 0m1.300s

real 0m0.111s
user 0m0.000s
sys 0m0.004s

$ filefrag -v testfile
Filesystem type is: ef53
File size of testfile is 2147483648 (524288 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0 21339392             512
 [... http://sundman.iki.fi/extents.txt ...]
 282  523520  1618176  1568000    768 eof
testfile: 282 extents found
$
----8<---------------------------

Many extents are around 400 blocks(?) -- is this good or bad? (This partition has a fragmentation score of 0 according to e4defrag.)

Regards,
Marcus

#23 lore raw
Quoted 98 lines · Marcus Sundman
Quoted 64 lines · Jan Kara
Quoted 45 lines · Marcus Sundman
Quoted 25 lines · Jan Kara
Quoted 18 lines · Marcus Sundman
Quoted 4 lines · Theodore Ts'o
Quoted 3 lines · Dave Chinner
Quoted 1 line
Quoted 1 line

/dev/sda6 /home ext4 rw,noatime,discard 0 0

                                   ^^^^^^^

I'd say that's your problem....

Quoted 8 lines

Looks like the Sandisk U100 is a good SSD for me to put on my personal "avoid" list:

http://thessdreview.com/our-reviews/asus-zenbook-ssd-review-not-necessarily-sandforce-driven-shows-significant-speed-bump/

There are a number of SSD's which do not implement "trim" efficiently, so these days, the recommended way to use trim is to run the "fstrim" command out of crontab.

OK. Removing 'discard' made it much better (the 60-600 second freezes are now 1-50 second freezes), but it's still at least an order of magnitude worse than a normal HD. When writing, that is -- reading is very fast (when there's no writing going on).

Quoted 6 lines

So, after reading up a bit on this trimming I'm thinking maybe my filesystem's block sizes don't match up with my SSD's blocks (or whatever its write unit is called). Then writing a FS block would always write to multiple SSD blocks, causing multiple read-erase-write sequences, right? So how can I check this, and how can I make the FS blocks match the SSD blocks?

Quoted 8 lines

As Ted wrote, alignment isn't usually a problem with SSDs. And even if it
was, it would be at most a factor 2 slow down and we don't seem to be at that fine grained level :)

At this point you might try mounting the fs with nobarrier mount option (I know you tried that before but without discard the difference could be more visible), switching IO scheduler to CFQ (for crappy SSDs it actually isn't a bad choice), and we'll see how much we can squeeze out of your drive...

I repartitioned the drive and reinstalled ubuntu and after that it gladly wrote over 100 MB/s to the SSD without any hangs. However, after a couple of months I noticed it had degraded considerably, and it keeps degrading. Now it's slowly becoming completely unusable again, with write speeds of the magnitude 1 MB/s and dropping.

As far as I can tell I have not made any relevant changes. Also, the amount of free space hasn't changed considerably, but it seems that the longer it's been since I reformatted the drive the more free space is required for it to perform well.

Quoted 6 lines

So, maybe the cause is fragmentation? I tried running e4defrag and then fstrim, but it didn't really help (well, maybe a little bit, but after a couple of days it was back in unusable-land). Also, "e4defrag -c" gives a fragmenation score of less than 5, so...

Any ideas?

So now you run without 'discard' mount option, right? My guess then would
be that the FTL layer on your SSD is just crappy and as the erase blocks get more fragmented as the filesystem is used it cannot keep up. But it's easy to put blame on someone else :)

You can check whether this is a problem of Linux or your SSD by writing a large file (few GB or more) like 'dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct'. What is the throughput? If it is bad, check output of 'filefrag -v testfile'. If the extents are reasonably large (1 MB and more), then the problem is in your SSD firmware. Not much we can do about it in that case...

Quoted 2 lines

If it really is SSD's firmware, maybe you could try f2fs or similar flash oriented filesystem which should put lower load on the disk's FTL.

----8<---------------------------

$ grep LABEL /etc/fstab
LABEL=system    /        ext4    errors=remount-ro,nobarrier,noatime 0 1
LABEL=home    /home        ext4    defaults,nobarrier,noatime 0    2
$ df -h|grep home
/dev/sda3       104G   98G  5.1G  96% /home
$ sync && time dd if=/dev/zero of=testfile bs=1M count=2048
oflag=direct && time sync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 404.571 s, 5.3 MB/s

real 6m44.575s
user 0m0.000s
sys 0m1.300s

real 0m0.111s
user 0m0.000s
sys 0m0.004s

$ filefrag -v testfile
Filesystem type is: ef53
File size of testfile is 2147483648 (524288 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0 21339392             512
 [... http://sundman.iki.fi/extents.txt ...]
 282  523520  1618176  1568000    768 eof
testfile: 282 extents found
$
----8<---------------------------
Quoted 2 lines

Many extents are around 400 blocks(?) -- is this good or bad? (This partition has a fragmentation score of 0 according to e4defrag.)

The free space is somewhat fragmented but given how full the fs is this
is understandable. The extents are large enough that the drive shouldn't have problems processing them better than at 5 MB/s (standard rotating disk would achieve much better throughput with this layout I believe). So my conclusion is that really FTL on your drive sucks (or possibly the drive doesn't have enough "hidden" additional space to ease the load on FTL when the disk gets full).

And with this full filesystem fstrim isn't going to help you because we can trim only free blocks and there aren't that many of those. Sorry.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#24 lore raw
Quoted 56 lines · Jan Kara
Quoted 45 lines · Marcus Sundman
Quoted 12 lines · Jan Kara

So now you run without 'discard' mount option, right? My guess then would
be that the FTL layer on your SSD is just crappy and as the erase blocks get more fragmented as the filesystem is used it cannot keep up. But it's easy to put blame on someone else :)

You can check whether this is a problem of Linux or your SSD by writing a large file (few GB or more) like 'dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct'. What is the throughput? If it is bad, check output of 'filefrag -v testfile'. If the extents are reasonably large (1 MB and more), then the problem is in your SSD firmware. Not much we can do about it in that case...

Quoted 2 lines

If it really is SSD's firmware, maybe you could try f2fs or similar flash oriented filesystem which should put lower load on the disk's FTL.

----8<---------------------------

$ grep LABEL /etc/fstab
LABEL=system    /        ext4    errors=remount-ro,nobarrier,noatime 0 1
LABEL=home    /home        ext4    defaults,nobarrier,noatime 0    2
$ df -h|grep home
/dev/sda3       104G   98G  5.1G  96% /home
$ sync && time dd if=/dev/zero of=testfile bs=1M count=2048
oflag=direct && time sync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 404.571 s, 5.3 MB/s

real 6m44.575s
user 0m0.000s
sys 0m1.300s

real 0m0.111s
user 0m0.000s
sys 0m0.004s

$ filefrag -v testfile
Filesystem type is: ef53
File size of testfile is 2147483648 (524288 blocks, blocksize 4096)
  ext logical physical expected length flags
    0       0 21339392             512
  [... http://sundman.iki.fi/extents.txt ...]
  282  523520  1618176  1568000    768 eof
testfile: 282 extents found
$
----8<---------------------------
Quoted 2 lines

Many extents are around 400 blocks(?) -- is this good or bad? (This partition has a fragmentation score of 0 according to e4defrag.)

The free space is somewhat fragmented but given how full the fs is this
is understandable. The extents are large enough that the drive shouldn't have problems processing them better than at 5 MB/s (standard rotating disk would achieve much better throughput with this layout I believe). So my conclusion is that really FTL on your drive sucks (or possibly the drive doesn't have enough "hidden" additional space to ease the load on FTL when the disk gets full).

Quoted 2 lines

And with this full filesystem fstrim isn't going to help you because we can trim only free blocks and there aren't that many of those. Sorry.

OK, but why does it become worse over time?
And can I somehow "reset" whatever it is that is making it worse so that it becomes good again? That way I could spend maybe 1 hour once every few months to get it back to top speed. Any other ideas how I could make this (very expensive and fairly new ZenBook) laptop usable?
Also, why doesn't this happen with USB memory sticks?

And many thanks for all your help with this issue! And thanks also to Sprouse and Ts'o!

Best regards,
Marcus

#25 lore raw
Quoted 57 lines · Marcus Sundman
Quoted 56 lines · Jan Kara
Quoted 45 lines · Marcus Sundman
Quoted 12 lines · Jan Kara

So now you run without 'discard' mount option, right? My guess then would
be that the FTL layer on your SSD is just crappy and as the erase blocks get more fragmented as the filesystem is used it cannot keep up. But it's easy to put blame on someone else :)

You can check whether this is a problem of Linux or your SSD by writing a large file (few GB or more) like 'dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct'. What is the throughput? If it is bad, check output of 'filefrag -v testfile'. If the extents are reasonably large (1 MB and more), then the problem is in your SSD firmware. Not much we can do about it in that case...

Quoted 2 lines

If it really is SSD's firmware, maybe you could try f2fs or similar flash oriented filesystem which should put lower load on the disk's FTL.

----8<---------------------------

$ grep LABEL /etc/fstab
LABEL=system    /        ext4    errors=remount-ro,nobarrier,noatime 0 1
LABEL=home    /home        ext4    defaults,nobarrier,noatime 0    2
$ df -h|grep home
/dev/sda3       104G   98G  5.1G  96% /home
$ sync && time dd if=/dev/zero of=testfile bs=1M count=2048
oflag=direct && time sync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 404.571 s, 5.3 MB/s

real 6m44.575s
user 0m0.000s
sys 0m1.300s

real 0m0.111s
user 0m0.000s
sys 0m0.004s

$ filefrag -v testfile
Filesystem type is: ef53
File size of testfile is 2147483648 (524288 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0 21339392             512
 [... http://sundman.iki.fi/extents.txt ...]
 282  523520  1618176  1568000    768 eof
testfile: 282 extents found
$
----8<---------------------------
Quoted 2 lines

Many extents are around 400 blocks(?) -- is this good or bad? (This partition has a fragmentation score of 0 according to e4defrag.)

The free space is somewhat fragmented but given how full the fs is this
is understandable. The extents are large enough that the drive shouldn't have problems processing them better than at 5 MB/s (standard rotating disk would achieve much better throughput with this layout I believe). So my conclusion is that really FTL on your drive sucks (or possibly the drive doesn't have enough "hidden" additional space to ease the load on FTL when the disk gets full).

Quoted 4 lines
Quoted 2 lines

And with this full filesystem fstrim isn't going to help you because we can trim only free blocks and there aren't that many of those. Sorry.

OK, but why does it become worse over time?

So my theory is the following. Initially we begin with empty disk and the
firmware knows the disk is empty because mkfs.ext4 discards the whole disk before creating the filesystem. Thus FTL has relatively easy work when we write a block because it has a plenty of unused erase blocks where block can be stored. As the time passes and disk gets written, erase blocks get more fragmented. After some time (especially when the disk is almost full), each erase block has most of the blocks used and a couple of free blocks. Thus when we write new block, FTL has to do a full read-modify-write cycle of the whole erase block to write a single block.

Good SSDs have quite a bit of additional space over the declared size (I've heard upto 50%) to make the erase block fragmentation problem (and also lifetime of NAND flash) easier. Also the FTL can be more or less smart regarding how to avoid fragmentation of erase blocks.

Quoted 5 lines

And can I somehow "reset" whatever it is that is making it worse so that it becomes good again? That way I could spend maybe 1 hour once every few months to get it back to top speed.
Any other ideas how I could make this (very expensive and fairly new ZenBook) laptop usable?

Well, I believe if you used like 70% or less of the disk and regularly
(like once in a few days) run fstrim command, I belive the disk performance should stay at a usable level.

Quoted 1 line

Also, why doesn't this happen with USB memory sticks?

It does happen. Try running a distro from a USB stick. It is pretty slow.
Why you don't observe problems with USB sticks is that you don't use them the way you use your / or /home. Usually you just write a big chunk of data to the USB stick, it stays there for a while and then you delete it. This is much easier on the FTL because all blocks in an erase block tend to have the same lifetime and thus in most cases either the whole erase block is used or free.

Quoted 2 lines

And many thanks for all your help with this issue! And thanks also to Sprouse and Ts'o!

You are welcome.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#26 lore raw
Quoted 9 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

And can I somehow "reset" whatever it is that is making it worse so that it becomes good again? That way I could spend maybe 1 hour once every few months to get it back to top speed.
Any other ideas how I could make this (very expensive and fairly new ZenBook) laptop usable?

Well, I believe if you used like 70% or less of the disk and regularly
(like once in a few days) run fstrim command, I belive the disk performance should stay at a usable level.

At 128 GB it is extremely small as it is, and I'm really struggling to fit all on it. Most of my stuff is on my NAS (which has almost 10 TB space), but still I need several code repositories and the development environment and a virtual machine etc on this tiny 128 GB thing.

So, if I used some other filesystem, might that allow me to use a larger portion of the SSD without this degradation? Or with a much slower rate of degradation?

And at some point it will become unusable again, so what can I do then? If I move everything to my NAS (and maybe even re-create the filesystem?) and move everything back, might that get rid of the FTL fragmentation? Or could I somehow defragment the FTL without moving away everything?

Regards,
Marcus

#27 lore raw
Quoted 11 lines · Marcus Sundman
Quoted 9 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

And can I somehow "reset" whatever it is that is making it worse so that it becomes good again? That way I could spend maybe 1 hour once every few months to get it back to top speed.
Any other ideas how I could make this (very expensive and fairly new ZenBook) laptop usable?

Well, I believe if you used like 70% or less of the disk and regularly
(like once in a few days) run fstrim command, I belive the disk performance should stay at a usable level.

Quoted 5 lines

At 128 GB it is extremely small as it is, and I'm really struggling to fit all on it. Most of my stuff is on my NAS (which has almost 10 TB space), but still I need several code repositories and the development environment and a virtual machine etc on this tiny 128 GB thing.

I see. I have like 70 GB disk and 50% of it are free :) But I have test
machines with much larger drives where I have VMs etc. This one is just for email and coding.

Quoted 3 lines

So, if I used some other filesystem, might that allow me to use a larger portion of the SSD without this degradation? Or with a much slower rate of degradation?

You might try f2fs. That is designed for low end flash storage so it
might work better than ext4. But it is a new filesystem so backup often.

Quoted 4 lines

And at some point it will become unusable again, so what can I do then? If I move everything to my NAS (and maybe even re-create the filesystem?) and move everything back, might that get rid of the FTL fragmentation?

Yes, that should get rid of it. But since you have only a few GB free,
I'm afraid the fragmentation will reappear pretty quickly. But I guess it's worth a try.

Quoted 1 line

Or could I somehow defragment the FTL without moving away everything?

I don't know about such way.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#28 lore raw
Quoted 30 lines · Jan Kara
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

And can I somehow "reset" whatever it is that is making it worse so that it becomes good again? That way I could spend maybe 1 hour once every few months to get it back to top speed.
Any other ideas how I could make this (very expensive and fairly new ZenBook) laptop usable?

Quoted 8 lines
Quoted 3 lines

Well, I believe if you used like 70% or less of the disk and regularly
(like once in a few days) run fstrim command, I belive the disk performance should stay at a usable level.

At 128 GB it is extremely small as it is, and I'm really struggling to fit all on it. Most of my stuff is on my NAS (which has almost 10 TB space), but still I need several code repositories and the development environment and a virtual machine etc on this tiny 128 GB thing.

I see. I have like 70 GB disk and 50% of it are free :) But I have test
machines with much larger drives where I have VMs etc. This one is just for email and coding.

Quoted 3 lines

So, if I used some other filesystem, might that allow me to use a larger portion of the SSD without this degradation? Or with a much slower rate of degradation?

You might try f2fs. That is designed for low end flash storage so it
might work better than ext4. But it is a new filesystem so backup often.

Quoted 4 lines

And at some point it will become unusable again, so what can I do then? If I move everything to my NAS (and maybe even re-create the filesystem?) and move everything back, might that get rid of the FTL fragmentation?

Quoted 6 lines

Yes, that should get rid of it. But since you have only a few GB free,
I'm afraid the fragmentation will reappear pretty quickly. But I guess it's worth a try.

Quoted 1 line

Or could I somehow defragment the FTL without moving away everything?

I don't know about such way.

How about triggering the garbage collection on the drive, is that possible?

- Marcus

#29 lore raw
Quoted 31 lines · Marcus Sundman
Quoted 30 lines · Jan Kara
Quoted 7 lines · Marcus Sundman
Quoted 6 lines · Jan Kara
Quoted 5 lines · Marcus Sundman

And can I somehow "reset" whatever it is that is making it worse so that it becomes good again? That way I could spend maybe 1 hour once every few months to get it back to top speed.
Any other ideas how I could make this (very expensive and fairly new ZenBook) laptop usable?

Quoted 8 lines
Quoted 3 lines

Well, I believe if you used like 70% or less of the disk and regularly
(like once in a few days) run fstrim command, I belive the disk performance should stay at a usable level.

At 128 GB it is extremely small as it is, and I'm really struggling to fit all on it. Most of my stuff is on my NAS (which has almost 10 TB space), but still I need several code repositories and the development environment and a virtual machine etc on this tiny 128 GB thing.

I see. I have like 70 GB disk and 50% of it are free :) But I have test
machines with much larger drives where I have VMs etc. This one is just for email and coding.

Quoted 3 lines

So, if I used some other filesystem, might that allow me to use a larger portion of the SSD without this degradation? Or with a much slower rate of degradation?

You might try f2fs. That is designed for low end flash storage so it
might work better than ext4. But it is a new filesystem so backup often.

Quoted 4 lines

And at some point it will become unusable again, so what can I do then? If I move everything to my NAS (and maybe even re-create the filesystem?) and move everything back, might that get rid of the FTL fragmentation?

Quoted 8 lines
Quoted 6 lines

Yes, that should get rid of it. But since you have only a few GB free,
I'm afraid the fragmentation will reappear pretty quickly. But I guess it's worth a try.

Quoted 1 line

Or could I somehow defragment the FTL without moving away everything?

I don't know about such way.

How about triggering the garbage collection on the drive, is that possible?

No, I don't know about any way to do that.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#30 lore raw
In reply to Marcus Sundman
Quoted 10 lines · Marcus Sundman

At 128 GB it is extremely small as it is, and I'm really struggling to fit all on it. Most of my stuff is on my NAS (which has almost 10 TB space), but still I need several code repositories and the development environment and a virtual machine etc on this tiny 128 GB thing.

So, if I used some other filesystem, might that allow me to use a larger portion of the SSD without this degradation? Or with a much slower rate of degradation?

What model are you using? It's possible that your flash device was designed as a cache driver for windows. As such, it might have been optimized for a read-mostly workload and not something for a lot of random small writes.

The f2fs file system is designed for crappy flash drives with crappy FTL's, so it might work be better for you. But let me ask you this --- how much is your data worth? How much would it cost to replace your flash device with something better?

I tend to get very nervous with crappy storage devices, and it sounds like your flash drive isn't a particularly good one. I'd strongly suggest doing regular backups, because when flash devices die, they can die in extremely catastrphic ways.

Regards,

- Ted

#31 lore raw
Quoted 7 lines · Theodore Ts'o
Quoted 6 lines · Marcus Sundman

At 128 GB it is extremely small as it is, and I'm really struggling to fit all on it. Most of my stuff is on my NAS (which has almost 10 TB space), but still I need several code repositories and the development environment and a virtual machine etc on this tiny 128 GB thing.

Quoted 7 lines
Quoted 3 lines

So, if I used some other filesystem, might that allow me to use a larger portion of the SSD without this degradation? Or with a much slower rate of degradation?

What model are you using? It's possible that your flash device was designed as a cache driver for windows. As such, it might have been optimized for a read-mostly workload and not something for a lot of random small writes.

It's a SanDisk SSD U100.

Quoted 2 lines

The f2fs file system is designed for crappy flash drives with crappy FTL's, so it might work be better for you.

OK, I'll probably try it when I have time to switch.

Quoted 3 lines

But let me ask you this
--- how much is your data worth? How much would it cost to replace your flash device with something better?

A lot. I have lsyncd running here most of the time, backing up to my raid-z NAS which in turn uses a versioned off-site backup system. Anyway, this is an Asus ZenBook computer and can't be opened. Well, it can, but that will void the warranty at the very least.

- Marcus

#32 lore raw
Quoted 2 lines · Marcus Sundman

It's a SanDisk SSD U100.

Here's one report of someone who had an experience with U100 in an Asus UX31E:

https://communities.intel.com/thread/32515

Quoted 4 lines

A lot. I have lsyncd running here most of the time, backing up to my raid-z NAS which in turn uses a versioned off-site backup system. Anyway, this is an Asus ZenBook computer and can't be opened. Well, it can, but that will void the warranty at the very least.

Personally, I avoid like the plague laptop computers where I can't replace the storage devices (or upgrade memory, etc.) without voiding the warrantee....

I generally order laptops without the flash device, and carefully select for high quality flash devices. Otherwise, you end up paying too much for what is generally shoddy equipment....

- Ted