Chronic resource starvation.
I've dealt with applications taking extended time off for a number of years. I typically attribute it to applications being overly zealous about eating memory as most every application tends to do these days. I had always figured that there a likely plenty of ppl complaining and I didn't want to get the boiler plat answer that resources are cheap.
I refuse to buy into the idea that Z computers can get an additional Y resource to run application X, when instead application X could be engineered once and for all. This ideology is not sustainable and eventually will crash upon it's self. I call this Z * Y < X. The application source becomes the single location where every computers resources can be increased at the cost of much less then to adjust the running environment of every location that the code may run.
Here is a 84MB video that demonstrates the issue.
http://j.mp/wavbCO
http://bitly.com/wavbCO+
You can see at the start an application that should update regularly is frozen and that moving windows and even focus is difficult. The system does recover, but after far too long.
I'll welcome any further testing as I'm sure it's needed. Keep in mind that I can't reproduce this reliably, but it does happen, so it will take time to collect any amount of data... A gkrellm plugin that illustrates the test is the most perfered, killing two birds with one stone as it will help with this case and many many future cases. If it's important wouldn't your test already be part of a tool like gkrellm?
Quoted 14 lines · Mike Mestnik
I've dealt with applications taking extended time off for a number of years. I typically attribute it to applications being overly zealous about eating memory as most every application tends to do these days. I had always figured that there a likely plenty of ppl complaining and I didn't want to get the boiler plat answer that resources are cheap.
I refuse to buy into the idea that Z computers can get an additional Y resource to run application X, when instead application X could be engineered once and for all. This ideology is not sustainable and eventually will crash upon it's self. I call this Z * Y < X. The application source becomes the single location where every computers resources can be increased at the cost of much less then to adjust the running environment of every location that the code may run.
Quoted 3 lines
Here is a 84MB video that demonstrates the issue.
http://j.mp/wavbCO
http://bitly.com/wavbCO+
I'm glad to see a number of you have clicked on this link.
Does this behavior look normal or is it just my system? If it is normal how difficult would it be to make corrections and would those corrections likely be kernel or application related?
Quoted 11 lines
You can see at the start an application that should update regularly is frozen and that moving windows and even focus is difficult. The system does recover, but after far too long.
I'll welcome any further testing as I'm sure it's needed. Keep in mind that I can't reproduce this reliably, but it does happen, so it will take time to collect any amount of data... A gkrellm plugin that illustrates the test is the most perfered, killing two birds with one stone as it will help with this case and many many future cases. If it's important wouldn't your test already be part of a tool like gkrellm?
Quoted 15 lines · Mike Mestnik
Quoted 14 lines · Mike Mestnik
I've dealt with applications taking extended time off for a number of years. I typically attribute it to applications being overly zealous about eating memory as most every application tends to do these days. I had always figured that there a likely plenty of ppl complaining and I didn't want to get the boiler plat answer that resources are cheap.
I refuse to buy into the idea that Z computers can get an additional Y resource to run application X, when instead application X could be engineered once and for all. This ideology is not sustainable and eventually will crash upon it's self. I call this Z * Y < X. The application source becomes the single location where every computers resources can be increased at the cost of much less then to adjust the running environment of every location that the code may run.
Quoted 8 lines
Quoted 3 lines
Here is a 84MB video that demonstrates the issue.
http://j.mp/wavbCO
http://bitly.com/wavbCO+I'm glad to see a number of you have clicked on this link.
Does this behavior look normal or is it just my system? If it is normal how difficult would it be to make corrections and would those corrections likely be kernel or application related?
That "Backup complete" makes me suspect a classic case of IO-itis. If bits of your GUI were pushed out or ram (or weren't previously used), and live on a disk you're beating hell out of, you get to experience horrid interactivity while those missing bits are being retrieved.
-Mike
Quoted 21 lines · Mike Galbraith
Quoted 20 lines · Mike Mestnik
Quoted 14 lines · Mike Mestnik
I've dealt with applications taking extended time off for a number of years. I typically attribute it to applications being overly zealous about eating memory as most every application tends to do these days. I had always figured that there a likely plenty of ppl complaining and I didn't want to get the boiler plat answer that resources are cheap.
I refuse to buy into the idea that Z computers can get an additional Y resource to run application X, when instead application X could be engineered once and for all. This ideology is not sustainable and eventually will crash upon it's self. I call this Z * Y < X. The application source becomes the single location where every computers resources can be increased at the cost of much less then to adjust the running environment of every location that the code may run.
Quoted 3 lines
Here is a 84MB video that demonstrates the issue.
http://j.mp/wavbCO
http://bitly.com/wavbCO+I'm glad to see a number of you have clicked on this link.
Quoted 7 lines
Quoted 3 lines
Does this behavior look normal or is it just my system? If it is normal how difficult would it be to make corrections and would those corrections likely be kernel or application related?
That "Backup complete" makes me suspect a classic case of IO-itis. If bits of your GUI were pushed out or ram (or weren't previously used), and live on a disk you're beating hell out of, you get to experience horrid interactivity while those missing bits are being retrieved.
Thank you for the reply!
That's DVDShrink, copies from optical drive(slow) to disk(fast-ish?).
http://www.dvdshrink.info/
That wouldn't explain why the monitor(gkrellm) stopped updating, every part of that is pooling like top would. Gkrellm indicates disk activity and it shows the backup complete on sr1 and a backup in progress on sr0, however neither of these operations rate on Disk or sda.
I wouldn't suspect that 16MB/s could saturate an HD disk, perhaps I can buffer this better?
Quoted 2 lines
-Mike
Quoted 11 lines · Mike Mestnik
Quoted 7 lines · Mike Galbraith
Quoted 3 lines
Does this behavior look normal or is it just my system? If it is normal how difficult would it be to make corrections and would those corrections likely be kernel or application related?
That "Backup complete" makes me suspect a classic case of IO-itis. If bits of your GUI were pushed out or ram (or weren't previously used), and live on a disk you're beating hell out of, you get to experience horrid interactivity while those missing bits are being retrieved.
Thank you for the reply!
Quoted 7 lines
That's DVDShrink, copies from optical drive(slow) to disk(fast-ish?).
http://www.dvdshrink.info/That wouldn't explain why the monitor(gkrellm) stopped updating, every part of that is pooling like top would. Gkrellm indicates disk activity and it shows the backup complete on sr1 and a backup in progress on sr0, however neither of these operations rate on Disk or sda.
No, dvdshrink wouldn't do enough IO to matter much. Heavy IO can (or could) cause symptoms like your video though, all of it. I used to be able to trigger loss of GUI control for 30 minutes at a time. All it took was something poking fdatasync after a large streaming IO had been running for a while.
Another IO thing that can cause horrible interactivity is IO error recovery. That can (or again, could) make a box lurch like hell. If that were the cause, you'd have kernel gripes in dmesg output.
-Mike
Quoted 18 lines · Mike Galbraith
Quoted 10 lines · Mike Mestnik
Quoted 7 lines · Mike Galbraith
Quoted 3 lines
Does this behavior look normal or is it just my system? If it is normal how difficult would it be to make corrections and would those corrections likely be kernel or application related?
That "Backup complete" makes me suspect a classic case of IO-itis. If bits of your GUI were pushed out or ram (or weren't previously used), and live on a disk you're beating hell out of, you get to experience horrid interactivity while those missing bits are being retrieved.
Thank you for the reply!
Quoted 7 lines
That's DVDShrink, copies from optical drive(slow) to disk(fast-ish?).
http://www.dvdshrink.info/That wouldn't explain why the monitor(gkrellm) stopped updating, every part of that is pooling like top would. Gkrellm indicates disk activity and it shows the backup complete on sr1 and a backup in progress on sr0, however neither of these operations rate on Disk or sda.
Quoted 5 lines
No, dvdshrink wouldn't do enough IO to matter much. Heavy IO can (or could) cause symptoms like your video though, all of it. I used to be able to trigger loss of GUI control for 30 minutes at a time. All it took was something poking fdatasync after a large streaming IO had been running for a while.
Ahhh, as I had suspected I'm not the only one.
Does fdatasync still cause this problem? I'm sure there must be some way to 'group' applications together that should be allowed to avoid this effect and even insert IO requests during the operation if given a high enough priority.
Quoted 4 lines
Another IO thing that can cause horrible interactivity is IO error recovery. That can (or again, could) make a box lurch like hell. If that were the cause, you'd have kernel gripes in dmesg output.
Sure, this happens as a 'feature' of commercial DVDs and some times on accident:
[5636213.987232] sr 4:0:0:0: [sr0] Device not ready [5636213.987235] sr 4:0:0:0: [sr0] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [5636213.987238] sr 4:0:0:0: [sr0] Sense Key : Not Ready [current] [5636213.987240] sr 4:0:0:0: [sr0] Add. Sense: Medium not present - tray closed [5636213.987243] sr 4:0:0:0: [sr0] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00 [5636213.987248] end_request: I/O error, dev sr0, sector 0 [5636213.987251] Buffer I/O error on device sr0, logical block 0 [5636213.987253] Buffer I/O error on device sr0, logical block 1 [5636213.987260] Buffer I/O error on device sr0, logical block 2 [5636213.987261] Buffer I/O error on device sr0, logical block 3 [5636213.987263] Buffer I/O error on device sr0, logical block 4 [5636213.987265] Buffer I/O error on device sr0, logical block 5 [5636213.987266] Buffer I/O error on device sr0, logical block 6 [5636213.987267] Buffer I/O error on device sr0, logical block 7 [5636213.997003] sr 4:0:0:0: [sr0] Device not ready [5636213.997005] sr 4:0:0:0: [sr0] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [5636213.997007] sr 4:0:0:0: [sr0] Sense Key : Not Ready [current] [5636213.997010] sr 4:0:0:0: [sr0] Add. Sense: Medium not present - tray closed [5636213.997012] sr 4:0:0:0: [sr0] CDB: Read(10): 28 00 00 00 00 00 00 00 02 00 [5636213.997016] end_request: I/O error, dev sr0, sector 0 [5636213.997018] Buffer I/O error on device sr0, logical block 0
Quoted 2 lines
-Mike
Quoted 1 line · Mike Mestnik
Ahhh, as I had suspected I'm not the only one.
Far from it, IO related stalls have come up many times over the years.
Quoted 4 lines
Does fdatasync still cause this problem? I'm sure there must be some way to 'group' applications together that should be allowed to avoid this effect and even insert IO requests during the operation if given a high enough priority.
A lot of work was done, and it improved things a LOT, but beating hell out of your interactive application's home has always been a bad idea, and will likely always be so.
-Mike
Quoted 4 lines · Mike Mestnik
Does fdatasync still cause this problem? I'm sure there must be some way to 'group' applications together that should be allowed to avoid this effect and even insert IO requests during the operation if given a high enough priority.
And btw, that resource prioritization exists now for both CPU and IO resources, look into cgroups and ionice for non-cgroup IO. For the CPU scheduler, you can automate task grouping by enabling SCHED_AUTOGROUP, and some distros use a userland automation solution via systemd. I don't know if systemd does IO group automation, but groups won't help defective spinning media anyway.
-Mike