truncated files
Hi!
Today on one try to hibernate via tuxonice it machine appeared dead. I am not completely sure if it was. But I was in a hurry and had no time to try to ping or SSH it from a different machine and thus I just switched it off hard.
After booting again which worked fine, parts of my KDE configuration appeared broken. Color scheme and window position was lost. And KMail did not know how to sent out a mail anymore, all transports were missing.
Thus I checked my /home filesystem on /dev/sda5 this evening. It appeared absolutely fine with xfs_check and xfs_repair -n from xfsprogs 2.9.8 (grml 2008.11rc).
But comparing some of the config files of KDE with the versions in my backup showed truncated files:
martin@shambhala:~> ls -l .kde-backup-2008-11-22/share/config/kmailrc -rw------- 1 martin martin 247680 22. Nov
18:29 .kde-backup-2008-11-22/share/config/kmailrc
martin@shambhala:~> ls -l .kde-broken-2008-11-25/share/config/kmailrc -rw------- 1 martin martin 116902 25. Nov
21:16 .kde-broken-2008-11-25/share/config/kmailrc
But no hole in it:
martin@shambhala:~>
su -c "xfs_bmap -v .kde-broken-2008-11-25/share/config/kmailrc" Passwort:
.kde-broken-2008-11-25/share/config/kmailrc:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL
0: [0..231]: 22444768..22444999 0 (22444768..22444999) 232
Similar stuff happened on the encfs encrypted KDE user that I use for professional stuff - encfs is layered on top of XFS via FUSE:
shambhala:~ms> ls -l .kde-broken-2008-11-25/share/config/kdeglobals -rw------- 1 ms teamix 4772 25. Nov
20:35 .kde-broken-2008-11-25/share/config/kdeglobals shambhala:~ms> ls -l .kde-backup-2008-11-22/share/config/kdeglobals -rw------- 1 ms teamix 6621 18. Nov
16:13 .kde-backup-2008-11-22/share/config/kdeglobals
Unfortunately I cannot check this one for holes directly as its on encfs:
shambhala:~ms> xfs_bmap -v .kde-broken-2008-11-25/share/config/kdeglobalsxfs_bmap: specified file [".kde-broken-2008-11-25/share/config/kdeglobals"] is not on an XFS filesystem
(Since encfs encrypted pathes as well and the file length may differ a bit, the encrypted file could be difficult to find... well maybe by date.)
And well my basket index was lost (http://basket.kde.org is a note taking application):
shambhala:~ms>
ls -l .kde-backup-2008-11-22/share/apps/basket/baskets/baskets.xml -rw-r--r-- 1 ms teamix 10271 18. Nov
15:03 .kde-backup-2008-11-22/share/apps/basket/baskets/baskets.xml shambhala:~ms>
ls -l .kde-broken-2008-11-25/share/apps/basket/baskets/baskets.xml -rw-r--r-- 1 ms teamix 1905 25. Nov
13:24 .kde-broken-2008-11-25/share/apps/basket/baskets/baskets.xml
But since I found no holes in the other file and AFAIK the truncated file problem has long been solved - anyone has a idea, how this could have happened?
I fixed it by copying ~/.kde/share/config for my private user, and complete ~/.kde for my company user from the backup last weekend. In ~/.kde/share/apps for the private too much has changed since the backup, thus for the moment I left it. I do not miss anything in there right now, but feel a tad bit uncomfortable that files might be truncated in there also and I might only find out later when I already worked with them for some time.
I thought that KDE maybe was writing those files at the moment. But its a bit wide-spread for that. And it must have happened that it wrote to config / app files on two user accounts. Might be the explaination, but I am not completely convinced.
I did not have seen a thing like this since at least a year I think.
This is with:
martin@shambhala:~> cat /proc/version
Linux version 2.6.27.7-tp42-toi-3.0-rc7a (martin@shambhala) (gcc version 4.3.2 (Debian 4.3.2-1) ) #1 PREEMPT Mon Nov 24 11:30:39 CET 2008
I did not find anything cumbersone in /var/log/syslog... no XFS crash or something like this.
If no one has any idea I take this as really bad luck. Might really be that I turned off the machine while quite some KDE files were being written out. I know that some KDE applications make backups prior to overwriting. Maybe something could be improved in KDE applications. If so, I would be interested to hear and I would file bug reports with KDE.
Well actually thats my bet already. XFS appears to be fine.
Ciao,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Martin Steigerwald
Hi!
Today on one try to hibernate via tuxonice it machine appeared dead. I am
^^^^^^^^^
When (not if) suspend to disk/resume fails, you get to keep all the broken pieces of your filesystem. It works most of the time, but it has some fundamentally broken corner cases that you probably just hit....
Quoted 2 lines
Similar stuff happened on the encfs encrypted KDE user that I use for professional stuff - encfs is layered on top of XFS via FUSE:
IIRC, FUSE cannot be suspended safely at all, so expect corruption of FUSE filesystems on a failed suspend.
I've never had a system that suspends reliably (let alone resumes from the suspend) so it's no real surprise that I don't trust suspend to disk....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am Dienstag 25 November 2008 schrieb Dave Chinner:
Quoted 11 lines · Martin Steigerwald
Quoted 4 lines
Hi!
Today on one try to hibernate via tuxonice it machine appeared dead. I am
^^^^^^^^^When (not if) suspend to disk/resume fails, you get to keep all the broken pieces of your filesystem. It works most of the time, but it has some fundamentally broken corner cases that you probably just hit....
Well I use TuxOnIce for a reason! I had uptimes of up to 70 days with it already. And they are usually only interrupted by kernel updates or manual shutdowns. I was never convinced by in-kernel solutions for hibernate.
Quoted 5 lines
Quoted 2 lines
Similar stuff happened on the encfs encrypted KDE user that I use for professional stuff - encfs is layered on top of XFS via FUSE:
IIRC, FUSE cannot be suspended safely at all, so expect corruption of FUSE filesystems on a failed suspend.
Ok, fair enough. My private data isn't on encfs tough.
Quoted 3 lines
I've never had a system that suspends reliably (let alone resumes from the suspend) so it's no real surprise that I don't trust suspend to disk....
Well I take it as bad luck then, especially since there are no hints that XFS had a problem. I am not sure whether the machine really was dead, but I can't reproduce what exactly happened. So thats it.
Ciao,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 13 lines · Martin Steigerwald
Am Dienstag 25 November 2008 schrieb Dave Chinner:
Quoted 11 lines · Martin Steigerwald
Quoted 4 lines
Hi!
Today on one try to hibernate via tuxonice it machine appeared dead. I am
^^^^^^^^^When (not if) suspend to disk/resume fails, you get to keep all the broken pieces of your filesystem. It works most of the time, but it has some fundamentally broken corner cases that you probably just hit....
Quoted 4 lines
Well I use TuxOnIce for a reason! I had uptimes of up to 70 days with it already. And they are usually only interrupted by kernel updates or manual shutdowns. I was never convinced by in-kernel solutions for hibernate.
Sure, though I'm not convinced that TuxOnIce is any better because it still uses the same fundamental design as the in-kernel ones.
Quoted 7 lines
Quoted 3 lines
I've never had a system that suspends reliably (let alone resumes from the suspend) so it's no real surprise that I don't trust suspend to disk....
Well I take it as bad luck then, especially since there are no hints that XFS had a problem. I am not sure whether the machine really was dead, but I can't reproduce what exactly happened. So thats it.
And therein lies the problem. I can't get suspend/resume to work reliably on anything I own, so I can't do anything about problems reported as a result of suspend/resume. Hell, I even considered running linux on my new laptop inside a virtual machine on windows just so I could have functioning suspend/resume....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am Mittwoch 26 November 2008 schrieb Dave Chinner:
Quoted 14 lines · Martin Steigerwald
Quoted 13 lines
Am Dienstag 25 November 2008 schrieb Dave Chinner:
Quoted 11 lines · Martin Steigerwald
Quoted 4 lines
Hi!
Today on one try to hibernate via tuxonice it machine appeared dead. I am
^^^^^^^^^When (not if) suspend to disk/resume fails, you get to keep all the broken pieces of your filesystem. It works most of the time, but it has some fundamentally broken corner cases that you probably just hit....
Quoted 7 lines
Quoted 4 lines
Well I use TuxOnIce for a reason! I had uptimes of up to 70 days with it already. And they are usually only interrupted by kernel updates or manual shutdowns. I was never convinced by in-kernel solutions for hibernate.
Sure, though I'm not convinced that TuxOnIce is any better because it still uses the same fundamental design as the in-kernel ones.
Might be.
But something is fishy here. I had it a second time today. This time I know for sure that the machine freezed hard. Mouse pointer froze and the machine didn't even respond to a ping anymore. Nothing in logs - doesn't surprise me.
I didn't have this issue with 2.6.26, and I also don't think I had it with 2.6.27.5. I will downgrade to 2.6.27.5 now.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
I've never had a system that suspends reliably (let alone resumes from the suspend) so it's no real surprise that I don't trust suspend to disk....
Well I take it as bad luck then, especially since there are no hints that XFS had a problem. I am not sure whether the machine really was dead, but I can't reproduce what exactly happened. So thats it.
Quoted 5 lines
And therein lies the problem. I can't get suspend/resume to work reliably on anything I own, so I can't do anything about problems reported as a result of suspend/resume. Hell, I even considered running linux on my new laptop inside a virtual machine on windows just so I could have functioning suspend/resume....
Then you really had back luck. TuxOnIce works stable for me - at least it did so for a long long while, until my recent kernel.
Ciao,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am Freitag 28 November 2008 schrieb Martin Steigerwald:
Quoted 4 lines
Am Mittwoch 26 November 2008 schrieb Dave Chinner:
Quoted 3 lines · Martin Steigerwald
Quoted 2 lines
Am Dienstag 25 November 2008 schrieb Dave Chinner:
Quoted 1 line
On Tue, Nov 25, 2008 at 10:44:14PM +0100, Martin Steigerwald
Quoted 21 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Hi!
Today on one try to hibernate via tuxonice it machine appeared dead. I am
^^^^^^^^^When (not if) suspend to disk/resume fails, you get to keep all the broken pieces of your filesystem. It works most of the time, but it has some fundamentally broken corner cases that you probably just hit....
Quoted 7 lines
Quoted 4 lines
Well I use TuxOnIce for a reason! I had uptimes of up to 70 days with it already. And they are usually only interrupted by kernel updates or manual shutdowns. I was never convinced by in-kernel solutions for hibernate.
Sure, though I'm not convinced that TuxOnIce is any better because it still uses the same fundamental design as the in-kernel ones.
Might be.
Quoted 7 lines
But something is fishy here. I had it a second time today. This time I know for sure that the machine freezed hard. Mouse pointer froze and the machine didn't even respond to a ping anymore. Nothing in logs - doesn't surprise me.
I didn't have this issue with 2.6.26, and I also don't think I had it with 2.6.27.5. I will downgrade to 2.6.27.5 now.
I wonder about those truncated files nonetheless. As I don't think that KDE is writing config files all the time. Well I might be wrong, but I didn't even change KDE configuration during time of the crash... OTOH XFS uses a in memory inode size and should be safe with the point in time when it writes the size to disk as far as I read here. Well this time at least again the file "kdeglobals" was affected and this file might be written rather often.
Anyway, since I am not even using an official TuxOnIce patch... lets see what the downgrade to 2.6.27.5 brings - no time to really dig deeper into that right now.
Maybe I should update notebook kernels a little less. 2.6.25.3,6,8,10 worked nicer than 2.6.26.5 which has had hangs on USB sound on my T23 Amarok machine, especially after several hibernate cycles.
Or use the Debian distro kernel, but I never had much luck with hibernate with that. Might be worth trying again.
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
hangs with MTRR_SANITIZER? (was: Re: truncated files)
Hi!
CC'd to linux-kernel mailinglist, as that might be something that goes beyond any possible TuxOnIce or XFS issues. I know I am using TuxOnIce which is not part of the mainline kernel. And I am even using an inofficial patch - which I will use again unchanged for the non MTRR_SANITIZER kernel, in order to know whether its the MTRR_SANITIZER thing. And anyway before knowing whether it might be MTRR_SANITIZER related I need to run the non MTRR_SANITIZER kernel for at least a week and have quite some hibernate cycles. If someone else had issues with MTRR_SANITIZER I would like to hear about it. Also if someone thinks I am completely off track on trying to track this down I appreciate a hint.
Links to my posts on xfs and tuxonice-devel mailing lists:
http://oss.sgi.com/pipermail/xfs/2008-November/037399.html
http://lists.tuxonice.net/pipermail/tuxonice-devel/2008-November/000421.html
Am Freitag 28 November 2008 schrieb Martin Steigerwald:
Quoted 36 lines
Am Freitag 28 November 2008 schrieb Martin Steigerwald:
Quoted 4 lines
Am Mittwoch 26 November 2008 schrieb Dave Chinner:
Quoted 3 lines · Martin Steigerwald
Quoted 2 lines
Am Dienstag 25 November 2008 schrieb Dave Chinner:
Quoted 1 line
On Tue, Nov 25, 2008 at 10:44:14PM +0100, Martin Steigerwald
Quoted 21 lines · wrote:
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Hi!
Today on one try to hibernate via tuxonice it machine appeared dead. I am
^^^^^^^^^When (not if) suspend to disk/resume fails, you get to keep all the broken pieces of your filesystem. It works most of the time, but it has some fundamentally broken corner cases that you probably just hit....
Quoted 7 lines
Quoted 4 lines
Well I use TuxOnIce for a reason! I had uptimes of up to 70 days with it already. And they are usually only interrupted by kernel updates or manual shutdowns. I was never convinced by in-kernel solutions for hibernate.
Sure, though I'm not convinced that TuxOnIce is any better because it still uses the same fundamental design as the in-kernel ones.
Might be.
Quoted 7 lines
But something is fishy here. I had it a second time today. This time I know for sure that the machine freezed hard. Mouse pointer froze and the machine didn't even respond to a ping anymore. Nothing in logs - doesn't surprise me.
I didn't have this issue with 2.6.26, and I also don't think I had it with 2.6.27.5. I will downgrade to 2.6.27.5 now.
Quoted 7 lines
I wonder about those truncated files nonetheless. As I don't think that KDE is writing config files all the time. Well I might be wrong, but I didn't even change KDE configuration during time of the crash... OTOH XFS uses a in memory inode size and should be safe with the point in time when it writes the size to disk as far as I read here. Well this time at least again the file "kdeglobals" was affected and this file might be written rather often.
Okay, I thought about this a bit more in my dreams this night it seems. I think it even hangs before starting much of hibernate.
I had this pre-hibernate script:
shambhala:/etc/acpi> bzr cat -r304 hibernate-tuxonice.sh #!/bin/sh
# Änderung der Netzwerkumgebung erkennen und Zeitserver handeln /etc/init.d/ifplugd stop
ifdown eth0/etc/init.d/chrony stop
# Gutnacht
hibernate
# Änderung der Netzwerkumgebung erkennen und Zeitserver handeln /etc/init.d/chrony start/etc/init.d/ifplugd start
Thus its logical that pinging the machine did not work anymore, since its first thing it does is to disable the network in order to detect changes in network environment between snapshot cycles reliably.
Then it froze hard even before the desktop was locked. AFAIR not even the hibernater / suspend LED of my ThinkPad T42 started to blink. Thus I guess it froze way before any serious hibernation work started. It also didn't yet switch to console. And actually somewhere along the line of this it might fail.
I at least have the idea that it could have to do with:
│ CONFIG_MTRR_SANITIZER: │ │ Convert MTRR layout from continuous to discrete, so X drivers can │ add writeback entries. │ │ Can be disabled with disable_mtrr_cleanup on the kernel command line. │ The largest mtrr entry size for a continous block can be set with │ mtrr_chunk_size. │ │ If unsure, say N.
Especially as some earlier description of this config option adds an important detail:
Quoted 6 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+config MTRR_SANITIZER + def_bool y + prompt "MTRR cleanup support" + depends on MTRR + help + Convert MTRR layout from continuous to discrete, so some X
driver
Quoted 4 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ could add WB entries. + + Say N here if you see bootup problems (boot crash, boot hang, + spontaneous reboots).
This one at least sounds interesting. Especially in combintion with the sentence about X drivers. But then it only speaks about boot related issues. And here it hanged shortly after I pressed Fn-F12 to start a snapshot cycle.
shambhala:/etc/acpi> lspci -nn | grep -i vga
01:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV350 [Mobility Radeon 9600 M10] [1002:4e50]
It didn't yet hang with my IBM ThinkPad T23 (SuperSavage) and my workstation at work (newer ATI Radeon). So that might be another hint.
On the first occurence the machine did not respond to user input very early, even before serious hibernating work started, too.
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + could be disabled with disable_mtrr_cleanup. also
mtrr_chunk_size
http://lkml.org/lkml/2008/4/28/685
Thus I am compiling 2.6.27.7 without MTRR_SANITIZER support, but elsewise unchanged. And will test whether those hangs will be gone. It didn't happen with 2.6.27.5 tough. That might just be concurrence or it might hint at those BIOS corruption prevention patch that came in between 2.6.27.5 and 2.6.27.7. But actually I doubt that the BIOS corruption prevention patch is at play here.
To avoid the truncated files problems, I will try this:
shambhala:/etc/acpi> cat hibernate-tuxonice.sh
#!/bin/sh
# Zur Sicherheit gleich am Anfang alle ausstehenden Änderungen schreiben sync
# Änderung der Netzwerkumgebung erkennen und Zeitserver handeln /etc/init.d/ifplugd stop
ifdown eth0/etc/init.d/chrony stop
# Zur Sicherheit hier nochmal alle ausstehenden Änderungen schreiben sync
# Gutnacht
hibernate
# Änderung der Netzwerkumgebung erkennen und Zeitserver handeln /etc/init.d/chrony start/etc/init.d/ifplugd start
Ciao,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: hangs with MTRR_SANITIZER? - no, something else
Am Samstag 29 November 2008 schrieb Martin Steigerwald:
Quoted 13 lines
Hi!
CC'd to linux-kernel mailinglist, as that might be something that goes beyond any possible TuxOnIce or XFS issues. I know I am using TuxOnIce which is not part of the mainline kernel. And I am even using an inofficial patch - which I will use again unchanged for the non
MTRR_SANITIZERkernel, in order to know whether its theMTRR_SANITIZERthing. And anyway before knowing whether it might beMTRR_SANITIZERrelated I need to run the nonMTRR_SANITIZERkernel for at least a week and have quite some hibernate cycles. If someone else had issues withMTRR_SANITIZERI would like to hear about it. Also if someone thinks I am completely off track on trying to track this down I appreciate a hint.
Ok, its not MTRR_SANITIZER. It hung again on hibernate, again before any serious hibernating work has started. I will add debug output to my pre-hibernate script as it might hang already in there, maybe while disabling the network. I want to know whether it hangs before calling the hibernate script or after it. I think I will go for the latest official hibernate patch instead of using the inofficial one, although I am not convinced that it makes much of a difference. Lets see.
The syncs I added to my pre-hibernate seemed to help. KDE configuration is intact. As a safeguard I rsync ~/.kde to a backup directory before hibernating anyway.
Lets see what ideas I have to continue that Sherlock Holmes game ;)
I am puzzled that it only happens on my ThinkPad T42, not on the T23 and neither on the Dell workstation - till now.
Goodnight ;-),
Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Re: [TuxOnIce-devel] hangs with MTRR_SANITIZER? - solved
Am Sonntag, 30. November 2008 schrieb Martin Steigerwald:
Quoted 23 lines
Am Samstag 29 November 2008 schrieb Martin Steigerwald:
Quoted 13 lines
Hi!
CC'd to linux-kernel mailinglist, as that might be something that goes beyond any possible TuxOnIce or XFS issues. I know I am using TuxOnIce which is not part of the mainline kernel. And I am even using an inofficial patch - which I will use again unchanged for the non
MTRR_SANITIZERkernel, in order to know whether its theMTRR_SANITIZERthing. And anyway before knowing whether it might beMTRR_SANITIZERrelated I need to run the nonMTRR_SANITIZERkernel for at least a week and have quite some hibernate cycles. If someone else had issues withMTRR_SANITIZERI would like to hear about it. Also if someone thinks I am completely off track on trying to track this down I appreciate a hint.Ok, its not
MTRR_SANITIZER. It hung again on hibernate, again before any serious hibernating work has started. I will add debug output to my pre-hibernate script as it might hang already in there, maybe while disabling the network. I want to know whether it hangs before calling the hibernate script or after it. I think I will go for the latest official hibernate patch instead of using the inofficial one, although I am not convinced that it makes much of a difference. Lets see.
Quoted 3 lines
The syncs I added to my pre-hibernate seemed to help. KDE configuration is intact. As a safeguard I rsync ~/.kde to a backup directory before hibernating anyway.
Okay... thats solved now.
Conclusions:
1) There was no XFS problem as the sync I added at the beginning of my pre hibernate script did avoid the truncated files the one time I still had the hang. Thus those appear to have been IO in flight.
2) Its not MTRR_SANITIZER as explained above nor any other mainline problem.
3) Instead problems gone, when I replaced the inofficial TuxOnIce rc7a for 2.6.26 to 2.6.27 forward port patch I used[1] with the official but still not officially released current tuxonice for 2.6.27 patch[2].
So sorry for the noise. I just learned to prefer official upstream patches. Whether they'd be officially released or not. Can ask whether they appear to be stable before trying one. ;)
[1]
http://lists.tuxonice.net/pipermail/tuxonice-devel/2008-November/000357.html
[2] http://www.tuxonice.net/downloads/all/current-tuxonice-2.6.27.patch.bz2
Ciao,
Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs